Repository navigation
feat(wfp2): token-aware (wfp2) opt-in snippet matching - #129
Open
mscasso-scanoss wants to merge 4 commits into
Open
mscasso-scanoss wants to merge 4 commits into
mscasso-scanoss wants to merge 4 commits into
Conversation
Add optional wfp2 (token-aware: identifiers/types collapsed, comments removed, language-aware) snippet matching alongside the classic wfp1, as an opt-in that leaves default behavior unchanged. - initialize_ldb_tables detects an optional `wfp2` table in the KB and loads oss_wfp2 (sets wfp2_present). - Parse a `wfp2=` block in the .wfp payload into scan->hashes2/lines2. - Compute wfp2 fingerprints in-engine for physical-file scans (src/wfp2.c: language detection -> comment removal -> token-aware normalization -> winnowing, GRAM 20/WINDOW 40, CRC32C), a C port of go-minr's snippetsx. File CRC64 hash is handled by oss_file.hash_calc. - Snippet scan accumulates wfp1 and wfp2 into a single matchmap (snippet_scan_pass over multiple wfp_source_t): hits are summed and ranges unified per file by add_file_to_matchmap, and the min-hits threshold is applied once over the combined evidence. Without wfp2 hashes the result is byte-identical to the classic wfp1-only scan. - Advertise the capability as "wfp2": true/false in the server block so the API can gate the opt-in by capability. Note: go-minr/scanoss-go DetectLanguage is non-deterministic on score ties (scanoss/go-minr#227); the in-engine port is deterministic but its tie order must be synced with the miner once that is fixed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The /scan/snippets API endpoint (consumed by batchScanner) invokes the engine via -S (snippet_scan_stream), a separate WFP parser from the -w (wfp_scan) path. It did not recognise the wfp2= marker, so wfp2 snippet lines were parsed as classic wfp1 hashes and matched against the wrong table. Mirror the -w handling: route lines after a wfp2= marker into scan->hashes2/lines2 so ldb_scan_snippets accumulates both sources. Without a wfp2 table or wfp2 lines the behaviour is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…wfp1+wfp2 sum snippet_scan_pass fed adjust_tolerance the summed hash_count across all sources (wfp1 + wfp2). wfp1 and wfp2 are two fingerprints of the SAME file, so the sum overstates the file size and inflates min_match_hits / min_match_lines and tightens range_tolerance above the original (classic) rule. That eats the detection gain of the combined scan and could even push the combined result below the classic wfp1 one. Use the classic wfp1 fingerprint count (sources[0], always oss_wfp) as the file-size proxy, exactly as before wfp2 existed. wfp2 must not move the threshold; it only ADDS hits into the same per-file matchmap (hits summed), which is where combined detection is gained: a file whose wfp1 hits alone fall short can cross the unchanged bar once wfp2 hits add up. With a single source this equals the classic count, so classic scans stay byte-identical (regression-verified). Validated against ossv2_wfp2 on real KB files mutated with the acceptance mutators: combined (wfp+wfp2) >= classic wfp in every case, with gains up to +98 points (match.c heavy/extreme 0% -> 98%). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds optional wfp2 (token-aware: identifiers/types collapsed, comments removed,
language-aware) snippet matching alongside the classic wfp1, as an opt-in that
leaves default behaviour unchanged. Part of the epic scanoss/roadmap#58 (engine =
scope item #2).
How it works
initialize_ldb_tablesdetects an optionalwfp2tablein the KB (like the
pathtable) and loadsoss_wfp2(wfp2_present).wfp2=marker line (empty value, likefh2=/hpsm=)after the classic wfp1 snippet lines switches the following
N=hash,...linesinto the wfp2 set, until the next
file=.single matchmap (
snippet_scan_passover multiplewfp_source_t). The onlyper-table part is the fetch;
add_file_to_matchmapdedups by file id, so hitsare summed and ranges unified, and the min-hits threshold is applied once over
the combined evidence. Both WFP parsers handle it:
-w(wfp_scan) and-S(
snippet_scan_stream, used by the API's/scan/snippets).wfp2 itself (
src/wfp2.c: language detection → comment removal → token-awarenormalization → winnowing GRAM 20 / WINDOW 40, CRC32C), a C port of go-minr's
snippetsx. The element/type tables are generated from the Go source.
"wfp2": true|falsein itsserverblock so the API can gate the opt-in by capability (not just a flag).
Commits
feat(wfp2)— opt-in snippet matching: table detection, combined matchmap,in-engine wfp2 computation, server-block capability.
fix(wfp2)— parsewfp2=blocks in the-Ssnippet-scan path (the one/scan/snippetsuses; without it wfp2 lines were misparsed as wfp1).fix(wfp2)— keep the snippet threshold on the classic wfp1 count, not thewfp1+wfp2 sum.
chore(release)— bump version to 6.2.0-crc64.Compatibility / validation
hash_count2 == 0) the combinedmap is a single wfp1 source. Regression-verified byte-identical vs the base on
unmodified files; the
-Spath verified to not pollute when the KB has no wfp2table.
combined scan holds at 93% (surviving wfp1 hashes + wfp2 accumulate in the same
matchmap). Same result via
-wand via-S.Notes
is fixed as
wfp2=.DetectLanguageis non-deterministic on score ties(scanoss/go-minr#227), which makes the current wfp2 KB internally inconsistent.
The in-engine port is deterministic but its tie order must be synced with the
miner once that is fixed and the KB re-mined.
🤖 Generated with Claude Code