Skip to content

feat(wfp2): token-aware (wfp2) opt-in snippet matching - #129

Open
mscasso-scanoss wants to merge 4 commits into
crc64from
feat/wfp2-integration
Open

mscasso-scanoss wants to merge 4 commits into
crc64from
feat/wfp2-integration

Conversation

@mscasso-scanoss

Copy link
Copy Markdown
Contributor

What

Adds optional wfp2 (token-aware: identifiers/types collapsed, comments removed,
language-aware) snippet matching alongside the classic wfp1, as an opt-in that
leaves default behaviour unchanged. Part of the epic scanoss/roadmap#58 (engine =
scope item #2).

How it works

  • Table detection — initialize_ldb_tables detects an optional wfp2 table
    in the KB (like the path table) and loads oss_wfp2 (wfp2_present).
  • Payload — per file, a wfp2= marker line (empty value, like fh2=/hpsm=)
    after the classic wfp1 snippet lines switches the following N=hash,... lines
    into the wfp2 set, until the next file=.
  • Combined matchmap — the snippet scan accumulates wfp1 and wfp2 into a
    single matchmap (snippet_scan_pass over multiple wfp_source_t). The only
    per-table part is the fetch; add_file_to_matchmap dedups by file id, so hits
    are summed and ranges unified, and the min-hits threshold is applied once over
    the combined evidence. Both WFP parsers handle it: -w (wfp_scan) and -S
    (snippet_scan_stream, used by the API's /scan/snippets).
  • In-engine wfp2 computation — for physical-file scans the engine computes
    wfp2 itself (src/wfp2.c: language detection → comment removal → token-aware
    normalization → winnowing GRAM 20 / WINDOW 40, CRC32C), a C port of go-minr's
    snippetsx. The element/type tables are generated from the Go source.
  • Capability — the engine advertises "wfp2": true|false in its server
    block so the API can gate the opt-in by capability (not just a flag).

Commits

  • feat(wfp2) — opt-in snippet matching: table detection, combined matchmap,
    in-engine wfp2 computation, server-block capability.
  • fix(wfp2) — parse wfp2= blocks in the -S snippet-scan path (the one
    /scan/snippets uses; without it wfp2 lines were misparsed as wfp1).
  • fix(wfp2) — keep the snippet threshold on the classic wfp1 count, not the
    wfp1+wfp2 sum.
  • chore(release) — bump version to 6.2.0-crc64.

Compatibility / validation

  • No opt-in = unchanged: without wfp2 hashes (hash_count2 == 0) the combined
    map is a single wfp1 source. Regression-verified byte-identical vs the base on
    unmodified files; the -S path verified to not pollute when the KB has no wfp2
    table.
  • wfp2 value: a file with renamed identifiers drops wfp1 93% → 22% but the
    combined scan holds at 93% (surviving wfp1 hashes + wfp2 accumulate in the same
    matchmap). Same result via -w and via -S.
  • Memory-safe (valgrind: no leaks/errors attributable to the new code).

Notes

  • Opt-in flag/param naming (roadmap 5.3.3 #58 scope 6) is still TBD; the payload marker
    is fixed as wfp2=.
  • go-minr's DetectLanguage is non-deterministic on score ties
    (scanoss/go-minr#227), which makes the current wfp2 KB internally inconsistent.
    The in-engine port is deterministic but its tie order must be synced with the
    miner once that is fixed and the KB re-mined.

🤖 Generated with Claude Code

mscasso-scanoss and others added 4 commits November 17, 2026 22:41
Add optional wfp2 (token-aware: identifiers/types collapsed, comments
removed, language-aware) snippet matching alongside the classic wfp1,
as an opt-in that leaves default behavior unchanged.

- initialize_ldb_tables detects an optional `wfp2` table in the KB and
  loads oss_wfp2 (sets wfp2_present).
- Parse a `wfp2=` block in the .wfp payload into scan->hashes2/lines2.
- Compute wfp2 fingerprints in-engine for physical-file scans
  (src/wfp2.c: language detection -> comment removal -> token-aware
  normalization -> winnowing, GRAM 20/WINDOW 40, CRC32C), a C port of
  go-minr's snippetsx. File CRC64 hash is handled by oss_file.hash_calc.
- Snippet scan accumulates wfp1 and wfp2 into a single matchmap
  (snippet_scan_pass over multiple wfp_source_t): hits are summed and
  ranges unified per file by add_file_to_matchmap, and the min-hits
  threshold is applied once over the combined evidence. Without wfp2
  hashes the result is byte-identical to the classic wfp1-only scan.
- Advertise the capability as "wfp2": true/false in the server block so
  the API can gate the opt-in by capability.

Note: go-minr/scanoss-go DetectLanguage is non-deterministic on score
ties (scanoss/go-minr#227); the in-engine port is deterministic but its
tie order must be synced with the miner once that is fixed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The /scan/snippets API endpoint (consumed by batchScanner) invokes the
engine via -S (snippet_scan_stream), a separate WFP parser from the -w
(wfp_scan) path. It did not recognise the wfp2= marker, so wfp2 snippet
lines were parsed as classic wfp1 hashes and matched against the wrong
table. Mirror the -w handling: route lines after a wfp2= marker into
scan->hashes2/lines2 so ldb_scan_snippets accumulates both sources.
Without a wfp2 table or wfp2 lines the behaviour is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…wfp1+wfp2 sum

snippet_scan_pass fed adjust_tolerance the summed hash_count across all sources
(wfp1 + wfp2). wfp1 and wfp2 are two fingerprints of the SAME file, so the sum
overstates the file size and inflates min_match_hits / min_match_lines and
tightens range_tolerance above the original (classic) rule. That eats the
detection gain of the combined scan and could even push the combined result
below the classic wfp1 one.

Use the classic wfp1 fingerprint count (sources[0], always oss_wfp) as the
file-size proxy, exactly as before wfp2 existed. wfp2 must not move the
threshold; it only ADDS hits into the same per-file matchmap (hits summed),
which is where combined detection is gained: a file whose wfp1 hits alone fall
short can cross the unchanged bar once wfp2 hits add up.

With a single source this equals the classic count, so classic scans stay
byte-identical (regression-verified). Validated against ossv2_wfp2 on real KB
files mutated with the acceptance mutators: combined (wfp+wfp2) >= classic wfp
in every case, with gains up to +98 points (match.c heavy/extreme 0% -> 98%).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant