Repository navigation
fix: derive L2 fullmap keys through one shared Unicode-aware filter - #252
Merged
Merged
Conversation
…nd query Level-two fullmap keys were derived by an ASCII-only filter at build time (rust/src/fullmap.rs level_two) but by a Unicode-aware polars regex at query time (nlp.level_two), so non-ASCII terms could never match at level two while level one stayed unaffected. Both sides now share ONE implementation, matching the level-one contract: - fullmap::level_two keeps Unicode alphanumeric characters plus underscore instead of stripping every non-ASCII byte, and is pub(crate) so nlp.rs can route through it. - A new normalize_terms_l2 batch pyfunction exposes that exact filter to Python (GIL released, rayon, index-aligned, same shape as normalize_terms). - nlp.level_two's default pattern routes through rs.normalize_terms_l2 via map_batches (nulls preserved, one Rust call per column); a custom regex keeps the vectorized polars path with no build-side contract. - Golden pins regenerated deliberately: build_golden loses the three ASCII-stripped level-two twins of non-ASCII names (caf, nave, t -- each level-two form now equals its level-one form) and gains beta-catenin / betacatenin from a new Greek-letter fixture name; the Python test_fullmap_golden mirror updates in lockstep (54 rows / 51 probes). Existing fullmaps must be rebuilt (build-fullmap) after upgrading: their stored level-two keys predate the fix, so non-ASCII terms cannot resolve at level two against them. The on-disk schema tag is unchanged. Fixes #251
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
SkyeAv
added a commit
that referenced
this pull request
Nov 9, 2026
Cut 22.3.0 and bump the package version in pyproject.toml, CITATION.cff, and uv.lock. Minor: add mechanical patch/minor/major release targets (#250) and share one Unicode-aware level-two filter between fullmap build and query paths, fixing non-ASCII term resolution (#252). Existing fullmaps must be rebuilt with build-fullmap after upgrading; the on-disk schema tag is unchanged. The real-checkout bump test now derives its expected patch version from pyproject.toml instead of pinning 22.2.2. Changelog: - Added the 22.3.0 - 2026-11-09 section covering #250 and #252 with PR links and the fullmap rebuild requirement. Docs: added the PyPI downloads badge to README.md. Testing: - make check -> Ruff, formatting, Pyright, and Clippy passed; Python 1355 passed, 1 skipped; Rust 158 passed, 1 ignored. - uv lock --check -> resolved 139 packages. - uv run mkdocs build --strict -> documentation built. - npx markdownlint-cli2 -> 0 issues in 24 files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Level-two fullmap keys were derived by an ASCII-only filter at build time but by a Unicode-aware polars regex at query time, so non-ASCII terms could never match at level two while level one stayed unaffected. Both sides now share one implementation, mirroring the single-source-of-truth contract level one already has.
Shared Filter
fullmap::level_twonow keeps Unicode alphanumeric characters plus underscore (c.is_alphanumeric() || c == '_') instead of stripping every non-ASCII byte, and ispub(crate)so the normalizer module can route through it.normalize_terms_l2pyfunction inrust/src/nlp.rsexposes that exact filter (GIL released viapy.detach, rayon fan-out, index-aligned, same shape asnormalize_terms), registered inrust/src/lib.rsand declared insrc/tablassert/rs.pyi.nlp.level_two's default pattern routes throughrs.normalize_terms_l2via the samemap_batcheswhole-column shapelevel_oneuses (nulls never reach Rust, one GIL-released call per column); a customregexkeeps the vectorized polars path and is documented as a generic cleaner with no build-side key contract.Golden Regeneration
build_golden: loses the three ASCII-stripped level-two twins of non-ASCII names (caf,nave,t; each level-two form now equals its level-one form and stores no extra key) and gainsβ-catenin/βcateninfrom a new Greek-letter fixture name inrust/tests/common/mod.rs.tests/test_fullmap_golden.pyupdates in lockstep (54 golden rows / 51 probes) and gains the issue's reproduction table astests/test_nlp.pyparity tests againstrs.normalize_terms_l2.Docs
docs/api/fullmap.md,docs/api/lib.md: thelevel_twosections now describe the shared compiled Rust filter instead of the\W+polars regex.docs/fullmap.md: notes that fullmaps built before this change must be rebuilt (build-fullmap) for non-ASCII level-two lookups; the on-disk schema tag is unchanged.Testing
level_two(level_one(...))andrs.normalize_terms_l2both returnaucafélait/βcateninforCafé au lait/β-catenin.make check-> ruff clean, pyright0 errors, pytest1355 passed, 1 skipped(the env-gated config-corpus sweep),cargo test129 + 11 + 17 + 1pass across lib/integration suites,clippy -D warningsclean.uv run mkdocs build --strict-> built;markdownlint-cli2->0 issues.Fixes #251