From 816c3f791cf1e0cac7ee92b97854c8bc3f1bd856 Mon Sep 17 00:00:00 2001 From: SkyeAv Date: Thu, 5 Nov 2026 12:21:11 -0800 Subject: [PATCH] fix: share one Unicode-aware level-two filter between fullmap build and query Level-two fullmap keys were derived by an ASCII-only filter at build time (rust/src/fullmap.rs level_two) but by a Unicode-aware polars regex at query time (nlp.level_two), so non-ASCII terms could never match at level two while level one stayed unaffected. Both sides now share ONE implementation, matching the level-one contract: - fullmap::level_two keeps Unicode alphanumeric characters plus underscore instead of stripping every non-ASCII byte, and is pub(crate) so nlp.rs can route through it. - A new normalize_terms_l2 batch pyfunction exposes that exact filter to Python (GIL released, rayon, index-aligned, same shape as normalize_terms). - nlp.level_two's default pattern routes through rs.normalize_terms_l2 via map_batches (nulls preserved, one Rust call per column); a custom regex keeps the vectorized polars path with no build-side contract. - Golden pins regenerated deliberately: build_golden loses the three ASCII-stripped level-two twins of non-ASCII names (caf, nave, t -- each level-two form now equals its level-one form) and gains beta-catenin / betacatenin from a new Greek-letter fixture name; the Python test_fullmap_golden mirror updates in lockstep (54 rows / 51 probes). Existing fullmaps must be rebuilt (build-fullmap) after upgrading: their stored level-two keys predate the fix, so non-ASCII terms cannot resolve at level two against them. The on-disk schema tag is unchanged. Fixes #251 --- docs/api/fullmap.md | 4 +-- docs/api/lib.md | 2 +- docs/fullmap.md | 2 +- rust/src/fullmap.rs | 60 +++++++++++++++++++++++++++--------- rust/src/lib.rs | 8 +++-- rust/src/nlp.rs | 53 ++++++++++++++++++++++++++++++- rust/tests/build_golden.rs | 5 ++- rust/tests/common/mod.rs | 5 +-- src/tablassert/nlp.py | 42 +++++++++++++++++++++++-- src/tablassert/rs.pyi | 1 + tests/test_fullmap_golden.py | 25 +++++++++------ tests/test_nlp.py | 42 +++++++++++++++++++++++++ 12 files changed, 209 insertions(+), 40 deletions(-) diff --git a/docs/api/fullmap.md b/docs/api/fullmap.md index 5ae69702..03a81b1e 100644 --- a/docs/api/fullmap.md +++ b/docs/api/fullmap.md @@ -93,7 +93,7 @@ Suffix appended to `col` to locate the `level_two` output column. `resolve()` expects the LazyFrame to already have two NLP columns applied upstream: - `col`: the canonical `level_one` output (cleaned, Unicode-lowercased, Porter2-stemmed, deduplicated, and byte-sorted by token) -- `col + tag`: the `level_two` output (non-word characters removed via `\W+`) +- `col + tag`: the `level_two` output (non-word characters removed by the same compiled Rust filter that derives the build-side level-two keys) The default `"_two"` matches `level_two`'s default tag. @@ -207,7 +207,7 @@ Schema-v5 fullmap databases are rejected by the current resolver. Rebuild existi **`level_two` output** (column: `col + "_two"`): -- All non-word characters (`\W+`) removed from the `level_one` result +- All non-word characters removed from the `level_one` result: Unicode alphanumeric characters and underscores survive, while punctuation, symbols, combining marks, and whitespace are dropped. The default path runs the exact compiled Rust filter the build path stores level-two keys with, so query-side and build-side keys are byte-identical for every input (non-ASCII letters such as accents and Greek symbols are kept, fixing the divergence tracked in issue #251). - Used as fallback when `level_one` produces no match - Preferred for disease names and free text diff --git a/docs/api/lib.md b/docs/api/lib.md index 7d1138dc..008fda6a 100644 --- a/docs/api/lib.md +++ b/docs/api/lib.md @@ -149,7 +149,7 @@ for row in result: ### NLP Processing -`resolve_many()` applies the same level-one normalizer used to build and query the fullmap, writing its result to `{col}`, then applies `level_two` to write `{col}_two`. Level one cleans surrounding quotes and whitespace, lowercases with Unicode rules, tokenizes on whitespace, stems only all-ASCII-alphabetic tokens with English Porter2, passes digit- and punctuation-bearing tokens through, removes duplicate tokens, sorts tokens by UTF-8 byte value, and joins them with one space. Nulls remain null in Python LazyFrame columns; empty values remain empty. Level one is preferred; level two is the fallback for terms with punctuation or special characters. +`resolve_many()` applies the same level-one normalizer used to build and query the fullmap, writing its result to `{col}`, then applies `level_two` to write `{col}_two`. Level one cleans surrounding quotes and whitespace, lowercases with Unicode rules, tokenizes on whitespace, stems only all-ASCII-alphabetic tokens with English Porter2, passes digit- and punctuation-bearing tokens through, removes duplicate tokens, sorts tokens by UTF-8 byte value, and joins them with one space. Level two removes non-word characters through the same compiled Rust filter the build path stores level-two keys with, keeping Unicode alphanumeric characters and underscores. Nulls remain null in Python LazyFrame columns; empty values remain empty. Level one is preferred; level two is the fallback for terms with punctuation or special characters. The resolver requires a schema-v6 fullmap. Existing schema-v5 databases are rejected and must be rebuilt. Preferred-name ranking compares normalized forms under the same level-one rules. diff --git a/docs/fullmap.md b/docs/fullmap.md index 71296fe3..1d822587 100644 --- a/docs/fullmap.md +++ b/docs/fullmap.md @@ -175,7 +175,7 @@ parallel (releasing the GIL, one reader per non-empty shard, re-merged into inpu missing tag raises rather than silently reading incompatible data. Databases built under the older `v1`/`v2`/`v3`/`v4`/`v5` schemas are rejected: there is no automatic schema migration, so a schema bump (the v3 to v4 move to sharded files, the v4 to v5 move to the redb 4 engine, and the v5 to v6 move to normalized level-one keys) requires rebuilding -via `tablassert build-fullmap`. +via `tablassert build-fullmap`. Rebuilding is also required after a key-derivation fix even when the schema tag is unchanged: fullmaps built before the issue #251 level-two correction stripped non-ASCII characters from level-two keys, so a pre-existing database cannot resolve non-ASCII terms at level two until it is rebuilt with a current extension. Readers open every fullmap file READ-ONLY with a SHARED file lock (redb 4 `ReadOnlyDatabase`), so any number of processes can run lookups against the same fullmap concurrently; only a `build-fullmap` rebuild diff --git a/rust/src/fullmap.rs b/rust/src/fullmap.rs index 9ba5710f..ab2325f4 100644 --- a/rust/src/fullmap.rs +++ b/rust/src/fullmap.rs @@ -327,11 +327,19 @@ fn level_one(value: &str) -> Cow<'_, str> { } } -fn level_two(value: &str) -> Cow<'_, str> { - // Fast path: level-two keeps [A-Za-z0-9_]; for an already-lowercase ASCII - // input that is exactly [a-z0-9_], so when every byte qualifies the filter is - // a no-op and we borrow unchanged (the common case post-level_one). Any - // uppercase or non-ASCII byte falls through to the filtering slow path. +/// Derive the level-two key of a (usually level-one) term: keep every Unicode +/// alphanumeric character or underscore, drop everything else. This is THE +/// single level-two key definition: `emit_term` stores build-side keys through +/// it and `nlp::normalize_terms_l2` derives query-side keys through it, so the +/// two sides cannot drift apart (the query side used a Unicode-aware polars +/// regex while the build stripped all non-ASCII, which made non-ASCII terms +/// unmatchable at level two). +pub(crate) fn level_two(value: &str) -> Cow<'_, str> { + // Fast path: for an already-lowercase ASCII input, keeping Unicode word + // characters is exactly keeping [a-z0-9_], so when every byte qualifies the + // filter is a no-op and we borrow unchanged (the common case + // post-level_one). Any uppercase or non-ASCII byte falls through to the + // filtering slow path. if value .bytes() .all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'_') @@ -341,7 +349,7 @@ fn level_two(value: &str) -> Cow<'_, str> { Cow::Owned( value .chars() - .filter(|c| c.is_ascii_alphanumeric() || *c == '_') + .filter(|c| c.is_alphanumeric() || *c == '_') .collect(), ) } @@ -3902,6 +3910,26 @@ mod tests { assert_eq!(level_two("brca-1"), "brca1"); } + #[test] + fn level_two_keeps_non_ascii_word_characters() { + // WHY: the build side and the Python query side must derive the SAME + // level-two key for every level-one string (issue #251). The filter + // keeps Unicode alphanumeric characters (accents, Greek letters, + // non-ASCII digits) plus underscore, and drops punctuation, symbols, + // combining marks, and whitespace -- matching the query side because it + // IS the implementation the query side calls. + assert_eq!(level_two("au café lait"), "aucafélait"); + assert_eq!(level_two("β-catenin"), "βcatenin"); + assert_eq!(level_two("naïve"), "naïve"); + assert_eq!(level_two("p_value"), "p_value"); + // Combining marks are neither alphanumeric nor underscore: a decomposed + // form must collapse exactly like the query side does. + assert_eq!(level_two("cafe\u{301}"), "cafe"); + // Unchanged ASCII behavior: punctuation still splits, empty stays empty. + assert_eq!(level_two("hello-world"), "helloworld"); + assert_eq!(level_two(""), ""); + } + /// `term_shard` is the single routing oracle shared by the writer and the /// reader, so it must be deterministic (same term -> same shard every call, /// or reads would look in the wrong file), agree with the raw xxh64-mask @@ -6259,9 +6287,9 @@ mod tests { Some([(1, 0), (2, 0)].as_slice()), "both spellings must land on one normalized level-one key" ); - // Level-two still composes AFTER level-one: it is the `[a-z0-9_]` strip - // of the NORMALIZED key ("aspirin oral" -> "aspirinoral"), not of the - // raw term (which would have been "oralaspirin"). + // Level-two still composes AFTER level-one: it is the Unicode-word- + // character strip of the NORMALIZED key ("aspirin oral" -> "aspirinoral"), + // not of the raw term (which would have been "oralaspirin"). assert_eq!( terms.get("aspirinoral").map(Vec::as_slice), Some([(1, 0), (2, 0)].as_slice()) @@ -6275,16 +6303,20 @@ mod tests { assert_eq!(terms.len(), 1, "expected only the key `gene`: {terms:?}"); assert!(terms.contains_key("gene")); - // Digit- and punctuation-bearing tokens survive verbatim (never stemmed) - // and non-ASCII folds case but skips stemming, so the level-two strip is - // the only place those characters disappear. + // Digit- and punctuation-bearing tokens survive verbatim (never stemmed). + // Level-two keeps Unicode word characters (é, β) and drops punctuation, + // so "café" derives NO distinct level-two key (it equals its level-one + // form) while "β-catenin" derives the level-two key "βcatenin" -- the + // same key the Python query side derives through + // `nlp::normalize_terms_l2` (issue #251). let mut terms: HashMap> = HashMap::new(); emit_term("TNF-alpha", (4, 0), &mut terms); emit_term("café", (5, 0), &mut terms); - for key in ["tnf-alpha", "tnfalpha", "café", "caf"] { + emit_term("β-catenin", (5, 0), &mut terms); + for key in ["tnf-alpha", "tnfalpha", "café", "β-catenin", "βcatenin"] { assert!(terms.contains_key(key), "missing key {key:?}: {terms:?}"); } - assert_eq!(terms.len(), 4); + assert_eq!(terms.len(), 5); // The gates run BEFORE stemming: "Hypothetical Protein" is rejected on // its cleaned spelling (post-normalization it reads "hypothet protein" diff --git a/rust/src/lib.rs b/rust/src/lib.rs index 8f2ec670..54bf4bcd 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -63,9 +63,10 @@ pub use fullmap::{ }; // Level-one normalization. `normalize_l1` is a plain Rust function consumed // by `fullmap::emit_term` (US-003) and by integration tests; `normalize_terms` -// is the batch pyo3 surface the Python side calls (US-002). Both are -// intentional public exports, not dead code. -pub use nlp::{normalize_l1, normalize_terms}; +// is the batch pyo3 surface the Python side calls (US-002), and +// `normalize_terms_l2` is the batch pyo3 surface for the shared level-two key +// filter (issue #251). All are intentional public exports, not dead code. +pub use nlp::{normalize_l1, normalize_terms, normalize_terms_l2}; #[pymodule] fn rs(module: &Bound<'_, PyModule>) -> PyResult<()> { @@ -84,6 +85,7 @@ fn rs(module: &Bound<'_, PyModule>) -> PyResult<()> { module.add_function(wrap_pyfunction!(fullmap::taxon_allowlist_identity, module)?)?; module.add_function(wrap_pyfunction!(ndjson::dedup_ndjson, module)?)?; module.add_function(wrap_pyfunction!(nlp::normalize_terms, module)?)?; + module.add_function(wrap_pyfunction!(nlp::normalize_terms_l2, module)?)?; module.add_function(wrap_pyfunction!(uuid::namespace_uuid, module)?)?; module.add_function(wrap_pyfunction!(xxh64, module)?)?; module.add_function(wrap_pyfunction!(xxh64_file, module)?)?; diff --git a/rust/src/nlp.rs b/rust/src/nlp.rs index ec2ea6d7..92062cb7 100644 --- a/rust/src/nlp.rs +++ b/rust/src/nlp.rs @@ -116,9 +116,32 @@ pub fn normalize_terms(py: Python<'_>, terms: Vec) -> Vec { }) } +/// Batch pyo3 surface for the fullmap `level_two` key filter: derive the +/// level-two form of a list of (usually level-one) terms, returning an +/// INDEX-ALIGNED list of strings (rayon's `into_par_iter` with `collect` +/// preserves input order). +/// +/// This is the query-side twin of the build-side key derivation: Python's +/// `nlp.level_two` default path calls this, and this calls the exact +/// `fullmap::level_two` filter `emit_term` stores build-side keys through, so +/// query and build level-two keys are byte-identical by construction. Before +/// issue #251 the query side ran a Unicode-aware polars `\\W+` regex while the +/// build stripped every non-ASCII character, so non-ASCII terms could never +/// match at level two. The GIL is released for the whole batch via `py.detach` +/// (the same pattern as `normalize_terms`). +#[pyfunction] +pub fn normalize_terms_l2(py: Python<'_>, terms: Vec) -> Vec { + py.detach(move || { + terms + .into_par_iter() + .map(|term| crate::fullmap::level_two(&term).into_owned()) + .collect() + }) +} + #[cfg(test)] mod tests { - use super::{normalize_l1, normalize_terms}; + use super::{normalize_l1, normalize_terms, normalize_terms_l2}; use pyo3::prelude::*; use std::borrow::Cow; use std::time::Instant; @@ -259,6 +282,34 @@ mod tests { }); } + #[test] + fn normalize_terms_l2_batch_is_index_aligned_and_handles_empty() { + // WHY: the Python level-two path pairs input and output positionally; + // output[i] must be fullmap::level_two(input[i]) -- including "" and an + // empty batch -- and non-ASCII terms must survive with the SAME keys the + // build side stores (issue #251: query and build L2 keys must never + // diverge again, which this pins against literal spellings). + pyo3::Python::initialize(); + pyo3::Python::attach(|py| { + let func = pyo3::wrap_pyfunction!(normalize_terms_l2, py).unwrap(); + let input = vec![ + "au café lait".to_string(), + "β-catenin".to_string(), + "hello-world".to_string(), + String::new(), + ]; + let output: Vec = func.call1((input,)).unwrap().extract().unwrap(); + assert_eq!(output, vec!["aucafélait", "βcatenin", "helloworld", ""]); + + let empty: Vec = func + .call1((Vec::::new(),)) + .unwrap() + .extract() + .unwrap(); + assert!(empty.is_empty()); + }); + } + #[test] fn normalize_terms_performance_one_million_terms() { // WHY: level-one normalization is a hot path for large tabular inputs; diff --git a/rust/tests/build_golden.rs b/rust/tests/build_golden.rs index c77918af..52ea0200 100644 --- a/rust/tests/build_golden.rs +++ b/rust/tests/build_golden.rs @@ -61,7 +61,6 @@ alpha gene|HGNC:1 alphagene|HGNC:1 back\slash|HGNC:3 backslash|HGNC:3 -caf|HGNC:2 café|HGNC:2 doid999|MONDO:1 doid:999|MONDO:1 @@ -95,7 +94,6 @@ mondo:3|MONDO:3 multi|HGNC:10,MONDO:3 name quot|HGNC:3 namequot|HGNC:3 -nave|HGNC:2 naïve|HGNC:2 ncbigene100|HGNC:1 ncbigene101|HGNC:1 @@ -104,8 +102,9 @@ ncbigene:101|HGNC:1 nullnam|HGNC:7 realnam|HGNC:4 share|HGNC:6,MONDO:2 -t|HGNC:3 été|HGNC:3 +β-catenin|HGNC:1 +βcatenin|HGNC:1 "#; // --------------------------------------------------------------------------- diff --git a/rust/tests/common/mod.rs b/rust/tests/common/mod.rs index c2a18c4b..46ed192f 100644 --- a/rust/tests/common/mod.rs +++ b/rust/tests/common/mod.rs @@ -61,7 +61,8 @@ pub struct SourceRow { // --------------------------------------------------------------------------- // Fixed embedded fixture. Deliberately covers: plain ASCII names, unicode names -// (café / naïve), escaped JSON (quotes, backslashes, \uXXXX), alias fields +// (café / naïve), a Greek-letter name (β-Catenin, the issue #251 level-two pin), +// escaped JSON (quotes, backslashes, \uXXXX), alias fields // (id/name/categories/taxon), dead terms (12345/none/nan), equivalent // identifiers, multiple names per row, an empty names array, a null // preferred_name, a row with no names array, and a class row with no @@ -70,7 +71,7 @@ pub struct SourceRow { // --------------------------------------------------------------------------- pub const SYNONYM_LINES: &[&str] = &[ - r#"{"curie":"HGNC:1","preferred_name":"Alpha Gene","names":["Alpha Gene","alpha"],"types":["Gene"],"taxa":["NCBITaxon:9606"]}"#, + r#"{"curie":"HGNC:1","preferred_name":"Alpha Gene","names":["Alpha Gene","alpha","β-Catenin"],"types":["Gene"],"taxa":["NCBITaxon:9606"]}"#, // includes a Greek-letter name (issue #251 L2 pin) r#"{"curie":"HGNC:2","preferred_name":"café","names":["café","naïve"],"types":["Gene"],"taxa":["NCBITaxon:9606"]}"#, r#"{"curie":"HGNC:3","preferred_name":"Esc","names":["\"Quoted Name\"","back\\slash","\u00e9t\u00e9"],"types":["Gene"],"taxa":["NCBITaxon:9606"]}"#, r#"{"id":"MONDO:1","name":"Alias Disease","names":["alias disease"],"categories":["biolink:Disease"],"taxon":["NCBITaxon:0"]}"#, diff --git a/src/tablassert/nlp.py b/src/tablassert/nlp.py index 4e9cc494..8553b9ca 100644 --- a/src/tablassert/nlp.py +++ b/src/tablassert/nlp.py @@ -71,6 +71,30 @@ def level_one(lf: pl.LazyFrame, col: str) -> pl.LazyFrame: return lf.with_columns(expr.alias(col)) +def _level_two_series(values: pl.Series) -> pl.Series: + """Level-two normalize one whole string column through the Rust level-two filter. + + Args: + values: Raw column batch (already cast to string by the caller). + + Returns: + Series with every non-null value replaced by its level-two form, + nulls kept exactly where they were. + + Notes: + Shares the exact batch shape of ``_normalize_terms_series`` (nulls never + reach Rust; one GIL-released Rust call per column instead of per row, + ``map_elements`` is forbidden in this repo) but applies the fullmap + build-side level-two key filter instead of the level-one pipeline, so + the query-side ``{col}_two`` keys are byte-identical to the keys + ``emit_term`` stores at build time (issue #251). + """ + text: pl.Series = values.cast(pl.String) + indices: pl.Series = text.is_not_null().arg_true() + normalized: pl.Series = pl.Series(rs.normalize_terms_l2(text.drop_nulls().to_list()), dtype=pl.String) + return text.scatter(indices, normalized) + + def level_two( lf: pl.LazyFrame, col: str, # pyright: ignore @@ -82,16 +106,28 @@ def level_two( The cleaned values are written to a new column named ``f"{col}{tag}"`` rather than overwriting the source column. + With the default pattern the column is cleaned through the compiled Rust + filter (``rs.normalize_terms_l2``) that derives the fullmap build-side + level-two keys, so query-side and build-side keys are byte-identical by + construction: non-ASCII word characters (accents, Greek letters) survive + while punctuation, symbols, and combining marks are dropped on both sides. + A custom ``regex`` keeps the vectorized Polars path and is a generic text + cleaner with no build-side key contract. + Args: lf: Source LazyFrame. col: Name of the source column to clean. - regex: Pattern whose matches are removed. Defaults to runs of non-word - characters (``\\W+``). + regex: Pattern whose matches are removed. Only the default non-word + pattern (``\\W+``) routes through the shared Rust level-two filter; + any other pattern is applied with Polars directly. tag: Suffix appended to ``col`` to form the output column name. Returns: LazyFrame with the new tagged column added. """ - expr: pl.Expr = pl.col(col).str.replace_all(regex, "") + if regex == r"\W+": + expr: pl.Expr = pl.col(col).cast(pl.String).map_batches(_level_two_series, return_dtype=pl.String) + else: + expr = pl.col(col).str.replace_all(regex, "") col: str = col + tag return lf.with_columns(expr.alias(col)) diff --git a/src/tablassert/rs.pyi b/src/tablassert/rs.pyi index 86453135..c0279c64 100644 --- a/src/tablassert/rs.pyi +++ b/src/tablassert/rs.pyi @@ -26,6 +26,7 @@ def hydrate_sources(db: Path) -> list[str]: ... def lookup_fullmap_terms(db: Path, terms: list[str], return_format: str = "rows") -> list[dict[str, Any]]: ... def namespace_uuid(domain: str, values: list[str]) -> str: ... def normalize_terms(terms: list[str]) -> list[str]: ... +def normalize_terms_l2(terms: list[str]) -> list[str]: ... def taxon_allowlist_identity(taxon_allowlist: list[int]) -> str: ... def xxh64(data: str) -> str: ... def xxh64_file(path: str) -> str: ... diff --git a/tests/test_fullmap_golden.py b/tests/test_fullmap_golden.py index 80a9137c..f24db791 100644 --- a/tests/test_fullmap_golden.py +++ b/tests/test_fullmap_golden.py @@ -43,12 +43,13 @@ def class_row(curie: str, equivalents: list[str]) -> dict[str, Any]: # Fixed embedded fixture — logically identical to rust/tests/build_golden.rs. -# Covers plain ASCII, unicode (café/naïve/été), escaped JSON (quotes/backslash/ +# Covers plain ASCII, unicode (café/naïve/été), a Greek-letter name (β-Catenin, +# the issue #251 level-two pin), escaped JSON (quotes/backslash/ # \uXXXX), alias fields (id/name/categories/taxon), dead terms (12345/none/nan), # equivalent identifiers, multiple names per row, an empty names array, a null # preferred_name, a row with no names array, and a class row with no equivalents. SYNONYMS: list[dict[str, Any]] = [ - synonym_row("HGNC:1", "Alpha Gene", ["Alpha Gene", "alpha"], "Gene"), + synonym_row("HGNC:1", "Alpha Gene", ["Alpha Gene", "alpha", "β-Catenin"], "Gene"), synonym_row("HGNC:2", "café", ["café", "naïve"], "Gene"), synonym_row("HGNC:3", "Esc", ['"Quoted Name"', "back\\slash", "été"], "Gene"), {"id": "MONDO:1", "name": "Alias Disease", "names": ["alias disease"], "categories": ["biolink:Disease"], "taxon": ["NCBITaxon:0"]}, @@ -78,6 +79,12 @@ def class_row(curie: str, equivalents: list[str]) -> dict[str, Any]: # re-derived independently from the RAW fixture spellings through # `rs.normalize_terms` (the same call the Python query side makes) and # cross-checked against the regenerated Rust golden, not copied from a build. +# Issue #251 (one shared level-two filter) then removed the three ASCII-stripped +# level-two twins of non-ASCII names ("caf", "nave", "t": the filter now keeps +# the Unicode letters, so each level-two form equals its level-one form and +# stores no extra key) and added "β-catenin" / "βcatenin" for the fixture's new +# Greek-letter name, each re-derived through `rs.normalize_terms` / +# `rs.normalize_terms_l2`. PROBES: list[str] = [ "alia diseas", "aliadiseas", @@ -86,7 +93,6 @@ def class_row(curie: str, equivalents: list[str]) -> dict[str, Any]: "alphagene", "back\\slash", "backslash", - "caf", "café", "doid999", "doid:999", @@ -120,7 +126,6 @@ def class_row(curie: str, equivalents: list[str]) -> dict[str, Any]: "multi", "name quot", "namequot", - "nave", "naïve", "ncbigene100", "ncbigene101", @@ -129,8 +134,9 @@ def class_row(curie: str, equivalents: list[str]) -> dict[str, Any]: "nullnam", "realnam", "share", - "t", "été", + "β-catenin", + "βcatenin", ] # The pinned hydrated lookup output (SOURCE_VERSION asserted separately). @@ -143,7 +149,6 @@ def class_row(curie: str, equivalents: list[str]) -> dict[str, Any]: {"term": "alphagene", "CURIE": "HGNC:1", "PREFERRED_NAME": "Alpha Gene", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, {"term": "back\\slash", "CURIE": "HGNC:3", "PREFERRED_NAME": "Esc", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, {"term": "backslash", "CURIE": "HGNC:3", "PREFERRED_NAME": "Esc", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, - {"term": "caf", "CURIE": "HGNC:2", "PREFERRED_NAME": "café", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, {"term": "café", "CURIE": "HGNC:2", "PREFERRED_NAME": "café", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, {"term": "doid999", "CURIE": "MONDO:1", "PREFERRED_NAME": "Alias Disease", "CATEGORY_NAME": "Disease", "TAXON_ID": 0, "SOURCE_NAME": "SRC"}, {"term": "doid:999", "CURIE": "MONDO:1", "PREFERRED_NAME": "Alias Disease", "CATEGORY_NAME": "Disease", "TAXON_ID": 0, "SOURCE_NAME": "SRC"}, @@ -178,7 +183,6 @@ def class_row(curie: str, equivalents: list[str]) -> dict[str, Any]: {"term": "multi", "CURIE": "MONDO:3", "PREFERRED_NAME": "Multi B", "CATEGORY_NAME": "Disease", "TAXON_ID": 0, "SOURCE_NAME": "SRC"}, {"term": "name quot", "CURIE": "HGNC:3", "PREFERRED_NAME": "Esc", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, {"term": "namequot", "CURIE": "HGNC:3", "PREFERRED_NAME": "Esc", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, - {"term": "nave", "CURIE": "HGNC:2", "PREFERRED_NAME": "café", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, {"term": "naïve", "CURIE": "HGNC:2", "PREFERRED_NAME": "café", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, {"term": "ncbigene100", "CURIE": "HGNC:1", "PREFERRED_NAME": "Alpha Gene", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, {"term": "ncbigene101", "CURIE": "HGNC:1", "PREFERRED_NAME": "Alpha Gene", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, @@ -188,8 +192,9 @@ def class_row(curie: str, equivalents: list[str]) -> dict[str, Any]: {"term": "realnam", "CURIE": "HGNC:4", "PREFERRED_NAME": "Dead", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, {"term": "share", "CURIE": "HGNC:6", "PREFERRED_NAME": "Shared Hit", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, {"term": "share", "CURIE": "MONDO:2", "PREFERRED_NAME": "Shared Disease", "CATEGORY_NAME": "Disease", "TAXON_ID": 0, "SOURCE_NAME": "SRC"}, - {"term": "t", "CURIE": "HGNC:3", "PREFERRED_NAME": "Esc", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, {"term": "été", "CURIE": "HGNC:3", "PREFERRED_NAME": "Esc", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, + {"term": "β-catenin", "CURIE": "HGNC:1", "PREFERRED_NAME": "Alpha Gene", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, + {"term": "βcatenin", "CURIE": "HGNC:1", "PREFERRED_NAME": "Alpha Gene", "CATEGORY_NAME": "Gene", "TAXON_ID": 9606, "SOURCE_NAME": "SRC"}, ] EXPECTED_CURIES: list[str] = [ @@ -254,8 +259,8 @@ def test_golden_lookup_is_pinned(golden_db: Path) -> None: assert all(r["SOURCE_VERSION"] == source_version for r in rows) canon: list[dict[str, object]] = _canonical(rows) - assert len(canon) == len(GOLDEN_ROWS) == 55 - assert len({r["term"] for r in canon}) == len(PROBES) == 52 # every probe resolved + assert len(canon) == len(GOLDEN_ROWS) == 54 + assert len({r["term"] for r in canon}) == len(PROBES) == 51 # every probe resolved assert canon == GOLDEN_ROWS, "fullmap lookup output diverged from the pinned golden" diff --git a/tests/test_nlp.py b/tests/test_nlp.py index a39389b2..bf8cda5c 100644 --- a/tests/test_nlp.py +++ b/tests/test_nlp.py @@ -7,6 +7,7 @@ import polars as pl +from tablassert import rs from tablassert.nlp import level_one, level_two GOLDEN_FIXTURE: Path = Path(__file__).resolve().parents[1] / "rust" / "tests" / "fixtures" / "nlp_golden.tsv" @@ -126,6 +127,47 @@ def test_level_two_custom_tag() -> None: assert result["name_clean"].to_list() == ["helloworld"] +def test_level_two_non_ascii_matches_rust_build_side() -> None: + """level two keeps non-ASCII word characters exactly like the build side. + + WHY: fullmap level-two keys were derived by an ASCII-only filter at build + time but a Unicode-aware regex at query time, so non-ASCII terms could + never match at level two (issue #251). Both sides now share one Rust + filter; this pins the issue's reproduction table on the query path. + """ + lf: pl.LazyFrame = pl.DataFrame({"name": ["au café lait", "β-catenin"]}).lazy() + result: pl.DataFrame = level_two(lf, "name").collect() + assert result["name_two"].to_list() == ["aucafélait", "βcatenin"] + + +def test_level_two_pipeline_matches_rust_batch() -> None: + """The level-one then level-two query pipeline equals the Rust batch filter. + + WHY: mirrors the issue's reproduction command end to end: the strings the + query side derives from a column must equal what ``rs.normalize_terms_l2`` + derives from the level-one forms, or fullmap level-two lookups silently + miss. + """ + lf: pl.LazyFrame = pl.DataFrame({"t": ["Café au lait", "β-catenin"]}).lazy() + result: pl.DataFrame = level_two(level_one(lf, "t"), "t").collect() + expected: list[str] = rs.normalize_terms_l2(["au café lait", "β-catenin"]) + assert result["t_two"].to_list() == ["aucafélait", "βcatenin"] + assert result["t_two"].to_list() == expected + + +def test_level_two_keeps_nulls_and_empty_through_rust_path() -> None: + """level two keeps nulls in place and empties empty strings. + + WHY: the default path round-trips through Rust (nulls never reach it), so + null positions and blank cells must come back exactly where they went in, + or term alignment across the ``{col}`` and ``{col}_two`` columns would + silently corrupt downstream lookups. + """ + lf: pl.LazyFrame = pl.DataFrame({"name": ["hello world", None, "a-b", ""]}).lazy() + result: pl.DataFrame = level_two(lf, "name").collect() + assert result["name_two"].to_list() == ["helloworld", None, "ab", ""] + + def test_level_one_token_order_insensitive() -> None: """level one canonicalizes token order.