fix: derive L2 fullmap keys through one shared Unicode-aware filter

docs/api/fullmap.md

@@ -93,7 +93,7 @@ Suffix appended to `col` to locate the `level_two` output column.
 `resolve()` expects the LazyFrame to already have two NLP columns applied upstream:
 
 - `col`: the canonical `level_one` output (cleaned, Unicode-lowercased, Porter2-stemmed, deduplicated, and byte-sorted by token)
-- `col + tag`: the `level_two` output (non-word characters removed via `\W+`)
+- `col + tag`: the `level_two` output (non-word characters removed by the same compiled Rust filter that derives the build-side level-two keys)
 
 The default `"_two"` matches `level_two`'s default tag.
 
@@ -207,7 +207,7 @@ Schema-v5 fullmap databases are rejected by the current resolver. Rebuild existi
 
 **`level_two` output** (column: `col + "_two"`):
 
-- All non-word characters (`\W+`) removed from the `level_one` result
+- All non-word characters removed from the `level_one` result: Unicode alphanumeric characters and underscores survive, while punctuation, symbols, combining marks, and whitespace are dropped. The default path runs the exact compiled Rust filter the build path stores level-two keys with, so query-side and build-side keys are byte-identical for every input (non-ASCII letters such as accents and Greek symbols are kept, fixing the divergence tracked in issue #251).
 - Used as fallback when `level_one` produces no match
 - Preferred for disease names and free text
 

docs/api/lib.md

@@ -149,7 +149,7 @@ for row in result:
 
 ### NLP Processing
 
-`resolve_many()` applies the same level-one normalizer used to build and query the fullmap, writing its result to `{col}`, then applies `level_two` to write `{col}_two`. Level one cleans surrounding quotes and whitespace, lowercases with Unicode rules, tokenizes on whitespace, stems only all-ASCII-alphabetic tokens with English Porter2, passes digit- and punctuation-bearing tokens through, removes duplicate tokens, sorts tokens by UTF-8 byte value, and joins them with one space. Nulls remain null in Python LazyFrame columns; empty values remain empty. Level one is preferred; level two is the fallback for terms with punctuation or special characters.
+`resolve_many()` applies the same level-one normalizer used to build and query the fullmap, writing its result to `{col}`, then applies `level_two` to write `{col}_two`. Level one cleans surrounding quotes and whitespace, lowercases with Unicode rules, tokenizes on whitespace, stems only all-ASCII-alphabetic tokens with English Porter2, passes digit- and punctuation-bearing tokens through, removes duplicate tokens, sorts tokens by UTF-8 byte value, and joins them with one space. Level two removes non-word characters through the same compiled Rust filter the build path stores level-two keys with, keeping Unicode alphanumeric characters and underscores. Nulls remain null in Python LazyFrame columns; empty values remain empty. Level one is preferred; level two is the fallback for terms with punctuation or special characters.
 
 The resolver requires a schema-v6 fullmap. Existing schema-v5 databases are rejected and must be rebuilt. Preferred-name ranking compares normalized forms under the same level-one rules.
 

docs/fullmap.md

@@ -175,7 +175,7 @@ parallel (releasing the GIL, one reader per non-empty shard, re-merged into inpu
 missing tag raises rather than silently reading incompatible data. Databases built under the older
 `v1`/`v2`/`v3`/`v4`/`v5` schemas are rejected: there is no automatic schema migration, so a schema bump
 (the v3 to v4 move to sharded files, the v4 to v5 move to the redb 4 engine, and the v5 to v6 move to normalized level-one keys) requires rebuilding
-via `tablassert build-fullmap`.
+via `tablassert build-fullmap`. Rebuilding is also required after a key-derivation fix even when the schema tag is unchanged: fullmaps built before the issue #251 level-two correction stripped non-ASCII characters from level-two keys, so a pre-existing database cannot resolve non-ASCII terms at level two until it is rebuilt with a current extension.
 
 Readers open every fullmap file READ-ONLY with a SHARED file lock (redb 4 `ReadOnlyDatabase`), so any
 number of processes can run lookups against the same fullmap concurrently; only a `build-fullmap` rebuild

rust/src/fullmap.rs

@@ -327,11 +327,19 @@ fn level_one(value: &str) -> Cow<'_, str> {
     }
 }
 
-fn level_two(value: &str) -> Cow<'_, str> {
-    // Fast path: level-two keeps [A-Za-z0-9_]; for an already-lowercase ASCII
-    // input that is exactly [a-z0-9_], so when every byte qualifies the filter is
-    // a no-op and we borrow unchanged (the common case post-level_one).  Any
-    // uppercase or non-ASCII byte falls through to the filtering slow path.
+/// Derive the level-two key of a (usually level-one) term: keep every Unicode
+/// alphanumeric character or underscore, drop everything else.  This is THE
+/// single level-two key definition: `emit_term` stores build-side keys through
+/// it and `nlp::normalize_terms_l2` derives query-side keys through it, so the
+/// two sides cannot drift apart (the query side used a Unicode-aware polars
+/// regex while the build stripped all non-ASCII, which made non-ASCII terms
+/// unmatchable at level two).
+pub(crate) fn level_two(value: &str) -> Cow<'_, str> {
+    // Fast path: for an already-lowercase ASCII input, keeping Unicode word
+    // characters is exactly keeping [a-z0-9_], so when every byte qualifies the
+    // filter is a no-op and we borrow unchanged (the common case
+    // post-level_one).  Any uppercase or non-ASCII byte falls through to the
+    // filtering slow path.
     if value
         .bytes()
         .all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'_')
@@ -341,7 +349,7 @@ fn level_two(value: &str) -> Cow<'_, str> {
         Cow::Owned(
             value
                 .chars()
-                .filter(|c| c.is_ascii_alphanumeric() || *c == '_')
+                .filter(|c| c.is_alphanumeric() || *c == '_')
                 .collect(),
         )
     }
@@ -3902,6 +3910,26 @@ mod tests {
         assert_eq!(level_two("brca-1"), "brca1");
     }
 
+    #[test]
+    fn level_two_keeps_non_ascii_word_characters() {
+        // WHY: the build side and the Python query side must derive the SAME
+        // level-two key for every level-one string (issue #251). The filter
+        // keeps Unicode alphanumeric characters (accents, Greek letters,
+        // non-ASCII digits) plus underscore, and drops punctuation, symbols,
+        // combining marks, and whitespace -- matching the query side because it
+        // IS the implementation the query side calls.
+        assert_eq!(level_two("au café lait"), "aucafélait");
+        assert_eq!(level_two("β-catenin"), "βcatenin");
+        assert_eq!(level_two("naïve"), "naïve");
+        assert_eq!(level_two("p_value"), "p_value");
+        // Combining marks are neither alphanumeric nor underscore: a decomposed
+        // form must collapse exactly like the query side does.
+        assert_eq!(level_two("cafe\u{301}"), "cafe");
+        // Unchanged ASCII behavior: punctuation still splits, empty stays empty.
+        assert_eq!(level_two("hello-world"), "helloworld");
+        assert_eq!(level_two(""), "");
+    }
+
     /// `term_shard` is the single routing oracle shared by the writer and the
     /// reader, so it must be deterministic (same term -> same shard every call,
     /// or reads would look in the wrong file), agree with the raw xxh64-mask
@@ -6259,9 +6287,9 @@ mod tests {
             Some([(1, 0), (2, 0)].as_slice()),
             "both spellings must land on one normalized level-one key"
         );
-        // Level-two still composes AFTER level-one: it is the `[a-z0-9_]` strip
-        // of the NORMALIZED key ("aspirin oral" -> "aspirinoral"), not of the
-        // raw term (which would have been "oralaspirin").
+        // Level-two still composes AFTER level-one: it is the Unicode-word-
+        // character strip of the NORMALIZED key ("aspirin oral" -> "aspirinoral"),
+        // not of the raw term (which would have been "oralaspirin").
         assert_eq!(
             terms.get("aspirinoral").map(Vec::as_slice),
             Some([(1, 0), (2, 0)].as_slice())
@@ -6275,16 +6303,20 @@ mod tests {
         assert_eq!(terms.len(), 1, "expected only the key `gene`: {terms:?}");
         assert!(terms.contains_key("gene"));
 
-        // Digit- and punctuation-bearing tokens survive verbatim (never stemmed)
-        // and non-ASCII folds case but skips stemming, so the level-two strip is
-        // the only place those characters disappear.
+        // Digit- and punctuation-bearing tokens survive verbatim (never stemmed).
+        // Level-two keeps Unicode word characters (é, β) and drops punctuation,
+        // so "café" derives NO distinct level-two key (it equals its level-one
+        // form) while "β-catenin" derives the level-two key "βcatenin" -- the
+        // same key the Python query side derives through
+        // `nlp::normalize_terms_l2` (issue #251).
         let mut terms: HashMap<String, Vec<(u32, u8)>> = HashMap::new();
         emit_term("TNF-alpha", (4, 0), &mut terms);
         emit_term("café", (5, 0), &mut terms);
-        for key in ["tnf-alpha", "tnfalpha", "café", "caf"] {
+        emit_term("β-catenin", (5, 0), &mut terms);
+        for key in ["tnf-alpha", "tnfalpha", "café", "β-catenin", "βcatenin"] {
             assert!(terms.contains_key(key), "missing key {key:?}: {terms:?}");
         }
-        assert_eq!(terms.len(), 4);
+        assert_eq!(terms.len(), 5);
 
         // The gates run BEFORE stemming: "Hypothetical Protein" is rejected on
         // its cleaned spelling (post-normalization it reads "hypothet protein"

rust/src/lib.rs

@@ -63,9 +63,10 @@ pub use fullmap::{
 };
 // Level-one normalization.  `normalize_l1` is a plain Rust function consumed
 // by `fullmap::emit_term` (US-003) and by integration tests; `normalize_terms`
-// is the batch pyo3 surface the Python side calls (US-002).  Both are
-// intentional public exports, not dead code.
-pub use nlp::{normalize_l1, normalize_terms};
+// is the batch pyo3 surface the Python side calls (US-002), and
+// `normalize_terms_l2` is the batch pyo3 surface for the shared level-two key
+// filter (issue #251).  All are intentional public exports, not dead code.
+pub use nlp::{normalize_l1, normalize_terms, normalize_terms_l2};
 
 #[pymodule]
 fn rs(module: &Bound<'_, PyModule>) -> PyResult<()> {
@@ -84,6 +85,7 @@ fn rs(module: &Bound<'_, PyModule>) -> PyResult<()> {
     module.add_function(wrap_pyfunction!(fullmap::taxon_allowlist_identity, module)?)?;
     module.add_function(wrap_pyfunction!(ndjson::dedup_ndjson, module)?)?;
     module.add_function(wrap_pyfunction!(nlp::normalize_terms, module)?)?;
+    module.add_function(wrap_pyfunction!(nlp::normalize_terms_l2, module)?)?;
     module.add_function(wrap_pyfunction!(uuid::namespace_uuid, module)?)?;
     module.add_function(wrap_pyfunction!(xxh64, module)?)?;
     module.add_function(wrap_pyfunction!(xxh64_file, module)?)?;

rust/src/nlp.rs

@@ -116,9 +116,32 @@ pub fn normalize_terms(py: Python<'_>, terms: Vec<String>) -> Vec<String> {
     })
 }
 
+/// Batch pyo3 surface for the fullmap `level_two` key filter: derive the
+/// level-two form of a list of (usually level-one) terms, returning an
+/// INDEX-ALIGNED list of strings (rayon's `into_par_iter` with `collect`
+/// preserves input order).
+///
+/// This is the query-side twin of the build-side key derivation: Python's
+/// `nlp.level_two` default path calls this, and this calls the exact
+/// `fullmap::level_two` filter `emit_term` stores build-side keys through, so
+/// query and build level-two keys are byte-identical by construction. Before
+/// issue #251 the query side ran a Unicode-aware polars `\\W+` regex while the
+/// build stripped every non-ASCII character, so non-ASCII terms could never
+/// match at level two. The GIL is released for the whole batch via `py.detach`
+/// (the same pattern as `normalize_terms`).
+#[pyfunction]
+pub fn normalize_terms_l2(py: Python<'_>, terms: Vec<String>) -> Vec<String> {
+    py.detach(move || {
+        terms
+            .into_par_iter()
+            .map(|term| crate::fullmap::level_two(&term).into_owned())
+            .collect()
+    })
+}
+
 #[cfg(test)]
 mod tests {
-    use super::{normalize_l1, normalize_terms};
+    use super::{normalize_l1, normalize_terms, normalize_terms_l2};
     use pyo3::prelude::*;
     use std::borrow::Cow;
     use std::time::Instant;
@@ -259,6 +282,34 @@ mod tests {
         });
     }
 
+    #[test]
+    fn normalize_terms_l2_batch_is_index_aligned_and_handles_empty() {
+        // WHY: the Python level-two path pairs input and output positionally;
+        // output[i] must be fullmap::level_two(input[i]) -- including "" and an
+        // empty batch -- and non-ASCII terms must survive with the SAME keys the
+        // build side stores (issue #251: query and build L2 keys must never
+        // diverge again, which this pins against literal spellings).
+        pyo3::Python::initialize();
+        pyo3::Python::attach(|py| {
+            let func = pyo3::wrap_pyfunction!(normalize_terms_l2, py).unwrap();
+            let input = vec![
+                "au café lait".to_string(),
+                "β-catenin".to_string(),
+                "hello-world".to_string(),
+                String::new(),
+            ];
+            let output: Vec<String> = func.call1((input,)).unwrap().extract().unwrap();
+            assert_eq!(output, vec!["aucafélait", "βcatenin", "helloworld", ""]);
+
+            let empty: Vec<String> = func
+                .call1((Vec::<String>::new(),))
+                .unwrap()
+                .extract()
+                .unwrap();
+            assert!(empty.is_empty());
+        });
+    }
+
     #[test]
     fn normalize_terms_performance_one_million_terms() {
         // WHY: level-one normalization is a hot path for large tabular inputs;

rust/tests/build_golden.rs

@@ -61,7 +61,6 @@ alpha gene|HGNC:1
 alphagene|HGNC:1
 back\slash|HGNC:3
 backslash|HGNC:3
-caf|HGNC:2
 café|HGNC:2
 doid999|MONDO:1
 doid:999|MONDO:1
@@ -95,7 +94,6 @@ mondo:3|MONDO:3
 multi|HGNC:10,MONDO:3
 name quot|HGNC:3
 namequot|HGNC:3
-nave|HGNC:2
 naïve|HGNC:2
 ncbigene100|HGNC:1
 ncbigene101|HGNC:1
@@ -104,8 +102,9 @@ ncbigene:101|HGNC:1
 nullnam|HGNC:7
 realnam|HGNC:4
 share|HGNC:6,MONDO:2
-t|HGNC:3
 été|HGNC:3
+β-catenin|HGNC:1
+βcatenin|HGNC:1
 "#;
 
 // ---------------------------------------------------------------------------

rust/tests/common/mod.rs

@@ -61,7 +61,8 @@ pub struct SourceRow {
 
 // ---------------------------------------------------------------------------
 // Fixed embedded fixture.  Deliberately covers: plain ASCII names, unicode names
-// (café / naïve), escaped JSON (quotes, backslashes, \uXXXX), alias fields
+// (café / naïve), a Greek-letter name (β-Catenin, the issue #251 level-two pin),
+// escaped JSON (quotes, backslashes, \uXXXX), alias fields
 // (id/name/categories/taxon), dead terms (12345/none/nan), equivalent
 // identifiers, multiple names per row, an empty names array, a null
 // preferred_name, a row with no names array, and a class row with no
@@ -70,7 +71,7 @@ pub struct SourceRow {
 // ---------------------------------------------------------------------------
 
 pub const SYNONYM_LINES: &[&str] = &[
-    r#"{"curie":"HGNC:1","preferred_name":"Alpha Gene","names":["Alpha Gene","alpha"],"types":["Gene"],"taxa":["NCBITaxon:9606"]}"#,
+    r#"{"curie":"HGNC:1","preferred_name":"Alpha Gene","names":["Alpha Gene","alpha","β-Catenin"],"types":["Gene"],"taxa":["NCBITaxon:9606"]}"#, // includes a Greek-letter name (issue #251 L2 pin)
     r#"{"curie":"HGNC:2","preferred_name":"café","names":["café","naïve"],"types":["Gene"],"taxa":["NCBITaxon:9606"]}"#,
     r#"{"curie":"HGNC:3","preferred_name":"Esc","names":["\"Quoted Name\"","back\\slash","\u00e9t\u00e9"],"types":["Gene"],"taxa":["NCBITaxon:9606"]}"#,
     r#"{"id":"MONDO:1","name":"Alias Disease","names":["alias disease"],"categories":["biolink:Disease"],"taxon":["NCBITaxon:0"]}"#,

src/tablassert/nlp.py

@@ -71,6 +71,30 @@ def level_one(lf: pl.LazyFrame, col: str) -> pl.LazyFrame:
     return lf.with_columns(expr.alias(col))
 
 
+def _level_two_series(values: pl.Series) -> pl.Series:
+    """Level-two normalize one whole string column through the Rust level-two filter.
+
+    Args:
+        values: Raw column batch (already cast to string by the caller).
+
+    Returns:
+        Series with every non-null value replaced by its level-two form,
+        nulls kept exactly where they were.
+
+    Notes:
+        Shares the exact batch shape of ``_normalize_terms_series`` (nulls never
+        reach Rust; one GIL-released Rust call per column instead of per row,
+        ``map_elements`` is forbidden in this repo) but applies the fullmap
+        build-side level-two key filter instead of the level-one pipeline, so
+        the query-side ``{col}_two`` keys are byte-identical to the keys
+        ``emit_term`` stores at build time (issue #251).
+    """
+    text: pl.Series = values.cast(pl.String)
+    indices: pl.Series = text.is_not_null().arg_true()
+    normalized: pl.Series = pl.Series(rs.normalize_terms_l2(text.drop_nulls().to_list()), dtype=pl.String)
+    return text.scatter(indices, normalized)
+
+
 def level_two(
     lf: pl.LazyFrame,
     col: str,  # pyright: ignore
@@ -82,16 +106,28 @@ def level_two(
     The cleaned values are written to a new column named ``f"{col}{tag}"`` rather
     than overwriting the source column.
 
+    With the default pattern the column is cleaned through the compiled Rust
+    filter (``rs.normalize_terms_l2``) that derives the fullmap build-side
+    level-two keys, so query-side and build-side keys are byte-identical by
+    construction: non-ASCII word characters (accents, Greek letters) survive
+    while punctuation, symbols, and combining marks are dropped on both sides.
+    A custom ``regex`` keeps the vectorized Polars path and is a generic text
+    cleaner with no build-side key contract.
+
     Args:
         lf: Source LazyFrame.
         col: Name of the source column to clean.
-        regex: Pattern whose matches are removed. Defaults to runs of non-word
-            characters (``\\W+``).
+        regex: Pattern whose matches are removed. Only the default non-word
+            pattern (``\\W+``) routes through the shared Rust level-two filter;
+            any other pattern is applied with Polars directly.
         tag: Suffix appended to ``col`` to form the output column name.
 
     Returns:
         LazyFrame with the new tagged column added.
     """
-    expr: pl.Expr = pl.col(col).str.replace_all(regex, "")
+    if regex == r"\W+":
+        expr: pl.Expr = pl.col(col).cast(pl.String).map_batches(_level_two_series, return_dtype=pl.String)
+    else:
+        expr = pl.col(col).str.replace_all(regex, "")
     col: str = col + tag
     return lf.with_columns(expr.alias(col))

src/tablassert/rs.pyi

@@ -26,6 +26,7 @@ def hydrate_sources(db: Path) -> list[str]: ...
 def lookup_fullmap_terms(db: Path, terms: list[str], return_format: str = "rows") -> list[dict[str, Any]]: ...
 def namespace_uuid(domain: str, values: list[str]) -> str: ...
 def normalize_terms(terms: list[str]) -> list[str]: ...
+def normalize_terms_l2(terms: list[str]) -> list[str]: ...
 def taxon_allowlist_identity(taxon_allowlist: list[int]) -> str: ...
 def xxh64(data: str) -> str: ...
 def xxh64_file(path: str) -> str: ...