diff --git a/docs/customize.rst b/docs/customize.rst index 38595d46..6f8a3ddc 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -50,11 +50,21 @@ them. Names outside that list, ``TITLES`` among them, are unchanged. Every 1.x name still imports, with a ``DeprecationWarning``, until 3.0 — see :doc:`migrate` for the full mapping. -Vocabulary entries are matched one word at a time (``given_name_titles`` -excepted), so a multi-word entry like ``titles={"grand moff"}`` can +Vocabulary entries are matched one word at a time, with two +exceptions, so a multi-word entry like ``titles={"grand moff"}`` can never match; the constructor warns when it sees one (``capitalization_exceptions`` keys included — they are looked up per -word too). +word too). The exceptions are ``given_name_titles``, looked up as the +space-joined run of words already read as titles, and +``maiden_markers``, matched by lookahead over the words as written: +``maiden_markers={"z domu"}`` matches the pair and neither word alone, +which is how the shipped Polish entry works. The words have to stand +together — a bracketed clause or a comma between them ends the run, and +the first word is then an ordinary name word. Where a phrase entry and a +word entry starting with it are both configured, the phrase wins where +it matches and the word matches everywhere else. No warning is raised +for a multi-word entry in either of these two fields, since there it is +not a mistake. Removing works the same way, and drops the word from recognition: diff --git a/docs/design/AGENTS.md b/docs/design/AGENTS.md index e2fa9c70..3a599113 100644 --- a/docs/design/AGENTS.md +++ b/docs/design/AGENTS.md @@ -27,6 +27,6 @@ Skip what the suite already enforces. `tests/v2/test_rules_doc.py` and `tests/v2 7. **General clause vs. adjudicated scope.** Enumerate the vocabulary a shape-clause reaches. (P6's precedence was argued for `vd` and swept in `do` and `mc`.) 8. **Guard docstring vs. what the guard enforces**, especially when the promise spans two test modules. 9. **Prose is input to the doc parsers.** A line starting with `"` inside a rule block is an example; a comment's quoted values join the citation block above them. -10. **Re-run every capability claim.** Axis 1 recomputes numbers; this one recomputes CAN and CANNOT. A statement that the parser never or cannot produce some reading is measured by a parse run written now, never derived from a mechanism — most of all when it justifies building something. (#291 derived "a multi-word credential is unparseable" from the true "vocabulary is matched one token at a time", and specified a vocabulary set, a matching unit, a predicate and a `Lexicon` field on it; `parse("John Smith, MD PhD").suffix` had been `'MD PhD'` since 1.4.0.) +10. **Re-run every capability claim.** Axis 1 recomputes numbers; this one recomputes CAN and CANNOT. A statement that the parser never or cannot produce some reading is measured by a parse run written now, never derived from a mechanism — most of all when it justifies building something. (#291 derived "a multi-word credential is unparseable" from "vocabulary is matched one token at a time" — which `given_name_titles` had been an exception to since 2026-07-19, a month before that arc, and `maiden_markers` is a second since #434 — and specified a vocabulary set, a matching unit, a predicate and a `Lexicon` field on it; `parse("John Smith, MD PhD").suffix` had been `'MD PhD'` since 1.4.0.) Also: before reporting "N names move", report the size of the population that COULD move — a small count over a corpus blind to the shape is evidence about the corpus. For rules.md#P6: of 782 corpus names, 245 carry a comma, two of those end in a particle, and one of the two clears the words-to-spare guard. Running axis 1 over this very list caught an earlier wording of that sentence conflating the population with the movers, so the count above is the corrected one. diff --git a/docs/design/decisions.md b/docs/design/decisions.md index b2817497..696ee4bc 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -142,6 +142,30 @@ the fullwidth-colon marker (旧姓：佐藤 arrives as one word; the head-peel q - 2026-08-25 #410 — the maiden name no longer makes H1 decline; the rule counts name words and a marker-announced name is not one. The decision and its width are recorded at decisions.md#H1, since the term it removed was never maiden-specific. +- 2026-08-26 #434 — a maiden marker may be a PHRASE, and the Polish "z domu" ships as the first one. The rule had assumed a marker is one word everywhere it was written down, and the library's own advice made that assumption load-bearing: the dead-entry warning told a caller with a multi-word entry to "split it into separate entries", which for this field produces two wrong readings rather than one inert entry. With z and domu stored separately, "Maria Kowalska z domu Nowak" reads maiden "domu Nowak" — the marker's own second word inside the value #434 reports — and "Anna z Nowak" loses its family name outright, because a bare preposition is then a marker and M2 hands a marker every word after it. So the warning was not merely unhelpful for this field; following it was the defect. A phrase entry matched whole is the only shape that fixes both: z is an ordinary Polish preposition and domu an ordinary noun, and neither ever claims anything alone. + + A run predicate — identify a run of marker words, then look the joined run up — was rejected, and the reason is the difference between the two exceptions this makes. given_name_titles is the STORAGE precedent and NOT the matching one. Its entries are stored space-joined and folded per word (_lexicon._title_key, since whole-phrase normalization would leave 'lt. col' with its interior period), and markers copy exactly that. But its MATCHING identifies the run per word first — 'lt' and 'col' are each title vocabulary, so the parse has already read both as titles before anything is joined — and z and domu are markers in no vocabulary at all, so no run can be identified before the lookup. Markers therefore need genuine multi-token lookahead, longest first, which is a different mechanism wearing the same storage rule. Worth knowing that until this change NO shipped vocabulary set held a single multi-word entry, given_name_titles included: the exception existed for caller-configured vocabulary only, and "z domu" is the first phrase the library itself ships. + + That the phrase is safe where its words are not is the C-i question answered one level up, and the contrast worth drawing is with the neighbour in this section's Excluded block. #vocabulary-collisions C-i asks whether a word is borne as an ordinary name in the position the vocabulary claim acts on; roz fails it — Roz is a diminutive of Rosalind and stands exactly where M2's claim acts, the word after a given name — and MAIDEN_MARKERS has no ambiguous subset to express a partial answer in, so the only remedies available there were ship and do not ship, and it was removed. z fails C-i at least as badly. What changes is that the claim need not be made about z at all: the phrase is the unit that claims, and "z domu" is borne as no name in any position. C-i is a test on the claiming unit, and a phrase entry is a way of choosing a different unit rather than a way of passing the test with the old one. This is not a general escape — a phrase whose words are common and whose pairing is also common would fail C-i as a phrase — but it is the answer whenever the collision is with a word that only a longer phrase makes unambiguous. + + Markers are consulted at FOUR sites, which is one more than the title-run key's three, and _title_key's own docstring names the failure mode ("a divergence between them fails silently: the entry simply stops matching"). So one predicate answers "does a marker start here, and where does it end" — _pipeline._vocab.maiden_marker_run, in the vocabulary layer because it is a question about token TEXTS, per mechanisms.md#ONE-PREDICATE-PER-QUESTION's rule that the destination follows the layer. The four are classify's tagging, extract's clause test, and group's TWO — the piece test M2's take is built on and the drop inside an extracted clause (#329), which is the pair that makes four where titles have three. classify calls the predicate in a sequence pass, tagging a run's head "vocab:maiden-marker" as before and its continuations "vocab:maiden-marker-cont"; extract calls it over a clause's whitespace words; both of group's sites read those tags rather than re-deriving the run, which is the same mechanism's recorded-answer half, available because group runs after classify. tests/v2/pipeline/test_group.py::test_every_marker_site_ends_the_run_in_the_same_place is the contract, modelled on the P5/H1 title-run test: it collects each site's answer to "how many words is the marker" over nine spellings and asserts they cannot disagree, and each of the four sites has been mutated to confirm it fails. + + One divergence between those sites is deliberately PRESERVED and must survive any future unification. extract splits a clause on WHITESPACE while the tokenizer splits punctuation off, so "née," is marker-led to classify and not to M3. That is what keeps group's role filter reachable, and cases.py::marker_glued_to_punctuation_keeps_the_clause_a_nickname is the row that pins it — 'Jane "née, Janie" Smith (née Jones)' reads nickname 'née Janie', maiden 'Jones'. Sharing the predicate did not merge the two questions, because each site hands it a different sequence of words; confirmed by mutation, where making _maiden_marked strip commas leaves that row (and its facade twin) the only failures. + + The commit order carries a finding about this repo's tests worth stating on its own. Taken in the plan's order — predicate, then storage plus the warning carve-out, then the sites — the middle commit would have exempted maiden_markers from the dead-entry warning before anything could match a phrase. That is not merely untidy: pyproject sets filterwarnings = ["error"], so a test that configures a phrase entry FAILS until the exemption lands, which means the carve-out cannot be deferred past the sites either. The two constraints leave one arrangement — the predicate alone, then storage and the four sites together, then the shipped entry — and the middle commit is fused for that reason rather than by preference. Shipping "z domu" last is what keeps any bisected revision from holding a phrase entry nothing can match. + + Review found one real defect, and it is the one this whole design exists to prevent. A marker run is TAGGED over the whole span-sorted token stream and CONSUMED over one segment, and the two populations are not the same: _segment.py:31 keeps only tokens extract has not already given a role, then buckets what is left by the commas before them. So a phrase written across a bracketed clause — "Anna z (domu) Nowak" — had both its words tagged while the piece walk could see only the first, and M2 took a proper PREFIX of the phrase for the whole marker: family 'Anna', maiden 'Nowak', a bare Polish preposition eating the name, and "Maria z (domu) Kowalska Nowak" losing its family name outright. Reachable on shipped vocabulary under the default policy, and it falsified three of this branch's own claims at once, in three files. The fix is a rule about the TAG rather than a guard on the consumer: classify refuses to tag a run whose tokens are not structurally contiguous — same role, so a clause edge ends a run, and the same comma bucket, computed the way segment computes it. Refusing rather than truncating, because truncating hands M2 the same wrong prefix one word shorter. The lookahead is bounded at the boundary before the predicate is asked, so a two-word entry refused at a clause edge still leaves a one-word entry starting there free to match, and an all-single-word vocabulary skips the walk entirely. rules.md#M2 now states the qualifier — a phrase is recognized only whole AND only where its words stand together — which it did not, and that gap is part of why the defect was invisible. + + The defensive complement was declined and the reason is worth recording: having _marker_run_pieces re-derive the run from token indices and refuse a mismatch would be a guard that, with the tagging rule in place, can never fire — untestable code standing in for an invariant one stage already guarantees. What it would have bought is written into that function's docstring instead, which now names _segment.py:31 as the second reason `seen` can skip a token; the sentence that said the continuation "is always in `seen`" without naming that reason is what made the gap invisible to review. + + The contract test should have caught this and did not, and the hole is worth naming because it is a property of how the contract was FORMULATED. It varied the marker's SPELLING across nine forms and held its PLACEMENT fixed, so every run it built was structurally contiguous by construction and no probe ever asked what the sites do when a run straddles a boundary — which is precisely where two sites walking different token populations can disagree. The placement axis is now its own test over eight placements, asserting two things no spelling can reach: that every tagged run has one role and one comma bucket, and that group drops a run whole or declines it. Mutation-checked in three pieces, which is what shows the two halves of the rule are both load-bearing: removing the contiguity bound outright fails both new case rows and all three straddling placements; removing only the ROLE half fails the two clause placements and both rows; removing only the COMMA half fails exactly one placement and NO case row, because a comma-split run happens to be declined downstream for an unrelated reason today. That last one is the argument for a tag-level contract test in its own right — the only witness to half the rule is an assertion about tags, not about fields. + + Perf, and how it was measured, because the first attempt at it was wrong in a way worth recording. Timing four source trees in separate processes and comparing them across sessions gave figures whose spread was the size of the effect: merged master alone measured 0.579s, 0.593s and 0.607s on one machine inside an hour. Every number below is a median of five INTERLEAVED rounds — every variant timed once per round, in order, so drift lands on all of them alike — over 8000 parses of a four-name mix (plain, titled-with-initials, phrase-marked, comma-and-particle). Against master: the run pass alone +6.8%, a head-word fast path inside the predicate +1.4%, the contiguity rule on top of that +1.6%, and after the simplify round -0.3% and -1.3% on two separate five-round runs. So the shipped mechanism has no cost this benchmark can resolve. That is not a free lunch and should not be read as one: classify no longer does the per-token `n in lexicon.maiden_markers` membership test master did — the marker decision moved into one pass that answers for every token at once — so what the marker pass adds and what it removed are within noise of each other. The reviewer reproduced the +7.2% and +1.6% stages independently on other hardware, which is the corroboration a percentage of this size needs. + + Four structural choices got it there, and each is a smaller version of the same move: ask the question once and pass the answer along. The fast path lives INSIDE the predicate rather than at its callers, so it stays an implementation detail and not a second answer. maiden_marker_head takes the FOLD rather than the text (suffix_as_written's shape, "n is _normalize(text), passed in so callers normalize once"), and classify folds once per token for the marker pass and the vocabulary tags together, which took the common token from two folds to one — fewer than before this work. maiden_marker_run folds each word once and keys from a prefix, where _title_key(words[:n]) per candidate length re-folds the whole prefix and made a ONE-word hit cost more than a two-word one; sixteen of the seventeen shipped entries are one word. And the marker pass returns its decision as index -> tag for classify's single token-building loop to write, rather than rebuilding every marker token a second time with dataclasses.replace, which is the most expensive call in a parse. The comma-bucket sweep the contiguity rule needs is deferred until a marker head is actually found, so a single-word vocabulary and a phrase vocabulary over a name holding no marker both skip it entirely. + + One limit is recorded rather than closed. The contiguity test is role plus comma bucket, and role is one-directional: a role change is always a clause edge, but two ADJACENT clauses of the same role are indistinguishable, so 'Jane (z) (domu) Jones' does tag a run across two separate nickname clauses. Nothing reads it — the piece walk never sees role-bearing tokens and the clause drop is scoped to one span — and the parse is identical with and without the phrase mechanism. The clean fix is a per-token structural UNIT id derived from state.segments and state.extracted, which would replace role-plus-bucket with one comparison and close the adjacent-clause case as a side effect. Deliberately not taken here: it changes a mechanism four correctness reviewers had just cleared, inside a pass whose stated business was not to change behavior. It is the shape to reach for if this test ever needs to grow a third term. + ### N3 — the lone-word nickname rule - 2026-07 (v2 core, PR #288; recorded plan deviation #2 of the core plan) — v1's rule counted pieces before grouping; the v2 port fires only when the nickname accompanies exactly ONE piece in total — a title counts against it, so "'Smitty' Dr. Jones" reaches H1 with a title and one name word left standing — through 2.1 that meant given="Jones" with the family empty, and since #410 (2026-08-25) H1 names the family, so it reads family="Jones". The count is unchanged; what moved is what happens after it declines. The rule lives in assignment because that is where the piece count is settled. (An earlier wording here said "one non-title piece", predicting the opposite output; the coherence review measured the truth.) @@ -404,7 +428,7 @@ Excluded (given-name titles and post-nominals — the 2026-07-19 transliteration - Hebrew bare רב (an ordinary word, "many") and בר (Bar is a common modern Israeli given name) — deferred, #269's territory. - Latin sri/shri deliberately absent while Devanagari श्री ships: the transliteration collides (Sri Mulyani), the native script cannot. -Excluded (multi-word vocabulary entries — every set matches one written word, so a multi-word entry is silently inert and now warns at configuration): +Excluded (multi-word vocabulary entries — a set outside #M2's two exceptions matches one written word, so a multi-word entry there is silently inert and warns at configuration; given_name_titles and, since #434, maiden_markers are the exceptions, and they match phrases by different mechanisms): - Eight entries shipped unmatchable from 2013 to 2.0. chargé d'affaires was SPLIT into the chainable chargé + d'affaires, plus unaccented charge (the attaché/attache precedent, Derek's call); leed ap, nicet i–iv and psm i/ii were REMOVED rather than split, on the name-swallowing measurements recorded in #291 (see the comma-suffix-arc Declined entry). The multi-word UserWarning is this story's enforcement. @@ -432,6 +456,7 @@ Excluded (MAIDEN_MARKERS, per nameparser/config/maiden_markers.py): surname)" as open; the question resolved silently with the shipped set. Recorded here because the included risky member deserves its analysis as much as the excluded ones; M1's (Nee) boundary covers only the enclosure path, not this marker path. - "born" — never shipped: a release-log drafting invention, caught by the 2.0 milestone audit and corrected (5ccf9f3). Recorded so nobody "restores" it; if ever proposed for real, Max Born is the counterexample to analyze. - Scandinavian "f." — collides with the initial F.; only the full participles (født/fødd/född) are safe. Czech masculine "rozený" awaits the same vetting, on its own merits rather than following the feminine form in. +- Polish "z domu" — INCLUDED 2026-08-26 as the set's first phrase entry, and the contrast to draw against the roz removal below rather than against any included word. Its words fail C-i separately and it passes as a pair; the argument is in the #434 entry under #M2. The point to carry away is that the entry below could not have been rescued the same way: roz is a whole marker as written, so there is no longer unit to move the claim onto. - Czech/Slovak "roz." — SHIPPED through 2.1 and removed 2026-08-26 (Derek's call). Roz is an ordinary English diminutive of Rosalind, and matching is whole-token, case-folded and period-insensitive, so "Roz", "roz" and "roz." are one string to this set. The collision is not theoretical and it is not new: on the bare path, which predates the 2.2 work entirely, "Rosalind Roz Smith" read maiden "Smith" and NO family name at all, and "Rosalind Roz Jones Smith" read maiden "Jones Smith", because M2 hands the marker every word after it. Grounded in #vocabulary-collisions C-i — a word belongs in its set's ambiguous subset iff it is borne as an ordinary name IN THE POSITION THE VOCABULARY CLAIM ACTS ON, and under uncertainty default to ambiguous. Roz is borne exactly there: the word after a given name, which is the only position M2's claim acts on. What C-i cannot do for this set is the marking: MAIDEN_MARKERS has no ambiguous subset, and no code path consults one, so the only two expressions of C-i available here are ship and do not ship. That is why the remedy is removal rather than an ambiguous marking, and it is worth knowing before someone proposes marking the next collision instead. The full participle rozená stays and is unambiguous; a Czech caller who needs the abbreviation adds it to their own Lexicon, which is the same answer this section gives for every locale-specific vocabulary. The cost is accepted and stated: "Anna Nováková roz. Svobodová" now reads middle "Nováková roz.", family "Svobodová" — which is how 1.4.0 read it — and "Rosalind Roz Smith" is back to 1.4.0's given/middle/family. Pinned by tests/v2/cases.py::diminutive_that_was_a_marker_keeps_the_family and ::full_participle_marker_still_consumes. @@ -495,6 +520,10 @@ Declined: - 2026-08-26 #335 — correction to the 2026-08-05 entry above, which reasoned from "the #329 change was gate-visible". It was not. Under the default policy the corpora run, no name reached the drop pass at all, so 山田 花子（旧姓 佐藤） moving between 2.0.0 and 2.1.0 was the East Asian order flip and not #329. Nor is #329 gate-visible now that M3 routes marker-led clauses to maiden without configuration: a ledger rule narrows by which FIELDS move and never by what they hold, so reverting the drop leaves the same six names moving the same fields and all three gates green (measured by mutation 2026-08-26). The {nickname, maiden} pair is the 2.1.0 ledger's; the other two baselines classify the CJK name differently, which is why those ledgers are split and must stay so. The deferral #329 argued for was still the right call — it was about a Latin-affecting change riding under a gate report, and that half held — but the premise is corrected here rather than left to be re-used. tools/differential/README.md carries the same correction beside the table it belongs to. - 2026-08-26 #335 (M3) — the opt-in this section opens with narrowed, and the boundary is not where the 2026-07-03 entry drew it. A clause whose content announces itself no longer needs its pair declared, so what configuring maiden_delimiters buys is now markerless content and one-word clauses. See #M3 for the whole reasoning; M1 still governs every clause whose pair is configured, and M3 defers to it for the marker drop itself. +- 2026-08-26 #434 — correction to the 2026-08-04 #329 entry above, on the two statements a phrase entry falsified, and a decision that entry's reasoning does not reach. Its "dropped from a multi-word clause" is no longer the condition: "(z domu)" under a configured pair is a two-word clause whose marker is NOT dropped, because both words ARE the marker. The condition has always been "the clause holds a word past its marker" and reads that way in the code; with one-word markers the two were indistinguishable, which is why the entry could say the wrong one for two years without cost. Nor is there a "clause-size guard" any more, load-bearing or otherwise — the containment test now spans the whole marker run, so its subject is the token AFTER the run rather than the second token of the clause. What survives unchanged is the REASON that guard was written: Nee is an attested surname (Irish Ní/Nee, a Chinese romanization), so a clause of nothing but its marker keeps its words, and that argument is mutation-proven still. Recorded here rather than only under #M2, because a future PR reading M1's own history pointer would otherwise find "the clause-size guard" described as load-bearing and restore something that reintroduces the phrase defect while passing every doc test. + + The decision the rewrite forces, stated so a later PR has something to weigh against. Under a configured maiden pair, "Maria Kowalska (z domu)" emits maiden "z domu" — the marker itself as the whole maiden name. That is not what the phrase argument under #M2 says about those words: "z domu" is borne as no name in any position, which is exactly why shipping it as a phrase is safe. So the value is a name nobody has. It is emitted anyway, and deliberately: M1's carve-out is that a clause holding nothing past its marker keeps its WORDS, and the parser has no way to know a caller's configured pair did not enclose an unfamiliar surname — the same ignorance that makes "(Nee)" keep its word. The alternative readings, an empty maiden or a nickname, both require the parser to judge its own vocabulary's words unnameable, which nothing else in the library does. Cheap to revisit: the shape is one rules.md example and one case row away from a decision either way, and no corpus name has it. + ### M3 — the marker-led clause - 2026-08-26 #335 (M3, PR on fix/335-maiden-marked-clause) — a bracketed clause whose content opens with a recognized marker word and carries a word after it reads as the maiden name, whichever bucket the enclosing pair sits in. The parse already had the information: "Jane Smith née Jones" gave maiden "Jones" while "Jane Smith (née Jones)" gave nickname "née Jones", because ( ) is a nickname pair by default and nothing looked inside it. The clause says "maiden" out loud, so the caller should not have to say it in Policy. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 8bc04b10..12aa08a4 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -55,7 +55,7 @@ Problem shape. "Which stage does X?" — asked before attributing behavior in pr ## ONE-PREDICATE-PER-QUESTION — one predicate answers it, and every other site calls that -Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix, and is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, is_leading_title, leading_titles, peel_walk, peel_trailing and segment_suffix_reading are called by both stages — the last of those is #430's instance, where THREE readers share one answer, the render join being the third — while is_title_piece and trailing_start are called by group alone — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at. tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. +Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, is_leading_title, leading_titles, peel_walk, peel_trailing and segment_suffix_reading are called by both stages — the last of those is #430's instance, where THREE readers share one answer, the render join being the third — while is_title_piece and trailing_start are called by group alone — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at. tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. ## CLAUSE-CONTENT-OVERRULES-DELIMITER — content wins diff --git a/docs/design/rules.md b/docs/design/rules.md index 78afa97c..f2ee4df6 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -432,7 +432,7 @@ P6. Rationale: a particle ending the name has nothing to link ## Suffixes: generational & credentials (S) -Background: what follows a name is one of two different things — generational suffixes (Jr., III), which attach to the name itself, and credentials (PhD, MD, MBA), which are earned attachments. All vocabulary sets match one written word at a time: a multi-word entry can never match anything and is warned about at configuration (the eight that shipped dead for years are the Excluded story in decisions.md). CLDR personNames keeps them as separate fields (`generation`, `credentials`) and formats them differently; this library currently reports both in one `suffix` field, a merge #326 examines. The vocabulary is largely split already: a generational word list and a credential acronym list, plus a short list of acronyms that are also ordinary names (MA, BA) and so are AMBIGUOUS as bare words. +Background: what follows a name is one of two different things — generational suffixes (Jr., III), which attach to the name itself, and credentials (PhD, MD, MBA), which are earned attachments. The suffix sets match one written word at a time: a multi-word entry there can never match anything and is warned about at configuration. Only two sets are exempt from that rule -- given_name_titles and, since #434, maiden_markers -- and neither is a suffix set, so within the suffix vocabulary the one-word rule is absolute. The eight multi-word entries that shipped dead for years span the suffix sets and the titles alike and are the Excluded story in decisions.md. CLDR personNames keeps them as separate fields (`generation`, `credentials`) and formats them differently; this library currently reports both in one `suffix` field, a merge #326 examines. The vocabulary is largely split already: a generational word list and a credential acronym list, plus a short list of acronyms that are also ordinary names (MA, BA) and so are AMBIGUOUS as bare words. S1. Rationale: brackets set off more than nicknames — credentials are routinely written parenthesized after a name, and a @@ -542,7 +542,7 @@ N3. Rationale: a person set down as a nickname plus one name word is ## Maiden names (M) -Background: a maiden name is written beside the current name, set off by a marker word or by enclosure. Markers are attested across French née/né, German geb./geborene, Dutch geboren, Czech/Slovak rozená (the abbreviation roz. shipped through 2.1 and was removed in 2.2 -- it collides with the English diminutive Roz, and a caller who needs it adds it to their own Lexicon), Scandinavian født/fødd/född, Russian урожд. (both ё and е spellings), and Japanese 旧姓 — both grammatical genders where attested. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3) — unless its content is suffix-shaped, which S1 takes ahead of both. A lone marker word has said nothing, and neither has one the colon spelling above glues to the name. +Background: a maiden name is written beside the current name, set off by a marker or by enclosure. Markers are attested across French née/né and the unaccented nee English writing uses for both, German geb./geborene, Dutch geboren, Czech/Slovak rozená (the abbreviation roz. shipped through 2.1 and was removed in 2.2 -- it collides with the English diminutive Roz, and a caller who needs it adds it to their own Lexicon), Scandinavian født/fødd/född, Russian урожд. and its full participles урождённая/урождённый (the participles in both the ё and е spellings, which case normalization does not fold), Japanese 旧姓, and Polish z domu — both grammatical genders where attested. Every one of these is a marker wherever it stands, the unaccented nee included: an enclosure holding a word past it reads as the maiden name and the marker is dropped, so a configured pair reading "(Nee Jones)" gives maiden Jones and not Nee Jones. A marker need not be one word. z domu is two, and a phrase marker is recognized only whole and only where its words stand together: neither of its words is a marker standing alone, and neither is the pair once a bracketed clause or a comma divides them. That is what makes shipping it safe where shipping its words would not be — z is an ordinary Polish preposition, and a name that merely contains one keeps its family name. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3) — unless its content is suffix-shaped, which S1 takes ahead of both. A lone marker has said nothing, and neither has one the colon spelling above glues to the name. M1. Rationale: an enclosure the caller has declared to mean maiden holds the former family name; a recognized marker word inside it @@ -550,20 +550,23 @@ M1. Rationale: an enclosure the caller has declared to mean maiden With a delimiter pair configured for maiden names, its enclosed clause reads as the maiden name — unless the content is suffix-shaped, which S1 takes first — a leading recognized - marker word inside a multi-word clause being dropped; a one-word - clause keeps its word, which may itself be a surname (Nee). + marker being dropped where the clause holds a word past it; a + clause of nothing but its marker keeps its words, which may + themselves be a surname (Nee). Clauses are independent: two enclosures read as one maiden name, each dropping or keeping its own marker. A pair configured for both maiden and nickname reads maiden. Configuring the pair is what this rule needs for a clause that does not announce itself - — markerless content, and a lone marker word alike; a clause of - two words or more led by a recognized marker reads as the maiden - name inside a nickname pair as well (M3), the suffix-shaped - content S1 takes excepted there as it is here. + — markerless content, and a lone marker alike, whether that + marker is one word or several; a clause holding a word past its + marker reads as the maiden name inside a nickname pair as well + (M3), the suffix-shaped content S1 takes excepted there as it is + here. "Jane Smith (née Jones)" maiden-parens → maiden="Jones" "Jane Smith (Nee)" maiden-parens → maiden="Nee" · boundary "Jane Smith (Nee) (Jones)" maiden-parens → maiden="Nee Jones" "Andrew Perkins (MBA)" maiden-parens → suffix="MBA" · boundary + "Maria Kowalska (z domu)" maiden-parens → maiden="z domu" · boundary history: decisions.md#M1 · interacts: S1, M2, M3 · implemented: nameparser/_pipeline/_extract.py, nameparser/_pipeline/_group.py M2. Rationale: a maiden marker announces that what follows it is the @@ -576,6 +579,14 @@ M2. Rationale: a maiden marker announces that what follows it is the maiden name, and the marker itself is dropped. A marker with nothing after it, or nothing before it, is just a word. + A marker may be more than one word, and is then recognized only + whole and only where its words stand together: its own first word + standing without the rest is not that marker, and neither is one + the writer divided from the rest by a bracketed clause or by a + comma. Whether the first word takes anything on its own is a + separate question with a separate answer — it does exactly when it + is a recognized marker in its own right, and the longer entry wins + wherever both could match. A marker taken this way also bounds a particle join arriving from its left (P2), so the family name's particles stop at the marker instead of absorbing it; a marker left as a word bounds nothing. @@ -597,6 +608,9 @@ M2. Rationale: a maiden marker announces that what follows it is the "Jane van der Berg née PhD" → family="van der Berg née" "Jane van der Berg née y Jones" → maiden="y Jones" "van der Berg, abdul née Jones" → maiden="Jones" + "Maria Kowalska z domu Nowak" → maiden="Nowak" + "Anna z Nowak" → family="Nowak" · boundary + "Anna z (domu) Nowak" → family="Nowak" · boundary Accepted: the fullwidth-colon spelling arrives as one word, so the marker inside it goes unrecognized; #317 tracks whether it should peel. @@ -620,7 +634,7 @@ M3. Rationale: an enclosure says nothing about whether it means announces itself, so the caller does not have to declare the pair. A bracketed clause whose content opens with a recognized marker - word and carries a word after it reads as the maiden name, + and carries a word after it reads as the maiden name, whichever bucket the enclosing pair sits in — unless the content is suffix-shaped, which S1 takes first — the marker itself dropped, as M1 drops it. A marker with no word after it is just @@ -636,6 +650,8 @@ M3. Rationale: an enclosure says nothing about whether it means "Jane (née Jones) Smith" → family="Smith" "Jane Smith (née Jr.)" → suffix="Jr." "Jane Smith (née)" → nickname="née" · boundary + "Maria Kowalska (z domu Nowak)" → maiden="Nowak" + "Maria Kowalska (z domu)" → nickname="z domu" · boundary Accepted: the word taken after the marker is not tested for being a name word, so unlike M2's bare take this one does not stop at a suffix word — the same two words read one way diff --git a/docs/release_log.rst b/docs/release_log.rst index 91a1bfbc..b0882ce0 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -45,9 +45,11 @@ Release Log - Fix a bound given name joining past a credential that the suffix rule then takes, leaving no family: ``"abdul Smith Jr Ma"`` read given ``abdul Smith`` with no family, where ``"John Smith Jr Ma"`` reads family ``Smith`` with suffix ``Jr, Ma``. The join's reserve had re-derived the suffix rule's trailing walk by hand and left out one of its two forks -- a bare acronym such as ``Ma`` is a credential only with words to spare -- so it counted the acronym as a name word, joined, and the suffix rule then peeled it. The two stages now share one walk: the reserve tries the join, runs the suffix rule's own peel over the pieces as the join would leave them, and joins only when two name words became one and nothing else changed. ``"abdul Smith Jr Ma"`` reads family ``Smith``, suffix ``Jr, Ma``; ``"abdul Smith Ma"`` reads family ``Smith``, suffix ``Ma`` rather than joining the pair and keeping the credential as the family -- both as 1.4.0 read them and as ``John`` reads in their place; ``"abdul Smith Berg Ma"`` keeps its join, and a title word standing after the bound word joins again as it did in 1.4.0 (``"Berg, abdul Sir"`` reads given ``abdul Sir``) without the pair ever being read as a title (``"abdul Sheikh and Ahmad Bakar"`` reads given ``abdul Sheikh and Ahmad``). No name of the differential corpora has the shapes; the ``Ma`` rules examples carry the fix, byte-identical to 1.4.0 and moving against 2.0.0 and 2.1.0, and the title-word shapes are pinned in tests (closes #425) - - Remove the Czech/Slovak abbreviation ``roz.`` from the default maiden markers, where it had been eating English given names. Marker matching is whole-token, case-folded and period-insensitive, so ``Roz`` -- the ordinary diminutive of Rosalind -- was the same string as the marker, and a maiden marker takes every word after it: ``"Rosalind Roz Smith"`` read maiden ``Smith`` with **no family name at all**, and ``"Rosalind Roz Jones Smith"`` read maiden ``Jones Smith``. Both now read the way 1.4.0 read them, given ``Rosalind`` with ``Roz`` a middle name and ``Smith`` the family. The defect is as old as the marker vocabulary and belongs to the bare marker rule, not to the bracketed-clause change in the bullet below; it was found while reviewing that change, which would have widened it -- with ``roz`` still shipped, ``"Jane Smith (Roz Jones)"`` reads maiden ``Jones`` where 2.1.0 read nickname ``Roz Jones``. What stops working is the abbreviation itself, for Czech and Slovak callers: ``"Anna Nováková roz. Svobodová"`` now reads middle ``Nováková roz.``, family ``Svobodová`` rather than family ``Nováková``, maiden ``Svobodová`` -- which is also how 1.4.0 read it, so this is a return to the v1 reading rather than a new one. The full participle is untouched and needs no configuration: ``"Anna Nováková rozená Svobodová"`` still reads family ``Nováková``, maiden ``Svobodová``, and it is only the abbreviation that was ambiguous. A caller who wants the abbreviation back adds it to their own lexicon -- ``Parser(lexicon=dataclasses.replace(lex, maiden_markers=lex.maiden_markers | {"roz"}))`` -- which restores the old reading exactly. The default set is now 16 entries. No differential corpus name contains the string in any case or position, so nothing moves against any baseline (closes the collision found in #335's review) + - Remove the Czech/Slovak abbreviation ``roz.`` from the default maiden markers, where it had been eating English given names. Marker matching is whole-token, case-folded and period-insensitive, so ``Roz`` -- the ordinary diminutive of Rosalind -- was the same string as the marker, and a maiden marker takes every word after it: ``"Rosalind Roz Smith"`` read maiden ``Smith`` with **no family name at all**, and ``"Rosalind Roz Jones Smith"`` read maiden ``Jones Smith``. Both now read the way 1.4.0 read them, given ``Rosalind`` with ``Roz`` a middle name and ``Smith`` the family. The defect is as old as the marker vocabulary and belongs to the bare marker rule, not to the bracketed-clause change in the bullet below; it was found while reviewing that change, which would have widened it -- with ``roz`` still shipped, ``"Jane Smith (Roz Jones)"`` reads maiden ``Jones`` where 2.1.0 read nickname ``Roz Jones``. What stops working is the abbreviation itself, for Czech and Slovak callers: ``"Anna Nováková roz. Svobodová"`` now reads middle ``Nováková roz.``, family ``Svobodová`` rather than family ``Nováková``, maiden ``Svobodová`` -- which is also how 1.4.0 read it, so this is a return to the v1 reading rather than a new one. The full participle is untouched and needs no configuration: ``"Anna Nováková rozená Svobodová"`` still reads family ``Nováková``, maiden ``Svobodová``, and it is only the abbreviation that was ambiguous. A caller who wants the abbreviation back adds it to their own lexicon -- ``Parser(lexicon=dataclasses.replace(lex, maiden_markers=lex.maiden_markers | {"roz"}))`` -- which restores the old reading exactly. This removal takes the default set to 16 entries. No differential corpus name contains the string in any case or position, so nothing moves against any baseline (closes the collision found in #335's review) - - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. This is about every delimiter pair the parser ships, not only the parentheses these examples use: quotes are how a nickname is most often written, and ``"Jane Smith "née Jones""`` and ``"Jane Smith 'née Jones'"`` move from nickname to maiden exactly as the parenthesized spelling does. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. One more shape moves with it, and it is the one to know about if you parse names that are a surname and a clause: a name that is ONLY a marker clause and one name word now reads that word as the given name rather than the family. ``"Smith (née Jones)"`` reads given ``Smith``, maiden ``Jones`` where every earlier version read family ``Smith``, nickname ``née Jones``. That is the bracketed form agreeing with the bare one -- ``"Smith née Jones"`` has read given ``Smith`` since 1.4.0 -- because a marker-led clause is not a nickname clause and so does not reach the rule that would have made the word a family name. The empty ``family`` it leaves behind is the bare form's behavior and is not new here, though it is not 1.4.0's either -- v1 read ``"Smith née Jones"`` as first ``Smith``, middle ``née``, last ``Jones``, and #445 tracks whether the family should be kept. Seven of the 1,080 differential corpus names move against the 2.1.0 and 2.0.0 baselines and six against 1.4.0: the Japanese name is the odd one, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Eight corpus names have the shape and seven move (measured); the eighth is ``"Jane Smith (née Jr.)"``, which the suffix rule takes first. So the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is -- six of the seven are hand-written rule or issue examples, only the Japanese name arriving from elsewhere, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) + - Add the Polish maiden marker ``z domu`` to the default vocabulary, and let a ``maiden_markers`` entry be more than one word. ``"Maria Kowalska z domu Nowak"`` now reads family ``Kowalska``, maiden ``Nowak``, where every earlier version read the marker as part of the name (1.4.0: middle ``Kowalska z domu``, family ``Nowak``). The bracketed spelling moves with it -- ``"Maria Kowalska (z domu Nowak)"`` read nickname ``z domu Nowak`` and now reads family ``Kowalska``, maiden ``Nowak`` -- while a clause holding nothing but the marker stays a nickname, as a lone ``(née)`` does: ``"Maria Kowalska (z domu)"`` keeps nickname ``z domu``. **This changes what a caller who worked around the old limit should have configured.** Through 2.1 a multi-word entry in any vocabulary field could never match, and the ``UserWarning`` it raised said to split it into separate entries. For maiden markers that advice was wrong, not merely unhelpful: with ``z`` and ``domu`` stored as two entries, ``"Maria Kowalska z domu Nowak"`` reads maiden ``domu Nowak`` -- the marker's own second word inside the value -- and ``"Anna z Nowak"`` reads maiden ``Nowak`` with **no family name at all**, because a bare preposition is then a marker and a marker takes every word after it. If you added ``z`` and ``domu`` separately, remove both: the shipped phrase does the job, and dropping the two word entries is what restores ``"Anna z Nowak"`` to given ``Anna``, middle ``z``, family ``Nowak`` -- which is how it reads on the default vocabulary, unchanged from 1.4.0. The same applies to any other field-splitting workaround for this one field. A phrase is matched over whole tokens that stand together -- a bracketed clause or a comma between the words ends the run, so ``"Anna z (domu) Nowak"`` keeps its family name too -- and longest first, so a phrase and a word entry that starts it can both be configured: with ``geb`` shipped and ``geb von`` added, ``"Jane Smith geb von Braun"`` reads maiden ``Braun`` and ``"Jane Smith geb Braun"`` still reads maiden ``Braun``. ``given_name_titles`` was the only field exempt from the multi-word warning; ``maiden_markers`` is now the second, and the warning still fires for every other field. This addition takes the default marker set to 17 entries. Two differential corpus names move, at all three baselines, and both are examples this change adds to the rules doc; no corpus name held ``domu`` or a bare ``z`` before it (#434) + + - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. This is about every delimiter pair the parser ships, not only the parentheses these examples use: quotes are how a nickname is most often written, and ``"Jane Smith "née Jones""`` and ``"Jane Smith 'née Jones'"`` move from nickname to maiden exactly as the parenthesized spelling does. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. One more shape moves with it, and it is the one to know about if you parse names that are a surname and a clause: a name that is ONLY a marker clause and one name word now reads that word as the given name rather than the family. ``"Smith (née Jones)"`` reads given ``Smith``, maiden ``Jones`` where every earlier version read family ``Smith``, nickname ``née Jones``. That is the bracketed form agreeing with the bare one -- ``"Smith née Jones"`` has read given ``Smith`` since 1.4.0 -- because a marker-led clause is not a nickname clause and so does not reach the rule that would have made the word a family name. The empty ``family`` it leaves behind is the bare form's behavior and is not new here, though it is not 1.4.0's either -- v1 read ``"Smith née Jones"`` as first ``Smith``, middle ``née``, last ``Jones``, and #445 tracks whether the family should be kept. Seven of the 1,085 differential corpus names move against the 2.1.0 and 2.0.0 baselines and six against 1.4.0: the Japanese name is the odd one, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Eight corpus names have the shape and seven move (measured); the eighth is ``"Jane Smith (née Jr.)"``, which the suffix rule takes first. So the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is -- six of the seven are hand-written rule or issue examples, only the Japanese name arriving from elsewhere, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) - Fix a particle chain and a maiden name taking a trailing generational numeral as a name word: ``"John van der Berg V"`` read family ``van der Berg V`` and ``"John née Jones Smith V"`` read maiden ``Jones Smith V``, where ``"John Smith V"`` reads suffix ``V``. Both stops asked "is this a suffix?" with the test that vetoes a bare ``V`` as an initial -- the same question the bound given-name join asked until #401 -- and now ask the suffix rule's own walk, over the name as they would leave it: family ``van der Berg``, maiden ``Jones Smith``, suffix ``V``, for ``I`` and ``X`` alike; ``"John van der J. V"`` and ``"J. née Jones Smith V"`` keep their readings, the word before the numeral being an initial as written or as left. The chain stops before a bare credential with words to spare too, as ``"John Smith Ma"`` does, where the suffix rule will still find them once the chain has run: ``"John van der Berg Ma"`` reads suffix ``Ma``, which is how 1.4.0 read it, while ``"Freiherr von Berg MA"`` keeps family ``von Berg MA``; a maiden name keeps such a credential, since the words the marker takes are the very words that made it one. A particle that is also suffix vocabulary now ends the chain where it opens the trailing run: ``"John Smith Mc V"`` reads suffix ``Mc, V``. The chain also no longer swallows the given name behind an unlisted abbreviation: ``"Xyz. van Johnson"`` and ``"Esq. van Gogh"`` read given ``van``, as ``"Dr. van Johnson"`` has since #367, the chain and the bound given-name join now asking the same test that reads the abbreviation as a title (closes #424) diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index 5c0a2cf6..74d5ab5d 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -75,6 +75,25 @@ ) +#: The fields whose entries may be PHRASES: stored space-joined and +#: folded per word (_title_key), and exempt from the dead-entry warning +#: below. Every other field is matched one word at a time, where a +#: multi-word entry can never match. +#: +#: The two are not the same mechanism, and the difference is why the +#: exemption is a list rather than a rule. A given_name_titles run is +#: identified per word FIRST -- 'lt' and 'col' are each title +#: vocabulary -- and only then joined and looked up. A maiden marker +#: phrase has no such per-word foothold: 'z' and 'domu' are not markers +#: individually, and adding them separately (which this warning used to +#: advise) reads 'Maria Kowalska z domu Nowak' as maiden 'domu Nowak' +#: and strips 'Anna z Nowak' of its family name. So markers are matched +#: by a genuine multi-token lookahead instead -- +#: _pipeline._vocab.maiden_marker_run, longest first -- and only the +#: STORAGE rule is shared with titles. +_PHRASE_FIELDS = ("given_name_titles", "maiden_markers") + + def _normalize(word: str) -> str: """Lowercase, strip whitespace and EDGE periods -- v1's lc() semantics. Interior periods survive on purpose: 'J.R.' must not @@ -192,13 +211,14 @@ def _normset( raise TypeError( f"Lexicon.{field_name} entries must be strings, got {w!r}" ) - # given_name_titles is the one field whose entries are matched as - # a multi-word run, so it folds per word: stored as the same key - # post_rules builds, or 'lt. col' would be kept verbatim and - # never match anything (a silent no-op on the config surface). - # Every other field holds single words, where the two folds - # agree. - n = _title_key(w.split()) if field_name == "given_name_titles" \ + # A phrase field's entries are matched as a multi-word run, so + # they fold per word: stored as the same key the match site + # builds, or 'lt. col' (and 'z. domu') would be kept verbatim + # and never match anything -- a silent no-op on the config + # surface. Every other field holds single words, where the two + # folds agree. See _PHRASE_FIELDS for how the two differ once + # stored. + n = _title_key(w.split()) if field_name in _PHRASE_FIELDS \ else _normalize(w) # "." or "" is a data bug (stray split artifact, empty CSV # cell); dropping it silently would also let a data-module typo @@ -208,7 +228,7 @@ def _normset( f"Lexicon.{field_name} entry {w!r} normalizes to empty " f"(lowercase + strip periods/whitespace leaves nothing)" ) - # Every field but given_name_titles is matched one word at a + # Every field outside _PHRASE_FIELDS is matched one word at a # time, so a multi-word entry can never match -- the library # itself shipped eight such dead entries for years (repaired # 2026-07-26). Warn, never raise: an inert entry produces @@ -218,8 +238,10 @@ def _normset( # new instance's __post_init__; remove() stores nothing, so # warning there would name entries the caller is trying to get # RID of, with "split it" advice that makes no sense for a - # no-op. - if (warn and field_name != "given_name_titles" + # no-op. Both invariants are indifferent to WHICH fields are + # exempt: the exemption only decides whether a warning exists to + # be emitted once or suppressed. + if (warn and field_name not in _PHRASE_FIELDS # interior whitespace test; split() covers all Unicode # whitespace and n != "".join(n.split())): @@ -304,8 +326,12 @@ class Lexicon: Entries are normalized at construction -- lowercased, edge periods stripped -- so matching is case-insensitive. Vocabulary entries are single words -- a multi-word entry warns at construction and can - never match (``given_name_titles``, matched as a space-joined run, - is the one exception). Field docs below show examples, not full + never match. Two fields are exempt, and they differ in HOW they + match: ``given_name_titles`` is looked up as the space-joined run + of words the parse has ALREADY read as titles, while + ``maiden_markers`` is matched by lookahead, longest first, over + words that need not be markers on their own (``"z domu"``). + Field docs below show examples, not full contents; inspect any field's shipped vocabulary directly, e.g. ``Lexicon.default().conjunctions``.""" @@ -366,7 +392,12 @@ class Lexicon: #: :data:`~nameparser.config.bound_given_names.BOUND_GIVEN_NAMES`. bound_given_names: frozenset[str] = frozenset() #: Marker words introducing a birth surname, routed to the maiden - #: field ("née", "geb.", "rozená", ...). Full default list: + #: field ("née", "geb.", "rozená", ...). An entry may be a PHRASE + #: ("z domu"): entries are matched by lookahead, longest first, so + #: a phrase wins where it matches and a word entry that starts one + #: still matches on its own everywhere else. A phrase matches only + #: where its words stand together -- a bracketed clause or a comma + #: between them ends the run. Full default list: #: :data:`~nameparser.config.maiden_markers.MAIDEN_MARKERS`. maiden_markers: frozenset[str] = frozenset() #: Family names for the unspaced-name segmentation stage (#271), diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index d27d6964..4aa73964 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -1,6 +1,8 @@ """Stage: classify. -Consumes: tokens. +Consumes: tokens, comma_offsets (with token roles, the two halves of +the structural-boundary test the marker pass applies -- see +_tag_marker_runs). Produces: tokens with vocabulary tags added (text/span/role unchanged). Reads: every Lexicon vocabulary field; no Policy FIELD is consulted (is_initial does consult the _policy module's _NO_INITIALS constant, @@ -9,7 +11,13 @@ Tags emitted -- stable (API): "particle", "conjunction", "initial"; namespaced (unstable): "vocab:title", "vocab:given-title", "vocab:suffix", "vocab:suffix-word", "vocab:suffix-ambiguous", -"vocab:particle-ambiguous", "vocab:bound-given", "vocab:maiden-marker". +"vocab:particle-ambiguous", "vocab:bound-given", "vocab:maiden-marker", +"vocab:maiden-marker-cont". +"vocab:maiden-marker" tags the HEAD of a maiden marker, which is a +whole marker whenever the marker is one word; the continuation tag +carries the rest of a PHRASE marker ("z domu"), so a site asking +"does a marker start here" reads the same tag it always did and a site +asking "where does it end" walks the continuations. "vocab:suffix" means "counts as a suffix as written": unambiguous suffix vocabulary, or an ambiguous acronym written with periods -- at the TAG level 'M.A.' gets "vocab:suffix" while 'Ma' gets only @@ -22,14 +30,16 @@ from __future__ import annotations import dataclasses +from collections.abc import Sequence from nameparser._lexicon import _normalize from nameparser._pipeline._state import ( - ParseState, PendingAmbiguity, WorkToken, + ParseState, PendingAmbiguity, WorkToken, comma_bucket, ) from nameparser._types import AmbiguityKind, Role from nameparser._pipeline._vocab import ( - is_initial, period_joined_vocab, suffix_as_written, + _longest_marker, is_initial, maiden_marker_head, maiden_marker_run, + period_joined_vocab, suffix_as_written, ) @@ -42,10 +52,17 @@ # abbreviation shape any word can wear and does not. A # bare ambiguous acronym is consumed only when the name has words to # spare" -def _tags_for(token: WorkToken, state: ParseState) -> frozenset[str]: +def _tags_for(token: WorkToken, n: str, state: ParseState, + marker_tag: str | None) -> frozenset[str]: + """`n` is _normalize(token.text), folded once by the caller and + shared with the marker pass; `marker_tag` is what that pass decided + for this token, or None. The marker DECISION is entirely + _tag_marker_runs'; only the writing happens here, so the two tokens + of a phrase are built once rather than replaced twice.""" lex = state.lexicon - n = _normalize(token.text) tags = set(token.tags) + if marker_tag is not None: + tags.add(marker_tag) if n in lex.titles: tags.add("vocab:title") if n in lex.given_name_titles: @@ -66,8 +83,11 @@ def _tags_for(token: WorkToken, state: ParseState) -> frozenset[str]: tags.add("conjunction") if n in lex.bound_given_names: tags.add("vocab:bound-given") - if n in lex.maiden_markers: - tags.add("vocab:maiden-marker") + # maiden markers are NOT tagged here: an entry may be a phrase whose + # words are not markers on their own, and this function sees one + # token with no neighbours. _tag_marker_runs below does the whole + # field, single words included, so there is one place that decides + # it (mechanisms.md#ONE-PREDICATE-PER-QUESTION). if is_initial(token.text): tags.add("initial") # v1's period-joined derivation (parse_pieces): a token with a @@ -84,10 +104,111 @@ def _tags_for(token: WorkToken, state: ParseState) -> frozenset[str]: return frozenset(tags) +def _tag_marker_runs(state: ParseState, + folded: Sequence[str]) -> dict[int, str]: + """Which tokens are maiden marker runs: index -> "vocab:maiden-marker" + for a run's head, "vocab:maiden-marker-cont" for the rest. + + Returns the decision rather than rewriting the tokens; classify + writes it into the one pass that builds them, so a marker token is + not replaced twice. `folded` is _normalize per token, computed once + for this pass and the vocabulary tags alike. + + The one sequence pass in this stage, and it has to be one: a marker + entry may be a PHRASE whose words are not markers individually + ('z', 'domu'), so no per-token membership test can find it. + Left to right, longest first at each position, then skip past what + the run claimed -- a second marker cannot start inside the first. + + This is where the tag is DECIDED for the two stages that read it + afterwards. group runs later and asks its questions of these tags + rather than re-deriving the run (the recorded-answer half of + mechanisms.md#ONE-PREDICATE-PER-QUESTION); extract runs EARLIER, + before tokens exist, so it calls the predicate itself over the + clause's whitespace words. + + A tagged run is structurally contiguous, and the test is + one-directional: a role change IS a clause edge, so no run spans + one, but not every clause edge is a role change -- two ADJACENT + clauses of the same role are indistinguishable here, and + 'Jane (z) (domu) Jones' does tag a run across them. Both consumers + refuse that run for reasons of their own (the piece walk never sees + role-bearing tokens at all; the clause drop is scoped to one + clause's span), so no reading depends on it today, and the claim + this pass can honestly make is the weaker one. What it does + guarantee is what _group._marker_run_pieces needs: a run inside the + MAIN stream stays inside one segment. Without it this pass walked + the whole span-sorted stream while group walked one segment -- + _segment keeps only role-less tokens and buckets them by the commas + before them -- so a run half inside a bracketed clause was tagged + whole and consumed as a proper PREFIX of itself, and + 'Anna z (domu) Nowak' read family 'Anna', maiden 'Nowak': the bare + preposition eating the name, which is the exact damage the phrase + entry exists to prevent. Refusing to tag such a run is the fix; + truncating it instead would hand M2 the same wrong prefix one word + shorter. + """ + markers = state.lexicon.maiden_markers + # the lookahead the vocabulary actually needs; 0 for an empty set, + # which skips the pass entirely + cap = _longest_marker(markers) + if not cap: + return {} + tokens = state.tokens + n_tokens = len(tokens) + # Deferred, not computed up front: only the contiguity walk reads + # it, only a phrase vocabulary runs that walk, and only at a token + # that opens an entry -- so a single-word vocabulary, and a + # phrase vocabulary over a name holding no marker, never pay the + # sweep at all. + buckets: list[int] | None = None + tags: dict[int, str] = {} + i = 0 + while i < n_tokens: + # The predicate's own head test first, over the fold the caller + # already has: almost no token opens any entry, and for those + # there is nothing to assemble. Same function maiden_marker_run + # consults, so a token skipped here is one it would refuse. + if not maiden_marker_head(folded[i], markers): + i += 1 + continue + # Bound the lookahead at the first structural boundary, so the + # predicate is asked over the words that could form one run and + # answers longest-first WITHIN them -- a two-word entry refused + # at a clause edge still leaves a one-word entry starting there + # free to match. + limit = 1 + if cap > 1: + if buckets is None: + buckets = [comma_bucket(t.span.start, state.comma_offsets) + for t in tokens] + role, bucket = tokens[i].role, buckets[i] + while (limit < cap and i + limit < n_tokens + and tokens[i + limit].role is role + and buckets[i + limit] == bucket): + limit += 1 + run = maiden_marker_run( + [tokens[k].text for k in range(i, i + limit)], markers) + if not run: + i += 1 + continue + tags[i] = "vocab:maiden-marker" + for k in range(i + 1, i + run): + tags[k] = "vocab:maiden-marker-cont" + i += run + return tags + + def classify(state: ParseState) -> ParseState: + # One fold per token, shared by the marker pass and the vocabulary + # tags -- the shape suffix_as_written already asks for ("n is + # _normalize(text), passed in so callers normalize once"). + folded = [_normalize(t.text) for t in state.tokens] + marker_tags = _tag_marker_runs(state, folded) tokens = tuple( - dataclasses.replace(t, tags=_tags_for(t, state)) - for t in state.tokens) + dataclasses.replace( + t, tags=_tags_for(t, folded[i], state, marker_tags.get(i))) + for i, t in enumerate(state.tokens)) # Delimited content whose vocabulary cannot settle it: extract's # escape sends an UNambiguous suffix straight through ("(MBA)" -> # suffix) and keeps everything else as a nickname, so an AMBIGUOUS diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index da5f876c..a0f365dd 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -4,9 +4,10 @@ Produces: extracted (role + inner span per delimited region), masked (full regions incl. delimiter chars, skipped by tokenize), UNBALANCED_DELIMITER ambiguities for opens with no close. -A Role.MAIDEN region is the WHOLE inner span, marker word included -- -nothing here strips one. classify tags a marker inside it like any -other token, and group drops it from a multi-token clause (#329). +A Role.MAIDEN region is the WHOLE inner span, marker included (all of +it, a phrase marker being several words) -- nothing here strips one. +classify tags a marker inside it like any other token, and group drops +it from a multi-token clause (#329). A region reaches that role two ways: the pair that matched sits in Policy.maiden_delimiters (M1), or the content itself opens with a marker word (M3), which reassigns the role after the match and so is @@ -38,6 +39,7 @@ from nameparser._pipeline._state import ( COMMA_CHARS, ParseState, PendingAmbiguity, ) +from nameparser._pipeline._vocab import maiden_marker_run from nameparser._types import AmbiguityKind, Role, Span @@ -58,21 +60,36 @@ def _suffix_shaped(content: str, lexicon: Lexicon) -> bool: # rules.md#M3: "a bracketed clause whose content opens with a -# recognized marker word and carries a word after it reads as the +# recognized marker and carries a word after it reads as the # maiden name, whichever bucket the enclosing pair sits in" def _maiden_marked(content: str, lexicon: Lexicon) -> bool: """The clause says 'maiden' out loud, so the caller does not have to say it in Policy. Requires a word AFTER the marker: a lone marker in brackets is a word in brackets, and M1 deliberately keeps a one-word - clause's word (it may be the surname Nee). The word after is not - tested for anything -- M3's Accepted line, and the reason a + clause's word (it may be the surname Nee). A PHRASE marker is one + marker, so the word after is the word after the whole run -- + '(z domu Nowak)' has one, '(z domu)' has none. The word after is + not tested for anything -- M3's Accepted line, and the reason a bracketed '(née V)' reads maiden 'V' where the bare 'née V' gives - M2 a suffix. Whitespace-split, so a marker the writer glued to - punctuation is not one here ('née,'); the tokenizer splits that - comma off and still tags the token, which is what keeps _group's - Role.MAIDEN filter reachable.""" + M2 a suffix. + + This stage runs before tokenize, so it calls maiden_marker_run + itself rather than reading the tags classify will set. The two + questions are deliberately not the same one: this splits on + WHITESPACE, so a marker the writer glued to punctuation is not one + here ('née,'); the tokenizer splits that comma off and classify + still tags the token, which is what keeps _group's Role.MAIDEN + filter reachable. Sharing the PREDICATE does not merge the two + questions -- each hands it a different sequence of words.""" words = content.split() - return len(words) > 1 and _normalize(words[0]) in lexicon.maiden_markers + # A one-word clause cannot satisfy the condition whatever the + # vocabulary says -- `run > 0 and 1 > run` is unsatisfiable -- so + # refuse it before any fold or lookup, as the pre-#434 words[0] + # test did for free. + if len(words) < 2: + return False + run = maiden_marker_run(words, lexicon.maiden_markers) + return run > 0 and len(words) > run # rules.md#N2: "a quote whose open and close are the same character diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 16d39f51..552f2bbc 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -36,7 +36,7 @@ import bisect import dataclasses -from collections.abc import Sequence, Set +from collections.abc import Iterable, Sequence, Set from enum import IntEnum from nameparser._lexicon import _title_key @@ -59,8 +59,9 @@ # wholly-suffix post-comma run, both of which see the merged reading Piece = list[int] -#: What the marker pass took out of a segment: the marker piece (to be -#: dropped) and the maiden-name pieces (to take Role.MAIDEN). +#: What the marker pass took out of a segment: the marker's tokens (to +#: be dropped -- more than one where the entry is a phrase, 'z domu') +#: and the maiden-name pieces (to take Role.MAIDEN). MaidenTake = tuple[Piece, list[Piece]] @@ -125,12 +126,75 @@ def _is_maiden_marker_piece(piece: Sequence[int], and "vocab:maiden-marker" in tokens[piece[0]].tags) +# mechanisms.md#ONE-PREDICATE-PER-QUESTION: "where the reader comes +# AFTER the decider, record the answer on the state instead" -- which is +# what the marker tags are. classify owns the phrase lookahead; group, +# running later, reads what it wrote. +def marker_run_length(following: Iterable[Set[str]]) -> int: + """How many tokens a marker run spans, given the tag sets of what + FOLLOWS its head, in order: 1 plus the leading continuations. + + One walk for both places group asks -- over pieces below and over + token indices in the clause drop at the end of this module, 700 + lines apart and differing only in index space. `following` is + consumed lazily and only until the run ends, so a caller passes a + generator over the rest of its sequence rather than materializing + one.""" + run = 1 + for tags in following: + if "vocab:maiden-marker-cont" not in tags: + break + run += 1 + return run + + +def _marker_run_pieces(seen: Sequence[int], pieces: Sequence[Sequence[int]], + tokens: Sequence[WorkToken], m: int) -> int: + """How many of `seen`'s pieces the marker run at seen[m] spans: 1 + for a single-word marker, more for a phrase entry ('z domu'). + + classify already decided where the run ends and recorded it on the + tokens -- "vocab:maiden-marker" on the head, + "vocab:maiden-marker-cont" on the rest. + + Each continuation is the NEXT piece and is always in `seen`, and + that holds because classify REFUSES to tag a run whose tokens are + not structurally contiguous. It is not a property of this walk, and + the reasons `seen` can skip a token are wider than they look. No + join has run yet, so a piece is one token. `seen` itself skips a + tail segment's delimiter cores, and a core between two marker words + would be a token between them, which classify would not have tagged + as a run. But `pieces` comes from a SEGMENT, and segment keeps only + the tokens no stage has given a role, bucketed by the commas before + them -- so a run half inside a bracketed clause, or split across a + structure comma, is one no segment holds whole. Walking cont tags + without classify's refusal read a proper PREFIX of the phrase as + the whole marker, and 'Anna z (domu) Nowak' lost its given name to + a bare preposition. Any future stage that removes a token from a + segment owes this the same refusal. + + What classify guarantees is exactly the main stream's half of that + and no more: a run of ROLE-LESS tokens stays inside one segment. It + can also tag a run across two adjacent clauses of the same role, + which this walk never sees because a role-bearing token is in no + segment at all -- see _tag_marker_runs, which states the limit. + """ + return marker_run_length( + tokens[pieces[seen[k]][0]].tags for k in range(m + 1, len(seen))) + + def _maiden_take(pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken], - cores: Set[str]) -> list[int] | None: - """The indices of the pieces the marker pass removes, the marker - first, or None. + cores: Set[str]) -> tuple[list[int], list[int]] | None: + """The piece indices the marker pass removes, split the way + MaidenTake declares them: the MARKER's pieces (one, or several for + a phrase entry like 'z domu') and the maiden name's. None when the + pass declines. + + Split here rather than by the caller because `run` is known here + and nowhere else -- it comes off the continuation tags, which are + read to find the walk's starting point in the first place. Computed before any join (the Ph. D. merge aside), so "up to any trailing suffix" means the first suffix WORD after the marker: a @@ -158,6 +222,8 @@ def _maiden_take(pieces: Sequence[Sequence[int]], if _is_maiden_marker_piece(pieces[seen[v]], tokens)), None) if m is None: return None + # the marker may be a phrase, in which case it is several pieces + run = _marker_run_pieces(seen, pieces, tokens, m) # "up to any trailing suffix": a suffix WORD anywhere after the # marker ends the maiden name, and so does the trailing numeral as # assign will read it, which the suffix-piece test does not see @@ -190,14 +256,18 @@ def _maiden_take(pieces: Sequence[Sequence[int]], view, view_tags, tokens, numeral_only=True) == len(view): trailing = len(pieces) - j = m + 1 + j = m + run while (j < len(seen) and seen[j] < trailing and not is_suffix_piece(pieces[seen[j]], ptags[seen[j]], tokens)): j += 1 - # j == m + 1 means nothing followed the marker but a suffix, so the - # pass declines and the marker stays an ordinary word (rules.md#M2). - return seen[m:j] if j > m + 1 else None + # j == m + run means nothing followed the marker but a suffix, so + # the pass declines and the marker stays ordinary words + # (rules.md#M2). The walk starts past the WHOLE marker: a phrase's + # second word is the marker, not the first word it takes. + if j <= m + run: + return None + return seen[m:m + run], seen[m + run:j] # rules.md#P3: "a recognized connective joins its neighbors into one @@ -330,8 +400,10 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), taken: MaidenTake | None = None take = _maiden_take(pieces, ptags, tokens, cores) if take is not None: - taken = (pieces[take[0]], [pieces[k] for k in take[1:]]) - for k in reversed(take): + marker_ks, maiden_ks = take + taken = ([i for k in marker_ks for i in pieces[k]], + [pieces[k] for k in maiden_ks]) + for k in reversed(marker_ks + maiden_ks): del pieces[k] del ptags[k] @@ -791,9 +863,10 @@ def group(state: ParseState) -> ParseState: tokens[i], tags=tokens[i].tags | {"joined"}) all_pieces.append(tuple(tuple(p) for p in pieces)) all_ptags.append(tuple(frozenset(t) for t in ptags)) - # rules.md#M1: "a leading recognized marker word inside a - # multi-word clause being dropped; a one-word clause keeps its - # word" — a marker inside EXTRACTED maiden content (#329). + # rules.md#M1: "a leading recognized marker being dropped where + # the clause holds a word past it; a clause of nothing but its + # marker keeps its words" — a marker inside EXTRACTED maiden + # content (#329). # classify tags # such a marker like any other token -- what the #274 rule above # lacks is not the TAG but the token: extract claims a delimited @@ -813,13 +886,16 @@ def group(state: ParseState) -> ParseState: # * Separate clauses are separate content. In "(Nee) (Jones)" the # two land as one contiguous run of maiden tokens, so only the # clause bound keeps the lone "(Nee)" intact. - # Drop the clause's FIRST token only when the clause holds more - # than one: `Nee` is a real surname (Irish Ni/Nee, and a Chinese - # romanization), so a one-token "(Nee)" is a maiden name, not a - # marker. FIRST token and no more, whatever the clause holds past - # it: cases.py's maiden_marker_delimited_three_token_clause is the - # row that bounds this in both directions, every other delimited - # row having a two-token clause where the two readings agree. + # Drop the clause's LEADING MARKER only when the clause holds a + # word past it: `Nee` is a real surname (Irish Ni/Nee, and a + # Chinese romanization), so a one-token "(Nee)" is a maiden name, + # not a marker. The marker and no more, whatever the clause holds + # past it: cases.py's maiden_marker_delimited_three_token_clause is + # the row that bounds this in both directions, every other + # delimited row having a two-token clause where the two readings + # agree. The marker is one token for a word entry and the whole run + # for a phrase one ('z domu'), which is why the loop below counts + # continuation tags rather than assuming one. # Spans index the original string by the anti-#100 # invariant, and script_segment only ever splits a token into # sub-slices, so containment stays exact. @@ -843,15 +919,25 @@ def group(state: ParseState) -> ParseState: # first token starting at or after the clause opens; tokens # are span-sorted and group never reorders or resizes them first = bisect.bisect_left(starts, clause.start) - # Testing the SECOND token's end proves BOTH are inside: - # tokens do not overlap, so first.end <= second.start, and - # bisect already put first.start at or after clause.start. - # That is also the "more than one token" test, since the - # tokens inside a clause are contiguous in index order. - if (first + 1 < len(tokens) - and tokens[first + 1].span.end <= clause.end - and "vocab:maiden-marker" in tokens[first].tags): - dropped.append(first) + if (first >= len(tokens) + or "vocab:maiden-marker" not in tokens[first].tags): + continue + # How many tokens the marker is: one, or the whole run of a + # phrase entry, as classify recorded it. Same walk the + # piece test above makes, in token-index space. + run = marker_run_length( + tokens[k].tags for k in range(first + 1, len(tokens))) + # Testing the end of the token AFTER the run proves the run + # AND that token are all inside: tokens do not overlap and + # are index-ordered, so every earlier one ends no later, + # and bisect already put first.start at or after + # clause.start. That is also the "more than the marker" + # test, since the tokens inside a clause are contiguous in + # index order -- a clause holding nothing but its marker + # keeps its words, exactly as a one-token "(Nee)" does. + if (first + run < len(tokens) + and tokens[first + run].span.end <= clause.end): + dropped.extend(range(first, first + run)) return dataclasses.replace( state, tokens=tuple(tokens), pieces=tuple(all_pieces), piece_tags=tuple(all_ptags), dropped=tuple(dropped), diff --git a/nameparser/_pipeline/_segment.py b/nameparser/_pipeline/_segment.py index a82c7feb..5e7ae560 100644 --- a/nameparser/_pipeline/_segment.py +++ b/nameparser/_pipeline/_segment.py @@ -16,10 +16,11 @@ """ from __future__ import annotations -import bisect import dataclasses -from nameparser._pipeline._state import ParseState, PendingAmbiguity, Structure +from nameparser._pipeline._state import ( + ParseState, PendingAmbiguity, Structure, comma_bucket, +) from nameparser._pipeline._vocab import is_wholly_suffix from nameparser._types import AmbiguityKind @@ -37,11 +38,12 @@ def segment(state: ParseState) -> ParseState: structure=Structure.NO_COMMA) buckets: list[list[int]] = [[] for _ in range(len(state.comma_offsets) + 1)] for i in main: - # comma_offsets is sorted and no offset ever equals a token - # start, so bisect_left counts the commas before this token - start = state.tokens[i].span.start - bucket = bisect.bisect_left(state.comma_offsets, start) - buckets[bucket].append(i) + # _state.comma_bucket, not a local bisect: classify asks the + # same question of the same offsets to keep a marker run inside + # one segment, and the two must be one expression rather than + # two that agree + buckets[comma_bucket(state.tokens[i].span.start, + state.comma_offsets)].append(i) groups = [tuple(b) for b in buckets] # v1 strips exactly ONE trailing comma as cosmetic (parser.py's # collapse_whitespace); every other empty bucket is STRUCTURAL and diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 5382543b..a8ea880c 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -10,6 +10,8 @@ """ from __future__ import annotations +import bisect +from collections.abc import Sequence from dataclasses import dataclass from enum import Enum, auto @@ -23,6 +25,23 @@ # boundaries) cannot drift apart. COMMA_CHARS = frozenset({",", "\u060c", "\uff0c"}) + +def comma_bucket(start: int, comma_offsets: Sequence[int]) -> int: + """Which comma-delimited part of the name a token starting at + `start` falls in: the number of commas before it. + + Shared here for the reason COMMA_CHARS is, and the sharing is + load-bearing in the same way. segment BUILDS the segments with this + (comma_offsets is sorted and no offset ever equals a token start, + so bisect_left counts the commas before the token); classify asks + it to decide whether two tokens could be in one segment, which is + half of what stops a maiden marker run from spanning a boundary no + segment holds. Two tokens agree here iff segment would put them in + one segment, and that is an identity rather than a resemblance + only while both sides ask this function. + """ + return bisect.bisect_left(comma_offsets, start) + @dataclass(frozen=True, slots=True) class WorkToken: """One tokenized word. role stays None until assign; extracted diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index f9e697d2..da342eff 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -15,10 +15,17 @@ more than the config parameter saves. Still no state: Policy is frozen config, not pipeline state. +maiden_marker_run is run-level for the first of those reasons and not +the second: a maiden marker may be a PHRASE ('z domu'), so how far one +reaches is a question about a run of words that no per-word membership +test can answer, and the vocabulary reaches it as a plain frozenset +field like every other predicate here. + Layering: imports _lexicon, _types, and _policy only. """ from __future__ import annotations +import functools import re import unicodedata from collections.abc import Callable, Iterable, Sequence @@ -288,6 +295,108 @@ def counts_as_suffix(text: str) -> bool: return all(counts_as_suffix(t) for t in merged) +# The two derived views of a marker vocabulary, cached per-vocabulary +# on _script_segment._longest_entry's precedent -- same function shape +# (a scalar derived from a vocabulary frozenset), same key space, and +# its reasoning for maxsize=16 carries over verbatim: a process holds +# the default vocabulary plus one per constructed pack parser, so 16 +# bounds many-lexicon churn without ever evicting in normal use. +# (NOT _extract._delimiter_chars' precedent, which these once cited: +# that one is consulted once per parse, so it says nothing about a +# lookup on the per-token path.) Keying on the frozenset costs a +# cached hash, not a sweep of its contents. +@functools.lru_cache(maxsize=16) +def _longest_marker(markers: frozenset[str]) -> int: + """How many words the longest entry of `markers` spans -- the + lookahead bound, computed from the vocabulary rather than fixed at a + literal so a caller's four-word entry works and an all-single-word + set leaves the common path at one lookup. Entries are stored + space-joined with single separators, so the space count IS the word + count.""" + return max((entry.count(" ") + 1 for entry in markers), default=0) + + +@functools.lru_cache(maxsize=16) +def _marker_heads(markers: frozenset[str]) -> frozenset[str]: + """The first word of every entry -- the set a run can possibly open + with. Entries are already stored per-word folded, so an entry's + first word is the same fold the lookup builds.""" + return frozenset(entry.split(" ", 1)[0] for entry in markers) + + +def maiden_marker_head(n: str, markers: frozenset[str]) -> bool: + """Could a maiden marker run START here? `n` is _normalize(word), + passed in so callers fold once -- suffix_as_written's shape exactly + ("`n` is `_normalize(text)`, passed in so callers normalize once"), + and with no raw-text sibling for the same reason it has none: every + caller is on the per-token path and has the fold in hand already. + + A SUPERSET test. True means only that some entry opens with this + word, never that a run matches -- maiden_marker_run is the answer, + and it calls this function, so the two cannot drift. Exported + because a caller scanning a whole token stream needs to know + whether assembling a candidate sequence is worth doing at all, and + for almost every token it is not: _classify's pass would otherwise + walk structural boundaries once per token to build a lookahead the + predicate discards on this very test. + """ + return n in _marker_heads(markers) + + +def maiden_marker_run(words: Sequence[str], markers: frozenset[str]) -> int: + """How many of `words` a maiden marker claims, longest first; 0 for + none. + + Phrases are stored space-joined and per-word normalized (_title_key's + storage rule), so the key is rebuilt the same way here -- normalizing + the joined phrase instead would leave interior periods. + + `words` must be words that stand TOGETHER -- one clause's, or one + segment's. This answers only what the vocabulary says about the + sequence it is handed; whether a sequence is a sequence is the + caller's, and classify's contiguity rule is where that is decided + for the token stream. + + Longest first, so a caller configuring both 'geb' and 'geb von' gets + the phrase where it matches and the bare word everywhere else. The + one answer to "does a marker start here, and where does it end": the + stages that can call it do (classify over token texts, extract over + a clause's whitespace words), and the stage that runs after classify + reads the tags classify recorded instead + (mechanisms.md#ONE-PREDICATE-PER-QUESTION). + """ + # Fast path first: no entry opens with this word, so no length can + # match. It cannot hide a match -- every key the loop builds opens + # with _normalize(words[0]) unless that word folds away, and a + # folded-away first word fails the n-word test below for every + # n > 1 and is not a stored entry for n == 1. + if not words: + return 0 + head = _normalize(words[0]) + if not maiden_marker_head(head, markers): + return 0 + cap = min(len(words), _longest_marker(markers)) + # Fold each word ONCE, then key from a prefix. _title_key(words[:n]) + # per candidate length re-folds the whole prefix every time, which + # is quadratic in cap and makes a one-word hit cost more than a + # two-word one -- the longest key is always built and discarded + # first, and all but one shipped entry is a single word. The join + # below IS _title_key's body over pre-folded words (per-word fold, + # empties dropped, space-joined); test_vocab pins that the two + # agree, since this is a copy of a fold whose definition lives in + # _lexicon. + folded = [head] + [_normalize(w) for w in words[1:cap]] + for n in range(cap, 0, -1): + key = " ".join(filter(None, folded[:n])) + # a word that folds away is DROPPED from the key, so 'née' + # followed by a lone '.' would key as 'née' and a one-word + # marker would claim the period as part of its run. n words in, + # n words out: anything else is not this key's n-word phrase. + if key.count(" ") == n - 1 and key in markers: + return n + return 0 + + def _normalized_for_script(text: str) -> str | None: """The guard AND the NFC normalization single_script and effective_script's license path both need, single-sourced so they diff --git a/nameparser/_policy.py b/nameparser/_policy.py index a3000f04..8700d8df 100644 --- a/nameparser/_policy.py +++ b/nameparser/_policy.py @@ -630,10 +630,12 @@ class Policy: #: pair encloses it, so "Jane Smith (née Jones)" needs no #: configuration -- unless the content is suffix-shaped, which is #: taken first, the brackets dropped and the content read as if - #: written bare. A maiden_markers word opening the enclosed - #: content is dropped from the value, but only where that content - #: holds more than one token: a lone "(Nee)" is a maiden NAME, - #: not a marker (#329). + #: written bare. A maiden_markers entry opening the enclosed + #: content is dropped from the value -- all of it, an entry being + #: allowed to span more than one word ("z domu") -- but only where + #: a word stands past it: a clause of nothing but its marker keeps + #: its words, since a lone "(Nee)" is a maiden NAME and not a + #: marker (#329). maiden_delimiters: frozenset[tuple[str, str]] = frozenset() #: Additional separators that split suffix groups (e.g. " - " for #: "Jane Smith, RN - CRNA"). Additive only: the comma always diff --git a/nameparser/config/_invariants.py b/nameparser/config/_invariants.py index 98969132..366981b6 100644 --- a/nameparser/config/_invariants.py +++ b/nameparser/config/_invariants.py @@ -12,6 +12,26 @@ module cannot import ``_lexicon`` anyway (``_lexicon`` imports these constants). The relationship checks between constants stay in the modules that own them -- those encode facts about the data, not hygiene. + +Interior whitespace is checked but not forbidden. A PHRASE entry is +legitimate in the two fields ``_lexicon._PHRASE_FIELDS`` names, and +``MAIDEN_MARKERS``'s ``'z domu'`` is the first one shipped; what this +asserts of one is that it is stored as the single-spaced, lowercase, +edge-clean form, so ``'z domu'`` and ``'Z domu '`` fail here rather +than becoming entries nothing can match. The remaining half of a +phrase's storage rule -- that each WORD is separately stripped of its +periods, so ``'z. domu'`` is stored ``'z domu'`` -- is +``_lexicon._title_key``'s and is deliberately NOT checked here. The +reason is altitude, not layering: importing ``_lexicon`` from here +would in fact work (its config imports are all inside +``_default_lexicon()``, nothing at module scope), but a data module +asserting things with the parser's own fold makes the constant's +hygiene depend on the parser, and the question worth asking is not +"is this entry pre-folded" but "does ``Lexicon`` store it unchanged". +That is one assertion over ``_PHRASE_FIELDS`` in +``tests/v2/test_ledger_guards.py``, the suite that already imports the +parser's fold for exactly this purpose, and it is where an entry +written ``'z. domu'`` is caught. """ from __future__ import annotations @@ -19,9 +39,15 @@ def assert_normalized(name: str, entries: Iterable[str]) -> None: - """Assert every entry is stored lowercase and free of edge whitespace.""" - offenders = sorted(w for w in entries if w != w.strip().lower()) + """Assert every entry is stored lowercase, edge-clean, and -- where + it is a phrase -- single-spaced.""" + # " ".join(w.split()) rather than w.strip(): it subsumes the strip + # and additionally collapses interior runs, which is what a phrase + # entry made worth checking. For a single word the two are + # identical, so no existing constant's verdict changes. + offenders = sorted(w for w in entries if w != " ".join(w.lower().split())) assert not offenders, ( - f"{name} entries must be stored lowercase and whitespace-free; " + f"{name} entries must be stored lowercase, without edge " + f"whitespace, and single-spaced if a phrase; " f"offending: {offenders}" ) diff --git a/nameparser/config/maiden_markers.py b/nameparser/config/maiden_markers.py index e9d26265..afd0543f 100644 --- a/nameparser/config/maiden_markers.py +++ b/nameparser/config/maiden_markers.py @@ -16,26 +16,45 @@ 'урожденная', 'урождённый', 'урожденный', + 'z domu', '旧姓', }) """ -Marker words that introduce a birth surname, e.g. "Jane Smith née Jones" +Markers that introduce a birth surname, e.g. "Jane Smith née Jones" (#274). French née/né/nee, German geb./geborene, Dutch geboren, Czech/Slovak rozená, Danish/Norwegian født (Nynorsk fødd), Swedish född, Russian урожд./урождённая/урождённый (both ё and е spellings — case normalization does not fold them, and running text routinely -writes е). Both grammatical genders are listed where #274 or review -attested them (née/né, урождённая/урождённый); Czech masculine rozený -awaits the same vetting, and would be vetted on its own merits rather -than following the feminine form in — the abbreviation roz. shared by -both genders is deliberately absent, below. Entries are stored -normalized: lowercase, no periods. +writes е), and Polish "z domu" (#434). Both grammatical genders are +listed where #274 or review attested them (née/né, +урождённая/урождённый); Czech masculine rozený awaits the same +vetting, and would be vetted on its own merits rather than following +the feminine form in — the abbreviation roz. shared by both genders is +deliberately absent, below. Entries are stored normalized: lowercase, +no periods. + +An entry may be a PHRASE, and "z domu" ("of the house") is the first +one shipped. Phrases are stored space-joined and folded per word, and +matched by lookahead, longest first, over a run of whole tokens that +stand TOGETHER -- a phrase divided by a bracketed clause or a comma is +not that phrase, so "Anna z (domu) Nowak" keeps its family name just as +"Anna z Nowak" does. That mechanism is what makes this entry safe where +its words are not: z is an ordinary Polish preposition and domu an +ordinary noun, and the whole-phrase claim never fires on either alone. +Splitting a phrase into its words is what the library used to advise +(the dead-entry warning, which no longer fires for this field), and it +produced exactly the misreadings the phrase avoids: +maiden "domu Nowak" for "Maria Kowalska z domu Nowak", and no family +name at all for "Anna z Nowak". A caller who added z and domu +separately should remove both and let the shipped entry work. Japanese 旧姓 is here rather than in locales.JA, on the rule that admitted the Cyrillic entries: a native-script marker cannot collide with a Latin-script name, and matching is whole-token, so it is safe -as a default. Neither character appears in any shipped surname, title, -suffix, conjunction, particle or bound-given vocabulary. locales.JA +as a default. (Whole-token is still the rule for a phrase entry: it +claims a run of whole tokens, one per word, and no part of one.) +Neither character appears in any shipped surname, title, suffix, +conjunction, particle or bound-given vocabulary. locales.JA is for what needs the my-data-is-Japanese declaration -- segmentation, where a pure-Han string cannot say which language wrote it -- and this needs none, since it can only ever match Han text. @@ -73,8 +92,7 @@ Consumed by the 2.0 parser's default lexicon. The 1.x parser does not read this module. -Deliberately absent: Polish "z domu" (a two-token marker; pending the -2.0 pipeline's multi-token matching decision), the Scandinavian +Deliberately absent: the Scandinavian abbreviation "f." (collides with the initial "F." — only the full participles are safe), and the Czech/Slovak abbreviation "roz." (shipped through 2.1 and removed in 2.2). Roz is an ordinary English diff --git a/tests/v2/cases.py b/tests/v2/cases.py index b78b278f..7c2334f1 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -795,6 +795,88 @@ def __post_init__(self) -> None: {"given": "Jane", "family": "Smith", "maiden": "Jones"}, classification="fix(#274)", notes="v1 mangles to middle='Smith née'"), + Case("phrase_marker_takes_the_maiden_name", + "Maria Kowalska z domu Nowak", + {"given": "Maria", "family": "Kowalska", "maiden": "Nowak"}, + classification="fix(#434)", + notes="the fork a PHRASE entry opens: a marker matched over " + "more than one token. 'z domu' is the first multi-word " + "entry any shipped vocabulary set holds, so this is the " + "row where the multi-token branch of the lookahead is " + "taken at all -- and the row that pins how many tokens " + "the take drops. With the marker run forced back to one " + "piece this reads maiden 'domu Nowak' (measured " + "2026-08-26), which is also what the library's former " + "advice produced: the dead-entry warning used to say " + "'split it into separate entries', and z plus domu as " + "two entries gives exactly that. 1.4.0 read first Maria " + "/ middle 'Kowalska z domu' / last Nowak (2026-08-26) " + "-- the marker inside the name, its ordinary reading of " + "every marker. phrase_marker_partial_is_not_a_marker " + "and preposition_alone_is_not_a_marker below are the " + "boundaries"), + Case("phrase_marker_partial_is_not_a_marker", + "Maria Kowalska z Nowak", + {"given": "Maria", "middle": "Kowalska z", "family": "Nowak"}, + notes="the boundary above it: the phrase's first word with " + "the second one missing. A lookahead that settled for a " + "PREFIX of an entry would find a marker here and read " + "family Kowalska, maiden Nowak (measured 2026-08-26 " + "with the match relaxed to entries starting with the " + "key). Nothing about z is marker-ish on its own -- it " + "is an ordinary Polish preposition -- which is the " + "whole reason the entry is a phrase. Parity"), + Case("preposition_alone_is_not_a_marker", "Anna z Nowak", + {"given": "Anna", "middle": "z", "family": "Nowak"}, + notes="the same boundary with only one name word ahead of " + "the preposition, which is where the damage of getting " + "it wrong is worst: under the split-entry workaround " + "the library used to advise, a bare z IS a marker and " + "M2 hands it every word after it, so this name reads " + "maiden 'Nowak' with NO family name at all (measured " + "2026-08-26). This is the row that would have caught " + "that advice. It is also the shape " + "diminutive_that_was_a_marker_keeps_the_family pins for " + "the roz collision -- a marker entry that is also an " + "ordinary word eats the family name, and a phrase entry " + "is how that is avoided rather than accepted " + "(decisions.md#vocabulary-collisions). Parity"), + Case("phrase_marker_split_by_a_clause_is_not_a_marker", + "Anna z (domu) Nowak", + {"given": "Anna", "middle": "z", "family": "Nowak", + "nickname": "domu"}, + notes="the fork that is about WHERE a run stands rather than " + "how it is spelled: the phrase's two words with a " + "bracketed clause between them. A marker run is tagged " + "over the whole token stream and consumed over one " + "SEGMENT, and a segment holds neither -- extract gives " + "a clause's tokens a role and segment keeps only the " + "role-less ones -- so a run written across a clause " + "edge is one the consuming walk cannot see whole. " + "Tagged anyway, the walk read the first word as the " + "entire marker and this name came back family 'Anna', " + "maiden 'Nowak' (measured 2026-08-26): the bare " + "preposition eating the name, which is the exact damage " + "the phrase entry exists to prevent. classify refuses " + "to tag a run that crosses such a boundary, so the " + "clause is an ordinary nickname and nothing else moves. " + "Parity: 1.4.0 read first Anna / middle z / last Nowak " + "/ nickname domu (2026-08-26)"), + Case("phrase_marker_split_by_a_clause_keeps_the_family", + "Maria z (domu) Kowalska Nowak", + {"given": "Maria", "middle": "z Kowalska", "family": "Nowak", + "nickname": "domu"}, + notes="the row above with a family name to lose, and it is " + "kept as its own row for the reason " + "preposition_alone_is_not_a_marker is: the two fail the " + "same mutation and record different damage from it. " + "Where that one shuffles fields, this one came back " + "family 'Maria', maiden 'Kowalska Nowak' -- the real " + "family name inside the maiden value and gone from its " + "own field (measured 2026-08-26). Losing a family name " + "is the consequence worth pinning, not the shuffle. " + "Parity: 1.4.0 read first Maria / middle 'z Kowalska' / " + "last Nowak / nickname domu (2026-08-26)"), Case("diminutive_that_was_a_marker_keeps_the_family", "Rosalind Roz Smith", {"given": "Rosalind", "middle": "Roz", "family": "Smith"}, @@ -1301,6 +1383,68 @@ def __post_init__(self) -> None: "Policy(maiden_delimiters=...) and reads identically -- " "what M3 adds is the DEFAULT reading, where 1.4.0 and " "2.1 alike gave nickname 'née Jones'"), + Case("phrase_marker_delimited_clause", + "Maria Kowalska (z domu Nowak)", + {"given": "Maria", "family": "Kowalska", "maiden": "Nowak"}, + classification="fix(#434)", + notes="the phrase fork at the OTHER drop site. M3 reads the " + "clause and #329's clause pass drops the marker from " + "inside it, and that pass counts tokens of its own -- " + "phrase_marker_takes_the_maiden_name above exercises " + "M2's pieces walk instead, so a count that stayed at " + "one token in either place is caught by exactly one of " + "the two rows (measured 2026-08-26: forcing the clause " + "pass back to a single token moves this row to maiden " + "'domu Nowak' and leaves the bare row alone, and " + "forcing the pieces walk back does the mirror). 1.4.0 " + "read first Maria / last Kowalska / nickname 'z domu " + "Nowak' (2026-08-26), the clause a nickname because " + "nothing looked inside it"), + Case("phrase_marker_delimited_alone_stays_a_nickname", + "Maria Kowalska (z domu)", + {"given": "Maria", "family": "Kowalska", "nickname": "z domu"}, + notes="M3's word-after condition, asked of a PHRASE: the word " + "must come after the whole marker run, not after the " + "clause's first word. " + "maiden_marked_clause_one_word_stays_a_nickname holds " + "the same boundary for a one-word marker, and cannot " + "reach this one -- a condition written as 'more than " + "one word in the clause' satisfies that row and turns " + "this clause into a maiden one. What it produces is " + "maiden 'z domu' (measured 2026-08-26): the same two " + "words, in the other field, which is the whole of what " + "M3 decides here. The VALUE does not move, because the " + "#329 drop is a separate site and declines for its own " + "reason -- there is no word past the run inside the " + "clause -- so this row catches the mutation on the " + "field alone. Parity: 1.4.0 read nickname 'z domu'"), + Case("phrase_marker_delimited_alone_keeps_its_words", + "Maria Kowalska (z domu)", + {"given": "Maria", "family": "Kowalska", "maiden": "z domu"}, + policy=Policy(maiden_delimiters=frozenset({("(", ")")})), + notes="the same string as " + "phrase_marker_delimited_alone_stays_a_nickname above, " + "under the pair M1 governs, and it reaches a branch " + "that row cannot: with the pair configured the clause " + "IS the maiden name, so the #329 drop runs and has to " + "decide whether a word stands past the marker. It does " + "not -- both words ARE the marker -- so M1's carve-out " + "keeps them, which is what the value shows. That " + "containment test spans the whole marker run, and " + "reading it one token in instead deletes this clause's " + "text: the marker is dropped and nothing is left. " + "Measured 2026-08-26, that mutation fails exactly one " + "test in the suite, M1's own doc example -- which is a " + "doc, edited by the commit that changes behavior, so " + "the pin belongs here too (the argument " + "build_rules_corpus.py makes for the rules corpus, one " + "layer down). Deleting the guard OUTRIGHT is caught by " + "the one-word rows instead -- " + "maiden_marker_delimited_two_clauses and M1's (Nee) " + "examples -- so this row is the phrase half of the " + "branch and not the branch. Parity: 1.4.0 under the " + "bucket-move idiom read first Maria / last Kowalska / " + "maiden 'z domu' (2026-08-26)"), Case("maiden_marked_clause_interior_keeps_the_family", "Jane (née Jones) Smith", {"given": "Jane", "family": "Smith", "maiden": "Jones"}, diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index 67d9298d..1152bd69 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -1,13 +1,17 @@ +import bisect import dataclasses +import pytest + from nameparser._lexicon import Lexicon from nameparser._pipeline._classify import classify -from nameparser._pipeline._extract import extract_delimited -from nameparser._pipeline._group import group +from nameparser._pipeline._extract import extract_delimited, _maiden_marked +from nameparser._pipeline._group import group, marker_run_length from nameparser._pipeline._script_segment import script_segment from nameparser._pipeline._segment import segment from nameparser._pipeline._state import ParseState from nameparser._pipeline._tokenize import tokenize +from nameparser._pipeline._vocab import maiden_marker_run from nameparser._policy import Policy, Script from nameparser._types import Role @@ -937,3 +941,156 @@ def test_an_unlisted_abbreviation_is_as_transparent_as_a_title() -> None: assert _piece_texts(out) == [["Xyz.", "abdul John", "Smith"]] out = _grouped("Berg, Xyz. abdul van") assert _piece_texts(out)[1] == ["Xyz.", "abdul van"] + + +# -- the marker sites' cross-site contract -------------------------- + +# A maiden marker is asked about at FOUR sites, and _title_key's +# docstring names what a divergence between two of them costs: "a +# divergence between them fails silently: the entry simply stops +# matching". This lexicon carries a word entry, a phrase entry, and a +# phrase whose FIRST WORD is itself an entry -- the longest-first case, +# where a site that stopped at the first word would still find a +# marker and simply find a shorter one. +_MARKER_LEX = Lexicon( + maiden_markers=frozenset({"née", "z domu", "geb", "geb von"})) + + +def _marker_sites(spelling: str) -> dict[str, int]: + """Every site's answer to "how many words is the marker here", for + a name whose marker is written `spelling`.""" + clause = spelling.split() + ["Jones"] + answers = {"predicate": maiden_marker_run(clause, + _MARKER_LEX.maiden_markers)} + + def staged(text: str) -> ParseState: + return classify(segment(tokenize(extract_delimited(ParseState( + original=text, lexicon=_MARKER_LEX, policy=Policy()))))) + + # classify: the run it tagged, head plus continuations + bare = staged(f"Jane Smith {spelling} Jones") + head = next(i for i, t in enumerate(bare.tokens) + if "vocab:maiden-marker" in t.tags) + answers["classify"] = marker_run_length( + t.tags for t in bare.tokens[head + 1:]) + # group's piece test and the M2 take built on it: the pass drops + # the marker and nothing else, so the count of dropped tokens is + # where that walk decided the marker ends + answers["group piece walk"] = len(group(bare).dropped) + # group's OTHER drop, the one inside an extracted clause (#329) -- + # a separate site reached only through a delimited name + answers["group clause drop"] = len( + group(staged(f"Jane Smith ({spelling} Jones)")).dropped) + # extract's clause test, which is a boolean: a clause is marker-led + # exactly while a word remains past the run, so the LONGEST prefix + # it still declines is the run itself. Read this way rather than as + # "the first prefix it accepts", which would answer 1 for every + # phrase whose first word is also an entry. + answers["extract"] = max( + k for k in range(len(clause) + 1) + if not _maiden_marked(" ".join(clause[:k]), _MARKER_LEX)) + return answers + + +@pytest.mark.parametrize("spelling", [ + "née", "Née", "née.", # a word entry + "z domu", "Z Domu", "z. domu", "z domu.", # a phrase entry + "geb", "geb von", # both, longest first +]) +def test_every_marker_site_ends_the_run_in_the_same_place( + spelling: str) -> None: + # The contract is AGREEMENT, not four expected numbers: a site that + # drifted would keep passing its own tests while disagreeing with + # the others about an input neither covers, which is exactly how + # the title-run key's two builders could have gone wrong (#369) and + # why P5 and H1 have a contract test of their own. + # + # This test varies the marker's SPELLING and holds its PLACEMENT + # fixed, so every run it builds is structurally contiguous by + # construction. That was a hole: the sites can also disagree + # because they walk different token populations, which no spelling + # reaches. test_a_tagged_marker_run_is_contiguous_and_drops_whole + # below is the placement axis, added after a real disagreement got + # through here. + answers = _marker_sites(spelling) + assert len(set(answers.values())) == 1, answers + # never vacuous: the parametrization's own precondition is that + # each spelling is WHOLLY a marker, so the agreed answer is its + # word count -- an all-zero agreement would otherwise pass + assert answers["predicate"] == len(spelling.split()) + + +# Placements, not spellings: the axis the test above does not vary. A +# marker run is tagged over the whole span-sorted token stream and +# consumed over one SEGMENT, and the two populations differ -- extract +# gives a delimited clause's tokens a role and _segment.py:31 keeps +# only role-None tokens, then buckets those by the commas before them. +# So a run written across a clause edge or a structure comma is one the +# piece walk cannot see whole. Every row here writes 'z domu' at a +# different place relative to those boundaries. +_MARKER_PLACEMENTS = [ + "Jane Smith z domu Jones", # contiguous, bare + "Jane Smith (z domu Jones)", # contiguous, wholly inside a clause + 'Jane Smith "z domu Jones"', # the same, in the other default pair + "Jane z (domu) Jones", # straddles a clause OPEN + "Jane Smith (z) domu Jones", # straddles a clause CLOSE + "Jane z, domu Jones", # straddles a structure comma + "Jane Smith née Jones", # the one-word control + "Jane Smith (née Jones)", # the one-word control, delimited +] + + +def _tagged_runs(state: ParseState) -> list[list[int]]: + """Every maiden marker run classify tagged, as token indices.""" + runs = [] + for i, token in enumerate(state.tokens): + if "vocab:maiden-marker" not in token.tags: + continue + length = marker_run_length(t.tags for t in state.tokens[i + 1:]) + runs.append(list(range(i, i + length))) + return runs + + +@pytest.mark.parametrize("text", _MARKER_PLACEMENTS) +def test_a_tagged_marker_run_is_contiguous_and_drops_whole(text: str) -> None: + state = ParseState(original=text, lexicon=_MARKER_LEX, policy=Policy()) + classified = classify(segment(tokenize(extract_delimited(state)))) + runs = _tagged_runs(classified) + commas = classified.comma_offsets + + def bucket(i: int) -> int: + return bisect.bisect_left(commas, classified.tokens[i].span.start) + + # 1. What classify tags is what a segment can hold: one role and one + # comma bucket throughout. _group._marker_run_pieces asserts this in + # prose and depends on it in fact -- with a run allowed to straddle, + # its walk stops at the boundary and hands M2 a proper PREFIX of the + # phrase as if it were the whole marker, which is how + # 'Anna z (domu) Nowak' came to read family 'Anna', maiden 'Nowak'. + for run in runs: + roles = {classified.tokens[i].role for i in run} + assert len(roles) == 1, (text, run, roles) + assert len({bucket(i) for i in run}) == 1, (text, run) + + # 2. And the consumer takes the whole run or declines it. A count + # that disagrees with the tagged length is the cross-site + # disagreement itself, whatever produced it. + dropped = set(group(classified).dropped) + for run in runs: + taken = dropped & set(run) + assert taken in (set(), set(run)), (text, run, sorted(taken)) + + +def test_the_marker_placements_reach_both_answers() -> None: + # Neither half of the pin above is vacuous: some placements must + # tag a run (or the contiguity assertion quantifies over nothing) + # and some must tag none (or the straddling rows have stopped + # straddling and the regression they pin is unwatched). + tagged = {} + for text in _MARKER_PLACEMENTS: + state = ParseState(original=text, lexicon=_MARKER_LEX, policy=Policy()) + runs = _tagged_runs(classify(segment(tokenize( + extract_delimited(state))))) + tagged[text] = bool(runs) + assert sum(tagged.values()) >= 4 + assert sum(not v for v in tagged.values()) >= 3 diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 3ba574d2..2ffd6aaa 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -1,9 +1,12 @@ import unicodedata -from nameparser._lexicon import Lexicon +import pytest + +from nameparser._lexicon import Lexicon, _normalize, _title_key from nameparser._pipeline._vocab import ( effective_script, is_initial, is_initial_shaped, is_suffix_lenient, - is_suffix_strict, is_wholly_suffix, resolve_script_set, single_script, + is_suffix_strict, is_wholly_suffix, maiden_marker_run, + resolve_script_set, single_script, ) from nameparser._policy import (Policy, Script, _NO_INITIALS, _SCRIPT_RANGES) @@ -222,6 +225,84 @@ def test_is_wholly_suffix_is_not_the_plural_of_is_post_nominal() -> None: ["V."], lex, Policy(lenient_comma_suffixes=False)) +# Stored form: space-joined, per-word normalized -- what _normset +# writes for this field. Built as bare frozensets rather than through a +# Lexicon so these exercise the predicate and nothing else. +_WORD_ONLY = frozenset({"née"}) +_PHRASE = frozenset({"z domu"}) +_BOTH = frozenset({"geb", "geb von"}) + + +def test_maiden_marker_run_has_nothing_to_claim() -> None: + assert maiden_marker_run([], _WORD_ONLY) == 0 + assert maiden_marker_run(["née", "Jones"], frozenset()) == 0 + assert maiden_marker_run(["Smith", "Jones"], _WORD_ONLY) == 0 + + +def test_maiden_marker_run_claims_one_word() -> None: + assert maiden_marker_run(["née", "Jones"], _WORD_ONLY) == 1 + # the marker alone is still a run of one; the caller decides + # whether a word has to follow it + assert maiden_marker_run(["née"], _WORD_ONLY) == 1 + + +def test_maiden_marker_run_claims_a_phrase() -> None: + assert maiden_marker_run(["z", "domu", "Nowak"], _PHRASE) == 2 + + +def test_maiden_marker_run_takes_the_longest_match_first() -> None: + # A caller configuring both a word and a phrase starting with it + # gets the phrase where it matches and the bare word everywhere + # else. Shortest-first would claim 'geb' and leave 'von' a name + # word. + assert maiden_marker_run(["geb", "von", "Braun"], _BOTH) == 2 + assert maiden_marker_run(["geb", "Braun"], _BOTH) == 1 + + +def test_maiden_marker_run_declines_a_partial_phrase() -> None: + # 'z' alone is not a marker -- the whole point of shipping the + # phrase rather than its words. This is the reading the library's + # own split-it-into-separate-entries advice produced. + assert maiden_marker_run(["z", "Nowak"], _PHRASE) == 0 + # and a phrase with nothing after its first word cannot match either + assert maiden_marker_run(["z"], _PHRASE) == 0 + + +def test_maiden_marker_run_folds_per_word() -> None: + # _title_key's storage rule, rebuilt at match time: case folds, and + # each word loses its own edge periods. Normalizing the JOINED + # phrase would leave 'z.' with its period and never match. + assert maiden_marker_run(["Z", "Domu", "Nowak"], _PHRASE) == 2 + assert maiden_marker_run(["z", "domu.", "Nowak"], _PHRASE) == 2 + assert maiden_marker_run(["z.", "domu", "Nowak"], _PHRASE) == 2 + + +@pytest.mark.parametrize("words", [ + ("z", "domu"), ("Z.", "Domu"), ("née",), ("née", "."), (".", "z"), + ("z", "", "domu"), ("GEB", "VON"), (), +]) +def test_the_marker_key_is_title_keys_own_fold(words: tuple[str, ...]) -> None: + """maiden_marker_run builds its lookup key inline so each word is + folded once rather than once per candidate length. That makes it a + COPY of a fold whose definition lives in _lexicon, and this is the + pin that keeps the copy honest: change _title_key and this fails + rather than the entry quietly ceasing to match.""" + assert " ".join(filter(None, (_normalize(w) for w in words))) \ + == _title_key(words) + + +def test_maiden_marker_run_does_not_claim_a_word_that_folds_away() -> None: + # _title_key drops a word that normalizes to nothing, so ['née', + # '.'] keys as 'née' -- a one-word entry would match a two-word + # run and the drop would take the period with it. + assert maiden_marker_run(["née", "."], _WORD_ONLY) == 1 + # and a word that folds away cannot OPEN a run either -- the + # question the head fast path has to answer the same way the loop + # would + assert maiden_marker_run([".", "née"], _WORD_ONLY) == 0 + assert maiden_marker_run([".", "z", "domu"], _PHRASE) == 0 + + def test_single_script_requires_every_char_in_one_script() -> None: assert single_script("毛泽东") is Script.HAN assert single_script("諸葛") is Script.HAN # traditional diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index b5391d03..5cfe5cf3 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -43,11 +43,11 @@ class declares, which members an alternation offers. Those are exact # assert_normalized touches -- looser in the dangerous direction, and a # hand copy of a constant with a source of truth, inside the module # written to forbid exactly that. -from nameparser._lexicon import _normalize +from nameparser._lexicon import _PHRASE_FIELDS, _normalize from nameparser.config.conjunctions import CONJUNCTIONS from nameparser.config.maiden_markers import MAIDEN_MARKERS from nameparser.config.particles import PARTICLES -from nameparser.config.titles import TITLES +from nameparser.config.titles import GIVEN_NAME_TITLES, TITLES from nameparser.config.suffixes import ( GLUED_HONORIFICS, SUFFIX_ACRONYMS_AMBIGUOUS, SUFFIX_WORDS) @@ -68,6 +68,35 @@ class declares, which members an alternation offers. Those are exact _SANCTIONED_EXTRAS = frozenset({(0xFF65, 0xFF65)}) +@pytest.mark.parametrize("field", _PHRASE_FIELDS) +def test_a_shipped_phrase_entry_is_stored_as_written(field: str) -> None: + """The half of a phrase entry's storage rule config's own + assert_normalized cannot see. + + It checks the single-spaced lowercase form; the per-WORD period + strip is _lexicon._title_key's, and a data module asserting with + the parser's fold would make a constant's hygiene depend on the + parser. The question that actually matters is asked here instead, + and it is stronger than the fold: does Lexicon store the shipped + constant UNCHANGED? An entry written 'z. domu' passes import-time + hygiene and is silently rewritten to 'z domu' -- inert as a lookup + key nobody wrote, and invisible everywhere until now. + + Over _PHRASE_FIELDS rather than over maiden_markers, so the + given_name_titles half is covered by the same assertion and a third + phrase field arrives already pinned. Nothing else in the suite pins + this equality for any field. + """ + from nameparser import Lexicon + shipped = {"maiden_markers": MAIDEN_MARKERS, + "given_name_titles": GIVEN_NAME_TITLES}[field] + assert getattr(Lexicon.default(), field) == frozenset(shipped), ( + f"{field}: Lexicon rewrote the shipped constant, so the entries " + f"it stores are not the ones the data module wrote -- " + f"{sorted(frozenset(shipped) - getattr(Lexicon.default(), field))} " + f"were folded away or changed") + + def test_the_corpus_population_is_not_degenerate() -> None: """The floors themselves live in compare.py and are asserted by tests/v2/test_differential.py, which already checks every shipped @@ -845,11 +874,14 @@ class _LatinCopy(NamedTuple): #: removal: drop an entry a member covers and the snapshot shrinks. #: #: Three nearby counts differ and are easy to conflate, all for -#: fix(#274) specifically, and all four numbers moved in 2.2 -- recount -#: rather than adjust them: MAIDEN_MARKERS ships 16 entries (roz left -#: the vocabulary); that rule's members reach 3 of them; the corpora -#: contain 4 markers in total (geb, nee, née, 旧姓 -- nee arrived with -#: #414's rules corpus), 3 of which it covers. +#: fix(#274) specifically, and every one of them moved in 2.2 -- twice, +#: so recount rather than adjust them: MAIDEN_MARKERS ships 17 entries +#: (roz left the vocabulary, z domu joined it); that rule's members +#: reach 3 of them; the corpora contain 5 markers in total (geb, nee, +#: née, z domu, 旧姓 -- nee arrived with #414's rules corpus and z domu +#: with #434's examples), 3 of which it covers. Counting z domu at all +#: needs the RUN reading _carries uses: it is two tokens, and a +#: per-token count cannot see it. _LATIN_ALTERNATION_SOURCES: dict[str, _LatinCopy] = { "fix(#274)": _LatinCopy( vocabulary=MAIDEN_MARKERS, @@ -1153,8 +1185,20 @@ def _carries(name: str, vocabulary: frozenset[str]) -> bool: """ delimiters = "".join({ch for pair in DEFAULT_NICKNAME_DELIMITERS for ch in pair}) - tokens = {_normalize(token.strip(delimiters)) for token in name.split()} - return bool(tokens & vocabulary) or any( + tokens = [_normalize(token.strip(delimiters)) for token in name.split()] + # A vocabulary may hold PHRASE entries ('z domu'), which no set of + # single tokens can contain: 'Maria Kowalska z domu Nowak' carries + # a marker and not one of its words is one. So the membership test + # runs over every consecutive token RUN up to the longest entry -- + # which is the single tokens themselves when no entry is a phrase, + # leaving every count above unchanged. Still textual and still + # wider than the parser (no longest-first, no fold-away rule): + # tools/differential/README.md says that is this helper's job. + longest = max((entry.count(" ") + 1 for entry in vocabulary), default=1) + runs = {" ".join(tokens[i:i + n]) + for n in range(1, longest + 1) + for i in range(len(tokens) - n + 1)} + return bool(runs & vocabulary) or any( entry in name for entry in vocabulary if not entry.isascii()) @@ -1277,6 +1321,10 @@ def _claim(rule: dict) -> _Claim: "expected_since_1.4.0.toml": { "fix(#335) a marker-led clause leaves the one name word its bare reading": _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), + "fix(#434) a multi-word maiden marker takes the maiden name": + _Claim(1, ('family', 'maiden', 'middle'), "c428798fc6ef"), + "fix(#434) a multi-word marker leads a bracketed clause to the maiden name": + _Claim(1, ('maiden', 'nickname'), "0b3ef183f283"), "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": @@ -1316,7 +1364,7 @@ def _claim(rule: dict) -> _Claim: "fix(comma-precomma-family) pre-comma run reads as family, not given": _Claim(279, ('family', 'given'), "28a62b622a48"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(1080, ('family', 'given', 'suffix'), "0cb2cda1ed6b"), + _Claim(1085, ('family', 'given', 'suffix'), "0df8e4a51a54"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1391,6 +1439,10 @@ def _claim(rule: dict) -> _Claim: "expected_since_2.0.0.toml": { "fix(#335) a marker-led clause leaves the one name word its bare reading": _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), + "fix(#434) a multi-word maiden marker takes the maiden name": + _Claim(1, ('family', 'maiden', 'middle'), "c428798fc6ef"), + "fix(#434) a multi-word marker leads a bracketed clause to the maiden name": + _Claim(1, ('maiden', 'nickname'), "0b3ef183f283"), "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), "fix(#335) a marker-led bracketed clause reads as the maiden name, compounding with the CJK order flip": @@ -1501,6 +1553,10 @@ def _claim(rule: dict) -> _Claim: "expected_since_2.1.0.toml": { "fix(#335) a marker-led clause leaves the one name word its bare reading": _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), + "fix(#434) a multi-word maiden marker takes the maiden name": + _Claim(1, ('family', 'maiden', 'middle'), "c428798fc6ef"), + "fix(#434) a multi-word marker leads a bracketed clause to the maiden name": + _Claim(1, ('maiden', 'nickname'), "0b3ef183f283"), "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": _Claim(6, ('maiden', 'nickname'), "d0e857deddb2"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": @@ -1851,7 +1907,14 @@ class _Excluded(NamedTuple): ("fix(comma-family)", "fix(comma-precomma-family)", "fix(suffix-routing)")), '(^|[\\w.]\\s+)[("\'][^)"\']+[)"\'](\\s+\\w|\\s*$)': - _Excluded(51, "770738271273", ()), + # 51 -> 54 as rules.md gained the bracketed Polish examples + # (#434): 'Maria Kowalska (z domu Nowak)', 'Maria Kowalska + # (z domu)', and the boundary 'Anna z (domu) Nowak' M2 gained + # when the clause-straddling defect was fixed. Growth in the + # corpus, not in the exclusion -- its regex is untouched -- and + # `absorbed_by` stayed empty, so no rule reaches the protected + # shape. + _Excluded(54, "e2924f45c9d8", ()), } diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index 6fce8c0d..f4765574 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -7,7 +7,8 @@ from nameparser import Parser from nameparser._lexicon import ( - Lexicon, _VOCAB_FIELDS, _default_lexicon, _normalize, _title_key, + Lexicon, _PHRASE_FIELDS, _VOCAB_FIELDS, _default_lexicon, _normalize, + _title_key, ) from nameparser._policy import Script, _SCRIPT_RANGES @@ -471,7 +472,18 @@ def test_every_lexicon_entry_point_rejects_a_buffer_with_a_decode_hint( build(value) -_PER_WORD_FIELDS = [f for f in _VOCAB_FIELDS if f != "given_name_titles"] +# Derived from the exemption, not written out beside it: an exemption +# that widens must take fields OUT of this list, where the parametrized +# test below stops covering them and the count assertion notices. +_PER_WORD_FIELDS = [f for f in _VOCAB_FIELDS if f not in _PHRASE_FIELDS] + + +def test_only_the_phrase_fields_are_exempt_from_the_warning() -> None: + # The exemption is exactly the kind of edit that silently widens: + # one more field in _PHRASE_FIELDS costs nothing at construction and + # turns a dead entry back into a silent no-op. Two fields, named. + assert _PHRASE_FIELDS == ("given_name_titles", "maiden_markers") + assert len(_PER_WORD_FIELDS) == len(_VOCAB_FIELDS) - 2 @pytest.mark.parametrize("field", _PER_WORD_FIELDS) @@ -497,6 +509,26 @@ def test_multiword_given_name_title_does_not_warn() -> None: Lexicon.empty().add(given_name_titles=["lt col"]) +def test_multiword_maiden_marker_does_not_warn() -> None: + # The second exemption, and the one whose warning was wrong rather + # than merely unhelpful: markers are matched by lookahead, and + # "split it into separate entries" reads 'Maria Kowalska z domu + # Nowak' as maiden 'domu Nowak'. + with warnings.catch_warnings(): + warnings.simplefilter("error") + Lexicon.empty().add(maiden_markers=["z domu"]) + + +def test_phrase_fields_store_the_phrase_folded_per_word() -> None: + # The shared storage rule: space-joined, each word normalized on + # its own. Whole-phrase _normalize would keep 'z.' interior period + # and the entry would never match. + lex = Lexicon.empty().add(maiden_markers=["Z. Domu"], + given_name_titles=["Lt. Col"]) + assert lex.maiden_markers == frozenset({"z domu"}) + assert lex.given_name_titles == frozenset({"lt col"}) + + def test_multiword_capitalization_key_warns() -> None: with pytest.warns(UserWarning, match="matched one word at a time"): dataclasses.replace( diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 55e75da6..f9ba35a8 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1311,21 +1311,30 @@ def test_stacked_activation_warns_only_for_uncovered_scripts() -> None: def _clause_free_latin_corpus_names() -> list[str]: from nameparser import DEFAULT_NICKNAME_DELIMITERS + from nameparser._pipeline._vocab import maiden_marker_run from nameparser.config.maiden_markers import MAIDEN_MARKERS from ._differential_fixtures import _CORPUS_NAMES - # A marker glued to a delimiter character is still a marker, and - # the membership test is per WORD, so '(geb.' must lose the - # bracket as well as the abbreviating period before it is asked. - # Stripping only the period admitted every corpus name that - # brackets its marker, and once rules.md#M3 read such a clause as - # the maiden name, six of them had a maiden clause of their own -- - # two clauses, and the appended one no longer the only variable. - # The strip turns away TEN names in all: those six, plus four that - # M3 declines and that would have been safe to keep -- the - # one-word '(Nee)', '(Nee) (Jones)' and '(née)', and '(née Jr.)', - # which S1 takes before M3 sees it. Textual, and so deliberately - # conservative in exactly that direction. + # "Does this name already carry a marker" is the parser's own + # question, so it is asked with the parser's own predicate rather + # than by word membership. A word test cannot see a PHRASE entry: + # no word of 'z domu' is a marker, so 'Maria Kowalska z domu Nowak' + # sat in this corpus with a maiden clause of its own and the + # appended one was not the only variable + # (mechanisms.md#ONE-PREDICATE-PER-QUESTION -- a test guard is as + # able to write the mirroring condition as a stage is). + # + # The strip stays, and runs BEFORE the predicate: a marker glued to + # a delimiter character is still a marker, and neither the + # predicate nor the vocabulary strips a bracket, so '(geb.' must + # lose it before being asked. Stripping only the period admitted + # every corpus name that brackets its marker, and once rules.md#M3 + # read such a clause as the maiden name, six of them had a maiden + # clause of their own. The strip turns away TEN names in all: those + # six, plus four that M3 declines and that would have been safe to + # keep -- the one-word '(Nee)', '(Nee) (Jones)' and '(née)', and + # '(née Jr.)', which S1 takes before M3 sees it. Textual, and so + # deliberately conservative in exactly that direction. # # Every count in this comment quantifies over the corpus, so one # added corpus row falsifies it silently. Recount rather than @@ -1335,34 +1344,45 @@ def _clause_free_latin_corpus_names() -> list[str]: # uv run python -c " # import re, sys; sys.path.insert(0, 'tests') # from nameparser import DEFAULT_NICKNAME_DELIMITERS as D + # from nameparser._pipeline._vocab import maiden_marker_run as run # from nameparser.config.maiden_markers import MAIDEN_MARKERS as M # from v2._differential_fixtures import _CORPUS_NAMES # base = [n for n in _CORPUS_NAMES if re.match(r'^[\x00-\u024f]*$', n) and ',' not in n] # strip = ''.join({c for p in D for c in p}) + '.' - # keep = lambda f: [n for n in base if not any(f(w) in M for w in n.split())] + # words = lambda n, f: [f(w) for w in n.split()] + # keep = lambda f: [n for n in base if not any(w in M for w in words(n, f))] # old = keep(lambda w: w.lower().rstrip('.')) # new = keep(lambda w: w.lower().strip(strip)) - # print(len(old), len(new), sorted(set(old) - set(new)))" + # ws = lambda n: words(n, lambda w: w.lower().strip(strip)) + # pred = [n for n in base if not any(run(ws(n)[i:], M) for i in range(len(ws(n))))] + # print(len(old), len(new), len(pred), sorted(set(old) - set(new)), sorted(set(new) - set(pred)))" # # Delimiter characters come from the shipped set rather than a # literal, so a pair added there cannot quietly reopen this. strip = "".join({ch for pair in DEFAULT_NICKNAME_DELIMITERS for ch in pair}) + "." + + def marked(name: str) -> bool: + words = [word.lower().strip(strip) for word in name.split()] + return any(maiden_marker_run(words[i:], MAIDEN_MARKERS) + for i in range(len(words))) + return [name for name in _CORPUS_NAMES - if _LATIN.match(name) and "," not in name - and not any(word.lower().strip(strip) in MAIDEN_MARKERS - for word in name.split())] + if _LATIN.match(name) and "," not in name and not marked(name)] def test_the_clause_free_corpus_is_not_empty() -> None: """The invariant below is parametrized over a FILTERED corpus, and an empty parametrization passes as a skip rather than failing -- - the shape #329 left behind. The filter has been widened once - already (the delimiter strip, 2026-08-26, which took it from 638 - names to 628 -- the one-liner in that filter's comment recounts - both), so the floor is what says a future widening emptied - it. Deliberately far below today's count: this asks whether the - filter still selects a corpus, not what the corpus holds.""" + the shape #329 left behind. The filter has been widened twice + already -- the delimiter strip, then the move from word + membership to the marker predicate, which a phrase entry made + necessary -- and the one-liner in that filter's comment recounts + every stage: 642 names before either, 632 after the strip, 629 + once the predicate decides. So the floor is what says a future + widening emptied it. Deliberately far below today's count: this + asks whether the filter still selects a corpus, not what the + corpus holds.""" assert len(_clause_free_latin_corpus_names()) > 100 @@ -1405,3 +1425,22 @@ def test_a_maiden_clause_changes_nothing_else(name: str) -> None: assert getattr(with_clause, field) == getattr(base, field), ( f"{name!r}: {field} reads {getattr(base, field)!r} alone and " f"{getattr(with_clause, field)!r} with a maiden clause") + + +def test_a_phrase_marker_outranks_the_word_it_starts_with() -> None: + # Longest first, and the row cases.py cannot hold: a Case carries a + # Policy or a Locale, never a Lexicon, so the one marker fork that + # needs two entries at once lives here. + # + # 'geb' ships and 'geb von' is the caller's addition. The phrase + # must win where it matches -- maiden 'Braun', the particle being + # part of the marker -- and the bare word must still match + # everywhere else. Shortest-first would take 'geb' in both and read + # the first as maiden 'von Braun', which is also what the default + # vocabulary reads, so the control below is what makes the first + # assertion mean anything. + configured = Parser(lexicon=Lexicon.default().add( + maiden_markers=["geb von"])) + assert configured.parse("Jane Smith geb von Braun").maiden == "Braun" + assert configured.parse("Jane Smith geb Braun").maiden == "Braun" + assert parse("Jane Smith geb von Braun").maiden == "von Braun" diff --git a/tools/differential/README.md b/tools/differential/README.md index 6c2f45cf..56677a19 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -412,38 +412,54 @@ The corpora run under the **default policy**, so any behavior gated behind a non-default `Policy` field is invisible here. Default *vocabulary* is a different matter: it is fully in EFFECT, never gated off the way a `Policy` field is, so a change to it can show up -here. That is not the same as coverage -- only 4 of the 16 shipped +here. That is not the same as coverage -- only 5 of the 17 shipped `maiden_markers` and 8 of the 15 `honorific_tails` appear anywhere in the corpora (re-measured 2026-08-26; the marker count was 3 until -#414's rules corpus brought in a parenthesized `Nee`, and the -denominator was 17 until `roz` left the vocabulary in 2.2 -- it -appeared in no corpus name, so only the denominator moved), so an -entry no corpus name exercises is as invisible as an opt-in policy. - -Those two numbers count WHOLE TOKENS, delimiters stripped. The strip -is what earns exactly one of the four: `nee`, which appears in the -corpora only inside brackets -- `Jane Smith (Nee)` and -`Jane Smith (Nee) (Jones)` -- where the token carries them until they -come off. `née` needs no strip, appearing bare in many names, and `né` -is not counted at all: it occurs only as a substring of `née`, never -as a token. The convention matters because the neighbouring guard +#414's rules corpus brought in a parenthesized `Nee`, the denominator +was 17 until `roz` left the vocabulary in 2.2 -- it appeared in no +corpus name, so only the denominator moved -- and #434's `z domu` then +put both back up by one), so an entry no corpus name exercises is as +invisible as an opt-in policy. + +Those two numbers count WHOLE TOKENS, delimiters stripped, and +consecutive RUNS of them: an entry may be a PHRASE (`z domu`), which no +single token can equal. The run half earns exactly one of the five, and +it is the reason the count is not simply per word -- `z` and `domu` +each appear in corpus names that carry no marker at all. The strip +earns another: `nee`, which appears in the corpora only inside brackets +-- `Jane Smith (Nee)` and `Jane Smith (Nee) (Jones)` -- where the token +carries them until they come off. `née` needs no strip, appearing bare +in many names, and `né` is not counted at all: it occurs only as a +substring of `née`, never as a token. The convention matters because the neighbouring guard `tests/v2/test_ledger_guards.py::_carries` deliberately asks a wider question -- it also matches a non-ASCII entry anywhere inside a name, since 旧姓 is written flush against the name it marks -- and under that -reading the marker count is 5, not 4. Both are right about different -questions. Recompute: +reading the marker count is 6, not 5: it adds `né`, which occurs only +as a substring of `née` and never as a token of its own. Both are right +about different questions. Recompute: + +A heredoc, not `python -c "..."`: the strip set contains a double +quote, which closes the `-c` string and leaves the rest to the shell. +Paste this as it stands. ``` -uv run python -c " +uv run python - <<'PY' import glob, json from nameparser import Parser from nameparser._lexicon import _normalize L = Parser().lexicon names = [json.loads(l) for f in glob.glob('tools/differential/corpus*.jsonl') for l in open(f, encoding='utf-8') if l.strip()] -toks = {_normalize(t.strip('()\'"«»“”„「」『』（）')) for n in names for t in n.split()} for s in ('maiden_markers', 'honorific_tails'): v = getattr(L, s) - print(s, len(toks & v), 'of', len(v), sorted(toks & v))" + longest = max(e.count(' ') + 1 for e in v) + runs = set() + for n in names: + t = [_normalize(w.strip('()\'"«»“”„「」『』（）')) for w in n.split()] + runs |= {' '.join(t[i:i+k]) for k in range(1, longest+1) for i in range(len(t)-k+1)} + carries = set(runs & v) | {e for e in v if not e.isascii() and any(e in n for n in names)} + print(s, len(runs & v), 'of', len(v), sorted(runs & v), + '| _carries reading:', len(carries), sorted(carries)) +PY ``` @@ -459,15 +475,18 @@ is opt-in about them is narrower than it looks (rows measured Row 1 carries no marker word, so it isolates the delimiter: the brackets alone route their content to `maiden`, and only once the policy says they do. Row 2 carries no brackets, so it isolates the -marker: `Lexicon.maiden_markers` ships 16 entries by default, `nee` +marker: `Lexicon.maiden_markers` ships 17 entries by default, `nee` among them, and the bare form needs no configuration at all. So what is opt-in is neither the marker words nor the delimited path as a whole: it is the delimited path for content that does not -announce itself. Since #335 a clause of two words or more led by a -marker reads as the maiden name whichever bucket its pair sits in +announce itself. Since #335 a clause holding a word past its marker +reads as the maiden name whichever bucket its pair sits in (`rules.md#M3`), so `Jane Smith (née Jones)` needs no configuration -either. Row 1 is exactly the shape that still does — a markerless +either. A word past the MARKER, not a second word in the clause: since +#434 a marker may be a phrase, and `Maria Kowalska (z domu)` is a +two-word clause that stays a nickname because both its words are the +marker. Row 1 is exactly the shape that still does — a markerless clause — along with a one-word clause like `Jane Smith (Nee)`, where nothing in the content says maiden and only a caller who knows the data can. diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 136ff033..655bbdb3 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -16,6 +16,8 @@ "Anh Do" "Anh Van Do" "Anna () Smith" +"Anna z (domu) Nowak" +"Anna z Nowak" "Asst. Vice Chancellor John Smith" "Beethoven, Ludwig van" "Berg, Jan van der" @@ -108,6 +110,9 @@ "Juan y Garcia" "Juan y Garcia née Jones" "Mari' Aube'" +"Maria Kowalska (z domu Nowak)" +"Maria Kowalska (z domu)" +"Maria Kowalska z domu Nowak" "Marquess of Bath" "Mary Beth Smith" "Mc Donald" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 3b76a3f8..7916b1da 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -67,15 +67,22 @@ issue = "fix(#274) maiden markers consumed" # tests/v2/test_ledger_guards.py: every alternative must match at least # one entry the config ships, and none may match ordinary name text. # -# Its members reach 3 of the 16 entries -- geb, nee, née -- which is -# not the same set as the markers the corpora contain (geb, nee, née -# and 旧姓; the first three are covered here). Thirteen entries go -# unreached, and the carve-out in that arithmetic is load-bearing: -# 旧姓 is one of the thirteen and DOES appear as a corpus token, which -# is why it has fix(cjk-maiden-marker) of its own below rather than an -# alternative here. Growing this rule toward the other TWELVE buys -# nothing: none of them appears as a token in any corpus, so there is -# no diff for them to classify. The roz alternative was +# Its members reach 3 of the 17 entries -- geb, nee, née -- which is +# not the same set as the markers the corpora contain (geb, nee, née, +# z domu and 旧姓; the first three are covered here). Fourteen entries +# go unreached, and the carve-outs in that arithmetic are +# load-bearing: TWO of the fourteen do appear in the corpora and each +# is classified somewhere other than this alternation. 旧姓 is one, +# and it has fix(cjk-maiden-marker) of its own below rather than an +# alternative here. z domu is the other, and it cannot be an +# alternative here at all: this is a per-WORD regex over a name, and +# the words of a phrase entry are an ordinary Polish preposition and +# an ordinary noun -- an alternative matching either would claim names +# carrying no marker, which is the one thing the pin above forbids. +# Its two names are classified by the fix(#434) rules, keyed on the +# literal. Growing this rule toward the other TWELVE buys nothing: +# none of them appears as a token in any corpus, so there is no diff +# for them to classify. The roz alternative was # dropped in 2.2 with the vocabulary entry it copied (see # maiden_markers.py's Deliberately absent note): no corpus name # contains the string in any case or position, so it classified @@ -1430,6 +1437,47 @@ issue = "fix(#410) the maiden flavor, where 1.4.0 read the marker as a middle na name_regex = "(?i)^Dr\\. Smith n[ée]e Jones$" fields = ["given", "middle", "family", "maiden"] +[[change]] +issue = "fix(#434) a multi-word maiden marker takes the maiden name" +# 'Maria Kowalska z domu Nowak': maiden_markers gained PHRASE entries +# and the Polish "z domu" is the first one any shipped set holds, so +# rules.md#M2's take fires on a marker that spans two tokens. Every +# earlier version read the marker as name text -- middle 'Kowalska z +# domu', family 'Nowak' -- because neither z nor domu is a marker +# alone and nothing looked at the pair. +# +# `fields` is the three the diff moves and no more. The marker leaves +# the name, so `middle` empties and `family` moves back one word, +# which is why this cannot share a rule with the bracketed name below: +# there the clause changes role and the name words around it do not +# move at all. One rule over both would have to declare the union and +# would pre-excuse a middle/family regression on the bracketed +# spelling -- the #444 failure the fix(#335) rules above spell out. +# +# Keyed on the literal, like those rules and for their reason: the +# shape is "a marker phrase inside a name", a question about the +# parse, and a fields-only rule standing in for it would pre-excuse +# every future middle/family/maiden confusion in the corpus. +name_regex = "(?i)^Maria Kowalska z domu Nowak$" +fields = ["middle", "family", "maiden"] + +[[change]] +issue = "fix(#434) a multi-word marker leads a bracketed clause to the maiden name" +# 'Maria Kowalska (z domu Nowak)': the same vocabulary change at the +# other site. rules.md#M3 reads a marker-led clause as the maiden name +# whatever pair encloses it, and #335 gave it that reach in 2.2 -- but +# the reach alone leaves this name a nickname, since M3 asks whether +# the content opens with a recognized marker and through 2.1 no +# vocabulary answered yes here. The phrase entry is what moves it, so +# it is #434's diff and not #335's, and it is deliberately NOT a +# seventh alternative in that rule's six literals. +# +# `fields` is the pair that moves. The clause changes role; the name +# words keep their readings, so a `given`, `middle` or `family` +# movement here is a regression and stays loud. +name_regex = "(?i)^Maria Kowalska \\(z domu Nowak\\)$" +fields = ["nickname", "maiden"] + [[change]] issue = "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it" # 'Jane Smith (née Jones)', 'Jane (née Jones) Smith', 'Jane Smith diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 5f49b2aa..d5336caf 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -950,6 +950,47 @@ issue = "fix(#410) a title and one name word name the family, whatever annotatio name_regex = "(?i)^(?:Senator \"Rick\" Edmonds|Xyz\\. \\(Bud\\) Smith|Dr\\. Smith n[ée]e Jones|'Smitty' Dr\\. Jones)$" fields = ["given", "family"] +[[change]] +issue = "fix(#434) a multi-word maiden marker takes the maiden name" +# 'Maria Kowalska z domu Nowak': maiden_markers gained PHRASE entries +# and the Polish "z domu" is the first one any shipped set holds, so +# rules.md#M2's take fires on a marker that spans two tokens. Every +# earlier version read the marker as name text -- middle 'Kowalska z +# domu', family 'Nowak' -- because neither z nor domu is a marker +# alone and nothing looked at the pair. +# +# `fields` is the three the diff moves and no more. The marker leaves +# the name, so `middle` empties and `family` moves back one word, +# which is why this cannot share a rule with the bracketed name below: +# there the clause changes role and the name words around it do not +# move at all. One rule over both would have to declare the union and +# would pre-excuse a middle/family regression on the bracketed +# spelling -- the #444 failure the fix(#335) rules above spell out. +# +# Keyed on the literal, like those rules and for their reason: the +# shape is "a marker phrase inside a name", a question about the +# parse, and a fields-only rule standing in for it would pre-excuse +# every future middle/family/maiden confusion in the corpus. +name_regex = "(?i)^Maria Kowalska z domu Nowak$" +fields = ["middle", "family", "maiden"] + +[[change]] +issue = "fix(#434) a multi-word marker leads a bracketed clause to the maiden name" +# 'Maria Kowalska (z domu Nowak)': the same vocabulary change at the +# other site. rules.md#M3 reads a marker-led clause as the maiden name +# whatever pair encloses it, and #335 gave it that reach in 2.2 -- but +# the reach alone leaves this name a nickname, since M3 asks whether +# the content opens with a recognized marker and through 2.1 no +# vocabulary answered yes here. The phrase entry is what moves it, so +# it is #434's diff and not #335's, and it is deliberately NOT a +# seventh alternative in that rule's six literals. +# +# `fields` is the pair that moves. The clause changes role; the name +# words keep their readings, so a `given`, `middle` or `family` +# movement here is a regression and stays loud. +name_regex = "(?i)^Maria Kowalska \\(z domu Nowak\\)$" +fields = ["nickname", "maiden"] + [[change]] issue = "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it" # 'Jane Smith (née Jones)', 'Jane (née Jones) Smith', 'Jane Smith diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 43c6bf6e..adf4880c 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -880,6 +880,47 @@ issue = "fix(#410) a title and one name word name the family, whatever annotatio name_regex = "(?i)^(?:Senator \"Rick\" Edmonds|Xyz\\. \\(Bud\\) Smith|Dr\\. Smith n[ée]e Jones|'Smitty' Dr\\. Jones)$" fields = ["given", "family"] +[[change]] +issue = "fix(#434) a multi-word maiden marker takes the maiden name" +# 'Maria Kowalska z domu Nowak': maiden_markers gained PHRASE entries +# and the Polish "z domu" is the first one any shipped set holds, so +# rules.md#M2's take fires on a marker that spans two tokens. Every +# earlier version read the marker as name text -- middle 'Kowalska z +# domu', family 'Nowak' -- because neither z nor domu is a marker +# alone and nothing looked at the pair. +# +# `fields` is the three the diff moves and no more. The marker leaves +# the name, so `middle` empties and `family` moves back one word, +# which is why this cannot share a rule with the bracketed name below: +# there the clause changes role and the name words around it do not +# move at all. One rule over both would have to declare the union and +# would pre-excuse a middle/family regression on the bracketed +# spelling -- the #444 failure the fix(#335) rules above spell out. +# +# Keyed on the literal, like those rules and for their reason: the +# shape is "a marker phrase inside a name", a question about the +# parse, and a fields-only rule standing in for it would pre-excuse +# every future middle/family/maiden confusion in the corpus. +name_regex = "(?i)^Maria Kowalska z domu Nowak$" +fields = ["middle", "family", "maiden"] + +[[change]] +issue = "fix(#434) a multi-word marker leads a bracketed clause to the maiden name" +# 'Maria Kowalska (z domu Nowak)': the same vocabulary change at the +# other site. rules.md#M3 reads a marker-led clause as the maiden name +# whatever pair encloses it, and #335 gave it that reach in 2.2 -- but +# the reach alone leaves this name a nickname, since M3 asks whether +# the content opens with a recognized marker and through 2.1 no +# vocabulary answered yes here. The phrase entry is what moves it, so +# it is #434's diff and not #335's, and it is deliberately NOT a +# seventh alternative in that rule's six literals. +# +# `fields` is the pair that moves. The clause changes role; the name +# words keep their readings, so a `given`, `middle` or `family` +# movement here is a regression and stays loud. +name_regex = "(?i)^Maria Kowalska \\(z domu Nowak\\)$" +fields = ["nickname", "maiden"] + [[change]] issue = "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it" # 'Jane Smith (née Jones)', 'Jane (née Jones) Smith', 'Jane Smith