diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 377d31c6..9171771b 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -78,7 +78,7 @@ Open: [#360](https://github.com/derek73/python-nameparser/issues/360) which part - Group distribution is load-bearing, not incidental: the v1 Portuguese tests (test_portuguese_dos, test_portuguese_prefixes) require multiple particle groups to land across middle and family, while #132 wanted the combined double-surname reading — the same shape with opposing wants, which is why the combined reading lives in the surnames VIEW and the split in the fields. -- 2026-08-22 #424 — the chain stops where assign's trailing peel begins. The chain ran "until a trailing suffix begins" and asked with the suffix-piece test, which vetoes a bare `V` as an initial — the #401 question at a third site — so `John van der Berg V` read family 'van der Berg V' (1.4.0 read it so too: shipped since 1.x), and `John van der Berg Ma` read family 'van der Berg Ma' where 1.4.0 read suffix 'Ma' — a 2.0 regression on S2's other fork. Both now stop where `_trailing_start` says the run begins: the S2 peel (`_peel_trailing` over `_peel_walk`, shared with assign and P5's reserve since #425) read over the pieces as they stand from the first name piece, once per segment, kept as a length from the end that the chain's merges ahead of it do not move (the benchmark's `particles` shape is the guard). The chain takes both forks — its merges leave the acronym at least the three pieces the fork counted, so assign peels it after as the walk did before — where the maiden walk takes only the numeral (the M2 entry says why). The fork reads the piece before the numeral as it stands, so `John van der J. V` keeps family 'van der J. V', as `John J. V` reads the V as a name; and a numeral with a suffix behind it is not last in the walk, so `John van der Berg V Jr` keeps family 'van der Berg V'. No differential corpus name had either shape; the rules examples carry them — the numeral against every baseline, the acronym against 2.0.0 and 2.1.0. Two more sites were asking assign's question with a test of their own, both found by the reviews of this change. The walk's starting point and P4's leading-particle scan asked group's `title()`, which does not see H2's unlisted abbreviations — assign peels those as titles all the same — so `Xyz. van Johnson` chained where `Dr. van Johnson` did not (#367 had keyed the exception on the first piece of the name, by group's test), and `Xyz. van Berg MA` read family 'van Berg MA' on every 2.x tree — the chain had taken the acronym, as it took every acronym; the draft that stopped before it left assign two pieces where the fork had counted three, and MA became the family. Both ask assign's own test now — `_is_leading_title` and the period-abbreviation pattern moved down into group, with `_leading_titles` as the one definition of where the name begins: the chain's walk, the bound join's first name piece (the second docs review found that one still on group's test — `Xyz. abdul John Smith` joined nothing where `Dr. abdul John Smith` read given 'abdul John') and, through `_is_leading_title`, the scan and the PARTICLE_OR_GIVEN report — so `Xyz. van …` reads as `Dr. van …` does. `Esq. van Gogh`, a corpus name, moves with it (1.4.0 read family 'van Gogh' for both, as #367's rule records; its own rule at every baseline, and one for the P4 example). A particle of the unambiguous suffix vocabulary too (vd, mc) is a suffix piece to the peel: where it opens the trailing run the chain stops before it as before any suffix word and the peel takes it — `John Smith Mc V` reads suffix 'Mc, V' where master and 1.4.0 read family 'Mc V', a class the second docs review found unrecorded (2,554 of its 117,306 constructed names, an eighteen-word pool that puts the words in every position) — and where it continues a prefix run, the run takes it as a particle, so `John van Mc` keeps family 'van Mc', every baseline's reading and the one P6 chose for the shape after a comma. Both are Accepted under P2. The fork's count was the last thing the chain asked on its own authority, and the reviews found it wrong behind a title-and-particle word: #367 stops the leading-particle scan at `Freiherr`, `St`, `Do` — the name's own first piece — where assign's title peel steps over them, so the chain takes the name's first word, and the acronym the fork counted with three pieces met assign with two: `Freiherr von Berg MA` read given 'von Berg', family 'MA' (1.4.0's reading, as it happens; master read family 'von Berg MA'). So the chain asks the peel again over the pieces it leaves — `chain(tail)`, then `_trailing_start` over what it built, and a second run without the stop where the verdict changed, the snapshot being one copy per segment with a trailing run — and takes what assign will not peel: family 'von Berg MA'. The numeral cannot flip, a chain group never being initial-shaped, so `Freiherr von Richthofen V` keeps suffix 'V', and its chain, the one name piece left, reads as `Dr. Smith V` reads — given 'von Richthofen', no family, where 1.4.0 and master read family 'von Richthofen V'; the code review asked for that boundary to be on record, and P2 carries it. +- 2026-08-22 #424 — the chain stops where assign's trailing peel begins. The chain ran "until a trailing suffix begins" and asked with the suffix-piece test, which vetoes a bare `V` as an initial — the #401 question at a third site — so `John van der Berg V` read family 'van der Berg V' (1.4.0 read it so too: shipped since 1.x), and `John van der Berg Ma` read family 'van der Berg Ma' where 1.4.0 read suffix 'Ma' — a 2.0 regression on S2's other fork. Both now stop where `_trailing_start` says the run begins: the S2 peel (`_peel_trailing` over `_peel_walk`, shared with assign and P5's reserve since #425) read over the pieces as they stand from the first name piece, once per segment, kept as a length from the end that the chain's merges ahead of it do not move (the benchmark's `particles` shape is the guard). The chain takes both forks — its merges leave the acronym at least the three pieces the fork counted, so assign peels it after as the walk did before — where the maiden walk takes only the numeral (the M2 entry says why). The fork reads the piece before the numeral as it stands, so `John van der J. V` keeps family 'van der J. V', as `John J. V` reads the V as a name; and a numeral with a suffix behind it is not last in the walk, so `John van der Berg V Jr` keeps family 'van der Berg V'. No differential corpus name had either shape; the rules examples carry them — the numeral against every baseline, the acronym against 2.0.0 and 2.1.0. Two more sites were asking assign's question with a test of their own, both found by the reviews of this change. The walk's starting point and P4's leading-particle scan asked group's `title()`, which does not see H2's unlisted abbreviations — assign peels those as titles all the same — so `Xyz. van Johnson` chained where `Dr. van Johnson` did not (#367 had keyed the exception on the first piece of the name, by group's test), and `Xyz. van Berg MA` read family 'van Berg MA' on every 2.x tree — the chain had taken the acronym, as it took every acronym; the draft that stopped before it left assign two pieces where the fork had counted three, and MA became the family. Both ask assign's own test now — `_is_leading_title` and the period-abbreviation pattern moved down into group, with `_leading_titles` as the one definition of where the name begins: the chain's walk, the bound join's first name piece (the second docs review found that one still on group's test — `Xyz. abdul John Smith` joined nothing where `Dr. abdul John Smith` read given 'abdul John') and, through `_is_leading_title`, the scan and the PARTICLE_OR_GIVEN report — so `Xyz. van …` reads as `Dr. van …` does. `Esq. van Gogh`, a corpus name, moves with it (1.4.0 read family 'van Gogh' for both, as #367's rule records; its own rule at every baseline, and one for the P4 example). A particle of the unambiguous suffix vocabulary too (vd, mc) is a suffix piece to the peel: where it opens the trailing run the chain stops before it as before any suffix word and the peel takes it — `John Smith Mc V` reads suffix 'Mc, V' where master and 1.4.0 read family 'Mc V', a class the second docs review found unrecorded (2,554 of its 117,306 constructed names, an eighteen-word pool that puts the words in every position) — and where it continues a prefix run, the run takes it as a particle, so `John van Mc` keeps family 'van Mc', every baseline's reading and the one P6 chose for the shape after a comma. Both are Accepted under P2. The fork's count was the last thing the chain asked on its own authority, and the reviews found it wrong behind a title-and-particle word: #367 stops the leading-particle scan at `Freiherr`, `St`, `Do` — the name's own first piece — where assign's title peel steps over them, so the chain takes the name's first word, and the acronym the fork counted with three pieces met assign with two: `Freiherr von Berg MA` read given 'von Berg', family 'MA' (1.4.0's reading, as it happens; master read family 'von Berg MA'). So the chain asks the peel again over the pieces it leaves — `chain(tail)`, then `_trailing_start` over what it built, and a second run without the stop where the verdict changed, the snapshot being one copy per segment with a trailing run — and takes what assign will not peel: family 'von Berg MA'. The numeral cannot flip, a chain group never being initial-shaped, so `Freiherr von Richthofen V` keeps suffix 'V', and its chain, the one name piece left, reads as `Dr. Smith V` reads — where 1.4.0 and master read family 'von Richthofen V'; the code review asked for that boundary to be on record, and P2 carries it. #410 (2026-08-25) then moved the name half of both: a title with one name word behind it names the family whatever stands beside it, so the pair reads family 'von Richthofen' and family 'Smith' today, where each read `given` with no family when this entry was written. The boundary is unchanged — the chain still stops before the numeral, and the two names still read alike; only the field the one name piece lands in moved. This name is the fifth corpus name #410 moves, and the only one already classified, its ledger rule's `fields` spanning the narrower diff. ### P6 — the trailing orphan particle @@ -132,19 +132,19 @@ Open: [#380](https://github.com/derek73/python-nameparser/issues/380) covers "Be #417's question — whether `van der Berg née` is a family name — is answered by consistency rather than decided: a marker the consumer declines is M2's "just a word" and rides inside whatever join reaches it, which is the reading `Jane van der Berg née` already had. The fields of `Jane van der Berg née Jr Jones` do not change at all (middle 'van der Berg née Jr', family 'Jones'); only the pieces do, which is why its pin is at the piece level. The invariant test that hid the shape now lists it. - Review added two pins the first cut had left to prose. A corpus-wide assertion that appending a maiden clause moves no other field (tests/v2/test_parser.py), with #410's lone-residual shape as its one boundary — `Dr. Jane` reads family 'Jane' and `Dr. Jane née Smith` given 'Jane', which predates this and is untouched by it, so the claim is stated for names of two or more name words. And a field-level row for the chain-carried marker. One guard is deliberately left without a pin: `_is_maiden_marker_piece`'s lone-piece half has no default-vocabulary input that reaches it now that the pass runs first (dropping it leaves the suite and a 337k-name review sweep identical), and it stays as M2's own definition — "standing as a word of its own" — rather than as a guard a test holds. + Review added two pins the first cut had left to prose. A corpus-wide assertion that appending a maiden clause moves no other field (tests/v2/test_parser.py), with #410's lone-residual shape as its one boundary until #410 closed it on 2026-08-25 — `Dr. Jane née Smith` reads family 'Jane' now, as `Dr. Jane` does, so the assertion covers every corpus name that parses to anything, and the only skip left is M2's own (nothing at all before the marker). And a field-level row for the chain-carried marker. One guard is deliberately left without a pin: `_is_maiden_marker_piece`'s lone-piece half has no default-vocabulary input that reaches it now that the pass runs first (dropping it leaves the suite and a 337k-name review sweep identical), and it stays as M2's own definition — "standing as a word of its own" — rather than as a guard a test holds. Open (M2): [#317](https://github.com/derek73/python-nameparser/issues/317) the fullwidth-colon marker (旧姓：佐藤 arrives as one word; the head-peel question). -[#410](https://github.com/derek73/python-nameparser/issues/410) -a maiden name makes H1 decline, so a title-plus-surname name reports no family — pre-existing, and widened by the stop. (#411, the same empty family from P5's reserve, closed 2026-08-21.) - 2026-08-22 #424 — the walk stops before the trailing numeral assign reads as the suffix. "Up to any trailing suffix" was asked with the suffix-piece test, whose initial veto does not see a bare `V`: `John née Jones Smith V` read maiden 'Jones Smith V' where `John Smith V` reads suffix 'V'. The walk now also stops at `_trailing_start` — the S2 peel read from the marker on, over the pieces as they stand, stepping over a tail segment's delimiter cores as the walk does — but at the NUMERAL only. The bare-acronym fork counts pieces, and the walk removes the very pieces it counted: `John née Jones Smith Ma` peeled over the pieces as they stand reads the acronym as a credential with words to spare, and once 'Jones Smith' has left the name it is the family of a two-piece name — maiden 'Jones Smith', family 'Ma', which is worse than the maiden 'Jones Smith Ma' it had always read. The numeral fork reads the piece before the numeral, and the take removes that very piece: afterwards assign sees the piece before the MARKER there, and where that is initial-shaped the fork does not fire — the first draft stopped the walk all the same, and `J. née Jones Smith V` read maiden 'Jones Smith', family 'V' (both reviews found it: 27 of the design-docs review's 144 marker shapes, every one a head ending in an initial; the pool is unrecorded). The walk runs the peel again over the view the take would leave — the pieces before the marker and the numeral on, built as P5's reserve builds its view — so it stops only where the numeral reads as the suffix both as written and as left: `J. née Jones Smith V` keeps maiden 'Jones Smith V', as `J. V` reads the V as a name, and `Dr. née Jones Smith V` keeps maiden 'Jones Smith V', since assign peels the title first and a numeral alone is no fork (the first re-ask checked the preceding piece only and handed that V to the given name — the code review). As written too: `Jane née Jones J. V` keeps maiden 'Jones J. V' though the take would leave `Jane V`, the words as written being what the marker takes; the second docs review found the statement saying only the second half. The acronym is left to assign, recorded as Accepted. The peel is read from the marker rather than after it, so `Jane Smith née V` declines as `Jane Smith née PhD` does — nothing after the marker but a suffix, the marker stays a word, suffix 'V' — which is 1.4.0's reading (the first draft read maiden 'V', and the 1.4.0 gate is what showed it); and `Jane née Jones J. V` keeps maiden 'Jones J. V' (the piece before is initial-shaped). 1.4.0 had no maiden support, so the examples fall under the #274 rule at that baseline where its fields reach, and `J. née Jones Smith V` (v1 read suffix 'V') under an accepted rule of its own; against 2.0.0 and 2.1.0 the examples that move are classified — `John née Jones Smith V` in maiden and suffix, `Jane Smith née V` in the five fields the marker's survival touches — and `J. née Jones Smith V` is 2.x parity. +- 2026-08-25 #410 — the maiden name no longer makes H1 decline; the rule counts name words and a marker-announced name is not one. The decision and its width are recorded at decisions.md#H1, since the term it removed was never maiden-specific. + ### N3 — the lone-word nickname rule -- 2026-07 (v2 core, PR #288; recorded plan deviation #2 of the core plan) — v1's rule counted pieces before grouping; the v2 port fires only when the nickname accompanies exactly ONE piece in total — a title counts against it, which is why "'Smitty' Dr. Jones" reads given="Jones" with the family empty (rules.md#N3's Accepted consequence) rather than family="Jones". The rule lives in assignment because that is where the piece count is settled. (An earlier wording here said "one non-title piece", predicting the opposite output; the coherence review measured the truth.) +- 2026-07 (v2 core, PR #288; recorded plan deviation #2 of the core plan) — v1's rule counted pieces before grouping; the v2 port fires only when the nickname accompanies exactly ONE piece in total — a title counts against it, so "'Smitty' Dr. Jones" reaches H1 with a title and one name word left standing — through 2.1 that meant given="Jones" with the family empty, and since #410 (2026-08-25) H1 names the family, so it reads family="Jones". The count is unchanged; what moved is what happens after it declines. The rule lives in assignment because that is where the piece count is settled. (An earlier wording here said "one non-title piece", predicting the opposite output; the coherence review measured the truth.) ### N2 — same-character quotes and shared-character conventions @@ -289,6 +289,10 @@ Declined (ambiguity kinds for script-resolved names, 2026-07-27): - A "han-script" zh-vs-ja kind — applying the ZH pack IS the disambiguation; per-name flags after an explicit opt-in are noise. - ORDER emission for script-resolved names — native-script order is convention, not a guess; the reserved kind stays unemitted. SEGMENTATION fires only on a genuine multi-split vocabulary fork (夏侯惇, 남궁민수). +### H1 — a title and one name word + +- 2026-08-25 (#410) — H1's "and nothing else" counted a suffix, a nickname and a maiden name as further name words, so a title-plus-surname name reported no family the moment any of them stood beside it: `Dr. Smith` reads family 'Smith' and `Dr. Smith née Jones` read given 'Smith' with the family empty. #410 reports the maiden flavor; measured, all three roles suppress the rule identically and the fix is one term in one guard, so the term went rather than the maiden role being special-cased. What decided the width is H1's own rationale — a title addresses by surname — which says nothing about what stands BESIDE the name. The rationale is stated for H1 rather than as a doc-wide principle about what a suffix is, deliberately: N3 counts a suffix the other way (`'Smitty' Jones Jr.` reads given 'Jones', family ''), and which of the two readings is right for a nickname-led name is not decided here. #399 is what made this urgent rather than latent: stopping the particle chain at the marker routed the canonical title-and-particle shape (`Freiherr von Richthofen geb. Albrecht`) into H1 for the first time. N3 moves with it, without N3 changing: `'Smitty' Dr. Jones` declines N3's one-piece count as it always did, and H1 now names the family behind it, so it reads family 'Jones' where it read given 'Jones' through 2.1. Five corpus names change reading at every baseline, of which four arrive as new diffs and are classified with the fix. The fifth, `Freiherr von Richthofen V`, was already classified under the `fix(#424)` rule: #410 narrows its diff from {given, family, suffix} to {family, suffix}, and `classify()` takes the first rule whose declared `fields` are a SUPERSET of the observed diff, so a shrinking diff kept matching and no run ever named it. A real movement behind a green gate, and the second of that kind here: the fields-only `fix(suffix-routing)` catch-all was absorbing three names at 1.4.0 the same way, its heading growing with nothing to announce it. Both are one shape — a rule broader than the diff it explains — and that is the lesson worth keeping rather than either fix. It is also, being a title, one name word and a suffix, the suffix flavor of this very shape, which two ledger comments had claimed no corpus carries. At 1.4.0 three of the four were being absorbed by the fields-only `fix(suffix-routing)` rule (its heading went 14 -> 17) — invisible to `_CORPUS_CLAIMS`, which records the whole corpus for a rule with no `name_regex`, so the guard could not see them arrive. They have their own rule now, as #372's names got one off the same catch-all. The v1 suite shipped the nickname reading as a strict xfail (`tests/test_nicknames.py::test_nickname_and_last_name_with_title`), which passes now. Known gap between the statement and the guard: H1 says "exactly one name word" and "that word", but the guard never counts units. It tests which ROLES are unoccupied — no middle, no family — and then retags every given token, so a name whose given run is two name WORDS (units, not tokens) reaches it and moves both: `Freiherr de V Jr` reads family 'de V'. The distinction matters for reading the claim that follows: a multi-token given run is ordinary — the rules.md example `Freiherr von Richthofen V` has one, its two tokens being a single particle unit — and it is a run of two UNITS that no real input produces. A 58,338-name sweep found no corpus name and no rules.md example with one — but it is recorded because decisions.md elsewhere leans on H1's "exactly one name word" as a settled scope when narrowing P5's licence, and that reading of H1 is stronger than the code behind it. Not fixed here: the widening this entry records is about which roles disqualify the rule, not about how it counts. + ### H2 — the leading-abbreviation title - 2026-06-30 (leading-period-title design; v2 core, PR #288) — the shape test is v1 parity (period_abbreviation): two-plus letters then a period, bare initials exempt. Its site is the head of the part CARRYING THE GIVEN NAME — the whole name, or the post-comma part under a family comma — not "the head of the name"; that scope correction is PR #315 (2026-08-01, docs-only), verified against 1.4.0 from PyPI, so the parity claim is real and the narrower description never was. The extraction litmus (2026-08-15): the spec drafted this rule as diff --git a/docs/design/rules.md b/docs/design/rules.md index 2f62e2e7..69f17dc9 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -29,18 +29,22 @@ Background: an honorific title precedes a name and is not itself part of it; it H1. Rationale: a title normally addresses by surname, so a title followed by a single name word usually names the family; but a - given-name title addresses by given name. - A title followed by exactly one name word and nothing else makes - that word the family name, unless the title is a given-name - title, which keeps it the given name. + given-name title addresses by given name. What stands beside + that word — a suffix, a nickname, a maiden name — does not make + the name any longer, so it does not decide this reading. + A title followed by exactly one name word makes that word the + family name, whatever suffix, nickname or maiden name stands + beside it, unless the title is a given-name title, which keeps + it the given name. "Mr. Johnson" → family="Johnson" "Mrs. Garcia" → family="Garcia" + "Dr. Smith née Jones" → family="Smith" "Sir John" → given="John" · boundary Accepted: a given-name title plus one name word leaves the family empty — the input names no family, and inventing one would be worse. "Sir John" → family="" - interacts: P5 · implemented: nameparser/_pipeline/_post_rules.py + history: decisions.md#H1 · interacts: P2, P3, P5, M2, S1, S2, N1, N3 · implemented: nameparser/_pipeline/_post_rules.py H2. Rationale: before a name, an abbreviation is almost always a title — "Rev.", "Ing.", "Mag." — and no vocabulary can list @@ -513,11 +517,17 @@ N3. Rationale: a person set down as a nickname plus one name word is positional reading applies. "'Smitty' Jones" → family="Jones" "'Smitty' John Jones" → given="John" · boundary - Accepted: the count does not set suffixes or titles aside, so a - nickname plus one name word plus a suffix reads the name word - as given and leaves the family empty. + Accepted: the count does not set suffixes aside, so a nickname + plus one name word plus a suffix reads the name word as given + and leaves the family empty. A title counts against the count + too, but H1 then reads the title-plus-one-word name that is + left, so the family is named after all — unless the title is a + given-name title, which keeps the word in `given` and leaves no + family, exactly as it does anywhere else. "'Smitty' Jones Jr." → family="" - history: decisions.md#N3 · implemented: nameparser/_pipeline/_assign.py + "'Smitty' Dr. Jones" → family="Jones" + "'Smitty' Sir John" → given="John" + history: decisions.md#N3 · interacts: H1 · implemented: nameparser/_pipeline/_assign.py ## Maiden names (M) @@ -587,7 +597,7 @@ M2. Rationale: a maiden marker announces that what follows it is the is maiden text all the same — the count it needs includes the very words the marker removes, so the reading is left to assign. "John née Jones Smith Ma" → maiden="Jones Smith Ma" - history: decisions.md#M2 · interacts: P2, P3, P5, R2, M1, S2 · implemented: nameparser/_pipeline/_group.py + history: decisions.md#M2 · interacts: P2, P3, P5, R2, M1, S2, H1 · implemented: nameparser/_pipeline/_group.py ## Commas & structure (C) diff --git a/docs/release_log.rst b/docs/release_log.rst index 92c37019..30d701ee 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -12,11 +12,15 @@ Release Log Nothing moved between vocabularies. The rename itself changes no parse at all; the parsing changes below are separate fixes, and - only one of them reaches the default name order -- a title no - longer changing how the name behind it is read, which moves four - names of the differential corpora. Both family-first orders change too, - below -- seven of those from the family-first fix, plus the names - the title fix moves in every order. The corpora have roughly + most of them reach the default name order -- each bullet names + the shapes and the corpus names it moves, and the gate output is + the source of record. The unusual part is the family-first + orders, which change too: seven names from the family-first fix, + plus the names #367's title fix moves, which it moves in every + order. The other title fix below, #410, is default-order only -- + under either family-first order the word behind the title is + already the family, so the rule is a no-op there and moves + nothing (measured over all four corpora). The corpora have roughly doubled across this cycle (#414 added the rules doc's examples, #413 taught the issue harvester to read backticks), so the counts in the bullets below are the classified summary at the time each @@ -69,7 +73,9 @@ Release Log - Fix a bound given-name join leaving no family name when the name also carries a maiden clause, and stop the join absorbing the marker itself. ``"abdul Berg née Jones"`` read given ``abdul Berg`` with an EMPTY family, where ``"abdul Berg"`` alone correctly reads given ``abdul``, family ``Berg``. The join reserves a name word so it always leaves a family name behind -- two name words alone do not join -- but the reserve was counted while the maiden marker and the maiden name were still part of the name, and the pass that removes them runs afterwards. Four words counted, the join fired, and when the two departed nothing was left for the family. The words a marker will take no longer count toward the reserve, so that name reads given ``abdul``, family ``Berg``, maiden ``Jones``. Excluding them is not sufficient on its own: where a suffix sits INSIDE the name the marker walk stops early, enough words survive to clear the reserve, and the join fired and took the marker -- ``"abd née Jones Jr Smith Berg"`` read given ``abd née``. The join now declines outright when the piece it would absorb is a marker, which is what the rule meant all along: it joins the bound word to a name word, and a marker is not one. A name with a word to spare is untouched -- ``"abd Allah Smith née Jones"`` still joins, giving given ``abd Allah``, family ``Smith``, maiden ``Jones``. This is pre-existing for the spellings that were already vocabulary: ``abdul``, ``abdel`` and ``abdal`` read this way at 2.0.0 and 2.1.0 alike. ``abd`` is different -- it reads CORRECTLY at 2.1.0 and acquired the defect only when this same release made it a bound given-name word, so for that spelling this fixes a regression introduced earlier in the cycle; and the particle spellings reached the reserve for the first time through the maiden-marker chain stop above. Two consequences worth knowing. The bound-given join no longer swallows a maiden marker at all, so ``"van der Berg, abdul née Jones"`` reads given ``abdul``, family ``van der Berg``, maiden ``Jones`` where it read given ``abdul née``, middle ``Jones``; the connective join is the remaining case, tracked at `#412 `_. And where the bound word is ALSO suffix vocabulary, a declining join after a family comma leaves the post-nominal reading and the name has no given name: ``"Berg, abd née Jones"`` reads family ``Berg``, suffix ``abd``, maiden ``Jones``, matching how ``"Berg, abd"`` alone has always parsed. ``abd`` is the only shipped word in both sets (closes #411) - - Fix a maiden clause changing how the rest of the name is read, and a connective join keeping the marker in the surname. A maiden marker and the name it introduces are not part of the name they follow, but the grouping rules that count a name's words -- the three-word test that keeps a single-letter connective a name word, and the bound given-name reserve -- counted them, because the pass that removes them ran after those rules. ``"juan y garcia"`` reads given ``juan``, middle ``y``, family ``garcia``, but ``"juan y garcia nee jones"`` counted five words, joined the ``y``, and read given ``juan y garcia`` with NO family name at all; ``"John e Smith nee jones"``, ``"Lt.Gov. juan e garcia nee jones"`` and six more differential corpus names lose the family the same way under an appended clause. The marker pass now runs before every join, so the joins and the counts see only the name that remains, and a name of two or more name words reads as it reads without its maiden clause, plus the maiden name: ``"juan y garcia nee jones"`` is given ``juan``, middle ``y``, family ``garcia``, maiden ``jones``. (A title plus ONE name word is the pre-existing #410 shape, where the lone word sits in ``family`` alone and in ``given`` once a maiden name follows -- ``"Dr. Jane"`` against ``"Dr. Jane née Smith"`` -- and is unchanged here.) The same order closes the last of the join-swallows. The connective join used to merge the marker into a multi-word piece before the marker rule could see it, so ``"Jane van der Berg née y Jones"`` kept family ``van der Berg née y Jones`` with no maiden name; it now reads family ``van der Berg``, maiden ``y Jones`` -- the marker takes the words after it, connective included -- and ``"Jane née and Jones Smith"`` reads given ``Jane``, maiden ``and Jones Smith`` where it read middle ``née and Jones``, family ``Smith``. It also retires the particle chain's marker stop from the #399 fix above, whose condition restated the marker rule's and disagreed with it one suffix later (``"Jane van der Berg née Jr Jones"``, #417): a marker the rule declines -- nothing after it but a suffix -- is an ordinary word and rides inside the chain, which is the reading ``"Jane van der Berg née"`` already had, and the parsed fields of that name do not change. Two limits. A bound given-name word still never joins onto a marker standing as a word of its own, so ``"Berg, abdul née PhD"`` keeps given ``abdul``; a declined marker the particle chain has taken travels with the chain, so ``"Abd van der Berg née Jr Jones"`` reads given ``Abd van der Berg née`` where it read given ``Abd van der Berg``, middle ``née Jr``. And the marker rule now sees the words as written rather than as joined, so a suffix-vocabulary word inside the maiden name stops it even with a connective beside it: ``"Jane née Jr y Jones"`` read maiden ``Jr y Jones`` and now reads family ``Jr y Jones`` with no maiden name, and ``"Jane Smith née Jones Jr y Smith"`` read maiden ``Jones Jr y Smith`` and now reads maiden ``Jones``, family ``Jr y Smith``. One consequence: ``"abd née Jones Jr Smith Berg"`` reads given ``abd Jr`` -- as ``"abd Jr Smith Berg"`` already does -- where the #411 fix above read given ``abd``. Four of the 1025 differential corpus names move. The two connective-join names move at the 2.0.0 and 2.1.0 baselines alike; at 1.4.0 both were already classified as maiden-marker changes, 1.4 having no maiden field. ``"Juan y Garcia née Jones"`` and ``"Jane née Jr y Jones"``, the examples this fix adds to the rules doc and so to the rules corpus, move at all three and at 2.0.0/2.1.0 respectively. The eight corpus names that lose a family only under an APPENDED clause are not in the corpora in that form, so the gate cannot see them; the case table pins that half (closes #412, closes #417, closes #418) + - Fix a maiden clause changing how the rest of the name is read, and a connective join keeping the marker in the surname. A maiden marker and the name it introduces are not part of the name they follow, but the grouping rules that count a name's words -- the three-word test that keeps a single-letter connective a name word, and the bound given-name reserve -- counted them, because the pass that removes them ran after those rules. ``"juan y garcia"`` reads given ``juan``, middle ``y``, family ``garcia``, but ``"juan y garcia nee jones"`` counted five words, joined the ``y``, and read given ``juan y garcia`` with NO family name at all; ``"John e Smith nee jones"``, ``"Lt.Gov. juan e garcia nee jones"`` and six more differential corpus names lose the family the same way under an appended clause. The marker pass now runs before every join, so the joins and the counts see only the name that remains, and a name of two or more name words reads as it reads without its maiden clause, plus the maiden name: ``"juan y garcia nee jones"`` is given ``juan``, middle ``y``, family ``garcia``, maiden ``jones``. (A title plus ONE name word is the #410 shape, fixed separately below: ``"Dr. Jane née Smith"`` reads family ``Jane`` now, as ``"Dr. Jane"`` does.) The same order closes the last of the join-swallows. The connective join used to merge the marker into a multi-word piece before the marker rule could see it, so ``"Jane van der Berg née y Jones"`` kept family ``van der Berg née y Jones`` with no maiden name; it now reads family ``van der Berg``, maiden ``y Jones`` -- the marker takes the words after it, connective included -- and ``"Jane née and Jones Smith"`` reads given ``Jane``, maiden ``and Jones Smith`` where it read middle ``née and Jones``, family ``Smith``. It also retires the particle chain's marker stop from the #399 fix above, whose condition restated the marker rule's and disagreed with it one suffix later (``"Jane van der Berg née Jr Jones"``, #417): a marker the rule declines -- nothing after it but a suffix -- is an ordinary word and rides inside the chain, which is the reading ``"Jane van der Berg née"`` already had, and the parsed fields of that name do not change. Two limits. A bound given-name word still never joins onto a marker standing as a word of its own, so ``"Berg, abdul née PhD"`` keeps given ``abdul``; a declined marker the particle chain has taken travels with the chain, so ``"Abd van der Berg née Jr Jones"`` reads given ``Abd van der Berg née`` where it read given ``Abd van der Berg``, middle ``née Jr``. And the marker rule now sees the words as written rather than as joined, so a suffix-vocabulary word inside the maiden name stops it even with a connective beside it: ``"Jane née Jr y Jones"`` read maiden ``Jr y Jones`` and now reads family ``Jr y Jones`` with no maiden name, and ``"Jane Smith née Jones Jr y Smith"`` read maiden ``Jones Jr y Smith`` and now reads maiden ``Jones``, family ``Jr y Smith``. One consequence: ``"abd née Jones Jr Smith Berg"`` reads given ``abd Jr`` -- as ``"abd Jr Smith Berg"`` already does -- where the #411 fix above read given ``abd``. Four of the 1025 differential corpus names move. The two connective-join names move at the 2.0.0 and 2.1.0 baselines alike; at 1.4.0 both were already classified as maiden-marker changes, 1.4 having no maiden field. ``"Juan y Garcia née Jones"`` and ``"Jane née Jr y Jones"``, the examples this fix adds to the rules doc and so to the rules corpus, move at all three and at 2.0.0/2.1.0 respectively. The eight corpus names that lose a family only under an APPENDED clause are not in the corpora in that form, so the gate cannot see them; the case table pins that half (closes #412, closes #417, closes #418) + + - Fix a title-plus-surname name losing its family name whenever anything stood beside it. ``"Dr. Smith"`` reads family ``Smith``, but ``"Dr. Smith née Jones"`` read given ``Smith`` with no family at all, and so did ``"Dr. Smith PhD"`` and ``"Dr. "Smitty" Smith"``. A title followed by a single name word names the family, and that rule declined whenever the name also carried a suffix, a nickname or a maiden name -- it counted them as further name words, which none of them is: each stands beside the name rather than in it, and none makes the name any longer. The rule now counts name words alone. ``"Dr. Smith née Jones"`` reads family ``Smith``, maiden ``Jones``; ``"Dr. Smith PhD"`` family ``Smith``, suffix ``PhD``; ``"Dr. "Smitty" Smith"`` family ``Smith``, nickname ``Smitty``; and the particle spelling this was found through, ``"Freiherr von Richthofen geb. Albrecht"``, reads family ``von Richthofen``, maiden ``Albrecht`` -- 2.2's own particle-chain stop had routed that canonical shape into the rule for the first time (#399). A given-name title is unchanged and still names no family: ``"Sir John née Jones"`` keeps given ``John`` with an empty family, exactly as ``"Sir John"`` does. One name moves where the nickname LEADS, rather than standing beside a name that already reads: ``"'Smitty' Dr. Jones"``. The lone-word nickname rule declines there because the title counts against its piece count, which leaves a title and one name word for this fix to read as the family -- given ``Jones`` through 2.1, family ``Jones`` now. Five names of the differential corpora change reading, the same five at every baseline. Four of them arrive as new diffs and are classified with this fix: ``"Dr. Smith née Jones"``, ``"Senator "Rick" Edmonds"``, ``"Xyz. (Bud) Smith"`` and ``"'Smitty' Dr. Jones"``. The fifth, ``"Freiherr von Richthofen V"``, was already classified under 2.2's particle-chain fix and stays there -- it is the suffix flavor in corpus form, and reads family ``von Richthofen`` with suffix ``V`` where it read given ``von Richthofen`` before. v1 read the two nicknames as 2.1 did; it had no maiden support at all, reading ``"Dr. Smith née Jones"`` as first ``Smith``, middle ``née``, last ``Jones``. The v1 test suite shipped the correct reading for the nickname shape as a known-failing test, which now passes (closes #410) **Deprecations** diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 74adca1c..6c1436be 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -9,7 +9,7 @@ Lexicon.given_name_titles. Implements rules H1, P1, O1, O2 and O3 of docs/design/rules.md; each -is cited at its code below, and P1/O1/O2's history lives in +is cited at its code below, and H1/P1/O1/O2's history lives in docs/design/decisions.md. """ from __future__ import annotations @@ -174,13 +174,15 @@ def post_rules(state: ParseState) -> ParseState: givens = _idx(tokens, Role.GIVEN) middles = _idx(tokens, Role.MIDDLE) families = _idx(tokens, Role.FAMILY) - others = any(t.role in (Role.SUFFIX, Role.NICKNAME, Role.MAIDEN) - for t in tokens) - # rules.md#H1: "a title followed by exactly one name word and - # nothing else makes that word the family name, unless the title - # is a given-name title" (v1 handle_firstnames) - if titles and givens and not middles and not families and not others: + # rules.md#H1: "a title followed by exactly one name word makes + # that word the family name, whatever suffix, nickname or maiden + # name stands beside it, unless the title is a given-name title, + # which keeps it the given name" -- counting those three as + # further name words is what emptied the family (#410) + # (known gap: the guard tests which roles are unoccupied, it does + # not count units -- decisions.md#H1) (v1 handle_firstnames) + if titles and givens and not middles and not families: joined = _title_key(tokens[i].text for i in titles) if joined not in state.lexicon.given_name_titles: for i in givens: diff --git a/tests/test_nicknames.py b/tests/test_nicknames.py index e7207653..62ccd8dd 100644 --- a/tests/test_nicknames.py +++ b/tests/test_nicknames.py @@ -153,7 +153,6 @@ def test_nickname_and_last_name(self) -> None: self.m(hn.last, "Edmonds", hn) self.m(hn.nickname, "Rick", hn) - @pytest.mark.xfail def test_nickname_and_last_name_with_title(self) -> None: hn = HumanName('Senator "Rick" Edmonds') self.m(hn.title, "Senator", hn) diff --git a/tests/test_titles.py b/tests/test_titles.py index 0cf6ce6d..19e68396 100644 --- a/tests/test_titles.py +++ b/tests/test_titles.py @@ -372,7 +372,8 @@ def test_leading_period_abbreviation_case_insensitive(self) -> None: def test_leading_period_abbreviation_with_nickname(self) -> None: hn = HumanName("Xyz. (Bud) Smith") self.m(hn.title, "Xyz.", hn) - self.m(hn.first, "Smith", hn) + self.m(hn.first, "", hn) + self.m(hn.last, "Smith", hn) self.m(hn.nickname, "Bud", hn) def test_charge_daffaires_chains_as_title(self) -> None: diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 055a1deb..82f13360 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -162,6 +162,100 @@ def __post_init__(self) -> None: "1b folds the name into the family. 1.4.0 and 2.1 gave " "first 'de Mesnil' with no family, because the chain " "left 1b nothing standing alone to fire on"), + Case("title_plus_one_word_with_maiden", "Dr. Smith née Jones", + {"title": "Dr.", "family": "Smith", "maiden": "Jones"}, + classification="fix(#410)", + notes="H1's 'and nothing else' counted a maiden name as a " + "further name word, so adding a maiden clause moved the " + "surname into `given` and emptied `family`: 'Dr. Smith' " + "reads family 'Smith' and 'Dr. Smith née Jones' read " + "given 'Smith'. A maiden name is announced beside the " + "name, not part of it. 1.4.0 had no maiden SUPPORT -- " + "the field exists, but no default marker vocabulary " + "routes to it -- and read first 'Smith', middle 'née', " + "last 'Jones'"), + Case("title_plus_one_word_with_maiden_particle_spelling", + "Freiherr von Richthofen geb. Albrecht", + {"title": "Freiherr", "family": "von Richthofen", + "maiden": "Albrecht"}, + classification="fix(#410)", + ambiguities=("particle-or-given",), + notes="the spelling #410 was found through: #399 stopped the " + "particle chain at the marker, which routed the " + "canonical title-and-particle shape into H1 for the " + "first time and so into this bug. 1.4.0 read the whole " + "tail as one surname, last 'von Richthofen geb. " + "Albrecht'"), + Case("given_name_title_plus_one_word_with_maiden", + "Sir John née Jones", + {"title": "Sir", "given": "John", "maiden": "Jones"}, + classification="fix(#274)", + notes="H1's carve-out is untouched by #410: a given-name " + "title addresses by given name, so the one name word " + "stays `given` and the family stays empty, exactly as " + "'Sir John' reads -- in the DEFAULT order, which is " + "what this row pins: under either family-first order " + "the same name reads family 'John', H1 being a no-op " + "there because assign has already placed the word. " + "The boundary of the widening, and " + "the row that fails if the retag is made unconditional. " + "The fix classification is #274's marker consumption, " + "which is what makes this differ from 1.4.0 (first " + "'John', middle 'née', last 'Jones')"), + Case("title_plus_one_word_comma_suffix", "Dr. King, Jr.", + {"title": "Dr.", "family": "King", "suffix": "Jr."}, + classification="fix(#410)", + notes="the most ordinary shape H1's widening touches, and " + "the one the suite could least afford to leave " + "unpinned: mutating H1 to decline on any name carrying " + "a comma left all 5819 tests green. Nothing else sees " + "the class -- the corpus-wide maiden-clause property " + "test filters commas out of its parametrization " + "(_clause_free_latin_corpus_names), and " + "'Smith, Dr.' takes its family from the comma rule " + "(C1), not from H1. 1.4.0 had the same empty family " + "here (title 'Dr.', first 'King', last ''), so this " + "row records a v1 bug fixed, not a v2 divergence"), + Case("title_plus_one_word_multi_word_maiden", + "Dr. Smith née Mary Jones", + {"title": "Dr.", "family": "Smith", "maiden": "Mary Jones"}, + classification="fix(#410)", + notes="the maiden name at two words rather than one. H1 " + "counts what stands in the NAME, so the arity of the " + "clause beside it is irrelevant -- a rule that " + "declined on a second maiden token would pass every " + "single-word row above. 1.4.0 read first 'Smith', " + "middle 'née Mary', last 'Jones'"), + Case("given_name_title_plus_one_word_multi_word_maiden", + "Sir John née Mary Jones", + {"title": "Sir", "given": "John", "maiden": "Mary Jones"}, + classification="fix(#410)", + notes="the carve-out at the same arity: 'whatever maiden " + "name stands beside it' has to leave the given-name " + "title alone however long the clause is. Without this " + "row the `whatever` is asserted only at one word. " + "1.4.0 read first 'John', middle 'née Mary', last " + "'Jones'"), + Case("title_plus_one_word_two_suffixes", "Dr. Smith PhD Jr.", + {"title": "Dr.", "family": "Smith", "suffix": "PhD, Jr."}, + classification="fix(#410)", + notes="two suffix pieces, not one. The removed term asked " + "whether ANY token carried a suffix role, so a guard " + "rebuilt to decline on the SECOND one would look " + "correct against every row that carries a single " + "credential. 1.4.0 split them, reading first 'Smith', " + "last 'PhD', suffix 'Jr.'"), + Case("title_plus_one_word_nickname_and_suffix", + "Dr. (Bud) Smith Jr.", + {"title": "Dr.", "family": "Smith", "suffix": "Jr.", + "nickname": "Bud"}, + classification="fix(#410)", + notes="two DIFFERENT annotations at once, which is the " + "combination the single-annotation rows cannot reach: " + "a guard declining only where a nickname and a suffix " + "are both present passes all of them. 1.4.0 read first " + "'Smith', last 'Jr.' with nickname 'Bud' -- the " + "credential taking the family slot"), # The smallest shape in which the two family-first orders can # disagree about this rule's leftovers: with one leftover both # send it to `given`, and two or more is what separates them (the @@ -349,7 +443,7 @@ def __post_init__(self) -> None: "skipped 'St'/'Do'/'Freiherr' and collapsed the " "untitled 'St John Smith' into one given name"), Case("titled_ambiguous_particle_no_op_chain", "St Van Jr.", - {"title": "St", "given": "Van", "suffix": "Jr."}, + {"title": "St", "family": "Van", "suffix": "Jr."}, notes="the piece after the particle is a suffix, so the chain " "scan never advances and the merge is a no-op -- nothing " "was chained, so there is no fork to report (the emitter " @@ -367,7 +461,10 @@ def __post_init__(self) -> None: "Not parity, and not #367's doing either: 1.4.0 reads " "'Do Van Jr.' as first 'Do Van', last 'Jr.', so the " "divergence is 2.0's suffix routing plus 'do' being a " - "title -- both older than this row's respelling", + "title -- both older than this row's respelling" + " -- and since #410 the one name word left standing " + "behind the title reads as the family, the suffix no " + "longer counting as something else in the name", classification="fix"), Case("initial_shaped_not_conjunction", "john e. smith", {"given": "john", "middle": "e.", "family": "smith"}, @@ -532,10 +629,14 @@ def __post_init__(self) -> None: notes="suffix-ACRONYM membership alone strips periods (v1 " "is_suffix parity)"), Case("nickname_rule_counts_whole_segment", "Xyz. (Bud) Smith", - {"title": "Xyz.", "given": "Smith", "nickname": "Bud"}, + {"title": "Xyz.", "family": "Smith", "nickname": "Bud"}, + classification="fix(#410)", notes="v1's lone-piece nickname rule counts the segment " "BEFORE title peeling (parser.py:1285, pinned live " - "2026-07-17)"), + "2026-07-17), which is what this row pins and what " + "#410 does not change. The FIELD moved: a nickname is " + "not a further name word, so the one name word behind " + "the title is the family. 1.4.0 read first 'Smith'"), Case("suffix_comma_decided_by_first_segment", "Dr. John P. Doe-Ray, CLU, CFP, LUTC", {"title": "Dr.", "given": "John", "middle": "P.", @@ -881,8 +982,9 @@ def __post_init__(self) -> None: "now sees the name that remains, so the clause changes " "nothing about how the rest of this name reads -- the " "reading is 'juan y garcia' plus a maiden name. " - "test_parser.py asserts that over the corpus, with " - "#410's lone-residual shape as the one boundary"), + "test_parser.py asserts that over the corpus, and " + "since #410 the only names it steps over are those " + "that parse to nothing at all"), Case("bound_given_reserve_excludes_a_multi_word_maiden_name", "Abd Berg née Mary Jones", {"given": "Abd", "family": "Berg", "maiden": "Mary Jones"}, @@ -1379,11 +1481,14 @@ def __post_init__(self) -> None: "(first=Johnson last=PhD); v2 keeps recognized " "suffixes in suffix"), Case("suffix_stays_suffix_title", "Mr. Johnson PhD", - {"title": "Mr.", "given": "Johnson", "suffix": "PhD"}, - classification="fix(suffix-routing)", - notes="v1 routes a lone trailing suffix to family " - "(title=Mr. first=Johnson last=PhD); v2 keeps " - "recognized suffixes in suffix"), + {"title": "Mr.", "family": "Johnson", "suffix": "PhD"}, + classification="fix(#410)", + notes="two fixes meet here. v1 routed a lone trailing suffix " + "to family (title 'Mr.', first 'Johnson', last 'PhD') " + "and v2 keeps recognized suffixes in `suffix` " + "(fix(suffix-routing)); that left 'Johnson' in `given` " + "with an empty family until #410 stopped counting the " + "suffix as a further name word"), Case("family_comma_lone_title", "Smith, Dr.", {"title": "Dr.", "family": "Smith"}, classification="fix(comma-family)", diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index f81cfa2f..819c5c57 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1239,6 +1239,10 @@ def _claim(rule: dict) -> _Claim: #: both is growth into names the rule genuinely describes. _CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { "expected_since_1.4.0.toml": { + "fix(#410) a title and one name word name the family, whatever annotation stands beside it": + _Claim(3, ('family', 'given'), "24d6223e472f"), + "fix(#410) the maiden flavor, where 1.4.0 read the marker as a middle name": + _Claim(1, ('family', 'given', 'maiden', 'middle'), "309e39fc2475"), "fix(#432) a dotted numeral behind a name is a middle initial, not the generation": _Claim(1, ('middle', 'suffix'), "e9f282da0d0f"), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": @@ -1272,7 +1276,7 @@ def _claim(rule: dict) -> _Claim: "fix(comma-precomma-family) pre-comma run reads as family, not given": _Claim(279, ('family', 'given'), "28a62b622a48"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(1073, ('family', 'given', 'suffix'), "4109210ea38c"), + _Claim(1075, ('family', 'given', 'suffix'), "97934f29bdc8"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1338,13 +1342,15 @@ def _claim(rule: dict) -> _Claim: "fix(#424) accepted: the chain keeps an acronym assign will not peel behind a title-and-particle word": _Claim(1, ('family', 'given'), "faa4bedda537"), "fix(#424) a title-led chain before the numeral is the one name piece": - _Claim(1, ('family', 'given', 'suffix'), "5b3a743f9e35"), + _Claim(1, ('family', 'suffix'), "5b3a743f9e35"), "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": _Claim(1, ('family', 'middle', 'suffix'), "a564b97f7162"), "fix(#360) ste moved into the never-given particles with mc": _Claim(1, ('family', 'given'), "e62caedec864"), }, "expected_since_2.0.0.toml": { + "fix(#410) a title and one name word name the family, whatever annotation stands beside it": + _Claim(4, ('family', 'given'), "da1dd1473145"), "fix(#430) a credential run does not end at the roman numeral describing it": _Claim(2, ('given', 'suffix'), "3c8fa6bc827a"), "fix(#432) a dotted numeral behind a name is a middle initial, not the generation": @@ -1404,7 +1410,7 @@ def _claim(rule: dict) -> _Claim: "fix(#424) the particle chain stops before a bare acronym with words to spare": _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b"), "fix(#424) a title-led chain before the numeral is the one name piece": - _Claim(1, ('_ambiguities', 'family', 'given', 'suffix'), "5b3a743f9e35"), + _Claim(1, ('_ambiguities', 'family', 'suffix'), "5b3a743f9e35"), "fix(comma-family) a comma followed only by titles keeps the given/family split": _Claim(2, ('family', 'given'), "5bd9c6d96c38"), "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": @@ -1447,6 +1453,8 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'maiden'), "2150936a8c55"), }, "expected_since_2.1.0.toml": { + "fix(#410) a title and one name word name the family, whatever annotation stands beside it": + _Claim(4, ('family', 'given'), "da1dd1473145"), "fix(#430) a credential run does not end at the roman numeral describing it": _Claim(2, ('given', 'suffix'), "3c8fa6bc827a"), "fix(#432) a dotted numeral behind a name is a middle initial, not the generation": @@ -1492,7 +1500,7 @@ def _claim(rule: dict) -> _Claim: "fix(#424) the particle chain stops before a bare acronym with words to spare": _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b"), "fix(#424) a title-led chain before the numeral is the one name piece": - _Claim(1, ('_ambiguities', 'family', 'given', 'suffix'), "5b3a743f9e35"), + _Claim(1, ('_ambiguities', 'family', 'suffix'), "5b3a743f9e35"), "fix(comma-family) a comma followed only by titles keeps the given/family split": _Claim(2, ('family', 'given'), "5bd9c6d96c38"), "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": @@ -1793,7 +1801,7 @@ class _Excluded(NamedTuple): ("fix(comma-family)", "fix(comma-precomma-family)", "fix(suffix-routing)")), '(^|[\\w.]\\s+)[("\'][^)"\']+[)"\'](\\s+\\w|\\s*$)': - _Excluded(44, "203608eec291", ()), + _Excluded(46, "71eb2fa94553", ()), } @@ -1897,7 +1905,7 @@ def test_a_fields_narrowing_actually_narrows_something() -> None: Measured: deleting `fields = ["nickname", "middle"]` from the ASCII-pairs entry passes every other check in this tree. The entry - then refuses ANY diff on the 34 corpus names it captures -- + then refuses ANY diff on the 46 corpus names it captures -- including 'Jenny (Johnson) Baker' and 'Lon (Jr.) Williams', whose parens are a maiden name and a suffix, both under active development. Nothing failed, because none of those names diffs diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 3868539a..5181f818 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -631,9 +631,9 @@ def test_the_chain_and_the_walk_stop_where_the_peel_begins() -> None: # the numeral keeps its three pieces behind the same word, and the # chain, now the one name piece, reads as 'Dr. Smith V' reads n = parse("Freiherr von Richthofen V") - assert (n.given, n.family, n.suffix) == ("von Richthofen", "", "V") + assert (n.given, n.family, n.suffix) == ("", "von Richthofen", "V") n = parse("Dr. Smith V") - assert (n.given, n.family, n.suffix) == ("Smith", "", "V") + assert (n.given, n.family, n.suffix) == ("", "Smith", "V") # the walk takes the numeral only: an acronym between the maiden # name and the numeral is maiden text n = parse("Jane Smith née Jones Ma V") @@ -1332,15 +1332,25 @@ def test_a_maiden_clause_changes_nothing_else(name: str) -> None: Before the marker pass moved ahead of the joins, seven of these names failed this. - One boundary, and it is not the grouping stage's: a name whose - residual is a single name piece (a title plus one word, 'Dr. - Jane') places that piece in `family` alone and in `given` once a - maiden name exists -- #410's shape, pre-existing, so those names - are stepped over rather than asserted either way. + One skip, and it is M2's own boundary: a corpus name that parses + to no name word ('', '(', '()') gives the appended marker nothing + to stand behind, so M2 leaves it a word. A name that is only a + NICKNAME is in that class too, which is why the test below asks + about five fields and not about `nickname`: '(Bud)' parses to a + nickname alone, and '(Bud) née Jones' reads given 'née', family + 'Jones' and no maiden at all -- the marker stayed a word, so + there is no clause to assert. A title-only or suffix-only name is + NOT in that class -- 'Coach née Jones' reads maiden 'Jones' -- + and is asserted like any other. #410's lone-residual shape used + to be skipped here too -- 'Dr. Jane' read family 'Jane' and 'Dr. + Jane née Smith' given 'Jane' -- and no longer moves, so the + assertion now covers every name that reaches the marker with + something to stand behind. """ base = parse(name) - if base.given == "": - pytest.skip("#410: a lone residual piece moves between fields") + if not (base.given or base.middle or base.family + or base.title or base.suffix): + pytest.skip("nothing before the marker at all: M2 leaves it a word") with_clause = parse(name + " née Jones") assert with_clause.maiden == "Jones" for field in ("title", "given", "middle", "family", "suffix", diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 6f44d141..b365291f 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -1,6 +1,8 @@ +"'Smitty' Dr. Jones" "'Smitty' John Jones" "'Smitty' Jones" "'Smitty' Jones Jr." +"'Smitty' Sir John" "(" ".," "Abu Bakar" @@ -25,6 +27,7 @@ "Del Toro" "Dr. John van der Berg" "Dr. Juan Q. Xavier de la Vega III" +"Dr. Smith née Jones" "Dr. Smith, John" "Dr. abdul salam" "Esq. Smith" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index a9bcca8c..194b8a11 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -984,10 +984,22 @@ fields = ["given", "family"] issue = "fix(#424) a title-led chain before the numeral is the one name piece" # 'Freiherr von Richthofen V': rules.md#P2 boundary -- the chain stops # before the numeral, and the chain is then the one name piece, which -# reads as 'Dr. Smith V' reads. last 'von Richthofen V' -> given 'von -# Richthofen', suffix 'V'. A rules.md example. +# reads as 'Dr. Smith V' reads. v1 read last 'von Richthofen V'; #424 +# split the numeral off, and #410 then moved the one name piece behind +# the title out of `given`, so today it is family 'von Richthofen', +# suffix 'V' -- 'Dr. Smith V' moved the same way, so the comparison +# still holds. A rules.md example. +# +# The diff was {given, family, suffix} before #410 and is {family, +# suffix} after it. `fields` declared all three until this round, and +# that width is exactly why the movement never surfaced: the subset +# test kept matching as the diff shrank, so a real change of reading +# passed through a green run unnamed. The declaration is narrowed to +# what the diff actually is, which is what makes a future `given` +# regression on this name loud instead of absorbed. A `fields` list +# wider than the diff it explains is a silent absorber. name_regex = "(?i)^freiherr\\s+von\\s+richthofen\\s+v$" -fields = ["given", "family", "suffix"] +fields = ["family", "suffix"] [[change]] issue = "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece" @@ -1220,7 +1232,8 @@ why = "feat(#273) recognizes TYPOGRAPHIC nickname delimiters; the ASCII pairs we # ('"Rick" Edmonds'), medial ('John (Jack) Kennedy') or trailing # ('Franklin, Benjamin (Ben)'), with a word character or a period as # the inner flank so a middle initial counts ('Cherice J. (Johnson) -# Williams'). 34 corpus names. +# Williams'). 46 corpus names (recounted 2026-08-25; the figure is +# the one test_ledger_guards.py pins as this entry's `captures`). # # It was medial-only for three rounds, on the theory that the trailing # position is where credentials live and that widening would silence @@ -1248,10 +1261,11 @@ why = "feat(#273) recognizes TYPOGRAPHIC nickname delimiters; the ASCII pairs we # '김민준씨 (Jimmy)' {first,last,suffix}. Zero classifications lost. # # A bare [("'] class is still wrong, for the reason the role -# narrowing cannot fix: it reaches 47 names, 11 of them on an -# apostrophe that is part of the surname ("Brian O'connor", +# narrowing cannot fix: it reaches 63 names, 9 of them on an +# apostrophe that is part of the word ("Brian O'connor", # "Harietta Keopuolani Nahi'ena'ena"), which is not a delimited run at -# all. +# all. (Both figures recounted 2026-08-25, against the corpora as they +# now stand; they were 47 and 11 when first written.) # # By role: ASCII parens mark nicknames, maiden names, suffixes and # credentials alike, and no regex tells them apart. The shape reaches @@ -1303,3 +1317,59 @@ issue = "fix(#432) a dotted numeral behind a name is a middle initial, not the g # would be dormant, which the gate reports as EXPLAINED NOTHING. name_regex = "(?i)^smith,\\s*john\\s+v\\.$" fields = ["middle", "suffix"] + +[[change]] +issue = "fix(#410) a title and one name word name the family, whatever annotation stands beside it" +# rules.md#H1 reads a title plus exactly one name word as the family +# ('Mr. Johnson' -> family 'Johnson'). The guard implementing it also +# declined whenever ANY token carried a suffix, nickname or maiden +# role, so the very same name with an annotation beside it kept the +# word in `given` and reported no family at all. None of the three is +# a name word, so the term went and these names move `given` -> +# `family`, reading as their un-annotated twins already did. +# +# Keyed on the names by literal rather than by shape. The shape is "a +# title, one name word, and an annotation" -- a parse tree, not a +# string -- and a fields-only rule standing in for it would classify +# every future given/family swap in this corpus. +# +# Its own rule because these have nothing to do with suffix routing: +# they were falling to the fields-only fix(suffix-routing) below, on a +# rule whose every other name moves a trailing token into `suffix`, +# while these move no suffix at all. Same surgery #372 did on the same +# catch-all, and its heading returns to 14 once this rule takes them. +# +# THREE names here, where the 2.0.0 and 2.1.0 ledgers classify four: +# at this baseline 'Dr. Smith née Jones' also moves `middle` and +# `maiden`, because 1.4.0 has no maiden support and read the marker as +# a middle name. It gets the separate rule below rather than widening +# `fields` here, which would let these three move a maiden name +# unnoticed. +# +# Four classified across the two rules, out of FIVE names that change +# reading. The fifth is 'Freiherr von Richthofen V' -- a title, one +# name word and a suffix, so this same shape -- and the fix(#424) rule +# further down has always claimed it. #410 only narrows its diff, from +# {given, family, suffix} to {family, suffix}, and that rule declares +# all three, so the gate stayed green and never named it. It is not +# moved here; the classification was right and the count was not. +name_regex = "(?i)^(?:Senator \"Rick\" Edmonds|Xyz\\. \\(Bud\\) Smith|'Smitty' Dr\\. Jones)$" +fields = ["given", "family"] + +[[change]] +issue = "fix(#410) the maiden flavor, where 1.4.0 read the marker as a middle name" +# 'Dr. Smith née Jones': the same H1 widening as the rule above, but +# against a baseline with no maiden support, so the diff is four +# fields rather than two. 1.4.0 had nowhere to put a birth surname -- +# no default marker vocabulary routes to `maiden` -- so it read first +# 'Smith', middle 'née', last 'Jones'. 2.x extracts the clause +# (`middle` and `maiden`), and #410 is what moves 'Smith' from `given` +# to `family` on top of that. +# +# Separate from the rule above because `fields` is the union of two +# different changes here, and a single rule carrying all four would +# let the three names up there move a `middle` or a `maiden` without +# anything noticing. This is the one name in the corpus where the +# marker extraction and the H1 fix land together. +name_regex = "(?i)^Dr\\. Smith n[ée]e Jones$" +fields = ["given", "middle", "family", "maiden"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 1ecddf86..3931348a 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -427,10 +427,24 @@ fields = ["given", "family", "_ambiguities"] issue = "fix(#424) a title-led chain before the numeral is the one name piece" # 'Freiherr von Richthofen V': rules.md#P2 boundary -- the chain stops # before the numeral, and the chain is then the one name piece, which -# reads as 'Dr. Smith V' reads. family 'von Richthofen V' -> given -# 'von Richthofen', suffix 'V'. A rules.md example. +# reads as 'Dr. Smith V' reads. The baseline read family 'von +# Richthofen V'; #424 split the numeral off as the suffix, and #410 +# then moved the one name piece behind the title out of `given`, so +# the reading today is family 'von Richthofen', suffix 'V' -- and +# 'Dr. Smith V' moved with it, so the comparison still holds. A +# rules.md example. +# +# #410 narrowed this diff rather than adding one: {given, family, +# suffix} before it, {family, suffix} after. `fields` went on +# declaring the wider set, and because the match is a subset test the +# rule kept claiming the name as its diff shrank -- a real change of +# reading crossing a green gate with nothing to name it. The +# declaration now matches the diff, so `given` moving on this name +# again would be unexplained rather than absorbed. The general form: +# a `fields` list wider than the diff it explains silently absorbs +# whatever else lands inside it. name_regex = "(?i)^freiherr\\s+von\\s+richthofen\\s+v$" -fields = ["given", "family", "suffix", "_ambiguities"] +fields = ["family", "suffix", "_ambiguities"] [[change]] issue = "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece" @@ -892,3 +906,46 @@ issue = "fix(#432) a dotted numeral behind a name is a middle initial, not the g # `given` and `family` are out of the field list; neither moves. name_regex = "(?i)^smith,\\s*john\\s+v\\.$" fields = ["middle", "suffix"] + +[[change]] +issue = "fix(#410) a title and one name word name the family, whatever annotation stands beside it" +# rules.md#H1 reads a title plus exactly one name word as the family +# ('Mr. Johnson' -> family 'Johnson'). The guard implementing it also +# declined whenever ANY token carried a suffix, nickname or maiden +# role, so the very same name with an annotation beside it kept the +# word in `given` and reported no family at all. None of the three is +# a name word, so the term went, and the four names below move +# `given` -> `family` to read as their un-annotated twins already did. +# +# Keyed on the four names by literal rather than by shape. The shape +# is "a title, one name word, and an annotation" -- a parse tree, not +# a string -- and a fields-only rule standing in for it would classify +# every future given/family swap in this corpus. +# +# `fields` is exactly the pair that moves: no annotation changes hands +# here, which is the whole point of the fix, so a diff in `nickname`, +# `maiden` or `suffix` on these names stays loud. +# +# Four names are classified here, out of five that move. The one left +# out is 'Freiherr von Richthofen V', which the fix(#424) rule below +# has claimed all along; #410 shrinks its diff (the name piece stops +# moving through `given`) without shrinking it out of that rule's +# fields, which is why no run ever mentioned it. Left where it is +# deliberately -- the classification is right, only the counting was +# wrong. +# +# These four are therefore the names of this shape that nothing else +# explains, not every name of this shape: two with a nickname, one with +# a maiden clause, one where the nickname leads. That fifth name is +# also the suffix flavor, so the corpus does carry one; it is the bare +# 'Dr. Smith PhD' spelling that appears nowhere, and +# tests/v2/cases.py::suffix_stays_suffix_title pins that. +# +# The change is in the 2.2 cycle, so 2.0.0 sees the same four diffs +# 2.1.0 does. The RULE is shared verbatim with that ledger -- +# identical name_regex and fields, identical _CORPUS_CLAIMS digest -- +# and was checked against this baseline's own run rather than assumed +# from it. These comments are not shared: each ledger says what its +# own baseline sees. +name_regex = "(?i)^(?:Senator \"Rick\" Edmonds|Xyz\\. \\(Bud\\) Smith|Dr\\. Smith n[ée]e Jones|'Smitty' Dr\\. Jones)$" +fields = ["given", "family"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 6413ce27..af8dbb11 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -121,10 +121,24 @@ fields = ["given", "family", "_ambiguities"] issue = "fix(#424) a title-led chain before the numeral is the one name piece" # 'Freiherr von Richthofen V': rules.md#P2 boundary -- the chain stops # before the numeral, and the chain is then the one name piece, which -# reads as 'Dr. Smith V' reads. family 'von Richthofen V' -> given -# 'von Richthofen', suffix 'V'. A rules.md example. +# reads as 'Dr. Smith V' reads. The baseline read family 'von +# Richthofen V'; #424 split the numeral off as the suffix, and #410 +# then moved the one name piece behind the title out of `given`, so +# the reading today is family 'von Richthofen', suffix 'V' -- and +# 'Dr. Smith V' moved with it, so the comparison still holds. A +# rules.md example. +# +# #410 narrowed this diff rather than adding one: {given, family, +# suffix} before it, {family, suffix} after. `fields` went on +# declaring the wider set, and because the match is a subset test the +# rule kept claiming the name as its diff shrank -- a real change of +# reading crossing a green gate with nothing to name it. The +# declaration now matches the diff, so `given` moving on this name +# again would be unexplained rather than absorbed. The general form: +# a `fields` list wider than the diff it explains silently absorbs +# whatever else lands inside it. name_regex = "(?i)^freiherr\\s+von\\s+richthofen\\s+v$" -fields = ["given", "family", "suffix", "_ambiguities"] +fields = ["family", "suffix", "_ambiguities"] [[change]] issue = "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece" @@ -829,3 +843,39 @@ issue = "fix(#432) a dotted numeral behind a name is a middle initial, not the g # `given` and `family` are out of the field list; neither moves. name_regex = "(?i)^smith,\\s*john\\s+v\\.$" fields = ["middle", "suffix"] + +[[change]] +issue = "fix(#410) a title and one name word name the family, whatever annotation stands beside it" +# rules.md#H1 reads a title plus exactly one name word as the family +# ('Mr. Johnson' -> family 'Johnson'). The guard implementing it also +# declined whenever ANY token carried a suffix, nickname or maiden +# role, so the very same name with an annotation beside it kept the +# word in `given` and reported no family at all. None of the three is +# a name word, so the term went, and the four names below move +# `given` -> `family` to read as their un-annotated twins already did. +# +# Keyed on the four names by literal rather than by shape. The shape +# is "a title, one name word, and an annotation" -- a parse tree, not +# a string -- and a fields-only rule standing in for it would classify +# every future given/family swap in this corpus. +# +# `fields` is exactly the pair that moves: no annotation changes hands +# here, which is the whole point of the fix, so a diff in `nickname`, +# `maiden` or `suffix` on these names stays loud. +# +# FIVE corpus names change reading under #410. These four are the ones +# that arrive as new diffs and so need classifying here. The fifth, +# 'Freiherr von Richthofen V', was already classified by the fix(#424) +# rule below and stays there: #410 narrows its diff from three fields +# to two, and that rule's declared fields are a superset either way, so +# nothing ever reported it. A real movement behind a green gate. +# +# So the claim is not "the only names of this shape" -- it is the only +# ones not already spoken for. Of the four, two carry a nickname, one a +# maiden clause, one a nickname leading the name. The suffix flavor is +# NOT absent from the corpus: that fifth name is a title, one name word +# and a suffix. What no corpus carries +# is the plain spelling 'Dr. Smith PhD', pinned by +# tests/v2/cases.py::suffix_stays_suffix_title instead. +name_regex = "(?i)^(?:Senator \"Rick\" Edmonds|Xyz\\. \\(Bud\\) Smith|Dr\\. Smith n[ée]e Jones|'Smitty' Dr\\. Jones)$" +fields = ["given", "family"]