feat(vocab): multi-word maiden markers, and the Polish z domu (#434) - #448
Merged
Conversation
maiden_marker_run answers "does a marker start at these words, and where does it end", longest first. Markers have four match sites and _title_key's docstring already names what a hand-written mirror between two of them costs -- "a divergence between them fails silently: the entry simply stops matching" -- so the answer gets one home before any site starts asking a phrase-shaped question. _vocab, not _pieces: mechanisms.md#ONE-PREDICATE-PER-QUESTION puts the destination on the LAYER rather than the topic, and this takes a sequence of token TEXTS plus a vocabulary set, which is text-level. The import direction allows it -- test_layering's _PIPELINE_STAGE_ALLOWED lets every stage import nameparser._pipeline.*, _group already imports _vocab, and _vocab reaches _lexicon for the shared fold. Nothing calls it yet and no phrase is stored yet, so this commit changes no reading. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A maiden_markers entry may now be a PHRASE. Storage follows given_name_titles (space-joined, folded per word through _title_key), but the MATCHING cannot: a title run is identified per word first -- 'lt' and 'col' are each title vocabulary -- and only then joined and looked up, where 'z' and 'domu' are markers only together. So markers get a genuine multi-token lookahead, and every site asks the one predicate for it. The dead-entry warning loses maiden_markers in the same commit that gives the field something to match, so no revision ships an exemption without the mechanism, or a warning that is wrong. Its advice was the bug: split into separate entries, 'Maria Kowalska z domu Nowak' reads maiden 'domu Nowak' and 'Anna z Nowak' loses its family name outright. The four sites: * classify tags the run in a sequence pass -- head "vocab:maiden-marker" as before, the rest "vocab:maiden-marker-cont" -- because _tags_for sees one token with no neighbours and a phrase's words are not markers alone. Single-word markers moved into the same pass, so one place decides the field. * group's piece test is unchanged (it reads the head tag); the M2 take walks the continuations and drops N pieces, and the walk for the maiden name starts past the whole marker. * group's extracted-clause drop (#329) removes the whole run and re-derives its containment argument for N: the token AFTER the run proves the run and that token are all inside the clause. * extract calls the predicate over the clause's whitespace words and still requires a word after the RUN. Both group sites read the tags classify recorded rather than re-deriving the run -- mechanisms.md#ONE-PREDICATE-PER-QUESTION's "record the answer on the state instead", available because group runs after classify. extract runs before tokenize and so calls the predicate itself. The divergence between _maiden_marked and the tokenizer is preserved, and deliberately: they share the predicate but hand it different word sequences, so 'née,' is still marker-led to one and not the other. Confirmed by mutation -- making _maiden_marked strip commas leaves cases.py's marker_glued_to_punctuation_keeps_the_clause_a_nickname the sole failure. test_every_marker_site_ends_the_run_in_the_same_place is the contract, modelled on the P5/H1 title-run test: it collects each site's answer to "how many words is the marker" and asserts they cannot disagree. Mutation-checked at all four -- capping classify's lookahead, stopping group's piece walk at one, dropping one clause token, and restoring extract's words[0] test each fail 5 of its 9 spellings. No shipped vocabulary changes here: every phrase reading above is reachable only through a caller-configured entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Last on purpose: the mechanism and the warning carve-out landed in the previous commit, so no revision in this branch stores a phrase nothing can match. "z domu" is the first multi-word entry any shipped vocabulary set holds. The docstring listed it under Deliberately absent, "pending the 2.0 pipeline's multi-token matching decision" -- resolved, so it moves into the attested list with the reason it is safe as a phrase where its words are not: z is a Polish preposition and domu a noun, neither a marker alone, and the lookahead only ever claims the pair. "Anna z Nowak" is unchanged, family Nowak. The docstring's whole-token claims all survive: a phrase claims a run of whole tokens, one per word, and no part of one. The 旧姓 and roz. paragraphs are about single-word entries and read as before. Measured on this commit: "Maria Kowalska z domu Nowak" and the bracketed "Maria Kowalska (z domu Nowak)" both read family Kowalska, maiden Nowak; "Maria Kowalska (z domu)" stays a nickname, having no word past the marker. All three differential gates (2.1.0, 2.0.0, 1.4.0) report no movement -- no corpus name contains domu or a bare z. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The predicate is called once per token now, where classify used to do one frozenset membership test, and the fold-plus-join it does costs more than that test did. Measured over 8000 parses of a four-name mix (plain, titled-with-initials, phrase-marked, comma-and-particle), best of five: 0.579s on merged master, 0.619s with the run pass, 0.592s here -- 6.7% down to 2.2%, in line with the 1.2-2.2% #429 recorded for the same kind of second evaluation. The fast path lives INSIDE the predicate rather than at its callers, so it is an implementation detail and not a second answer to the question. It cannot hide a match: every key the loop builds opens with _normalize(words[0]), and a first word that folds away matches nothing at any length -- pinned both ways in test_vocab. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five rows, and each one is a FORK rather than a vocabulary member (mechanisms.md#VOCABULARY-EXERCISES-FORKS): single-token match against multi-token match, and the boundaries on either side of it. Every expected value measured on the 1.4.0 wheel first, under --isolated --no-project so the checkout cannot shadow it. phrase_marker_takes_the_maiden_name fix(#434) phrase_marker_partial_is_not_a_marker parity preposition_alone_is_not_a_marker parity phrase_marker_delimited_clause fix(#434) phrase_marker_delimited_alone_stays_a_nickname parity 1.4.0 read the two fixes as first Maria / middle 'Kowalska z domu' / last Nowak and as nickname 'z domu Nowak' -- the marker inside the name, its ordinary reading of every marker. The three parity rows read identically on both sides. Mutation matrix, whole suite, restores verified by comparison: cap the lookahead at one word -> both fix rows (it is upstream of both sites), nothing else in cases bare take drops N-1 -> phrase_marker_takes_the_maiden_name ALONE clause pass drops one token -> phrase_marker_delimited_clause ALONE word-after not run-relative -> phrase_marker_delimited_alone_... ALONE -- the one-word marker row it sits beside satisfies that mutation a prefix of an entry matches -> phrase_marker_partial_is_not_a_marker and preposition_alone_is_not_a_marker, jointly, and no other row in the suite The last pair are joint catchers, not sole ones: they fail the same mutation and record different damage from it -- one loses the middle/family split, the other loses its family outright, which is what the split-entry workaround did to it. Reported rather than papered over. Shortest-first has no cases row and cannot have one: it needs a vocabulary configuring both a word and a phrase starting with it, and a Case carries a Policy or a Locale, never a Lexicon. It is pinned at parse level instead by test_a_phrase_marker_outranks_the_word_it_starts_with, the sole catcher. The one-name-word shape ('Maria z domu Nowak', family '') gets no row. It is M2 as written -- the same branch 'Smith nee Jones' takes, keyed on the marker's position and not on its length -- so a phrase-spelled twin would demonstrate the entry rather than exercise a fork. rules.md#N3's Accepted line covers it and #445 tracks whether the empty family is right. Gates unmoved: Latin-script case rows do not enter any corpus (corpus_cjk.jsonl is generated from the case table but is CJK-only). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rules.md. The M Background gains Polish and the fact a marker need not
be one word. Three statements were wrong for a phrase and are fixed,
each with its citing code comment in this same commit
(test_doc_citations quotes them verbatim):
M1 "a leading recognized marker word inside a multi-word clause
being dropped; a one-word clause keeps its word" -- a phrase
makes '(z domu)' a multi-word clause whose marker is NOT
dropped. Now keyed on a word standing past the marker, which is
what the code has always tested.
M2 gains a sentence: a marker recognized only whole, its own first
word standing alone being an ordinary name word.
M3 "opens with a recognized marker word" -> "a recognized marker".
Four examples land with them and so enter the rules corpus. Two move:
'Maria Kowalska z domu Nowak' (middle/family/maiden) and 'Maria
Kowalska (z domu Nowak)' (nickname/maiden), at all three baselines,
each with its own ledger rule declaring exactly its own fields -- not
the union, which is #444's post-mortem, and not an alternative inside
the fix(#335) rule, which this is not: M3's reach alone leaves the
bracketed name a nickname, and the phrase entry is what moves it.
'Anna z Nowak' and 'Maria Kowalska (z domu)' are parity and move
nothing. Gates green at 2.1.0, 2.0.0 and 1.4.0.
Two guards had the same blind spot the change itself had, and both are
real: they asked "is any WORD of this name a marker", which no phrase
entry can answer yes to.
tests/v2/test_parser.py's clause-free corpus filter let 'Maria
Kowalska z domu Nowak' through, so the appended clause was not the
only variable and the invariant failed. It asks the predicate now;
the recount is in its comment (642 names, 632 after the delimiter
strip, 629 once the predicate decides).
test_ledger_guards._carries said the same name carries no maiden
vocabulary, which would have refused the ledger rule above. It
matches consecutive token RUNS up to the longest entry now, which
is the single tokens themselves for any vocabulary of words, so
every count in that module is unchanged.
Re-recorded, each because the corpus grew and not because a rule did:
fix(suffix-routing)'s whole-corpus claim (1080 -> 1084), the
parenthesized-clause exclusion (51 -> 53 captures, absorbed_by still
empty), and the two new #434 claims.
decisions.md, under M2 and unwrapped: why a run predicate was rejected
(z and domu are markers in no vocabulary, so no run can be identified
before the lookup, where 'lt' and 'col' each are); that
given_name_titles is the storage precedent and not the matching one,
and that no shipped set held a multi-word entry before this; C-i
answered at the phrase level, with the roz removal as the contrast --
a phrase changes the unit that CLAIMS rather than passing the test
with the old one, and roz has no longer unit to move onto; the four
sites, the contract test, and the preserved whitespace/token
divergence with the row that pins it; the commit ordering, including
that filterwarnings = ["error"] is what makes the warning carve-out
load-bearing rather than tidy; and the measured 6.7% -> 2.2% cost with
its method.
Swept for the rest of the single-word claim rather than stopping at
the two files already fixed: docs/customize.rst (the caller-facing
statement, now naming both exceptions and how they differ),
rules.md's S Background, decisions.md's Excluded (multi-word) header,
_policy.maiden_delimiters' field doc, tools/differential/README.md's
vocabulary-coverage count and its recompute one-liner (4 of 16 -> 5 of
17, counting runs), and docs/design/AGENTS.md's #291 anecdote, whose
premise was true when it was written.
Release log: what works, that the previously advised workaround was
wrong and what it produced, and the migration note to remove z and
domu if they were added separately. Every value in it measured on this
commit, on the 1.4.0 wheel and on the 2.1.0 wheel; the roz bullet's
"now 16 entries" became "this removal takes the default set to 16" so
the two bullets do not contradict inside one release.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A phrase written across a bracketed clause was tagged as one run and
consumed as a prefix of itself. 'Anna z (domu) Nowak' read family
'Anna', maiden 'Nowak' and 'Maria z (domu) Kowalska Nowak' lost its
family name outright -- a bare Polish preposition eating the name,
which is the exact damage the phrase entry exists to prevent.
Reachable on shipped vocabulary under the default policy.
The gap is between two token populations. _tag_marker_runs walks the
whole span-sorted stream, delimited-clause tokens included;
_marker_run_pieces walks one SEGMENT, and _segment.py:31 is
`main = [i for i, t in enumerate(state.tokens) if t.role is None]`,
which drops every token extract has already given a role, then buckets
what is left by the commas before it. So the piece walk could see only
the run's first word and handed M2 that word as the whole marker. The
docstring asserting each continuation "is always in `seen`" named only
delimiter cores as a reason `seen` skips a token; it now names this
one, which is what made the gap invisible.
The fix is a rule about the TAG, not a guard on a consumer: classify
refuses to tag a run whose tokens are not structurally contiguous --
same role, so a clause edge ends a run, and the same comma bucket,
computed exactly as _segment.py computes it. Refusing, not truncating:
truncating hands M2 the same wrong prefix one word shorter. The
lookahead is bounded at the boundary BEFORE the predicate is asked, so
a two-word entry refused at a clause edge leaves a one-word entry
starting there free to match.
The defensive complement -- having _marker_run_pieces re-derive the run
by token index and refuse a mismatch -- is declined. With the tagging
rule in place it can never fire, and a guard that cannot fire is its
own problem. Its argument lives in that function's docstring instead.
Should the contract test have caught this? Yes, and the hole is in how
the contract was FORMULATED: it varied the marker's SPELLING across
nine forms and held its PLACEMENT fixed, so every run it built was
contiguous by construction. The placement axis is now its own test over
eight placements, asserting two things no spelling can reach -- every
tagged run has one role and one comma bucket, and group drops a run
whole or declines it -- with a vacuity pin so the straddling rows
cannot quietly stop straddling. The six earlier mutations still fail
the original test, so nothing was traded away.
Two case rows, both parity, both measured on the 1.4.0 wheel:
phrase_marker_split_by_a_clause_is_not_a_marker and
..._keeps_the_family. Mutation matrix, whole suite:
contiguity bound removed -> both rows, all three straddling
placements, and M2's new example
ROLE half removed -> the two clause placements and both rows
COMMA half removed -> exactly one placement and NO case row
That last line is the argument for the tag-level test standing on its
own: a comma-split run is declined downstream today for an unrelated
reason, so nothing about fields can witness half of this rule.
rules.md#M2 gains the qualifier it lacked -- a phrase is recognized
only whole AND only where its words stand together -- with a boundary
example, and the M Background, customize.rst, the maiden_markers
docstring, the Lexicon field doc and the release-log bullet all say so
now. Those three of them were falsified by the defect; re-verified by
measurement, and the sweep covers every place the branch states it.
Perf, re-measured because the first figures were not interleaved and
their spread was the size of the effect (master alone measured 0.579s,
0.593s and 0.607s within an hour). Medians of five interleaved rounds:
master 0.599s, run pass alone +6.8%, fast path +1.4%, shipped
mechanism +1.6% -- matching the reviewer's independent +7.2%/+1.6%.
The contiguity walk cost +3.5% until classify was made to ask
maiden_marker_head first; that is the predicate's own fast-path test,
exported from _vocab and called BY the predicate so the two cannot
drift, and documented as a deliberate superset.
Also: assert_normalized now checks the single-spaced form rather than
just strip().lower(), so 'z domu' fails import-time hygiene, and its
message no longer claims entries are whitespace-free. The per-word
period half of the storage rule needs _lexicon._title_key, which this
module may not import; that gap is recorded in its docstring rather
than closed. The duplicate _normalize per token is noted at the fast
path with the measurement that says it is not worth threading away.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The condition restated where the fix did not reach. rules.md#M1's body
and tools/differential/README.md both still said a clause of "two
words or more" led by a marker reads as maiden -- which M3's own
boundary example eleven lines below contradicts, 'Maria Kowalska (z
domu)' being a two-word marker-led clause that stays a nickname. Both
now say "a clause holding a word past its marker", which is what M1's
and M3's repaired statements already say and what the code has always
tested. M1's "a lone marker word" became "a lone marker ... whether
that marker is one word or several", which its own new example needs.
Statements that overreached:
* M2 said a phrase's first word standing alone "is an ordinary name
word and takes nothing". With geb shipped and geb von configured it
is a marker and does take -- the setup the release note and
test_a_phrase_marker_outranks_the_word_it_starts_with both document.
Now: it is not THAT marker, and whether it takes anything is a
separate question with a separate answer.
* The M Background kept "a marker word" twice, two sentences after
announcing markers need not be one word, and its attested list
omitted the unaccented nee -- which is live and output-visible, so a
reader predicted maiden 'Nee Jones' where a configured pair gives
'Jones'. It also gave "both ё and е spellings" to урожд., which has
neither; the four full participles carry that.
Counts, every one recomputed rather than adjusted, since the corpus
moved twice in this branch:
README's _carries reading 5 -> 6 (it adds né, a substring
of née and never a token)
README's shipped entries 16 -> 17
1.4 ledger's fix(#274) block 3 of 16 -> 3 of 17; thirteen
unreached -> fourteen; and the
arithmetic's carve-out is now TWO
names, not one -- z domu appears in
the corpora and cannot ever be an
alternative here, its words being an
ordinary preposition and an ordinary
noun
test_ledger_guards' roster 16 -> 17 entries, 4 -> 5 corpus
markers, with the note that counting
z domu at all needs the RUN reading
release log's #335 bullet 1,080 -> 1,085 corpus names (the
Seven and the six are unchanged --
re-derived from the gate output)
release log's marker count restated as a delta, the shape the
roz bullet beside it already uses
The differential README's recompute recipe did not run as pasted: a
double quote inside the strip set closes `python -c "`. It is a
heredoc now, prints the _carries reading beside the token one, and was
confirmed by pasting it under zsh -- both printed figures reproduce.
Records that would have misled later. decisions.md#M1's 2026-08-04
entry still described "dropped from a multi-word clause" and a
"clause-size guard ... load-bearing and mutation-proven"; a phrase
falsifies the first and there is no such guard any more. Corrected in
place under M1 rather than only under M2, because a future PR reading
M1's own history pointer would restore something that reintroduces the
phrase defect and passes every doc test. The same entry records the
decision M1's rewrite forces and nothing had weighed: a configured
pair emits maiden 'z domu', two words the phrase argument calls
nobody's name, and it is emitted deliberately -- the alternatives ask
the parser to judge its own vocabulary unnameable, which nothing else
does.
docs/design/AGENTS.md's "then-true" is dropped. given_name_titles has
folded per word since 2026-07-19 (d13bf5c), a month before the #291
arc, so an exception already existed -- and "then-true" retrofits
exactly the excuse that axis exists to forbid.
Smaller: mechanisms.md's _vocab enumeration now names
maiden_marker_run, which decisions.md cites as the module's clearest
two-stage case; the FOUR-sites sentence enumerates four instead of
three, group being two of them; rules.md's narrowed S Background no
longer attaches "the eight" to a subject that excludes chargé
d'affaires; customize.rst says the warning does not fire for the two
exempt fields; two 120-column lines in maiden_markers.py are
rewrapped.
And one measurement correction of my own. The mutation figure on
phrase_marker_delimited_alone_stays_a_nickname claimed maiden 'domu'.
Measured: the extract-only mutation gives maiden 'z domu', and so does
that mutation plus the clause drop reverted -- the value never moves,
because the #329 drop declines for its own reason. The row catches its
mutation on the FIELD alone, and the note says so now.
One row added, and the reviewers' premise for it corrected. They
recommended pinning the clause-drop containment branch on the ground
that only a rules-doc example holds it. Measured: deleting the guard
outright fails maiden_marker_delimited_two_clauses, a group unit test
and three M1 doc examples. What has a single pin is the RUN-RELATIVE
half -- reading the containment one token in instead of past the whole
marker deletes this clause's text and fails exactly M1-4. So
phrase_marker_delimited_alone_keeps_its_words is the phrase half of
the branch and not the branch, measured parity against 1.4.0 through
the bucket-move idiom, and it now fails that mutation in both runners.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #448 +/- ##
==========================================
+ Coverage 98.61% 98.64% +0.03%
==========================================
Files 45 45
Lines 3095 3171 +76
==========================================
+ Hits 3052 3128 +76
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Eleven findings applied, two of them overturning trades I had accepted
on reasoning the module's own precedents contradict. No behavior
changes: all eleven mutations from the earlier rounds still fail, with
the same catchers.
Ask once, pass the answer along -- the same move at four scales:
* maiden_marker_head takes the FOLD, not the text. The trade I had
accepted ("an extra parameter on the shared predicate") does not
apply: this module already ships _is_suffix_strict_n beside
is_suffix_strict, and suffix_as_written's docstring asks for exactly
this ("n is _normalize(text), passed in so callers normalize once").
maiden_marker_run's signature is untouched.
* classify folds once per token and shares it between the marker pass
and the vocabulary tags. Instrumented: an ordinary token is folded
ONCE now, against twice before this round and three times for a
marker head -- fewer than master, which folded once and then did a
per-token maiden_markers membership test the marker pass replaced.
* maiden_marker_run folds each word once and keys from a prefix.
_title_key(words[:n]) per candidate length re-folds the whole prefix,
which is quadratic in cap and made a ONE-word hit cost more than a
two-word one -- the longest key always built and discarded first,
with sixteen of seventeen shipped entries a single word. The exact
form, not the faster variant that assumes no folded word contains a
space. It is a copy of a fold defined in _lexicon, so test_vocab now
pins that the two agree.
* _tag_marker_runs returns index -> tag and classify writes it in the
one loop that builds tokens, instead of replacing every marker token
a second time with dataclasses.replace. The DECISION stays entirely
in _tag_marker_runs; only the writing moved.
And the comma-bucket sweep is deferred until a marker head is actually
found -- it was built unconditionally and read only under cap > 1, so a
single-word vocabulary paid a bisect per token and discarded it, while
the comment above claimed the walk was skipped. The tri-state is gone
with it.
comma_bucket moved to _state.py, beside COMMA_CHARS and for its stated
reason ("Shared here so tokenize and extract cannot drift apart").
segment builds the segments with it and classify asks it whether two
tokens could be in one -- their being the SAME EXPRESSION is the
invariant the contiguity fix rests on, and it was held by prose in two
files plus a `_segment.py:31` line reference that would rot. The
reference is gone.
Also: one continuation walk (marker_run_length) for the two sites 740
lines apart in _group and the two open-coded copies in its tests;
_maiden_take splits where `run` is known and returns the two lists
MaidenTake already declares; _maiden_marked refuses a one-word clause
before any fold, the free short-circuit master had; the caches cite
_script_segment._longest_entry, whose shape and key space they share
and whose maxsize=16 reasoning carries over, instead of
_extract._delimiter_chars, which is consulted once per parse and so
says nothing about a per-token lookup; and classify's stage header
declares comma_offsets, which it now reads.
Two false claims, both measured false before fixing:
* "a run never crosses a STRUCTURAL boundary" is stronger than the
code, and _marker_run_pieces rested its walk on the strong form.
role is one-directional: a role change IS a clause edge, but two
ADJACENT clauses of the same role are indistinguishable, so
'Jane (z) (domu) Jones' tags a run across two nickname clauses.
Nothing reads it -- the piece walk never sees role-bearing tokens,
the clause drop is scoped to one span -- and the parse is identical
with and without the phrase mechanism. Both docstrings state the
true form and name the limit.
* _invariants said the per-word period fold "needs the fold this
module may not import". False: _lexicon's config imports are all
inside _default_lexicon(). The import is still not added -- the
right altitude is a test -- and the gap is closed where it belongs:
test_a_shipped_phrase_entry_is_stored_as_written, parametrized over
_PHRASE_FIELDS, asserts Lexicon stores each shipped constant
UNCHANGED. Nothing pinned that for any field. Verified by mutation:
shipping 'z. domu' fails it by name.
Deviation from the brief, and its reason: the raw-text
maiden_marker_head wrapper is not kept, because it would have had zero
callers. suffix_as_written is the precedent for a fold-taking
predicate with no raw-text sibling in this module, and an unused
wrapper is the kind of thing a simplify round should not add.
Perf, re-measured with the same interleaved-median method (five
rounds, every variant timed once per round). Against master: run pass
alone +6.8%, fast path +1.4%, contiguity rule +1.6%, and after this
round -0.3% and -1.3% on two separate five-round runs -- no cost this
benchmark can resolve. Not a free lunch: master did a per-token
maiden_markers membership test that the marker pass replaced, so what
was added and what was removed are within noise of each other.
decisions.md carries the figures, the method, and the structural-unit-id
follow-up recorded rather than taken.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #434.
parse("Maria Kowalska z domu Nowak")reports familyKowalska, maidenNowak.maiden_markersaccepts multi-word entries, and the Polishz domuships — the first multi-word entry any shipped vocabulary set holds.Not the run predicate the issue offers
#434 frames the choice as "either a marker run predicate, or markers stay single-word". The run predicate is what the library's own warning used to recommend — "Split it into separate entries" — and splitting is the defect:
zis an ordinary Polish preposition anddomuan ordinary noun; neither may claim anything alone. That is what a phrase entry buys, and it is the same C-i questionrozwas removed for two PRs ago — answered at the phrase level rather than the word level.given_name_titlesis the storage precedent (space-joined, per-word normalized,_title_key) but not the matching one: its run is identified per word first, becauseltandcolare each title vocabulary. Nothing can be identified first here, so markers need real multi-token lookahead.One predicate, four sites, one contract
_vocab.maiden_marker_run(words, markers) -> int, longest-first with a head-word fast path. Four sites ask it or read what it recorded: classify's sequence pass, group's piece walk, group's clause drop, extract's clause test._title_key's docstring warns that a divergence between match sites "fails silently — the entry simply stops matching", and titles have three sites where markers have four, sotest_every_marker_site_ends_the_run_in_the_same_placeasserts all five answers agree across nine spellings and eight placements.A divergence is deliberately preserved:
_maiden_markedasks a whitespace-split question and the tokenizer a token-level one, which is what keeps_group's role filter reachable (pinned by a row #446 added).A regression found in review, and fixed
Two reviewers independently found that a run could form across a bracket, quote or comma — classify walks the token stream,
_marker_run_pieceswalks a segment, and a segment excludes every tokenextractalready roled. A barezthen claimed the name:Fixed by refusing to tag a run whose tokens are not structurally contiguous — same role and same comma bucket, the expression
_segment.pyitself uses. Refusing rather than truncating: truncating hands the same wrong prefix one word shorter. The contract test should have caught it and did not, because it varied the marker's spelling while holding its placement fixed; it now varies both.Two coverage gaps closed with it, each a mutation that passed 5,977 tests: M2's phrase decline was unpinned (breaking it made
z domuvanish from every field), and shipping the first phrase raised_longest_markerfrom 1 to 2, newly exposing the fold-away guard to all sixteen other markers through the default vocabulary.Verification
Suite 6001;
compare.pyexits 0 at 1.4.0, 2.0.0 and 2.1.0; mypy, ruff and the Sphinx doctest build clean; every commit independently green from a clean archive. Two corpus names move, both examples this change adds.Perf, medians of five interleaved rounds (the first measurement had a spread the size of the effect): the run pass costs +6.8%, reduced to +1.6% by a head-word gate that classify asks first and the predicate itself calls, so the two cannot drift.
Four review rounds. The one real defect is the contiguity regression above; the rest were prose and coverage.
🤖 Generated with Claude Code