Some search improvements - #2227
Open
fastfadingviolets wants to merge 12 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
…rboston#79) - scripts/search-eval: seed/run/label/compare CLI scoring recall@10, MRR, nDCG@10 per category against the app's exact bill-search params - Corpus: 7337 real bills extracted from tests/integration/exportedTestData via the production search converter (regenerate: yarn search-eval:corpus; hash-pinned in committed meta.json, JSONL itself gitignored) - Golden set: 70 queries (exact-bill-id, topic, synonym, misspelling, member-committee) with rot-proof predicate labels + provisional graded ids - Baseline scorecard recorded vs Typesense 0.24.0 (pre-upgrade defaults) - searchTop10 spreads the shared billsSearchParams (minus exclude_fields, which conflicts with the eval's pinned include_fields), so ranking params added to the shared module reach Typesense in the eval; rule-based label resolution stays unweighted — it resolves ground truth, not the ranking under test - useBillSort wired to the shared billsRelevanceSort constant Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dapter - typesense/typesense image 0.24.0 -> 30.2 in infra/Dockerfile.search (local docker-compose) and the repo-checks CI service container; the dormant k8s chart's 0.24.0 pin is marked not-deployed and the two live pins cross-referenced so they stay in sync - typesense JS client ^1.2.2 -> ^3.0.6 in root and functions (+ @babel/runtime peer dep); typesense-instantsearch-adapter ^2.4.0 -> ^3.0.2 - SearchIndexer: use documents(id).delete() (3.x reserves documents().delete() for delete-by-query); narrow import failures via filter(!success) and log the fail response's top-level id, falling back to document.id - functions typescript 4.5.5 -> 5.3.3 (typesense 3.x type defs require TS >= 5.0); node engines pinned to ^22 and CI setup-node 20 -> 22 (adapter 3.x requires node >= 22, and node 20 is past EOL) - transpilePackages the adapter: 3.0.2 ships ESM syntax in .js files with extensionless relative imports under a CJS package.json, so Next's server externalization hit ERR_MODULE_NOT_FOUND and /bills, /testimony and /hearings returned 500; webpack resolves the extensionless imports fine The live dev/prod Typesense servers (AWS ECS, managed in maple-testimony/ infra) must be upgraded to 30.2 before deploying the functions in this change. Eval scorecard recorded: relevance-neutral vs the 0.24 baseline — zero delta on recall@10/MRR/nDCG@10 across all categories (overall 0.651/0.767/0.525 unchanged); only 4/70 queries differ in top-10 and none move a relevant document. Closes the eval gate on codeforboston#77. Also verified locally against 30.2: functions typecheck and unit tests, the skipped search integration suite run unskipped, and the full CI integration test list (51/51). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three pieces of reindex infrastructure (codeforboston#85, codeforboston#114): convertVersion escape hatch for stale index detection (codeforboston#85). SearchIndexer names each collection after a hash of the config, but the hash covers convert.toString() only — the bodies of module-level helpers convert calls are invisible, so rewriting one changes what gets indexed without ever scheduling a reindex. Not hypothetical: ce137bd (Nov 2024) rewrote buildTopicsForSearch and the live bills collection went per-document divergent for 21 months. CollectionConfig gains an optional convertVersion to force a rebuild; when unset the key is omitted from the hashed object, so introducing the field leaves every existing collection name untouched (verified: bills, hearings and publishedTestimony all hash exactly as before). Naming moves out of the constructor into searchCollectionName so the invariant is testable without firebase-admin or a Typesense client; one test pins a literal hash so an accidental key addition fails in CI rather than renaming all three collections in production. The one live blind spot is marked at buildTopicsForSearch, and the doc widened: the blind spot is the whole import graph convert reaches. Chunked backfill (codeforboston#114). performUpgrade did create -> backfill -> upgradeAlias in one invocation with the cursor living only inside a generator: nothing resumed, nothing retried, and on dev bills is ~49,000 documents, so the run either fit in one timeout or never converged. The backfill is now a chain of chunk documents under the upgrade document, each carrying the cursor it resumes from (the scraper.ts idiom). upgradeSearchIndex only starts the run; runSearchBackfillChunk moves as much as its wall-clock budget allows, writes progress back, and either chains the next chunk or swaps the alias. failurePolicy gives retries; a chunk is idempotent (it upserts into a collection nothing is aliased to, from a cursor its own document fixes). Guards: a chunk from a superseded run is dropped (identity checked before the give-up age check), a chunk still failing after 30 minutes fails the run, the chain stops at 500. Chunk triggers are per-alias, emitted by createSearchIndexer with the config captured in the closure — no prefix sniffing, no registry lookup, no load-order requirement. Bookkeeping is absolute, not incremental, so a replayed chunk converges on the same totals (verified by verbatim replay: 30 batches / 7337 documents, not 59 / 14424). Firestore page size (250) is decoupled from the import payload, which is sliced by serialized bytes (6 MB against the AWS API Gateway 10 MB cap) and measured in bytes, not UTF-16 code units. The cursor is the last document's idField value — the value Firestore actually compares. numBatches keeps working as a whole-run budget, so {"check":true,"numBatches":1} still builds a one-page index and swaps the alias. The upgrade timeout takes the full 540s v1 ceiling. Verified end to end on the emulator against a local Typesense 30.2: full run, numBatches lever, resume from cursor, chunk replay, both supersede guards. Status tooling. typesense-admin status prints the server version, each alias with its live collection and document count, and any collection no alias points at — what an abandoned backfill leaves behind. yarn firebase-admin run-script searchUpgradeStatus prints each alias's run: status, target collection, documents and batches so far, and the cursor it is working from; it enumerates /search rather than hardcoding the alias triple, so a run left by a renamed alias still shows up. Verifying a deploy previously meant a REPL session against the cluster. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bills: query_by gains primarySponsor, cosponsors, currentCommittee (already indexed as display names — no schema change) with weights 10,5,4,3,4,1. Member/committee queries were the worst eval category because those fields were unsearchable; number stays far above all so bill-number mentions in other bills' text can't outrank the bill itself, and title stays above committee so committee text doesn't crowd out topic matches (committee at 5 regressed 'mental health'). num_typos and drop_tokens_threshold probes were exact no-ops, so no typo knobs ship. Eval vs the typesense-30 baseline (committed as typesense-30-weights.json): member-committee recall@10 0.093 -> 0.887, nDCG@10 0.119 -> 0.864; overall nDCG@10 0.525 -> 0.687; all other categories unchanged (topic +0.008). Testimony: billId and author name matches now outrank incidental content mentions (10,6,3,1). Hearings: billNumbers/title/chairNames weighted above long agenda/description text (10,8,6,5,2,2,2) — hearings always sort by startsAt, so this governs matching/tie behavior, not ordering. All three pages' params verified accepted against the hosted dev gateway (still Typesense 0.24), so this is safe to ship ahead of the server upgrade. The params are hoisted to searchParams.ts and hardened after review: query_by/query_by_weights derive from one ordered field-to-weight map, so reordering fields can't silently reassign weights, and each export is checked with satisfies against the adapter's param type so misspelled param keys fail to compile. SearchPage keys its Typesense client memo on the params' JSON content instead of object identity — the inline literal in the hearings page was defeating the memo and recreating the client every render. search-eval types its params as the typesense client's own SearchParams, so future ranking knobs flow through without type edits. Cleanup verified behavior-identical: eval metrics and top-10s byte-equal to typesense-30-weights.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The eval harness measured bills only, so extending relevance work to the other collections would have shipped unmeasured (codeforboston#82). Registry (scripts/search-eval/collections.ts): one entry per alias carrying the app's search params, its relevance sort, the fields golden rules resolve against, and the categories that make sense for it. Corpora and golden sets become per-alias (corpus/<alias>/docs.jsonl.gz, goldens/<alias>.json), selected with --alias; everything defaults to bills, so existing invocations are unchanged. loadGoldens throws when a set's defaults.sort_by is not the collection's relevance sort — hearings motivates it: every sort option there was date-ordered, so nDCG would have measured recency. This adds a relevance sort to the hearings UI (and gives "past oldest" an eventId tiebreak instead of the duplicated startsAt:asc,startsAt:asc key), shared as constants between app and harness. Verified behaviour-neutral on bills: zero delta on every category. Corpora from prod, no credentials (yarn search-eval corpus --env --alias): firestore.rules already grants unauthenticated read on events and the publishedTestimony collection group, so the web client SDK pages the live collections and the production converter runs locally — still the property that matters, since a converter change needs source documents. The source collection derives from the config's documentTrigger so it cannot drift from the indexer's, and config.filter stays the authority on membership. Client-SDK timestamps are rebuilt as firebase-admin ones (the converters check InstanceOf against the admin class, and root vs functions/ node_modules hold separate copies). Bills keeps the emulator path: its rule is path-scoped and denies collection-group reads. Committed corpora: 962 hearings and 1,056 published testimony from prod, zero conversion failures; testimony authorUid hashed and fullName replaced by authorDisplayName — neither is in any query_by. Re-exporting bills still reproduces its corpus byte for byte. Golden sets: 48 hearings queries (nDCG@10 0.907) and 39 testimony queries (0.856) with committed baselines; every rule resolved against the corpus before being written, so no query silently skips for want of judgments. Both synonym categories sit at or near zero by design — neither collection sets synonym_sets yet, so those queries are the headroom codeforboston#82 has to close. Testimony bill-id is 0.848 for the same reason bills was before numberVariants. The testimony topic category was rewritten before commit: broad queries ("housing") scored a flat 1.000 that could not detect a regression; narrow ones ("solitary confinement") put recall@10's denominator at the set size, where a missed document costs score. Two golden-authoring traps documented: a short `contains` matches inside words ("gun" matches "begun"), and rule values compare with punctuation and spaces stripped, so "health care" and "healthcare" are one rule. tests/search-eval/README.md records what the new goldens settled about weights and variants. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Synonym set (codeforboston#79 follow-on, codeforboston#90, codeforboston#116, codeforboston#117). A curated legislative set (29 items) applied at query time via the Typesense 30 synonym-sets API, sent as synonym_sets=legislative from all three collections' params. Multi-way items only where both vocabularies are genuinely interchangeable in MA bill titles (firearm/gun, educator/teacher, cannabis/marijuana/ marihuana, liquor/alcoholic beverage); everything else is one-way root=colloquial -> statutory (doctors->physician, seniors->elderly, ...) so precise statutory queries stay unexpanded — multi-way variants of those items regressed 7 golden queries because Typesense scores synonym matches equal to query terms. Multi-way groups include their own head term (codeforboston#90): alcohol, climate and vehicles never did, so "liquor" only reached "alcohol" content via typo-tolerance coincidence. The set lives in functions/src/search/synonyms.ts, built by two constructors that make the data invariants unrepresentable — multiWay always includes its head term, oneWay derives its id from the root and keeps the root query-side (codeforboston#117). checkSearchIndexVersion re-upserts it on every deploy, after upgrades are scheduled so a failure cannot lose a reindex, and throws so the invocation fails visibly (codeforboston#116): the functions already hold the Typesense key through their Secret Manager binding, so CI never touches it. typesense-admin keeps upsert-synonyms (local Typesense, recovery) pointed at the same module, and status prints the server's synonym sets, flagging a missing set by name. SYNONYM_SET_NAME flows from one exported constant into searchParams.ts. The set is server-level state and survives SearchIndexer's hash-named collection swaps. Bill-number variants (codeforboston#84, codeforboston#82). Typesense indexes "H100" as one token and space is a hard separator, so the query "H 100" matched nothing. convert emits a numberVariants field holding the spaced form, in query_by at weight 10 beside number, kept out of hits via exclude_fields (which also cuts response size 9-14%). The logic is shared as functions/src/bills/numberVariants.ts exporting billNumberVariantsVersion, and both configs using it set convertVersion to that constant — the collection-name hash cannot see shared helper bodies, and the failure mode is divergent indexes, not stale ones. Bills and testimony get the field; HEARINGS DELIBERATELY DOES NOT: billNumbers is multi-valued and Typesense pools tokens across array elements, so a hearing listing both some H-bill and some *2391 bill becomes a false positive for "H 2391". Bills and testimony are safe because each document has at most one variant. Hyphen tokenization (codeforboston#91). token_separators: ["-"] on all three schemas: "vote-by-mail" indexed as a single token, so the spaced query and the hyphenated one returned disjoint sets and neither "vote" nor "mail" could reach inside the compound. Verified symmetric on a scratch collection. Hearings gets the setting despite measuring flat so tokenization does not differ per collection. Eval, per change, all against committed baselines: - synonyms on bills: synonym recall@10 0.79->0.96, nDCG 0.826->0.973, overall nDCG@10 0.687->0.709, zero regressions - synonyms on testimony/hearings: hearings synonym 0.000->0.810 (overall 0.907->0.957), testimony synonym 0.673->0.968 (overall 0.856->0.909); query-side only, so the deltas are exact - number variants: bills exact-bill-id 0.800->1.000 (overall 0.709->0.752); testimony bill-id 0.848->1.000 (overall 0.909->0.940), gain confirmed query-side via a control run on the re-exported corpus - hyphen splitting: bills hyphenation 0.574->0.955 (overall 0.738->0.766, after adding six hyphenated goldens the old set never asked about), testimony overall 0.948->0.980, hearings flat - head terms: fixing codeforboston#90 regresses syn-009 ("liquor") because Typesense credits a single-word synonym match with the longest token-count across the resolved group; no clean fix exists without losing real recall, so the group stays and syn-009 is regraded to genuine relevance (bills nDCG@10 0.766->0.754, new baseline). Mechanism documented in tests/search-eval/README.md. Deploy notes: the synonym set requires the ECS Typesense 30 upgrade; numberVariants and token_separators are schema changes, so collections reindex asynchronously and the alias stays on the old collection until the backfill finishes — Typesense 404s the whole request on an unknown query_by field, so confirm the alias swapped before shipping the frontend. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…deforboston#87, codeforboston#76) Bills was the weakest collection (nDCG@10 0.754 against hearings' 0.957 and testimony's 0.980) and the deficit was ordering, not retrieval: every topical term a bill states plainly was diluted in body, 1683 median characters of statutory prose searched at weight 1. Pinslip (codeforboston#87): the clerk's petition blurb — 256 characters saying what the bill is for, present on 96.2% of bills, disjoint from body, indexed nowhere. Weight 3 (2-4 return byte-identical results; at 5+, where pinslip meets title, it costs 0.011 — the constraint is the ceiling). Twenty bills (0.3%) are procedural study orders whose Pinslip is a concatenated docket up to 28,049 characters against a median of 224; indexed whole they matched nearly any topical query (the Judiciary order at rank 6 for "eviction"), so convert drops a pinslip over 2000 characters. The threshold is inline because SearchIndexer hashes convert.toString() and would not see a named constant change. Eval: topic recall@10 0.775->0.800, exact-bill-id and member-committee unchanged. The goldens could not see the change fairly — topic queries grade by title-contains rules, so a thin-titled bill with a topical pinslip counted as a regression where it worked; measured raw against a pinslip-present-but-unqueried control first, then graded only what moved. One per-query regression stands deliberately (topic-016 "vaccines", a rank 1/2 swap; regrading the promoted bill upward would be the self-fulfilling move measuring-before-grading exists to avoid). Summary (codeforboston#76): the LLM plain-language description written by bill_on_document_created, on 87.8% of prod bills. Weight 2, between pinslip and body (1-4 plateau; at 5 topic falls 0.500->0.394). convert withholds summary for Order and Extension Order types — indexing it took mc-014 "Education Committee" from 0.558 to 0.064 by promoting extension orders that then matched a committee query in four fields, and fields_matched sits in the lowest bits of _text_match breaking ties; withholding restores 0.558 exactly. Only those two types: Resolves and constitutional amendments are legislation people search for. Six new plain-language goldens (ground truth identified from title/pinslip/body, never summary): pl category 0.508->0.569, recall +0.100, MRR +0.111 — concentrated in pl-001 "tenants facing eviction", a retrieval win where the control found four documents, two of them clean-energy bills. text_match_type: max_weight was measured and rejected (collapses 24 documents onto one score and testimonyCount decides the list; the same field pair needs opposite priority for committee and topic queries). Two per-query regressions stand, both fields_matched displacement. The corpus grows an enrich subcommand: re-exporting bills from prod wholesale is wrong (court 192 has drained out of its policy committees, which deletes mc-013 outright), so enrich joins only the missing field from the live project — readable without credentials via per-court collection reads, since the generalCourts rule denies collection-group scope — and every other value stays frozen; meta.enriched records each join. enrich and corpus share one remoteSetup preamble, configForAlias lives with the registry, and the per-collection regeneration recipe moves onto EvalCollection — the generic recipe printed on md5 mismatch would have silently rewritten what the member-committee goldens mean. Cleanup verified behavior-identical: reproduces bills-summary.json exactly, and enrich through the refactored path rebuilds the corpus byte for byte. BillHit prefers the summary snippet, falling back to pinslip, guarded on matchLevel — Typesense omits a non-matching field from highlight and the adapter falls back to the raw value, so an unguarded Snippet prints petition boilerplate under every result. Neither field joins exclude_fields (the adapter builds _snippetResult by walking the returned document); highlight_full_fields is pinned to "title" so the full copy of every searched field stops shipping at ~2.4KB per page. Deploying moves the bills collection hash; the reindex is asynchronous and the alias stays on the old collection until the backfill finishes — confirm the swap before shipping the frontend (codeforboston#94). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…odeforboston#95) Split out of codeforboston#76, where indexing the LLM summary promoted a wall of "Extension Order - Education" documents into a committee query and had to be fixed narrowly by withholding the field. This fixes the thing underneath: procedural documents were never what a committee search wanted, and they had already distorted a measurement twice — the 2000-character pinslip cap in codeforboston#87 was the first workaround for the same population. convert emits content.LegislationTypeName as a faceted legislationType, and billsRelevanceSort opens with _eval(legislationType:!=[`Order`,`Extension Order`]):desc. bills nDCG@10 0.739 -> 0.752 recall@10 0.880 -> 0.890, MRR 0.932 -> 0.949 member-committee 0.864 -> 0.934 every other category exactly 0.000 A control run with the field indexed but neither sorted nor queried reproduced bills-summary.json exactly, so the delta is the sort alone. Demoting beats dropping: a config.filter scores worse on both counts and SearchIndexer deletes filtered documents from the index, making the orders unfindable rather than merely outranked. Only Order and Extension Order: demoting everything that is not a Bill scores higher still but pushes down Resolves and constitutional amendment proposals. Verified by hand (no golden names an order): lookup by number survives — no bill shares an order's number — but searching the type by name does not, which is why the legislationType refinement ships with the sort instead of after it. _eval conditional sort needs Typesense 26+; the server upgrade this branch already requires is tracked in codeforboston#94. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs/search-deployment.md records where the live Typesense servers actually are (AWS ECS via maple-testimony/infra), the four merges that ship this branch — infra then code, for dev, then again for prod, with no manual steps: both pipelines already do the work — the alias-swap/reindex mechanics and how to watch a chunked backfill (yarn firebase-admin run-script searchUpgradeStatus, yarn typesense-admin status), and the two failure modes that are quiet: a missing synonym set regresses relevance with no error (deploys re-upsert it from checkSearchIndexVersion, and the confirm step prints the server's sets), and a prod frontend with unset NEXT_PUBLIC_TYPESENSE_* silently searches the hardcoded dev cluster. Prod deploys from the prod branch — the same commit main already ran on dev. Linked from the search-eval and scripts READMEs. Everything mechanical lives at its own call site: collection-name hashing, alias-swap internals and the convertVersion escape hatch; measurements live on codeforboston#73. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fastfadingviolets
force-pushed
the
search-weights
branch
from
August 26, 2026 19:28
bc93791 to
9a1619f
Compare
Contributor
Author
|
The vercel preview is broken because the typesense indexes aren't updated yet, but I've checked this works locally. |
fastfadingviolets
force-pushed
the
search-weights
branch
from
August 27, 2026 14:41
6d7399d to
16b64cc
Compare
…on#120) - backfill pages on FieldPath.documentId() with the doc path as cursor, so duplicate idField values (e.g. H100 across courts) no longer skip documents at page boundaries; regression test added - chunk chaining and startUpgrade are replay-safe under at-least-once delivery: ALREADY_EXISTS treated as success, recorded runId kept, age gate runs before any parsing, run-doc writes use update() so a racing chunk can't resurrect a deleted run - billsRelevanceSort rewritten bracket-free (qs corrupted routed URLs); 82/82 bills goldens byte-identical to baseline - SortBy falls back to the default option for unknown routed sort values instead of crashing into the error boundary - VirtualFilters includes the legislationType refinement - typesense-admin status: catch the synonym-sets probe immediately, single collections listing instead of N+1 - backfill serializes each doc once (raw JSONL import) with failure detection that handles the client's string-form responses Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fastfadingviolets
force-pushed
the
search-weights
branch
from
August 27, 2026 14:43
16b64cc to
b102aaf
Compare
fastfadingviolets
changed the title
fastfadingviolets
marked this pull request as ready for review
August 27, 2026 16:35
fastfadingviolets
requested review from
mertbagt,
mvictor55,
nesanders,
sashamaryl and
timblais
as code owners
August 27, 2026 16:35
fastfadingviolets
requested review from
Mephistic,
alexjball and
kiminkim724
as code owners
August 27, 2026 16:35
fastfadingviolets
force-pushed
the
search-weights
branch
from
August 27, 2026 16:39
b102aaf to
27cd194
Compare
- TestimonyHit renders content through <Snippet> instead of the raw trimContent() call, matching how BillHit/HearingHit already render matched text. testimonySearchParams sets highlight_full_fields="" since no testimony field uses <Highlight> — left at the adapter's default it would send a second, full-length marked copy of `content` on every hit for nothing. - infra/Dockerfile.node, infra/Dockerfile.firebase, and .nvmrc bumped 20 -> 22, matching package.json's engines pin. That pin moved to ^22 in 1c222ee (Typesense JS client 3.x needs node >=22) along with CI's setup-node, but missed these three files, leaving `yarn dev:up` unable to install. - typesense-admin: add `preview-eval-corpus`, which points the local app's Typesense aliases at search-eval's seeded `_eval` collections so `yarn dev:up` can browse the frozen golden-set corpus instead of the dev-workflow backfill. Kept out of `search-eval seed` itself so eval runs never repoint app aliases as a side effect. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fastfadingviolets
force-pushed
the
search-weights
branch
from
August 27, 2026 18:52
27cd194 to
2a1792b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
These tackle a few bits of low-hanging fruit in the search pipeline, namely:
Checklist
firestore.indexes.json(Please do not only create indexes through the Firebase Web UI, even though the error messages may reccommend it - indexes created this way may be obliterated by subsequent deploys)Screenshots
Add some screenshots highlighting your changes.
Known issues
If you've run against limitations or caveats, include them here. Include follow-up issues as well.
Steps to test/reproduce
For each feature or bug fix, create a step by step list for how a reviewer can test it out. E.g.: