Skip to content

Add column-level evaluation layer and rank proposals in propose - #137

Open
yhong123 wants to merge 12 commits into
mainfrom
yhong123/112-column_evaluators
Open

Add column-level evaluation layer and rank proposals in propose#137
yhong123 wants to merge 12 commits into
mainfrom
yhong123/112-column_evaluators

Conversation

@yhong123

@yhong123 yhong123 commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Resolved #112

Summary

Adds a column-level evaluation layer that scores and ranks candidate generators against real data, and uses it to improve the propose command in configure-generators.

  • New datafaker.evaluators package: ColumnEvaluator samples a real column, picks an evaluation profile (short text / free text / categorical / identifier) from its statistics, and scores each candidate proposer's synthetic output against it using novelty, diversity, and statistical fidelity metrics (metrics.py, statistical_fidelity.py, feature_extractors.py, distribution_builders.py).
  • proposal_ranking.py: normalises the heterogeneous raw fit scores into a probabilistic ranking and truncates the displayed list with entropy-based cutoff, so propose no longer dumps every candidate unranked.
  • propose command (interactive/generators.py): now surfaces ranked proposals (capped at MAX_PROPOSERS_SHOWN = 10, propose all still shows everything) and filters out continuous-float proposers for genuinely Integer-typed columns, since inserting a non-whole float into an Integer column is backend-dependent.
  • IncrementProposer (proposers/sequence.py): a new proposer for integer primary keys that continues the real data's max value, reusing the same generic.column_value_provider.increment mechanism make.py already uses as its default for integer PKs. (Works towards Add a UniqueSequenceProposer for unique/ID columns — no current proposer guarantees non-duplicate values #134)
  • dialects.py: adds WordCount/SentenceCount (and related) SQL expression helpers, with Postgres/DuckDB and MSSQL compilation, used by the new feature extractors.
  • Docs: new builtin_generators.rst and choosing_a_generator.rst, glossary entries for Generator/Proposer/Role, and TOC updates. (Work towards https://github.com/alan-turing-institute/DataMatryoshka/issues/14)

Implements the "Enhancement to generators selection layer" half of #112 (keyword/ranking improvements to propose, without changing generator/fit logic) plus the groundwork column-level fidelity metrics for the evaluation layer described there. The evaluate command and table/dataset-level utility & privacy metrics from #112 are not part of this PR -- a new issue will be created for this.

Further details on the generator evaluation and ranking methods can be found here: https://safehr-data.github.io/datafaker/builtin_generators.html#evaluating-and-ranking-proposals

Note

@tim-band your original fit-value table is left in the codebase alongside the new ranked table — I didn't want to remove it without your sign-off. Specifically, this PR does not touch:

  • interactive/generators.py:1072-1090 — the old per-proposer loop that prints PROPOSE_GENERATOR_SAMPLE_TEXT using prop.fit(-1)
  • Proposer.fit in proposers/base.py:140, which that loop calls

Once you've reviewed and are happy with the ranking table (RANKED_SAMPLE_TEXT) replacing this output, let me know and I'll remove both in a follow-up — or if you'd rather keep both tables shown side by side permanently, let me know that too and I'll leave them as-is.

Test plan

  • poetry run pytest tests/test_evaluators_*.py — 169 passed
  • Exercise configure-generatorspropose interactively against a real table to confirm ranked output and propose all still work as expected

@yhong123
yhong123 requested a review from tim-band September 11, 2026 12:46
@yhong123 yhong123 self-assigned this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enhancement to generators selection layer and new evaluation layer

1 participant