Add column-level evaluation layer and rank proposals in propose - #137
Open
yhong123 wants to merge 12 commits into
Open
Add column-level evaluation layer and rank proposals in propose#137yhong123 wants to merge 12 commits into
yhong123 wants to merge 12 commits into
Conversation
added 12 commits
September 9, 2026 17:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolved #112
Summary
Adds a column-level evaluation layer that scores and ranks candidate generators against real data, and uses it to improve the
proposecommand inconfigure-generators.datafaker.evaluatorspackage:ColumnEvaluatorsamples a real column, picks an evaluation profile (short text / free text / categorical / identifier) from its statistics, and scores each candidate proposer's synthetic output against it using novelty, diversity, and statistical fidelity metrics (metrics.py,statistical_fidelity.py,feature_extractors.py,distribution_builders.py).proposal_ranking.py: normalises the heterogeneous raw fit scores into a probabilistic ranking and truncates the displayed list with entropy-based cutoff, soproposeno longer dumps every candidate unranked.proposecommand (interactive/generators.py): now surfaces ranked proposals (capped atMAX_PROPOSERS_SHOWN = 10,propose allstill shows everything) and filters out continuous-float proposers for genuinelyInteger-typed columns, since inserting a non-whole float into an Integer column is backend-dependent.IncrementProposer(proposers/sequence.py): a new proposer for integer primary keys that continues the real data's max value, reusing the samegeneric.column_value_provider.incrementmechanismmake.pyalready uses as its default for integer PKs. (Works towards Add a UniqueSequenceProposer for unique/ID columns — no current proposer guarantees non-duplicate values #134)dialects.py: addsWordCount/SentenceCount(and related) SQL expression helpers, with Postgres/DuckDB and MSSQL compilation, used by the new feature extractors.builtin_generators.rstandchoosing_a_generator.rst, glossary entries for Generator/Proposer/Role, and TOC updates. (Work towards https://github.com/alan-turing-institute/DataMatryoshka/issues/14)Implements the "Enhancement to generators selection layer" half of #112 (keyword/ranking improvements to
propose, without changing generator/fit logic) plus the groundwork column-level fidelity metrics for the evaluation layer described there. Theevaluatecommand and table/dataset-level utility & privacy metrics from #112 are not part of this PR -- a new issue will be created for this.Further details on the generator evaluation and ranking methods can be found here: https://safehr-data.github.io/datafaker/builtin_generators.html#evaluating-and-ranking-proposals
Note
@tim-band your original fit-value table is left in the codebase alongside the new ranked table — I didn't want to remove it without your sign-off. Specifically, this PR does not touch:
interactive/generators.py:1072-1090— the old per-proposer loop that printsPROPOSE_GENERATOR_SAMPLE_TEXTusingprop.fit(-1)Proposer.fitinproposers/base.py:140, which that loop callsOnce you've reviewed and are happy with the ranking table (
RANKED_SAMPLE_TEXT) replacing this output, let me know and I'll remove both in a follow-up — or if you'd rather keep both tables shown side by side permanently, let me know that too and I'll leave them as-is.Test plan
poetry run pytest tests/test_evaluators_*.py— 169 passedconfigure-generators→proposeinteractively against a real table to confirm ranked output andpropose allstill work as expected