fix(dbt): emit expr '1' for COUNT(*) metrics - #432
LukasSchwarzlmueller wants to merge 4 commits into
Conversation
ossie-to-msi turned COUNT(*) into a count metric with expr '*'. MetricFlow renders count metrics as SUM(CASE WHEN <expr> IS NOT NULL THEN 1 ELSE 0 END), so the generated query contained CASE WHEN * IS NOT NULL, which is invalid SQL. Use the constant 1 instead. It is never null, so every row is counted, which is COUNT(*) semantics. COUNT(dataset.*) is handled the same way.
|
|
||
| # COUNT(*) → count of the constant 1 (MetricFlow cannot render a bare * inside a count) | ||
| if isinstance(tree, exp.Count) and _is_star(tree.this): | ||
| return AggregationType.COUNT, "1", None, False |
There was a problem hiding this comment.
I believe this hands "1" back as a column name, and the caller resolves it like one.
_convert_metric passes bare_col to _find_dataset_for_col, which for "1" finds no qualifier, matches no field, and falls through to datasets[0].name.
Two ways that goes wrong:
- Doc with
datasets: [customers, orders]andorder_count = COUNT(*)producesmetric_aggregation_params.semantic_model == "customers". Before this PR, the emitted*made MetricFlow generateCASE WHEN * IS NOT NULLand the query failed to compile. Now it compiles and returns the customers row count under the nameorder_count. In a "ratio", it's worse:(SUM(amount)) / (COUNT(*))over those same two datasets gives numeratorordersand denominatorcustomers, so MetricFlow joins two unrelated models and the average is quietly wrong. - If any dataset has a field whose expression is the constant
1, the field expression scan matches it and everyCOUNT(*)in the document binds to that dataset.
1 is a constant, not a column, so it should not go through the column -> dataset resolver at all.
I suggest resolving the star case from the qualifier only, and refusing when it's ambiguous, which is that the Microsoft converter already does:
# _sql_to_dax.py:208-210
table = resolve_table()
if table is None:
return None, "'COUNT(*)' needs exactly one dataset to count rows of"Trading a compile error for a plausible-looking wrong number is the part I would like to avoid here. Wdyt?
There was a problem hiding this comment.
You're right, thanks for catching this. Fixed in 664e3a1.
One thing i was unsure of: it raises instead of skipping the metric, because the converter has no per-metric skip. I can switch to skip-with-a-warning if you prefer.
There was a problem hiding this comment.
Up to you. I think skip-with-a-warning is fine.
There was a problem hiding this comment.
Can you update the PR that way? Thanks!
|
|
||
| def _is_star(node: exp.Expression) -> bool: | ||
| """Return True for ``*`` and for a qualified ``dataset.*``.""" | ||
| return isinstance(node, exp.Star) or (isinstance(node, exp.Column) and isinstance(node.this, exp.Star)) |
There was a problem hiding this comment.
This accepts any multi-part qualified star, and the qualifier is then used verbatim as a semantic model name.
COUNT(db.orders, *) satisfies isinstance(node.this, exp.Star), so we take the star branch and _get_dataset_qualifier joins part[:-1] into "db.orders". The metric comes out with metric_aggregation_params.semantic_model == "db.orders" while the manifest's model is named orders, and MetricFlow can't resolve the metric owning model. Unlike the plain column path there is no bare column fallback to recover from it, because the star branch discards the node entirely.
I suggest to either restrict _is_star to a single part qualifier, or have the star path resolve the qualifier last segment against the actual dataset names and refuse when there is no match.
There was a problem hiding this comment.
Went with your second option. Fixed in 664e3a1.
- Restricting
_is_starwould senddb.orders.*down the column path and give*again. - Your literal
COUNT(db.orders, *)example is caught by the extra-argument check from the other thread.
|
@LukasSchwarzlmueller thanks for the updates! I will do a new pass. Much appreciated 😄 |
Summary
ossie-to-msiturnedCOUNT(*)into acountmetric withtype_params.expr: '*'. MetricFlow renders everycountmetric asSUM(CASE WHEN <expr> IS NOT NULL THEN 1 ELSE 0 END), so the generated query containedCASE WHEN * IS NOT NULL, which is not valid SQL (DuckDB:STAR expression is only allowed as the root element of an expression).COUNT(*)andCOUNT(<dataset>.*)now become acountmetric with the constantexpr: '1'.1is never null, so every row is counted, which isCOUNT(*)semantics, and it is the form MetricFlow itself documents for counting all rows.Which dataset the rows belong to is decided by the qualifier only, never by guessing:
COUNT(orders.*)binds to the datasetorders; a schema prefix such asdb.orders.*is matched on its last segment, and a name that matches no dataset (or several) is skipped with a warning, see below.COUNT(*)is accepted only when the document has a single dataset. With several, that metric is skipped with aROW_COUNT_METRIC_DROPPEDwarning instead of silently counting the first dataset's rows. A ratio that uses it is skipped as a whole, so nothing references a missing metric.ossie-to-msinow prints these warnings the waymsi-to-ossiealready does. Callers using the library directly should checkresult.issuesto see skipped metrics.COUNT(1)is treated likeCOUNT(*), since they are equivalent.COUNT(DISTINCT *)and multi-argumentCOUNT(COUNT(t.*, x)) are not row counts; they take the existing raw-expression fallback instead of emitting*or dropping an argument.Round trip: MetricFlow's loader turns
countintoSUM, and leavesexpr: 1asSUM(1), which loses the dataset.msi_to_ossienow remembers which metrics were row counts before that transform and emitsCOUNT(<dataset>.*), so a metric stays on its dataset through Ossie → MSI → Ossie → MSI. This applies to anycountmetric withexpr: 1, somsi-to-ossieoutput changes for existing dbt projects that use it. A filtered row count is unchanged: it still comes back asSUM(CASE WHEN <filter> THEN 1 END), which has nodataset.*form and can lose its dataset, as filtered metrics could before.Checked end to end on DuckDB (
dbt run, thenmf query; the manifest was also patched foragg_time_dimensionand the time spine, which are separate converter gaps this PR does not touch). WithCOUNT(orders.*)the metric lands onorderswithout any patch. Before,mf query --metrics order_countfails with the binder error above; after, it returns 4 for four orders, and the per-segment counts, revenue and average match a plain SQL query on the same tables.Changes:
expression_utils.py:ROW_COUNT_EXPR, an_is_starhelper, and a singleCOUNTblock that rejects extra arguments, unwrapsDISTINCT, then checks for a star.ossie_to_msi.py:_find_dataset_for_row_count, so the row count does not go through the column → dataset lookup.converter_issues.py,cli.py: the newROW_COUNT_METRIC_DROPPEDissue type, and warnings printed forossie-to-msi.msi_to_ossie.py: remembers row-count metrics before the MetricFlow transform and re-emitsCOUNT(<dataset>.*).README.md: one line under the Ossie → MSI conversion choices.test_ossie_to_msi.pyandtest_msi_to_ossie.py, including a two-dataset round trip, the skipped-with-a-warning cases, and a CLI test for the warning.The full
converters/dbtsuite passes on Python 3.11, 3.12, 3.13 and 3.14 (124 tests). Existing snapshots are unchanged.Related Issues
None yet.
Checklist
Converters
converters/is updated to reflect spec or ontology changesDocumentation
converters/dbt/README.mdis updated to reflect the user-facing changeTests
pytest/ CI green)Compliance
Specification, Ontology, Validation and Examples: not applicable.