Skip to content

fix(dbt): emit expr '1' for COUNT(*) metrics - #432

Open
LukasSchwarzlmueller wants to merge 4 commits into
apache:mainfrom
LukasSchwarzlmueller:fix/dbt-count-star
Open

LukasSchwarzlmueller wants to merge 4 commits into
apache:mainfrom
LukasSchwarzlmueller:fix/dbt-count-star

Conversation

@LukasSchwarzlmueller

@LukasSchwarzlmueller LukasSchwarzlmueller commented Sep 19, 2026

Copy link
Copy Markdown

Summary

ossie-to-msi turned COUNT(*) into a count metric with type_params.expr: '*'. MetricFlow renders every count metric as SUM(CASE WHEN <expr> IS NOT NULL THEN 1 ELSE 0 END), so the generated query contained CASE WHEN * IS NOT NULL, which is not valid SQL (DuckDB: STAR expression is only allowed as the root element of an expression).

COUNT(*) and COUNT(<dataset>.*) now become a count metric with the constant expr: '1'. 1 is never null, so every row is counted, which is COUNT(*) semantics, and it is the form MetricFlow itself documents for counting all rows.

Which dataset the rows belong to is decided by the qualifier only, never by guessing:

  • COUNT(orders.*) binds to the dataset orders; a schema prefix such as db.orders.* is matched on its last segment, and a name that matches no dataset (or several) is skipped with a warning, see below.
  • A bare COUNT(*) is accepted only when the document has a single dataset. With several, that metric is skipped with a ROW_COUNT_METRIC_DROPPED warning instead of silently counting the first dataset's rows. A ratio that uses it is skipped as a whole, so nothing references a missing metric. ossie-to-msi now prints these warnings the way msi-to-ossie already does. Callers using the library directly should check result.issues to see skipped metrics.
  • COUNT(1) is treated like COUNT(*), since they are equivalent.
  • COUNT(DISTINCT *) and multi-argument COUNT (COUNT(t.*, x)) are not row counts; they take the existing raw-expression fallback instead of emitting * or dropping an argument.

Round trip: MetricFlow's loader turns count into SUM, and leaves expr: 1 as SUM(1), which loses the dataset. msi_to_ossie now remembers which metrics were row counts before that transform and emits COUNT(<dataset>.*), so a metric stays on its dataset through Ossie → MSI → Ossie → MSI. This applies to any count metric with expr: 1, so msi-to-ossie output changes for existing dbt projects that use it. A filtered row count is unchanged: it still comes back as SUM(CASE WHEN <filter> THEN 1 END), which has no dataset.* form and can lose its dataset, as filtered metrics could before.

Checked end to end on DuckDB (dbt run, then mf query; the manifest was also patched for agg_time_dimension and the time spine, which are separate converter gaps this PR does not touch). With COUNT(orders.*) the metric lands on orders without any patch. Before, mf query --metrics order_count fails with the binder error above; after, it returns 4 for four orders, and the per-segment counts, revenue and average match a plain SQL query on the same tables.

Changes:

  • expression_utils.py: ROW_COUNT_EXPR, an _is_star helper, and a single COUNT block that rejects extra arguments, unwraps DISTINCT, then checks for a star.
  • ossie_to_msi.py: _find_dataset_for_row_count, so the row count does not go through the column → dataset lookup.
  • converter_issues.py, cli.py: the new ROW_COUNT_METRIC_DROPPED issue type, and warnings printed for ossie-to-msi.
  • msi_to_ossie.py: remembers row-count metrics before the MetricFlow transform and re-emits COUNT(<dataset>.*).
  • README.md: one line under the Ossie → MSI conversion choices.
  • Tests in test_ossie_to_msi.py and test_msi_to_ossie.py, including a two-dataset round trip, the skipped-with-a-warning cases, and a CLI test for the warning.

The full converters/dbt suite passes on Python 3.11, 3.12, 3.13 and 3.14 (124 tests). Existing snapshots are unchanged.

Related Issues

None yet.

Checklist

Converters

  • Converter logic in converters/ is updated to reflect spec or ontology changes
  • New converters include tests under the converter's test directory

Documentation

  • converters/dbt/README.md is updated to reflect the user-facing change

Tests

  • All existing tests pass (pytest / CI green)
  • New functionality is covered by tests

Compliance

  • ASF license headers are present on all new source files (no new files added)
  • No third-party dependencies are added without PMC/IPMC approval

Specification, Ontology, Validation and Examples: not applicable.

ossie-to-msi turned COUNT(*) into a count metric with expr '*'. MetricFlow
renders count metrics as SUM(CASE WHEN <expr> IS NOT NULL THEN 1 ELSE 0 END),
so the generated query contained CASE WHEN * IS NOT NULL, which is invalid SQL.

Use the constant 1 instead. It is never null, so every row is counted, which
is COUNT(*) semantics. COUNT(dataset.*) is handled the same way.
@jbonofre
jbonofre self-requested a review September 20, 2026 05:10

# COUNT(*) → count of the constant 1 (MetricFlow cannot render a bare * inside a count)
if isinstance(tree, exp.Count) and _is_star(tree.this):
return AggregationType.COUNT, "1", None, False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this hands "1" back as a column name, and the caller resolves it like one.

_convert_metric passes bare_col to _find_dataset_for_col, which for "1" finds no qualifier, matches no field, and falls through to datasets[0].name.
Two ways that goes wrong:

  1. Doc with datasets: [customers, orders] and order_count = COUNT(*) produces metric_aggregation_params.semantic_model == "customers". Before this PR, the emitted * made MetricFlow generate CASE WHEN * IS NOT NULL and the query failed to compile. Now it compiles and returns the customers row count under the name order_count. In a "ratio", it's worse: (SUM(amount)) / (COUNT(*)) over those same two datasets gives numerator orders and denominator customers, so MetricFlow joins two unrelated models and the average is quietly wrong.
  2. If any dataset has a field whose expression is the constant 1, the field expression scan matches it and every COUNT(*) in the document binds to that dataset.

1 is a constant, not a column, so it should not go through the column -> dataset resolver at all.

I suggest resolving the star case from the qualifier only, and refusing when it's ambiguous, which is that the Microsoft converter already does:

# _sql_to_dax.py:208-210
table = resolve_table()
if table is None:
    return None, "'COUNT(*)' needs exactly one dataset to count rows of"

Trading a compile error for a plausible-looking wrong number is the part I would like to avoid here. Wdyt?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, thanks for catching this. Fixed in 664e3a1.

One thing i was unsure of: it raises instead of skipping the metric, because the converter has no per-metric skip. I can switch to skip-with-a-warning if you prefer.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Up to you. I think skip-with-a-warning is fine.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you update the PR that way? Thanks!

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread converters/dbt/src/ossie_dbt/expression_utils.py Outdated

def _is_star(node: exp.Expression) -> bool:
"""Return True for ``*`` and for a qualified ``dataset.*``."""
return isinstance(node, exp.Star) or (isinstance(node, exp.Column) and isinstance(node.this, exp.Star))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This accepts any multi-part qualified star, and the qualifier is then used verbatim as a semantic model name.

COUNT(db.orders, *) satisfies isinstance(node.this, exp.Star), so we take the star branch and _get_dataset_qualifier joins part[:-1] into "db.orders". The metric comes out with metric_aggregation_params.semantic_model == "db.orders" while the manifest's model is named orders, and MetricFlow can't resolve the metric owning model. Unlike the plain column path there is no bare column fallback to recover from it, because the star branch discards the node entirely.

I suggest to either restrict _is_star to a single part qualifier, or have the star path resolve the qualifier last segment against the actual dataset names and refuse when there is no match.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went with your second option. Fixed in 664e3a1.

  • Restricting _is_star would send db.orders.* down the column path and give * again.
  • Your literal COUNT(db.orders, *) example is caught by the extra-argument check from the other thread.

Comment thread converters/dbt/src/ossie_dbt/expression_utils.py Outdated
@jbonofre

Copy link
Copy Markdown
Member

@LukasSchwarzlmueller thanks for the updates! I will do a new pass. Much appreciated 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants