[dagster-dbt] Column-level lineage is not produced for dbt snapshots (reads compiled SQL only from target/compiled/)

[themattmorris]

What's the issue?

Column-level lineage (CLL) is never produced for dbt snapshots. In the Dagster UI, snapshot assets do not get the column-lineage icon that regular dbt models get, and no dagster/column_lineage metadata is attached to their materializations.

Root cause

This is fixable on the Dagster side. The nuance: dbt intentionally does not write compiled snapshot SQL to target/compiled/ — see dbt/compilation.py Compiler._write_node, which returns early (without writing the file) when node.resource_type in (NodeType.Snapshot, NodeType.Seed). For snapshots the compiled SQL is only available:

• in the manifest, as manifest.nodes[...]["compiled_code"] (always present after compile), and
• on disk under target/run/<package>/snapshots/<file>.sql (only after a dbt build/dbt snapshot run, not a bare dbt compile).

Dagster, however, reads compiled SQL exclusively from a hardcoded target/compiled/... filesystem path and has no fallback. When building column lineage, _build_column_lineage_metadata hardcodes the compiled subdirectory:

python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_event.py

package_name = dbt_resource_props["package_name"]
node_sql_path = target_path.joinpath(
    "compiled",                                   # <-- hardcoded
    package_name,
    dbt_resource_props["original_file_path"].replace("\\", "/"),
)
optimized_node_ast = cast(
    "exp.Query",
    optimize(
        parse_one(sql=node_sql_path.read_text(), dialect=sql_dialect),  # FileNotFoundError for snapshots
        ...
    ),
)

Note that "snapshot" is included in REFABLE_NODE_TYPES (compat.py), so snapshots pass the resource-type gate at the top of the function and reach this path-construction logic — they are not intentionally excluded. There's a telling asymmetry: seeds (which dbt also does not write to compiled/) are explicitly short-circuited earlier in the function (if unique_id.startswith("seed"): return {}), so they never hit the missing-file read. Snapshots fall through to the read and fail silently.

For a snapshot, target/compiled/<package>/snapshots/<file>.sql never exists (dbt does not write it there — see above), so node_sql_path.read_text() raises FileNotFoundError. That exception is caught and swallowed in _fetch_column_metadata:

python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_event_iterator.py

except Exception as e:
    logger.warning(
        "An error occurred while building column lineage metadata for the dbt resource"
        f" `{dbt_resource_props['original_file_path']}`."
        " Lineage metadata will not be included in the event.\n\n"
        f"Exception: {e}",
        exc_info=True,
    )

So lineage_metadata stays {}, no column lineage is attached, and the UI shows no CLL icon for the snapshot.

Original stack trace (from #30491):

FileNotFoundError: [Errno 2] No such file or directory:
  '/.../target/name-.../compiled/workspace/snapshots/model.sql'

The file exists at /.../target/name-.../run/workspace/snapshots/model.sql.

Prior history

• [dagster-dbt] fetch_column_metadata fails for dbt snapshots #30491 ("[dagster-dbt] fetch_column_metadata fails for dbt snapshots") reported the same underlying failure but was auto-closed as stale / not planned without a fix.
• [dagster-dbt] fetch_column_metadata fails for dbt snapshots #30491 #30493 was a community PR ("fall back to run folder") that attempted the fix but was closed without being merged.

The bug still reproduces on current master.

What did you expect to happen?

Column-level lineage should be produced for dbt snapshots the same way it is for models, by resolving the compiled SQL from a source that actually exists for snapshots (the manifest's compiled_code, or target/run/...) instead of only reading target/compiled/....

Suggested fix

Have _build_column_lineage_metadata fall back to the compiled SQL that dbt does provide when the compiled/ file is absent. The most robust source is the manifest's dbt_resource_props["compiled_code"], which is always populated for snapshots regardless of whether a dbt compile or a full dbt build/run was performed. (The closed PR #30493 proposed a narrower fallback to the target/run/... directory, which only works after a run.)

How to reproduce?

1. Define a dbt snapshot in a dbt project.
2. Materialize it in Dagster with column lineage enabled, e.g.:

   yield from dbt.cli(["build"], context=context).stream().fetch_column_metadata()

3. Observe a warning in the logs ("An error occurred while building column lineage metadata ...") and that the snapshot asset has no column lineage / no CLL icon in the UI, while models in the same project do.