Skip to content

[dagster-dbt] Relax sqlglot upper bound after fixing 28.1+ column-lineage compatibility #34097

Description

@sebfest

Summary

dagster-dbt currently declares sqlglot[rs]<28.1.0, which prevents environments that need a newer SQLGlot release from resolving. This affects coexistence with newer acryl-datahub[dbt] releases, which pin SQLGlot in the 30.x line.

Root cause

The upper bound was introduced in PR #32947 after SQLGlot 28.1.0 caused an undocumented failure in Dagster's dbt column-lineage path. The failure is reproducible for CTE/join queries:

AttributeError: 'str' object has no attribute 'copy'

SQLGlot 28.1 changed its qualify_tables implementation to assume aliases are expression objects, while Dagster passes an already-optimized AST to sqlglot.lineage(); optimized CTE/join aliases can be strings.

Proposed fix

Serialize the optimized AST to SQL before passing it to lineage(). This lets SQLGlot reparse the query into the expected expression representation. With that change, Dagster's existing duplicate_column_dep_orders column-lineage case passes on SQLGlot 28.1.0, 29.0.1, and 30.16.0.

Then remove the <28.1.0 upper bound and regenerate the lockfile.

Validation

  • Existing Dagster dbt column-lineage integration case passes on SQLGlot 28.1.0.
  • The same case passes on SQLGlot 29.0.1.
  • SQLGlot 30.16.0 parser/optimizer/lineage smoke test passes.
  • ruff and the package lock check pass.

Proposed PR

#34098 contains the proposed change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions