Summary
dagster-dbt currently declares sqlglot[rs]<28.1.0, which prevents environments that need a newer SQLGlot release from resolving. This affects coexistence with newer acryl-datahub[dbt] releases, which pin SQLGlot in the 30.x line.
Root cause
The upper bound was introduced in PR #32947 after SQLGlot 28.1.0 caused an undocumented failure in Dagster's dbt column-lineage path. The failure is reproducible for CTE/join queries:
AttributeError: 'str' object has no attribute 'copy'
SQLGlot 28.1 changed its qualify_tables implementation to assume aliases are expression objects, while Dagster passes an already-optimized AST to sqlglot.lineage(); optimized CTE/join aliases can be strings.
Proposed fix
Serialize the optimized AST to SQL before passing it to lineage(). This lets SQLGlot reparse the query into the expected expression representation. With that change, Dagster's existing duplicate_column_dep_orders column-lineage case passes on SQLGlot 28.1.0, 29.0.1, and 30.16.0.
Then remove the <28.1.0 upper bound and regenerate the lockfile.
Validation
- Existing Dagster dbt column-lineage integration case passes on SQLGlot 28.1.0.
- The same case passes on SQLGlot 29.0.1.
- SQLGlot 30.16.0 parser/optimizer/lineage smoke test passes.
ruff and the package lock check pass.
Proposed PR
#34098 contains the proposed change.
Summary
dagster-dbtcurrently declaressqlglot[rs]<28.1.0, which prevents environments that need a newer SQLGlot release from resolving. This affects coexistence with neweracryl-datahub[dbt]releases, which pin SQLGlot in the 30.x line.Root cause
The upper bound was introduced in PR #32947 after SQLGlot 28.1.0 caused an undocumented failure in Dagster's dbt column-lineage path. The failure is reproducible for CTE/join queries:
SQLGlot 28.1 changed its
qualify_tablesimplementation to assume aliases are expression objects, while Dagster passes an already-optimized AST tosqlglot.lineage(); optimized CTE/join aliases can be strings.Proposed fix
Serialize the optimized AST to SQL before passing it to
lineage(). This lets SQLGlot reparse the query into the expected expression representation. With that change, Dagster's existingduplicate_column_dep_orderscolumn-lineage case passes on SQLGlot 28.1.0, 29.0.1, and 30.16.0.Then remove the
<28.1.0upper bound and regenerate the lockfile.Validation
ruffand the package lock check pass.Proposed PR
#34098 contains the proposed change.