Skip to content

sql_database source: timestamp/datetime column precision is never reflected, at any reflection_level #4316

Description

@geforce123

Title

sql_database source: timestamp/datetime column precision is never reflected, at any reflection_level

Environment

  • dlt 1.29.1
  • SQLAlchemy 2.0.51 (reproduces with any dialect that reports .precision on its
    DateTime type -- reproduced concretely against Oracle 19.28, oracledb 4.0.2)

Description

sqla_col_to_column_schema (dlt/sources/sql_database/schema_types.py) has explicit
precision handling for Numeric, SmallInteger, String, and _Binary types, gated
by add_precision = reflection_level == "full_with_precision". The DateTime branch
has no equivalent handling at all:

elif isinstance(sql_t, sqltypes.DateTime):
    col["data_type"] = "timestamp"
    col["timezone"] = sql_t.timezone or sql_t.__visit_name__ in ("DATETIMEOFFSET",)
    # <- no precision/scale read here, at any reflection_level

This means a source column's declared fractional-second precision (e.g. Oracle
TIMESTAMP(0), TIMESTAMP(6), TIMESTAMP(9)) is silently dropped during reflection,
even with reflection_level="full_with_precision" and detect_precision_hints=True.
The destination then falls back to whatever its default timestamp precision is
(e.g. Snowflake's default is scale 9), regardless of what the source actually declared
or contains.

Notably, the raw driver-level metadata does have this information --
oracledb's cursor.description reports the correct scale (6, 9, etc.) for these
columns -- so the data needed is available further down the stack, just not
propagated into dlt's own type-mapping layer.

Minimal repro

from dlt.sources.sql_database import sql_database

source = sql_database(
    "oracle+oracledb://user:pass@host:1521/?service_name=XXX",
    schema="SOME_SCHEMA",
    table_names=["some_table"],  # has a TIMESTAMP(6) column
    reflection_level="full_with_precision",
    detect_precision_hints=True,
)
pipeline = dlt.pipeline(pipeline_name="repro", destination="duckdb", dataset_name="repro")
pipeline.run(source)
print(pipeline.default_schema.get_table("some_table")["columns"]["some_ts_column"])
# -> no 'precision' key at all, regardless of reflection_level

Expected

For dialects whose reflected DateTime type object exposes .precision (Oracle's
does, via SQLAlchemy's Oracle dialect), full_with_precision reflection should carry
it into col["precision"], the same way it already does for Numeric/String.

Workaround we're using in the meantime

A supplementary ALL_TAB_COLUMNS catalog query, with precision applied afterward via
resource.apply_hints(columns={...}) -- since the SQLAlchemy type object doesn't
expose .precision for Oracle TIMESTAMP either (confirmed via vars() on the
reflected type), this can't currently be fixed purely with a type_adapter_callback.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions