Title
sql_database source: timestamp/datetime column precision is never reflected, at any reflection_level
Environment
- dlt 1.29.1
- SQLAlchemy 2.0.51 (reproduces with any dialect that reports
.precision on its
DateTime type -- reproduced concretely against Oracle 19.28, oracledb 4.0.2)
Description
sqla_col_to_column_schema (dlt/sources/sql_database/schema_types.py) has explicit
precision handling for Numeric, SmallInteger, String, and _Binary types, gated
by add_precision = reflection_level == "full_with_precision". The DateTime branch
has no equivalent handling at all:
elif isinstance(sql_t, sqltypes.DateTime):
col["data_type"] = "timestamp"
col["timezone"] = sql_t.timezone or sql_t.__visit_name__ in ("DATETIMEOFFSET",)
# <- no precision/scale read here, at any reflection_level
This means a source column's declared fractional-second precision (e.g. Oracle
TIMESTAMP(0), TIMESTAMP(6), TIMESTAMP(9)) is silently dropped during reflection,
even with reflection_level="full_with_precision" and detect_precision_hints=True.
The destination then falls back to whatever its default timestamp precision is
(e.g. Snowflake's default is scale 9), regardless of what the source actually declared
or contains.
Notably, the raw driver-level metadata does have this information --
oracledb's cursor.description reports the correct scale (6, 9, etc.) for these
columns -- so the data needed is available further down the stack, just not
propagated into dlt's own type-mapping layer.
Minimal repro
from dlt.sources.sql_database import sql_database
source = sql_database(
"oracle+oracledb://user:pass@host:1521/?service_name=XXX",
schema="SOME_SCHEMA",
table_names=["some_table"], # has a TIMESTAMP(6) column
reflection_level="full_with_precision",
detect_precision_hints=True,
)
pipeline = dlt.pipeline(pipeline_name="repro", destination="duckdb", dataset_name="repro")
pipeline.run(source)
print(pipeline.default_schema.get_table("some_table")["columns"]["some_ts_column"])
# -> no 'precision' key at all, regardless of reflection_level
Expected
For dialects whose reflected DateTime type object exposes .precision (Oracle's
does, via SQLAlchemy's Oracle dialect), full_with_precision reflection should carry
it into col["precision"], the same way it already does for Numeric/String.
Workaround we're using in the meantime
A supplementary ALL_TAB_COLUMNS catalog query, with precision applied afterward via
resource.apply_hints(columns={...}) -- since the SQLAlchemy type object doesn't
expose .precision for Oracle TIMESTAMP either (confirmed via vars() on the
reflected type), this can't currently be fixed purely with a type_adapter_callback.
Title
sql_database source: timestamp/datetime column precision is never reflected, at any
reflection_levelEnvironment
.precisionon itsDateTimetype -- reproduced concretely against Oracle 19.28,oracledb4.0.2)Description
sqla_col_to_column_schema(dlt/sources/sql_database/schema_types.py) has explicitprecision handling for
Numeric,SmallInteger,String, and_Binarytypes, gatedby
add_precision = reflection_level == "full_with_precision". TheDateTimebranchhas no equivalent handling at all:
This means a source column's declared fractional-second precision (e.g. Oracle
TIMESTAMP(0),TIMESTAMP(6),TIMESTAMP(9)) is silently dropped during reflection,even with
reflection_level="full_with_precision"anddetect_precision_hints=True.The destination then falls back to whatever its default timestamp precision is
(e.g. Snowflake's default is scale 9), regardless of what the source actually declared
or contains.
Notably, the raw driver-level metadata does have this information --
oracledb'scursor.descriptionreports the correct scale (6, 9, etc.) for thesecolumns -- so the data needed is available further down the stack, just not
propagated into dlt's own type-mapping layer.
Minimal repro
Expected
For dialects whose reflected
DateTimetype object exposes.precision(Oracle'sdoes, via SQLAlchemy's Oracle dialect),
full_with_precisionreflection should carryit into
col["precision"], the same way it already does forNumeric/String.Workaround we're using in the meantime
A supplementary
ALL_TAB_COLUMNScatalog query, with precision applied afterward viaresource.apply_hints(columns={...})-- since the SQLAlchemy type object doesn'texpose
.precisionfor OracleTIMESTAMPeither (confirmed viavars()on thereflected type), this can't currently be fixed purely with a
type_adapter_callback.