[refactor](fe) Remove the JDBC data-source implementation from fe-core; serve it from fe-connector-jdbc by morningman · Pull Request #68037 · apache/doris

morningman · 2026-09-15T19:07:26Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #67916 (the same SQL Server type-mapping fix had to be made twice, once per copy)

Problem Summary:

fe-core still carried a full copy of the JDBC data-source implementation next to the
fe-connector-jdbc plugin: 13 dialect clients with their own type mappings
(datasource/jdbc/client/*, ~3000 lines), the driver-jar policy and url normalization
inside JdbcResource, and an engine hook (ConnectorValidationContext .validateAndResolveDriverPath/computeDriverChecksum) that lent that policy to plugins.
The only production user of the fe-core clients was the streaming/CDC framework, and the
two copies had drifted: three streaming-driven fixes (#64740 composite PK order, #63402
PostgreSQL neighbour-table leak, #61433 PostgreSQL array types) never reached the
connector, and #67916 had to patch both.

After this PR fe-core has no JDBC implementation; what remains is generic SPI use plus
two @Deprecated persistence shells (JdbcResource, JdbcTable).

SPI (major 9.0) — ConnectorTableMetadataOps.getPrimaryKeys,
ConnectorPassthroughSqlOps.executeQuery(session, sql, params) +
ConnectorQueryResult, ConnectorContext.fetchPluginFile (a cloud deployment's
object-store copy of a missing plugin file); ConnectorValidationContext loses its two
driver methods. DriverUrlPolicy (fe-connector-spi) is the one driver-jar policy —
url grammar, jdbc_driver_url_white_list, structural jdbc_driver_secure_path
matching, bare-name resolution across the current and pre-2.1 drivers directories,
cloud fetch, checksum — applied by the jdbc, iceberg and paimon connectors.
fe-connector-jdbc — implements the two new methods; applies DriverUrlPolicy
at CREATE and at load; ports the three fixes above (KEY_SEQ ordering in the base
client, PostgreSQL exact schema/table filter, bpchar[] elements as STRING).
Streaming — StreamingSourceClient (fe-core) opens a temporary connector through
ConnectorFactory with the job's source properties (same names as a JDBC catalog),
goes through the PluginDrivenMetadata funnel and ConnectorColumnConverter, and
runs the framework's probe queries with bound parameters. The four call sites keep
their logic. The fe-core clients and their tests are deleted.
JdbcResource — @Deprecated shell: Gson tag, configs, property list and
defaults, SHOW RESOURCES rows unchanged; validation and the driver checksum come
from the jdbc plugin. Dead JdbcTransactionManager and *_JDBC_* plan/rule types
removed; the "jdbc" gate around the schema-load debug point removed.
Gate — build-support/check-fe-core-jdbc-free.sh (validate phase, self-tested)
forbids org.apache.doris.datasource.jdbc.*, com.zaxxer.hikari.* and java.sql
connection/statement imports in fe-core main sources (httpv2/, JDBC to the FE
itself, exempt). fe-core drops its HikariCP dependency.

Release note

A JDBC catalog now applies jdbc_driver_secure_path / jdbc_driver_url_white_list
when it loads its driver, not only at CREATE; a catalog whose driver sits outside a
since-tightened allow-list fails to connect instead of loading the jar.
CREATE RESOURCE ... type=jdbc is deprecated in favour of
CREATE CATALOG ... "type"="jdbc"; it keeps working and SHOW RESOURCES shows
jdbc_url as written instead of a normalized form.
Streaming jobs need the jdbc connector plugin (bundled by default). The FE-side
metadata connection is built by the connector (its url normalization forces
tinyInt1isBit=false / yearIsDateType=false); the persisted job url and the BE CDC
reader are unchanged. PostgreSQL array columns whose element type the old client
rejected now map to ARRAY<STRING> (as the CDC client already did), empty tables no
longer fail array-dimension detection, and char(n)[] maps to ARRAY<STRING> in
JDBC catalogs again.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Unit tests run locally: fe-connector-spi (164, incl. 23 new DriverUrlPolicyTest),
fe-connector-jdbc (240; new PostgreSQL/OceanBase/base-client tests, pre-create validation
and load-time policy tests), fe-connector-paimon (574), fe-connector-iceberg (1431, one
failure pre-existing on master: IcebergWritePlanProviderTest .planMergePreservesExplicitlyEmptyReadAcrossConcurrentFirstAppend), the full fe-core
suite (10993 tests; the 12 failures in 10 classes are unrelated to this change and fail
identically on the base commit on the same machine), build-support/tests/run.sh.
Regression suites to watch in CI: external_table_p0/jdbc, job_p0/streaming_job/cdc,
query_p0/show/test_nereids_show_resources, manager/test_manager_interface_3,
test_cloud_plugin_auto_download (cloud).
Behavior changed:
- No.
- Yes. See the release note.
Does this need documentation?
- No.
- Yes. Deprecate CREATE RESOURCE type=jdbc;
  note that the driver allow-lists are enforced when a JDBC catalog loads its driver.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

🤖 Generated with Claude Code

hello-stephen · 2026-09-15T19:07:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2026-09-15T19:07:34Z

run buildall

morningman · 2026-09-15T19:07:36Z

/review

github-actions · 2026-09-15T19:40:59Z

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Sep 19th, 2026 8:24 AM.
Workflow run: https://github.com/apache/doris/actions/runs/35011738058

The selected account is excluded until 2026-09-19T08:24:00Z. Please trigger /review again; another configured account may be available.

… and add getPrimaryKeys/executeQuery Issue Number: close #xxx Related PR: #xxx Problem Summary: Part 1 of removing every JDBC data-source implementation from fe-core: the connector SPI grows what the streaming/CDC framework needs from a JDBC source, and the driver-jar policy stops being an engine service. - `ConnectorTableMetadataOps.getPrimaryKeys` (a JDBC table answers `isKey` for every column, so the real constraint needs its own method) and `ConnectorPassthroughSqlOps.executeQuery(session, sql, params)` with the `ConnectorQueryResult` value type; parameters bind positionally so a name a caller took from user input never becomes SQL text. - `ConnectorValidationContext` loses `validateAndResolveDriverPath` and `computeDriverChecksum`. The whole policy (url grammar, url white list, structural secure-path matching, bare-name resolution across the current and pre-2.1 drivers directories, checksum) moves to `DriverUrlPolicy` in fe-connector-spi, applied by the jdbc, iceberg and paimon connectors themselves; fe.conf's `jdbc_driver_secure_path` / `jdbc_driver_url_white_list` reach it through the engine environment. The only engine service left is `ConnectorContext.fetchPluginFile`, the cloud deployment's object-store copy of a missing driver jar. - The jdbc connector applies the policy at load time as well as at CREATE (the pre-SPI `JdbcClient` did too), so a catalog whose driver sits outside a since-tightened allow-list fails to connect instead of loading the jar. - Three fixes that had only landed in the fe-core client are ported to the connector clients: composite primary keys ordered by KEY_SEQ (apache#64740), the PostgreSQL neighbour-table column leak through LIKE wildcards (apache#63402), and PostgreSQL array element types (apache#61433; `bpchar[]` elements are STRING). - Plugin API version 8.0 -> 9.0, both surface baselines regenerated. A JDBC catalog now applies `jdbc_driver_secure_path` / `jdbc_driver_url_white_list` when it loads its driver, not only when it is created. PostgreSQL `char(n)[]` columns of a JDBC catalog map to `ARRAY<STRING>` again instead of `ARRAY<CHAR(n)>`. - Test: Unit Test - fe-connector-spi suite (incl. new DriverUrlPolicyTest), fe-connector-jdbc suite (new PostgreSQL/OceanBase/base-client tests, pre-create validation and load-time policy tests), fe-connector-paimon and fe-connector-iceberg suites. - Behavior changed: Yes (see release note) - Does this need documentation: No Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…tor SPI and delete the fe-core JDBC clients ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Part 2: the streaming/CDC framework was the last user of the fe-core copy of the JDBC dialect clients (`datasource/jdbc/client/*`, 13 classes and ~3000 lines that duplicated fe-connector-jdbc and had drifted from it). It now goes through the same connector plugin a JDBC catalog uses: - `StreamingSourceClient` (fe-core, `job/util`) opens a temporary connector via `ConnectorFactory` with the job's own source properties, acquires metadata through the `PluginDrivenMetadata` funnel, converts the connector schema with `ConnectorColumnConverter`, and runs the framework's probe queries through `executeQuery`. The four call sites (`generateCreateTableCmds`, the `cdc_stream` TVF, the OceanBase compatibility-mode check and the PostgreSQL slot/publication validator) keep their logic; only the client changes. - The fe-core clients, `JdbcFieldSchema` and their six tests are deleted; `JdbcClientException.getAllExceptionMessages` becomes `Util.getAllExceptionMessages`. - A streaming job now needs the jdbc connector plugin (bundled by default) and fails with a clear message without it. ### Release note None ### Check List (For Author) - Test: Unit Test - StreamingSourceClientTest (new), StreamingJobUtilsTest, DataSourceConfigValidatorTest, CdcStreamTableValuedFunctionTest, PostgresResourceValidatorTest (four new cases), UtilTest. - Behavior changed: Yes. The FE-side metadata connection of a streaming job is built by the connector, whose URL normalization forces `tinyInt1isBit=false` and `yearIsDateType=false`; the job's persisted URL and the BE CDC reader are unchanged. PostgreSQL array columns whose element type the old client rejected now map to `ARRAY<STRING>`, as the CDC client already did. - Does this need documentation: No Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…and drop dead JDBC plan/rule types ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Part 3: `JdbcResource` carried the last JDBC logic in fe-core — url normalization, dialect constants, driver-jar resolution, white list, secure path, checksum and cloud download — although a resource no longer backs anything (`CREATE CATALOG ... WITH RESOURCE` is disallowed by default and reads nothing from a JDBC resource). It is now a `@Deprecated` shell: the Gson tag, the persisted `configs` field, the property list and defaults and the `SHOW RESOURCES` rows are unchanged, `CREATE RESOURCE type=jdbc` keeps working, and property validation plus the driver checksum are done by the jdbc connector plugin exactly as for a JDBC catalog. The stored `jdbc_url` is no longer rewritten. Also removed: the empty `JdbcTransactionManager`, the unreferenced `LOGICAL/PHYSICAL_JDBC_*` `PlanType` values and `*_JDBC_*` `RuleType` values, and the `"jdbc"` source-name gate around the schema-load debug point in `PluginDrivenExternalTable` (the debug point now applies to any plugin table). ### Release note `CREATE RESOURCE ... type=jdbc` is deprecated; use `CREATE CATALOG ... "type"="jdbc"`. It keeps working, validated by the jdbc connector plugin, and `SHOW RESOURCES` shows the `jdbc_url` as written instead of a normalized form. ### Check List (For Author) - Test: Unit Test - JdbcResourceTest (rewritten for the shell: defaults, replay, row count, connector validation, missing plugin, password masking), GrantResourcePrivilegeCommandTest, RevokeResourcePrivilegeCommandTest, CreateResourceCommandTest, PluginDrivenInsertExecutorTest. - Behavior changed: Yes (see release note) - Does this need documentation: Yes (deprecate CREATE RESOURCE type=jdbc) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… HikariCP dependency ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Part 4: keep the JDBC implementation from drifting back into fe-core. `build-support/check-fe-core-jdbc-free.sh` runs in fe-core's `validate` phase next to the metadata-funnel gate and fails the build on an import of `org.apache.doris.datasource.jdbc.*`, `com.zaxxer.hikari.*` or a `java.sql` connection/statement type in fe-core main sources; `httpv2/` (the FE's HTTP SQL gateway, JDBC to the FE itself) is the only exemption. Self-test in `build-support/tests/`. fe-core's HikariCP dependency goes with the deleted clients (the plugin bundles its own), and the fe-connector docs list the new gate. ### Release note None ### Check List (For Author) - Test: Unit Test - build-support/tests/run.sh (all gate self-tests), fe-core validate phase. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

morningman · 2026-09-15T23:45:34Z

run buildall

hello-stephen · 2026-09-16T01:25:50Z

TPC-H: Total hot run time: 16913 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5dbcecc3a909c6872fafa87dc8f1ee4aa243136a, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17576	3010	3003	3003
q2	2082	254	233	233
q3	10237	867	513	513
q4	4671	259	202	202
q5	7670	574	391	391
q6	140	117	97	97
q7	524	502	394	394
q8	9258	886	866	866
q9	3735	2424	2459	2424
q10	6522	864	720	720
q11	407	196	180	180
q12	625	259	193	193
q13	18120	1562	1159	1159
q14	161	149	142	142
q15	q16	436	395	373	373
q17	1273	909	822	822
q18	3143	2318	2301	2301
q19	1264	860	726	726
q20	379	281	196	196
q21	5666	1748	1885	1748
q22	333	270	230	230
Total cold run time: 94222 ms
Total hot run time: 16913 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3357	3298	3294	3294
q2	503	394	383	383
q3	2334	2334	2203	2203
q4	1220	1199	894	894
q5	2254	2189	2166	2166
q6	162	120	86	86
q7	1046	938	865	865
q8	1592	1410	1398	1398
q9	3221	3213	3204	3204
q10	1879	1829	1696	1696
q11	355	276	258	258
q12	457	431	347	347
q13	1496	1533	1166	1166
q14	170	182	168	168
q15	q16	401	403	362	362
q17	3704	3385	3376	3376
q18	4955	4580	5042	4580
q19	949	839	872	839
q20	1027	1002	862	862
q21	3936	3184	3218	3184
q22	395	348	307	307
Total cold run time: 35413 ms
Total hot run time: 31638 ms

hello-stephen · 2026-09-16T01:32:27Z

TPC-DS: Total hot run time: 83633 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5dbcecc3a909c6872fafa87dc8f1ee4aa243136a, data reload: false

query5	4254	422	351	351
query6	395	137	126	126
query7	4920	414	238	238
query8	286	127	123	123
query9	8691	2926	2940	2926
query10	398	226	186	186
query11	5409	1067	923	923
query12	117	73	74	73
query13	1186	437	324	324
query14	6103	2329	2228	2228
query14_1	2135	2113	2117	2113
query15	173	130	119	119
query16	914	388	365	365
query17	804	469	365	365
query18	2331	340	242	242
query19	175	144	113	113
query20	89	73	82	73
query21	206	105	90	90
query22	5653	5510	5547	5510
query23	7031	6493	6127	6127
query23_1	6277	6272	6268	6268
query24	7278	1108	777	777
query24_1	779	790	819	790
query25	434	300	266	266
query26	1234	239	134	134
query27	2777	402	259	259
query28	4679	1504	1523	1504
query29	932	453	359	359
query30	252	160	140	140
query31	817	426	342	342
query32	129	79	81	79
query33	463	233	192	192
query34	976	814	484	484
query35	414	418	357	357
query36	566	575	505	505
query37	122	85	71	71
query38	1038	873	846	846
query39	488	493	499	493
query39_1	469	479	482	479
query40	216	95	82	82
query41	61	58	56	56
query42	78	77	74	74
query43	248	256	217	217
query44	1001	535	561	535
query45	118	109	104	104
query46	781	834	531	531
query47	778	773	715	715
query48	307	301	239	239
query49	544	253	198	198
query50	782	274	208	208
query51	8261	7977	7841	7841
query52	73	71	59	59
query53	194	196	158	158
query54	228	169	143	143
query55	81	65	58	58
query56	191	160	161	160
query57	696	688	680	680
query58	184	167	166	166
query59	1269	1308	1158	1158
query60	275	189	154	154
query61	112	129	121	121
query62	344	207	183	183
query63	176	142	139	139
query64	2748	643	581	581
query65	1689	1658	1672	1658
query66	1764	259	202	202
query67	10034	10109	9938	9938
query68	3031	1217	731	731
query69	358	236	209	209
query70	657	598	613	598
query71	238	175	169	169
query72	2258	1738	1509	1509
query73	666	611	349	349
query74	1996	1248	1156	1156
query75	1218	1122	992	992
query76	2374	715	529	529
query77	268	264	212	212
query78	4331	3748	3343	3343
query79	2862	820	583	583
query80	1602	328	262	262
query81	508	160	140	140
query82	670	120	94	94
query83	297	217	193	193
query84	294	109	86	86
query85	772	349	275	275
query86	475	176	198	176
query87	1065	1017	939	939
query88	3172	2117	2105	2105
query89	280	200	179	179
query90	2140	137	136	136
query91	131	118	98	98
query92	101	71	65	65
query93	2735	1044	676	676
query94	646	249	226	226
query95	524	256	309	256
query96	817	590	293	293
query97	1096	1092	1057	1057
query98	181	140	140	140
query99	425	352	314	314
Total cold run time: 182258 ms
Total hot run time: 83633 ms

hello-stephen · 2026-09-16T01:36:44Z

ClickBench: Total hot run time: 14.78 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5dbcecc3a909c6872fafa87dc8f1ee4aa243136a, data reload: false

query1	0.01	0.00	0.01
query2	0.08	0.04	0.03
query3	0.25	0.11	0.11
query4	1.60	0.10	0.09
query5	0.18	0.16	0.15
query6	1.26	0.69	0.68
query7	0.03	0.01	0.00
query8	0.04	0.02	0.03
query9	0.29	0.21	0.22
query10	0.36	0.37	0.34
query11	0.17	0.11	0.12
query12	0.16	0.12	0.12
query13	0.31	0.31	0.31
query14	0.45	0.46	0.44
query15	0.38	0.35	0.36
query16	0.22	0.23	0.22
query17	0.68	0.74	0.67
query18	0.18	0.17	0.16
query19	1.25	1.11	1.18
query20	0.02	0.01	0.01
query21	15.44	0.15	0.12
query22	5.07	0.05	0.05
query23	16.17	0.25	0.10
query24	3.06	0.32	0.28
query25	0.11	0.04	0.03
query26	0.78	0.16	0.13
query27	0.05	0.03	0.03
query28	3.67	0.54	0.26
query29	12.44	3.22	2.57
query30	0.27	0.11	0.12
query31	2.76	0.37	0.18
query32	3.50	0.33	0.23
query33	1.41	1.46	1.44
query34	15.40	2.26	1.81
query35	1.81	1.78	1.79
query36	0.48	0.29	0.29
query37	0.07	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.03	0.03
query40	0.11	0.07	0.08
query41	0.07	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 90.73 s
Total hot run time: 14.78 s

hello-stephen · 2026-09-16T02:03:07Z

FE UT Coverage Report

Increment line coverage 79.86% (111/139) 🎉
Increment coverage report
Complete coverage report

morningman requested review from 924060929, CalvinKirs, englefly, morrySnow and starocean999 as code owners September 15, 2026 19:07

morningman and others added 4 commits September 16, 2026 07:39

morningman force-pushed the wt-remove-jdbc-fe-core branch from e60bc62 to 5dbcecc Compare September 15, 2026 23:45

Navigation Menu

Uh oh!

[refactor](fe) Remove the JDBC data-source implementation from fe-core; serve it from fe-connector-jdbc - #68037

[refactor](fe) Remove the JDBC data-source implementation from fe-core; serve it from fe-connector-jdbc#68037
morningman wants to merge 4 commits into
apache:masterfrom
morningman:wt-remove-jdbc-fe-core

morningman commented Sep 15, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Sep 15, 2026

Uh oh!

morningman commented Sep 15, 2026

Uh oh!

morningman commented Sep 15, 2026

Uh oh!

github-actions Bot commented Sep 15, 2026

Uh oh!

morningman commented Sep 15, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

morningman commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 15, 2026

Uh oh!

morningman commented Sep 15, 2026

Uh oh!

morningman commented Sep 15, 2026

Uh oh!

github-actions Bot commented Sep 15, 2026

Uh oh!

morningman commented Sep 15, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

FE UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

morningman commented Sep 15, 2026 •

edited

Loading