Skip to content

[fix](meta path) Preserve metadata access path type for map_entries - #68040

Merged
924060929 merged 1 commit into
apache:masterfrom
linrrzqqq:tranform-v-fix
Sep 17, 2026
Merged

924060929 merged 1 commit into
apache:masterfrom
linrrzqqq:tranform-v-fix

Conversation

@linrrzqqq

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

When transform_values operates on physical nested Map columns with nested-column pruning enabled:

  • MAP<..., ARRAY<...>> may fail because VALUES.OFFSET is incorrectly marked as a DATA access path.
  • MAP<..., STRUCT<...>> may crash the BE because VALUES.NULL is incorrectly treated as a Struct data field, causing all physical Struct child iterators to be pruned.

Root cause

AccessPathExpressionCollector.visitMapEntries creates a new CollectorContext while translating access paths from map_entries.

The new context did not inherit the original ColumnAccessPathType, so META paths such as OFFSET and NULL fell back to the default DATA type.

What is changed?

Propagate the original access-path type to the new context in visitMapEntries.

Added regression coverage for:

  • transform_values((k, v) -> size(v), map<int, array<int>>)
  • transform_values((k, v) -> v is null, map<int, struct<...>>)

The tests verify that VALUES.OFFSET and VALUES.NULL remain META access paths.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16774 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2bed8f26417c6b5d968ec4bf68b52b1bca489e27, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17590	3009	3021	3009
q2	2081	263	236	236
q3	10239	903	512	512
q4	4676	254	199	199
q5	7679	557	390	390
q6	136	113	94	94
q7	519	483	382	382
q8	9231	891	987	891
q9	3492	2418	2373	2373
q10	6536	871	714	714
q11	398	202	188	188
q12	610	260	203	203
q13	18129	1532	1155	1155
q14	155	151	137	137
q15	q16	430	392	377	377
q17	1350	907	794	794
q18	3036	2264	2247	2247
q19	1124	879	755	755
q20	365	278	194	194
q21	5279	1694	1856	1694
q22	324	277	230	230
Total cold run time: 93379 ms
Total hot run time: 16774 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3408	3307	3342	3307
q2	494	386	369	369
q3	2223	2352	2267	2267
q4	1191	1161	891	891
q5	2151	2112	2125	2112
q6	170	118	85	85
q7	1011	932	864	864
q8	1583	1394	1412	1394
q9	3136	3107	3120	3107
q10	1853	1809	1626	1626
q11	358	270	254	254
q12	450	431	347	347
q13	1479	1518	1150	1150
q14	173	166	169	166
q15	q16	404	403	363	363
q17	3646	3350	3168	3168
q18	4792	4416	4769	4416
q19	876	805	919	805
q20	1031	980	824	824
q21	3925	3132	3282	3132
q22	397	356	324	324
Total cold run time: 34751 ms
Total hot run time: 30971 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82124 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2bed8f26417c6b5d968ec4bf68b52b1bca489e27, data reload: false

query5	4263	412	351	351
query6	378	150	123	123
query7	4939	423	236	236
query8	293	126	122	122
query9	8671	2918	2888	2888
query10	394	225	190	190
query11	5374	1040	924	924
query12	118	70	76	70
query13	1197	430	311	311
query14	6181	2204	2112	2112
query14_1	1976	1963	1980	1963
query15	173	116	116	116
query16	937	374	353	353
query17	806	449	357	357
query18	2329	334	228	228
query19	161	137	108	108
query20	73	72	68	68
query21	201	102	92	92
query22	5349	5283	5304	5283
query23	6753	6214	6096	6096
query23_1	6220	6068	6185	6068
query24	7308	1110	769	769
query24_1	774	777	777	777
query25	422	303	265	265
query26	1225	245	125	125
query27	2787	418	252	252
query28	4698	1530	1510	1510
query29	931	438	355	355
query30	251	162	139	139
query31	830	399	332	332
query32	126	79	82	79
query33	471	229	181	181
query34	984	846	465	465
query35	407	407	350	350
query36	574	573	529	529
query37	121	82	72	72
query38	1008	847	831	831
query39	501	486	465	465
query39_1	485	438	489	438
query40	203	94	84	84
query41	59	56	57	56
query42	79	73	73	73
query43	238	243	216	216
query44	994	539	560	539
query45	123	109	101	101
query46	794	855	540	540
query47	772	749	699	699
query48	312	326	218	218
query49	554	236	203	203
query50	742	262	197	197
query51	8266	8046	8275	8046
query52	67	67	65	65
query53	194	202	159	159
query54	217	153	159	153
query55	75	57	54	54
query56	184	192	218	192
query57	670	690	628	628
query58	220	150	178	150
query59	1225	1247	1107	1107
query60	256	210	189	189
query61	132	130	125	125
query62	340	208	177	177
query63	179	150	148	148
query64	2765	691	568	568
query65	1604	1629	1639	1629
query66	1881	259	223	223
query67	10024	9632	9833	9632
query68	3041	1187	666	666
query69	350	225	205	205
query70	672	607	613	607
query71	256	173	163	163
query72	2451	1737	1551	1551
query73	616	621	323	323
query74	2005	1233	1128	1128
query75	1197	1111	955	955
query76	2401	736	528	528
query77	260	255	201	201
query78	3845	3733	3176	3176
query79	2175	851	600	600
query80	1629	323	268	268
query81	497	155	132	132
query82	649	124	101	101
query83	285	212	191	191
query84	296	113	87	87
query85	783	341	273	273
query86	399	173	182	173
query87	1027	984	896	896
query88	2792	2122	2129	2122
query89	280	200	180	180
query90	2041	130	135	130
query91	133	119	97	97
query92	85	72	73	72
query93	1517	1114	697	697
query94	629	259	228	228
query95	525	251	294	251
query96	821	615	291	291
query97	1086	1033	991	991
query98	173	132	141	132
query99	417	347	315	315
Total cold run time: 178598 ms
Total hot run time: 82124 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.67 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2bed8f26417c6b5d968ec4bf68b52b1bca489e27, data reload: false

query1	0.00	0.01	0.00
query2	0.08	0.04	0.04
query3	0.25	0.12	0.11
query4	1.63	0.10	0.10
query5	0.18	0.16	0.16
query6	1.25	0.70	0.71
query7	0.03	0.01	0.00
query8	0.04	0.03	0.02
query9	0.29	0.23	0.22
query10	0.34	0.34	0.33
query11	0.16	0.11	0.12
query12	0.16	0.12	0.13
query13	0.30	0.31	0.32
query14	0.46	0.46	0.47
query15	0.37	0.36	0.34
query16	0.21	0.22	0.23
query17	0.69	0.72	0.71
query18	0.17	0.16	0.16
query19	1.14	1.20	1.10
query20	0.02	0.01	0.01
query21	15.44	0.18	0.13
query22	5.06	0.05	0.04
query23	16.18	0.26	0.11
query24	2.98	0.32	0.23
query25	0.11	0.04	0.03
query26	0.76	0.17	0.11
query27	0.04	0.03	0.03
query28	3.63	0.61	0.28
query29	12.41	3.23	2.60
query30	0.26	0.12	0.12
query31	2.75	0.36	0.17
query32	3.52	0.32	0.23
query33	1.37	1.44	1.37
query34	15.36	2.15	1.77
query35	1.73	1.74	1.72
query36	0.46	0.30	0.28
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.12	0.07	0.08
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 90.23 s
Total hot run time: 14.67 s

@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval opinion: no blocking or correctness issues found.

Checkpoint conclusions:

  • Goal and data correctness: the change preserves ColumnAccessPathType.META while translating the synthetic map_entries entry field to KEYS or VALUES. The lowered transform_values path now reaches FE pruning and BE OFFSET_ONLY or NULL_MAP_ONLY readers without treating OFFSET or NULL as data fields, while the required map keys remain a DATA path.
  • Scope and focus: the production fix is one focused context-state copy. No additional user-provided focus was supplied; the full PR was reviewed.
  • Tests: the FE tests require the exact META paths and reject the pre-fix same-spelling DATA paths. The regression cases enable nested pruning and cover populated, empty, and NULL maps, array offsets, and nullable Struct values with deterministic ordering. Coverage is present but was reviewed statically only; no local build or test was run under the review-runner contract.
  • Concurrency and lifecycle: no shared mutable state, thread interaction, lock, or nontrivial lifecycle change is introduced. CollectorContext is request-local, and all of its state is now preserved or deliberately translated.
  • Configuration and compatibility: no production configuration, persistence format, function symbol, or wire schema changes. Existing typed Thrift access paths are reused, and non-OLAP readers retain their safe META-to-DATA fallback. Rolling-upgrade behavior is unchanged.
  • Parallel paths and conditions: all MapEntries-based map-lambda lowerings share the corrected translation. DATA versus META remains enum-discriminated, so real fields named NULL or OFFSET are not confused with metadata. Key-side NULL and OFFSET routing was also checked against nullable map-child encoding.
  • Error handling, observability, transactions, and writes: no new error boundary, transaction, persistence, or data-write path is added. Existing path validation and reader diagnostics remain applicable; no additional metric or log is needed for this local fix.
  • Performance and other risks: the added enum assignment is negligible and restores the intended metadata-only read modes. No additional performance, memory, or correctness issue was substantiated.

Review completion: complete after one converged round. Both normal full-review passes and the separate risk-focused pass returned NO_NEW_VALUABLE_FINDINGS; all initial risk items were independently resolved, with no inline comments required.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 1.35% (1/74) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow changed the title [fix](nereids) Preserve metadata access path type for map_entries [fix](meta path) Preserve metadata access path type for map_entries Sep 17, 2026
@924060929
924060929 merged commit a4c062c into apache:master Sep 17, 2026
44 checks passed
@linrrzqqq
linrrzqqq deleted the tranform-v-fix branch September 17, 2026 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants