Skip to content

Reload derived-column transforms without freezing or stale star-trees - #18977

Open
Vamsi-klu wants to merge 13 commits into
apache:masterfrom
Vamsi-klu:fix-derived-column-reload-transform
Open

Vamsi-klu wants to merge 13 commits into
apache:masterfrom
Vamsi-klu:fix-derived-column-reload-transform

Conversation

@Vamsi-klu

@Vamsi-klu Vamsi-klu commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

PR flow

Persist transform provenance to decide BACKFILL vs UPDATE; trigger star-tree rebuild on UPDATE; minion ignores transform actions.

flowchart TD
  N0["Persist transform function provenance #40;F3#41;"]:::stAdded
  N1["Compute transform action #40;BACKFILL#47;UPDATE#41; #40;F6#41;"]:::stAdded
  N2["Get columns with pending transform value changes #40;F6#41;"]:::stAdded
  N3["Capture columns with pending transform value changes #40;F5#41;"]:::stAdded
  N4["Determine if star#45;tree needs rebuild due to transform value change #40;F5#41;"]:::stAdded
  N5["Force rebuild of existing star#45;trees on transform value change #40;F10#41;"]:::stAdded
  N6["Ignore transform BACKFILL#47;UPDATE for minion record#45;replay #40;F2#41;"]:::stAdded
  N7["Skip transform actions in minion refresh #40;F1#41;"]:::stAdded
  N0 -->|"persisted function used to decide action"| N1
  N1 -->|"UPDATE action leads to value#45;change columns"| N2
  N2 -->|"SegmentPreProcessor captures changed columns"| N3
  N3 -->|"used to decide star#45;tree rebuild"| N4
  N4 -->|"force rebuild if star#45;tree needs update"| N5
  N7 -->|"call needPreprocess with false flag"| N6
  N6 -->|"flag passed to needProcessStarTrees"| N4
  classDef stAdded fill:#dafbe1,stroke:#1a7f37,color:#1f2328,stroke-width:2px
  classDef stModified fill:#fff8c5,stroke:#9a6700,color:#1f2328,stroke-width:2px
  classDef stRemoved fill:#ffebe9,stroke:#cf222e,color:#1f2328,stroke-width:2px
  classDef stUnchanged fill:#f6f8fa,stroke:#656d76,color:#1f2328,stroke-width:1px
Loading

AI-generated · Green: added · Yellow: modified · Red: removed · Gray: existing

Partial evidence: 0 file patches omitted; 1 truncated.

Diff evidence
  • F1: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/refreshsegment/RefreshSegmentTaskExecutor.java — before · after
  • F2: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java — before · after
  • F3: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java — before · after
  • F5: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java — before · after
  • F6: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java — before · after
  • F10: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/startree/v2/builder/MultipleTreesBuilder.java — before · after
  • Regenerate PR flow

Fixes #9989

What changed

Persist derived-column transform provenance in segment column metadata and use it on reload to decide BACKFILL vs UPDATE.

  • transformFunction is written only when stored values were generated from that expression.
  • transformFunctionBackfilled stores the compatibility expression as a string when values were not regenerated (legacy metadata-only BACKFILL). Change detection uses firstNonNull(transformFunction, transformFunctionBackfilled).
  • One-release reader still accepts the earlier boolean "true" marker and treats the sibling transformFunction as backfilled.
  • Expressions are read with getProperty(), not getString(), so Groovy ${...} interpolation cannot rewrite the expression against other metadata keys.

Server SegmentPreProcessor + DefaultColumnHandler is the apply path:

  • Transform BACKFILL/UPDATE does not drive minion RefreshSegment record-replay rebuild. That path copies existing values, stamps autoGenerated=false, and freezes the column.
  • ImmutableSegmentLoader.needPreprocess(..., includeTransformFunctionActions=false) is the minion signal. We do not set ExpressionTransformer._overwriteExistingValues=true.
  • There is no "new segments only" flag. Reload applies to auto-generated derived columns; ingest-time columns (autoGenerated=false) are left alone.

Star-trees: UPDATE_*_TRANSFORM_FUNCTION (VALUES_CHANGED) on a star-tree dimension or aggregation input forces a rebuild. MultipleTreesBuilder skips reuse when configs match but values changed. BACKFILL does not rebuild star-trees.

Fail closed if replaceSpecialCharacterInPropertyValue cannot persist the expression (UTF-16 surrogates). Silent drop would loop BACKFILL forever.

transformFunctionDefaulted is not added in this PR (no retry path for missing-arg default materialization).

Why it matters

Derived columns come from transform expressions. If table config changes the expression, reloaded auto-generated columns must pick up the new formula. Without stored provenance, Pinot can see that the column exists but not whether it matches the current transform.

Compatibility

  • Older metadata without these keys still loads (getTransformFunction() is null).
  • First reload after upgrade is BACKFILL (metadata only). The next real expression change regenerates values.
  • Mixed cluster: old servers ignore the new keys and never regenerate.
  • Chained derived columns are not invalidated when a parent rebuilds unless the child's expression string also changed.
  • RefreshSegment may still rebuild for structural reasons (ADD column, type change). That remaining record-replay path can still freeze auto-generated derived columns if it runs.

Testing

./mvnw -pl pinot-segment-local -am \
  -Dtest=DerivedColumnStarTreeReloadTest,DefaultColumnHandlerTest,ColumnMetadataImplTest,IngestionConfigUtilsTest,StarTreeBuilderUtilsTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

./mvnw -pl pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks -am \
  -Dtest=RefreshSegmentTaskExecutorTest,RefreshSegmentTaskGeneratorTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

Covers metadata round-trip, distinct backfill field, fail-closed persist, minion skip of transform-only work, star-tree rebuild on UPDATE, and no star-tree rebuild on BACKFILL.

@Vamsi-klu
Vamsi-klu marked this pull request as ready for review July 11, 2026 23:38
@Vamsi-klu

Copy link
Copy Markdown
Contributor Author

Review request: @Jackie-Jiang

GitHub did not allow this account/integration to add requested reviewers through the reviewer section, so I am tagging the relevant reviewer here for visibility.

This PR fixes #9989 by persisting derived-column transform-function metadata into segment column metadata and using that metadata during reload to regenerate auto-generated derived columns when the configured transform changes. Review would be especially useful because the implementation follows the issue guidance to persist the transform function and compare it against the table config during reload.


Drafted-by: Codex (GPT-5); human-reviewed by Vamsi-klu before posting

@Jackie-Jiang Jackie-Jiang added ingestion Related to data ingestion pipeline feature New functionality labels Aug 5, 2026
@Jackie-Jiang
Jackie-Jiang requested a lite review from Copilot August 5, 2026 01:37
@Jackie-Jiang

Copy link
Copy Markdown
Contributor

@J-HowHuang Could you please help review this PR?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes derived-column staleness on segment reload by persisting the configured transform expression into per-column segment metadata and using it to decide whether auto-generated derived columns must be regenerated when transforms change.

Changes:

  • Add transformFunction as an optional persisted column-metadata property and expose it via ColumnMetadata#getTransformFunction().
  • Persist transform-function provenance during segment generation and during default-column/derived-column materialization (including default-value fallback when transform inputs are missing).
  • Update default-column reload decisioning to treat transform-function changes (or missing legacy metadata) as a regeneration trigger, and add tests covering round-trip/back-compat and reload behaviors.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImplTest.java Adds tests for transform-function metadata round-trip, config read, and missing-property back-compat.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/V1Constants.java Introduces the persisted metadata key transformFunction.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/EmptyColumnMetadata.java Implements getTransformFunction() for empty-column metadata.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java Adds storage, (de)serialization, and equality support for transform-function metadata.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/ColumnMetadata.java Adds the public API surface (getTransformFunction() default method).
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/DefaultColumnHandlerTest.java Adds tests validating persistence and reload regeneration behavior for derived columns when transforms change or metadata is missing.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/map/SimpleColumnMetadata.java Implements getTransformFunction() for simple in-memory metadata.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java Adds transform-function change detection into reload action mapping and persists transform metadata on default-value fallback.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java Persists transform-function metadata during segment generation via extended column-metadata writing.

@Vamsi-klu
Vamsi-klu force-pushed the fix-derived-column-reload-transform branch from c2382b2 to 6cbca04 Compare August 6, 2026 03:33
@codecov-commenter

codecov-commenter commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.84466% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.93%. Comparing base (411bdc8) to head (1650595).
⚠️ Report is 11 commits behind head on master.

Files with missing lines Patch % Lines
...loader/defaultcolumn/BaseDefaultColumnHandler.java 67.64% 16 Missing and 6 partials ⚠️
...local/segment/creator/impl/BaseSegmentCreator.java 85.00% 0 Missing and 3 partials ⚠️
.../local/segment/index/map/SimpleColumnMetadata.java 0.00% 1 Missing ⚠️
...a/org/apache/pinot/segment/spi/ColumnMetadata.java 0.00% 1 Missing ⚠️
...segment/spi/index/metadata/ColumnMetadataImpl.java 91.66% 0 Missing and 1 partial ⚠️
...egment/spi/index/metadata/EmptyColumnMetadata.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             master   #18977   +/-   ##
=========================================
  Coverage     66.93%   66.93%           
  Complexity     1423     1423           
=========================================
  Files          3452     3452           
  Lines        218447   218508   +61     
  Branches      34718    34737   +19     
=========================================
+ Hits         146221   146267   +46     
- Misses        60547    60563   +16     
+ Partials      11679    11678    -1     
Flag Coverage Δ
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-25 66.93% <71.84%> (+<0.01%) ⬆️
lane-a 100.00% <ø> (ø)
lane-b 0.00% <ø> (ø)
temurin 66.93% <71.84%> (+<0.01%) ⬆️
unittests 66.93% <71.84%> (+<0.01%) ⬆️
unittests1 57.69% <35.92%> (+0.01%) ⬆️
unittests2 39.02% <61.16%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ments

Segments created before the transform function was tracked in the segment
metadata report null for it. Treating that as a change forced a one-time
rebuild of every auto-generated derived column on the first reload after an
upgrade, and for those segments there is no way to tell whether the values are
actually stale.

Reloading such a segment is now a no-op for the column values: the configured
transform function is recorded into the column metadata through the regular
reload persistence path (new metadata-only BACKFILL_TRANSFORM_FUNCTION action),
so the next actual transform function change is detected and rebuilt exactly
once. A transform function that is present in the metadata and differs from the
config (including being removed from the config) still triggers the rebuild.

Also annotate BaseDefaultColumnHandler#getTransformFunctionForColumn with
@nullable, matching the BaseSegmentCreator copy.

@J-HowHuang J-HowHuang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall.

So the new behavior will be, once we modify (remove/update) the transform function from the table config, all the existing segment will be affected retrospectively once reloaded.
If anyone wants to only apply the change to the newly generated segments, is there a way to do it? Is it worth it to also leave a door for this use case? cc @Jackie-Jiang

@Jackie-Jiang
Jackie-Jiang requested a balanced review from Copilot August 13, 2026 20:45
@Jackie-Jiang Jackie-Jiang added release-notes Referenced by PRs that need attention when compiling the next release notes backward-incompat Introduces a backward-incompatible API or behavior change labels Aug 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (3)

pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java:349

  • This legacy path preserves stale derived values while recording the current expression as though those values were generated from it. For the issue scenario, a pre-upgrade segment with no stored expression and values from the old formula will become a no-op on every later reload with the current config, so the promised legacy regeneration never occurs. Treat null metadata plus a configured transform as a transform change and rebuild the column; update the legacy test to assert recomputed values rather than metadata-only backfill.
        // Segments created before the transform function was tracked in the metadata report null for it. Their values
        // cannot be told apart from up-to-date ones, so instead of regenerating them, record the configured transform
        // function in the metadata (values untouched) so that the NEXT transform function change is detected.
        // Tradeoff: a transform function change that lands in the very same reload as this backfill is not applied to
        // the existing values (which matches the behavior before the transform function was tracked at all); operators
        // who need those values regenerated can force it with one more change to the expression.
        if (!defaultColumnActionMap.containsKey(column) && columnMetadata.getTransformFunction() == null
            && getTransformFunctionForColumn(column) != null) {
          defaultColumnActionMap.put(column, DefaultColumnAction.BACKFILL_TRANSFORM_FUNCTION);

pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:805

  • This setter accepts null for backward-compatible metadata—the production parser calls it with the nullable result of extractTransformFunction()—but its parameter is non-null by default. Mark the parameter @Nullable so the new public builder contract matches its actual use and the built object's nullable field.
    public Builder setTransformFunction(String transformFunction) {

pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java:718

  • replaceSpecialCharacterInPropertyValue() explicitly returns null for values containing UTF-16 surrogate characters, but this branch silently omits the metadata. A valid transform containing a supplementary Unicode character in a string literal can therefore be evaluated while its provenance is never persisted, causing reload decisions to repeatedly treat the column as legacy and preventing reliable change detection. Either use a reversible encoding that supports the full expression or reject the transform with a contextual exception instead of silently dropping it.
      String validTransformFunction =
          CommonsConfigurationUtils.replaceSpecialCharacterInPropertyValue(transformFunction);
      if (validTransformFunction != null) {
        properties.setProperty(getKeyFor(column, TRANSFORM_FUNCTION), validTransformFunction);
      }

@Jackie-Jiang Jackie-Jiang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly good. Well done

}

/// Adds column metadata information to the properties configuration.
public static void addColumnMetadataInfo(PropertiesConfiguration properties, String column,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(minor) Remove the old method, and change the callers to adapt to the new API

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in ec240375. Callers now use the transformFunction overload, including OpenStruct and the column metadata tests.


@Nullable
@SuppressWarnings("deprecation")
private String getTransformFunctionForColumn(String column) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(minor) Instead of looping over the list multiple times (once per column), we can first build a map from column to transform function

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ec240375. BaseSegmentCreator builds the column-to-transform map once via IngestionConfigUtils.getTransformFunctionByColumn.


/// Returns the transform function expression used to generate the column, if persisted in the segment metadata.
@Nullable
default String getTransformFunction() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to provide default impl. This is an internal interface

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in ec240375. ColumnMetadata.getTransformFunction() is abstract now, with implementations on ColumnMetadataImpl, EmptyColumnMetadata, and SimpleColumnMetadata. Same for getTransformFunctionBackfilled().

_lengthOfLongestElement, _isAscii, _totalNumberOfEntries, _maxNumberOfMultiValues, _maxRowLengthInBytes,
_bitsPerElement, _partitionFunction, _partitions, _autoGenerated, _parentColumn, _sparseKeys,
_bitsPerElement, _partitionFunction, _partitions, _autoGenerated, _transformFunction, _parentColumn,
_sparseKeys,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(minor) Reformat

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reformatted when the backfilled field was added to the constructor and builder.


@Nullable
private static String extractTransformFunction(String column, PropertiesConfiguration config) {
Object transformFunctionProperty = config.getProperty(Column.getKeyFor(column, Column.TRANSFORM_FUNCTION));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use getString()?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried getString() first. Commons Configuration interpolates ${...}, and a Groovy transform can contain that. I added a test where ${x} was rewritten to another metadata key's value.

So the expression itself is still read with getProperty(), same reason min/max avoid getString(). The boolean backfill marker is read with getString() because it is just "true".


@Nullable
@SuppressWarnings("deprecation")
private String getTransformFunctionForColumn(String column) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here. We can create a map from column to transform functions. Suggest extracting it into a util method to be shared

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ec240375. Both BaseDefaultColumnHandler and BaseSegmentCreator use IngestionConfigUtils.getTransformFunctionByColumn so the list is scanned once.

// Metadata-only action: record the configured transform function for an auto-generated column created before the
// transform function was tracked in the segment metadata. No values are regenerated, and it is handled entirely
// within updateDefaultColumns(), i.e. it is never dispatched to updateDefaultColumn().
BACKFILL_TRANSFORM_FUNCTION;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When backfilling the transform function, we might want to fill a different field so that we can differentiate the case of backward compatible handling vs actual backfilled value. Currently there is no way to differentiate them

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. BACKFILL now writes the configured expression only to transformFunctionBackfilled. transformFunction is reserved for expressions that actually produced the stored values.

Change detection uses the stored transform if present, otherwise the backfilled one, so a later config change still regenerates exactly once.

Readers also accept the earlier boolean "true" marker from the Aug 15 commit on this branch. In that case the companion transformFunction value is treated as the compat expression, not a real stored transform.

Tests cover backfill vs stored so they cannot be confused, the boolean marker still detecting a later change, and a real transform change still rebuilding values.

cursoragent and others added 3 commits August 15, 2026 05:04
Remove the unused 8-arg addColumnMetadataInfo overload, share a
column-to-transform map via IngestionConfigUtils, drop the default
ColumnMetadata.getTransformFunction() impl, and persist
transformFunctionBackfilled so legacy metadata backfill is distinct
from a transform that actually produced the stored values.

Co-authored-by: deepinsight coder <Vamsi-klu@users.noreply.github.com>
Co-authored-by: deepinsight coder <Vamsi-klu@users.noreply.github.com>
Co-authored-by: deepinsight coder <Vamsi-klu@users.noreply.github.com>
cursoragent and others added 3 commits September 12, 2026 02:05
Keep the derived-column transformFunction field alongside master's lazy
index-size list, and keep both test suites.

Co-authored-by: deepinsight coder <Vamsi-klu@users.noreply.github.com>
Write the compatibility expression only to transformFunctionBackfilled so
a backward-compat reload cannot be mistaken for a transform that actually
produced the stored values. Keep accepting the earlier boolean "true"
marker for one release. Read the field with getString().

Co-authored-by: deepinsight coder <Vamsi-klu@users.noreply.github.com>
getString() rewrites Groovy ${...} against other metadata keys. Keep
getString() for the boolean backfill marker and read expressions with
getProperty(), matching min/max. Also wrap a long test line.

Co-authored-by: deepinsight coder <Vamsi-klu@users.noreply.github.com>
@Vamsi-klu

Copy link
Copy Markdown
Contributor Author

@J-HowHuang Yes, once the transform is added, updated, or removed, reload applies that to existing auto-generated derived columns. Legacy segments that have no stored transform are a metadata-only backfill on first reload, then the next real change rebuilds values.

There is no opt-out in this PR for keeping the old transform on old segments. If we want that door, I would rather do it as a follow-up than add a public flag here.

cursoragent and others added 2 commits September 13, 2026 04:48
Transform BACKFILL/UPDATE must not drive a record-replay rebuild. That path
copies existing values, stamps autoGenerated=false, and freezes the column.
Server SegmentPreProcessor plus DefaultColumnHandler is the apply path.

Force a star-tree rebuild when a star-tree column gets
UPDATE_*_TRANSFORM_FUNCTION. BACKFILL does not rebuild star-trees.

Fail closed if transform metadata cannot be persisted (UTF-16 surrogates),
so BACKFILL does not loop forever.

Co-authored-by: deepinsight coder <Vamsi-klu@users.noreply.github.com>
MultipleTreesBuilder reused existing trees when configs matched, so
UPDATE_*_TRANSFORM_FUNCTION left stale aggregates. Skip reuse when
processStarTrees sees VALUES_CHANGED. BACKFILL still does not rebuild.

Read star-tree SUM as double from the root aggregated doc id.

Co-authored-by: deepinsight coder <Vamsi-klu@users.noreply.github.com>
@Vamsi-klu Vamsi-klu changed the title Regenerate derived columns when transforms change Reload derived-column transforms without freezing or stale star-trees Sep 13, 2026
Empty commit so pull_request workflows run without action_required.
No code change.
@Vamsi-klu

Copy link
Copy Markdown
Contributor Author

@Jackie-Jiang @J-HowHuang Ready for another look on HEAD d858652.

Since the Aug 13 notes: BACKFILL writes only transformFunctionBackfilled, expressions stay on getProperty(), minion RefreshSegment skips transform BACKFILL/UPDATE so it cannot freeze the column, star-trees rebuild on VALUES_CHANGED, and persist fails closed on surrogates. Apache CI is green on this head.

J-How's earlier +1 was on 16505955 and does not cover this head. Formal reviewer requests 403 from this account, so tagging here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backward-incompat Introduces a backward-incompatible API or behavior change feature New functionality ingestion Related to data ingestion pipeline release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Derivated column not reloaded when formula changes

6 participants