branch-4.1: [fix](cloud) refresh tablet meta of continuously ingested tablets #67972 - #68044
Merged
yiguolei merged 1 commit intoSep 16, 2026
Merged
Conversation
…ache#67972) Problem Summary: `CloudTabletMgr::sync_tablets()` selected the tablets it would work on by `last_sync_time_s`, then did two things to each one: `sync_meta()` followed by `sync_rowsets()`. But that clock only tracks how stale a tablet's **rowsets** are. It is advanced inside `sync_tablet_rowsets()`, and only when that actually issues its RPC. A tablet under continuous ingest therefore keeps it permanently fresh, never falls below the staleness bound, and never has `sync_meta()` called on it at all. Its `TabletMeta` then stays at whatever it was built with for the lifetime of the object, so properties that only arrive through the tablet meta are never picked up. `ttl_seconds` is the one that shows: `CloudTablet::sync_meta()` is the only thing that refreshes it, and it feeds the file cache expiration computed on the write path (`CloudRowsetBuilder`, compaction output), the read path (`TabletReader`, `OlapScanner`) and warm-up. An `ALTER TABLE ... SET ("file_cache_ttl_seconds" = ...)` on a table under load therefore has no effect on those blocks, which keep landing in the wrong queue with an expiration derived from the stale value. Fixed tablet metadata never being refreshed for tablets under continuous ingest, which left properties such as `file_cache_ttl_seconds` stale on those tablets after an ALTER. **1. Give meta staleness its own clock.** `last_sync_tablet_meta_time_s` is advanced only by `sync_meta()`. **2. Rename `last_sync_time_s` to `last_sync_rowsets_time_s`** (separate commit, mechanical). The old name says "sync" while the clock only ever tracked rowsets, and reading it as "when this tablet was last synced" is exactly the mistake that let meta work be gated on it. With a second clock alongside it the old name would be actively misleading. This also keeps the field named the same as on branch-3.1. **3. `sync_tablets()` decides per tablet which of the two RPCs it is due for**, rather than sorting tablets into a single bucket: - rowsets stale -> sync rowsets, and the meta too. Pulling rowsets implies pulling the meta, which is the relationship the single pass had: the rowsets are only as trustworthy as the meta they belong to. - rowsets fresh but meta stale -> sync the meta only, one RPC instead of the two a full sync costs. This is the case that used to be skipped entirely. - both fresh -> skip. Work is still ordered by the older of the two clocks, so a mid-run stop has already served the tablets that waited longest. New bvars `sync_tablets_meta_num` and `sync_tablets_rowsets_num` split what `num_sync` used to lump together, and the finish log reports both. (cherry picked from commit fe92389)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Sep 16, 2026
Contributor
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pick #67972