[server] Incrementally clean historical KV state - #4199
Draft
luoyuxia wants to merge 8 commits into
Draft
Conversation
Route writes for expired partitions through internal historical targets while preserving original partition metadata across PUT_KV and PRODUCE_LOG. Tier historical KV and log records back to their original Paimon partitions, fail writes to confirmed missing targets, and safely clean fully tiered historical KV overlays with leader-epoch and offset guards. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 1469/1469 AI-Contributed/UT: 2215/2215
Simplify historical write routing, request handling, and Paimon tiering integration while removing redundant tests. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 446/446 AI-Contributed/UT: 576/576
Bind historical write state to the active KV overlay and defer idle cleanup until its deadline after lake progress catches up. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 234/234 AI-Contributed/UT: 129/129
Return a retriable KV storage error while local historical KV state is being initialized or rebuilt. Clarify cleanup-state naming and terminology, and extend the Paimon integration test through post-recovery writes and restarted tiering. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 244/244 AI-Contributed/UT: 239/239
Refresh registered Paimon partition-bucket file sets when the required lake snapshot changes, allowing unchanged local lookup files to remain reusable. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 161/215 AI-Contributed/UT: 95/125
Exclude the plugin-facing LakeTableLookuper interface from the aggregate per-class coverage rule. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 3/3 AI-Contributed/UT: 0/0
Remove the coupled historical KV cleanup mechanism, defer lake lookup refresh I/O until lookup initialization, and simplify historical write version and routing checks. Reroute missing original targets when no request remains in flight, abort only the affected target when handoff is unsafe, and document the remaining retirement ambiguity. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5.6-sol AI-Contributed/Feature: 56/56 AI-Contributed/UT: 2/2
luoyuxia
force-pushed
the
fip28-incremental-historical-kv-cleanup
branch
from
September 2, 2026 08:52
a6bd609 to
ac5dde1
Compare
Preserve historical write lookup results so compaction between lookup and apply cannot change merge, delete, or changelog semantics. Tag historical values and tombstones with their producing WAL offsets, advance an exclusive cleanup offset after publishing the covering lake snapshot, and reuse the RocksDB compaction filter for best-effort cleanup. Perform lake lookups between the local probe and apply lock sections, then expose the complete request-scoped result through an in-memory historical value lookup. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 480/480 AI-Contributed/UT: 102/102
luoyuxia
force-pushed
the
fip28-incremental-historical-kv-cleanup
branch
from
September 2, 2026 11:42
ac5dde1 to
f691450
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: #4159.
Depends on #4120. This is a stacked PR; until #4120 merges, review commit
f6914503aonly.Historical writes retain local KV values and tombstones until their WAL offsets are covered by the lake snapshot. This change makes entries below the confirmed lake progress eligible for incremental reclamation, without waiting for the lake and local log end offsets to become equal or rebuilding the entire RocksDB instance.
Cleanup is best effort: the cleanup offset controls eligibility, while physical deletion occurs during normal RocksDB compactions. The implementation does not schedule an additional full-range compaction when the offset advances.
Brief change log
Tests
fluss-serverunit tests:KvValueLayoutTest,RowTtlCompactionFilterTest,HistoricalKvCompactionFilterTest,KvTabletTest, andHistoricalPartitionManagerTest(56 tests passed)../mvnw -pl fluss-server spotless:checkgit diff --check./mvnw verifywas attempted. It stopped influss-serverafter 1,303 tests because the local disk was 95% full: fiveReplicaFetcherThreadTestcases hit the 85% RocksDB disk guard, and two unrelatedReplicaTestlocal-log metric assertions failed. The historical-KV and compaction-filter tests passed in that run.API and Format
No public API changes.
The internal RocksDB value format for historical KV state now carries an 8-byte WAL-offset tag; a historical tombstone consists only of that tag. Lake values remain in the plain value layout.
Documentation
No user-facing documentation changes. The design and trade-offs are described in #4159 and this PR.
Generative AI disclosure