Add cross-store ordering and recovery to tiered storage - #1074
Open
enigbe wants to merge 11 commits into
Open
Conversation
This commit adds `TierStore`, a tiered `KVStore` implementation that routes node persistence across three storage roles: - a primary store for durable, authoritative data - an optional backup store for a second durable copy of primary-backed data - an optional ephemeral store for rebuildable cached data such as the network graph and scorer TierStore routes ephemeral cache data to the ephemeral store when configured, while durable data remains primary and backup. Reads do not consult the backup store during normal operation. Unpaginated listings expose the logical contents of the primary and ephemeral tiers without consulting the backup store; paginated listings currently expose only the primary tier. For primary+backup writes and removals, this implementation treats the backup store as part of the persistence success path rather than as a best-effort background mirror. Earlier designs used asynchronous backup queueing to avoid blocking the primary path, but that weakens the durability contract by allowing primary success to be reported before backup persistence has completed. TierStore now issues primary and backup operations together and only returns success once both complete. This gives callers a clearer persistence guarantee when a backup store is configured: acknowledged primary+backup mutations have been attempted against both durable stores. The tradeoff is that dual-store operations are not atomic across stores, so an error may still be returned after one store has already been updated. Additionally, adds unit coverage for the current contract, including: - basic read/write/remove/list persistence - routing of ephemeral data away from the primary store - backup participation in the foreground success path for writes and removals Assisted-by: Amp (AI coding agent)
TierStore needs a single persistent ordering domain before it can provide correct pagination across primary and ephemeral stores because the wrapped stores' native ordering and pagination tokens are not comparable. In this commit, we create an internal SQLite index database automatically whenever ephemeral storage is configured and give it a persistent identity as well as an exclusive SQLite lock for TierStore's lifetime. This is a prefactor and only establishes the internal storage plumbing; listing behavior remains unchanged. Assisted-by: Amp (AI coding agent)
Primary and ephemeral stores maintain independent creation orders, so their paginated listings cannot be merged while preserving the PaginatedKVStore ordering contract. In this commit, the local TierStore index records logical key membership in one SQLite ordering domain. We initialize each namespace from the primary store on first use, preserving its existing paginated order, then maintain the index after writes and removals. We also route both list methods through the index so keys from either tier share a consistent creation order. Assisted-by: Amp (AI coding agent)
Value-store and index updates cannot be committed atomically, so failures can leave keys partially created or removed across storage tiers. In this commit we: - Persist creation and removal intent in the local index database before applying external changes. We recover pending operations before listing or modifying their namespace, including rechecking recovery under the per-key lock to prevent queued operations from racing with newly recorded journal entries. - Roll "creates" forward to all required stores, remove keys from the listing index before deleting their value copies, and retain failed operations for retry. Keep ordinary updates unjournaled and reject unindexed existing values as corruption. - Add deterministic failure and synchronization instrumentation to test recovery across interrupted writes, removals, primary/backup divergence, and queued operations. Assisted-by: Amp (AI coding agent)
The local index store supplies an opaque pagination token, but returning that token directly would allow callers to reuse it with another logical namespace or a different index database. In this commit we wrap the index token in a versioned TierStore token containing the logical namespace identity and persistent index database ID and we validate this context before passing the opaque token back to the index store, rejecting any malformed, unsupported, or mismatched tokens. Assisted-by: Amp (AI coding agent)
Nodes adopting tiered storage may already hold cache values in primary storage, while a missing index leaves existing ephemeral values without a recoverable position in the cross-store ordering. For this commit, we: - Prepare namespaces on reads as well as writes and listings so first access cannot bypass migration or journal recovery. - Rebuild missing indexes from primary ordering, discard ephemeral values whose ordering cannot be recovered, and move indexed cache values to ephemeral storage without changing their index positions. - Copy values before removing primary and backup copies, and persist a completion marker so interrupted reconciliation can safely resume without repeating it on every access. - Preserve namespace-specific cache routing for future namespace changes. Assisted-by: Amp (AI coding agent)
Preparing an entire namespace before every read allows one unrecoverable journal entry to block reads of unrelated keys. Cache migration has similar coupling because readiness is recorded for the whole namespace and all cache keys are reconciled together. In this commit, we: - Initialize the ordering index before reads, then hold the requested key's operation lock continuously while recovering its journal entry, reconciling its cache placement, and reading its value. - Track cache readiness per logical key and retain the original key identity as metadata to detect hash collisions. - Keep writes, removals, and listings on the existing namespace-wide preparation path for now (addressed in follow up). - Add coverage for unrelated pending operations, pending removals, independent cache migration, and lock re-entry deadlocks. Assisted-by: Amp (AI coding agent)
Preparing an entire namespace before every write or removal allows one unrecoverable journal entry to block mutations of unrelated keys sharing that namespace. In this commit, we: - Initialize the ordering index before mutations, then recover journal state and reconcile cache placement for only the requested key while holding its existing operation lock. - Reserve complete namespace recovery and cache reconciliation for listing operations. - Remove the obsolete journal-list snapshot gate and add coverage showing that a stuck key does not block unrelated writes or removals, while listings retain their deliberate fail-fast behavior. - Verify that touching one cache key no longer migrates another cache key in the same namespace. Assisted-by: Amp (AI coding agent)
TierStore remains internal until NodeBuilder provisions and installs its storage backends. Add builder options for local ephemeral and backup SQLite stores, wrap the configured primary store in TierStore, and pass the resulting store into node construction. When ephemeral storage is enabled, automatically create the persistent ordering index in the node's storage directory and require TierStore to own it exclusively. Update filesystem-backed tests and add integration coverage confirming that configured backup storage receives durable primary-backed data. Assisted-by: Amp (AI coding agent)
Add the opt-in storage-tier feature, gate tier-specific APIs and implementation, and preserve direct store usage when disabled. Document the feature, expose it on docs.rs, and run targeted tier storage tests in CI. AI-assisted: Developed with Amp.
Add a shared helper for creating temporary storage paths, loggers, and cleanup guards across TierStore tests.
|
I've assigned @tnull as a reviewer! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR is the second part of the tiered-storage work and is intended to be reviewed on top of #692. It's a first attempt at addressing #1039.
Motivation
#692 introduces
TierStore, routing durable data to primary and backup storage (if configured) and rebuildable cache data to ephemeral storage (if configured). Primary and ephemeral stores, however, maintain independent creation orders and issue pagination tokens that apply only to their own stores. Their listings therefore cannot be combined while preserving thePaginatedKVStoreordering contract.We considered modifying
PaginatedKVStoreto make independently paginated stores directly mergeable. Doing so would require exposing or standardizing ordering metadata currently encapsulated behind each store’s opaque pagination tokens. This could leak backend implementation details, constrain store implementations, and expand the trait contract for consumers that do not require cross-store pagination. Independent stores also have no natural shared creation sequence, so a coordinating ordering domain would still be required.TierStoreis currently the only "client" that combines independently ordered stores. This PR therefore takes a deliberately local first approach: maintain the shared ordering insideTierStore. Experience with this implementation can inform a broader trait-level design if other consumers eventually need the same capability.How it works
1. Cross-store ordering
When ephemeral storage is configured,
TierStorecreates a private SQLite index in the node’s storage directory. The index records logical key membership and creation order independently of the tier holding each value.On first access to an existing namespace,
TierStoreinitializes the index from the primary store’s paginated listing, preserving its established order. Subsequent creations and removals update index membership. Paginated and unpaginated listings can then be served from one ordering domain (the index) containing keys from both primary and ephemeral storage.TierStoreholds an exclusive lock on the index database for its lifetime, preventing concurrent instances from maintaining the same ordering state. Pagination tokens are bound to the index and logical namespace such that any malformed, unsupported, or mismatched tokens are rejected before reaching the index store.2. Recovering interrupted membership changes
Creating or removing a key changes both its backing stores (primary + ephemeral) and its index membership. These changes cannot be committed atomically when the stores are independent.
TierStoretherefore journals creation and removal intent in the index before changing external stores:Interrupted operations are rolled forward on later access, i.e.
TierStoreconsiders a primary-backed creation successful only after both primary and the configured backup have stored the value. If either write fails, the call returns an error, the journal entry remains, and the key is withheld from listings because its durability requirement (primary + backup) has not been met. Recovery replays the journaled value to both stores, bringing the failed copy into sync. Once both writes succeed,TierStoreadds the key to the listing index and clears the journal entry. For an interrupted removal, index membership is removed first; recovery finishes deleting any remaining primary and backup copies before clearing the journal.3. Migrating existing cache data
A node may enable ephemeral storage after cache values have already been written to primary storage. If an existing ephemeral store is reused after its corresponding index is lost or replaced, it may contain values whose original ordering cannot be reconstructed.
TierStorerebuilds missing index state from the authoritative primary ordering and discards ephemeral-only values whose positions cannot be recovered. Cache values recognized by the routing policy are copied to ephemeral storage before their primary and backup copies are removed, preserving their logical positions.Point operations (reads, writes, removals) reconcile only the requested key, while listings prepare the complete namespace. Per-key migration progress is persisted so interrupted work can resume safely.
4. Isolating recovery by key
Recovering an entire namespace before every point operation would allow one unrecoverable journal entry to block unrelated keys.
Reads, writes, and removals instead hold the requested key’s operation lock while recovering its journal state, reconciling cache placement, and performing the operation. Listings still prepare the complete namespace because they require a consistent view of every key.
5. Builder integration
This PR adds
NodeBuilderconfiguration for local backup and ephemeral SQLite stores. The configured primary store is wrapped inTierStore, and the private ordering index is created automatically when ephemeral storage is enabled.Tiered storage remains behind the opt-in
storage-tierfeature. Builds without it continue to use the configured store directly.Test coverage
Tests cover:
PR stack
TierStorebackup #1073: Backfill and resilver backup storageCo-authored with Amp.