fix(swim): fast-restart rejoin — persist incarnation, echo refutations, ping liveness - #243
Open
EnRaiha wants to merge 2 commits into
Open
fix(swim): fast-restart rejoin — persist incarnation, echo refutations, ping liveness#243EnRaiha wants to merge 2 commits into
EnRaiha wants to merge 2 commits into
Conversation
…s, ping liveness Fixes the SWIM fast-restart rejoin stick (epic NodeDB-Lab#165): a node that crashes while peers hold Dead(A, N) restarts at incarnation 0, announces Alive(0) exactly once, and can diverge from the cluster forever when the probabilistic refutation round-trip fails. - F4 (primary): persist the local incarnation in the catalog (KEY_SWIM_INCARNATION, u64 LE, same pattern as cluster_epoch). Bootstrap resumes at persisted + 1, so the first announcement already dominates any lingering Dead(A, N) rumour. Every self-refutation bump is persisted (fire-and-forget, never stalls the probe loop). New IncarnationStore trait: catalog-backed in production, in-memory in tests. Wired through SwimSubsystemConfig, spawn_with_subscribers, and register_default_subsystems (load + bump on start). - F1: ingest_piggyback returns refutations — whenever an update is refuted, the stored (newer) view is echoed deterministically on the reply piggyback (Ack + forwarded PingReq), instead of relying on probabilistic gossip fanout. - F2: a ping IS liveness evidence (SWIM paper §3) — the sender's Alive claim is applied before acking, clearing Dead/Suspect views; self-advertisement on the ack path is rate-limited to 500ms so a heavily-pinged node cannot flood its dissemination queue. - F3: a fresh Alive apply cancels the pending suspicion timer, so a stale expiry can no longer promote a just-seen-live node to Dead. Tests: 6 new regression tests pinning each fix + the combined deterministic recovery loop (A holds Dead(B,5) → echo → self-refute → converge Alive), 129/129 swim, 1015 cluster lib, clippy -D 0, maya-gate L1 clean. Refactor note (deferred): IncarnationTracker consolidation (P2-GLM53-REVIEW-RESOLUTION.md Improvement 1) — incarnation state is still split between detector/local_incarnation and the store; a single-owner tracker with atomic persist would remove the fire-and-forget save. MemberState::Left has no production sender yet; when graceful-leave lands, restart must resume above TerminalLeft or clean-shutdown restarts will stick forever.
21 tasks
EnRaiha
added a commit
to EnRaiha/nodedb
that referenced
this pull request
Aug 24, 2026
Drill/fix-plan/GLM-resolution docs (commit dd0c316) written pre-fix are now stale. Add resolution banners pointing to the implemented fixes: SWIM 8886846 (PR NodeDB-Lab#243), wire window e60a853 (PR NodeDB-Lab#244), epoch fence 4f92959 (PR NodeDB-Lab#245), lease GC 16d9916 (PR NodeDB-Lab#246). Full verification + refactor code in P2-REPORT.md.
load_swim_incarnation errors on corrupted metadata (len != 8) instead of silent Ok(None); register_default_subsystems propagates read errors instead of unwrap_or(None) — bootstrap no longer hides catalog IO/corruption problems at startup.
EnRaiha
added a commit
to EnRaiha/nodedb
that referenced
this pull request
Aug 24, 2026
/NodeDB-Lab#246 - catalog: load_swim_incarnation returns an ERROR on corrupted metadata (len != 8) instead of silently Ok(None); register_default_subsystems propagates the read error instead of unwrap_or(None) — bootstrap no longer hides catalog IO/corruption problems (PR NodeDB-Lab#243 critical items). - server: handle_stream drops StalePeerEpoch frames explicitly with a warn! log instead of ending the stream via generic ? — fenced frames are not transport failures; connection stays open for exempt traffic (PR NodeDB-Lab#245 critical item). - lease_gc: proposal failures logged at warn (operators can notice GC not happening); drain_propose doc comment translated to English (PR NodeDB-Lab#246 review items).
Contributor
Author
|
Addressed the two critical review items in 1876085:
Thanks for the review! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the SWIM fast-restart rejoin stick (epic #165, Medium #9): a node that crashes while peers hold Dead(A, N) restarts at incarnation 0, announces Alive(0) exactly once, and can diverge forever when the probabilistic refutation round-trip fails.
6 regression tests incl. deterministic recovery loop (A holds Dead(B,5) → echo → self-refute → converge Alive). Verification: swim 129/129, cluster lib 1023, clippy -D warnings 0, maya-gate L1 clean.
Closes #165 item: Medium — SWIM fast-restart rejoin.