fix(cluster): enforce cluster-epoch fence at the decode chokepoint - #245
Open
EnRaiha wants to merge 2 commits into
Open
fix(cluster): enforce cluster-epoch fence at the decode chokepoint#245EnRaiha wants to merge 2 commits into
EnRaiha wants to merge 2 commits into
Conversation
…pic NodeDB-Lab#165) cluster_epoch was stamp-only: write_frame stamps the local epoch on every outbound frame and parse_frame only OBSERVED the inbound stamp (fetch_max) — the module docs claimed stale-epoch peers are rejected, but no rejection existed anywhere. Make it true: - ClusterError::StalePeerEpoch { peer_epoch, local_epoch } (next to UnsupportedWireVersion) for operator-visible diagnostics. - validate_peer_cluster_epoch(rpc_type, peer_epoch) in cluster_epoch.rs: rejects peer_epoch < local unless the RPC type is exempt (EPOCH_EXEMPT_RPC_TYPES = JOIN_REQ/JOIN_RESP/PING/PONG — join is how a fenced peer re-adopts the epoch; ping/pong is the pre-join liveness channel and the discovery path for fenced peers). - Single enforcement point: parse_frame, right after the peer_epoch parse, before payload/CRC work — the only decode path for every rpc_codec frame in both directions (server requests + shuffle-push, client responses). No dispatch/transport changes needed; the new Err propagates through existing ? paths. Join responses still carry the leader's epoch and the existing observe (fetch_max) on the accept path is what re-adopts a rejoining node — no ordering change. - Exported from lib.rs for API symmetry. Tests (9 new): validate rejects stale non-exempt / accepts equal+newer / genesis 0-0 ok / join+ping exempt / topology+vote fenced; parse_frame rejects stale epoch (local mark untouched) / accepts newer + observes / join+ping exempt. EPOCH_TEST_LOCK serialises every test mutating the shared epoch global, including the header decode tests (fixed a latent parallel race in v3_frame_round_trips_with_epoch and parse_frame_accepts_current_version, which mutated the global without any lock). Verification: cluster_epoch 11/11 x5 runs, header 9/9 x5 runs, full lib 1016/1016 serial (parallel run has a PRE-EXISTING flaky race in transport::client::tests::insecure_transport_rejects_non_private_bind on a shared observability counter — passes solo, fails only when parallel; last touched by main commit 5f0cd2f), clippy -D warnings 0, maya-gate L1 clean 4/4. Refactor note (deferred): parse_frame could return the epoch and move validation into raft_rpc::decode, but keeping header.rs the single authority over frame-level rejection (version/size/CRC/epoch) is the deliberate design (P2-FIX-PLAN Fix 3 §5); observe-then-bump ordering invariant documented for the fetch_add(1) bump path.
11 tasks
EnRaiha
added a commit
to EnRaiha/nodedb
that referenced
this pull request
Aug 24, 2026
Drill/fix-plan/GLM-resolution docs (commit dd0c316) written pre-fix are now stale. Add resolution banners pointing to the implemented fixes: SWIM 8886846 (PR NodeDB-Lab#243), wire window e60a853 (PR NodeDB-Lab#244), epoch fence 4f92959 (PR NodeDB-Lab#245), lease GC 16d9916 (PR NodeDB-Lab#246). Full verification + refactor code in P2-REPORT.md.
…pilot) handle_stream matches StalePeerEpoch explicitly: the frame is dropped with a warn! log (fenced until rejoin) instead of propagating through generic ? which ended the stream task — fenced frames are NOT transport failures; the connection stays open for exempt join/ping traffic and the sender's raft retry converges via fetch_max re-observation.
EnRaiha
added a commit
to EnRaiha/nodedb
that referenced
this pull request
Aug 24, 2026
/NodeDB-Lab#246 - catalog: load_swim_incarnation returns an ERROR on corrupted metadata (len != 8) instead of silently Ok(None); register_default_subsystems propagates the read error instead of unwrap_or(None) — bootstrap no longer hides catalog IO/corruption problems (PR NodeDB-Lab#243 critical items). - server: handle_stream drops StalePeerEpoch frames explicitly with a warn! log instead of ending the stream via generic ? — fenced frames are not transport failures; connection stays open for exempt traffic (PR NodeDB-Lab#245 critical item). - lease_gc: proposal failures logged at warn (operators can notice GC not happening); drain_propose doc comment translated to English (PR NodeDB-Lab#246 review items).
Author
|
Addressed the critical transport-handling item in e9b1a46: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the cluster-epoch fence real (epic #165 High #6). cluster_epoch was stamp-only: docs claimed stale-epoch peers are rejected but no rejection existed.
9 new tests; cluster_epoch 11/11 + header 9/9 stable across 5 runs, full lib 1016/1016 serial, clippy 0, maya-gate clean 4/4.
Part of #165.