Skip to content

fix(zipper): rediscover backends for hot cached paths - #589

Merged
timtofan merged 2 commits into
masterfrom
atimofieiev/pathcache-rediscover-gh
Nov 7, 2026
Merged

timtofan merged 2 commits into
masterfrom
atimofieiev/pathcache-rediscover-gh

Conversation

@timtofan

@timtofan timtofan commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

Why

Reads can drop a whole series for a few minutes although a store still holds it. The zipper's per-backend path cache renews a routing entry from every successful response, including responses to requests it already filtered with that same cache. A path that is read more often than expireDelaySec therefore never re-runs discovery:

  • A backend that starts holding the path later (for example after the write ring changes) is never asked, so new points stay invisible until carbonapi restarts.
  • A replica that is unreachable for longer than expireDelaySec (a store restart or a network block) drops out and never rejoins. Hot paths shrink towards a single replica, and that replica's next timeout returns not-found for the series. With a response cache in front, the gap is served for another cache TTL.

Architecture

request ─▶ backend.Filter(paths) ─┬─ some backend Contains(path) → send only to those (filtered)
                                  └─ none                        → fan out to all eligible (discovery)
response ─▶ pathCache.setAll / setLeaves (validUntil = now + expireDelaySec)

The bug is in the feedback loop: responses to filtered requests also called setAll/setLeaves, so the set of routed backends could only shrink while a path stayed hot.

The fix adds a local PathCacheFiltered flag to RenderRequest/FindRequest, set where backend.Filter runs (upstream.go Find/Render, stable_render.go). Backends learn paths only from unfiltered (discovery) responses: buffered HTTP, streaming/packed HTTP, gRPC render and exact-name HTTP/gRPC find. The flag is never put on the wire. Stable and opportunistic render batching keep requests with different flags in separate batches, so a filtered request cannot renew entries through a shared batch. Expiry is exclusive at the deadline (now < validUntil).

Unchanged: TLD discovery (tldcache rebuilds per probe cycle and bypasses Filter), replica merge and mismatch modes, response-cache keys, absent-point handling.

Before / after

Before After
Who renews a routing entry any successful response, including cache-filtered ones discovery (unfiltered) responses only
Hot path after a replica outage longer than expireDelaySec routed to the remaining replicas forever; one more miss drops the series the returning replica rejoins within expireDelaySec
Hot path after the write ring changes routed to the old holder only fans out once per expireDelaySec and learns the new holder
Recovery restart carbonapi automatic within expireDelaySec
Steady-state cost R backend requests/render for R holders R + (N−R)/K (one discovery per window, K renders per window)

TestPathCacheSteadyStateFanout, 8 backends polled every 5s with a 120s TTL: 1 holder 1.29 requests/render (was 1; always-fan-out is 8), 3 holders 3.21 (was 3).

Validation

  • TestPathCacheReplicaRejoinsAfterOutage: one replica is unreachable for 125s, then returns. Without the fix it is never asked again in any of the four render modes (buffered, stream, packed, gRPC); with the fix it rejoins and the other replica's next miss no longer drops the series.
  • TestPathCacheHotRoutingRediscovery: 12 cases (four render modes × render-only / find-only / mixed) fail without the fix and pass with it.
  • TestUpstreamRoutingCacheMetadata and TestCollectRenderBatchBoundsAndIdentity/different_routing_cache_policy cover the upstream flag and batch separation.
  • go test -race ./... and the nativegzip net tests pass; golangci-lint v2.13.2: 0 issues.
  • Running in production on one frontend for 15 minutes against an identical peer at the same traffic (~430 requests/s): no 5xx (peer 6), mean render latency 21.6ms vs 21.4ms, backend render requests per render +4.6% (the bounded rediscovery cost).

Anton Timofieiev added 2 commits November 6, 2026 13:31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both replicas hold the series; one is unreachable for longer than the path
cache expiry, then returns. It must rejoin within one more expiry, or the
other replica stays the only route and its next miss drops the series.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@timtofan
timtofan merged commit aa4ad17 into master Nov 7, 2026
6 checks passed
@timtofan
timtofan deleted the atimofieiev/pathcache-rediscover-gh branch November 7, 2026 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants