Skip to content

fix(backend): recover unavailable replicas when peers return not-found - #598

Merged
timtofan merged 1 commit into
masterfrom
atimofieiev/replica-unavailable
Nov 14, 2026
Merged

timtofan merged 1 commit into
masterfrom
atimofieiev/replica-unavailable

Conversation

@timtofan

Copy link
Copy Markdown
Collaborator

Why

A temporarily unavailable replica can fail an otherwise answered render. When one replica returns HTTP 503 or times out and all other selected replica groups for the same cluster report that the target does not exist, the error fan-in currently turns the render into HTTP 500. This particularly affects missing targets in wildcard or multi-target expressions.

Architecture

Render replies are grouped by data center and cluster, and classified separately for each fetch. Extend the existing cross-replica connection-failure recovery to HTTP 503 and timeout errors. Recovery requires another selected replica group in the same cluster, and every other selected group in that cluster must have answered without data.

Successful data replies retain their existing behavior. Arbitrary HTTP 500 responses, malformed bodies, ungrouped backends, unrelated clusters, and wholly unavailable clusters retain their errors. This changes local fan-in classification and adds no retry or transport path.

Before / after

Before After
One replica returns 503 or times out; every same-cluster peer reports not-found: render fails with 500. Those availability errors share the peers' not-found result.
A peer returns data. Existing successful-data behavior is preserved.
No same-cluster peer answers, or the backend reports an arbitrary server/decoding error. The render still fails.

Validation

With GOTOOLCHAIN=go1.27.0, build, vet and tests pass for ./pkg/backend ./pkg/app/carbonapi.

Extended render-handler regression cases cover 503 with data, missing targets, wrapped errors, read timeouts and context deadlines, and verify that HTTP 500, malformed responses, unavailable peers, unrelated clusters and ungrouped backends still fail where required. Existing cases exercise ordinary and coalesced fetches.

@timtofan
timtofan merged commit 71af595 into master Nov 14, 2026
2 of 5 checks passed
@timtofan
timtofan deleted the atimofieiev/replica-unavailable branch November 14, 2026 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant