Repository navigation
fix(backend): prewarm store connections before a reload takes traffic - #596
Merged
Merged
Conversation
added 8 commits
November 13, 2026 16:31
When a replica group (stores sharing dc and cluster) answered the request, an error from another member of that group no longer fails a required fetch: the group's answer stands for the whole group. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A store restart refuses connections for a few seconds. Since the required-fetch rule (#593) and the replica-group recovery, a refused fetch with no answering replica group member fails the render, so every store restart became a burst of 500s. Fetches that failed before the backend sent any response (refused, reset or dial failure inside http.Client.Do) are now repeated against the same backend with a 250 ms doubling backoff capped at 2 s, for up to timeouts.connectRetry (default 10s) and never past the request deadline. Groups that another member already answered are not retried. Failures after the backend answered are not retried. When the window runs out, the required-fetch and replica-group rules apply unchanged. New counters: backend_connect_retries_total and backend_connect_retry_recoveries_total. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The connect retry retried a refused store's render fetch for up to timeouts.connectRetry whenever no member of that store's own replica group answered. Renders query every DC's group and fill missing points from any of them, so while one store refused, every render touching it waited the whole window even though the other DCs had already returned the same series. A production canary (30 s refusal of a one-store group) added p50 7.9 s and max 10.1 s per render and turned client give-ups into 133 HTTP 500s and 95 499s. A refused fetch is now given up as soon as a replica group of the same cluster has returned data for the target: before the first retry, and during an attempt, where repeats still in flight to other groups of that cluster are no longer awaited. Retries continue only while no other group has data (all copies briefly unreachable), which is what the connect retry was for. Not-found answers, data from another cluster and ungrouped backends never cancel a retry. A cancelled fetch keeps its plain connect error, so the required-fetch and replica-group rules decide the render exactly as before. New counter: backend_connect_retries_cancelled_total. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ast mismatches mergeMetrics used the first reply as the base and stopped healing at the first reply whose step or point count differed from it. A store that answered first with a shorter series (for example while replaying after a restart) became the base and disabled healing for every reply after it, so renders showed nulls that other replicas held. When the finest-step replies differ in point count, the freshest one (latest present point, then most present points, then reply order) now decides the shape, and the base is the first reply with that shape. Replies of another shape are skipped one by one instead of ending the list. Replies of one shape still merge in reply order, so values, majority voting and mismatch counts are bit-identical to before; a differential test against the old algorithm checks this. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Connect retries were given up only when another replica group of the same cluster returned data. Renders fan out to every cluster, so most fetches to a refused store are for names held in other clusters; there the refused cluster's other groups can only answer not-found, the fetch was never covered, and it still waited the whole connectRetry window. A production canary (30s refusal of a one-store replica group) got 41 HTTP 500 and 67 499 in 30s against 1 and 1 on a peer instance. Replica groups of one cluster hold the same names, so any definitive answer from another group of the cluster, data or not-found, now covers the target: the refused fetch is given up before the first retry, and repeats in flight to other groups of that cluster stop being awaited once one group answers. Retries continue only while no other group of the cluster has answered at all (every copy unreachable), the restart case the connect retry was for. Server errors, answers from another cluster and ungrouped backends still never cancel a retry. A cancelled fetch keeps its plain connect error, so the required-fetch and replica-group rules decide the render as before: a required fetch no group has data for still fails. backend_connect_retries_cancelled_total now also counts not-found cover. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…cluster answered not-found A production canary returned HTTP 500 for renders such as sumSeries(x, *) while the only store of a one-store replica group refused connections (and again while it timed out at dial). The literal target "*" got not-found from 93 stores and a connect error from one. Since the required-fetch rule the fan-in returns not-found only when every error is a not-found, and the not-found retry cancellation kept the plain connect error on the fetch it gave up, so the whole render failed. A peer instance on the previous release returned 200 with the other series. Replica groups of one cluster hold the same names. When every other replica group of the failed store's cluster gave a definitive not-found (some member answered, none returned data), the store's connect failures (refused, reset or timed out before any response) now count as the cluster's not-found, with or without connect retries. The connect error still fails the fetch when the group is its cluster's only group, when another group of the cluster is unreachable too or failed (HTTP 5xx), or when one returned data (the render has that data anyway). Server errors, failures after the backend accepted the request, slow-replica cuts and ungrouped backends are unchanged, so the required-fetch rule still rejects a fetch whose data could exist only on the unreachable stores. Tests: the canary fetch of "*" (94 stores, 93 not-found, one single-store group refused or timing out at dial) is not-found, and sumSeries(a.b.*, *) returns 200 with the a.b.* series; every carbon group unreachable still returns 500. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A go-carbon store that restarts accepts connections while it warms up but answers renders only after seconds. Renders query every DC's replica group and merge them, yet the fan-in waited for every backend up to afterStarted (15s in production). During one store restart about 300 renders per minute per instance took 10-15s although the other DCs had answered. The render fan-in now collects all replica groups concurrently. Once every other replica group of the same cluster has received all its replies and given a definitive answer (data or not-found), and at least one of them returned data, a group still silent after timeouts.slowReplicaGrace (default 1s, 0 keeps waiting) is no longer awaited: its outstanding fetches become a plain error, recovered as not-found when a member of that group answered, and the merged result of the other groups is returned. backend_slow_replica_cancelled_total counts the abandoned fetches. If every other group of the cluster said not-found, the slow group may be the only holder of the series, so it is awaited in full. This differs from the connect retry, which also gives up on a not-found cover: a refused store cannot answer until it is back, a slow one will deliver its copy. A group is also never cut while another group of its cluster is still pending or only refused or failed, data from another cluster never counts, and ungrouped backends are always awaited. The in-flight request is not cancelled because coalesced batches share one context with other targets. Message order stays by group, so mergeMetrics sees the same input order as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A release reload (systemctl reload -> SIGUSR2) re-execs carbonapi through gracehttp: the new process starts with an empty backend connection pool and takes full traffic at once. Every render then dials, thousands of store connects start within a second under the 200 ms connect timeout, and the ones that miss it fail renders after the connect-retry window. On one production instance that gave ~270 render 500s at a release and 46 on a repeat reload. The old process keeps serving until the new one calls gracehttp.Serve, so dial ahead of that: SetupZipper now dials prewarmConnsPerBackend (default 64) connections to every HTTP backend, round by round with at most 256 in flight and a 2 s connect budget, bounded by prewarmTimeout (default 5 s). The transport's DialContext hands those out before dialing; unused ones close after 10 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #594 (this PR contains its commits until it merges; review only the last commit).
Why
A graceful reload (
systemctl reload→ SIGUSR2) re-execs carbonapi through gracehttp. The new process starts with an empty backend connection pool and takes full traffic at once: every render dials, thousands of store connects start within a second under the 200ms connect timeout, and the ones that miss it fail renders once the connect-retry window runs out. On a production instance one release reload produced ~270 render 500s, and a repeat reload 46.Architecture
The old process keeps serving until the new one calls
gracehttp.Serve, so the new process can dial before it takes traffic:The warm dialer lives in
pkg/backend/net/warm_dialer.goand is wired inpkg/app/carbonapi/app.go. Unused prewarmed connections close after 10s, so the pool converges to the normal idle-connection behaviour. Non-HTTP backends are not affected.Before / after
prewarmConnsPerBackend(default 64,0disables) andprewarmTimeout(default 5s)Usage
Enabled by default through the new defaults in
pkg/cfg/common.go; setprewarmConnsPerBackend: 0to turn it off.Validation
warm_dialer_test.gocovers handing out prewarmed connections before dialing, idle expiry, skipping stale connections, and failure counting with context cancellation;backend_conn_test.gochecks that the first renders after a prewarm need no dial.go test ./pkg/backend/... ./pkg/app/carbonapi/... ./pkg/types/... ./pkg/cfg/...pass.