Skip to content

fix(backend): prewarm store connections before a reload takes traffic - #596

Merged
timtofan merged 8 commits into
masterfrom
atimofieiev/fe-reload-drain-gh
Nov 13, 2026
Merged

timtofan merged 8 commits into
masterfrom
atimofieiev/fe-reload-drain-gh

Conversation

@timtofan

Copy link
Copy Markdown
Collaborator

Depends on #594 (this PR contains its commits until it merges; review only the last commit).

Why

A graceful reload (systemctl reload → SIGUSR2) re-execs carbonapi through gracehttp. The new process starts with an empty backend connection pool and takes full traffic at once: every render dials, thousands of store connects start within a second under the 200ms connect timeout, and the ones that miss it fail renders once the connect-retry window runs out. On a production instance one release reload produced ~270 render 500s, and a repeat reload 46.

Architecture

The old process keeps serving until the new one calls gracehttp.Serve, so the new process can dial before it takes traffic:

SIGUSR2 → new process: SetupZipper
            └─ prewarm: dial prewarmConnsPerBackend conns to every HTTP backend
               (rounds, ≤256 in flight, 2s connect budget, bounded by prewarmTimeout)
          → gracehttp.Serve (old process stops accepting)
          → transport.DialContext hands out prewarmed conns first, dials only when none are left

The warm dialer lives in pkg/backend/net/warm_dialer.go and is wired in pkg/app/carbonapi/app.go. Unused prewarmed connections close after 10s, so the pool converges to the normal idle-connection behaviour. Non-HTTP backends are not affected.

Before / after

Before After
A reloaded process opens every backend connection on demand under full load 64 connections per backend are already open when traffic switches
~270 / 46 render 500s per reload on a production instance 0 render 5xx over 3 reloads on the same instance
No knob prewarmConnsPerBackend (default 64, 0 disables) and prewarmTimeout (default 5s)

Usage

Enabled by default through the new defaults in pkg/cfg/common.go; set prewarmConnsPerBackend: 0 to turn it off.

Validation

  • warm_dialer_test.go covers handing out prewarmed connections before dialing, idle expiry, skipping stale connections, and failure counting with context cancellation; backend_conn_test.go checks that the first renders after a prewarm need no dial.
  • go test ./pkg/backend/... ./pkg/app/carbonapi/... ./pkg/types/... ./pkg/cfg/... pass.
  • Production: three reloads of a canary instance gave 0 render 5xx and 0 dial errors, with 64 connections per backend prewarmed in ~0.2s; the unchanged build gave 46 render 500s on the same instance.

Anton Timofieiev added 8 commits November 13, 2026 16:31
When a replica group (stores sharing dc and cluster) answered the request,
an error from another member of that group no longer fails a required
fetch: the group's answer stands for the whole group.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A store restart refuses connections for a few seconds. Since the
required-fetch rule (#593) and the replica-group recovery, a refused fetch
with no answering replica group member fails the render, so every store
restart became a burst of 500s.

Fetches that failed before the backend sent any response (refused, reset
or dial failure inside http.Client.Do) are now repeated against the same
backend with a 250 ms doubling backoff capped at 2 s, for up to
timeouts.connectRetry (default 10s) and never past the request deadline.
Groups that another member already answered are not retried. Failures
after the backend answered are not retried. When the window runs out, the
required-fetch and replica-group rules apply unchanged.

New counters: backend_connect_retries_total and
backend_connect_retry_recoveries_total.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The connect retry retried a refused store's render fetch for up to
timeouts.connectRetry whenever no member of that store's own replica group
answered. Renders query every DC's group and fill missing points from any
of them, so while one store refused, every render touching it waited the
whole window even though the other DCs had already returned the same
series. A production canary (30 s refusal of a one-store group) added p50
7.9 s and max 10.1 s per render and turned client give-ups into 133 HTTP
500s and 95 499s.

A refused fetch is now given up as soon as a replica group of the same
cluster has returned data for the target: before the first retry, and
during an attempt, where repeats still in flight to other groups of that
cluster are no longer awaited. Retries continue only while no other group
has data (all copies briefly unreachable), which is what the connect retry
was for. Not-found answers, data from another cluster and ungrouped
backends never cancel a retry. A cancelled fetch keeps its plain connect
error, so the required-fetch and replica-group rules decide the render
exactly as before.

New counter: backend_connect_retries_cancelled_total.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ast mismatches

mergeMetrics used the first reply as the base and stopped healing at the
first reply whose step or point count differed from it. A store that
answered first with a shorter series (for example while replaying after a
restart) became the base and disabled healing for every reply after it, so
renders showed nulls that other replicas held.

When the finest-step replies differ in point count, the freshest one (latest
present point, then most present points, then reply order) now decides the
shape, and the base is the first reply with that shape. Replies of another
shape are skipped one by one instead of ending the list. Replies of one shape
still merge in reply order, so values, majority voting and mismatch counts
are bit-identical to before; a differential test against the old algorithm
checks this.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Connect retries were given up only when another replica group of the same
cluster returned data. Renders fan out to every cluster, so most fetches to
a refused store are for names held in other clusters; there the refused
cluster's other groups can only answer not-found, the fetch was never
covered, and it still waited the whole connectRetry window. A production
canary (30s refusal of a one-store replica group) got 41 HTTP 500 and 67
499 in 30s against 1 and 1 on a peer instance.

Replica groups of one cluster hold the same names, so any definitive answer
from another group of the cluster, data or not-found, now covers the target:
the refused fetch is given up before the first retry, and repeats in flight
to other groups of that cluster stop being awaited once one group answers.
Retries continue only while no other group of the cluster has answered at
all (every copy unreachable), the restart case the connect retry was for.
Server errors, answers from another cluster and ungrouped backends still
never cancel a retry. A cancelled fetch keeps its plain connect error, so
the required-fetch and replica-group rules decide the render as before: a
required fetch no group has data for still fails.

backend_connect_retries_cancelled_total now also counts not-found cover.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…cluster answered not-found

A production canary returned HTTP 500 for renders such as sumSeries(x, *)
while the only store of a one-store replica group refused connections (and
again while it timed out at dial). The literal target "*" got not-found
from 93 stores and a connect error from one. Since the required-fetch rule
the fan-in returns not-found only when every error is a not-found, and the
not-found retry cancellation kept the plain connect error on the fetch it
gave up, so the whole render failed. A peer instance on the previous
release returned 200 with the other series.

Replica groups of one cluster hold the same names. When every other replica
group of the failed store's cluster gave a definitive not-found (some member
answered, none returned data), the store's connect failures (refused, reset
or timed out before any response) now count as the cluster's not-found, with
or without connect retries. The connect error still fails the fetch when the
group is its cluster's only group, when another group of the cluster is
unreachable too or failed (HTTP 5xx), or when one returned data (the render
has that data anyway). Server errors, failures after the backend accepted the
request, slow-replica cuts and ungrouped backends are unchanged, so the
required-fetch rule still rejects a fetch whose data could exist only on the
unreachable stores.

Tests: the canary fetch of "*" (94 stores, 93 not-found, one single-store
group refused or timing out at dial) is not-found, and
sumSeries(a.b.*, *) returns 200 with the a.b.* series; every carbon group
unreachable still returns 500.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A go-carbon store that restarts accepts connections while it warms up but
answers renders only after seconds. Renders query every DC's replica group
and merge them, yet the fan-in waited for every backend up to afterStarted
(15s in production). During one store restart about 300 renders per minute
per instance took 10-15s although the other DCs had answered.

The render fan-in now collects all replica groups concurrently. Once every
other replica group of the same cluster has received all its replies and
given a definitive answer (data or not-found), and at least one of them
returned data, a group still silent after timeouts.slowReplicaGrace
(default 1s, 0 keeps waiting) is no longer awaited: its outstanding fetches
become a plain error, recovered as not-found when a member of that group
answered, and the merged result of the other groups is returned.
backend_slow_replica_cancelled_total counts the abandoned fetches.

If every other group of the cluster said not-found, the slow group may be
the only holder of the series, so it is awaited in full. This differs from
the connect retry, which also gives up on a not-found cover: a refused
store cannot answer until it is back, a slow one will deliver its copy.
A group is also never cut while another group of its cluster is still
pending or only refused or failed, data from another cluster never counts,
and ungrouped backends are always awaited. The in-flight request is not
cancelled because coalesced batches share one context with other targets.
Message order stays by group, so mergeMetrics sees the same input order as
before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A release reload (systemctl reload -> SIGUSR2) re-execs carbonapi through
gracehttp: the new process starts with an empty backend connection pool and
takes full traffic at once. Every render then dials, thousands of store
connects start within a second under the 200 ms connect timeout, and the
ones that miss it fail renders after the connect-retry window. On one
production instance that gave ~270 render 500s at a release and 46 on a
repeat reload.

The old process keeps serving until the new one calls gracehttp.Serve, so
dial ahead of that: SetupZipper now dials prewarmConnsPerBackend (default
64) connections to every HTTP backend, round by round with at most 256 in
flight and a 2 s connect budget, bounded by prewarmTimeout (default 5 s).
The transport's DialContext hands those out before dialing; unused ones
close after 10 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@timtofan
timtofan merged commit 215b192 into master Nov 13, 2026
4 of 6 checks passed
@timtofan
timtofan deleted the atimofieiev/fe-reload-drain-gh branch November 13, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant