Skip to content

fix(zipper): keep idle backend connections across query bursts - #590

Merged
timtofan merged 1 commit into
masterfrom
atimofieiev/store-conn-idle-gh
Nov 8, 2026
Merged

timtofan merged 1 commit into
masterfrom
atimofieiev/store-conn-idle-gh

Conversation

@timtofan

@timtofan timtofan commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

Why

Renders fail with HTTP 500 ("all backends failed") in short bursts several times an hour, and right after a carbonapi restart. Each failed request has one backend connect fail with dial tcp …: i/o timeout (connect timeout 200ms) on the only backend holding the series.

The backends and the network are not slow. In a packet capture of one burst, carbonapi sent 13.7k SYNs in 2s and every SYN was answered within 16ms (0.2ms in the same data centre), yet carbonapi logged 19 connect timeouts to a backend whose 517 SYNs were all answered within 0.2ms. With thousands of dials in flight, some dial goroutines run after their 200ms deadline.

The dials happen because the zipper closes a pooled backend connection after 3s idle (set in 2018 to stop idle connections piling up). Alerting queries the same backends every minute, so every burst finds an empty pool and opens its connections again: 140–180 new backend connections per second on average, up to 8k in one second.

Architecture

query burst ─▶ carbonapi (zipper) ─▶ per-backend connection pool
                                     ├─ idle connection → reuse
                                     └─ none → connect (connect timeout) ─▶ backend
only holder's connect times out + other backends not found → "all backends failed" → HTTP 500

The idle timeout becomes the idleConnTimeout zipper setting with a 90s default: longer than a minutely query interval. Keep it below the backends' own idle timeout (go-carbon idle-timeout). maxIdleConnsPerHost still bounds each pool, and pools are LIFO, so connections beyond what traffic keeps busy still close after idleConnTimeout. 0 keeps connections until the backend closes them. A reused connection that the backend closed is retried once, as before.

Before / after

Before After
Idle backend connection lifetime 3s 90s (idleConnTimeout)
New backend connections per frontend 137–180/s, bursts up to 8k/s ~4/s; one ~3s burst after a restart
Established backend connections per frontend ~1–3k ~3–4k (up to ~120 per busy backend)
Connect-timeout 5xx bursts several per hour only possible in the first seconds after a restart

Validation

  • TestBackendIdleConnTimeoutFromConfig: a render within the configured idle timeout reuses the connection and a render after it dials again; it fails without this change ("idle connection not closed"). TestBackendIdleConnTimeoutDefaultSpansMinutelyBursts keeps the default above a 1m query interval. Config parse tests cover the default and an override.
  • go test -race ./... passes; golangci-lint v2.13.2: 0 issues.
  • Running in production on one frontend against an identical peer at the same traffic (51.5k renders per 2 min each): new backend connections 4/s vs 137/s, mean render latency 21.4ms vs 22.6ms, no 5xx. A reload under traffic produced one 3s burst of 12.8k connects while the pool warmed, then about one per second, with no connect timeouts.

Pooled backend connections were closed after 3s idle (set in 2018 to
stop idle connections piling up). Alerts query the same stores every
minute, so every burst found an empty pool and opened hundreds of new
connections; a single connect that exceeded the 200ms connect timeout
on the only replica holding a series failed the render with HTTP 500.

Make the idle timeout configurable as idleConnTimeout and default it to
90s, longer than the minutely query interval and below the stores' own
idle timeout. MaxIdleConnsPerHost still bounds the pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@timtofan
timtofan merged commit 93e6fe6 into master Nov 8, 2026
6 checks passed
@timtofan
timtofan deleted the atimofieiev/store-conn-idle-gh branch November 8, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants