Repository navigation
fix(zipper): keep idle backend connections across query bursts - #590
Merged
Merged
Conversation
Pooled backend connections were closed after 3s idle (set in 2018 to stop idle connections piling up). Alerts query the same stores every minute, so every burst found an empty pool and opened hundreds of new connections; a single connect that exceeded the 200ms connect timeout on the only replica holding a series failed the render with HTTP 500. Make the idle timeout configurable as idleConnTimeout and default it to 90s, longer than the minutely query interval and below the stores' own idle timeout. MaxIdleConnsPerHost still bounds the pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
deniszh
approved these changes
Nov 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Renders fail with HTTP 500 ("all backends failed") in short bursts several times an hour, and right after a carbonapi restart. Each failed request has one backend connect fail with
dial tcp …: i/o timeout(connect timeout 200ms) on the only backend holding the series.The backends and the network are not slow. In a packet capture of one burst, carbonapi sent 13.7k SYNs in 2s and every SYN was answered within 16ms (0.2ms in the same data centre), yet carbonapi logged 19 connect timeouts to a backend whose 517 SYNs were all answered within 0.2ms. With thousands of dials in flight, some dial goroutines run after their 200ms deadline.
The dials happen because the zipper closes a pooled backend connection after 3s idle (set in 2018 to stop idle connections piling up). Alerting queries the same backends every minute, so every burst finds an empty pool and opens its connections again: 140–180 new backend connections per second on average, up to 8k in one second.
Architecture
The idle timeout becomes the
idleConnTimeoutzipper setting with a 90s default: longer than a minutely query interval. Keep it below the backends' own idle timeout (go-carbonidle-timeout).maxIdleConnsPerHoststill bounds each pool, and pools are LIFO, so connections beyond what traffic keeps busy still close afteridleConnTimeout. 0 keeps connections until the backend closes them. A reused connection that the backend closed is retried once, as before.Before / after
idleConnTimeout)Validation
TestBackendIdleConnTimeoutFromConfig: a render within the configured idle timeout reuses the connection and a render after it dials again; it fails without this change ("idle connection not closed").TestBackendIdleConnTimeoutDefaultSpansMinutelyBurstskeeps the default above a 1m query interval. Config parse tests cover the default and an override.go test -race ./...passes; golangci-lint v2.13.2: 0 issues.