Skip to content

fix(zipper): keep lookups not-found when a store refuses in every replica group - #597

Merged
timtofan merged 2 commits into
masterfrom
atimofieiev/lookup-refused-notfound
Nov 13, 2026
Merged

timtofan merged 2 commits into
masterfrom
atimofieiev/lookup-refused-notfound

Conversation

@timtofan

@timtofan timtofan commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Why

A /metrics/find that matches nothing returned HTTP 422 (all backends failed with mixed errors: ... connection refused:1 ... No matches found:N), and /info HTTP 500, whenever one store refused connections in every replica group of a cluster. That happens when the retired stores of a resized cluster are stopped while they are still listed in the zipper config: each replica group then has one refusing member and the rest answer not-found. Dashboards showed an error instead of "no data". Renders were not affected.

The previous rule recovered a refusal only when another replica group of the same cluster answered not-found from every member, so no group could vouch once each group had a stopped store.

Architecture

lookupGroupErrors (find and info fan-in) now asks one question per cluster, clusterReachableNotFound:

  • at least one member of the cluster answered with its own not-found,
  • no member of the cluster returned matches or info,
  • every other error of the cluster is a connect failure (connectFailure: refused, reset or dial timeout before any response).

When all three hold, the cluster's connect failures become the lookup's not-found error, so errorsFanIn reports not-found. This replaces the stricter "another group answered in full" check, which the new rule covers.

Still failing, unchanged:

  • a cluster where no store answered (unreachable cluster),
  • any member with a server error or a timeout after the store accepted the request (the whole cluster stays an error),
  • ungrouped backends, which never recover each other.

Before / after

Case Before After
find, one refusing store in every replica group, rest not-found 422 200 []
info, same 500 404
one-group cluster, one member refused, the others not-found 422 200 []
same, plus a 500 from any member of the cluster 422 422
every store of a cluster refused 422 422

Validation

  • TestFindInfoRefusedStoreNotFound: 15 cases through both handlers. On master the new refused_in_every_group, refused_in_every_group_dial_timeout and one_group_cluster_member_refused cases fail with find 422; with the change all 15 pass, and the 8 must-fail cases still return 422/500.

  • go test -race ./pkg/backend/... ./pkg/app/carbonapi/... and go vet on both packages pass; golangci-lint v2.14.0 (the CI version) reports 0 issues.

  • Controlled test on a production frontend: a side instance with the real three replica groups of one cluster (2 stores each) plus one refusing member (127.0.0.1:1/2/3) in every group, queries that match nothing:

    Binary find x.* find a.b.….prefix* info find matching data
    master before this PR 422 422 500 200
    this PR 200 [] 200 [] 404 200, same body
  • Single-host canary on that frontend (drained restart, about 5 minutes of real traffic, compared with a peer on master): find 39× 200 / 0× 422 (peer 33 / 0), render 1,990 / 0 5xx (peer 1,986 / 0), partial-fail share 28% vs 29%, no panics. Five find/info queries returned byte-identical bodies on the canary and the peer. The host is back on master.

The second commit drops a constant parameter of a test helper that golangci-lint (unparam) flags on master, so CI passes.

🤖 Generated with Claude Code

Anton Timofieiev added 2 commits November 13, 2026 16:52
…lica group

A find that matches nothing returned HTTP 422, and an info HTTP 500, when
one store refused connections in every replica group of a cluster, for
example after stopping the retired stores of a resized cluster. A refusal
counted as not-found only when another replica group of the cluster
answered not-found from every member, and no group could.

Treat a cluster's connect failures (refused, reset, dial timeout) as
not-found when every reachable store of the cluster answered not-found
and no member failed in any other way. An unreachable cluster, a server
error or a timeout after the store accepted the request still fail the
lookup, and ungrouped backends still never recover each other.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
golangci-lint (unparam) failed CI on master: every caller passed "dc2".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@timtofan
timtofan marked this pull request as ready for review November 13, 2026 16:03
@timtofan
timtofan merged commit 3634442 into master Nov 13, 2026
6 checks passed
@timtofan
timtofan deleted the atimofieiev/lookup-refused-notfound branch November 13, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant