Skip to content

[WRONG BRANCH] fix(pool): rotate Anthropic pool accounts on out-of-extra-usage 400 - #25

Merged
vadymhimself merged 3 commits into
release/2.65.0-vadymhimself.15from
gauge/pool-rotate-extra-usage-400
Nov 14, 2026
Merged

vadymhimself merged 3 commits into
release/2.65.0-vadymhimself.15from
gauge/pool-rotate-extra-usage-400

Conversation

@vadymhimself

Copy link
Copy Markdown
Collaborator

Why (AGE-167)

Anthropic answers an account whose plan and extra usage are spent with HTTP 400 ("You're out of extra usage" / "Third-party apps now draw from your extra usage"), not 429. Since #8/#11 classifyError maps both to insufficient_quota, so combos hop, but the Anthropic account pool only rotated on 429. One spent pool account failed the request while the other pool accounts sat idle; a hosted gateway (no combo) returned the error to the agent.

What

  • New src/oauth/anthropic-pool-rotation.ts: isPoolRotatableStatus() / peekAnthropicPoolRotation() → trigger "429" or "quota-400". A 400 counts only when the existing classifyError says insufficient_quota; every other 400 (malformed request) passes through untouched. Body read with the existing bounded peek (readQuotaResetAt, now also returns the text) and replayed.
  • Used at all rotation sites: adapter-dispatch, adapter-continuation, sidecar hook (sidecar-execution, images/loop, web-search/loop). On a quota-400 only the Anthropic pool rotates; key pools and generic OAuth still rotate on 429 alone.
  • rotateAnthropicAccountOn429(..., trigger): a quota-400 always cools the account (Retry-After → rejected-window reset → 10 min fallback, matching the combo layer's insufficient_quota cooldown instead of the 60 s rate-limit default).

Tests

  • New tests/adapters/anthropic/anthropic-pool-extra-usage-400.test.ts (9): out-of-extra-usage A→B succeeds; third-party-apps 400 rotates; role-order 400 does not rotate and reaches the caller; helper + cooldown units. 2 of them fail without the dispatch change.
  • Regression files green, run one at a time: always-on-429-failover 9, anthropic-account-pool 56, openrouter-quota-reset-cooldown-4024 18, terminal-continuation-owner-rotation 6, anthropic-sidecar-account-failover 2, web-search-sidecar-429 8, images/loop 56, generic-oauth-failover 53, web-search-timeout-contract 7, web-search 81, test-layout-tooling 16.
  • Full tsc left to CI (the 2 GB workspace cannot hold it); scoped type checks over every changed file are clean.

Eval added/updated: N/A gateway-internal pool rotation, no agent-visible behaviour scenario; covered by the unit tests above.

Known red: enforce-target "wrong base (release/…)" (as on every release-line PR).

🤖 Generated with Claude Code

Anthropic answers an account whose plan and extra usage are spent with
HTTP 400 ("You're out of extra usage", "Third-party apps now draw from
your extra usage"), not 429. Since #8/#11 src/lib/errors.ts classifies
both as insufficient_quota, so a combo hops on them, but the Anthropic
account pool still rotated on status 429 only (adapter-dispatch,
adapter-continuation, and the web-search/image sidecar loops). One spent
pool account therefore failed the request while the other pool accounts
sat idle; on a hosted gateway, which has no combo, the caller got the
error.

Add src/oauth/anthropic-pool-rotation.ts: isPoolRotatableStatus() and
peekAnthropicPoolRotation() decide the trigger ("429" or "quota-400").
A 400 counts only when the existing classifyError() says
insufficient_quota, so no message wording is duplicated, and any other
400 (a malformed request) is passed through untouched. The 400 body is
read with the bounded peek the key-pool 429 path already uses
(readQuotaResetAt, which now also returns the peeked text), and the
returned response replays it.

All three rotation sites use the helper. The sidecar hook moves only the
Anthropic pool on a quota-400; key pools and the generic OAuth roster
still rotate on 429 alone.

rotateAnthropicAccountOn429() takes the trigger. On a quota-400 it always
cools the account (the 400 is quota evidence even if the unified status
header still reads "allowed"), using the same Retry-After, then
rejected-window reset, as a quota 429. With neither, it falls back to 10
minutes, the cooldown the combo layer gives the same insufficient_quota
code, not the 60s rate-limit default that would send the next request
back to the spent account a minute later.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

⏳ DRAFT

  • wrong target branch (release/2.65.0-vadymhimself.15); retarget to dev.

What to do

  • Retarget this PR to dev — all contributions go to dev.

Its title has been prefixed with [WRONG BRANCH].
This pull request is being kept as a draft automatically. Once every issue above is resolved, it will be marked ready for review again.

@github-actions github-actions Bot changed the title fix(pool): rotate Anthropic pool accounts on out-of-extra-usage 400 [WRONG BRANCH] fix(pool): rotate Anthropic pool accounts on out-of-extra-usage 400 Nov 14, 2026
@github-actions
github-actions Bot marked this pull request as draft November 14, 2026 16:28
@vadymhimself

Copy link
Copy Markdown
Collaborator Author

qa-sec (Mars) at head 02203744a5edd369d6ec0eaba5de4032e84a3bb1: RED (2 items)

The direction is right: rotating a spent pool account on Anthropic's out-of-extra-usage 400, a 10 min fallback cooldown, and pass-through of other 400s. The peek and replay are bounded, an abort returns 499, and the sidecar keeps key pools and generic OAuth on 429 only. Two items before merge:

R1: any 400 that merely contains quota wording benches pool accounts, including text the client controls.
src/oauth/anthropic-pool-rotation.ts:19-26 (isQuotaExhaustedBadRequest) runs the whole 400 message through classifyError, which matches substrings anywhere in it (src/lib/errors.ts:426-437: insufficient_quota, quota exhausted, out of extra usage, ...). Anthropic 400s echo request values. For example, tool_use ids were found without tool_result blocks immediately after: <id> echoes a client-supplied tool id, and toolu_insufficient_quota fits the id pattern. One crafted request then counts as a quota-400 on every hop. rotateAnthropicAccountOn429 marks it limited (anthropic-routing.ts:802) and cools each account it touches for 10 min, up to ANTHROPIC_POOL_MAX_FAILOVERS_PER_REQUEST per request. A few repeats bench the whole shared pool for every tenant.
Exact change: in isQuotaExhaustedBadRequest, stop using classifyError. Parse the JSON body and require Anthropic's envelope (type === "error", error.type === "invalid_request_error"), with error.message starting with one of the two exact phrases, case-insensitive and anchored: You're out of extra usage or Third-party apps now draw from your extra usage. Return false for non-JSON bodies. Add the must-fail test: a 400 whose message echoes toolu_insufficient_quota (or contains "out of extra usage" in the middle) does not rotate and reaches the caller unchanged.

R2: the test payloads are hand-written (REAL-PAYLOAD-RATCHET, Vadym 20263).
tests/adapters/anthropic/anthropic-pool-extra-usage-400.test.ts:24-25 hand-writes the two Anthropic 400 messages and bodies. A test that imitates an outside payload must use a recorded real sample: put the full body and headers of a real out-of-extra-usage 400 (the AGE-167 incident, redacted) under fixtures/external/ and load it in the test. The anchored match in R1 depends on the exact wording, so the real sample matters here.

LOW (not blocking):

  • L1: the same substring problem predates this PR at the combo layer. classifyError → insufficient_quota gives a provider-scoped 10 min combo cooldown (errors.ts:664, combos/failover.ts), so an echoed phrase may already cool a whole provider. A follow-up card would harden classifyError for 400s the same way.
  • L2: images/loop.ts and web-search/loop.ts now pass trigger to deps.on429. Please confirm that executeResponsesSidecars is the only on429 implementation. Any other implementation that ignores trigger would rotate key pools on a quota-400.

A new push voids this. Re-review takes about 15 min after the fix.

…ct phrases

Security review (Mars, RED R1): isQuotaExhaustedBadRequest ran the whole
400 message through classifyError, which matches quota words anywhere in
the text. Anthropic's validation 400s echo client values back (tool ids,
model names, prompt text), so one crafted request such as a tool_use id
"toolu_insufficient_quota" could cool every pool account for 10 minutes.

The 400 now counts only when the body is Anthropic's JSON error envelope
(type "error", error.type "invalid_request_error") and error.message,
after leading whitespace, starts (case-insensitive) with "You're out of
extra usage" (straight or curly apostrophe) or "Third-party apps now draw
from your extra usage". A non-JSON body, another error type, a missing
envelope, or the wording anywhere but the start never rotates.
classifyError is no longer used here; combo classification is unchanged.

R2 (REAL-PAYLOAD-RATCHET): the third-party test payload is now the real
recorded Anthropic 400 (staging gateway log line, 2026-11-11), copied
byte-identical from AgentRadarLabs/agentradar-paperclip
fixtures/external/anthropic-extra-usage-thirdparty-400.txt @ 615ec8db69
into fixtures/external/anthropic/ with a README saying where it came
from. No real "You're out of extra usage" body has been recorded anywhere
(paperclip marks its sample "derived"), so that wording stays a typed
test string.

New must-fail tests (5 fail on the previous commit): a 400 echoing
toolu_insufficient_quota neither rotates nor changes on its way to the
caller; the wording mid-message does not rotate; the right wording in a
wrong envelope (rate_limit_error, api_error, missing type "error",
non-string message, array body) does not rotate; raw non-JSON text does
not rotate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vadymhimself

Copy link
Copy Markdown
Collaborator Author

qa-sec (Mars) re-review at head 0f57c1e6b62f361159af369023d806c52c977033: GREEN

I read the change from 0220374 to 0f57c1e (one commit):

  • R1 is fixed. isQuotaExhaustedBadRequest no longer uses classifyError. It needs a JSON body, the Anthropic envelope (type: "error", error.type: "invalid_request_error"), and error.message matching ^(?:you['’]re out of extra usage|third-party apps now draw from your extra usage) (case-insensitive) after trimStart. Non-JSON returns false. Anthropic validation messages start with a field path, so an echoed client value cannot reach the anchor.
  • Allowed path: both wordings rotate inside the envelope, including the curly apostrophe and upper case. Refusal path: echoed toolu_insufficient_quota (unit test, plus a routed test where the caller gets Anthropic's body byte for byte), the phrase in the middle of a message, and the wrong envelope type all stay null.
  • R2 is fixed as far as samples exist. The third-party wording loads the recorded capture (fixtures/external/anthropic/extra-usage-thirdparty-400.txt, byte-identical to paperclip's recorded sample, provenance in the README); the test only closes the JSON the logger cut. No real "You're out of extra usage" capture exists, and the README says so. Accepted: replace the typed string as soon as one is logged.
  • L2 is answered (the sidecar is the only real on429), and L1 has its own card, AGE-2650.

Merge only when Liza reopens release/* merges after the 1115a cut, at this exact head. A new push voids this GREEN.

The fix commits on this branch inherited 2.65.0-vadymhimself.21, which
names an already-published tag, so release-version-line failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vadymhimself

Copy link
Copy Markdown
Collaborator Author

qa-sec (Mars) re-stamp at head b07438614420d4e44b0dbf8f7c10561ee80cb4cb: GREEN (one commit on my 0f57c1e GREEN)

  • That commit changes only the version line, 2.65.0-vadymhimself.21 → .22, in package.json, desktop/src-tauri/Cargo.toml, Cargo.lock and tauri.conf.json, one line each. No code or dependency changed. My earlier review stands.
  • It needs CI green at this same head before merge. A new push voids this GREEN.

@vadymhimself
vadymhimself marked this pull request as ready for review November 14, 2026 23:16
@github-actions
github-actions Bot marked this pull request as draft November 14, 2026 23:16
@vadymhimself
vadymhimself merged commit 4577636 into release/2.65.0-vadymhimself.15 Nov 14, 2026
35 of 37 checks passed
vadymhimself pushed a commit that referenced this pull request Nov 14, 2026
…ient can echo

Rebuilt on the release line after #25 was squash-merged. The tree is
identical to the QA-approved head 4321ddc (tree 5f37668).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vadymhimself added a commit that referenced this pull request Nov 15, 2026
… words (AGE-2650) (#26)

Rebuilt on the release line after #25 was squash-merged. The tree is
identical to the QA-approved head 4321ddc (tree 5f37668).

Co-authored-by: Gauge <gauge@agentradar-house.staging.agentradar.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant