Skip to content

[WRONG BRANCH] fix(errors): hop on Anthropic 400 "out of extra usage" (OCX-LOW-400-NO-FAILOVER) - #8

Merged
vadymhimself merged 1 commit into
release/2.65.0-vadymhimself.15from
be-low-r8/out-of-extra-usage-failover
Nov 7, 2026
Merged

vadymhimself merged 1 commit into
release/2.65.0-vadymhimself.15from
be-low-r8/out-of-extra-usage-failover

Conversation

@vadymhimself

Copy link
Copy Markdown
Collaborator

Why

combo/waterfall-low called through the Mac gateway returns HTTP 400 "You're out of extra usage" from the Anthropic tier and does not fail over: classifyError maps the 400 to invalid_request_error, which comboFailureDecision treats as a hard stop (it hops only on 401/402/403/404/408/425/429/5xx and quota/rate codes). Found by core-be-r5, traced by be-low-r8. No ocx config can change this.

Change

src/lib/errors.ts classifyError: text containing "out of extra usage" maps to insufficient_quota (already a hop code, also feeds isRateLimitOrQuotaFailureMessage health blocking). Two asserts: classifyError (tests/server/error-fidelity.test.ts) and comboFailureDecision (tests/codex-integration/combos.test.ts). Both fail without the change (2 fail), pass with it (102 pass); tsc --noEmit clean.

Release

Targets release/2.65.0-vadymhimself.15 like #7 (the fork has no .16 branch yet); Liza cuts the tagged .16 and installs it (Vadym 19412: gateway upgrades ship as a tagged fork release). Not applied under /opt/homebrew.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Nov 7, 2026
@github-actions github-actions Bot changed the title fix(errors): hop on Anthropic 400 "out of extra usage" (OCX-LOW-400-NO-FAILOVER) [WRONG BRANCH] fix(errors): hop on Anthropic 400 "out of extra usage" (OCX-LOW-400-NO-FAILOVER) Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

⏳ DRAFT

  • wrong target branch (release/2.65.0-vadymhimself.15); retarget to dev.

What to do

  • Retarget this PR to dev — all contributions go to dev.

Its title has been prefixed with [WRONG BRANCH].
Automatic draft conversion failed (token cannot change draft status). Please convert this pull request to a draft manually. The required enforce-target check will keep failing until every issue above is resolved.

…ent_quota so combos hop

Anthropic answers a spent plan with HTTP 400, not 429. classifyError mapped it to
invalid_request_error, a hard stop in comboFailureDecision, so combo/waterfall-low
never failed over to its next tier. insufficient_quota is already a hop code.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@vadymhimself
vadymhimself force-pushed the be-low-r8/out-of-extra-usage-failover branch from 2839a7f to 03cb925 Compare November 7, 2026 18:33
@vadymhimself
vadymhimself marked this pull request as ready for review November 7, 2026 19:06
@vadymhimself
vadymhimself merged commit e987656 into release/2.65.0-vadymhimself.15 Nov 7, 2026
29 of 30 checks passed
vadymhimself added a commit that referenced this pull request Nov 7, 2026
…our extra usage" as insufficient_quota so combos hop (#11)

Same spent-plan state as "out of extra usage" (#8), newer Anthropic wording. Without it the
400 classifies as invalid_request_error and a combo stops instead of hopping to the next tier.

Co-authored-by: Claude Code <noreply@anthropic.com>
vadymhimself added a commit that referenced this pull request Nov 14, 2026
)

* fix(pool): rotate Anthropic pool accounts on out-of-extra-usage 400

Anthropic answers an account whose plan and extra usage are spent with
HTTP 400 ("You're out of extra usage", "Third-party apps now draw from
your extra usage"), not 429. Since #8/#11 src/lib/errors.ts classifies
both as insufficient_quota, so a combo hops on them, but the Anthropic
account pool still rotated on status 429 only (adapter-dispatch,
adapter-continuation, and the web-search/image sidecar loops). One spent
pool account therefore failed the request while the other pool accounts
sat idle; on a hosted gateway, which has no combo, the caller got the
error.

Add src/oauth/anthropic-pool-rotation.ts: isPoolRotatableStatus() and
peekAnthropicPoolRotation() decide the trigger ("429" or "quota-400").
A 400 counts only when the existing classifyError() says
insufficient_quota, so no message wording is duplicated, and any other
400 (a malformed request) is passed through untouched. The 400 body is
read with the bounded peek the key-pool 429 path already uses
(readQuotaResetAt, which now also returns the peeked text), and the
returned response replays it.

All three rotation sites use the helper. The sidecar hook moves only the
Anthropic pool on a quota-400; key pools and the generic OAuth roster
still rotate on 429 alone.

rotateAnthropicAccountOn429() takes the trigger. On a quota-400 it always
cools the account (the 400 is quota evidence even if the unified status
header still reads "allowed"), using the same Retry-After, then
rejected-window reset, as a quota 429. With neither, it falls back to 10
minutes, the cooldown the combo layer gives the same insufficient_quota
code, not the 60s rate-limit default that would send the next request
back to the spent account a minute later.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pool): anchor the quota-400 match to Anthropic's envelope and exact phrases

Security review (Mars, RED R1): isQuotaExhaustedBadRequest ran the whole
400 message through classifyError, which matches quota words anywhere in
the text. Anthropic's validation 400s echo client values back (tool ids,
model names, prompt text), so one crafted request such as a tool_use id
"toolu_insufficient_quota" could cool every pool account for 10 minutes.

The 400 now counts only when the body is Anthropic's JSON error envelope
(type "error", error.type "invalid_request_error") and error.message,
after leading whitespace, starts (case-insensitive) with "You're out of
extra usage" (straight or curly apostrophe) or "Third-party apps now draw
from your extra usage". A non-JSON body, another error type, a missing
envelope, or the wording anywhere but the start never rotates.
classifyError is no longer used here; combo classification is unchanged.

R2 (REAL-PAYLOAD-RATCHET): the third-party test payload is now the real
recorded Anthropic 400 (staging gateway log line, 2026-11-11), copied
byte-identical from AgentRadarLabs/agentradar-paperclip
fixtures/external/anthropic-extra-usage-thirdparty-400.txt @ 615ec8db69
into fixtures/external/anthropic/ with a README saying where it came
from. No real "You're out of extra usage" body has been recorded anywhere
(paperclip marks its sample "derived"), so that wording stays a typed
test string.

New must-fail tests (5 fail on the previous commit): a 400 echoing
toolu_insufficient_quota neither rotates nor changes on its way to the
caller; the wording mid-message does not rotate; the right wording in a
wrong envelope (rate_limit_error, api_error, missing type "error",
non-string message, array body) does not rotate; raw non-JSON text does
not rotate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(release): 2.65.0-vadymhimself.22

The fix commits on this branch inherited 2.65.0-vadymhimself.21, which
names an already-published tag, so release-version-line failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Gauge <gauge@agentradar-house.staging.agentradar.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant