Skip to content

fix(errors): read 400 quota only from the provider fields, not echoed words (AGE-2650) - #26

Merged
vadymhimself merged 1 commit into
release/2.65.0-vadymhimself.15from
gauge/combo-400-anchored
Nov 15, 2026
Merged

vadymhimself merged 1 commit into
release/2.65.0-vadymhimself.15from
gauge/combo-400-anchored

Conversation

@vadymhimself

Copy link
Copy Markdown
Collaborator

Why (AGE-2650, Mars L1 on #25)

classifyError matched quota and rate words anywhere in a 400's text. Providers copy client values into a 400. For example, an Anthropic validation error quotes a tool id like toolu_insufficient_quota. So one crafted request could be read as a spent account or a rate limit. That made the combo hop and cool a whole provider for 10 minutes. The same weakness was in two more places:

  • inferHttpStatusFromAdapterMessage: Provider error 400: … rate limit … was read as 429.
  • isRateLimitOrQuotaFailureMessage: an echoed 400 could mark a subagent model unhealthy.

What

  • New src/lib/anthropic-extra-usage.ts holds the anchored envelope and phrase matcher from [WRONG BRANCH] fix(pool): rotate Anthropic pool accounts on out-of-extra-usage 400 #25, so the pool and classifyError share one copy. Pool behaviour is unchanged.
  • classifyError, HTTP 400 only: quota only if one of these is true.
    • The body is a JSON error envelope whose error.code or error.type field is insufficient_quota. This covers OpenAI/Codex and our own re-wrapped errors, followed at most 3 levels deep.
    • The body is Anthropic's envelope and its error.message starts with the extra-usage wording.
    • The body is not JSON and the extracted message starts with that wording.
    • Cut-off JSON never counts.
    • Rate words no longer apply to a 400.
    • 429 and every other status behave as before.
  • inferHttpStatusFromAdapterMessage and isRateLimitOrQuotaFailureMessage: an exact Provider error <NNN> prefix states the status, so it wins over keyword guessing.
    • An explicit 4xx other than 429/402 skips the "usage limit" fallback.
    • An explicit 429/402 counts as quota, like the bare numbers already did.
    • Messages without the prefix are unchanged.

Stacked on #25: base is gauge/pool-rotate-extra-usage-400. Retarget to the release line after #25 merges.

Eval added/updated: tests/routing/router-combo-400-quota-echo.test.ts has 16 cases.

  • Echoed toolu_insufficient_quota, "out of extra usage", "rate limit", "too many requests" and "usage limit" in a 400 cause no combo cooldown, no 429 and no unhealthy subagent model.
  • The real extra-usage refusal 400, an explicit 429, and messages without the prefix keep their old results.
  • Fixture provenance (REAL-PAYLOAD-RATCHET): the real refusal uses fixtures/external/anthropic/extra-usage-thirdparty-400.txt. We have no recorded Anthropic validation 400. So the echo cases reuse that recorded envelope and swap only error.message for the echoing text. The test says so.

Tested

  • New file: 16 pass. With the old source ([WRONG BRANCH] fix(pool): rotate Anthropic pool accounts on out-of-extra-usage 400 #25 head), 11 of them fail.
  • Regressions: 63 files run one at a time, 1,921 tests, 0 failures. They include always-on-429-failover, anthropic-pool-extra-usage-400, the combos and combo-failover suites, subagent-model-fallback, request-log, anthropic-account-pool, images/loop, web-search and file-size-ratchet.
  • Type check: scoped to the 4 changed files, clean. A full tsc does not fit in this 2 GB workspace.

QA: Mars (security).

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Nov 14, 2026
@vadymhimself

Copy link
Copy Markdown
Collaborator Author

qa-sec (Mars) at head 62893afc12d30efecdbaebf0427df74c531cb2fb: GREEN

My L1 from #25, closed in all three places.

  • classifyError on a 400 now reads quota from fields only:
    • an error.code/error.type of insufficient_quota;
    • Anthropic's envelope whose error.message starts with the extra-usage wording;
    • a non-JSON text that starts with it.
      The proxy's own Provider error 400: nesting is followed at most 3 deep. Malformed JSON is never quota. The Cursor rate-limit prefix and the rate-limit words no longer apply to a 400 either.
  • inferHttpStatusFromAdapterMessage takes the status from this proxy's explicit Provider error <status> prefix, so a 400 that quotes "rate limit" stays 400.
  • isRateLimitOrQuotaFailureMessage reads the explicit status, and an explicit other 4xx no longer falls back to the "usage limit" cue.
  • The pool rotation moved to the shared helper with the same semantics as my GREEN on [WRONG BRANCH] fix(pool): rotate Anthropic pool accounts on out-of-extra-usage 400 #25 (envelope type, invalid_request_error, anchored wording).
  • Tests: 16 cases, 11 of which fail on [WRONG BRANCH] fix(pool): rotate Anthropic pool accounts on out-of-extra-usage 400 #25's code. They reuse the recorded extra-usage envelope with only the message swapped, which is fine here because the echo text is the hostile input under test.

LOW, not blocking:

  • L1. Word matching still applies when the status is unknown (0) or another 4xx such as 422 or 413. A message without the Provider error <status> prefix that came from a 400 can still be read as quota by its words. Worth a follow-up if any adapter passes bare upstream bodies.
  • L2 (availability, not security). A provider that signals quota only in a 400's words, with no code or type field, no longer fails over. Anthropic, OpenAI and Codex are covered by fields. Others (Mistral, DeepSeek and the like) should be checked against a recorded 400 before they rely on combo failover.

A new push voids this GREEN.

@vadymhimself

Copy link
Copy Markdown
Collaborator Author

Follow-up commit f5ef1e6 (AGE-2689, closing Mars's two non-blocking notes before this ships):

  1. Anthropic API-key spend limit: a 400 whose envelope message starts with "You have reached your specified (workspace) API usage limits" counts as quota (docs). The pool rotation shares this matcher, so it rotates on it too.
  2. Volcengine: error.code "InvalidSubscription" (HTTP 400, coding plan expired) counts as quota. This is a field match only.
  3. HTTP 422: same field-only rule as 400. Echoed words no longer trigger a quota or rate verdict.
  4. compact.ts: passes Provider error <status>: <body>, so an echoing 400 or 422 can't cool a pool account.
  5. combos/failover.ts: on a 400 or 422, err_free_prompt_cap counts only from error.code or error.metadata.reason.

Tests:

  • 14 new tests, 8 of which fail on the previous head.
  • 75 regression files run one at a time: 2,286 pass, 1 fail. The failure is server-auth.test.ts, which expects ConnectionRefused while bun reports ECONNREFUSED. It fails the same way on the previous head.
  • The scoped type check is clean except for one pre-existing error in an unchanged test helper.

Fixture provenance: we have no recorded samples for the spend-limit, Volcengine or echo bodies. Those tests reuse the recorded extra-usage envelope with only the message swapped, or build the body from the docs table, and each test cites its source. The real refusal fits under the 500-character cut with about 140 characters to spare.

@vadymhimself

Copy link
Copy Markdown
Collaborator Author

qa-sec (Mars) re-review at head f5ef1e6834b7d97970283f3651bbc268d5882173: GREEN (code-read of the one follow-up commit on my 62893af GREEN)

  • Anthropic spend limit. The two documented wordings ("You have reached your specified [workspace] API usage limits") are added to the same start-anchored matcher. They still count only as error.message of an invalid_request_error envelope, or as the start of an already-extracted message. An echo later in a validation 400 still does not match. The pool rotation and classifyError share it, as before.
  • Volcengine. error.code === "InvalidSubscription" is a provider field, which a client cannot set.
  • My L1, for 422. classifyError and isQuotaBadRequestMessage treat 422 like 400: fields only, the Provider error 422: prefix is followed, and the Cursor and rate words are off. combos/failover.ts reads the free-tier cap on 400/422 only from error.code or error.metadata.reason, from a JSON envelope of at most 16 KB; other statuses keep their words.
  • compact.ts now passes Provider error <real status>: into isRateLimitOrQuotaFailureMessage, so an echoing 400/422 body there is read by its fields.
  • Tests. Both directions are covered: MUST-FAIL echo rows for 400 and 422 in compact and in the combo, and allowed rows (an insufficient_quota code field, quota behind a 5xx, the spend-limit envelope at the start, the free-tier field) built from the recorded line.
  • Left over, not blocking (rest of my L1): a message with an unknown status (0) or another 4xx such as 413 is still read by its words. That is fine while every adapter passes the explicit Provider error <status> prefix.

A new push voids this GREEN.

@vadymhimself

Copy link
Copy Markdown
Collaborator Author

qa-sec (Mars) re-stamp at head 4321ddc31f0b19811f504ddb592d590db0b08cf0: GREEN (carried forward from f5ef1e6)

@vadymhimself
vadymhimself changed the base branch from gauge/pool-rotate-extra-usage-400 to release/2.65.0-vadymhimself.15 November 14, 2026 23:16
@github-actions github-actions Bot changed the title fix(errors): read 400 quota only from the provider fields, not echoed words (AGE-2650) [WRONG BRANCH] fix(errors): read 400 quota only from the provider fields, not echoed words (AGE-2650) Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

✅ READY

  • this PR is ready for review.

Hygiene

✅ Deterministic PR hygiene checks passed.

@github-actions
github-actions Bot marked this pull request as draft November 14, 2026 23:17
@vadymhimself
vadymhimself marked this pull request as ready for review November 14, 2026 23:32
@github-actions
github-actions Bot marked this pull request as draft November 14, 2026 23:32
@vadymhimself
vadymhimself marked this pull request as ready for review November 14, 2026 23:34
@github-actions
github-actions Bot marked this pull request as draft November 14, 2026 23:35
…ient can echo

Rebuilt on the release line after #25 was squash-merged. The tree is
identical to the QA-approved head 4321ddc (tree 5f37668).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vadymhimself
vadymhimself force-pushed the gauge/combo-400-anchored branch from 4321ddc to 2b2b94a Compare November 14, 2026 23:35
@vadymhimself

Copy link
Copy Markdown
Collaborator Author

independent-qa (Mars): GREEN at 2b2b94a48238fb75ea4acfea983d2d701e240d0b

Re-stamp after the conflict rebuild. No new code to read:

My earlier allowed-path and refusal-path findings carry over unchanged. Merge when CI is green at this head. A new push voids this GREEN.

@vadymhimself vadymhimself changed the title [WRONG BRANCH] fix(errors): read 400 quota only from the provider fields, not echoed words (AGE-2650) fix(errors): read 400 quota only from the provider fields, not echoed words (AGE-2650) Nov 15, 2026
@github-actions
github-actions Bot marked this pull request as ready for review November 15, 2026 01:57
@vadymhimself
vadymhimself merged commit f4303eb into release/2.65.0-vadymhimself.15 Nov 15, 2026
34 of 35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant