Skip to content

[WRONG BRANCH] fix(anthropic): Claude identity on OAuth replays; a request-shape 429 no longer cools the account (+ #14, .18) - #15

Merged
vadymhimself merged 7 commits into
release/2.65.0-vadymhimself.15from
core-be-r4b/oauth-replay-identity
Nov 10, 2026
Merged

vadymhimself merged 7 commits into
release/2.65.0-vadymhimself.15from
core-be-r4b/oauth-replay-identity

Conversation

@vadymhimself

@vadymhimself vadymhimself commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Why

AgentRadar card HERMES-IDENTITY-NORMALISE (from ANDREW-CLAUDE-429, Vadym 20795: "We think it's because Andrew runs in Hermes"). The lead's live A/B test (11-10 18:13Z, through the same customer gateway, opus-5-5):

  • Body A, with the Claude Code identity system block, got HTTP 200.
  • Body B, with a Hermes-style system string, got HTTP 429.
  • B's 429 also cooled BOTH of that customer's Anthropic accounts, so every client on them was refused until the cooldown ended.

A non-Claude-Code client that sends the Anthropic Messages shape is replayed byte for byte (src/adapters/anthropic.ts source-replay branch). On a subscription (OAuth) account, that request reached Anthropic with the client's own system prompt and no Claude Code identity block. The translated path has always prepended that block.

Change

  1. Identity on OAuth replays. prependClaudeIdentity (src/adapters/anthropic.ts) runs on OAuth replays only. It prepends {type:"text", text: CLAUDE_CODE_SYSTEM_INSTRUCTION} unless a system block already starts with a Claude Code or Agent SDK identity line.

    • It checks every block, because Claude Code puts its billing-header block first.
    • A string system becomes [identity, {type:"text", text: system}], and a missing system becomes [identity].
    • API-key replays are unchanged.
    • Replay test sources that stand for Claude Code (claude-messages-endpoint, claude-opus-output-limit) now carry Claude Code's identity line, so they still assert a byte-for-byte replay.
  2. A request-shape 429 no longer cools the account. In rotateAnthropicAccountOn429 (src/oauth/anthropic-routing.ts), the account is cooled (and the request fails over) only on a real limit:

    • a unified window marked rejected (status, 5h-status or 7d-status); or
    • a 429 with no unified status at all (unchanged legacy behaviour, with or without a Retry-After).

    A 429 whose unified status says the account is still allowed is treated as a refusal of this one request. It is logged as cooldown=none (request refused, account not limited), it goes back to the caller, and the account and its other sessions stay as they were. Before this change, any such 429 benched the whole account for every client on it.

  3. Also carries [STALE: REBASE FROM NEW FORK] anthropic-pool: per-model-family cooldown on seven_day_<family> 429s; log limit headers on every 429 #14: the per-family 429 cooldown, the unified-header logging and the 2.65.0-vadymhimself.18 version commit, so one .18 release ships all of it. [STALE: REBASE FROM NEW FORK] anthropic-pool: per-model-family cooldown on seven_day_<family> 429s; log limit headers on every 429 #14 can close as superseded once this merges.

Agent's-eye: what a Hermes agent sees when its call is refused

The gateway passes Anthropic's refusal straight to the client.

Before .18, a Hermes call on a subscription account got:

HTTP 429
{"type":"error","error":{"type":"rate_limit_error","message":"Error"}}

Because that 429 cooled the account (and the failover account answered the same way), the agent's next calls, and those of every other client on those accounts, got:

HTTP 429  Retry-After: <seconds>
{"type":"error","error":{"type":"rate_limit_error","message":"All Anthropic OAuth accounts are temporarily rate-limited"}}

Answering as that agent: I don't know what happened (the first message is just "Error", and the second blames a rate limit that does not exist). I don't know who refused me. There is no instruction except to wait and retry, and retrying cannot work. That fails the agent's-eye test.

After .18:

  • The same Hermes call carries the Claude Code identity block, so it is answered (HTTP 200), as body A was in the A/B.
  • If Anthropic still refuses a request for its shape, the agent gets Anthropic's own 429 body above for that one call. The account is no longer cooled, so the gateway never adds the false "All Anthropic OAuth accounts are temporarily rate-limited", and other clients on the account are unaffected.
  • That residual bare "Error" body is Anthropic's text, not ours. Rewriting it into a message with a CTA is a separate change. It is not in this PR, because with the identity fix the refusal should no longer occur. The live A/B re-test after the roll checks this.

Tests

  • New tests/adapters/anthropic/anthropic-replay-oauth-identity.test.ts asserts the exact rewritten body for:
    • a string system;
    • an array system (its cache_control is kept);
    • no system;
    • an already-identified Claude Code system (billing header + identity), which is not doubled, and an Agent SDK-identified one;
    • an API-key replay, which is untouched.
  • New case in tests/adapters/anthropic/anthropic-ratelimit-headers.test.ts: a 429 with unified-status: allowed, with and without a Retry-After, on two accounts, returns no failover, and a header-less 429 afterwards still fails over to the second account. The allowed 429s leave no cooldown and stay eligible, and the log line reads cooldown=none (request refused, account not limited). The existing cases (a drained 5h window, a seven_day_opus claim, a Retry-After-only 429, and the tests/routing failover cases with no headers at all) still cool as before.
  • Lane run core-be-r4b-ocx15-pr3 on this branch and on the release tip 1f93f91 (.17), each running tsc plus tests/adapters/anthropic, tests/oauth, tests/responses, tests/claude-integration, tests/routing and the file-size ratchet: tsc clean on both; PR 6217 pass / 109 fail (6327 tests), base 6209 pass / 109 fail (6319). The failing test names are identical (105 unique, comm empty both ways): socket, sidecar and continuation tests that fail on this Mac on the base too, and that pass in CI on the base. Then core-be-r4b-ocx15-pr4 at this head: anthropic-ratelimit-headers, anthropic-replay-oauth-identity, tests/routing/always-on-429-failover, tests/routing/anthropic-quorum-cache and the file-size ratchet, 89 pass / 0 fail.
  • The two tests that prove the fix:
    • tests/adapters/anthropic/anthropic-replay-oauth-identity.test.ts › "a Hermes-style string system gets the identity block first".
    • tests/adapters/anthropic/anthropic-ratelimit-headers.test.ts › "an allowed unified status fails only the request, while a header-less 429 still fails over".

Roll

No roll in this PR. Rolling .18 to the gateways and the live A/B re-test happen after the 11-11 08:00 WITA deploy.

Eval added/updated: N/A deterministic (unit/integration test: tests/adapters/anthropic/anthropic-replay-oauth-identity.test.ts, tests/adapters/anthropic/anthropic-ratelimit-headers.test.ts)

🤖 Generated with Claude Code

vadymhimself and others added 5 commits November 10, 2026 22:23
… log limit headers on every 429

A 429 whose representative claim is seven_day_<family> (e.g. seven_day_opus) for a request of that
family now cools only (account, family): the account keeps serving its other models, and an Opus
request is refused only when every account is out of Opus. Every other 429 still cools the whole
account. Each 429 logs model, representative-claim, unified/5h/7d status, retry-after, reset and the
chosen cooldown, and the no-eligible-account line now counts cooled, re-auth and unusable accounts
instead of claiming they are all in cooldown.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…nd the cooldown end on each 429

Co-Authored-By: Claude Code <noreply@anthropic.com>
…line

The 429 log line now prints on every 429 (pinned included), so the pin e2e
counts the '; failing over to' line that only a real account switch prints.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…m block

A replayed Anthropic Messages body from a non-Claude-Code client (Hermes) reached
the subscription endpoint with only its own system prompt. The translated path
always prepends the identity block; the replay path did not. Prepend it on OAuth
when no system block already carries a Claude Code or Agent SDK identity line,
so Claude Code bodies (identity after the billing-header block) are not doubled.
Replay test sources that stand for Claude Code now carry its identity line.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Nov 10, 2026
@github-actions github-actions Bot changed the title fix(anthropic): give an OAuth source replay the Claude identity system block (+ #14) [WRONG BRANCH] fix(anthropic): give an OAuth source replay the Claude identity system block (+ #14) Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

⏳ DRAFT

  • wrong target branch (release/2.65.0-vadymhimself.15); retarget to dev.

What to do

  • Retarget this PR to dev — all contributions go to dev.

Its title has been prefixed with [WRONG BRANCH].
Automatic draft conversion failed (token cannot change draft status). Please convert this pull request to a draft manually. The required enforce-target check will keep failing until every issue above is resolved.

…r cools the account

An account is cooled (and the request fails over) only on a real limit: a unified window
marked rejected, or a Retry-After when Anthropic sent no unified status. A 429 whose
unified status is still "allowed", or with no limit evidence, is a refusal of this request's
shape (e.g. a replay without the Claude Code identity); before, it benched every client on
the account. Also moves prependClaudeIdentity next to the identity constant in
src/oauth/anthropic.ts so src/adapters/anthropic.ts stays within its file-size cap.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@vadymhimself vadymhimself changed the title [WRONG BRANCH] fix(anthropic): give an OAuth source replay the Claude identity system block (+ #14) fix(anthropic): Claude identity on OAuth replays; a request-shape 429 no longer cools the account (+ #14, .18) Nov 10, 2026
@vadymhimself
vadymhimself marked this pull request as ready for review November 10, 2026 19:33
@github-actions github-actions Bot changed the title fix(anthropic): Claude identity on OAuth replays; a request-shape 429 no longer cools the account (+ #14, .18) [WRONG BRANCH] fix(anthropic): Claude identity on OAuth replays; a request-shape 429 no longer cools the account (+ #14, .18) Nov 10, 2026
@github-actions
github-actions Bot marked this pull request as draft November 10, 2026 19:33
Only a 429 whose unified status is present and says allowed skips the cooldown;
a header-less 429 keeps the legacy cooldown and failover the routing tests expect.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@vadymhimself
vadymhimself marked this pull request as ready for review November 10, 2026 23:08
@vadymhimself
vadymhimself merged commit b849dba into release/2.65.0-vadymhimself.15 Nov 10, 2026
58 of 65 checks passed
vadymhimself added a commit that referenced this pull request Nov 11, 2026
…in, still cooling nothing (.20) (#18)

#15 stopped cooling on a 429 whose unified status says "allowed". That was right, because such a 429 is a refusal of the request's shape and cooling on it benched every client on the account. But it also stopped the failover, so the 429 went back to the caller. A combo then moves to its next provider and model, which loses more cache than an account switch. 3f6b2f3 had fixed exactly that bounce.

The allowed 429 now moves this request to another account and cools no account. Rejected and header-less 429s cool and fail over as before. Pinned requests still never switch. Known ceiling: when every account repeats the refusal, the request can alternate accounts up to ANTHROPIC_POOL_MAX_FAILOVERS_PER_REQUEST times before the 429 returns. Refused calls bill no tokens.

Co-authored-by: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant