diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 74f5dfb8837..034232e333f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -820,6 +820,9 @@ jobs: - name: Install dependencies run: bash scripts/ci/bun-install-frozen.sh . gui + - name: Gateway quota and cache guardrails + run: bun run test:gateway-quota + - name: GUI lint if: needs.changes.outputs.gui == 'true' run: | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a5944a82f56..046d64a97ac 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,6 +8,7 @@ Thanks for helping with opencodex. - Current maintainer invariants live in [`structure/`](./structure); start at [`structure/INDEX.md`](./structure/INDEX.md) - Maintainer roles and merge policy live in [`MAINTAINERS.md`](./MAINTAINERS.md) - Attribution for work landed through a maintainer carry lives in [`CREDITS.md`](./CREDITS.md) +- AgentRadar fork maintenance and offline quota checks live in [`FORK.md`](./FORK.md) - Planning and investigation notes live in [`devlog/`](./devlog); older retired notes remain in git history - PR screenshots go in the pull request description, never on your branch: drag the image into the description editor, or, with push access, commit it to the `pr-assets` branch and link it by commit SHA diff --git a/FORK-DELTA.md b/FORK-DELTA.md new file mode 100644 index 00000000000..62d312871be --- /dev/null +++ b/FORK-DELTA.md @@ -0,0 +1,44 @@ +# AgentRadar refresh compatibility inventory + +Compared hosted release `v2.65.0-vadymhimself.27` (`d0f166aca52d0cbe3dd132601ae8798aba8e002f`) with upstream `6bae884cbc66306cfbf4e0c2d50ba19fb79305a3` on 2026-11-18. All 99 raw fork-only commits were assigned to behavior groups. Because old port commits squash several concerns, parity was proven per behavior and per test case, not per commit. + +## How parity was proven + +1. **AR test oracle.** Every test case the AR line added (369) or changed (103) relative to its fork base was extracted from `.27` and run against pure upstream: 96 passed unchanged. Each of the other 376 cases was then resolved with evidence: + - **Carried:** the `.27` test, with only import paths adapted and assertions unchanged, fails on upstream and passes on this branch. + - **Equivalent:** an upstream test asserts the same behavior, or the `.27` test passes unchanged. + - **Replaced-better:** upstream deliberately changed the behavior, with evidence that the replacement is safe and equal or better. + - **Not carried:** the behavior is unreachable for AR traffic or would cause harm. The reason is recorded. +2. **Full upstream suite, both modes.** With `OPENCODEX_AGENTRADAR` unset, the failures are a subset of the pristine-upstream environment failures (same machine, same runner). With `OPENCODEX_AGENTRADAR=1`, the production mode, every failure that does not occur without the flag was triaged. Real hosted regressions were fixed with mutation-checked tests. The remaining flag-only failures are upstream tests that encode upstream-only behavior which hosted mode deliberately replaces with `.27` behavior; each group cites the `.27` behavior. +3. **AR gate.** `bun run test:gateway-quota` runs the AR contract tests in the existing CI `gates` job. + +Working ledgers (per case, with commands and evidence) were kept outside the repository during the port; their dispositions are summarized below. + +## Hosted contracts carried (gated by `OPENCODEX_AGENTRADAR=1` where upstream deliberately differs) + +- **Per-request account pin** (`0c1162dae7` … `fbcc5b2285`): Messages, Chat and `count_tokens` only; refused when malformed or when hosted mode is off. Exact pinned account through every lane, including native Messages. No process-wide selection commit, no rotation off the pin, helpers refused or bound to the pin. The served-account header and the `[ar-acct]` line are written only after a physical send. Real loopback gateway e2e on fake and real-host base URLs; the real-host variant asserts that the native lane served the request. +- **Canonical Anthropic exact relay**: `https://api.anthropic.com/v1/messages` routes take the native lane. The caller's body is forwarded unchanged apart from the wire model and pooled-account metadata. No `ttl` injection, `context_management` is kept, caller betas pass. Upstream SSE and JSON bytes are relayed, with only Content-Type and a bounded Retry-After as headers. A strict SSE tap stops at the first terminal frame and records a root error as failed 502 and a stall as 504. Malformed frames fail before relay. Every rule is scoped to the canonical host; third-party Anthropic-compatible routes keep upstream handling. +- **Deterministic keyed placement** (`0428964be2`, `b60e434917`, `0c4ffad3e3`): rendezvous-hash seat ahead of restored affinity, with the hash runner-up on failover. +- **Pool policy**: a 16-account funded roster, always intersected with the request send budget. A header-less 429 cools the account for 60 seconds and rotates, but is reported to the client as a retryable rate limit, not usage-limit. Cached seven-day exhaustion is a hard hold, with no lazy probe. An org-disabled account rotates and is marked needs-reauth. Terminal re-asks rotate only on 429, quota-400 and org-disabled 403. +- **Quota and refusal handling for all installs**: account rotation on the exact extra-usage and spend-limit 400 envelopes (`45776369ce`). Field-only 400/422 quota evidence (`f4303eb19c`). Family-claim 429s. Pool 401 replay counted at the physical send. Credential-recovery attempt split (upstream #8882 / #8900). +- **Hosted combo policy**: verbatim final Anthropic errors, a 10-minute target cooldown cap (`d3ea59ccb9`) and a quorum-guarded stale-cooldown lift. Spend-limit and free-tier field hops. Combo aliases may carry typed tools to Anthropic targets. Restricted `web_search` keeps its translated fallback. +- **Observability**: the gated usage line (`8ca6f61ec5`) and the privacy-bounded refusal-shape log (`fae0ae4736`). Analytics `physicalBreakdown` and `redAlerts` plus the physical-attempt accounting used by summaries, health, the lab and route explainability. `p` Anthropic account labels (`326bdfd7f7`), `comboTargetAdvanced`, cache TTL attribution, the `codex-account-retry` kind, and request-incompatible route traces. +- **Providers and state**: the `xai-apikey` identity (`de6381aa3b`), lower-only response-store environment bounds (`db4cf5cbef`), credit-journal expiry (`326e1a77b7`, co-authored by Gauge), Kimi `usages` precedence (`aa80674b33`), quota explanation (`af51f855b4`), the 20-minute Antigravity paste window (`d6d040ed32`), a 60-second API-key 429 cap, `.27` plan vocabulary for hosted `?quota=1`, and `--code` as an alias of `--headless`. +- **Claude ingress**: hosted exact `count_tokens` (pin-respecting; the `.27` bug that counted on an unpinned pool account is fixed). `prompt_cache_key` on session and parent session. The 5-minute cache default in hosted mode. Image 413 shrink on the native lane. Native-only client tools are refused before send. + +## Replaced by upstream (do not reapply) + +Native Messages replay (#7374), the server-tool/cache/usage bundle (#7225), per-account recovery usage (#7224), WebSocket/key-429/usage (#7389), redirect refusal (#8868), funded roster (#8871), cache boundary (#8212), native Codex refresh/revocation (#6515/#6523), combo admission ladder (#7355), Claude session-id extraction, and modern catalog rows (#8893; upstream prices kept). Old port-repair commits, version stamps and old manifests are not runtime value. + +## Not carried, with reason + +- Physical `count_tokens` for non-hosted installs: upstream estimates locally and spends nothing. +- The `.27` lazy weekly usage probe: replaced by the cached gate plus in-request rotation, which avoids the stale-observation race. +- The 4,096-frame inspector cap: it disabled accounting after one bursty read. +- The translated-path citation carrier: unreachable for AR after the canonical native lane. +- Fable temperature 0.3: Anthropic rejects it. +- Old fork release publisher: incompatible with upstream CI policy; see `FORK.md`. + +## Known pre-existing failures + +The suites listed in `FORK.md` fail identically on pristine upstream on the same machine: Rust ocx-child link (Xcode licence), service and route-to-home sandbox ownership, the catalog `models_cache.json` serve-options row, cli-connect-readiness, codex composed acceptance, update-stop-first, bun-install-frozen and test-git-isolation. diff --git a/FORK-VERIFICATION.md b/FORK-VERIFICATION.md new file mode 100644 index 00000000000..a777ab0e19b --- /dev/null +++ b/FORK-VERIFICATION.md @@ -0,0 +1,46 @@ +# AR rebuild validation — 2026-11-19 + +## Inputs and boundaries + +- Fixed upstream base: `6bae884cbc66306cfbf4e0c2d50ba19fb79305a3` (upstream package 2.100.0). +- Compatibility reference: `v2.65.0-vadymhimself.27`, `d0f166aca52d0cbe3dd132601ae8798aba8e002f`. +- Superseding build version: `2.100.0-vadymhimself.2`. The initial `.1` artifact is on HOLD and must not be installed. +- Runtime: pinned Bun 1.4.4. Tests use isolated HOME/OPENCODEX_HOME, synthetic credentials and loopback providers. Running laptop, Mini and hosted gateways were not replaced or deployed. +- Git history was not rebased or force-pushed; this is a fresh upstream-derived branch with the AR overlay reapplied. + +## Evidence + +| Check | Result | Meaning | +| --- | --- | --- | +| Pristine upstream full suite | 69,990 pass / 753 skip / 33 fail | Same-machine baseline, not a green full suite. | +| Rebuild full suite, hosted flag unset | 70,276 pass / 753 skip / 32 fail | Every failing test also fails on pristine upstream; no new failure names. | +| Rebuild full suite, hosted flag enabled | 69,985 pass / 753 skip / 323 fail | Hosted-only failures were grouped and reviewed against the intentional `.27` contract. These include upstream-only expectations for rewriting native bytes, 502 vs 504, six-account rosters, one-hour cache TTL, and sanitizing canonical server-tool history. Not a green full-suite result. | +| Final repaired AR gate | 534 pass / 0 fail across 29 files | Real-handler budget, cache-prefix, account pin, native fidelity, quota evidence, accounting, count-tokens and final boundary regressions. The metadata-only `.2` version stamp follows these checks. | +| Typecheck, structure, privacy, diff check | Pass | Final version-stamped source. | +| Layout and file-size contracts | 38 pass / 0 fail | No baseline caps increased. | +| Dashboard/package build | Pass | Current upstream dependency versions and lockfiles retained. | +| Extracted package smoke | Pass | CLI reports the fork version, health endpoint returns 200, dashboard serves built HTML from an isolated home. | +| Live Claude cache parity | Blocked | User-authorized laptop probes hit Anthropic 429s. Waterfall used OpenAI; those cache hits do not establish Claude parity. | +| GitHub Actions | Unavailable | The fork rejects workflow dispatch with Actions disabled. No successful GitHub CI is claimed. | + +The complete suites preceded the final small sidecar unused-reservation refund and version stamp. Those final changes were validated by the final AR gate, including Error and AbortError refund regressions, and all static/layout/file-size checks. The package smoke uses the final packed files. + +## Compatibility method + +The committed fork contained 99 fork-only commits, many of them mixed historical port squashes. A semantic inventory classified those changes. An additional oracle extracted 369 fork-added test cases and 103 modified test cases relative to the common base. Port owners recorded whether each behavior was carried, equivalent upstream, intentionally replaced, or unreachable. Literal `.27` tests that depend on removed internal modules were adapted to current seams, with fixture/assertion changes recorded in the working ledgers. The current contract and intentional replacements are summarized in `FORK-DELTA.md`. + +Hosted-only full-suite failures were separately triaged across three disjoint slices. Confirmed defects found during that pass were fixed with focused regressions: canonical-only relay scope; response-header filtering; local error-frame identity preservation; combo-alias typed-tool eligibility; anchored quota evidence; classification before verbatim error presentation; retryable headerless 429s; restricted-search fallback; operator advisor opt-out; pinned count-tokens selection and unavailable-route fallback. Existing explicitly configured third-party native passthrough remains supported. + +This is evidence for the exercised contracts, not a guarantee that every possible production configuration has been tested. The hosted deployment's actual secret-bearing configuration was not copied into the test tree. Read `FORK.md` before adapting the release pipeline or cutting over an existing gateway. + +## Known environment failures + +The paired baseline identifies local Rust/Xcode licence failures, sandbox/service ownership and route-intent fixtures, missing Codex catalog cache, connected-runtime probe expectations, composed Codex acceptance and update-stop-first. Some additional timing tests fail in broad runs and pass alone. These are disclosed, not retried into a blanket green claim. + +## Delivery boundary + +The build is a tarball candidate, not an npm publication, GitHub release tag, merge, deployment or installed runtime replacement. The old fork publisher requires a successful exact-SHA push-CI event that current upstream intentionally no longer creates. No gate was weakened to disguise that incompatibility. + +## Final boundary-review correction + +A bounded final independent review found two blockers after the initial `.1` package: the strict SSE EOF tail bypassed frame validation, and hosted OAuth token counts could dispatch a captured credential after its account changed during pacing. The initial artifact was explicitly marked HOLD and PR #46 converted to draft. The superseding build validates EOF tails before enqueue and revalidates the same OAuth authority immediately before physical count dispatch. Controlled regressions cover malformed/non-object/invalid-UTF-8 EOF tails and queued pause/removal/credential replacement/cancellation. The token-count response also bounds Retry-After. Focused results for this repair are recorded in the PR; the full-suite results above predate the repair. diff --git a/FORK.md b/FORK.md new file mode 100644 index 00000000000..3677bd0cbf2 --- /dev/null +++ b/FORK.md @@ -0,0 +1,33 @@ +# AgentRadar gateway overlay + +This branch starts from upstream `lidge-jun/opencodex` development history, not from the old fork's accumulated port commits. The refresh baseline is `6bae884cbc66306cfbf4e0c2d50ba19fb79305a3` (2026-11-18). The source gateway release for the compatibility inventory is `v2.65.0-vadymhimself.27` (`d0f166aca52d0cbe3dd132601ae8798aba8e002f`). + +What was carried, replaced or dropped is in [FORK-DELTA.md](./FORK-DELTA.md). Exact validation results and their limits are in [FORK-VERIFICATION.md](./FORK-VERIFICATION.md). + +Hosted gateways run with `OPENCODEX_AGENTRADAR=1`. Every AR behavior that deliberately differs from upstream is behind that flag. Without it, this branch behaves like upstream except for generic defect fixes that apply to all installs. + +## Maintenance contract + +- Keep upstream implementation, dependency versions, generated catalogs and release machinery unless an explicitly documented hosted requirement needs an overlay. A different commit hash does not establish a missing fix: upstream often lands fixes through squash merges or maintainer carries. +- Keep hosted account binding, serving-account visibility and deterministic session placement in a small, tested overlay. Do not replace upstream routing or retry implementations wholesale to preserve these entry points. +- Preserve upstream request-wide physical-send accounting, cancellation, refusal evidence, credential ownership and committed-output boundaries. The hosted 16-account roster is always intersected with the request send budget. +- To branch new gateway work, start from this line. PRs against the old `release/2.65.0-vadymhimself.*` branches are stale and must be re-cut here. +- Keep every genuinely missing generic fix in a separately attributable change so it can be removed when the upstream implementation lands. Record the upstream PR and the original fork commit. +- Refresh from a fixed upstream SHA in a new isolated worktree. Compare the overlay against current upstream behavior, drop equivalents and apply only the remaining changes. Do not rewrite a published release branch, rebase open upstream PRs, force-push, or carry the old baseline's generated files and dependency manifests over newer upstream versions. +- Do not copy local credentials, account databases, logs or deployment configuration into a refresh worktree. Synthetic credentials and loopback mock providers are sufficient for compatibility tests. + +## Gateway checks + +Run `bun run test:gateway-quota` in addition to the affected suites and the standard typecheck, structure and privacy checks. The command is included in the existing CI `gates` job without changing its triggers, permissions or runner selection. + +The gateway gate exercises the AR account pin through real loopback gateway processes, exact canonical Anthropic relay, hosted pool and combo policy, physical-send bounds, funded-roster limits, sidecar reservation ownership, quota-echo resistance, `count_tokens`, analytics and usage logs. Each hosted test sets the flag itself, so the gate runs in the existing CI job without it. An offline pass proves the asserted wire/accounting contracts, not a live Anthropic cache-hit rate or production cost reduction. Missing GitHub checks are not a pass. + +For the next refresh, also run the full suite with `OPENCODEX_AGENTRADAR=1` and triage every failure that does not occur without the flag. Upstream tests encode upstream behavior, so a hosted failure is acceptable only when it matches a documented `.27` behavior. Pre-existing environment failures on this machine: Rust ocx-child link (Xcode licence), service and route-to-home sandbox ownership, the catalog `models_cache.json` serve-options row, cli-connect-readiness, codex composed acceptance, update-stop-first, bun-install-frozen and test-git-isolation. + +## Release boundary + +Preparing this branch does not replace the running gateway or authorize publication. Keep the current laptop build and hosted release line intact until the candidate is validated and deployment is explicitly approved. + +The old fork publishes through `.github/workflows/fork-release.yml` after successful exact-SHA push CI on its release line. Current upstream deliberately has no push CI trigger and uses a different release-evidence contract. These workflows are not interchangeable. The refresh does not copy the old publisher, enable Actions, weaken its successful-CI requirement, add release triggers or publish a tag. Release integration requires a separate reviewed change before any cutover. + +The initial publication of a new `release/2.100.0-vadymhimself` branch was rejected by the existing required `ci` rule while Actions was disabled. The pristine reference is `baseline/ar-upstream-2026-11-18`; the candidate overlay is `port/ar-on-upstream-2026-11-18`. The rebuild PR targets that reference for a bounded diff. No release-branch protection was bypassed or changed. diff --git a/desktop/src-tauri/Cargo.lock b/desktop/src-tauri/Cargo.lock index e8372b53288..9eaa7580eaa 100644 --- a/desktop/src-tauri/Cargo.lock +++ b/desktop/src-tauri/Cargo.lock @@ -2633,7 +2633,7 @@ dependencies = [ [[package]] name = "opencodex-desktop" -version = "2.100.0" +version = "2.100.0-vadymhimself.2" dependencies = [ "base64 0.22.1", "dbus", diff --git a/desktop/src-tauri/Cargo.toml b/desktop/src-tauri/Cargo.toml index cc960d1bdc9..901345b3f27 100644 --- a/desktop/src-tauri/Cargo.toml +++ b/desktop/src-tauri/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "opencodex-desktop" -version = "2.100.0" +version = "2.100.0-vadymhimself.2" description = "OpenCodex desktop shell" authors = ["OpenCodex contributors"] license = "MIT" diff --git a/desktop/src-tauri/tauri.conf.json b/desktop/src-tauri/tauri.conf.json index a9325d3a6fd..b8c12e3f7e2 100644 --- a/desktop/src-tauri/tauri.conf.json +++ b/desktop/src-tauri/tauri.conf.json @@ -1,7 +1,7 @@ { "$schema": "https://schema.tauri.app/config/2", "productName": "OpenCodex", - "version": "2.100.0", + "version": "2.100.0-vadymhimself.2", "identifier": "com.opencodex.desktop", "build": { "frontendDist": "../ui", diff --git a/docs-site/src/content/docs/es/getting-started/quickstart.md b/docs-site/src/content/docs/es/getting-started/quickstart.md index 77fa5fe88fb..47f49e3e3ad 100644 --- a/docs-site/src/content/docs/es/getting-started/quickstart.md +++ b/docs-site/src/content/docs/es/getting-started/quickstart.md @@ -23,7 +23,7 @@ ocx init `ocx init` te guía por estos pasos: -1. **Elegir un proveedor**: selecciona uno de los 108 preajustes del registro integrado o `custom` para introducir una URL base y un adaptador. +1. **Elegir un proveedor**: selecciona uno de los 109 preajustes del registro integrado o `custom` para introducir una URL base y un adaptador. 2. **Clave API**: pega una clave o indica una variable de entorno como `${ANTHROPIC_API_KEY}`. 3. **Modelo predeterminado**: para proveedores con clave, locales o personalizados, acepta el preajuste o introduce un identificador de modelo. 4. **Puerto del proxy**: `10100` de forma predeterminada. diff --git a/docs-site/src/content/docs/es/guides/api-key-catalog.md b/docs-site/src/content/docs/es/guides/api-key-catalog.md index 46909b6fe8d..a04485c5879 100644 --- a/docs-site/src/content/docs/es/guides/api-key-catalog.md +++ b/docs-site/src/content/docs/es/guides/api-key-catalog.md @@ -7,7 +7,7 @@ Esta página contiene la sección 3 de la guía de [Proveedores](/es/guides/prov con clave API incluidos y las notas específicas de cada proveedor. Los modos de autenticación, los inicios de sesión de cuentas y los proveedores locales siguen en esa página. -opencodex incluye 108 preajustes integrados: 87 con clave, 17 OAuth, tres locales y uno predeterminado de reenvío a ChatGPT. El selector **Añadir proveedor** abre el formulario del proveedor con clave, valida la credencial y la guarda; cada proveedor tiene su propia validación. Entre ellos: +opencodex incluye 109 preajustes integrados: 88 con clave, 17 OAuth, tres locales y uno predeterminado de reenvío a ChatGPT. El selector **Añadir proveedor** abre el formulario del proveedor con clave, valida la credencial y la guarda; cada proveedor tiene su propia validación. Entre ellos: **ClinePass** utiliza una clave API de Cline con el [catálogo oficial de suscripción](https://docs.cline.bot/getting-started/clinepass) y el [punto final de Chat Completions](https://docs.cline.bot/api/chat-completions), operados por Cline Bot Inc. bajo [los términos de Cline](https://cline.bot/tos). Un id enrutado como `cline-pass/cline-pass/kimi-k3` es intencional: el primer segmento selecciona el proveedor de opencodex, mientras que `cline-pass/kimi-k3` es el identificador de modelo completo que se envía aguas arriba. La cuota de ClinePass se comparte por cuenta entre límites rotativos de 5 horas, semanales y mensuales. Una sonda en vivo del 2026-08-13 verificó que todos los modelos estáticos de ClinePass aceptan `low`, `medium`, `high`, `xhigh` y `max` en el límite de entrada de la pasarela. opencodex conserva los niveles solicitados; cualquier normalización específica del backend sigue siendo responsabilidad de ClinePass. diff --git a/docs-site/src/content/docs/fr/getting-started/quickstart.md b/docs-site/src/content/docs/fr/getting-started/quickstart.md index 67917b74d53..fde2a097101 100644 --- a/docs-site/src/content/docs/fr/getting-started/quickstart.md +++ b/docs-site/src/content/docs/fr/getting-started/quickstart.md @@ -13,7 +13,7 @@ ocx init `ocx init` vous accompagne dans les étapes suivantes : -1. **Choix d’un fournisseur** — sélectionnez l’un des 108 préréglages intégrés au registre, ou `custom` pour saisir une +1. **Choix d’un fournisseur** — sélectionnez l’un des 109 préréglages intégrés au registre, ou `custom` pour saisir une URL de base et un adaptateur. 2. **Clé API** — collez une clé ou référencez une variable d’environnement telle que `${ANTHROPIC_API_KEY}`. 3. **Modèle par défaut** — pour les fournisseurs clés, locaux et personnalisés, acceptez le préréglage ou saisissez un identifiant de modèle. diff --git a/docs-site/src/content/docs/fr/guides/providers.md b/docs-site/src/content/docs/fr/guides/providers.md index 7a49dbd4227..5e7b9ba7169 100644 --- a/docs-site/src/content/docs/fr/guides/providers.md +++ b/docs-site/src/content/docs/fr/guides/providers.md @@ -310,7 +310,7 @@ existante n'est pas concernée. ## 3. Catalogue des clés API -opencodex fournit 108 préréglages intégrés : 87 à clé, 17 OAuth, trois locaux et un préréglage par défaut de +opencodex fournit 109 préréglages intégrés : 88 à clé, 17 OAuth, trois locaux et un préréglage par défaut de transfert ChatGPT. Dans le tableau de bord, le sélecteur **Ajouter un fournisseur** ouvre le tableau de bord du fournisseur à clé, valide la clé et l'enregistre ; la validation dépend du fournisseur. Parmi les entrées notables : diff --git a/docs-site/src/content/docs/getting-started/quickstart.md b/docs-site/src/content/docs/getting-started/quickstart.md index a1d830d4634..45a1813cbb1 100644 --- a/docs-site/src/content/docs/getting-started/quickstart.md +++ b/docs-site/src/content/docs/getting-started/quickstart.md @@ -21,7 +21,7 @@ ocx init `ocx init` walks you through: -1. **Pick a provider** — choose one of the 108 built-in registry presets or `custom` to type a base +1. **Pick a provider** — choose one of the 109 built-in registry presets or `custom` to type a base URL and adapter. 2. **API key** — paste a key, or reference an environment variable like `${ANTHROPIC_API_KEY}`. 3. **Default model** — for key, local, and custom providers, accept the preset or enter a model id. diff --git a/docs-site/src/content/docs/guides/api-key-catalog.md b/docs-site/src/content/docs/guides/api-key-catalog.md index f9f6a0ed9c0..610f6326349 100644 --- a/docs-site/src/content/docs/guides/api-key-catalog.md +++ b/docs-site/src/content/docs/guides/api-key-catalog.md @@ -7,7 +7,7 @@ This page holds section 3 of the [Providers](/guides/providers/) guide: the buil and the provider-specific notes that go with them. Auth modes, account logins, and local providers stay on that page. -opencodex ships 108 built-in presets: 87 key-based, 17 OAuth, three local, and one default +opencodex ships 109 built-in presets: 88 key-based, 17 OAuth, three local, and one default ChatGPT-forward preset. The dashboard's **Add provider** picker opens a key provider's dashboard, validates the key, and stores it; validation is provider-specific. Notable entries: diff --git a/docs-site/src/content/docs/ja/getting-started/quickstart.md b/docs-site/src/content/docs/ja/getting-started/quickstart.md index bd94bcd293e..e28f6eb3c95 100644 --- a/docs-site/src/content/docs/ja/getting-started/quickstart.md +++ b/docs-site/src/content/docs/ja/getting-started/quickstart.md @@ -18,7 +18,7 @@ ocx init `ocx init` では次の手順を説明します。 -1. **プロバイダーを選択してください** — 108 個の組み込みレジストリプリセットのいずれか、または `custom` を選択してベース URL とアダプターを入力します。 +1. **プロバイダーを選択してください** — 109 個の組み込みレジストリプリセットのいずれか、または `custom` を選択してベース URL とアダプターを入力します。 2. **API キー** — キーを貼り付けるか、`${ANTHROPIC_API_KEY}` のような環境変数を参照します。 3. **デフォルト モデル** — キー、ローカル、カスタム プロバイダーの場合は、プリセットを受け入れるか、モデル ID を入力します。 4. **プロキシ ポート** — デフォルトは `10100` です。 diff --git a/docs-site/src/content/docs/ja/guides/providers.md b/docs-site/src/content/docs/ja/guides/providers.md index 0a8a9abf884..34bf3e47966 100644 --- a/docs-site/src/content/docs/ja/guides/providers.md +++ b/docs-site/src/content/docs/ja/guides/providers.md @@ -210,7 +210,7 @@ Kiro のログインには Kiro CLI が必要です。Unix では `curl -fsSL ht ## 3. API キーカタログ -opencodex には組み込みプリセットが 108 個含まれています。キー方式 87、OAuth 17、ローカル 3、 +opencodex には組み込みプリセットが 109 個含まれています。キー方式 88、OAuth 17、ローカル 3、 デフォルト ChatGPT 転送プリセット 1 です。ダッシュボードの **Add provider** ピッカーはキー発行ページを開き、 入力したキーを検証した後保存します(検証はプロバイダー固有です)。主な項目は以下のとおりです: diff --git a/docs-site/src/content/docs/ko/getting-started/quickstart.md b/docs-site/src/content/docs/ko/getting-started/quickstart.md index f438cd502c2..89c8cc936b4 100644 --- a/docs-site/src/content/docs/ko/getting-started/quickstart.md +++ b/docs-site/src/content/docs/ko/getting-started/quickstart.md @@ -18,7 +18,7 @@ ocx init `ocx init`은 다음 과정을 안내합니다: -1. **프로바이더 선택** — 내장 레지스트리 프리셋 108개 중 하나를 고르거나 `custom`을 선택해 base URL과 adapter를 직접 입력합니다. +1. **프로바이더 선택** — 내장 레지스트리 프리셋 109개 중 하나를 고르거나 `custom`을 선택해 base URL과 adapter를 직접 입력합니다. 2. **API 키** — 키를 붙여넣거나 `${ANTHROPIC_API_KEY}` 같은 환경 변수를 참조합니다. 3. **기본 모델** — 키, 로컬, custom 프로바이더에서는 프리셋을 그대로 쓰거나 모델 ID를 직접 입력합니다. 4. **프록시 포트** — 기본값은 `10100`입니다. diff --git a/docs-site/src/content/docs/ko/guides/providers.md b/docs-site/src/content/docs/ko/guides/providers.md index 313b3af6d25..90a96ccdfc0 100644 --- a/docs-site/src/content/docs/ko/guides/providers.md +++ b/docs-site/src/content/docs/ko/guides/providers.md @@ -212,7 +212,7 @@ Kiro 로그인에는 Kiro CLI가 필요합니다. Unix에서는 `curl -fsSL http ## 3. API 키 카탈로그 -opencodex에는 빌트인 프리셋이 108개 들어 있습니다. 키 방식 87개, OAuth 17개, 로컬 3개, +opencodex에는 빌트인 프리셋이 109개 들어 있습니다. 키 방식 88개, OAuth 17개, 로컬 3개, 기본 ChatGPT 포워드 프리셋 1개입니다. 대시보드의 **Add provider** 선택기는 키 발급 페이지를 열고, 입력한 키를 검증한 뒤 저장합니다(검증은 프로바이더별로 다릅니다). 주요 항목은 다음과 같습니다: diff --git a/docs-site/src/content/docs/ru/getting-started/quickstart.md b/docs-site/src/content/docs/ru/getting-started/quickstart.md index 6e37fd9c57d..d6f241d44ae 100644 --- a/docs-site/src/content/docs/ru/getting-started/quickstart.md +++ b/docs-site/src/content/docs/ru/getting-started/quickstart.md @@ -18,7 +18,7 @@ ocx init `ocx init` проведёт вас по следующим шагам: -1. **Выбор провайдера** — выберите один из 108 встроенных пресетов реестра или `custom`, чтобы +1. **Выбор провайдера** — выберите один из 109 встроенных пресетов реестра или `custom`, чтобы ввести базовый URL и адаптер вручную. 2. **API-ключ** — вставьте ключ или сошлитесь на переменную окружения вида `${ANTHROPIC_API_KEY}`. 3. **Модель по умолчанию** — для провайдеров с ключом, локальных и `custom` примите значение из diff --git a/docs-site/src/content/docs/ru/guides/providers.md b/docs-site/src/content/docs/ru/guides/providers.md index 5c7c17067c9..e0b3cd5602b 100644 --- a/docs-site/src/content/docs/ru/guides/providers.md +++ b/docs-site/src/content/docs/ru/guides/providers.md @@ -222,7 +222,7 @@ Inline JSON и лишние позиционные аргументы откло ## 3. Каталог API-ключей -opencodex поставляется с 108 встроенными пресетами: 87 на основе ключей, 17 OAuth, три локальных и +opencodex поставляется с 109 встроенными пресетами: 88 на основе ключей, 17 OAuth, три локальных и один пресет ChatGPT-форварда по умолчанию. Селектор **Add provider** в дашборде открывает страницу выдачи ключей провайдера, проверяет ключ и сохраняет его; проверка зависит от провайдера. Наиболее заметные записи: diff --git a/docs-site/src/content/docs/tr/getting-started/quickstart.md b/docs-site/src/content/docs/tr/getting-started/quickstart.md index c6088681e63..e2277bfbeaf 100644 --- a/docs-site/src/content/docs/tr/getting-started/quickstart.md +++ b/docs-site/src/content/docs/tr/getting-started/quickstart.md @@ -13,7 +13,7 @@ ocx init `ocx init` adım adım size rehberlik eder: -1. **Bir sağlayıcı seçin** — yerleşik kayıt defterindeki 108 önayardan birini +1. **Bir sağlayıcı seçin** — yerleşik kayıt defterindeki 109 önayardan birini veya bir temel URL ile adaptör yazmak için `custom` seçeneğini belirleyin. 2. **API anahtarı** — bir anahtar yapıştırın veya `${ANTHROPIC_API_KEY}` gibi bir ortam değişkenine başvurun. diff --git a/docs-site/src/content/docs/tr/guides/providers.md b/docs-site/src/content/docs/tr/guides/providers.md index 4f99cef139f..22d5facd195 100644 --- a/docs-site/src/content/docs/tr/guides/providers.md +++ b/docs-site/src/content/docs/tr/guides/providers.md @@ -342,7 +342,7 @@ olmayan bir makineden oturum açmak bundan etkilenmez. ## 3. API anahtarı kataloğu -opencodex 108 yerleşik önayar ile birlikte gelir: 87 anahtar tabanlı, 17 +opencodex 109 yerleşik önayar ile birlikte gelir: 88 anahtar tabanlı, 17 OAuth, üç yerel ve bir varsayılan ChatGPT iletme önayarı. Kontrol panelinin **Sağlayıcı ekle** seçicisi bir anahtar sağlayıcısının kontrol panelini açar, anahtarı doğrular ve saklar; doğrulama sağlayıcıya özgüdür. Dikkate değer diff --git a/docs-site/src/content/docs/zh-cn/getting-started/quickstart.md b/docs-site/src/content/docs/zh-cn/getting-started/quickstart.md index 6d809a4847f..291b72d0f00 100644 --- a/docs-site/src/content/docs/zh-cn/getting-started/quickstart.md +++ b/docs-site/src/content/docs/zh-cn/getting-started/quickstart.md @@ -18,7 +18,7 @@ ocx init `ocx init` 会引导你完成: -1. **选择 provider** — 从内置 registry 的 108 个预设中选择一个,或选择 `custom` 手动输入 base URL 和 adapter。 +1. **选择 provider** — 从内置 registry 的 109 个预设中选择一个,或选择 `custom` 手动输入 base URL 和 adapter。 2. **API key** — 粘贴一个 key,或引用一个环境变量,例如 `${ANTHROPIC_API_KEY}`。 3. **默认模型** — 对于 key、本地和 custom provider,接受预设值或输入模型 id。 4. **代理端口** — 默认为 `10100`。 diff --git a/docs-site/src/content/docs/zh-cn/guides/providers.md b/docs-site/src/content/docs/zh-cn/guides/providers.md index 77638b70551..9aa81d1e017 100644 --- a/docs-site/src/content/docs/zh-cn/guides/providers.md +++ b/docs-site/src/content/docs/zh-cn/guides/providers.md @@ -222,7 +222,7 @@ API Key 提供商,也不负责让 WorkBuddy 桌面客户端使用 OpenCodex ## 3. API 密钥目录 -opencodex 内置 108 个预设:87 个密钥预设、17 个 OAuth 预设、3 个本地预设,以及 1 个默认的 +opencodex 内置 109 个预设:88 个密钥预设、17 个 OAuth 预设、3 个本地预设,以及 1 个默认的 ChatGPT 转发预设。仪表盘的 **Add provider** 选择器会打开密钥提供商的控制台,验证并保存密钥。 验证因提供商而异。主要条目包括: diff --git a/docs-site/src/content/docs/zh-tw/getting-started/quickstart.md b/docs-site/src/content/docs/zh-tw/getting-started/quickstart.md index 0108527935b..85f2e523f33 100644 --- a/docs-site/src/content/docs/zh-tw/getting-started/quickstart.md +++ b/docs-site/src/content/docs/zh-tw/getting-started/quickstart.md @@ -13,7 +13,7 @@ ocx init `ocx init` 會引導你完成: -1. **選擇 provider** —— 從內建 registry 的 108 個預設中選擇一個,或選擇 `custom` 手動輸入 +1. **選擇 provider** —— 從內建 registry 的 109 個預設中選擇一個,或選擇 `custom` 手動輸入 base URL 和 adapter。 2. **API key** —— 貼上一個 key,或引用一個環境變數,例如 `${ANTHROPIC_API_KEY}`。 3. **預設模型** —— 對於 API key、本機和 custom provider,可接受預設值或輸入模型 id。 diff --git a/docs-site/src/content/docs/zh-tw/guides/providers.md b/docs-site/src/content/docs/zh-tw/guides/providers.md index ee76c45f2c5..e7b4e8a569e 100644 --- a/docs-site/src/content/docs/zh-tw/guides/providers.md +++ b/docs-site/src/content/docs/zh-tw/guides/providers.md @@ -264,7 +264,7 @@ database 並移除目前的 WAL、SHM 與 journal sidecar,再發布先前的 s ## 3. API 金鑰目錄 -opencodex 內建 108 個 preset:87 個 key-based、17 個 OAuth、3 個 local,以及 1 個預設 ChatGPT-forward +opencodex 內建 109 個 preset:88 個 key-based、17 個 OAuth、3 個 local,以及 1 個預設 ChatGPT-forward preset。儀表板的 **Add provider** picker 會開啟 key provider 的 dashboard、驗證金鑰並儲存;驗證方式 依 provider 而異。主要條目如下。 diff --git a/gui/src/i18n/de.ts b/gui/src/i18n/de.ts index 277284c7701..3d8a7696d52 100644 --- a/gui/src/i18n/de.ts +++ b/gui/src/i18n/de.ts @@ -1331,6 +1331,7 @@ export const de: Record = { "logs.detail.attempt.recovery.key401": "API-Schlüssel erneut authentifiziert", "logs.detail.attempt.recovery.oauthAccount403": "Kontovalidierung erforderlich (403)", "logs.detail.attempt.recovery.oauthAccount429": "Konto rate-limitiert (429)", + "logs.detail.attempt.recovery.codexAccountRetry": "Codex-Konto nach verbrauchtem Fenster gewechselt", "logs.detail.attempt.recovery.oauthAccount402": "Kontingent des Kontos erschöpft (402)", "logs.detail.attempt.recovery.opaqueBlobRejection": "Veralteter verschlüsselter Zustand verworfen", "logs.detail.attempt.recovery.reasoningEffortDowngrade": "Reasoning-Aufwand reduziert", diff --git a/gui/src/i18n/en.ts b/gui/src/i18n/en.ts index 48f8299fffe..011aef37b74 100644 --- a/gui/src/i18n/en.ts +++ b/gui/src/i18n/en.ts @@ -1389,6 +1389,7 @@ export const en = { "logs.detail.attempt.recovery.key401": "API key re-authentication", "logs.detail.attempt.recovery.oauthAccount403": "Account validation required (403)", "logs.detail.attempt.recovery.oauthAccount429": "Account rate-limited (429)", + "logs.detail.attempt.recovery.codexAccountRetry": "Codex account rotated after a spent window", "logs.detail.attempt.recovery.oauthAccount402": "Account quota exhausted (402)", "logs.detail.attempt.recovery.opaqueBlobRejection": "Stale encrypted state dropped", "logs.detail.attempt.recovery.reasoningEffortDowngrade": "Reasoning effort downgraded", diff --git a/gui/src/i18n/fr.ts b/gui/src/i18n/fr.ts index 43af6e034ed..47e0c8f97dd 100644 --- a/gui/src/i18n/fr.ts +++ b/gui/src/i18n/fr.ts @@ -1365,6 +1365,7 @@ export const fr: Record = { "logs.detail.attempt.recovery.key401": "Ré-authentification de la clé API", "logs.detail.attempt.recovery.oauthAccount403": "Validation du compte requise (403)", "logs.detail.attempt.recovery.oauthAccount429": "Compte limité (429)", + "logs.detail.attempt.recovery.codexAccountRetry": "Compte Codex permuté après une fenêtre épuisée", "logs.detail.attempt.recovery.oauthAccount402": "Quota du compte épuisé (402)", "logs.detail.attempt.recovery.opaqueBlobRejection": "État chiffré obsolète abandonné", "logs.detail.attempt.recovery.reasoningEffortDowngrade": "Effort de raisonnement réduit", diff --git a/gui/src/i18n/ja.ts b/gui/src/i18n/ja.ts index 0be2a960b21..7ce02fdea86 100644 --- a/gui/src/i18n/ja.ts +++ b/gui/src/i18n/ja.ts @@ -1272,6 +1272,7 @@ export const ja: Record = { "logs.detail.attempt.recovery.key401": "API キーの再認証", "logs.detail.attempt.recovery.oauthAccount403": "アカウントの検証が必要 (403)", "logs.detail.attempt.recovery.oauthAccount429": "アカウントのレート制限 (429)", + "logs.detail.attempt.recovery.codexAccountRetry": "枠を使い切った Codex アカウントを切り替え", "logs.detail.attempt.recovery.oauthAccount402": "アカウントのクォータ枯渇 (402)", "logs.detail.attempt.recovery.opaqueBlobRejection": "古い暗号化状態を破棄", "logs.detail.attempt.recovery.reasoningEffortDowngrade": "推論強度を下げて再試行", diff --git a/gui/src/i18n/ko.ts b/gui/src/i18n/ko.ts index 54e706b75b3..299858ebc93 100644 --- a/gui/src/i18n/ko.ts +++ b/gui/src/i18n/ko.ts @@ -1370,6 +1370,7 @@ export const ko: Record = { "logs.detail.attempt.recovery.key401": "API 키 재인증", "logs.detail.attempt.recovery.oauthAccount403": "계정 확인 필요 (403)", "logs.detail.attempt.recovery.oauthAccount429": "계정 속도 제한 (429)", + "logs.detail.attempt.recovery.codexAccountRetry": "소진된 창 이후 Codex 계정 전환", "logs.detail.attempt.recovery.oauthAccount402": "계정 할당량 소진 (402)", "logs.detail.attempt.recovery.opaqueBlobRejection": "만료된 암호화 상태 제거", "logs.detail.attempt.recovery.reasoningEffortDowngrade": "추론 강도 하향", diff --git a/gui/src/i18n/pt.ts b/gui/src/i18n/pt.ts index 9b96e229b86..d2563657e13 100644 --- a/gui/src/i18n/pt.ts +++ b/gui/src/i18n/pt.ts @@ -1384,6 +1384,7 @@ export const pt: Record = { "logs.detail.attempt.recovery.key401": "Reautenticação da chave de API", "logs.detail.attempt.recovery.oauthAccount403": "Validação da conta necessária (403)", "logs.detail.attempt.recovery.oauthAccount429": "Conta com limite de taxa (429)", + "logs.detail.attempt.recovery.codexAccountRetry": "Conta Codex trocada após janela esgotada", "logs.detail.attempt.recovery.oauthAccount402": "Cota da conta esgotada (402)", "logs.detail.attempt.recovery.opaqueBlobRejection": "Estado criptografado obsoleto descartado", "logs.detail.attempt.recovery.reasoningEffortDowngrade": "Esforço de raciocínio reduzido", diff --git a/gui/src/i18n/ru.ts b/gui/src/i18n/ru.ts index 348606bb76d..0c7468b7e5e 100644 --- a/gui/src/i18n/ru.ts +++ b/gui/src/i18n/ru.ts @@ -1348,6 +1348,7 @@ export const ru: Record = { "logs.detail.attempt.recovery.key401": "Повторная аутентификация API-ключа", "logs.detail.attempt.recovery.oauthAccount403": "Требуется проверка аккаунта (403)", "logs.detail.attempt.recovery.oauthAccount429": "Ограничение частоты для аккаунта (429)", + "logs.detail.attempt.recovery.codexAccountRetry": "Смена аккаунта Codex после исчерпанного окна", "logs.detail.attempt.recovery.oauthAccount402": "Квота аккаунта исчерпана (402)", "logs.detail.attempt.recovery.opaqueBlobRejection": "Устаревшее зашифрованное состояние отброшено", "logs.detail.attempt.recovery.reasoningEffortDowngrade": "Уровень рассуждения понижен", diff --git a/gui/src/i18n/tr.ts b/gui/src/i18n/tr.ts index 187e7c7277f..0aea9421234 100644 --- a/gui/src/i18n/tr.ts +++ b/gui/src/i18n/tr.ts @@ -1375,6 +1375,7 @@ export const tr: Record = { "logs.detail.attempt.recovery.key401": "API anahtarı yeniden doğrulandı", "logs.detail.attempt.recovery.oauthAccount403": "Hesap doğrulaması gerekli (403)", "logs.detail.attempt.recovery.oauthAccount429": "Hesap hız sınırına takıldı (429)", + "logs.detail.attempt.recovery.codexAccountRetry": "Dolan pencere sonrası Codex hesabı değiştirildi", "logs.detail.attempt.recovery.oauthAccount402": "Hesap kotası tükendi (402)", "logs.detail.attempt.recovery.opaqueBlobRejection": "Eski şifreli durum bırakıldı", "logs.detail.attempt.recovery.reasoningEffortDowngrade": "Akıl yürütme düzeyi düşürüldü", diff --git a/gui/src/i18n/vi.ts b/gui/src/i18n/vi.ts index 2748f01b333..d932c3b7ce8 100644 --- a/gui/src/i18n/vi.ts +++ b/gui/src/i18n/vi.ts @@ -1369,6 +1369,7 @@ export const vi: Record = { "logs.detail.attempt.recovery.key401": "Xác thực lại khóa API", "logs.detail.attempt.recovery.oauthAccount403": "Cần xác minh tài khoản (403)", "logs.detail.attempt.recovery.oauthAccount429": "Tài khoản bị giới hạn tần suất (429)", + "logs.detail.attempt.recovery.codexAccountRetry": "Đổi tài khoản Codex sau khi hết hạn mức", "logs.detail.attempt.recovery.oauthAccount402": "Tài khoản đã hết hạn mức (402)", "logs.detail.attempt.recovery.opaqueBlobRejection": "Đã bỏ trạng thái mã hóa cũ", "logs.detail.attempt.recovery.reasoningEffortDowngrade": "Đã giảm mức suy luận", diff --git a/gui/src/i18n/zh-TW.ts b/gui/src/i18n/zh-TW.ts index cba839b5ed0..24c9d3db5a8 100644 --- a/gui/src/i18n/zh-TW.ts +++ b/gui/src/i18n/zh-TW.ts @@ -3293,6 +3293,7 @@ export const zhTW: Record = { "logs.detail.attempt.recovery.key401": "API 金鑰重新驗證", "logs.detail.attempt.recovery.oauthAccount403": "需要驗證帳號 (403)", "logs.detail.attempt.recovery.oauthAccount429": "帳號速率受限 (429)", + "logs.detail.attempt.recovery.codexAccountRetry": "額度用盡後切換 Codex 帳號", "logs.detail.attempt.recovery.oauthAccount402": "帳號配額已用盡 (402)", "logs.detail.attempt.recovery.opaqueBlobRejection": "已捨棄過期加密狀態", "logs.detail.attempt.recovery.reasoningEffortDowngrade": "已降低推理強度", diff --git a/gui/src/i18n/zh.ts b/gui/src/i18n/zh.ts index edfbcaccf86..7233bb8904d 100644 --- a/gui/src/i18n/zh.ts +++ b/gui/src/i18n/zh.ts @@ -1343,6 +1343,7 @@ export const zh: Record = { "logs.detail.attempt.recovery.key401": "API 密钥重新认证", "logs.detail.attempt.recovery.oauthAccount403": "需要验证账号 (403)", "logs.detail.attempt.recovery.oauthAccount429": "账号速率受限 (429)", + "logs.detail.attempt.recovery.codexAccountRetry": "额度用尽后切换 Codex 账号", "logs.detail.attempt.recovery.oauthAccount402": "账号配额已用尽 (402)", "logs.detail.attempt.recovery.opaqueBlobRejection": "已丢弃过期加密状态", "logs.detail.attempt.recovery.reasoningEffortDowngrade": "已降低推理强度", diff --git a/gui/src/pages/Logs.tsx b/gui/src/pages/Logs.tsx index dc6328a9167..eb6b135c9e2 100644 --- a/gui/src/pages/Logs.tsx +++ b/gui/src/pages/Logs.tsx @@ -365,6 +365,7 @@ const RECOVERY_KIND_KEYS = { "rate-limit-429": "logs.detail.attempt.recovery.rateLimit429", "anthropic-oauth-429": "logs.detail.attempt.recovery.anthropicOauth429", "oauth-account-429": "logs.detail.attempt.recovery.oauthAccount429", + "codex-account-retry": "logs.detail.attempt.recovery.codexAccountRetry", "oauth-account-402": "logs.detail.attempt.recovery.oauthAccount402", "image-413": "logs.detail.attempt.recovery.image413", "empty-completion": "logs.detail.attempt.recovery.emptyCompletion", diff --git a/package.json b/package.json index 2a9de5f4a41..383d6603351 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@bitkyc08/opencodex", - "version": "2.100.0", + "version": "2.100.0-vadymhimself.2", "description": "Universal provider proxy for OpenAI Codex & Claude Code — use any LLM with Codex CLI/App/SDK and Claude Code", "type": "module", "main": "./bin/package-main.mjs", @@ -51,6 +51,7 @@ "start": "bun run src/cli/index.ts start", "test": "bun scripts/test.ts", "test:changed": "bun scripts/test.ts --changed=dev", + "test:gateway-quota": "bun scripts/test.ts tests/adapters/anthropic/anthropic-quota-dispatch.test.ts tests/adapters/anthropic/anthropic-funded-roster.test.ts tests/adapters/anthropic/anthropic-sidecar-account-failover.test.ts tests/adapters/anthropic/anthropic-account-pool.test.ts tests/adapters/anthropic/anthropic-replay.test.ts tests/claude-integration/messages-native-combo.test.ts tests/claude-integration/claude-inbound-cache-stabilize.test.ts tests/responses/responses-anthropic-billing-send-budget.test.ts tests/responses/responses-send-budget-counts.test.ts tests/adapters/adapter-inner-send-budget.test.ts tests/usage/key-attribution.test.ts tests/server/agentradar-account-pin-e2e.test.ts tests/server/agentradar-account-pin-parity-e2e.test.ts tests/adapters/anthropic/anthropic-pool-extra-usage-400.test.ts tests/adapters/anthropic/anthropic-weekly-spent-gate.test.ts tests/adapters/anthropic/anthropic-agentradar-pool-parity.test.ts tests/adapters/anthropic/anthropic-family-claim-429.test.ts tests/server/terminal-guard-agentradar-rotation.test.ts tests/routing/router-combo-400-quota-echo.test.ts tests/responses/responses-compact-400-quota-echo.test.ts tests/routing/routing-analytics-physical.test.ts tests/lib/call-usage-line-log.test.ts tests/lib/upstream-refusal-log.test.ts tests/server/agentradar-account-pin-native-e2e.test.ts tests/claude-integration/agentradar-hosted-fidelity.test.ts tests/claude-integration/agentradar-hosted-tap.test.ts tests/adapters/anthropic/anthropic-agentradar-429.test.ts tests/server/image-bridge-runturn-pacing.test.ts tests/claude-integration/agentradar-count-tokens.test.ts", "typecheck": "bun check --no-script --noEmit", "typecheck:tsc": "bun x tsc --noEmit", "typecheck:gui:if-changed": "bun scripts/typecheck-gui-if-changed.ts", diff --git a/scripts/test-layout/layout-agentradar.json b/scripts/test-layout/layout-agentradar.json new file mode 100644 index 00000000000..31457eac849 --- /dev/null +++ b/scripts/test-layout/layout-agentradar.json @@ -0,0 +1,32 @@ +{ + "call-usage-line-log.test.ts": "lib", + "upstream-refusal-log.test.ts": "lib", + "routing-analytics-physical.test.ts": "routing", + "usage-log-physical.test.ts": "usage", + "usage-summary-physical.test.ts": "usage", + "request-log-physical.test.ts": "usage", + "management-api-logs-metrics-physical.test.ts": "server", + "responses-state-env-bounds.test.ts": "responses", + "router-combo-400-quota-echo.test.ts": "routing", + "responses-compact-400-quota-echo.test.ts": "responses", + "provider-registry-xai-apikey.test.ts": "providers", + "anthropic-agentradar-pool-parity.test.ts": "adapters/anthropic", + "anthropic-family-claim-429.test.ts": "adapters/anthropic", + "anthropic-pool-extra-usage-400.test.ts": "adapters/anthropic", + "anthropic-weekly-spent-gate.test.ts": "adapters/anthropic", + "agentradar-account-pin-parity-e2e.test.ts": "server", + "terminal-guard-agentradar-rotation.test.ts": "server", + "web-search-overflow-usage.test.ts": "web-search", + "anthropic-agentradar-429.test.ts": "adapters/anthropic", + "server-combo-local-pacing-cooldown.test.ts": "server", + "image-bridge-runturn-pacing.test.ts": "server", + "agentradar-hosted-fidelity.test.ts": "claude-integration", + "agentradar-hosted-tap.test.ts": "claude-integration", + "agentradar-account-pin-native-e2e.test.ts": "server", + "quota-explain.test.ts": "providers", + "combo-pool-stale-cooldown.test.ts": "routing", + "usage-anthropic-account-label.test.ts": "usage", + "server-combo-failover-advance.test.ts": "server", + "agentradar-count-tokens.test.ts": "claude-integration", + "combo-pool-cooldown-blast-radius.test.ts": "codex-integration" +} diff --git a/scripts/test-layout/layout-server-inbound.json b/scripts/test-layout/layout-server-inbound.json index ea06db0ab22..136ca83bb29 100644 --- a/scripts/test-layout/layout-server-inbound.json +++ b/scripts/test-layout/layout-server-inbound.json @@ -1,4 +1,5 @@ { + "agentradar-account-pin-e2e.test.ts": "server", "claude-translated-advisor-tool.test.ts": "claude-integration", "management-provider-forward-advisor-tool.test.ts": "server", "local-control-socket.test.ts": "server", diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index df3c2fe07d2..e5213b7f818 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1,7 +1,7 @@ { "version": 1, "root": "tests", - "explicitShards": ["layout-route-intent.json", "layout-release-signing.json", "layout-file-identity.json", "layout-clients.json", "layout-codex-collision.json", "layout-native-profiles.json", "layout-agent-assertion.json", "layout-chat.json", "layout-replay.json", "layout-adapters.json", "layout-jev.json", "layout-workbuddy.json", "layout-combos.json", "layout-credit-renewal.json", "layout-server-inbound.json", "layout-providers.json", "layout-responses.json", "layout-exec-boundary.json", "layout-persistent-dashboard.json", "layout-start-ownership.json", "layout-config.json", "layout-kiro-usage.json", "layout-usage.json", "layout-docs-guides.json", "layout-docs.json", "layout-claude-keychain.json", "layout-test-hygiene.json", "layout-codex-journal.json"], + "explicitShards": ["layout-agentradar.json", "layout-route-intent.json", "layout-release-signing.json", "layout-file-identity.json", "layout-clients.json", "layout-codex-collision.json", "layout-native-profiles.json", "layout-agent-assertion.json", "layout-chat.json", "layout-replay.json", "layout-adapters.json", "layout-jev.json", "layout-workbuddy.json", "layout-combos.json", "layout-credit-renewal.json", "layout-server-inbound.json", "layout-providers.json", "layout-responses.json", "layout-exec-boundary.json", "layout-persistent-dashboard.json", "layout-start-ownership.json", "layout-config.json", "layout-kiro-usage.json", "layout-usage.json", "layout-docs-guides.json", "layout-docs.json", "layout-claude-keychain.json", "layout-test-hygiene.json", "layout-codex-journal.json"], "explicit": { "kimi-monthly-quota.test.ts": "providers", "vertex-adc-verified.test.ts": "adapters/google", "sync-fanout.test.ts": "cli", "sync-command-fanout.test.ts": "cli", "link-pinned-sync-argv.test.ts": "clients", "sync-link-status.test.ts": "cli", "sync-fanout-process.test.ts": "cli", "codex-installation-acl.test.ts": "codex-integration", "codex-installation-id.test.ts": "codex-integration", "codex-installation-wire.test.ts": "codex-integration", "codex-installation-failover.test.ts": "responses", "service-desktop-recovery.test.ts": "service", "catalog-omit-disabled-native.test.ts": "codex-integration", "provider-input-modalities.test.ts": "providers", "combo-description.test.ts": "routing", "combo-description-catalog.test.ts": "codex-integration", "codex-provider-routing.test.ts": "codex-integration", "config-native-reasoning-retention.test.ts": "config", "server-native-reasoning-retention-merge.test.ts": "server", "server-native-reasoning-retention.test.ts": "server", "config-workflow-budget.test.ts": "config", "server-workflow-budget-settings.test.ts": "server", "responses-retained-reasoning-stream-recovery.test.ts": "responses", "openai-native-hosted-search-history.test.ts": "adapters/openai", "responses-native-compact-search-history.test.ts": "responses", "config-hosted-search-projection.test.ts": "config", "gcp-credential-boundary.test.ts": "adapters/google", "gcp-sa-marker.test.ts": "adapters/google", "gcp-sa-adapter.test.ts": "adapters/google", "management-gcp-credential-input.test.ts": "server", "codex-preserve-signed-in.test.ts": "codex-integration", "cli-account-preservation.test.ts": "cli", "cli-provider-list-active.test.ts": "cli", "service-state-isolation.test.ts": "service", "compaction-history-framing.test.ts": "responses", "catalog-native-max-context.test.ts": "codex-integration", "protocol-argon-thought-boundary.test.ts": "responses", "service-log.test.ts": "service", "messages-native-empty-inline.test.ts": "claude-integration", "catalog-auto-review-override.test.ts": "codex-integration", "management-auto-review-settings.test.ts": "server", "management-auto-review-auth.test.ts": "server", "prefer-model-alias-ids.test.ts": "server", "responses-mimo-envelope.test.ts": "responses", "responses-call-id-linkage.test.ts": "responses", "bridge-terminal-refusal.test.ts": "adapters", "loop-terminal-refusal.test.ts": "images", "web-search-terminal-refusal.test.ts": "web-search", "combo-terminal-refusal.test.ts": "routing", "server-terminal-refusal-wiring.test.ts": "server", "incomplete-reason-normalization.test.ts": "responses", "advisor-retention.test.ts": "advisor", "responses-native-key-cooldown.test.ts": "responses", "responses-run-turn-key-readmission.test.ts": "responses", "qoder-spawn-readmission.test.ts": "adapters", "api-key-selection-cooldown.test.ts": "providers", "api-key-auto-pause-distribution.test.ts": "providers", "explicit-key-pause-oauth-boundary.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", "messages-native-key-selection.test.ts": "claude-integration", "coding-agent-windows-dispatch.test.ts": "adapters", "coding-agent-native-empty-argv.test.ts": "windows", "cli-message-mcp-resume.test.ts": "cli", "messaging-mcp-resume.test.ts": "codex-integration", "pr-gates-workflow.test.ts": "ci-workflows", "stop-restore-history-ownership.test.ts": "service", "cli-stop-refusal-transport.test.ts": "cli", diff --git a/src/adapters/anthropic.ts b/src/adapters/anthropic.ts index d59c1a4294f..a3107befa5c 100644 --- a/src/adapters/anthropic.ts +++ b/src/adapters/anthropic.ts @@ -40,7 +40,7 @@ import { resolveThinkingEnabled } from "../clients/reasoning-controls"; import { resolveReasoningCapabilities } from "../providers/reasoning-capabilities"; import { declaresThinkingToggleWire, reasoningControlsAreUnsupported } from "./reasoning-controls"; import { applyAgentRouterLanguageFraming, isAgentRouterEndpoint } from "./agentrouter"; -import { anthropicCacheWrite1hTokens, resolveAnthropicCacheRetention, type AnthropicCacheRetention } from "./anthropic/cache-retention"; +import { anthropicCacheWrite1hTokens, anthropicPromptCacheTtlMs, resolveAnthropicCacheRetention, type AnthropicCacheRetention } from "./anthropic/cache-retention"; import { advisorServerExchange, advisorServerToolDestination, planAnthropicServerTools } from "./anthropic/advisor-tool"; import { applyAnthropicPerMessageEffort } from "./anthropic/per-message-effort"; import { claudeFamilyVersion, rejectsCombinedSampling, rejectsForcedToolChoice, rejectsSamplingParameters, supportsExplicitThinkingDisable, usesAdaptiveThinking, usesBetweenToolsFloor } from "./anthropic-model-contract"; @@ -1262,11 +1262,13 @@ export function createAnthropicAdapter(provider: OcxProviderConfig, cacheRetenti // Experimental, default off (#6895): pin the top-level effort and carry changes as effort messages. applyAnthropicPerMessageEffort({ provider, parsed, incoming, body, headers, url, isOAuth, chatLineage: resolveTranslatedAnthropicSessionKey(parsed, incoming) }); + const promptCacheTtlMs = anthropicPromptCacheTtlMs(body); const request = { url, method: "POST", headers, body: JSON.stringify(body), + ...(promptCacheTtlMs !== undefined ? { promptCacheTtlMs } : {}), // Adapter-owned: the registry wrapper would otherwise report the exact absence of a // tier field and mislabel every fast turn as downgraded. tierLog: createAdapterTierMetadata( diff --git a/src/adapters/anthropic/cache-retention.ts b/src/adapters/anthropic/cache-retention.ts index aca2a361107..fd3bac88148 100644 --- a/src/adapters/anthropic/cache-retention.ts +++ b/src/adapters/anthropic/cache-retention.ts @@ -14,6 +14,8 @@ export type AnthropicCacheRetention = "none" | "short" | "long"; * terms we cannot see. An explicit cacheRetention "long" still sends 1h anywhere. */ export function defaultAnthropicCacheRetention(provider: Pick): "short" | "long" { + // AgentRadar hosted gateways keep the .27 5-minute default: a 1 h write bills 2x instead of 1.25x. + if (process.env.OPENCODEX_AGENTRADAR === "1") return "short"; return credentialDomainFor(provider)?.firstPartyAnthropic ? "long" : "short"; } @@ -45,6 +47,12 @@ const MAX_DEPTH = 16; const isDataKey = (key: string): boolean => key === "cache_control" || key === "input" || key === "input_schema" || key === "input_examples"; +/** Effective prompt-cache lifetime of a wire body: 1 h only when every breakpoint is 1 h; none -> undefined. */ +export function anthropicPromptCacheTtlMs(body: unknown): number | undefined { + const ttls = collectAnthropicCacheTtls(body); + return ttls.length === 0 ? undefined : ttls.every(ttl => ttl === "1h") ? 3_600_000 : 300_000; +} + /** * Every caller cache_control breakpoint TTL in a Messages body: top-level automatic caching plus * blocks in system, tools and messages, including nested containers (tool_result, diff --git a/src/adapters/anthropic/passthrough.ts b/src/adapters/anthropic/passthrough.ts index 4b876858666..5f7b08c10f5 100644 --- a/src/adapters/anthropic/passthrough.ts +++ b/src/adapters/anthropic/passthrough.ts @@ -18,7 +18,7 @@ * for every destination decides from the full source again. */ import { mergeAnthropicBetaHeader } from "../../providers/anthropic-fast"; -import { applyClaudeToolPrefix, CLAUDE_CODE_SYSTEM_INSTRUCTION } from "../../oauth/anthropic"; +import { ANTHROPIC_OAUTH_BETA, applyClaudeToolPrefix, CLAUDE_CODE_SYSTEM_INSTRUCTION } from "../../oauth/anthropic"; import { credentialDomainFor, opaqueStateForDestination } from "../../protocols/opaque-state"; import type { OcxConfig, OcxProviderConfig } from "../../types"; import { @@ -93,6 +93,27 @@ export interface AnthropicMessagesPassthroughOptions { clientIdentity?: AnthropicClientIdentity; /** Optional conversation/session key for deterministic per-conversation X-Claude-Code-Session-Id. */ sessionKey?: string | null; + /** + * AgentRadar hosted contract, first-party Anthropic only: send the caller's body as sent (wire + * model, plus the OAuth identity line when no identity block exists) and every caller beta. No + * allowlist, cache TTL, account metadata or tool-name rewrite: each changes cache billing or drops + * a field the caller's beta enabled. + */ + exactSource?: boolean; +} + +/** Exactly Anthropic's own Messages endpoint: no other scheme, port, path prefix or look-alike host. */ +export function isCanonicalAnthropicMessagesUrl(provider: Pick): boolean { + try { return resolveAnthropicMessagesUrl(provider) === "https://api.anthropic.com/v1/messages"; } catch { return false; } +} + +const CLAUDE_IDENTITY_LINE = /^You are (Claude Code, Anthropic's official CLI for Claude|a Claude agent, built on Anthropic's Claude Agent SDK)/; + +function withClaudeIdentity(body: Rec): Rec { + const system = typeof body.system === "string" ? (body.system ? [{ type: "text", text: body.system }] : []) + : Array.isArray(body.system) ? body.system : []; + if (system.some(block => isRec(block) && typeof block.text === "string" && CLAUDE_IDENTITY_LINE.test(block.text.trimStart()))) return body; + return { ...body, system: [{ type: "text", text: CLAUDE_CODE_SYSTEM_INSTRUCTION }, ...system] }; } /** The allowlisted copy of `body` with `model` set to the wire model. Shallow: nothing is cloned. */ @@ -301,11 +322,14 @@ export function buildAnthropicMessagesPassthroughRequest( : "anthropic provider requires a non-empty apiKey (authMode: key)"); } const url = resolveAnthropicMessagesUrl(provider); - const native = anthropicMessagesNativeWireBody(provider, modelId, body, options); - const { strippedOpaqueState, oauthToolNames } = native; - const ttlBody = applyAnthropicNativeCacheTtlDefault( + const exact = options.exactSource === true && domain?.firstPartyAnthropic === true && url === "https://api.anthropic.com/v1/messages"; + const native = exact ? undefined : anthropicMessagesNativeWireBody(provider, modelId, body, options); + const { strippedOpaqueState = false, oauthToolNames } = native ?? {}; + const ttlBody = native && applyAnthropicNativeCacheTtlDefault( native.wireBody, resolveAnthropicCacheRetention(config?.cacheRetention, provider)); - const wireBody = oauth ? bindAnthropicAccountMetadata(ttlBody, options.providerAccountUuid) : ttlBody; + // Exact replay still isolates the pooled account's identity: its own account_uuid and derived device_id. + const wireBody = !ttlBody ? (oauth ? bindAnthropicAccountMetadata(withClaudeIdentity({ ...body, model: modelId }), options.providerAccountUuid) : { ...body, model: modelId }) + : oauth ? bindAnthropicAccountMetadata(ttlBody, options.providerAccountUuid) : ttlBody; const headers = anthropicBaseRequestHeaders(wireBody.stream === true); const resolvedSessionKey = options.sessionKey?.trim() || extractAnthropicCallerSessionId(body); if (oauth) applyAnthropicOAuthAuth(headers, provider.apiKey, resolvedSessionKey); @@ -323,6 +347,14 @@ export function buildAnthropicMessagesPassthroughRequest( } } if (domain?.firstPartyAnthropic) applyAnthropicClientIdentity(headers, options.clientIdentity, provider.headers); + if (exact) { + // .27 order: caller betas, then the operator's, then the OAuth pair. + const operator = Object.entries(provider.headers ?? {}).flatMap(([name, value]) => name.toLowerCase() === "anthropic-beta" ? [value] : []); + for (const name of Object.keys(headers)) if (name.toLowerCase() === "anthropic-beta") delete headers[name]; + mergeAnthropicBetaHeader(headers, [options.callerAnthropicBeta ?? "", ...operator, oauth ? ANTHROPIC_OAUTH_BETA : ""]); + return { url, headers, body: JSON.stringify(wireBody), wireBody, droppedBetas: false, droppedBetaNames: [], + droppedUnrecognizedBetas: 0, strippedOpaqueState: false }; + } const betas = allowlistAnthropicBetas(options.callerAnthropicBeta, domain?.firstPartyAnthropic ? "first-party" : "compatible", { inlineTools: hasInlineToolChanges(wireBody), observedNativeClient: hasObservedAnthropicClientIdentity(options.clientIdentity) }); mergeAnthropicBetaHeader(headers, betas.betas); diff --git a/src/adapters/base.ts b/src/adapters/base.ts index 23c4d31529f..05680dc37c2 100644 --- a/src/adapters/base.ts +++ b/src/adapters/base.ts @@ -137,6 +137,8 @@ export interface AdapterRequest { method: string; headers: Record; body: string; + /** Effective Anthropic prompt-cache lifetime of the final wire body (5 min or 1 h). */ + promptCacheTtlMs?: number; /** Caller tool identities replaced by hosted tools in this exact build; empty clears prior builds. */ hostedToolSubstitutions?: ReadonlyArray<{ name: string; namespace?: string; callType: string }>; /** Exact final wire model, captured before serialization for model-scoped transport gates. */ diff --git a/src/bridge/errors.ts b/src/bridge/errors.ts index 26248cc9a1b..f88e00f0605 100644 --- a/src/bridge/errors.ts +++ b/src/bridge/errors.ts @@ -14,6 +14,7 @@ import { CYBER_POLICY_ERROR_CODE, isCyberPolicyCode, isModelNotInPlanRefusal, + isQuotaBadRequestMessage, markVerifiedModelNotInPlanResponse, MODEL_NOT_IN_PLAN_CODE, REQUIRED_TOOL_CONTRACT_CODE, @@ -35,6 +36,26 @@ export function upstreamCodeForComboFailure(response: Response, bodyCode: string : bodyCode; } +// AgentRadar hosted gateways: a canonical Anthropic combo child's final error is relayed as Anthropic sent it. +const verbatimComboFailures = new WeakMap Response>(); +/** Register the exact client response a combo returns if this child failure ends the ladder. */ +export function markVerbatimComboFailure(response: Response, build: () => Response): Response { + verbatimComboFailures.set(response, build); + return response; +} +/** The registered exact response for a consumed combo child failure, if any. */ +export function verbatimComboFailure(response: Response): Response | undefined { + return verbatimComboFailures.get(response)?.(); +} +/** Classification keeps the formatted failure; only the client presentation is the registered one. */ +export function carryVerbatimComboFailure(source: Response, target: Response): void { + const build = verbatimComboFailures.get(source); + if (build) verbatimComboFailures.set(target, build); +} +export function hasVerbatimComboFailure(response: Response): boolean { + return verbatimComboFailures.has(response); +} + /** Preserve formatted quota-code provenance when lifecycle helpers rebuild a response. */ export function carryFormattedQuotaSource(source: Response, target: Response): Response { if (formattedQuotaSources.has(source)) formattedQuotaSources.set(target, formattedQuotaSources.get(source)!); @@ -119,7 +140,10 @@ export function formatErrorResponse( status: finalStatus, headers, }); - if (validationStatus && error.code === "insufficient_quota") { + // AgentRadar hosted gateways keep the .27 contract: a field-anchored quota verdict (insufficient_quota + // field, recorded extra-usage or spend-limit opening) is combo evidence on its own; echoed words never are. + if (validationStatus && error.code === "insufficient_quota" + && !(process.env.OPENCODEX_AGENTRADAR === "1" && isQuotaBadRequestMessage(message))) { formattedQuotaSources.set(response, options?.upstreamCode !== undefined ? options.upstreamCode : options?.classified ? null : options?.code ?? null); } diff --git a/src/claude/inbound-content-options.ts b/src/claude/inbound-content-options.ts index 7d8b1a898bf..7432426834f 100644 --- a/src/claude/inbound-content-options.ts +++ b/src/claude/inbound-content-options.ts @@ -73,7 +73,9 @@ export function toolsToResponses(tools: unknown): Rec[] | undefined { if (!restrictedSearch) out.push({ type: "web_search" }); // hosted sidecar path continue; } - if (typeof raw.name === "string" && raw.name.length > 0 && isRec(raw.input_schema)) { + // An Anthropic-typed tool (computer_*, bash_*, ...) is not a function even when it carries a + // schema: its typed fields (display size, ...) have no translated home (.27). + if ((type === "" || type === "custom") && typeof raw.name === "string" && raw.name.length > 0 && isRec(raw.input_schema)) { out.push({ type: "function", name: raw.name, @@ -117,6 +119,8 @@ export function declaredAnthropicServerTools(raw: unknown): Rec[] | undefined { * The Advisor rides the out-of-body lane and is optional: a destination without it simply runs the * turn unadvised, so it never makes a translated child native-only. */ +export const EXACT_REPLAY_REQUIRED_MESSAGE = "Anthropic-native tools and MCP servers require exact canonical Anthropic Messages replay; the selected route cannot carry them"; + export function hasNativeOnlyTool(raw: Rec): boolean { return Array.isArray(raw.tools) && raw.tools.some(tool => !toolsToResponses([tool]) && !declaredAnthropicServerTools({ tools: [tool] })); diff --git a/src/claude/inbound.ts b/src/claude/inbound.ts index c2396b389eb..cc1b4445ab5 100644 --- a/src/claude/inbound.ts +++ b/src/claude/inbound.ts @@ -388,6 +388,23 @@ function canonicalJson(value: unknown): string { return JSON.stringify(value) ?? "null"; } +function claudeCacheSessionUserId(userId: string): string { + const session = claudeSessionUserId(userId); + // AgentRadar hosted gateways key on the session id alone (.27): every other user_id field may vary. + const hosted = process.env.OPENCODEX_AGENTRADAR === "1"; + if (!hosted && !session.includes("parent_session_id")) return session; + try { + const parsed: unknown = JSON.parse(session); + if (isRec(parsed) && typeof parsed.parent_session_id === "string" && parsed.parent_session_id) { + if (hosted) return parsed.parent_session_id; + const { parent_session_id: parent, ...rest } = parsed; + return JSON.stringify({ ...rest, session_id: parent }); + } + if (hosted && isRec(parsed) && typeof parsed.session_id === "string" && parsed.session_id) return parsed.session_id; + } catch { /* not JSON */ } + return session; +} + /** Provenance of the generated prompt_cache_key (never serialized into the wire body). */ export type ClaudeCacheKeySource = "metadata" | "system" | null; @@ -601,8 +618,9 @@ function applyAnthropicSettings( // metadata.user_id embeds the session uuid, so hashing it yields a stable // per-session key with a bounded length/charset. Remote Claude Code adds a per-turn // attribution token (`tk`) to that blob; it is dropped, as Claude Code itself drops it, - // so every turn of one session keeps one key. - body.prompt_cache_key = createHash("sha256").update(claudeSessionUserId(raw.metadata.user_id)).digest("hex").slice(0, 32); + // so every turn of one session keeps one key. A fork keys on its parent session, whose + // cached prefix it shares. + body.prompt_cache_key = createHash("sha256").update(claudeCacheSessionUserId(raw.metadata.user_id)).digest("hex").slice(0, 32); cacheKeySource = "metadata"; } else if (typeof cacheSystem === "string" || cacheSystem.length > 0) { // Claude Desktop sends no metadata.user_id (H1, devlog 130): without any key the diff --git a/src/claude/outbound.ts b/src/claude/outbound.ts index b328948398f..93a2dab5cb5 100644 --- a/src/claude/outbound.ts +++ b/src/claude/outbound.ts @@ -1,4 +1,5 @@ import type { UpstreamConditionEvidence } from "../lib/upstream-condition"; +import { logTranslationBufferLimit } from "../lib/upstream-refusal-log"; import { ownedAnthropicErrorPayload, messagesOverflowPresentation } from "../lib/messages-error-body"; import { proxyMessagePresentation, markProxyError, hasProxyErrorMessage, hasProxyIncomplete } from "../lib/client-error-body"; import { admitMessagesFailure, collectMessagesFailure, type MessagesFailureVerdictSink } from "./failure-verdict"; @@ -621,6 +622,7 @@ export function responsesSseToAnthropicSse( }); const admitted = () => admitMessagesFailure(opts.failureVerdictSink, payload, legacy); if (code === "translation_buffer_limit") { + logTranslationBufferLimit("stream", { model, bytes: bufferBytes, limit: TRANSLATOR_MAX_TURN_BYTES }); releaseThinkingBuffer(open); if (open?.callId) translatorBudget.closeCall(open.callId); open = null; diff --git a/src/cli/dispatch.ts b/src/cli/dispatch.ts index 8065f69743e..fa369728228 100644 --- a/src/cli/dispatch.ts +++ b/src/cli/dispatch.ts @@ -384,9 +384,12 @@ const commandRunners: Record = { const loginArgs = deps.args.slice(1); // `--headless` is the only flag this command takes; the provider is the first bare word, so // `ocx login --headless anthropic` and `ocx login anthropic --headless` mean the same thing. - // It is deliberately not `--code`: that name carries a secret value everywhere else in the + // It is deliberately not named `--code`: that name carries a secret value everywhere else in the // CLI (`ocx account login --code `) and its operand is scrubbed from stderr. - const codeMode = loginArgs.includes("--headless"); + // A bare `--code` is still accepted as its alias, because it was this flag's earlier name and + // ignoring it silently starts a browser login on a host that has none. No value is read, and an + // unknown provider word prints the usage text, never the word. + const codeMode = loginArgs.includes("--headless") || loginArgs.includes("--code"); const acceptAntigravityRisk = loginArgs.includes("--accept-antigravity-risk"); const positional = loginArgs.filter(arg => !arg.startsWith("--")); // 'ocx login codex' is the command people type first, and until now it answered with diff --git a/src/codex/reset-credit-auto-redeem.ts b/src/codex/reset-credit-auto-redeem.ts index 2c6292368ad..3507bbb0548 100644 --- a/src/codex/reset-credit-auto-redeem.ts +++ b/src/codex/reset-credit-auto-redeem.ts @@ -93,9 +93,15 @@ function readJournal(path: string): Journal { } function writeJournal(path: string, journal: Journal, now: number): void { - // Keep only entries whose credit could still matter: settled ones older than a week are noise. - const cutoff = now - 7 * 24 * 60 * 60_000; - journal.entries = journal.entries.filter(e => e.state !== "settled" || e.updatedAt > cutoff); + // Keep only entries whose credit could still matter: a settled entry stays until the later of + // a week after its last update and the credit's own expiry, so it is never dropped while the + // credit it records can still be listed. An unparseable expiry falls back to the week rule. + const week = 7 * 24 * 60 * 60_000; + journal.entries = journal.entries.filter(e => { + if (e.state !== "settled") return true; + const expiry = Date.parse(e.expiresAt); + return now < Math.max(e.updatedAt + week, Number.isNaN(expiry) ? 0 : expiry); + }); atomicWriteFile(path, JSON.stringify(journal, null, 2)); } diff --git a/src/combos/failover.ts b/src/combos/failover.ts index a687b4b821e..75eb22b1977 100644 --- a/src/combos/failover.ts +++ b/src/combos/failover.ts @@ -161,6 +161,17 @@ export function parseRetryAfterMs( return options?.preserveImmediate ? 1 : undefined; } +/** + * Whether an account pool behind `provider` can serve right now. A 429/402 cooldown on a pooled + * target only says every account was spent when it was recorded; a newly connected or re-authed + * account, or one whose window rolled, makes it stale. Registered by the pool owner so this + * module keeps no import on the OAuth stores. + */ +let poolHasEligibleAccount: ((provider: string, now: number) => boolean) | undefined; +export function setComboPoolEligibilityProbe(probe: typeof poolHasEligibleAccount): void { + poolHasEligibleAccount = probe; +} + export function isComboTargetInCooldown( comboId: string, target: Pick, @@ -169,7 +180,11 @@ export function isComboTargetInCooldown( const key = cooldownMapKey(comboId, target); const entry = targetCooldowns.get(key); if (!entry) return false; - if (entry.cooldownUntil <= now) { + if (entry.cooldownUntil <= now + // AgentRadar hosted gateways only (.27): elsewhere an upstream 429 that is not the pool's own refusal + // parks the target in full even while an account is eligible (isAnthropicPoolLocalRefusal). + || (process.env.OPENCODEX_AGENTRADAR === "1" && (entry.status === 429 || entry.status === 402) + && poolHasEligibleAccount?.(target.provider, now))) { targetCooldowns.dropLocal(key); return false; } @@ -322,8 +337,11 @@ export function coolComboTarget( }) ? COMBO_REQUEST_RATE_COOLDOWN_MS : DEFAULT_COOLDOWN_MS); targetCooldowns.set(cooldownMapKey(comboId, target), { revision: ++nextCooldownRevision, - // Local fallbacks are capped at ten minutes; explicit server delays at one day. - cooldownUntil: now + (serverDelayMs ?? Math.min(Math.max(cooldownMs, 1), MAX_COOLDOWN_MS)), + // Local fallbacks are capped at ten minutes; explicit server delays at one day. AgentRadar hosted + // gateways cap server delays at ten minutes too (.27, d3ea59ccb9): one account's weekly + // Retry-After once parked a shared target for hours while its siblings were healthy. + cooldownUntil: now + (serverDelayMs !== undefined && process.env.OPENCODEX_AGENTRADAR !== "1" + ? serverDelayMs : Math.min(Math.max(serverDelayMs ?? cooldownMs, 1), MAX_COOLDOWN_MS)), status: options?.status, }); sweepExpiredOnWrite(now); @@ -777,7 +795,9 @@ function isProviderScopedQuotaCap( message: string, code?: string | null, ): boolean { - if (status === 400 || status === 422) return false; // Validation prose may reflect the caller's input. + // Validation prose may reflect the caller's input. AgentRadar hosted gateways keep the .27 rule: + // the free-tier cap still counts there as the provider's own field, never as words. + if (status === 400 || status === 422) return process.env.OPENCODEX_AGENTRADAR === "1" && hasFreePromptCapField(message); const normalizedCode = normalizedFailureCode(code); const text = message.toLowerCase(); if ( @@ -790,6 +810,22 @@ function isProviderScopedQuotaCap( || (text.includes("free tier") && text.includes("single request")); } +/** + * `err_free_prompt_cap` as the provider's own field: `error.code`, or `error.metadata.reason` as + * the OpenRouter-style free-tier refusal carries it, in a bounded JSON envelope after an optional + * `Provider error : ` prefix. Never a word anywhere in the text. + */ +function hasFreePromptCapField(message: string): boolean { + if (message.length > 16_384) return false; + let payload: unknown; + try { payload = JSON.parse(message.trim().replace(/^Provider error \d{3}:\s*/, "")); } catch { return false; } + const error = payload && typeof payload === "object" && !Array.isArray(payload) ? (payload as Record).error : undefined; + if (!error || typeof error !== "object" || Array.isArray(error)) return false; + const { code, metadata } = error as Record; + return code === "err_free_prompt_cap" || (!!metadata && typeof metadata === "object" && !Array.isArray(metadata) + && (metadata as Record).reason === "err_free_prompt_cap"); +} + /** A structured account-window exhaustion code; prose never qualifies (a request can echo it). */ function hasAccountExhaustionCode(code?: string | null): boolean { const normalized = normalizedFailureCode(code); @@ -808,7 +844,8 @@ function isRequestLocalFreePromptCap( message: string, code?: string | null, ): boolean { - if (status !== 400 && status !== 422) return false; + // .27 (AgentRadar hosted) read a 422 free-tier field as provider-wide, not request-local. + if (status !== 400 && (status !== 422 || process.env.OPENCODEX_AGENTRADAR === "1")) return false; const text = message.toLowerCase(); if (normalizedFailureCode(code) === "free_rate_limited") return true; if (text.includes("err_free_prompt_cap")) return true; diff --git a/src/combos/resolve.ts b/src/combos/resolve.ts index e8d878ba69a..5bc0fc28a1c 100644 --- a/src/combos/resolve.ts +++ b/src/combos/resolve.ts @@ -1,3 +1,4 @@ +import { createHash } from "node:crypto"; import { errorDestinationText, providerErrorDestinationLabel } from "../lib/error-destination"; import type { OcxComboTarget, OcxConfig } from "../types"; import { getCachedProviderRoutingQuota } from "../providers/quota-routing-cache"; @@ -455,6 +456,8 @@ export function pickComboTarget( now?: number; /** Inspect a round-robin choice without mutating its sticky/weight state. */ preview?: boolean; + /** A random combo's pick from this seed instead of Math.random (AgentRadar count/generation correlation). */ + randomSeed?: string; } = {}, ): ComboPick | null { const writerGeneration = captureConfigGeneration(); @@ -512,7 +515,8 @@ export function pickComboTarget( .filter(({ target }) => eligible(target)); if (eligibleTargets.length > 0) { const totalWeight = eligibleTargets.reduce((sum, entry) => sum + entry.target.weight, 0); - let random = Math.random() * totalWeight; + let random = (options.randomSeed === undefined ? Math.random() + : createHash("sha256").update(`${comboId}\0${options.randomSeed}`).digest().readUIntBE(0, 6) / 2 ** 48) * totalWeight; for (const entry of eligibleTargets) { random -= entry.target.weight; if (random <= 0) { @@ -688,6 +692,7 @@ export async function pickComboTargetWithWait( cooldownReprobeTargets?: Iterable; now?: number; sleep?: (ms: number, signal?: AbortSignal) => Promise; + randomSeed?: string; }, ): Promise { const now = options.now ?? Date.now(); @@ -719,6 +724,7 @@ export async function pickComboTargetWithWait( const normalOnly = (target: NormalizedComboTarget): boolean => !target.lastResort && (customEligible?.(target) ?? true); const normalPick = pickComboTarget(config, comboId, { + randomSeed: options.randomSeed, allowCooldownReprobe: false, cooldownReprobeTargets, exclude: excluded, @@ -751,6 +757,7 @@ export async function pickComboTargetWithWait( // The combo can be deleted or renamed while this request sleeps. if (!getCombo(config, comboId)) return null; const waited = pickComboTarget(config, comboId, { + randomSeed: options.randomSeed, allowCooldownReprobe: false, cooldownReprobeTargets, exclude: excluded, @@ -768,6 +775,7 @@ export async function pickComboTargetWithWait( const remainingWaitMs = options.waitForCooldownMs - spentWaitMs; const pick = pickComboTarget(config, comboId, { + randomSeed: options.randomSeed, allowCooldownReprobe: false, cooldownReprobeTargets, exclude: excluded, @@ -851,11 +859,13 @@ export function clearComboSelectionState(comboId?: string): void { selectionState.delete(comboId); } -export function tryPickComboModel(config: OcxConfig, modelId: string, preview = false): ComboPick | null { +export function tryPickComboModel( + config: OcxConfig, modelId: string, preview = false, eligible?: (target: NormalizedComboTarget) => boolean, randomSeed?: string, +): ComboPick | null { const comboId = resolveComboId(config, modelId); if (!comboId) return null; if (!getCombo(config, comboId)) throw new UnknownComboError(comboId); - const picked = pickComboTarget(config, comboId, { preview, allowCooldownReprobe: false }); + const picked = pickComboTarget(config, comboId, { preview, allowCooldownReprobe: false, eligible, randomSeed }); if (!picked) throw new NoAvailableComboTargetsError(comboId, comboExclusion(config, comboId)); return picked; } diff --git a/src/generated/model-metadata.ts b/src/generated/model-metadata.ts index 319385922fe..9821493232c 100644 --- a/src/generated/model-metadata.ts +++ b/src/generated/model-metadata.ts @@ -14,6 +14,7 @@ export interface ModelMetadata { const PROVIDER_ALIASES: Record = { "xai": "xai", + "xai-apikey": "xai", "anthropic": "anthropic", "anthropic2": "anthropic", "anthropic-apikey": "anthropic", diff --git a/src/images/loop.ts b/src/images/loop.ts index a6ffc138489..08025bb9de9 100644 --- a/src/images/loop.ts +++ b/src/images/loop.ts @@ -346,6 +346,13 @@ export interface ImageBridgeDeps { fetchForRequest?: (request: AdapterRequest, parsed: OcxParsedRequest, adapter: ProviderAdapter, recovery?: AttemptRecoveryKind) => typeof globalThis.fetch; /** Reserve the routed provider's next request-start slot before each adapter dispatch. */ waitForRequestSlot?: (signal?: AbortSignal) => Promise; + /** + * Physical fetch for one runTurn iteration, bound to the slot the loop reserved for it. A + * stateful transport (Cursor RunSSE + BidiAppend) sends several requests per turn; the caller + * makes later sends wait on start intervals only (`turnScopedPacing`), because the turn already + * holds the concurrency lease and a second one would wait on itself. + */ + runTurnFetch?: (pacingSlot: ProviderRequestSlot | undefined) => typeof globalThis.fetch; /** Raw adapter usage at the terminal event, pre wire-normalization (see bridgeToResponsesSSE onUsage). */ onUsage?: (usage: OcxUsage | undefined) => void; proxyErrorContext?: object; @@ -579,6 +586,7 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise { @@ -756,7 +764,7 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise 0 && attempts[attempts.length - 1]?.ordinal === attempt.ordinal; + return physicalUsageAttempts(entry.attempts ?? []).at(-1) === attempt; } function classifyOutcome(entry: PersistedUsageEntry, attempt: PersistedUsageAttempt): PassiveProductionOutcome { @@ -111,7 +111,8 @@ export function derivePassiveProductionSignals( scan: for (let rowIndex = scanRows.length - 1; rowIndex >= 0; rowIndex--) { const entry = scanRows[rowIndex]!; - const attempts = entry.attempts ?? []; + // Only real upstream sends are production evidence; the first send of an ordinal wins. + const attempts = physicalUsageAttempts(entry.attempts ?? []); for (let attemptIndex = attempts.length - 1; attemptIndex >= 0; attemptIndex--) { const signal = signalFor(entry, attempts[attemptIndex]!); if (signal?.subjectId !== subjectId) continue; diff --git a/src/lib/agentradar-account-pin.ts b/src/lib/agentradar-account-pin.ts new file mode 100644 index 00000000000..d665789fb05 --- /dev/null +++ b/src/lib/agentradar-account-pin.ts @@ -0,0 +1,41 @@ +import type { OcxConfig } from "../types"; + +const ACCOUNT_PIN = Symbol.for("agentradar.accountPin"); +export interface AgentRadarAccountPin { provider: "claude" | "codex" | "grok" | "kimi"; accountId: string } +type PinnedConfig = OcxConfig & { [ACCOUNT_PIN]?: AgentRadarAccountPin }; +const servedAccounts = new WeakMap(); + +/** Only the authenticated AgentRadar embedding bridge may supply these headers. */ +export function agentRadarAccountPin(headers: Headers): AgentRadarAccountPin | null | undefined { + const accountId = headers.get("x-agentradar-account-id"); + const provider = headers.get("x-agentradar-account-provider"); + if (accountId === null && provider === null) return undefined; + if (process.env.OPENCODEX_AGENTRADAR !== "1" || !accountId || !/^[!-~]{1,256}$/.test(accountId)) return null; + return provider === "claude" || provider === "codex" || provider === "grok" || provider === "kimi" ? { provider, accountId } : null; +} + +export function agentRadarPin(config: OcxConfig): AgentRadarAccountPin | undefined { + return (config as PinnedConfig)[ACCOUNT_PIN]; +} + +export function agentRadarPinnedConfig(config: T, pin: AgentRadarAccountPin): T { + return { ...config, [ACCOUNT_PIN]: pin, + ...(pin.provider === "claude" ? { webSearchSidecar: { ...config.webSearchSidecar, backend: "anthropic" as const, anthropicInstance: "anthropic" as const } } : {}) }; +} + +/** Request-local attribution only, never serialized into ordinary request logs. */ +export function noteAgentRadarAccountDispatch(context: object, accountId: string, reason?: string, sessionKey?: string | null): void { + if (process.env.OPENCODEX_AGENTRADAR !== "1") return; + servedAccounts.set(context, accountId); + if (reason) { + const digest = (value: string) => new Bun.CryptoHasher("sha256").update(value).digest("hex").slice(0, 8); + console.error(`[ar-acct] a=${digest(accountId)} r=${reason} s=${sessionKey ? digest(sessionKey) : "none"}`); + } +} + +export function withAgentRadarServedAccount(response: Response, context: object): Response { + const accountId = servedAccounts.get(context); + if (!accountId || process.env.OPENCODEX_AGENTRADAR !== "1") return response; + response.headers.set("x-agentradar-served-account-id", accountId); + return response; +} diff --git a/src/lib/errors.ts b/src/lib/errors.ts index 161834fe721..ff86ce55b92 100644 --- a/src/lib/errors.ts +++ b/src/lib/errors.ts @@ -1,5 +1,6 @@ import { formatRetryAfterAdvice, parseRetryAfterFromMessage } from "./retry-delay"; import { isMetaMusePlanFailure, META_MUSE_PLAN_REFUSAL_MESSAGE } from "../providers/meta-muse-plan-refusal"; +import { ANTHROPIC_SPEND_LIMIT, EXTRA_USAGE_ROTATION } from "../providers/anthropic-billing-refusal"; export interface OcxErrorPayload { message: string; @@ -426,6 +427,56 @@ export function isUpstreamResetReplayRefusedMessage(text: string): boolean { ); } +/** `Provider error ` or `Provider error : `, exactly as this proxy writes it. */ +const EXPLICIT_PROVIDER_STATUS = /^Provider error ([1-5]\d\d)(?::|$)/; +const PROVIDER_ERROR_400_PREFIX = /^Provider error (?:400|422):\s*/i; +// Anthropic's quota refusals, start-anchored because a validation 400 echoes client values after +// its own opening: a spent extra-usage balance and a spend limit. +const isAnthropicQuotaRefusal = (text: string): boolean => { + const trimmed = text.trimStart(); + return EXTRA_USAGE_ROTATION.test(trimmed) || ANTHROPIC_SPEND_LIMIT.test(trimmed); +}; + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +/** + * Quota evidence in an HTTP 400 or 422, read from fields, never from words anywhere in the text. + * A 400/422 is a verdict on the request and echoes client values ("...: toolu_insufficient_quota"), + * so matching words let one crafted request read as a spent account and cool a combo provider. + * After this proxy's `Provider error 400: ` / `422: ` prefix it is quota only as a JSON envelope + * whose `error.code`/`error.type` field is `insufficient_quota`, whose `error.code` is Volcengine + * Ark's `InvalidSubscription` (https://www.volcengine.com/docs/82379/1299023), or Anthropic's + * envelope whose `error.message` starts with a quota refusal wording (extra usage, spend limit); a nested `Provider error 400` + * message is read the same way, at most three deep. An `invalid_request_error: ` text counts only + * when it then starts with that wording (an already-extracted `error.message`). Truncated or + * malformed JSON is never quota. Other prose is this proxy's own adapter wording, not an echoing + * provider body, and keeps the word rules (see providerBadRequestBody). + */ +export function isQuotaBadRequestMessage(message: string): boolean { + let text = message.trim(); + for (let depth = 0; depth < 3; depth += 1) { + text = text.replace(PROVIDER_ERROR_400_PREFIX, ""); + if (!text.startsWith("{")) return isAnthropicQuotaRefusal(text.replace(/^invalid_request_error:\s*/i, "")); + let payload: unknown; + try { payload = JSON.parse(text); } catch { return false; } + if (!isRecord(payload) || !isRecord(payload.error)) return false; + const { code, type, message: inner } = payload.error; + if (code === "insufficient_quota" || type === "insufficient_quota" || code === "InvalidSubscription") return true; + if (typeof inner !== "string") return false; + if (payload.type === "error" && type === "invalid_request_error" && isAnthropicQuotaRefusal(inner)) return true; + if (!PROVIDER_ERROR_400_PREFIX.test(inner.trim())) return false; + text = inner.trim(); + } + return false; +} + +/** A 400/422 whose text is the provider's own body: a JSON envelope or an extracted typed message. */ +function providerBadRequestBody(message: string): boolean { + return /^(?:\{|invalid_request_error:)/i.test(message.trim().replace(PROVIDER_ERROR_400_PREFIX, "")); +} + export function classifyError(status: number, type: string, message: string): OcxErrorPayload { const text = message.toLowerCase(); if (type === SPEND_LEDGER_STORAGE_UNAVAILABLE_CODE) { @@ -494,10 +545,15 @@ export function classifyError(status: number, type: string, message: string): Oc // The Cursor adapter's classified rate-limit prefix is authoritative: its DETAIL may echo // quota wording ("... quota exhausted") that would otherwise hit the insufficient_quota // branch below and break the planned retry-with-backoff contract (WP3 review blocker 1). - if (text.includes("cursor rate limit exceeded")) { + // + // No quota or rate verdict on a 400 or 422 provider body comes from words anywhere in its text: + // see isQuotaBadRequestMessage. Those verdicts hop a combo and cool its target or provider. + const badRequest = (status === 400 || status === 422) && providerBadRequestBody(message); + if (!badRequest && text.includes("cursor rate limit exceeded")) { return { message, type: "rate_limit_error", code: "rate_limit_exceeded" }; } - if ( + // A caller's structured `type` is a field too, not words in the text. + if (badRequest ? type === "insufficient_quota" || isQuotaBadRequestMessage(message) : ( text.includes("insufficient_quota") || text.includes("exceeded your current quota") || text.includes("quota exhausted") || @@ -505,8 +561,10 @@ export function classifyError(status: number, type: string, message: string): Oc text.includes("account quota exceeded") || text.includes("monthly quota exceeded") || text.includes("weekly quota exceeded") || - text.includes("daily quota exceeded") - ) { + text.includes("daily quota exceeded") || + // Anthropic answers a spent extra-usage balance with HTTP 400 prose; only the opening counts. + isAnthropicQuotaRefusal(message) + )) { return { message, type: "insufficient_quota", code: "insufficient_quota" }; } // A refusal this proxy made itself, kept apart from the provider rate limits below. The HTTP @@ -518,14 +576,15 @@ export function classifyError(status: number, type: string, message: string): Oc return { message, type: "rate_limit_error", code: SEND_BUDGET_EXHAUSTED_CODE }; } if ( - status === 429 || - text.includes("rate limit") || - text.includes("rate limited") || - text.includes("too many requests") || - text.includes("resource_exhausted") || - text.includes("resource exhausted") || - text.includes("throttlingexception") || - text.includes("throttling") + status === 429 || (!badRequest && ( + text.includes("rate limit") || + text.includes("rate limited") || + text.includes("too many requests") || + text.includes("resource_exhausted") || + text.includes("resource exhausted") || + text.includes("throttlingexception") || + text.includes("throttling") + )) ) { return { message, type: "rate_limit_error", code: "rate_limit_exceeded" }; } @@ -617,7 +676,10 @@ export function classifyError(status: number, type: string, message: string): Oc export function isRateLimitOrQuotaFailureMessage(message: string): boolean { const normalized = String(message ?? "").trim(); if (!normalized) return false; - const numericStatus = Number(normalized); + // This proxy's `Provider error ` prefix states the status; the body after it echoes + // client values, so an explicit 400 is read by classifyError's 400 rules, not by its words. + const explicit = EXPLICIT_PROVIDER_STATUS.exec(normalized); + const numericStatus = explicit ? Number(explicit[1]) : Number(normalized); if (numericStatus === 429 || numericStatus === 402) return true; const statusHint = Number.isInteger(numericStatus) && numericStatus > 0 ? numericStatus : 0; const classified = classifyError(statusHint, "", normalized); @@ -629,7 +691,9 @@ export function isRateLimitOrQuotaFailureMessage(message: string): boolean { ) { return true; } - // Retained quota cue used by subagent health before classifyError covered it. + // Retained quota cue used by subagent health before classifyError covered it. Not for an + // explicit other 4xx: that is a verdict on the request, and the words may be the client's. + if (explicit && statusHint >= 400 && statusHint < 500) return false; return normalized.toLowerCase().includes("usage limit"); } @@ -642,6 +706,11 @@ export function inferHttpStatusFromAdapterMessage(message: string): number { if (isClientClosedMessage(lower)) return 499; // Codex Transport maps cyber_policy only on HTTP 400 (SSE is code-based). if (isCyberPolicyMessage(lower)) return 400; + // This proxy's own `Provider error ` prefix states the upstream status outright. The + // text after it echoes client values, so a 400 quoting "rate limit" must stay 400 rather than + // turn into a 429 that cools a combo target. + const explicit = EXPLICIT_PROVIDER_STATUS.exec(message.trimStart()); + if (explicit) return Number(explicit[1]); // See classifyError: this prefix now only means explicit request-size overflow (400); // quota-style Cursor resource exhaustion carries the rate-limit prefix and maps to 429. if (lower.includes("cursor resource limit exceeded")) return 400; diff --git a/src/lib/request-failure-model.ts b/src/lib/request-failure-model.ts index 1f78488789f..20396511ca6 100644 --- a/src/lib/request-failure-model.ts +++ b/src/lib/request-failure-model.ts @@ -225,6 +225,9 @@ const RECOVERY_KIND_CAUSE = { "rate-limit-429": "rate-limit", "anthropic-oauth-429": "rate-limit", "oauth-account-429": "rate-limit", + // The Codex pool rotated off an account whose window the body proved spent: the resend goes to + // a different credential, so it is funded, like its sibling `oauth-account-429`. + "codex-account-retry": "rate-limit", // A 402 is the account's quota, not a transient limit; only another account can serve it. "oauth-account-402": "quota-exhausted", "image-413": "payload-too-large", diff --git a/src/lib/upstream-refusal-log.ts b/src/lib/upstream-refusal-log.ts new file mode 100644 index 00000000000..ec55efc4bbc --- /dev/null +++ b/src/lib/upstream-refusal-log.ts @@ -0,0 +1,184 @@ +import { createHash } from "node:crypto"; +import { sanitizeLogMetadataString } from "./redact"; + +/** + * One structured log line per upstream 4xx, so an operator can learn which parameter or size limit + * a provider refuses without reading anything a client sent. + * + * The line carries only the HTTP status, the provider and model ids, the `type` / `code` / `param` + * of the provider's JSON error envelope, and the outgoing body size. It never carries the envelope's + * `message`, a request or response body, a header, a credential or an account id: `message` is free + * text that providers echo client content into, so no code path here ever reads it. `type`, `code` + * and `param` are provider strings too, so each is logged verbatim only when it is a short + * identifier (`invalid_request_error`, `reasoning.effort`, `tools[3].name`); anything else is + * logged as `#` plus 8 hex of SHA-256 over its redacted value, which still groups repeats. + * + * Written only on an AgentRadar gateway (`OPENCODEX_AGENTRADAR=1`), so a self-hosted gateway gains + * no new output. + */ +const enabled = () => process.env.OPENCODEX_AGENTRADAR === "1"; + +/** Each envelope field is cut to this many characters, newlines and control characters removed. */ +export const REFUSAL_FIELD_MAX_CHARS = 80; +/** An error envelope is small; read no more than this of a refusal body. */ +const ENVELOPE_READ_LIMIT_BYTES = 64 * 1024; +/** A refusal body that never finishes is abandoned, not held. */ +const ENVELOPE_READ_TIMEOUT_MS = 2_000; +/** Statuses the account and key rotation logs already report (`[key-failover]`, `[anthropic-pool]`). */ +const ROTATION_LOGGED_STATUSES: ReadonlySet = new Set([401, 429]); +/** An inbound body above this is logged at the gateway entry (a 6 MB request cap sits just above it). */ +export const INBOUND_SIZE_LOG_THRESHOLD_BYTES = 5 * 1024 * 1024; + +export interface RefusalEnvelopeFields { + type?: string; + code?: string; + param?: string; +} + +/** A short identifier; a 20+ character alphanumeric run (request id, token) is not one. */ +const IDENTIFIER_RE = /^[A-Za-z0-9][A-Za-z0-9_.:[\]-]{0,63}$/; +const OPAQUE_RUN_RE = /[A-Za-z0-9]{20,}/; + +function envelopeScalar(value: unknown): string | undefined { + const text = sanitizeLogMetadataString(typeof value === "number" && Number.isFinite(value) ? String(value) : value, REFUSAL_FIELD_MAX_CHARS); + if (text === undefined) return undefined; + return IDENTIFIER_RE.test(text) && !OPAQUE_RUN_RE.test(text) + ? text + : `#${createHash("sha256").update(text).digest("hex").slice(0, 8)}`; +} + +/** + * Read `type`, `code` and `param` from a provider's JSON error envelope (`error.*`, or the top level + * of a flat envelope). Anything else, including `message`, is left unread. A body that is not + * JSON yields no fields. + */ +export function refusalEnvelopeFields(text: string): RefusalEnvelopeFields { + let parsed: unknown; + try { + parsed = JSON.parse(text); + } catch { + return {}; + } + // Some providers wrap the envelope in a one-element array. + const root = Array.isArray(parsed) ? parsed[0] : parsed; + if (root === null || typeof root !== "object") return {}; + // `{"error": {...}}` is the common envelope. Some gateways answer flat, with the fields at the top + // level and no `error` object; an `error` that is a bare string is that flat shape too. + const nested = (root as { error?: unknown }).error; + const source = nested !== null && typeof nested === "object" && !Array.isArray(nested) ? nested : root; + const { type, code, param } = source as { type?: unknown; code?: unknown; param?: unknown }; + const fields: RefusalEnvelopeFields = {}; + const t = envelopeScalar(type); + const c = envelopeScalar(code); + const p = envelopeScalar(param); + if (t !== undefined) fields.type = t; + if (c !== undefined) fields.code = c; + if (p !== undefined) fields.param = p; + return fields; +} + +/** Byte length of an outgoing request body, or undefined when it is a stream or form that cannot be measured without reading it. */ +export function outgoingBodyBytes(body: unknown): number | undefined { + if (typeof body === "string") return Buffer.byteLength(body); + if (body instanceof ArrayBuffer || ArrayBuffer.isView(body)) return body.byteLength; + if (typeof Blob !== "undefined" && body instanceof Blob) return body.size; + if (body instanceof URLSearchParams) return Buffer.byteLength(body.toString()); + return undefined; +} + +export interface UpstreamRefusal extends RefusalEnvelopeFields { + status: number; + provider?: string; + model?: string; + /** Outgoing request body size in bytes, when known. */ + bytes?: number; +} + +function idField(value: string | undefined): string { + // Ids are configuration-controlled and can be token-shaped: redacted, bounded and JSON-escaped. + return JSON.stringify(sanitizeLogMetadataString(value, REFUSAL_FIELD_MAX_CHARS) ?? "-"); +} + +function envelopeField(name: string, value: string | undefined): string { + return value === undefined ? "" : ` ${name}=${JSON.stringify(value)}`; +} + +/** Write the one line. `console.warn`, like the rotation logs this complements. */ +export function logUpstreamRefusal(refusal: UpstreamRefusal): void { + if (!enabled()) return; + console.warn( + `[upstream-refusal] status=${refusal.status} provider=${idField(refusal.provider)} model=${idField(refusal.model)}` + + envelopeField("type", refusal.type) + envelopeField("code", refusal.code) + envelopeField("param", refusal.param) + + ` bytes=${refusal.bytes ?? "-"}`, + ); +} + +async function readEnvelopeText(response: Response): Promise { + const reader = response.body?.getReader(); + if (!reader) return ""; + const timer = setTimeout(() => { void reader.cancel().catch(() => undefined); }, ENVELOPE_READ_TIMEOUT_MS); + const chunks: Uint8Array[] = []; + let total = 0; + try { + while (total < ENVELOPE_READ_LIMIT_BYTES) { + const { done, value } = await reader.read(); + if (done) break; + chunks.push(value); + total += value.byteLength; + } + } finally { + clearTimeout(timer); + void reader.cancel().catch(() => undefined); + } + return Buffer.concat(chunks).toString("utf8"); +} + +/** + * Log the refusal shape of an upstream 4xx without disturbing it. The caller's response is returned + * untouched; the envelope is read from a clone, off the request path, and a failure to read it + * degrades to a line with the status, ids and size only. + */ +export function observeUpstreamRefusal( + response: Response, + context: { provider?: string; model?: string; body?: unknown }, +): void { + const { status } = response; + if (!enabled()) return; + if (status < 400 || status >= 500 || ROTATION_LOGGED_STATUSES.has(status)) return; + const base: UpstreamRefusal = { status, provider: context.provider, model: context.model }; + const bytes = outgoingBodyBytes(context.body); + if (bytes !== undefined) base.bytes = bytes; + let copy: Response; + try { + copy = response.clone(); + } catch { + logUpstreamRefusal(base); + return; + } + readEnvelopeText(copy).then( + text => logUpstreamRefusal({ ...base, ...refusalEnvelopeFields(text) }), + () => logUpstreamRefusal(base), + ); +} + +/** + * The fork's own 413: a request or response that would not fit the translation buffer. + * `limit` is the cap that was hit; `bytes` is the size when the caller knows it. + */ +export function logTranslationBufferLimit( + where: "request" | "response" | "stream", + context: { model?: string; bytes?: number; limit?: number }, +): void { + if (!enabled()) return; + console.warn( + `[upstream-refusal] status=413 provider="-" model=${idField(context.model)} code="translation_buffer_limit"` + + ` where=${where} bytes=${context.bytes ?? "-"} limit=${context.limit ?? "-"}`, + ); +} + +/** Log an inbound request above the threshold, from its declared Content-Length, before any body is read. */ +export function logOversizedInbound(pathname: string, declaredBytes: number): void { + if (!enabled()) return; + if (!Number.isFinite(declaredBytes) || declaredBytes <= INBOUND_SIZE_LOG_THRESHOLD_BYTES) return; + console.warn(`[inbound-size] path=${JSON.stringify(sanitizeLogMetadataString(pathname, REFUSAL_FIELD_MAX_CHARS) ?? "-")} bytes=${declaredBytes}`); +} diff --git a/src/lib/usage-line-log.ts b/src/lib/usage-line-log.ts new file mode 100644 index 00000000000..a23165ae5fc --- /dev/null +++ b/src/lib/usage-line-log.ts @@ -0,0 +1,79 @@ +import { createHash } from "node:crypto"; +import { baseProviderLabel } from "../providers/label"; +import { sanitizeLogMetadataString } from "./redact"; + +/** + * One structured log line per finished model call, so an operator can work out, per calling key, + * the share of calls and tokens on each model from the process log alone. + * + * The line carries only the HTTP status, the provider and model ids, three token counts, an 8-hex + * hash of the gateway API key's record id and the duration. It never carries message text, a + * request or response body, a header, a secret key value, an account label or an email: the input is + * the fork's own final request record, and only the fields named below are read from it. + * + * It is written only on an AgentRadar gateway (`OPENCODEX_AGENTRADAR=1`, the condition the + * `[ar-acct]` line uses), so a self-hosted gateway gains no new output. + */ + +/** Each id is cut to this many characters, newlines and control characters removed. */ +export const USAGE_LOG_ID_MAX_CHARS = 80; + +/** The fields of the final request record this line reads. Nothing else is looked at. */ +export interface UsageLogInput { + status: number; + provider?: string; + model?: string; + usage?: { + inputTokens?: number; + outputTokens?: number; + cachedInputTokens?: number; + cacheReadInputTokens?: number; + }; + /** The configured key's record id (never the secret key value). Absent for environment or loopback admission. */ + apiKeyId?: string; + durationMs?: number; +} + +/** The OAuth (`o`) and API-key (`k`) account labels `baseProviderLabel` leaves in place. */ +const ACCOUNT_SUFFIX_RE = /-(?:o[a-f0-9]{6}|k[a-f0-9]{32})$/; + +function idField(value: string | undefined): string { + return JSON.stringify(sanitizeLogMetadataString(value, USAGE_LOG_ID_MAX_CHARS) ?? "-"); +} + +/** + * A pooled provider is logged with the serving account's label as a suffix (`anthropic-p2e22d0`). + * That label names one account, which this line must not, so the suffix is dropped. + */ +function providerField(value: string | undefined): string { + const base = typeof value === "string" && value !== "" ? baseProviderLabel(value).replace(ACCOUNT_SUFFIX_RE, "") : value; + return idField(base); +} + +function count(value: number | undefined): string { + return typeof value === "number" && Number.isFinite(value) && value >= 0 ? String(Math.round(value)) : "-"; +} + +/** First 8 hex of sha256 of the key record id, or "-" when the call carried no configured key. */ +export function hashedKeyId(apiKeyId: string | undefined): string { + if (typeof apiKeyId !== "string" || apiKeyId === "") return "-"; + return createHash("sha256").update(apiKeyId).digest("hex").slice(0, 8); +} + +/** The line, without writing it. */ +export function usageLogLine(input: UsageLogInput): string { + const { usage } = input; + const status = Number.isInteger(input.status) ? String(input.status) : "-"; + return `[usage] status=${status} provider=${providerField(input.provider)} model=${idField(input.model)}` + + ` in=${count(usage?.inputTokens)} out=${count(usage?.outputTokens)}` + + ` cached=${count(usage?.cacheReadInputTokens ?? usage?.cachedInputTokens)}` + + ` key=${hashedKeyId(input.apiKeyId)} ms=${count(input.durationMs)}`; +} + +/** Write the one line, on an AgentRadar gateway only. Never throws: logging must not fail a request. */ +export function logUsageLine(input: UsageLogInput): void { + if (process.env.OPENCODEX_AGENTRADAR !== "1") return; + try { + console.info(usageLogLine(input)); + } catch { /* request logging must never fail a user request */ } +} diff --git a/src/oauth/anthropic-account-refusal.ts b/src/oauth/anthropic-account-refusal.ts index 8f03436bacc..7e17cca6b31 100644 --- a/src/oauth/anthropic-account-refusal.ts +++ b/src/oauth/anthropic-account-refusal.ts @@ -1,8 +1,8 @@ import { wasAccountAutoPaused } from "./account-auto-pause"; -import { anthropicModelFamily } from "./anthropic-model-quota"; +import { anthropicModelFamily, anthropicModelQuotaFor } from "./anthropic-model-quota"; /** Narrow pre-output account recovery, fenced to the bearer that physically sent the turn. */ import { readBoundedResponseBody } from "../lib/bounded-body"; -import { classifyAnthropic429, anthropicRetryAfterMs, anthropicRatePolicyFor, ANTHROPIC_SHORT_RETRY_MS, ANTHROPIC_MAX_INLINE_THROTTLE_MS } from "./anthropic-rate-limit-policy"; +import { anthropicClaimedFamily, classifyAnthropic429, anthropicRetryAfterMs, anthropicRatePolicyFor, ANTHROPIC_SHORT_RETRY_MS, ANTHROPIC_MAX_INLINE_THROTTLE_MS } from "./anthropic-rate-limit-policy"; import { isNonReplayableResponse, sleepWithAbort } from "../lib/upstream-retry"; import { credentialGeneration, getAccountCredentialWithStatus, markAccountNeedsReauthIfGeneration } from "./store"; import type { OAuthAccessSnapshot } from "./index"; @@ -10,6 +10,7 @@ import type { OcxConfig } from "../types"; import type { AnthropicRouteDecision } from "./anthropic-model-routes"; import { anthropicRoutingFor } from "./anthropic-routing"; import { configuredAnthropicInstance } from "../providers/anthropic-instance"; +import { readAnthropicExtraUsageRefusal } from "../providers/anthropic-billing-refusal"; import type { AnthropicInstanceId } from "../providers/anthropic-instance-id"; import { captureAnthropicPhysicalSendOwnership, anthropicPhysicalSendOwnershipIsCurrent, type AnthropicPhysicalSendOwnership } from "./anthropic-send-ownership"; @@ -17,7 +18,7 @@ import { captureAnthropicPhysicalSendOwnership, anthropicPhysicalSendOwnershipIs const responseCredentials = new WeakMap(); type RetryState = { firstAccountId: string; sameAccount: boolean; detour: boolean }; const retryStatesByInstance = new Map>(); -const verdicts = new WeakMap>(); +const verdicts = new WeakMap>(); /** Legacy compatibility only; physical callers must capture ownership before fetch and use ForSend. */ export function bindAnthropicRefusalCredential(response: Response, snapshot: OAuthAccessSnapshot, providerAccountUuid?: string): void { @@ -64,7 +65,8 @@ async function isRevokedOAuthToken(response: Response, signal?: AbortSignal): Pr } catch { return false; } } -async function isAccountRefusal(response: Response, signal?: AbortSignal): Promise { +/** True for an account-scoped refusal; "org-disabled" when the organization turned subscription access off. */ +async function isAccountRefusal(response: Response, signal?: AbortSignal): Promise { try { const body = await readBoundedResponseBody(response.clone(), { signal }); if (!body.displaySafe || body.truncated) return false; @@ -78,9 +80,10 @@ async function isAccountRefusal(response: Response, signal?: AbortSignal): Promi if ("code" in error && error.code != null && (typeof error.code !== "string" || !["subscription_required", "insufficient_quota", "permission_denied", "oauth_org_not_allowed"].includes(error.code))) return false; // The organization disabled subscription access: an account-scoped verdict, cooled like its siblings. - if (error.type === "oauth_org_not_allowed" || ("code" in error && error.code === "oauth_org_not_allowed")) return true; + if (error.type === "oauth_org_not_allowed" || ("code" in error && error.code === "oauth_org_not_allowed")) return "org-disabled"; const message = error.message.trim(); - if (error.type === "permission_error" && /^(?:Your account does not have access to Claude Code|Your (?:Claude )?subscription (?:has expired|is (?:expired|inactive))|Your account does not have an active subscription|(?:Your|This) organization has disabled Claude subscription access for Claude Code(?:\. Use an Anthropic API key instead, or ask your admin to enable access)?)[.!]?$/i.test(message)) return true; + if (error.type === "permission_error" && /^(?:Your|This) organization has disabled Claude subscription access for Claude Code(?:\. Use an Anthropic API key instead, or ask your admin to enable access)?[.!]?$/i.test(message)) return "org-disabled"; + if (error.type === "permission_error" && /^(?:Your account does not have access to Claude Code|Your (?:Claude )?subscription (?:has expired|is (?:expired|inactive))|Your account does not have an active subscription)[.!]?$/i.test(message)) return true; return (error.type === "billing_error" || error.type === "permission_error") && /^Your credit balance is too low to access the Anthropic API\.(?: Please go to Plans & Billing to upgrade or purchase credits\.)?$/i.test(message); } catch { @@ -104,6 +107,8 @@ export async function rotateAnthropicAccountOnResponseForInstance( requestKey?: object; allow429Recovery?: boolean; allowAccountRefusal?: boolean; + /** AgentRadar terminal re-ask (.27): of the account refusals only an org-disabled 403 rotates; no 401. */ + orgDisabledRefusalOnly?: boolean; /** Applies to alternate ranking only; the shared same-account throttle remains allowed. */ excludedAccountIds?: ReadonlySet; /** Native dispatch re-reads operator policy after asynchronous classification/waits. */ @@ -132,8 +137,9 @@ export async function rotateAnthropicAccountOnResponseForInstance( && credentialGeneration(row.credential) === sent.generation && (!sent.checkProviderUuid || row.credential.accountId === sent.providerAccountUuid) ? row : undefined; }; + let orgDisabled = false; if (response.status === 401) { - if (options.allowAccountRefusal === false) return null; + if (options.allowAccountRefusal === false || options.orgDisabledRefusalOnly) return null; let verdict = verdicts.get(response); if (!verdict) { verdict = isRevokedOAuthToken(response, options.signal); verdicts.set(response, verdict); } if (!await verdict || options.signal?.aborted || !ownedCurrent()) return null; @@ -158,9 +164,16 @@ export async function rotateAnthropicAccountOnResponseForInstance( const current = ownedCurrent(); if (!current) return null; options.currentDecision?.(); - const kind = classifyAnthropic429(response.headers); + const kind = classifyAnthropic429(response.headers, Date.now(), options.model); if (kind === "family-quota") { - if (anthropicModelFamily(options.model) !== "Fable" || !options.canRetry || options.allow429Recovery === false) return null; + // A claimed family has no 7d_oi header for the response recorder: record its exclusion here. + const claimed = anthropicClaimedFamily(response.headers, options.model); + if (claimed) { + const reset = Number(response.headers.get("anthropic-ratelimit-unified-reset")); + anthropicModelQuotaFor(instance).observeAnthropicFamilyQuota(sent.accountId, [{ label: claimed, scope: "model", percent: 100, + rejected: true, passiveObservedAt: Date.now(), ...(reset > 0 && Number.isFinite(reset) ? { resetAt: reset * 1000 } : {}) }], Date.now()); + } + if ((!claimed && anthropicModelFamily(options.model) !== "Fable") || !options.canRetry || options.allow429Recovery === false) return null; return pickAlternateAnthropicAccount(options.config, sent.accountId, Date.now(), (options.currentDecision ? options.currentDecision() : options.decision ?? null), options.model, options.excludedAccountIds); } if (kind !== "shared-quota") { @@ -176,7 +189,8 @@ export async function rotateAnthropicAccountOnResponseForInstance( // A concurrent committed selection may have moved the same-account proposal. if (state.firstAccountId !== sent.accountId) state.detour = true; const wait = Math.max(delay, (anthropicRatePauseUntil(sent.accountId) ?? now) - now); - if (!state.sameAccount && wait <= ANTHROPIC_MAX_INLINE_THROTTLE_MS && !current.paused) { + // AgentRadar gateways fail over at once, as the hosted contract did, instead of waiting in place. + if (!state.sameAccount && wait <= ANTHROPIC_MAX_INLINE_THROTTLE_MS && !current.paused && process.env.OPENCODEX_AGENTRADAR !== "1") { state.sameAccount = true; // Millisecond rounding can wake just before this deadline; later extensions still bind. const retryAt = now + wait; @@ -201,27 +215,39 @@ export async function rotateAnthropicAccountOnResponseForInstance( return null; } } else { - if (response.status !== 403 || options.allowAccountRefusal === false) return null; + // A 400 is account evidence only as Anthropic's exact spent-extra-usage envelope; any other + // 400 is the request's own fault and would fail the same way on every account. + if (response.status === 400 ? options.allow429Recovery === false : response.status !== 403 || options.allowAccountRefusal === false) return null; let verdict = verdicts.get(response); if (!verdict) { - verdict = isAccountRefusal(response, options.signal); + verdict = response.status === 400 ? readAnthropicExtraUsageRefusal(response, options.signal) : isAccountRefusal(response, options.signal); verdicts.set(response, verdict); } - if (!await verdict || options.signal?.aborted) return null; + const refusal = await verdict; + if (!refusal || options.signal?.aborted) return null; + if (options.orgDisabledRefusalOnly && response.status === 403 && refusal !== "org-disabled") return null; // A late refusal must not cool a new credential stored while its body was being read. const current = ownedCurrent(); if (!current || current.paused || current.needsReauth || credentialGeneration(current.credential) !== sent.generation || sent.checkProviderUuid && current.credential.accountId !== sent.providerAccountUuid) return null; + orgDisabled = refusal === "org-disabled" && process.env.OPENCODEX_AGENTRADAR === "1"; } const decision = options.currentDecision ? options.currentDecision() : options.decision; - const status = response.status === 429 ? 429 : 403; - if (!options.canRetry) { - recordAnthropicAccountRefusal(options.config, options.accountId, status, - response.headers.get("retry-after"), Date.now(), response.headers); - return null; + const status = response.status === 429 ? 429 : response.status === 400 ? 400 : 403; + const next = options.canRetry + ? rotateAnthropicAccountOnRefusal(options.config, options.accountId, status, + response.headers.get("retry-after"), options.sessionKey, Date.now(), response.headers, decision, options.model, options.excludedAccountIds) + : (recordAnthropicAccountRefusal(options.config, options.accountId, status, + response.headers.get("retry-after"), Date.now(), response.headers), null); + // AgentRadar gateways keep the hosted contract: an organization that turned subscription access off + // will not come back by waiting, so after rotating the account needs a new login, not a retry per cooldown. + if (orgDisabled) { + try { + await markAccountNeedsReauthIfGeneration(instance, sent.accountId, sent.generation, undefined, undefined, + store => anthropicPhysicalSendOwnershipIsCurrent(sent, store)); + } catch { /* the cooldown above still applies */ } } - return rotateAnthropicAccountOnRefusal(options.config, options.accountId, status, - response.headers.get("retry-after"), options.sessionKey, Date.now(), response.headers, decision, options.model, options.excludedAccountIds); + return next; } /** Compatibility entrypoint: legacy consumers always recover inside the primary pool. */ diff --git a/src/oauth/anthropic-identity.ts b/src/oauth/anthropic-identity.ts index 44c355c76f0..512e13d2403 100644 --- a/src/oauth/anthropic-identity.ts +++ b/src/oauth/anthropic-identity.ts @@ -25,9 +25,9 @@ export function normalizeAnthropicIdentity(value: unknown, access: string): Anth return proof.v === 1 && bound && proof.bearerSha256 === bound.bearerSha256 ? bound : undefined; } -/** Fixed destination, no redirects, bounded body and deadline; failures never include response data. */ -export const resolveAnthropicAccountIdentity: AnthropicIdentityResolver = async (access, signal) => { - if (!access || signal?.aborted) return undefined; +/** Fixed destination, no redirects, bounded body and deadline: the parsed profile or the refusal status. */ +async function readAnthropicProfile(access: string, signal?: AbortSignal): Promise<{ data?: unknown; status?: number }> { + if (!access || signal?.aborted) return {}; const deadline = AbortSignal.timeout(PROFILE_TIMEOUT_MS); const combined = signal ? AbortSignal.any([signal, deadline]) : deadline; try { @@ -39,30 +39,65 @@ export const resolveAnthropicAccountIdentity: AnthropicIdentityResolver = async }); if (!response.ok || response.redirected) { void response.body?.cancel().catch(() => {}); - return undefined; + return { status: response.status }; } const body = await readBoundedResponseBody(response, { signal: combined, maxBytes: PROFILE_MAX_BYTES, fatalUtf8: true, totalTimeoutMs: PROFILE_TIMEOUT_MS, inactivityTimeoutMs: PROFILE_TIMEOUT_MS, }); - if (body.truncated || body.oversized || body.timedOut || combined.aborted) return undefined; - const data: unknown = JSON.parse(body.text); - if (!data || typeof data !== "object" || !("account" in data)) return undefined; - const account = data.account; - const identity = account && typeof account === "object" && "uuid" in account - ? bindAnthropicIdentity(access, account.uuid) : undefined; - if (!identity) return undefined; - const organization = "organization" in data ? data.organization : undefined; - const org = organization && typeof organization === "object" - ? organization as Record : undefined; - const plan = normalizeAnthropicPlan(org?.rate_limit_tier) ?? normalizeAnthropicPlan(org?.organization_type); - return { ...identity, ...(plan ? { plan } : {}) }; + if (body.truncated || body.oversized || body.timedOut || combined.aborted) return {}; + return { data: JSON.parse(body.text) as unknown }; } catch { - // Unavailable identity is an explicit unresolved result, never an ownership guess. - return undefined; + return {}; } +} + +function profilePlan(data: unknown): string | undefined { + const organization = data && typeof data === "object" && "organization" in data ? data.organization : undefined; + const org = organization && typeof organization === "object" ? organization as Record : undefined; + return normalizeAnthropicPlan(org?.rate_limit_tier) ?? normalizeAnthropicPlan(org?.organization_type); +} + +/** Failures never include response data. */ +export const resolveAnthropicAccountIdentity: AnthropicIdentityResolver = async (access, signal) => { + const { data } = await readAnthropicProfile(access, signal); + // Unavailable identity is an explicit unresolved result, never an ownership guess. + if (!data || typeof data !== "object" || !("account" in data)) return undefined; + const account = data.account; + const identity = account && typeof account === "object" && "uuid" in account + ? bindAnthropicIdentity(access, account.uuid) : undefined; + if (!identity) return undefined; + const plan = profilePlan(data); + return { ...identity, ...(plan ? { plan } : {}) }; }; +export interface AnthropicPlanReading { plan: string | null; planUnknownReason?: string } +const planReadings = new Map }>(); + +/** + * A live plan read for an account whose credential has none stored yet (AgentRadar 20720). Keyed by + * the bearer's digest, so a refresh reads afresh; an hour on success, five minutes on failure. + */ +export function readAnthropicPlan(access: string): Promise { + const key = createHash("sha256").update(access).digest("hex"); + const now = Date.now(); + const hit = planReadings.get(key); + if (hit && now < hit.until) return hit.value; + // ponytail: cleared wholesale at 500 bearers; a pool never holds that many live logins. + if (planReadings.size >= 500) planReadings.clear(); + const value = readAnthropicProfile(access).then(({ data, status }): AnthropicPlanReading => { + if (status !== undefined) return { plan: null, planUnknownReason: `the plan lookup failed (HTTP ${status})` }; + if (data === undefined) return { plan: null, planUnknownReason: "the plan lookup did not answer" }; + const plan = profilePlan(data); + return plan ? { plan } : { plan: null, planUnknownReason: "Anthropic did not report a plan" }; + }); + planReadings.set(key, { until: now + 5 * 60_000, value }); + void value.then(result => { + if (result.plan && planReadings.get(key)?.value === value) planReadings.set(key, { until: now + 60 * 60_000, value }); + }); + return value; +} + /** A changed access bearer cannot inherit the previous bearer's proof. */ export function mergeAnthropicIdentity(fresh: OAuthCredentials, previous: OAuthCredentials): Pick { const before = normalizeAnthropicIdentity(previous.anthropicIdentity, previous.access); diff --git a/src/oauth/anthropic-rate-limit-policy.ts b/src/oauth/anthropic-rate-limit-policy.ts index d191e4139af..bdf4bb353fe 100644 --- a/src/oauth/anthropic-rate-limit-policy.ts +++ b/src/oauth/anthropic-rate-limit-policy.ts @@ -2,6 +2,7 @@ import type { AnthropicInstanceId } from "../providers/anthropic-instance-id"; import type { GenerationContext } from "../lib/state-store-sweeper"; import { credentialGeneration, getAccountCredential } from "./store"; +import { anthropicModelFamily, type AnthropicModelFamily } from "./anthropic-model-family"; export type AnthropicRateLimitKind = "shared-quota" | "family-quota" | "transient-rate" | "request-scoped-unknown"; export type RateLimitHeaders = Pick; export const ANTHROPIC_SHORT_RETRY_MS = 100; @@ -13,6 +14,16 @@ export const ANTHROPIC_MAX_INLINE_THROTTLE_MS = 1_000; */ export const ANTHROPIC_DETOUR_GRACE_MS = 5 * 60_000; +/** + * A unified refusal whose representative claim is the requested model's own weekly limit + * (`seven_day_opus` on an Opus request) spends only that family; the shared windows are open. + */ +export function anthropicClaimedFamily(headers: RateLimitHeaders, model?: string): AnthropicModelFamily | undefined { + const family = anthropicModelFamily(model); + return family && headers.get("anthropic-ratelimit-unified-status")?.trim() === "rejected" + && headers.get("anthropic-ratelimit-unified-representative-claim")?.trim() === `seven_day_${family.toLowerCase()}` ? family : undefined; +} + export function anthropicRetryAfterMs(value: string | null | undefined, now: number): number | undefined { const text = value?.trim(); if (!text || text.length > 128) return undefined; @@ -20,11 +31,14 @@ export function anthropicRetryAfterMs(value: string | null | undefined, now: num return Number.isFinite(new Date(at).getTime()) && at > now ? Math.ceil(at - now) : undefined; } -export function classifyAnthropic429(headers: RateLimitHeaders, now = Date.now()): AnthropicRateLimitKind { +export function classifyAnthropic429(headers: RateLimitHeaders, now = Date.now(), model?: string): AnthropicRateLimitKind { const rejected = (window: string) => headers.get(`anthropic-ratelimit-unified-${window}-status`)?.trim() === "rejected"; if (rejected("5h") || rejected("7d")) return "shared-quota"; - if (rejected("7d_oi")) return "family-quota"; + if (rejected("7d_oi") || anthropicClaimedFamily(headers, model)) return "family-quota"; if (headers.get("anthropic-ratelimit-unified-status")?.trim() === "rejected") return "shared-quota"; + // AgentRadar gateways keep the hosted contract: with no unified status at all, a 429 cools the + // account (Retry-After, else the default) and rotates, rather than a same-account retry. + if (process.env.OPENCODEX_AGENTRADAR === "1" && headers.get("anthropic-ratelimit-unified-status") === null) return "shared-quota"; // Retry-After is an account-local admission pause, not proof of spent subscription quota. if (anthropicRetryAfterMs(headers.get("retry-after"), now) !== undefined || headers.get("anthropic-ratelimit-unified-status")?.trim() === "allowed") return "transient-rate"; diff --git a/src/oauth/anthropic-routing.ts b/src/oauth/anthropic-routing.ts index 305a3d46417..0cc684521c4 100644 --- a/src/oauth/anthropic-routing.ts +++ b/src/oauth/anthropic-routing.ts @@ -1,3 +1,5 @@ +import { agentRadarPin } from "../lib/agentradar-account-pin"; +import { setComboPoolEligibilityProbe } from "../combos/failover"; import { clearAllAnthropicFamilyQuota, anthropicModelQuotaFor, anthropicModelPercents, anthropicModelRankPercent, anthropicModelWeeklyPercent } from "./anthropic-model-quota"; /** * Opt-in Anthropic OAuth account pool (#294). @@ -22,6 +24,8 @@ import { REFRESH_SKEW_MS } from "./refresh-policy"; import { captureOAuthAccountSelection, commitOAuthAccountSelection, credentialGeneration, getAccountSet, getAccountCredential, getAccountCredentialWithStatus } from "./store"; import type { OAuthAccessSnapshot } from "./index"; import { getCachedProviderAccountQuota } from "../providers/quota"; +import { recordAnthropicAccountQuotaFromHeadersForInstance } from "../providers/quota/account-cache"; +import { captureConfigGeneration } from "../lib/state-store-sweeper"; import { fallbackCodexAccountLogLabel } from "../codex/account-label"; import { normalizeAccountPoolStickyLimit, @@ -54,6 +58,8 @@ import { AnthropicSessionAffinityStore, anthropicAffinityDigest, AFFINITY_IDLE_T * hand it a plain `new Headers({...})`. */ export type AnthropicRateLimitHeaders = Pick; +/** 400 is Anthropic's exact spent-extra-usage refusal: quota evidence without a 429 classification. */ +export type AnthropicAccountRefusalStatus = 429 | 403 | 400; /** Backoff only when upstream supplies no usable deadline. */ const DEFAULT_COOLDOWN_MS = 60_000; @@ -122,6 +128,8 @@ interface AccountHealth { * Retry-After set the deadline. A usage-limit answer reports this, never an admission deadline. */ quotaResetAt?: number; + /** AgentRadar: a 429 with no unified status cools and rotates, but is not evidence of spent quota. */ + headerless?: boolean; } /** See `anthropicAccountHold`: the latest hold on one account and its known window reset. */ @@ -219,6 +227,8 @@ interface ScoredAccount { fiveHourTieBreak: number; /** True only under the opt-in weekly window; see compareScoredAccounts. */ knownFirst: boolean; + /** Weekly window only: a known 100% is a spent account and ranks behind an unknown reading. */ + spentLast: boolean; } function compareScoredAccounts(a: ScoredAccount, b: ScoredAccount): number { @@ -227,6 +237,8 @@ function compareScoredAccounts(a: ScoredAccount, b: ScoredAccount): number { // never opted in: an account measured at 100% would sort ahead of an unmeasured one purely // because it had a reading. The accepted scope preserves the five-hour default exactly. if (a.knownFirst && b.knownFirst && a.hasKnownUsage !== b.hasKnownUsage) { + // A known 100% is a spent account, not a preferred one: an unknown reading may have room. + if (a.spentLast && (a.hasKnownUsage ? a : b).score >= 100) return a.hasKnownUsage ? 1 : -1; return a.hasKnownUsage ? -1 : 1; } return a.score - b.score || a.fiveHourTieBreak - b.fiveHourTieBreak; @@ -257,6 +269,17 @@ export function formatAnthropicAccountOrdinal(accountId: string): string { return fallbackCodexAccountLogLabel(accountId); } +/** + * Name the serving Anthropic account in the account-qualified provider and, on AgentRadar gateways, + * also in the row/attempt `accountLogLabel` (`p`, the same digest), so per-account usage joins. + */ +export function stampAnthropicAccountForLog( + logCtx: { provider: string; accountLogLabel?: string }, instance: string, accountId: string, config?: OcxConfig, +): void { + logCtx.provider = formatAnthropicProviderForLog(instance, accountId, config); + if (process.env.OPENCODEX_AGENTRADAR === "1") logCtx.accountLogLabel = formatAnthropicAccountOrdinal(accountId); +} + export function formatAnthropicProviderForLog( providerName: string, accountId: string | null | undefined, @@ -283,6 +306,7 @@ export type AnthropicCooldownRecoverySettlement = "cleared" | "retained" | "supe export type AnthropicAccountSelectionReason = | "pool-disabled" | "affinity" + | "hash" | "active" | "lowest-usage" | "only-eligible" @@ -376,6 +400,7 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting // Undefined means this runtime has not admitted a selection yet; null means consumed. // The startup baseline comes from the authoritative store, never a second persisted pin. let manualPreference: OAuthAccountSelection | null | undefined; + let manualPreferenceIsBaseline = true; let manualSelectionGeneration = 0; let manualResetPending = false; /** Automatic pointer moves preserve affinity; manual selection revokes older bindings. */ @@ -536,6 +561,7 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting clearAnthropicCooldownGenerations(); sessionAffinity.resetMemory(); manualPreference = undefined; + manualPreferenceIsBaseline = true; manualSelectionGeneration++; manualResetPending = false; quorumCache = null; @@ -546,7 +572,21 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting } function isCooled(accountId: string, now: number): boolean { - return getAnthropicAccountHealthSnapshot(accountId, now) !== null; + return getAnthropicAccountHealthSnapshot(accountId, now) !== null || weeklySpentUntil(accountId, now) !== undefined; + } + + /** + * AgentRadar gateways only: a cached seven-day reading at 100% with its reset still ahead is a hard + * hold, so a fresh process (empty cooldown map, persisted quota cache) never seats a session on a + * spent account and an all-spent pool answers a usage limit instead of sending. Ordinary installs + * keep upstream's soft headroom preference. A reading without a reset proves nothing and fails open. + */ + function weeklySpentUntil(accountId: string, now: number): number | undefined { + if (process.env.OPENCODEX_AGENTRADAR !== "1") return undefined; + const quota = getCachedProviderAccountQuota(PROVIDER, accountId); + const resetAt = quota?.weeklyResetAt; + return typeof quota?.weeklyPercent === "number" && quota.weeklyPercent >= 100 + && typeof resetAt === "number" && resetAt > now ? resetAt : undefined; } function fiveHourKnown(accountId: string): boolean { @@ -743,8 +783,10 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting latest.quota = quota; }; const entry = getAnthropicAccountHealthSnapshot(accountId, now) ? upstreamHealth.get(accountId) : undefined; - consider(entry?.cooldownUntil, entry?.refusalStatus === 429, entry?.quotaResetAt); + consider(entry?.cooldownUntil, entry?.refusalStatus === 429 && !entry.headerless, entry?.quotaResetAt); consider(anthropicRatePauseUntil(accountId, now), false); + const weeklyReset = weeklySpentUntil(accountId, now); + consider(weeklyReset, true, weeklyReset); const family = anthropicFamilyHoldState(accountId, model); consider(anthropicFamilyRetryAt(accountId, model, now), family?.revalidating === false, family?.resetAt); if (latest.until === 0) return null; @@ -819,6 +861,7 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting // Every window EXCEPT the legacy five-hour default is an explicit opt-in, so // known-before-unknown applies to all of them and to none of the default path. knownFirst: window !== "five-hour", + spentLast: window === "weekly", })); let best = scored[0]!; for (let i = 1; i < scored.length; i++) { @@ -878,7 +921,7 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting model?: string, excludedAccountIds?: ReadonlySet, ): string | null { - if (!admitted(config)) return null; + if (!admitted(config) || agentRadarPin(config)) return null; const strategy = isAnthropicAccountPoolEnabled(config) ? anthropicPoolStrategy(config) : "quota"; const eligible = routeCandidates(getEligibleAnthropicAccounts(now, model), decision).filter(id => id !== excludeId && !excludedAccountIds?.has(id)); if (strategy === "round-robin") { @@ -964,6 +1007,24 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting return null; } + /** AgentRadar gateways only: a keyed round-robin session is placed by rendezvous hash, not by process state. */ + function agentRadarHashSession(sessionKey: string | null | undefined, config: OcxConfig): boolean { + return process.env.OPENCODEX_AGENTRADAR === "1" && !!normalizeAffinityComponent(sessionKey) + && anthropicPoolStrategy(config) === "round-robin"; + } + + /** Highest sha256(key \0 account) among candidates with headroom, else among all candidates. */ + function agentRadarHashSeat(sessionKey: string | null | undefined, candidates: string[], model?: string): string | null { + const headroom = model ? candidates.filter(id => !anthropicModelExhausted(id, model)) : candidates; + let winner: string | null = null; + let score = ""; + for (const id of headroom.length ? headroom : candidates) { + const candidate = new Bun.CryptoHasher("sha256").update(`${normalizeAffinityComponent(sessionKey)}\0${id}`).digest("hex"); + if (candidate > score) { winner = id; score = candidate; } + } + return winner; + } + /** * Resolve which Anthropic OAuth account should serve this session. * When the pool is disabled, retain the active account unless the operator paused it. @@ -1010,9 +1071,11 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting return { accountId: set.activeAccountId, reason: "pool-disabled" }; } + const deterministicSession = agentRadarHashSession(sessionKey, config); // A manual choice is a one-dispatch preference, not a lower-priority quota hint. // Consume it only after admission commits, so a failed token lookup cannot spend it. - if (manualPreference && !sessionAffinity.get(affinityKey(sessionKey), set, now)) { + if (manualPreference && !(deterministicSession && manualPreferenceIsBaseline) + && !sessionAffinity.get(affinityKey(sessionKey), set, now)) { if (manualPreference.accountId !== set.activeAccountId || manualPreference.revision !== set.selectionRevision) { manualPreference = null; } else { @@ -1026,6 +1089,8 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting } const key = affinityKey(sessionKey); + const seat = deterministicSession ? agentRadarHashSeat(sessionKey, eligible, model) : null; + if (seat) return { accountId: seat, reason: sessionAffinity.get(key, set, now)?.accountId === seat ? "affinity" : "hash", routePosition: decision?.position }; if (key) { const affined = sessionAffinity.get(key, set, now); if (affined && now - affined.lastUsedAt <= AFFINITY_IDLE_TTL_MS) { @@ -1169,7 +1234,7 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting function rotateAnthropicAccountOnRefusal( config: OcxConfig, failedAccountId: string, - status: 429 | 403, + status: AnthropicAccountRefusalStatus, retryAfterHeader: string | null | undefined, sessionKey?: string | null, now = Date.now(), @@ -1178,13 +1243,17 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting model?: string, excludedAccountIds?: ReadonlySet, ): string | null { - if (!recordAnthropicAccountRefusal(config, failedAccountId, status, retryAfterHeader, now, rateLimitHeaders)) return null; + if (!recordAnthropicAccountRefusal(config, failedAccountId, status, retryAfterHeader, now, rateLimitHeaders) || agentRadarPin(config)) return null; // The pool's strategy is a PROACTIVE policy. When the pool is disabled, reactive // presence-only recovery must not silently reactivate round-robin/fill-first merely // because those dormant values remain in config. The quota picker is the neutral // recovery policy already used by the default strategy. - const next = isAnthropicAccountPoolEnabled(config) + // A hashed session fails over to its hash runner-up, so warm and fresh processes agree on it. + const next = isAnthropicAccountPoolEnabled(config) && agentRadarHashSession(sessionKey, config) + ? agentRadarHashSeat(sessionKey, routeCandidates(getEligibleAnthropicAccounts(now, model), decision) + .filter(id => id !== failedAccountId && !excludedAccountIds?.has(id)), model) + : isAnthropicAccountPoolEnabled(config) ? pickAlternateAnthropicAccount(config, failedAccountId, now, decision, model, excludedAccountIds) : pickLowestUsage(config, failedAccountId, now, null, model, excludedAccountIds); if (!next) { @@ -1194,7 +1263,7 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting console.warn(status === 403 ? `[${instance}-pool] ${decision ? `route:#${decision.position} ` : ""}account unavailable (403); failing over` - : `[${instance}-pool] ${decision ? `route:#${decision.position} ` : ""}429 on ${formatAnthropicAccountOrdinal(failedAccountId)}; failing over to ${formatAnthropicAccountOrdinal(next)}`); + : `[${instance}-pool] ${decision ? `route:#${decision.position} ` : ""}${status} on ${formatAnthropicAccountOrdinal(failedAccountId)}; failing over to ${formatAnthropicAccountOrdinal(next)}`); return next; } @@ -1214,7 +1283,7 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting function recordAnthropicAccountRefusal( config: OcxConfig, failedAccountId: string, - status: 429 | 403, + status: AnthropicAccountRefusalStatus, retryAfterHeader: string | null | undefined, now = Date.now(), rateLimitHeaders?: AnthropicRateLimitHeaders | null, @@ -1245,9 +1314,10 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting // without that fallback such a refusal cools for the 60s default and the exhausted // account is back in the rotation a minute later. const parsedRetry = parseRetryAfterMs(retryAfterHeader, now); - const resetDerived = status === 429 && parsedRetry === undefined ? parseRateLimitReset(rateLimitHeaders, now) : undefined; - const quotaResetAt = status === 429 ? rejectedWindowsResetAt(rateLimitHeaders, now) : undefined; - const cooldownMs = parsedRetry ?? resetDerived?.delayMs ?? (status === 403 ? ACCOUNT_REFUSAL_COOLDOWN_MS : DEFAULT_COOLDOWN_MS); + const resetDerived = status !== 403 && parsedRetry === undefined ? parseRateLimitReset(rateLimitHeaders, now) : undefined; + const quotaResetAt = status !== 403 ? rejectedWindowsResetAt(rateLimitHeaders, now) : undefined; + // A spent-extra-usage 400 names no reset of its own: keep the account out as long as an entitlement refusal. + const cooldownMs = parsedRetry ?? resetDerived?.delayMs ?? (status === 429 ? DEFAULT_COOLDOWN_MS : ACCOUNT_REFUSAL_COOLDOWN_MS); const cooldownGeneration = noteAnthropicCooldownMutation(failedAccountId); upstreamHealth.set(failedAccountId, { cooldownUntil: now + cooldownMs, @@ -1257,11 +1327,22 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting ...(resetDerived ? { rejectedQuotaWindows: resetDerived.rejectedQuotaWindows } : {}), cooldownGeneration, recordedAt: now, - refusalStatus: status, - refusedQuotaWindows: status === 429 ? rejectedQuotaWindowsIn(rateLimitHeaders) : [], + refusalStatus: status === 403 ? 403 : 429, + refusedQuotaWindows: status !== 403 ? rejectedQuotaWindowsIn(rateLimitHeaders) : [], ...(quotaResetAt !== undefined ? { quotaResetAt } : {}), + ...(status === 429 && !rateLimitHeaders?.get("anthropic-ratelimit-unified-status") + && rejectedQuotaWindowsIn(rateLimitHeaders).length === 0 ? { headerless: true } : {}), }); sweepExpiredOnWrite(now); + // The cooldown is per process; the quota cache (and its disk snapshot) is what a fresh + // AgentRadar process reads. A rejected 429 may omit the utilization header, so state it. + const weeklyReset = rateLimitHeaders?.get("anthropic-ratelimit-unified-7d-reset"); + if (process.env.OPENCODEX_AGENTRADAR === "1" && status !== 403 && weeklyReset + && rateLimitHeaders?.get("anthropic-ratelimit-unified-7d-status") === "rejected") { + recordAnthropicAccountQuotaFromHeadersForInstance(instance, failedAccountId, new Headers({ + "anthropic-ratelimit-unified-7d-utilization": "1", "anthropic-ratelimit-unified-7d-reset": weeklyReset, + }), captureConfigGeneration()); + } clearAnthropicSessionAffinityForAccount(failedAccountId); notePoolRotationFailure(poolKey, failedAccountId); // The refused account changed the eligible roster even when no retry send remains. @@ -1300,7 +1381,7 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting const current = captureOAuthAccountSelection(PROVIDER); if (current?.accountId !== committed.accountId || current.revision !== committed.revision) return false; if (isAnthropicAccountPoolEnabled(options.config)) { - if (anthropicPoolStrategy(options.config) === "round-robin" && options.reason !== "affinity") { + if (anthropicPoolStrategy(options.config) === "round-robin" && options.reason !== "affinity" && options.reason !== "hash") { const limit = stickyLimitForPool(options.config); const picked = pickRoundRobinAccount(poolKey, routeCandidates(getEligibleAnthropicAccounts(Date.now(), options.model), options.routeDecision ?? null), limit); if (picked !== accountId) seedPoolRotationAccount(poolKey, accountId); @@ -1342,6 +1423,7 @@ function createAnthropicRouting(instance: AnthropicInstanceId): AnthropicRouting function resetAnthropicRoutingForManualSelection(accountId: string): void { requestAnthropicSessionAffinityReset(); manualPreference = captureOAuthAccountSelection(PROVIDER); + manualPreferenceIsBaseline = false; seedPoolRotationAccount(poolKey, accountId); // A manual account selection is an operator statement about the roster; do not answer the // next activation question from a count read before it. @@ -1489,6 +1571,10 @@ export function anthropicRoutingFor(instance: AnthropicInstanceId): AnthropicRou if (!routing) { routing = createAnthropicRouting(instance); instances.set(instance, routing); } return routing; } +// A 429/402 combo cooldown on the pool target outlives the pool state it described: lift it once another +// account can serve (a failover quorum with an eligible account); a lone account keeps the target cooled. +setComboPoolEligibilityProbe((provider, now) => (provider === "anthropic" || provider === "anthropic2") + && anthropicRoutingFor(provider).hasAnthropicFailoverQuorum(now) && anthropicRoutingFor(provider).getEligibleAnthropicAccounts(now).length > 0); export function sweepAllAnthropicRoutingHealth(now = Date.now()): number { let removed = 0; for (const routing of instances.values()) removed += routing.sweepExpiredAnthropicRoutingHealth(now); @@ -1656,7 +1742,7 @@ export function rotateAnthropicAccountOn429( export function rotateAnthropicAccountOnRefusal( config: OcxConfig, failedAccountId: string, - status: 429 | 403, + status: AnthropicAccountRefusalStatus, retryAfterHeader: string | null | undefined, sessionKey?: string | null, now = Date.now(), @@ -1681,7 +1767,7 @@ export function recordAnthropicAccount429( export function recordAnthropicAccountRefusal( config: OcxConfig, failedAccountId: string, - status: 429 | 403, + status: AnthropicAccountRefusalStatus, retryAfterHeader: string | null | undefined, now = Date.now(), rateLimitHeaders?: AnthropicRateLimitHeaders | null, diff --git a/src/oauth/generic-account-failover.ts b/src/oauth/generic-account-failover.ts index fd768d91d2a..e5a2e7dc608 100644 --- a/src/oauth/generic-account-failover.ts +++ b/src/oauth/generic-account-failover.ts @@ -1,3 +1,4 @@ +import { agentRadarPin } from "../lib/agentradar-account-pin"; import { clientFacingModelText, type ClientFacingModelLabel } from "../lib/error-destination"; import { isBuiltinAnthropicInstanceRow } from "../providers/anthropic-instance"; /** @@ -60,6 +61,14 @@ export const GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST = 3; * into a pool-wide 429 storm. */ export const GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST = 6; +/** + * AgentRadar gateways serve pooled subscriptions with no combo behind them, so a request walks up to + * sixteen funded accounts (the hosted .27 contract) before refusing. Still funded per account and + * intersected with every explicit send budget; ordinary installs keep six. + */ +export function oauthMaxAccountsPerRequest(): number { + return process.env.OPENCODEX_AGENTRADAR === "1" ? 16 : GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST; +} const DEFAULT_COOLDOWN_MS = 60_000; @@ -726,6 +735,7 @@ export function rotateGenericOAuthAccountOnRefusal( cooldownSource: parsed ? "retry-after" : "default", }); sweepExpiredOnWrite(now); + if (agentRadarPin(config)) return null; const eligible = eligibleFailoverAccounts(providerName, now, family).filter(id => id !== failedAccountId); if (eligible.length === 0) return null; diff --git a/src/oauth/google-antigravity.ts b/src/oauth/google-antigravity.ts index 3dcc84200fa..085b0386870 100644 --- a/src/oauth/google-antigravity.ts +++ b/src/oauth/google-antigravity.ts @@ -40,6 +40,9 @@ const REFRESH_SKEW_MS = 5 * 60 * 1000; const REQUEST_TIMEOUT_MS = 30_000; const ONBOARD_ATTEMPTS = 5; const ONBOARD_POLL_MS = 2_000; +// A person without a local browser copies the redirect URL from the address bar and pastes it back by +// hand, which takes far longer than the callback base class's default wait. +const ANTIGRAVITY_LOGIN_TIMEOUT_MS = 20 * 60 * 1000; function requestSignal(signal: AbortSignal | undefined): AbortSignal { const timeout = AbortSignal.timeout(REQUEST_TIMEOUT_MS); @@ -212,7 +215,7 @@ function credentialsFromPayload(payload: GoogleTokenPayload, refreshFallback = " }; } -class AntigravityOAuthFlow extends OAuthCallbackFlow { +export class AntigravityOAuthFlow extends OAuthCallbackFlow { #verifier = ""; #forceAccountSelect: boolean; @@ -223,6 +226,7 @@ class AntigravityOAuthFlow extends OAuthCallbackFlow { callbackHostname: "127.0.0.1", callbackBindHostname: "127.0.0.1", redirectUri: `http://127.0.0.1:${CALLBACK_PORT}${CALLBACK_PATH}`, + timeoutMs: ANTIGRAVITY_LOGIN_TIMEOUT_MS, } satisfies OAuthCallbackFlowOptions); this.#forceAccountSelect = opts?.forceAccountSelect === true; } diff --git a/src/providers/anthropic-billing-refusal.ts b/src/providers/anthropic-billing-refusal.ts index f25e5d62b77..8e825ce947e 100644 --- a/src/providers/anthropic-billing-refusal.ts +++ b/src/providers/anthropic-billing-refusal.ts @@ -43,7 +43,12 @@ export function isAnthropicBillingFailure(status: number, message: string, code? /** Inspect a clone, never the terminal body; the shared reader owns interruption/cleanup. */ export async function readAnthropicBillingRefusal(response: Response, signal?: AbortSignal): Promise { - if (signal?.aborted || (response.status !== 400 && response.status !== 402)) return false; + return (response.status === 400 || response.status === 402) + && readAnthropicErrorEnvelope(response, signal, payload => isAnthropicBillingRefusal(response.status, payload)); +} + +async function readAnthropicErrorEnvelope(response: Response, signal: AbortSignal | undefined, accepts: (payload: unknown) => boolean): Promise { + if (signal?.aborted) return false; const contentType = response.headers.get("content-type")?.split(";", 1)[0]?.trim().toLowerCase(); if (!contentType || !/^application\/(?:json|[a-z0-9.+-]+\+json)$/.test(contentType)) return false; try { @@ -54,7 +59,7 @@ export async function readAnthropicBillingRefusal(response: Response, signal?: A totalTimeoutMs: BILLING_REFUSAL_DEADLINE_MS, }); if (signal?.aborted || !body.displaySafe || body.truncated || body.timedOut || body.oversized) return false; - return isAnthropicBillingRefusal(response.status, JSON.parse(body.text)); + return accepts(JSON.parse(body.text)); } catch { // Untrusted malformed, unreadable and invalid UTF-8 bodies confer no authority. return false; @@ -69,11 +74,25 @@ export function isAnthropicExtraUsageFailure(status: number, message: string, co if (status !== 400 || Buffer.byteLength(message, "utf8") > MAX_BILLING_REFUSAL_BYTES) return false; if (code != null && code !== "invalid_request_error") return false; const text = withoutProviderPrefix(status, message); - try { - const payload: unknown = JSON.parse(text); - return record(payload) && payload.type === "error" && record(payload.error) - && payload.error.type === "invalid_request_error" && payload.error.code == null - && typeof payload.error.message === "string" && EXTRA_USAGE_REFUSAL.test(payload.error.message.trim()); - } catch { /* Exact prose needs a structured type. */ } + try { return isAnthropicExtraUsageRefusal(JSON.parse(text)); } catch { /* Exact prose needs a structured type. */ } return code === "invalid_request_error" && EXTRA_USAGE_REFUSAL.test(text); } + +function isAnthropicExtraUsageRefusal(payload: unknown, wording = EXTRA_USAGE_REFUSAL): boolean { + return record(payload) && payload.type === "error" && record(payload.error) + && payload.error.type === "invalid_request_error" && payload.error.code == null + && typeof payload.error.message === "string" && wording.test(payload.error.message.trim()); +} + +// Pool rotation keeps the hosted fork's start-anchored match: Anthropic has changed the suffix +// before, and an echoed client value can only appear after the message's own start. +export const EXTRA_USAGE_ROTATION = /^(?:You['\u2019]re out of extra usage|Third-party apps now draw from your extra usage)/i; +// Anthropic's spend limit (https://platform.claude.com/docs/en/api/rate-limits#setting-your-own-spend-limit), same anchoring. +export const ANTHROPIC_SPEND_LIMIT = /^you have reached your specified (?:workspace )?api usage limits/i; + +/** Account-scoped spent extra-usage or spend-limit evidence for pool rotation; same bounded clone read as billing. */ +export async function readAnthropicExtraUsageRefusal(response: Response, signal?: AbortSignal): Promise { + return response.status === 400 + && readAnthropicErrorEnvelope(response, signal, payload => isAnthropicExtraUsageRefusal(payload, EXTRA_USAGE_ROTATION) + || isAnthropicExtraUsageRefusal(payload, ANTHROPIC_SPEND_LIMIT)); +} diff --git a/src/providers/key-failover.ts b/src/providers/key-failover.ts index da6bbf877f4..947b29ef8f2 100644 --- a/src/providers/key-failover.ts +++ b/src/providers/key-failover.ts @@ -38,6 +38,9 @@ import { isDisplayOnlyQuota } from "./quota-types"; const DEFAULT_COOLDOWN_MS = 60_000; const MAX_COOLDOWN_MS = 10 * 60_000; // cap at 10 min for api-key rotation +// AgentRadar gateways rotate a pool of hosted keys and cap an upstream Retry-After at 60s, so a long +// header cannot park a key for ten minutes; a 401 hold keeps MAX_COOLDOWN_MS. +const retryAfterCapMs = (): number => process.env.OPENCODEX_AGENTRADAR === "1" ? 60_000 : MAX_COOLDOWN_MS; /** * Anthropic credit-balance refusal (#6958). An empty prepaid balance is account state, not a * rate signal: it clears only when someone buys credits or auto-reload fires, so the @@ -358,7 +361,7 @@ const DEFAULT_RESET_REPLAY = { /** * Parse an upstream `Retry-After` header: numeric seconds (including `0`) or an HTTP-date. - * Returns a bounded delay in ms (1..MAX_COOLDOWN_MS), or undefined when the value is + * Returns a bounded delay in ms (1..retryAfterCapMs()), or undefined when the value is * malformed. An HTTP-date already in the past yields an immediate (1 ms) retry. */ function parseRetryAfterMs(value: string | null | undefined, now = Date.now()): number | undefined { @@ -367,7 +370,7 @@ function parseRetryAfterMs(value: string | null | undefined, now = Date.now()): if (/^\d+(?:\.\d+)?$/.test(text)) { const seconds = Number(text); if (Number.isFinite(seconds) && seconds >= 0) { - return Math.min(Math.max(Math.ceil(seconds * 1000), 1), MAX_COOLDOWN_MS); + return Math.min(Math.max(Math.ceil(seconds * 1000), 1), retryAfterCapMs()); } } const timestamp = Date.parse(text); @@ -375,7 +378,7 @@ function parseRetryAfterMs(value: string | null | undefined, now = Date.now()): const delay = timestamp - now; // A valid HTTP-date whose retry time has already passed is an immediate retry, exactly like // numeric `Retry-After: 0` — never a malformed-header fallback to the fixed interval. - return Math.min(Math.max(delay, 1), MAX_COOLDOWN_MS); + return Math.min(Math.max(delay, 1), retryAfterCapMs()); } /** diff --git a/src/providers/label.ts b/src/providers/label.ts index 958859e5811..3f5fd11aa44 100644 --- a/src/providers/label.ts +++ b/src/providers/label.ts @@ -1,4 +1,4 @@ -import { ACCOUNT_LOG_LABEL_RE, CODEX_ACCOUNT_LOG_LABEL_RE, KEY_ACCOUNT_LOG_LABEL_RE, apiKeyAccountLogLabel, oauthAccountLogLabel } from "../codex/account-label"; +import { ACCOUNT_LOG_LABEL_RE, CODEX_ACCOUNT_LOG_LABEL_RE, KEY_ACCOUNT_LOG_LABEL_RE, apiKeyAccountLogLabel, fallbackCodexAccountLogLabel, oauthAccountLogLabel } from "../codex/account-label"; import type { OcxProviderConfig } from "../types"; import { captureCacheAffinityIdentity } from "./cache-affinity-identity"; import { isAnthropicInstanceId } from "./anthropic-instance-id"; @@ -99,7 +99,8 @@ export function anthropicAccountTagField(provider: string | undefined): { accoun * Two providers are skipped because they already have attribution: * - `openai` produces its own `p`-labels through `codexAuthContextLogLabel`. * - `anthropic` folds the account into the provider label (`formatAnthropicProviderForLog`); - * `anthropicAccountTag` exposes that same label as the separate `accountTag` field. + * `anthropicAccountTag` exposes that same label as the separate `accountTag` field. On an + * AgentRadar gateway (`OPENCODEX_AGENTRADAR=1`) it is also stamped, as that same `p`. * * It lives here rather than in `account-label.ts` because it needs `baseProviderLabel`, and this * module already imports from that one -- the reverse direction would be an import cycle. This @@ -115,6 +116,13 @@ export function stampOAuthAccountLabel( if (!accountId) return; if (provider.authMode !== "oauth") return; const base = baseProviderLabel(providerName); - if (base === "openai" || (base === "anthropic" || base === "anthropic2")) return; + if (base === "openai") return; + if (base === "anthropic" || base === "anthropic2") { + // AgentRadar gateways need the serving account on every row, including the plain active-account + // path that never rewrites the provider: `p`, the same digest `formatAnthropicProviderForLog` + // appends, so one account carries one identity in both fields. + if (process.env.OPENCODEX_AGENTRADAR === "1") logCtx.accountLogLabel = fallbackCodexAccountLogLabel(accountId); + return; + } logCtx.accountLogLabel = oauthAccountLogLabel(accountId, base); } diff --git a/src/providers/quota-explain.ts b/src/providers/quota-explain.ts new file mode 100644 index 00000000000..ad378b76f93 --- /dev/null +++ b/src/providers/quota-explain.ts @@ -0,0 +1,122 @@ +/** + * Turn a cached quota reading into a sentence a human can act on. + * + * Upstream answers a spent model-family window with the same generic `rate_limit_error` it uses + * for ordinary request-rate throttling -- "this request would exceed your account's rate limit, + * please try again later". That advice is wrong for an exhausted window: trying again later does + * not help until the window rolls, and the operator's real options (a different model, a + * different account) are invisible in it. + * + * This gateway already probes and caches the per-window percentages, so it can name the window + * that is actually spent. Nothing here infers or predicts: it reports a cached reading, says so, + * and stays silent when the cache has nothing to add. + */ +import { getCachedProviderAccountQuota } from "./quota/account-cache"; +import type { ProviderQuota, ProviderQuotaWindow } from "./quota-types"; + +/** A window counts as spent at 100% with a reset that has not yet passed. */ +function windowSpent(percent: number | undefined, resetAt: number | undefined, now: number): boolean { + if (typeof percent !== "number" || !Number.isFinite(percent) || percent < 100) return false; + return typeof resetAt !== "number" || !Number.isFinite(resetAt) || resetAt > now; +} + +function resetPhrase(resetAt: number | undefined, now: number): string { + if (typeof resetAt !== "number" || !Number.isFinite(resetAt) || resetAt <= now) return ""; + const minutes = Math.round((resetAt - now) / 60_000); + if (minutes < 60) return `, resets in ${Math.max(1, minutes)}m`; + const hours = minutes / 60; + // A trailing ".0" reads as spurious precision on a cached figure, so whole hours stay whole. + if (hours < 48) { + const rounded = hours < 10 ? Math.round(hours * 10) / 10 : Math.round(hours); + return `, resets in ${rounded}h`; + } + return `, resets in ${Math.round(hours / 24)}d`; +} + +/** + * Only a window the PRODUCER proved model-scoped is reported against a model name; everything + * else is provider-wide. Same rule the routing gate uses, and for the same reason: a label is + * upstream text and cannot be trusted to mean what it looks like. + */ +function windowLabel(window: ProviderQuotaWindow): string { + return window.scope === "model" ? `${window.label} window` : `${window.label}`; +} + +/** + * A short clause naming the spent window, or undefined when the cache cannot explain the refusal. + * + * Deliberately returns undefined rather than guessing: a 429 with no exhausted window in cache is + * ordinary rate limiting, and saying otherwise would send the operator after a quota problem that + * does not exist. + */ +export function explainQuotaRefusal( + quota: ProviderQuota | null, + model: string | undefined, + now = Date.now(), +): string | undefined { + if (!quota) return undefined; + + const spent: string[] = []; + // A model-scoped window is only evidence about a model whose id carries that family. + for (const window of quota.customWindows ?? []) { + // A `modelId` window is display evidence that exhaustion ignores; it cannot explain a refusal. + if (window.modelId !== undefined || !windowSpent(window.percent, window.resetAt, now)) continue; + if (window.scope === "model" && model !== undefined + && !model.toLowerCase().includes(window.label.trim().toLowerCase())) continue; + spent.push(`${windowLabel(window)} at 100%${resetPhrase(window.resetAt, now)}`); + } + if (windowSpent(quota.fiveHourPercent, quota.fiveHourResetAt, now)) { + spent.push(`5-hour window at 100%${resetPhrase(quota.fiveHourResetAt, now)}`); + } + if (windowSpent(quota.weeklyPercent, quota.weeklyResetAt, now)) { + spent.push(`weekly window at 100%${resetPhrase(quota.weeklyResetAt, now)}`); + } + if (spent.length === 0) return undefined; + return `${spent.join("; ")} (cached quota reading)`; +} + +/** + * The same explanation for an OAuth pool account, which is where a per-model window actually + * lives: the routing cache is key-auth only, so an Anthropic pool refusal has to be read from + * the account's own reading or it reads as nothing at all. + */ +export function explainAccountQuotaRefusal( + providerName: string, + accountId: string | undefined, + model: string | undefined, + now = Date.now(), +): string | undefined { + if (!accountId) return undefined; + const note = explainQuotaRefusal(getCachedProviderAccountQuota(providerName, accountId), model, now); + return note ? `${providerName}: ${note}` : undefined; +} + +/** + * Whether EVERY account in the pool has this model's window spent, and the soonest one rolls. + * + * This is the fact an operator actually needs on a pool refusal. "One account is spent" is not + * actionable -- the pool should have rotated. "All of them are spent" says the model is + * unavailable until a stated time, and that the answer is a different model or a new account + * rather than waiting a minute and retrying. + * + * Silent unless the reading is complete: an account the cache cannot speak for might have + * headroom, and claiming a pool-wide outage on partial evidence is worse than saying nothing. + */ +export function explainPoolQuotaRefusal( + providerName: string, + accountIds: readonly string[], + model: string | undefined, + now = Date.now(), +): string | undefined { + if (accountIds.length === 0) return undefined; + const notes: string[] = []; + for (const accountId of accountIds) { + const quota = getCachedProviderAccountQuota(providerName, accountId); + if (!quota) return undefined; + const note = explainQuotaRefusal(quota, model, now); + if (!note) return undefined; + notes.push(note); + } + const plural = accountIds.length === 1 ? "account" : `all ${accountIds.length} accounts`; + return `${providerName}: ${model ?? "this model"} is exhausted on ${plural} — ${notes[0]}`; +} diff --git a/src/providers/quota/vendor-probes-key.ts b/src/providers/quota/vendor-probes-key.ts index 8c061b4519f..35f40b77a90 100644 --- a/src/providers/quota/vendor-probes-key.ts +++ b/src/providers/quota/vendor-probes-key.ts @@ -1298,8 +1298,9 @@ function parseKimiQuotaPayload(value: unknown): ProviderQuota | null { let weekly = parseKimiQuotaRow(body.usage); const total = parseKimiQuotaRow(body.totalQuota); const nested = asRecord(root.data); + const containers = [root, nested, asRecord(root.quota), asRecord(nested?.quota)]; let monthly: { percent: number; resetAt?: number } | null = null; - for (const container of [root, nested, asRecord(root.quota), asRecord(nested?.quota)]) { + for (const container of containers) { const usages = asRecord(container?.usages); if (!usages) continue; monthly = parseKimiQuotaRow(usages.limit_month_total ?? usages.monthTotal) @@ -1322,6 +1323,20 @@ function parseKimiQuotaPayload(value: unknown): ProviderQuota | null { if (fiveHour && weekly) break; } } + // Kimi Code's own client reads usages.limit_5h / limit_7d (MoonshotAI/kimi-code managed-usage.ts). + // Beside them the legacy blocks can be unused placeholders, so a usages window wins; legacy + // counters win only over a zero ratio, which public kimi.ai key reports show contradicting them. + const usagesWindow = (key: string) => { + for (const container of containers) { + const row = parseKimiQuotaRow(asRecord(container?.usages)?.[key]); + if (row) return row; + } + return null; + }; + const preferUsages = (usages: typeof fiveHour, legacy: typeof fiveHour) => + legacy && usages?.percent === 0 && legacy.percent > 0 ? legacy : usages ?? legacy; + fiveHour = preferUsages(usagesWindow("limit_5h"), fiveHour); + weekly = preferUsages(usagesWindow("limit_7d"), weekly); const quota: ProviderQuota = { ...(fiveHour ? { fiveHourPercent: fiveHour.percent, diff --git a/src/providers/registry/entries-core.ts b/src/providers/registry/entries-core.ts index ba35d065897..6acc5ba260c 100644 --- a/src/providers/registry/entries-core.ts +++ b/src/providers/registry/entries-core.ts @@ -176,7 +176,7 @@ function anthropicOAuthEntry(id: AnthropicInstanceId, label: string): ProviderRe }); } -export const PROVIDER_REGISTRY_CORE: readonly ProviderRegistryEntry[] = [ +const CORE_ENTRIES: readonly ProviderRegistryEntry[] = [ { id: "openai", label: "OpenAI (Codex login)", @@ -1514,3 +1514,22 @@ export const PROVIDER_REGISTRY_CORE: readonly ProviderRegistryEntry[] = [ note: "Serverless Inference subscription API. Live discovery exposes only kimi-k2-instruct because Vultr documents it as the sole tool-calling model.", }, ]; + +// AgentRadar hosted gateways store an xAI API key under "xai-apikey" and regenerate their key-provider +// list from this registry, so the row must exist. It is the xai row's key-auth lane under its own +// identity, derived so capability fixes to xai reach it; Grok OAuth stays on "xai". +function xaiApiKeyEntry(xai: ProviderRegistryEntry): ProviderRegistryEntry { + const { allowKeyAuthOverride: _override, keyAuthServiceTier, oauthId: _oauthId, featured: _featured, ...shared } = xai; + return { + ...shared, + ...keyAuthServiceTier, + id: "xai-apikey", + label: "xAI Grok (API key)", + authKind: "key", + dashboardUrl: "https://console.x.ai", + note: "Direct xAI API billing — no Grok subscription", + }; +} + +export const PROVIDER_REGISTRY_CORE: readonly ProviderRegistryEntry[] = CORE_ENTRIES.flatMap(entry => + entry.id === "xai" ? [entry, xaiApiKeyEntry(entry)] : [entry]); diff --git a/src/responses/state.ts b/src/responses/state.ts index 7df4605d30c..c57f4536a38 100644 --- a/src/responses/state.ts +++ b/src/responses/state.ts @@ -22,6 +22,7 @@ import { collectReferencedSpillFileNames, snapshotReferencedSpillFileNames } fro import { selectSnapshotEntries, snapshotPayload } from "./state/snapshot-select"; import { expandStoredResponseInput } from "./state/replay-input"; import { normalizedClientThreadId, replayScopeMatches, validReplayOwnerKey } from "./state/replay-owner"; +import { envBoundAtMost } from "./state/env-bounds"; import { hasPendingClientToolCall } from "./state/unforced-store-false"; import { computeResponseStateMetrics, type ResponseStateMetrics } from "./state/metrics"; export type { ResponseStateMetrics } from "./state/metrics"; @@ -224,9 +225,11 @@ function deferSupersededSpill(ref: ResponseSpillRef | undefined): void { function byteCap(): number { - return byteCapOverride ?? MAX_STORED_RESPONSE_BYTES; + return byteCapOverride ?? envBoundAtMost("OPENCODEX_RESPONSE_STATE_MAX_BYTES", MAX_STORED_RESPONSE_BYTES); } +const responseTtlMs = (): number => envBoundAtMost("OPENCODEX_RESPONSE_STATE_TTL_MS", RESPONSE_TTL_MS); + /** Test-only: lower/restore the in-memory byte cap (null restores the default). */ export function setResponseStateByteCapForTests(bytes: number | null): void { byteCapOverride = bytes; @@ -249,7 +252,7 @@ export function getStoredResponseBytesForTests(): number { let spillByteCapOverride: number | null = null; function spillByteCap(): number { - return spillByteCapOverride ?? MAX_SPILLED_RESPONSE_BYTES; + return spillByteCapOverride ?? envBoundAtMost("OPENCODEX_RESPONSE_STATE_MAX_SPILL_BYTES", MAX_SPILLED_RESPONSE_BYTES); } /** @@ -939,7 +942,7 @@ function enforceSpilledResponseBudget(requiredHeadroomBytes?: number): number { function pruneResponses(at = now()): void { for (const [id, state] of states) { - if (at - state.createdAt > RESPONSE_TTL_MS) deleteEntry(id); + if (at - state.createdAt > responseTtlMs()) deleteEntry(id); } while (states.size > MAX_STORED_RESPONSES) { const oldest = states.keys().next().value; @@ -949,11 +952,12 @@ function pruneResponses(at = now()): void { // Unconditional RAM cap. Resident payloads demote durably; stubs/tombstones are // deleted only when even their bounded metadata cannot fit the override. while (storedResponseBytes > byteCap() && states.size > 0) { - const oldestResident = [...states].find(([id, entry]) => entry.kind === "resident" - && !spillQueueHoldsResidentCandidate(id, entry)); - const hasPendingResident = !oldestResident && [...states].some(([id, entry]) => entry.kind === "resident" - && spillQueueHoldsResidentCandidate(id, entry)); - if (hasPendingResident) break; + // One pass over the map, no copies: a lowered cap demotes on nearly every write. + let oldestResident: [string, ResidentResponseState] | undefined, hasPendingResident = false; + for (const [id, entry] of states) if (entry.kind === "resident") { + if (!spillQueueHoldsResidentCandidate(id, entry)) { oldestResident = [id, entry]; break; } else hasPendingResident = true; + } + if (!oldestResident && hasPendingResident) break; const oldestId = oldestResident?.[0] ?? states.keys().next().value as string | undefined; if (!oldestId) break; const entry = states.get(oldestId)!; @@ -988,7 +992,7 @@ function pruneResponses(at = now()): void { export function sweepExpiredResponseStates(at = now()): number { let removed = 0; for (const [id, state] of states) { - if (at - state.createdAt <= RESPONSE_TTL_MS) continue; + if (at - state.createdAt <= responseTtlMs()) continue; deleteEntry(id); removed += 1; } diff --git a/src/responses/state/env-bounds.ts b/src/responses/state/env-bounds.ts new file mode 100644 index 00000000000..3892a454cda --- /dev/null +++ b/src/responses/state/env-bounds.ts @@ -0,0 +1,15 @@ +/** + * Optional environment tightening of a built-in response-store bound: the resident byte cap + * (OPENCODEX_RESPONSE_STATE_MAX_BYTES), retention (OPENCODEX_RESPONSE_STATE_TTL_MS) and the spill + * cap (OPENCODEX_RESPONSE_STATE_MAX_SPILL_BYTES). Values above the default, or below 1 (a fraction + * would floor to 0 and empty the store), are ignored: the defaults are part of the documented + * app-owned memory ceiling, and a small instance needs a lower bound, never a higher one. Read per + * call, like the test overrides, so a changed environment takes effect without a restart. + */ +export function envBoundAtMost(name: string, defaultValue: number): number { + const raw = process.env[name]?.trim(); + if (!raw) return defaultValue; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed < 1) return defaultValue; + return Math.min(Math.floor(parsed), defaultValue); +} diff --git a/src/router.ts b/src/router.ts index b0282e3e337..1bbe7100376 100644 --- a/src/router.ts +++ b/src/router.ts @@ -74,6 +74,15 @@ export class NoEligiblePolicyCandidateError extends Error { } } +interface RouteModelOptions { + /** Restricts the combo pick; a target it rejects is traced ineligible. */ + comboEligible?: (target: ComboPick["target"]) => boolean; + /** Request-level compatibility; defaults to comboEligible. A rejected target is traced `request-incompatible`. */ + comboRequestCompatible?: (target: ComboPick["target"]) => boolean; + /** A random combo picks from this seed (AgentRadar: count_tokens and its generation agree). */ + comboRandomSeed?: string; +} + export interface RouteResult { providerName: string; provider: OcxProviderConfig; @@ -268,6 +277,8 @@ export function comboRouteDecisionTrace( comboId: string, pick: ComboPick, requestedModel: string, + eligible?: (target: ComboPick["target"]) => boolean, + requestCompatible?: (target: ComboPick["target"]) => boolean, ): RouteDecisionTraceV1 { const combo = getCombo(config, comboId); return buildRouteDecisionTrace({ @@ -282,7 +293,7 @@ export function comboRouteDecisionTrace( ? { tieBreak: combo.strategy } : {}), }, - candidates: combo ? comboRouteCandidates(config, pick, combo) : undefined, + candidates: combo ? comboRouteCandidates(config, pick, combo, eligible, requestCompatible) : undefined, }); } @@ -396,6 +407,8 @@ function comboRouteCandidates( config: OcxConfig, pick: NonNullable, combo: NormalizedComboConfig, + eligible?: (target: ComboPick["target"]) => boolean, + requestCompatible: (target: ComboPick["target"]) => boolean = eligible ?? (() => true), ): TraceCandidateInput[] { const now = Date.now(); return combo.targets.map((target, index) => { @@ -404,6 +417,7 @@ function comboRouteCandidates( const configured = provider !== undefined; const enabled = configured && provider.disabled !== true; const inCooldown = isComboTargetInCooldown(pick.comboId, target, now); + const compatible = requestCompatible(target); const isSelected = index === pick.targetIndex; // The pick's `attempted` list includes the winner itself; only non-selected // targets can be "already-attempted" (fallback picks exclude earlier tries). @@ -412,6 +426,7 @@ function comboRouteCandidates( if (!configured) exclusions.push({ code: "unconfigured" }); if (configured && !enabled) exclusions.push({ code: "disabled" }); if (inCooldown) exclusions.push({ code: "cooldown" }); + if (!compatible) exclusions.push({ code: "request-incompatible" }); if (isSelected && inCooldown) exclusions.push({ code: "selected-despite-cooldown" }); if (!isSelected && alreadyAttempted && exclusions.length === 0) { exclusions.push({ code: "already-attempted" }); @@ -420,7 +435,7 @@ function comboRouteCandidates( return { provider: target.provider, model: target.model, - eligible: enabled && !inCooldown && !alreadyAttempted, + eligible: enabled && !inCooldown && !alreadyAttempted && (eligible?.(target) ?? true) && compatible, exclusions, }; }); @@ -434,6 +449,7 @@ function routeModelInternal( allowCompactionNativeFallback = false, preview = false, redirectState: BlockedModelRedirectState = { visited: new Set(), edges: 0 }, + comboPick: RouteModelOptions = {}, ): RouteResult { const slash = modelId.indexOf("/"); // Policy namespace is system-reserved: an explicit `policy/` or a @@ -521,7 +537,7 @@ function routeModelInternal( } if (!bypassCombos && !preservesPhysicalComboProvider(config)) { - const combo = tryPickComboModel(config, modelId, preview); + const combo = tryPickComboModel(config, modelId, preview, comboPick.comboEligible, comboPick.comboRandomSeed); if (combo) { const concrete = `${combo.target.provider}/${combo.target.model}`; // The selected target is already a concrete provider/model reference. Resolve it without @@ -699,7 +715,9 @@ function routeModelInternal( throw new Error(`No provider configured for model: ${modelId}`); } -function routeWithDecisionTrace(config: OcxConfig, modelId: string, route: RouteResult): RouteResult { +function routeWithDecisionTrace( + config: OcxConfig, modelId: string, route: RouteResult, options: RouteModelOptions = {}, +): RouteResult { // Policy routes carry a full evaluation trace already; never rebuild it. if (route.routeDecision) return route; const accountRef = route.codexAccountNamespace; @@ -718,7 +736,7 @@ function routeWithDecisionTrace(config: OcxConfig, modelId: string, route: Route : {}), }, candidates: route.routeKind === "combo" && route.combo && combo - ? comboRouteCandidates(config, route.combo, combo) + ? comboRouteCandidates(config, route.combo, combo, options.comboEligible, options.comboRequestCompatible) : undefined, }); return route; @@ -728,9 +746,10 @@ export function routeModel( config: OcxConfig, modelId: string, policyEvidence?: PolicyRequestEvidence, + options: RouteModelOptions = {}, ): RouteResult { - const route = routeModelInternal(config, modelId, false, policyEvidence); - return routeWithDecisionTrace(config, modelId, route); + const route = routeModelInternal(config, modelId, false, policyEvidence, false, false, undefined, options); + return routeWithDecisionTrace(config, modelId, route, options); } /** Resolve a route for capability inspection without creating combo selection state. */ diff --git a/src/routing/analytics.ts b/src/routing/analytics.ts index 220a65e70f5..d0e2aaa5d29 100644 --- a/src/routing/analytics.ts +++ b/src/routing/analytics.ts @@ -11,19 +11,28 @@ * sample for the full history. */ -import { normalizePersistedUsageRow, type PersistedUsageEntry } from "../usage/log"; -import { computeEntryCost } from "../usage/summary"; +import { + normalizePersistedUsageRow, + physicalUsageAttempts, + type AttemptRecoveryKind, + type PersistedUsageAttempt, + type PersistedUsageEntry, + type UsageStatus, +} from "../usage/log"; +import { normalizeCostTokens } from "../usage/cost"; +import { computeEntryCost, type UsageSurface } from "../usage/summary"; import { withRequestHistoryRead } from "./history/indexer"; export const ANALYTICS_MAX_ROWS = 50_000; /** Default row cap for the management API (full cap remains available via `limit`). */ export const ANALYTICS_API_DEFAULT_ROWS = 5_000; +export const ANALYTICS_MAX_RED_ALERTS = 100; export interface RoutingAnalyticsFilters { provider?: string; model?: string; profileId?: string; - surface?: string; + surface?: UsageSurface; from?: number; to?: number; } @@ -54,6 +63,69 @@ export interface AnalyticsProfileRow { successRate: number | null; } +/** Attempt-reported input telemetry. It is not a provider invoice or raw wire capture. */ +export interface AnalyticsAttemptUsage { + inclusiveInputTokens: number; + rawInputTokens: number; + cacheReadInputTokens: number; + cacheWriteInputTokens: number; + rawInputShare: number | null; + cacheReadShare: number | null; + cacheWriteShare: number | null; +} + +export interface AnalyticsPhysicalUsageCoverage { + totalAttempts: number; + measuredAttempts: number; + reportedAttempts: number; + estimatedAttempts: number; + unreportedAttempts: number; + unsupportedAttempts: number; + ratio: number | null; + supportedRatio: number | null; +} + +export interface AnalyticsPhysicalBreakdownRow { + provider: string; + model: string; + accountRef?: string; + requests: number; + physicalAttempts: number; + physicalSends: number; + repeatedSendAttempts: number; + recoveryAttempts: number; + recoveryEvents: number; + recoveryRate: number | null; + comboFailoverRequests: number; + comboFailoverRate: number | null; + requestRatePerHour: number | null; + attemptUsage: AnalyticsAttemptUsage; + usageCoverage: AnalyticsPhysicalUsageCoverage; +} + +export type AnalyticsRedAlertKind = + | "consecutive-high-raw-input" + | "high-cache-write-after-warmup" + | "low-cache-read-share-after-warmup" + | "falling-cache-read" + | "recovery" + | "repeated-send" + | "combo-failover"; + +export interface AnalyticsRedAlert { + kind: AnalyticsRedAlertKind; + requestId: string; + timestamp: number; + conversationId?: string; + provider: string; + model: string; + accountRef?: string; + attemptOrdinal: number; + value?: number; + previousValue?: number; + recoveryKinds?: AttemptRecoveryKind[]; +} + export interface RoutingAnalyticsResult { generatedAt: number; totalRequests: number; @@ -66,6 +138,22 @@ export interface RoutingAnalyticsResult { fallbackRate: number | null; totalAttempts: number; averageAttemptsPerRequest: number | null; + physicalAttempts: number; + physicalSends: number; + repeatedSendAttempts: number; + recoveryAttempts: number; + recoveryEvents: number; + recoveryRate: number | null; + comboFailoverRequests: number; + comboFailoverRate: number | null; + requestRatePerHour: number | null; + attemptUsage: AnalyticsAttemptUsage; + physicalUsageCoverage: AnalyticsPhysicalUsageCoverage; + physicalBreakdown: AnalyticsPhysicalBreakdownRow[]; + redAlerts: AnalyticsRedAlert[]; + redAlertsPartial: boolean; + sequentialRoutes: number; + warmedRoutes: number; incompleteStreamRate: number | null; cooldownTriggeringFailures: number; durationMs: { @@ -84,6 +172,7 @@ export interface RoutingAnalyticsResult { }; estimatedCostUsdPerSuccessfulRequest: number | null; estimatedCostUsdTotalSuccessful: number | null; + /** Measured physical attempts divided by all physical attempts. */ usageCoverage: number | null; priceCoverage: number | null; breakdown: AnalyticsBreakdownRow[]; @@ -91,6 +180,8 @@ export interface RoutingAnalyticsResult { } interface ScannedRow { + requestId: string; + timestamp: number; provider: string; model: string; apiKeyId?: string | null; @@ -102,7 +193,6 @@ interface ScannedRow { closeReason?: string | null; terminalStatus?: string | null; usageStatus: string; - usageJson?: string | null; attemptCount: number; fallback: number; rowJson: string; @@ -114,6 +204,71 @@ interface Bucket extends AnalyticsBreakdownRow { costRows: number; } +interface UsageAccumulator { + inclusiveInputTokens: number; + rawInputTokens: number; + cacheReadInputTokens: number; + cacheWriteInputTokens: number; +} + +interface CacheDecomposition { + inclusive: number; + raw: number; + read: number; + write: number; +} + +interface CoverageAccumulator { + totalAttempts: number; + reportedAttempts: number; + estimatedAttempts: number; + unreportedAttempts: number; + unsupportedAttempts: number; +} + +interface PhysicalAccumulator { + provider: string; + model: string; + accountRef?: string; + requests: number; + physicalAttempts: number; + physicalSends: number; + repeatedSendAttempts: number; + recoveryAttempts: number; + recoveryEvents: number; + comboFailoverRequests: number; + usage: UsageAccumulator; + coverage: CoverageAccumulator; +} + +interface PhysicalAttempt { + ordinal: number; + provider: string; + model: string; + adapter?: string; + accountRef?: string; + sendCount: number; + recoveryKinds: AttemptRecoveryKind[]; + recoveryCount: number; + usageStatus: UsageStatus; + usageEstimated: boolean; + firstSendAt?: number; + promptCacheTtlMs?: number; + cache?: CacheDecomposition; +} + +interface ExpandedRow { + row: ScannedRow; + attempts: PhysicalAttempt[]; + fallback: boolean; + attemptCount: number; + comboFailover: boolean; + conversationId?: string; + kind: "success" | "failure" | "cancelled"; + rowCostUsd: number | null; + cooldownTriggering: boolean; +} + const COOLDOWN_RECOVERY_KINDS = new Set([ "rate-limit-429", "key-401", @@ -124,6 +279,41 @@ const COOLDOWN_RECOVERY_KINDS = new Set([ "oauth-account-429", "oauth-account-402", ]); +const USAGE_STATUSES = new Set([ + "reported", "unreported", "unsupported", "estimated", +]); +const HOUR_MS = 3_600_000; +const ANTHROPIC_DEFAULT_PROMPT_CACHE_TTL_MS = 5 * 60_000; +const ANTHROPIC_ACCOUNT_PROVIDER_RE = /^anthropic-(p[a-f0-9]{6})$/; + +/** + * One Anthropic account is one physical route, counted once. + * + * An Anthropic attempt can name its account in the account-qualified provider (`anthropic-p`), + * in the frozen `accountTag` (#8383, the same `p`), and on hosted rows in `accountLogLabel` + * (the same digest, also on the plain active account whose provider stays `anthropic`). Keyed on + * the raw provider, one account would split into an `anthropic` route and an `anthropic-p` + * route. So the route takes the account from exactly one source -- the tag or label the caller + * passes, else the provider suffix -- and the provider folds to `anthropic`. + */ +function physicalAccountRoute( + provider: string, + accountRef: string | undefined, +): { provider: string; accountRef?: string } { + const suffix = ANTHROPIC_ACCOUNT_PROVIDER_RE.exec(provider)?.[1]; + if (provider !== "anthropic" && suffix === undefined) return accountRef ? { provider, accountRef } : { provider }; + const account = accountRef ?? suffix; + return { provider: "anthropic", ...(account ? { accountRef: account } : {}) }; +} + +function usesAnthropicCacheAlertPolicy( + attempt: Pick, +): boolean { + return attempt.adapter === "anthropic" + || attempt.provider === "anthropic" + || attempt.provider === "anthropic-native" + || ANTHROPIC_ACCOUNT_PROVIDER_RE.test(attempt.provider); +} function percentile(sorted: number[], p: number): number | undefined { if (sorted.length === 0) return undefined; @@ -147,27 +337,185 @@ function parseEntry(rowJson: string): PersistedUsageEntry | null { } } +/** Account of an attempt: the frozen Anthropic tag, else the persisted account label. */ +function attemptAccount(source: { accountTag?: string; accountLogLabel?: string } | null | undefined): string | undefined { + return source?.accountTag ?? source?.accountLogLabel; +} + +function usageStatus(value: unknown): UsageStatus { + return typeof value === "string" && USAGE_STATUSES.has(value as UsageStatus) + ? value as UsageStatus + : "unreported"; +} + +function cacheDecomposition(value: PersistedUsageEntry["usage"] | undefined): CacheDecomposition | null { + if (!value) return null; + const tokens = normalizeCostTokens(value); + return tokens && { + inclusive: value.inputTokens, + raw: tokens.input, + read: tokens.cacheRead, + write: tokens.cacheWrite, + }; +} + +function physicalAttemptsFor( + row: ScannedRow, + entry: PersistedUsageEntry | null, +): PhysicalAttempt[] { + const attempts = entry?.attempts; + if (entry && attempts !== undefined) { + const durable = physicalUsageAttempts(attempts); + const legacyAccountAttempt = attemptAccount(entry) + ? durable.reduce((final, attempt) => + !final || attempt.ordinal > final.ordinal ? attempt : final, undefined) + : undefined; + const physical = durable + .map((attempt): PhysicalAttempt => { + const cache = cacheDecomposition(attempt.usage); + const route = physicalAccountRoute(attempt.provider, attemptAccount(attempt) + ?? (attempt === legacyAccountAttempt ? attemptAccount(entry) : undefined)); + return { + ordinal: attempt.ordinal, + model: attempt.model, + ...(attempt.adapter ? { adapter: attempt.adapter } : {}), + ...route, + sendCount: attempt.sendCount, + recoveryKinds: attempt.recoveryKinds, + recoveryCount: attempt.recoveryCount ?? attempt.recoveryKinds.length, + usageStatus: attempt.usageStatus, + usageEstimated: attempt.usage?.estimated === true, + ...(attempt.firstSendAt !== undefined ? { firstSendAt: attempt.firstSendAt } : {}), + ...(attempt.promptCacheTtlMs !== undefined + ? { promptCacheTtlMs: attempt.promptCacheTtlMs } + : {}), + ...(cache ? { cache } : {}), + }; + }) + .sort((a, b) => a.ordinal - b.ordinal); + const rootCache = cacheDecomposition(entry.usage); + const finalAttempt = physical.at(-1); + if (!durable.some(attempt => attempt.usage !== undefined) && finalAttempt && rootCache) { + finalAttempt.cache = rootCache; + finalAttempt.usageStatus = usageStatus(entry.usageStatus); + finalAttempt.usageEstimated = entry.usage?.estimated === true; + } + return physical; + } + const cache = cacheDecomposition(entry?.usage); + return [{ + ordinal: 1, + model: row.model, + ...physicalAccountRoute(row.provider, attemptAccount(entry)), + sendCount: 1, + recoveryKinds: [], + recoveryCount: 0, + usageStatus: usageStatus(entry?.usageStatus ?? row.usageStatus), + usageEstimated: entry?.usage?.estimated === true, + ...(cache ? { cache } : {}), + }]; +} + function cooldownTriggering(entry: PersistedUsageEntry | null, status: number): boolean { if (status === 429) return true; - if (!Array.isArray(entry?.attempts)) return false; - return entry.attempts.some((attempt: unknown) => { - if (!attempt || typeof attempt !== "object") return false; - const recoveryKinds = (attempt as { recoveryKinds?: unknown }).recoveryKinds; - return Array.isArray(recoveryKinds) - && recoveryKinds.some(kind => typeof kind === "string" && COOLDOWN_RECOVERY_KINDS.has(kind)); - }); + return physicalUsageAttempts(entry?.attempts ?? []).some(attempt => + attempt.recoveryKinds.some(kind => COOLDOWN_RECOVERY_KINDS.has(kind))); } -/** Same attempt-aware pricing as /api/usage, so a refused retry before a measured send is priced. */ +/** + * Same attempt-aware pricing as /api/usage, over physical attempts only. Legacy root usage with no + * per-attempt usage is priced on the final physical attempt, never added beside it. + */ function successCostUsd(entry: PersistedUsageEntry): number | null { - const { isPriced, costTotal } = computeEntryCost(entry); + let priced = entry; + if (entry.attempts !== undefined) { + const attempts = physicalUsageAttempts(entry.attempts).map(attempt => ({ ...attempt })); + const finalAttempt = attempts.at(-1); + if (finalAttempt && entry.usage && !attempts.some(attempt => attempt.usage !== undefined)) { + finalAttempt.usage = entry.usage; + finalAttempt.usageStatus = entry.usageStatus; + } + const { usage: _usage, totalTokens: _totalTokens, ...rest } = entry; + priced = { ...rest, attempts }; + } + const { isPriced, costTotal } = computeEntryCost(priced); return isPriced ? costTotal : null; } +function blankUsage(): UsageAccumulator { + return { inclusiveInputTokens: 0, rawInputTokens: 0, cacheReadInputTokens: 0, cacheWriteInputTokens: 0 }; +} + +function blankCoverage(): CoverageAccumulator { + return { totalAttempts: 0, reportedAttempts: 0, estimatedAttempts: 0, unreportedAttempts: 0, unsupportedAttempts: 0 }; +} + +function addAttemptUsage(usage: UsageAccumulator, cache: CacheDecomposition | undefined): void { + if (!cache) return; + usage.inclusiveInputTokens += cache.inclusive; + usage.cacheReadInputTokens += cache.read; + usage.cacheWriteInputTokens += cache.write; + usage.rawInputTokens += cache.raw; +} + +function addCoverage(coverage: CoverageAccumulator, attempt: PhysicalAttempt): void { + coverage.totalAttempts += 1; + if (attempt.usageStatus === "unsupported") { + coverage.unsupportedAttempts += 1; + } else if ( + attempt.cache + && (attempt.usageStatus === "reported" || attempt.usageStatus === "estimated") + ) { + if (attempt.usageEstimated || attempt.usageStatus === "estimated") coverage.estimatedAttempts += 1; + else coverage.reportedAttempts += 1; + } else { + coverage.unreportedAttempts += 1; + } +} + +function finalUsage(usage: UsageAccumulator): AnalyticsAttemptUsage { + const total = usage.inclusiveInputTokens; + return { + ...usage, + rawInputShare: total > 0 ? usage.rawInputTokens / total : null, + cacheReadShare: total > 0 ? usage.cacheReadInputTokens / total : null, + cacheWriteShare: total > 0 ? usage.cacheWriteInputTokens / total : null, + }; +} + +function finalCoverage(coverage: CoverageAccumulator): AnalyticsPhysicalUsageCoverage { + const measuredAttempts = coverage.reportedAttempts + coverage.estimatedAttempts; + const supportedAttempts = coverage.totalAttempts - coverage.unsupportedAttempts; + return { + ...coverage, + measuredAttempts, + ratio: coverage.totalAttempts > 0 ? measuredAttempts / coverage.totalAttempts : null, + supportedRatio: supportedAttempts > 0 ? measuredAttempts / supportedAttempts : null, + }; +} + +function physicalAccumulator(provider: string, model: string, accountRef?: string): PhysicalAccumulator { + return { + provider, + model, + ...(accountRef ? { accountRef } : {}), + requests: 0, + physicalAttempts: 0, + physicalSends: 0, + repeatedSendAttempts: 0, + recoveryAttempts: 0, + recoveryEvents: 0, + comboFailoverRequests: 0, + usage: blankUsage(), + coverage: blankCoverage(), + }; +} + export async function computeRoutingAnalytics( filters: RoutingAnalyticsFilters, options: { maxRows?: number } = {}, ): Promise { + const generatedAt = Date.now(); const maxRows = Math.min( Math.max(1, Math.trunc(options.maxRows ?? ANALYTICS_MAX_ROWS)), ANALYTICS_MAX_ROWS, @@ -182,23 +530,46 @@ export async function computeRoutingAnalytics( if (filters.provider !== undefined) add("provider = ?", filters.provider); if (filters.model !== undefined) add("model = ?", filters.model); if (filters.profileId !== undefined) add("profile_id = ?", filters.profileId); - if (filters.surface !== undefined) add("surface = ?", filters.surface); + if (filters.surface === "codex") where.push("surface IS NULL"); + else if (filters.surface === "claude") where.push("surface IN ('claude', 'claude-desktop')"); + else if (filters.surface === "grok") where.push("surface = 'grok'"); if (filters.from !== undefined) add("timestamp >= ?", filters.from); if (filters.to !== undefined) add("timestamp <= ?", filters.to); const whereSql = where.length > 0 ? ` WHERE ${where.join(" AND ")}` : ""; const rows = await withRequestHistoryRead(handle => handle.query( - `SELECT provider, model, api_key_id AS apiKeyId, profile_id AS profileId, + `SELECT request_id AS requestId, timestamp, provider, model, + api_key_id AS apiKeyId, profile_id AS profileId, profile_revision AS profileRevision, status, duration_ms AS durationMs, first_output_ms AS firstOutputMs, close_reason AS closeReason, terminal_status AS terminalStatus, - usage_status AS usageStatus, usage_json AS usageJson, - attempt_count AS attemptCount, fallback, row_json AS rowJson - FROM requests${whereSql} ORDER BY timestamp DESC LIMIT ?`, + usage_status AS usageStatus, attempt_count AS attemptCount, + fallback, row_json AS rowJson + FROM requests${whereSql} ORDER BY timestamp DESC, request_id DESC LIMIT ?`, ).all(...values, maxRows + 1) as ScannedRow[]); const scanned = rows.slice(0, maxRows); const historyTruncated = rows.length > maxRows; + const expanded: ExpandedRow[] = scanned.map(row => { + const entry = parseEntry(row.rowJson); + row.rowJson = ""; + const attempts = physicalAttemptsFor(row, entry); + const hasDurableAttempts = entry?.attempts !== undefined; + const kind = classifyRow(row); + return { + row, + attempts, + fallback: hasDurableAttempts ? attempts.length > 1 : row.fallback === 1, + attemptCount: hasDurableAttempts ? attempts.length : row.attemptCount, + comboFailover: entry?.comboTargetAdvanced === true, + ...(entry?.conversationId?.trim() + ? { conversationId: entry.conversationId.trim() } + : {}), + kind, + rowCostUsd: kind === "success" && entry ? successCostUsd(entry) : null, + cooldownTriggering: kind === "failure" && cooldownTriggering(entry, row.status), + }; + }); let successes = 0; let failures = 0; @@ -207,7 +578,6 @@ export async function computeRoutingAnalytics( let totalAttempts = 0; let incompleteStreams = 0; let cooldownFailures = 0; - let usageReported = 0; const durations: number[] = []; const firstOutputs: number[] = []; let costTotalUsd = 0; @@ -216,37 +586,32 @@ export async function computeRoutingAnalytics( const byKey = new Map(); const byProfile = new Map(); - for (const row of scanned) { - const kind = classifyRow(row); + for (const { + row, + fallback, + attemptCount, + kind, + rowCostUsd, + cooldownTriggering: triggersCooldown, + } of expanded) { if (kind === "success") successes += 1; else if (kind === "failure") failures += 1; else cancelled += 1; - if (row.fallback === 1) fallbacks += 1; - totalAttempts += row.attemptCount; + if (fallback) fallbacks += 1; + totalAttempts += attemptCount; if (row.terminalStatus === "incomplete") incompleteStreams += 1; durations.push(row.durationMs); if (row.firstOutputMs !== null && row.firstOutputMs !== undefined && row.firstOutputMs >= 0) { firstOutputs.push(row.firstOutputMs); } - if (row.usageStatus !== "unreported") usageReported += 1; - - let rowCostUsd: number | null = null; - if (kind === "success") { - const entry = parseEntry(row.rowJson); - if (entry) { - rowCostUsd = successCostUsd(entry); - if (rowCostUsd !== null) { - costTotalUsd += rowCostUsd; - costCount += 1; - } - } - } - if (kind === "failure") { - const failureEntry = parseEntry(row.rowJson); - if (cooldownTriggering(failureEntry, row.status)) cooldownFailures += 1; + if (rowCostUsd !== null) { + costTotalUsd += rowCostUsd; + costCount += 1; } + if (triggersCooldown) cooldownFailures += 1; + const key = `${row.provider}\0${row.model}\0${row.apiKeyId ?? ""}\0${row.profileId ?? ""}`; let bucket: Bucket | undefined = byKey.get(key); if (!bucket) { @@ -293,14 +658,207 @@ export async function computeRoutingAnalytics( profile.requests += 1; if (kind === "success") profile.successes += 1; else if (kind === "failure") profile.failures += 1; - if (row.fallback === 1) profile.fallbacks += 1; + if (fallback) profile.fallbacks += 1; } } + const overall = physicalAccumulator("", ""); + const byPhysicalRoute = new Map(); + let comboFailoverRequests = 0; + let alertCount = 0; + const redAlerts: AnalyticsRedAlert[] = []; + const appendAlert = (alert: AnalyticsRedAlert) => { + alertCount += 1; + if (redAlerts.length === ANALYTICS_MAX_RED_ALERTS) redAlerts.shift(); + redAlerts.push(alert); + }; + const sequenceObservations: Array<{ + row: ScannedRow; + attempt: PhysicalAttempt; + conversationId?: string; + measuredCache?: CacheDecomposition; + }> = []; + + for (let index = expanded.length - 1; index >= 0; index--) { + const { row, attempts, comboFailover, conversationId } = expanded[index]!; + if (comboFailover) comboFailoverRequests += 1; + const seenRoutes = new Set(); + for (const [attemptIndex, attempt] of attempts.entries()) { + const measuredCache = attempt.usageStatus === "reported" || attempt.usageStatus === "estimated" + ? attempt.cache + : undefined; + const routeKey = `${attempt.provider}\0${attempt.model}\0${attempt.accountRef ?? ""}`; + let route = byPhysicalRoute.get(routeKey); + if (!route) { + route = physicalAccumulator(attempt.provider, attempt.model, attempt.accountRef); + byPhysicalRoute.set(routeKey, route); + } + if (!seenRoutes.has(routeKey)) { + seenRoutes.add(routeKey); + route.requests += 1; + if (comboFailover) route.comboFailoverRequests += 1; + } + for (const accumulator of [overall, route]) { + accumulator.physicalAttempts += 1; + accumulator.physicalSends += attempt.sendCount; + if (attempt.sendCount > 1) accumulator.repeatedSendAttempts += 1; + if (attempt.recoveryCount > 0) accumulator.recoveryAttempts += 1; + accumulator.recoveryEvents += attempt.recoveryCount; + addAttemptUsage(accumulator.usage, measuredCache); + addCoverage(accumulator.coverage, attempt); + } + + const baseAlert = { + requestId: row.requestId, + timestamp: row.timestamp, + ...(conversationId ? { conversationId } : {}), + provider: attempt.provider, + model: attempt.model, + ...(attempt.accountRef ? { accountRef: attempt.accountRef } : {}), + attemptOrdinal: attempt.ordinal, + }; + if (attempt.recoveryCount > 0) { + appendAlert({ + ...baseAlert, + kind: "recovery", + value: attempt.recoveryCount, + ...(attempt.recoveryKinds.length > 0 ? { recoveryKinds: attempt.recoveryKinds } : {}), + }); + } + if (attempt.sendCount > 1) { + appendAlert({ ...baseAlert, kind: "repeated-send", value: attempt.sendCount }); + } + if (comboFailover && attemptIndex === attempts.length - 1) { + appendAlert({ ...baseAlert, kind: "combo-failover", value: attempts.length }); + } + if (usesAnthropicCacheAlertPolicy(attempt)) { + sequenceObservations.push({ + row, + attempt, + ...(conversationId ? { conversationId } : {}), + ...(measuredCache ? { measuredCache } : {}), + }); + } + } + } + + type CacheLineage = { + previousRead: number; + previousInclusive: number; + previousCacheable: number; + previousSendAt: number; + previousTtlMs?: number; + }; + const sequence = new Map(); + sequenceObservations.sort((a, b) => + (a.attempt.firstSendAt ?? a.row.timestamp) - (b.attempt.firstSendAt ?? b.row.timestamp) + || a.row.requestId.localeCompare(b.row.requestId) + || a.attempt.ordinal - b.attempt.ordinal); + for (const { row, attempt, conversationId, measuredCache } of sequenceObservations) { + if (!measuredCache) continue; + const routeKey = `${attempt.provider}\0${attempt.model}\0${attempt.accountRef ?? ""}`; + const sequentialKey = `${conversationId ?? row.requestId}\0${routeKey}`; + let routeState = sequence.get(sequentialKey); + if (!routeState) { + routeState = { lineages: [] }; + sequence.set(sequentialKey, routeState); + } + const sendAt = attempt.firstSendAt ?? row.timestamp; + routeState.lineages = routeState.lineages.filter(lineage => + sendAt - lineage.previousSendAt + < (lineage.previousTtlMs ?? ANTHROPIC_DEFAULT_PROMPT_CACHE_TTL_MS)); + if ( + routeState.previousRawSendAt !== undefined + && sendAt - routeState.previousRawSendAt + >= (routeState.previousRawTtlMs ?? ANTHROPIC_DEFAULT_PROMPT_CACHE_TTL_MS) + ) { + routeState.previousRaw = undefined; + } + + const baseAlert = { + requestId: row.requestId, + timestamp: row.timestamp, + ...(conversationId ? { conversationId } : {}), + provider: attempt.provider, + model: attempt.model, + ...(attempt.accountRef ? { accountRef: attempt.accountRef } : {}), + attemptOrdinal: attempt.ordinal, + }; + const { inclusive, raw, read, write } = measuredCache; + let lineageIndex = -1; + for (const [index, candidate] of routeState.lineages.entries()) { + if ( + candidate.previousCacheable > 0 + && read / candidate.previousCacheable >= 0.9 + && ( + lineageIndex === -1 + || candidate.previousCacheable > routeState.lineages[lineageIndex]!.previousCacheable + ) + ) { + lineageIndex = index; + } + } + const previous = lineageIndex >= 0 + ? routeState.lineages[lineageIndex] + : routeState.lineages.at(-1); + const newInputGrowth = Math.max(0, inclusive - (previous?.previousInclusive ?? inclusive)); + const excessWrite = Math.max(0, write - newInputGrowth); + const retainedPrefixShare = previous?.previousCacheable && previous.previousCacheable > 0 + ? Math.min(1, read / previous.previousCacheable) + : null; + if (raw > 100_000 && routeState.previousRaw !== undefined && routeState.previousRaw > 100_000) { + appendAlert({ ...baseAlert, kind: "consecutive-high-raw-input", value: raw, previousValue: routeState.previousRaw }); + } + if (previous) { + if (excessWrite > 10_000) { + appendAlert({ ...baseAlert, kind: "high-cache-write-after-warmup", value: excessWrite }); + } + if (retainedPrefixShare !== null && retainedPrefixShare < 0.9) { + appendAlert({ ...baseAlert, kind: "low-cache-read-share-after-warmup", value: retainedPrefixShare }); + } + if (read < previous.previousRead) { + appendAlert({ ...baseAlert, kind: "falling-cache-read", value: read, previousValue: previous.previousRead }); + } + } + routeState.previousRaw = raw; + routeState.previousRawSendAt = sendAt; + routeState.previousRawTtlMs = attempt.promptCacheTtlMs; + if (read > 0 || write > 0) { + const nextLineage: CacheLineage = { + previousRead: read, + previousInclusive: inclusive, + previousCacheable: read + write, + previousSendAt: sendAt, + ...(attempt.promptCacheTtlMs ? { previousTtlMs: attempt.promptCacheTtlMs } : {}), + }; + if (lineageIndex >= 0) routeState.lineages.splice(lineageIndex, 1); + routeState.lineages.push(nextLineage); + if (routeState.lineages.length > 8) routeState.lineages.shift(); + } + } + const sequentialRoutes = sequence.size; + const warmedRoutes = [...sequence.values()].filter(state => state.lineages.length > 0).length; + durations.sort((a, b) => a - b); firstOutputs.sort((a, b) => a - b); const total = scanned.length; const rate = (count: number): number | null => (total > 0 ? count / total : null); + const timestamps = scanned.map(row => row.timestamp).filter(Number.isFinite); + const observedStart = timestamps.length > 0 ? Math.min(...timestamps) : undefined; + const observedEnd = timestamps.length > 0 ? Math.max(...timestamps) : undefined; + const windowStart = filters.from ?? observedStart; + const windowEnd = filters.to ?? (filters.from !== undefined ? generatedAt : observedEnd); + const windowMs = windowStart !== undefined && windowEnd !== undefined && windowEnd > windowStart + ? windowEnd - windowStart + : 0; + const requestRatePerHour = (count: number): number | null => windowMs > 0 + ? count / (windowMs / HOUR_MS) + : null; const breakdown: AnalyticsBreakdownRow[] = [...byKey.values()].map(bucket => { const sorted = bucket.durations.sort((a, b) => a - b); @@ -329,12 +887,34 @@ export async function computeRoutingAnalytics( successRate: profile.requests > 0 ? profile.successes / profile.requests : null, })).sort((a, b) => b.requests - a.requests); + const physicalBreakdown: AnalyticsPhysicalBreakdownRow[] = [...byPhysicalRoute.values()].map(route => ({ + provider: route.provider, + model: route.model, + ...(route.accountRef ? { accountRef: route.accountRef } : {}), + requests: route.requests, + physicalAttempts: route.physicalAttempts, + physicalSends: route.physicalSends, + repeatedSendAttempts: route.repeatedSendAttempts, + recoveryAttempts: route.recoveryAttempts, + recoveryEvents: route.recoveryEvents, + recoveryRate: route.physicalAttempts > 0 ? route.recoveryAttempts / route.physicalAttempts : null, + comboFailoverRequests: route.comboFailoverRequests, + comboFailoverRate: route.requests > 0 ? route.comboFailoverRequests / route.requests : null, + requestRatePerHour: requestRatePerHour(route.requests), + attemptUsage: finalUsage(route.usage), + usageCoverage: finalCoverage(route.coverage), + })).sort((a, b) => b.physicalAttempts - a.physicalAttempts + || a.provider.localeCompare(b.provider) + || a.model.localeCompare(b.model) + || (a.accountRef ?? "").localeCompare(b.accountRef ?? "")); + const confidence: AnalyticsConfidence | null = total === 0 ? null : total >= 100 ? "high" : total >= 20 ? "medium" : "low"; + const physicalUsageCoverage = finalCoverage(overall.coverage); return { - generatedAt: Date.now(), + generatedAt, totalRequests: total, scannedRows: scanned.length, historyTruncated, @@ -345,6 +925,24 @@ export async function computeRoutingAnalytics( fallbackRate: rate(fallbacks), totalAttempts, averageAttemptsPerRequest: total > 0 ? totalAttempts / total : null, + physicalAttempts: overall.physicalAttempts, + physicalSends: overall.physicalSends, + repeatedSendAttempts: overall.repeatedSendAttempts, + recoveryAttempts: overall.recoveryAttempts, + recoveryEvents: overall.recoveryEvents, + recoveryRate: overall.physicalAttempts > 0 ? overall.recoveryAttempts / overall.physicalAttempts : null, + comboFailoverRequests, + comboFailoverRate: rate(comboFailoverRequests), + requestRatePerHour: requestRatePerHour(total), + attemptUsage: finalUsage(overall.usage), + physicalUsageCoverage, + physicalBreakdown, + redAlerts: redAlerts.sort((a, b) => b.timestamp - a.timestamp + || b.requestId.localeCompare(a.requestId) + || b.attemptOrdinal - a.attemptOrdinal), + redAlertsPartial: historyTruncated || alertCount > ANALYTICS_MAX_RED_ALERTS, + sequentialRoutes, + warmedRoutes, incompleteStreamRate: rate(incompleteStreams), cooldownTriggeringFailures: cooldownFailures, durationMs: { @@ -362,7 +960,7 @@ export async function computeRoutingAnalytics( }, estimatedCostUsdPerSuccessfulRequest: costCount > 0 ? costTotalUsd / costCount : null, estimatedCostUsdTotalSuccessful: costCount > 0 ? costTotalUsd : null, - usageCoverage: total > 0 ? usageReported / total : null, + usageCoverage: physicalUsageCoverage.ratio, priceCoverage: successes > 0 ? costCount / successes : null, breakdown, profileBreakdown, diff --git a/src/routing/health.ts b/src/routing/health.ts index af4b20f5e42..61bcd2a5be6 100644 --- a/src/routing/health.ts +++ b/src/routing/health.ts @@ -17,6 +17,7 @@ */ import type { OcxConfig } from "../types"; +import { physicalUsageAttempts } from "../usage/log"; import { openRequestHistoryIndexSync, reportRequestHistoryCorruption, requestHistoryDb } from "./history/indexer"; import { OPENAI_CODEX_PROVIDER_ID } from "../providers/openai-tiers"; import { @@ -89,8 +90,12 @@ interface HealthSample { status: number; closeReason: string | null; terminalStatus: string | null; + streamAborted?: boolean; durationMs: number; timestamp: number; + /** Ledger insertion order, then attempt order: ties between equal timestamps. */ + rowOrder?: number; + attemptOrder?: number; } interface HealthRow extends HealthSample { @@ -110,7 +115,7 @@ interface HealthRow extends HealthSample { * candidate) and `[]` when the row sent nothing upstream at all. */ function attemptSamplesFor( - row: Pick, + row: Pick, provider: string, model: string, ): HealthSample[] | undefined { @@ -128,25 +133,27 @@ function attemptSamplesFor( (attempt): attempt is Record => !!attempt && typeof attempt === "object" && !Array.isArray(attempt), ); - const sent = attempts.filter(attempt => attempt.locallyAnswered !== true && attempt.sendCount !== 0); + // Physical sends only, one per ordinal; a combo's single send still names its own target. + const sent = physicalUsageAttempts(attempts); if (sent.length === 0) return []; - const mine = attempts.filter(attempt => attempt.provider === provider && attempt.model === model); - if (attempts.length <= 1 || mine.length === 0) return undefined; const finalSend = sent.at(-1); const samples: HealthSample[] = []; - for (const record of mine) { - if (!sent.includes(record)) continue; + for (const [attemptOrder, record] of sent.entries()) { + if (record.provider !== provider || record.model !== model) continue; if (typeof record.status !== "number" || typeof record.durationMs !== "number") continue; const isFinal = record === finalSend; samples.push({ status: record.status, closeReason: isFinal && typeof parsed.closeReason === "string" ? parsed.closeReason : null, terminalStatus: isFinal && typeof parsed.terminalStatus === "string" ? parsed.terminalStatus : null, + ...(record.streamAborted === true ? { streamAborted: true } : {}), durationMs: record.durationMs, timestamp: row.timestamp, + rowOrder: row.rowOrder, + attemptOrder, }); } - return samples; + return samples.length > 0 ? samples : undefined; } catch { return undefined; } @@ -224,6 +231,7 @@ export function policyCandidateHealthEvidence( function classifySample(sample: HealthSample): "success" | "failure" | "neutral" { if (sample.closeReason === "client_cancel" || sample.status === 499) return "neutral"; + if (sample.streamAborted) return "failure"; // Invalid requests and policy refusals must not poison target health. if (sample.status >= 400 && sample.status < 500 && sample.status !== 429) return "neutral"; if (sample.terminalStatus === "incomplete") return "failure"; @@ -263,9 +271,9 @@ function computeHistoricalHealthEvidence( } const rows = handle.query( `SELECT status, close_reason AS closeReason, terminal_status AS terminalStatus, - duration_ms AS durationMs, timestamp, row_json AS rowJson + duration_ms AS durationMs, timestamp, rowid AS rowOrder, row_json AS rowJson FROM requests WHERE ${where.join(" AND ")} - ORDER BY timestamp DESC LIMIT ?`, + ORDER BY timestamp DESC, rowid DESC LIMIT ?`, ).all(...values, HEALTH_MAX_SAMPLES) as HealthRow[]; // Rows whose top-level target differs from this candidate may still carry // candidate attempts (combo/failover): expand those too. The serialized @@ -273,12 +281,12 @@ function computeHistoricalHealthEvidence( // rows that cannot contribute samples for this candidate. const escapeLike = (value: string): string => value.replace(/[\\%_]/g, match => `\\${match}`); const attemptRows = handle.query( - `SELECT timestamp, row_json AS rowJson - FROM requests WHERE timestamp >= ? AND attempt_count > 1 + `SELECT timestamp, rowid AS rowOrder, row_json AS rowJson + FROM requests WHERE timestamp >= ? AND attempt_count > 0 AND row_json LIKE ? ESCAPE '\\' AND row_json LIKE ? ESCAPE '\\' AND NOT (provider = ? AND model = ?) - ORDER BY timestamp DESC LIMIT ?`, + ORDER BY timestamp DESC, rowid DESC LIMIT ?`, ).all( now - HEALTH_WINDOW_MS, `%\"provider\":\"${escapeLike(input.provider)}\"%`, @@ -286,7 +294,7 @@ function computeHistoricalHealthEvidence( input.provider, input.model, HEALTH_MAX_SAMPLES, - ) as Array>; + ) as Array>; const samples: HealthSample[] = []; for (const row of rows) { @@ -295,9 +303,11 @@ function computeHistoricalHealthEvidence( for (const row of attemptRows) { samples.push(...(attemptSamplesFor(row, input.provider, input.model) ?? [])); } - // Newest first for the consecutive-failure walk; attempt samples inherit - // their row's timestamp. - samples.sort((a, b) => b.timestamp - a.timestamp); + // Newest physical send first for the consecutive-failure walk: insertion order breaks + // equal-timestamp row ties, attempt order breaks ties between retries in one request. + samples.sort((a, b) => b.timestamp - a.timestamp + || (b.rowOrder ?? 0) - (a.rowOrder ?? 0) + || (b.attemptOrder ?? 0) - (a.attemptOrder ?? 0)); let successes = 0; let failures = 0; @@ -317,7 +327,7 @@ function computeHistoricalHealthEvidence( failures += 1; weightedTotal += weight; } - if (sample.terminalStatus === "incomplete") incompleteStreams += 1; + if (sample.terminalStatus === "incomplete" || sample.streamAborted) incompleteStreams += 1; latencies.push(sample.durationMs); } // Consecutive failures: walk newest -> oldest until a success. diff --git a/src/routing/history/ingest.ts b/src/routing/history/ingest.ts index 429fe4dc974..ca472f6c440 100644 --- a/src/routing/history/ingest.ts +++ b/src/routing/history/ingest.ts @@ -11,6 +11,8 @@ import { closeSync, fstatSync, openSync, readSync, type BigIntStats } from "node import { encodePersistedRequestedModel, exactFileIdentity, + normalizedAttempts, + physicalUsageAttempts, type PersistedUsageEntry, type UsageLogRevision, } from "../../usage/log"; @@ -88,7 +90,8 @@ export function recordSourceMeta(dbHandle: Database, revision: UsageLogRevision /** Extract the `requests` row columns from a canonical persisted entry. */ function extractRow(entry: PersistedUsageEntry): Array { - const attempts = entry.attempts; + // Fallback and attempt count describe real upstream sends only (valid, sent, distinct ordinals). + const attempts = Array.isArray(entry.attempts) ? physicalUsageAttempts(normalizedAttempts(entry.attempts)) : undefined; return [ entry.requestId, entry.timestamp, diff --git a/src/server/agentradar-count-tokens.ts b/src/server/agentradar-count-tokens.ts new file mode 100644 index 00000000000..c59847f659a --- /dev/null +++ b/src/server/agentradar-count-tokens.ts @@ -0,0 +1,211 @@ +/** + * AgentRadar hosted gateways (.27): `/v1/messages/count_tokens` for a canonical Anthropic target is + * Anthropic's own exact count rather than the local estimate, because Claude Code decides when to + * compact from this number. The count sends the exact source the native lane would send, under the + * same eligibility, image policy, cooldowns and key scope as generation. A pinned request counts on + * its pinned account only; an unpinned one selects an account exactly as a dispatch would and + * commits nothing. It is neither billed nor request-logged. `undefined` keeps the local estimate. + */ +import { createHash } from "node:crypto"; +import { buildAnthropicMessagesPassthroughRequest, isCanonicalAnthropicMessagesUrl } from "../adapters/anthropic/passthrough"; +import { resolveInboundModel } from "../claude/inbound"; +import { anthropicErrorResponse } from "../claude/outbound"; +import { getCombo, pickComboTarget, resolveComboId } from "../combos"; +import { readBoundedResponseBytes } from "../lib/bounded-body"; +import { formatClientRetryAfter } from "../lib/client-retry-after"; +import { agentRadarAccountPin, agentRadarPinnedConfig } from "../lib/agentradar-account-pin"; +import { createTranslatorBudget, isTranslatorBudgetExceededError } from "../lib/translator-budget"; +import { applyUpstreamRecoveryInit } from "../lib/upstream-retry"; +import { anthropicSessionKeyFromParts } from "../oauth/anthropic-routing"; +import { publicOAuthAuthenticationErrorMessage } from "../oauth"; +import { createProtocolEnvelope } from "../protocols/envelope"; +import { featuresFromMessagesBody } from "../protocols/features"; +import { configuredAnthropicInstance } from "../providers/anthropic-instance"; +import { captureRouteStaticPolicy, routeConcreteModel, routeModel, type RouteResult } from "../router"; +import { POLICY_NAMESPACE, resolvePolicyProfileId } from "../routing/profile"; +import type { OcxConfig } from "../types"; +import { resolveWireProtocolOverride } from "./adapter-resolve"; +import { resolveAdmissionModelScope, routeAllowedByScope } from "./admission-model-scope"; +import type { DataPlaneAdmission } from "./auth-cors"; +import { nativeMessagesDeclineReason, type NativeMessagesSelector } from "./messages-native-eligibility"; +import { nativeOAuthBindingIsCurrent, resolveNativeOAuthBindingForInstance, type NativeOAuthBinding } from "./messages-native-oauth"; +import { conversationIdFromClaudeMetadata } from "./request-log-conversation"; +import { createComboProtocolLanes, hostedReplayIncompatible } from "./responses/core-combo-native"; +import { providerFetch } from "./responses/fetch-helpers"; + +type Rec = Record; +const COUNT_MAX_BYTES = 64 * 1024; +const isRec = (value: unknown): value is Rec => value !== null && typeof value === "object" && !Array.isArray(value); +// The fields count_tokens and its Messages turn share; sampling, stream and metadata differ. +const SEED_FIELDS = ["cache_control", "context_management", "mcp_servers", "messages", "model", "output_config", "system", "thinking", "tool_choice", "tools"]; +const canonicalJson = (value: unknown): string => Array.isArray(value) ? `[${value.map(canonicalJson).join(",")}]` + : isRec(value) ? `{${Object.keys(value).sort().map(key => `${JSON.stringify(key)}:${canonicalJson(value[key])}`).join(",")}}` + : JSON.stringify(value) ?? "null"; + +/** The .27 combo contract covers a combo only when one of its targets is canonical Anthropic. */ +export function agentRadarCanonicalCombo(config: OcxConfig, cc: OcxConfig["claudeCode"], body: Rec): boolean { + const combo = typeof body.model === "string" ? getCombo(config, resolveComboId(config, resolveInboundModel(body.model, cc)) ?? "") : undefined; + return !!combo?.targets.some(target => { + const provider = config.providers[target.provider]; + return provider?.adapter === "anthropic" && isCanonicalAnthropicMessagesUrl(provider); + }); +} + +/** + * One random-combo seed for a count_tokens request and the Messages turn it precedes (.27). Only a + * combo with a canonical Anthropic target is correlated; other combos keep independent picks. + */ +export function agentRadarComboSeed(config: OcxConfig, cc: OcxConfig["claudeCode"], body: Rec): string | undefined { + if (!agentRadarCanonicalCombo(config, cc, body)) return undefined; + return createHash("sha256").update(canonicalJson(Object.fromEntries(SEED_FIELDS.filter(key => Object.hasOwn(body, key)).map(key => [key, body[key]])))).digest("hex"); +} + +export async function agentRadarExactCountTokens( + req: Request, + config: OcxConfig, + cc: OcxConfig["claudeCode"], + raw: Rec, + fastRow: boolean, + admission?: DataPlaneAdmission, +): Promise { + const pin = agentRadarAccountPin(req.headers); + if (pin === null) return undefined; // refused by the caller before this runs + if (typeof raw.model !== "string") return undefined; + const selectorId = resolveInboundModel(raw.model, cc); + if (resolvePolicyProfileId(config, selectorId) !== null || selectorId.startsWith(`${POLICY_NAMESPACE}/`)) return undefined; + const scope = resolveAdmissionModelScope(config, admission); + const selector: NativeMessagesSelector = { fastRow, claudeCode: cc, webSearchBackendAvailable: false }; + const translatorBudget = createTranslatorBudget(); + try { + let route: RouteResult; + try { + const comboId = resolveComboId(config, selectorId); + if (comboId) { + // Generation refuses a pinned combo on the bridge; a pin never counts on a combo target. + if (pin) return undefined; + const combo = getCombo(config, comboId); + if (!combo) return undefined; + if (combo.imageInput === "disabled" && featuresFromMessagesBody(raw).has("request.images")) { + return anthropicErrorResponse(400, `Combo "${comboId}" does not accept image input`, "invalid_request_error"); + } + const lanes = createComboProtocolLanes({ + source: { + inbound: "messages", + envelope: createProtocolEnvelope({ inbound: "messages", body: raw, translatorBudget }), + selector, + dispatchNativeChild: () => Promise.reject(new Error("count_tokens never dispatches a combo child")), + }, + req, config, logCtx: { model: selectorId, provider: comboId }, admissionScope: scope, comboId, targets: combo.targets, + })!; + // Preview: the generation that follows owns any round-robin or cooldown-reprobe state. + const pick = pickComboTarget(config, comboId, { + eligible: target => lanes.pickable(target), preview: true, allowCooldownReprobe: false, randomSeed: agentRadarComboSeed(config, cc, raw), + }); + if (!pick) return lanes.refusal() ?? anthropicErrorResponse(503, `No available targets for combo: ${comboId}`, "api_error"); + route = routeConcreteModel(config, `${pick.target.provider}/${pick.target.model}`); + selector.routeSelector = `${pick.target.provider}/${pick.target.model}`; + } else { + route = routeModel(config, selectorId); + selector.routeSelector = selectorId; + // A pin serves only its own Anthropic OAuth account, exactly as generation's pinServes. + if (pin && !(pin.provider === "claude" && route.providerName === "anthropic" && route.provider.authMode === "oauth")) return undefined; + if (raw.model !== route.modelId && hostedReplayIncompatible(raw, route.modelId)) { + return anthropicErrorResponse(400, "request cannot be replayed exactly to canonical Anthropic", "invalid_request_error"); + } + } + } catch { + // An unavailable or invalid route has no exact counter; keep upstream's local estimate. + return undefined; + } + route.staticPolicy = captureRouteStaticPolicy(route.providerName, route.modelId, route.provider, route.staticPolicy.effectiveAlias, "anthropic"); + route.provider = resolveWireProtocolOverride(route.providerName, route.modelId, route.provider, "anthropic", route.staticPolicy); + const countConfig = pin ? agentRadarPinnedConfig(config, pin) : config; + if (route.provider.adapter !== "anthropic" || !isCanonicalAnthropicMessagesUrl(route.provider) + || !routeAllowedByScope(scope, route) + || nativeMessagesDeclineReason(route, raw, countConfig, selector) !== undefined) return undefined; + + let provider = route.provider; + let providerAccountUuid: string | undefined; + let binding: NativeOAuthBinding | undefined; + if (provider.authMode === "oauth") { + const instance = configuredAnthropicInstance(config, route.providerName); + if (!instance) return undefined; + try { + binding = await resolveNativeOAuthBindingForInstance(instance, countConfig, { + model: route.modelId, + readOnly: true, + sessionKey: anthropicSessionKeyFromParts({ + sessionIdHeader: req.headers.get("session_id")?.trim() || req.headers.get("x-claude-code-session-id"), + threadIdHeader: req.headers.get("thread_id") ?? req.headers.get("thread-id"), + clientThreadId: conversationIdFromClaudeMetadata(isRec(raw.metadata) ? raw.metadata : undefined) ?? null, + promptCacheKey: null, + }), + }); + provider = { ...provider, apiKey: binding.snapshot.accessToken }; + providerAccountUuid = binding.providerAccountUuid; + } catch (error) { + // Fail closed for a pin: never estimate as if its own account had answered. + if (pin) return anthropicErrorResponse(401, publicOAuthAuthenticationErrorMessage(error), "authentication_error"); + return undefined; + } + } + + let built: ReturnType; + try { + translatorBudget.chargeRetained(Buffer.byteLength(JSON.stringify(raw)), { kind: "request_copies" }); + built = buildAnthropicMessagesPassthroughRequest(provider, route.modelId, raw, config, { + callerAnthropicBeta: req.headers.get("anthropic-beta"), + providerAccountUuid, + exactSource: true, + }); + translatorBudget.chargeRetained(Buffer.byteLength(built.body), { kind: "request_copies" }); + } catch (error) { + if (!isTranslatorBudgetExceededError(error)) return undefined; + return anthropicErrorResponse(413, "request translation buffer exceeded the safe limit", "request_too_large", "translation_buffer_limit"); + } + const countUrl = new URL(built.url); + countUrl.pathname = "/v1/messages/count_tokens"; + const signal = AbortSignal.any([req.signal, AbortSignal.timeout(config.connectTimeoutMs ?? 200_000)]); + try { + const upstream = await providerFetch(provider, undefined, { + providerName: route.providerName, modelId: route.modelId, + dispatchOverride: async (input, init, execute, reacquirePacing) => { + // Pacing may outlive a pause, removal or credential replacement. A count never commits a + // selection; re-resolve the same pin (or the ordinary read-only pool) before the send. + for (let attempt = 0; binding && !nativeOAuthBindingIsCurrent(binding); attempt++) { + signal.throwIfAborted(); + if (attempt >= 3) throw new Error("Anthropic count credential changed before dispatch"); + binding = await resolveNativeOAuthBindingForInstance(binding.instance, countConfig, { + routeTarget: binding.routeTarget, sessionKey: binding.sessionKey, model: route.modelId, readOnly: true, + }); + provider = { ...route.provider, apiKey: binding.snapshot.accessToken }; + translatorBudget.releaseRetained(Buffer.byteLength(built.body), { kind: "request_copies" }); + built = buildAnthropicMessagesPassthroughRequest(provider, route.modelId, raw, config, { + callerAnthropicBeta: req.headers.get("anthropic-beta"), providerAccountUuid: binding.providerAccountUuid, exactSource: true, + }); + translatorBudget.chargeRetained(Buffer.byteLength(built.body), { kind: "request_copies" }); + await reacquirePacing?.(provider, route.modelId); + } + signal.throwIfAborted(); + return execute(input, applyUpstreamRecoveryInit({ ...init, headers: built.headers, body: built.body }, "connection-reset")); + }, + })( + countUrl, + applyUpstreamRecoveryInit({ method: "POST", headers: built.headers, body: built.body, signal }, "connection-reset"), + ); + const read = await readBoundedResponseBytes(upstream, { signal, maxBytes: COUNT_MAX_BYTES }); + if (read.oversized) return anthropicErrorResponse(502, "Anthropic token count body exceeded safe limit", "api_error"); + const retryAfter = formatClientRetryAfter(upstream.headers.get("retry-after")); + return new Response(read.bytes.slice(), { + status: upstream.status, + headers: { "Content-Type": upstream.headers.get("content-type") ?? "application/json", ...(retryAfter ? { "Retry-After": retryAfter } : {}) }, + }); + } catch { + if (req.signal.aborted) return anthropicErrorResponse(499, "client closed request during Anthropic token count", "api_error"); + if (signal.aborted) return anthropicErrorResponse(504, "Anthropic token count timed out", "timeout_error"); + return anthropicErrorResponse(502, "Anthropic token count failed", "api_error"); + } + } finally { + translatorBudget.dispose(); + } +} diff --git a/src/server/chat-completions.ts b/src/server/chat-completions.ts index 44691c9e57d..322e349c348 100644 --- a/src/server/chat-completions.ts +++ b/src/server/chat-completions.ts @@ -75,6 +75,7 @@ import { attachAdvisorReasoning } from "./inference/advisor-diagnostics"; import { directEncodersApply } from "./inference/client-encoder-delivery"; import { responseWithDeferredRequestLog } from "./relay"; import { handleResponses } from "./responses"; +import { agentRadarAccountPin, agentRadarPinnedConfig } from "../lib/agentradar-account-pin"; import { previewXaiOauthWireModel } from "./responses/core-normalize"; import { providerConsumesCallerAuthorization } from "../providers/caller-authorization"; import { captureExplicitOpenAiCallerAuth } from "../providers/openai-sidecar"; @@ -178,6 +179,11 @@ async function handleChatCompletionsWithBudget( advisorAdmissionPolicy?: CapturedAdmissionPolicy; }, ): Promise { + const agentRadarAccount = agentRadarAccountPin(req.headers); + if (agentRadarAccount === null) { + if (logIds) addFinalRequestLog(logIds.requestId, logIds.start, logCtx, 400, { closeReason: "non_stream" }); + return chatCompletionsErrorResponse(400, "Invalid AgentRadar account pin", "invalid_request_error"); + } const advisorInternal = isInternalCallCapability(req.headers.get(ADVISOR_INTERNAL_CAPABILITY_HEADER)); const sidecarInternal = isInternalCallCapability(req.headers.get(SIDECAR_INTERNAL_CAPABILITY_HEADER)); const callerPolicy = advisorInternal @@ -296,7 +302,7 @@ async function handleChatCompletionsWithBudget( nativeDecline = route.combo ? "combo-or-policy-route" : effortRow ? "effort-row" : nativeChatDeclineReason(route, chatBody, config); - if (nativeDecline === undefined) { + if (nativeDecline === undefined && !agentRadarAccount) { chatNativeRoute = route; // Reserve an input estimate for spend without recording it as usage: native Chat attempts // keep the provider-reported counts, as they did before the reservation existed. @@ -510,7 +516,7 @@ async function handleChatCompletionsWithBudget( } // Existing primary enrichment stays on non-caller-auth routes. Caller-auth routes defer // optional stored sidecar auth until the final helper plan actually needs it. - if (!callerAuthorizationRoute) { + if (!callerAuthorizationRoute && !agentRadarAccount) { // This enrichment is optional for routed/non-main providers. If native main // is fenced, omit it and let auth-context reject only a final physical-main // selection while healthy pool/provider routes continue. @@ -560,10 +566,11 @@ async function handleChatCompletionsWithBudget( linkRequestSessionLane(req, internalReq); const finalizeNativeLog = createFinalRequestLog(logIds, logCtx).finish; - const upstream = await handleResponses(internalReq, config, logCtx, { + const upstream = await handleResponses(internalReq, agentRadarAccount ? agentRadarPinnedConfig(config, agentRadarAccount) : config, logCtx, { + ...(agentRadarAccount ? { agentRadarAccount } : {}), clientModelLabel: clientFacingModelLabel(requestedModel), openAiSidecarAuth, - allowStoredOpenAiSidecarAuth: !!(callerAuthorizationRoute && settledRoute + allowStoredOpenAiSidecarAuth: !agentRadarAccount && !!(callerAuthorizationRoute && settledRoute && !isCanonicalOpenAiForwardProvider(settledRoute.provider)), nativeCallerAuth, callerDirectAuth, diff --git a/src/server/claude-messages.ts b/src/server/claude-messages.ts index 790493cac42..f0d17d02e40 100644 --- a/src/server/claude-messages.ts +++ b/src/server/claude-messages.ts @@ -45,7 +45,7 @@ import { stripOneMillionMarker } from "../claude/context-windows"; import { captureAdmissionPolicy } from "./capture-admission-policy"; import type { CapturedAdmissionPolicy } from "../types/admission-policy"; import { captureClaudeInbound } from "../claude/inbound-debug"; -import { declaredAnthropicServerTools, replayedAnthropicServerBlocks, systemCachePrefix } from "../claude/inbound-content-options"; +import { declaredAnthropicServerTools, EXACT_REPLAY_REQUIRED_MESSAGE, hasNativeOnlyTool, replayedAnthropicServerBlocks, systemCachePrefix } from "../claude/inbound-content-options"; import { linkAdvisorPlanConfig } from "./responses/advisor-plan-slot"; import { claudeCodeForIngress } from "../claude/intercept/model-bindings"; import { classifyInterceptClient } from "../claude/intercept/client-class"; @@ -98,6 +98,9 @@ import { clientWireOf } from "./inference/client-wire"; import { directEncodersApply } from "./inference/client-encoder-delivery"; import type { ClientEncoderOption } from "./responses/core-options"; import { handleResponses } from "./responses"; +import { agentRadarAccountPin, agentRadarPinnedConfig } from "../lib/agentradar-account-pin"; +import { isCanonicalOpenAiForwardProvider } from "../providers/openai-tiers-destination"; +import { logTranslationBufferLimit } from "../lib/upstream-refusal-log"; import { resolveComboId } from "../combos"; import { previewXaiOauthWireModel } from "./responses/core-normalize"; import { upstreamWireForAdapter } from "../protocols/contract"; @@ -146,6 +149,8 @@ type Rec = Record; /** Which listener a Claude Messages request arrived on. Only the intercept ingress honours bindings. */ export interface ClaudeIngressOptions { claudeIntercept?: boolean; + /** count_tokens only: the admitted caller, so an AgentRadar exact count keeps the key's model scope. */ + admission?: DataPlaneAdmission; } /** @@ -287,7 +292,7 @@ function wantsNativePassthrough( function shouldForwardNativeHeader(name: string, value: string, config: OcxConfig): boolean { const lowerName = name.toLowerCase(); - if (PASSTHROUGH_STRIP_HEADERS.has(lowerName)) return false; + if (PASSTHROUGH_STRIP_HEADERS.has(lowerName) || lowerName.startsWith("x-agentradar-")) return false; if (lowerName !== "authorization" && lowerName !== "x-api-key") return true; const token = singleCredentialToken(lowerName, value); return !!token && !isProxyAdmissionSecret(token, config); @@ -398,7 +403,9 @@ async function anthropicNativePassthrough( ...anthropicRateLimitHeaders(upstream.headers), ...(pathname === "/v1/messages" ? upstreamMessagesRequestIdHeaders(upstream.headers) : {}), }; - const bodyGuard = resolvePassthroughBodyGuard(config, req.signal); + const bodyGuard = { ...resolvePassthroughBodyGuard(config, req.signal), + // The .27 strict relay belongs to canonical Anthropic only. + strictRelay: process.env.OPENCODEX_AGENTRADAR === "1" && base === "https://api.anthropic.com" }; if (upstream.ok && contentType.includes("text/event-stream") && upstream.body) { const frames = new MessagesSseFrameContext(); return new Response(tapAnthropicSseForLog(upstream.body, logCtx, finalize, bodyGuard, frames, createMessagesSseErrorFilter(frames)), { @@ -644,6 +651,13 @@ async function handleClaudeMessagesWithBudget( ingress: ClaudeIngressOptions = {}, ): Promise { logCtx.surface = "claude"; + const agentRadarAccount = agentRadarAccountPin(req.headers); + if (agentRadarAccount === null) { + if (logIds) addFinalRequestLog(logIds.requestId, logIds.start, logCtx, 400, { closeReason: "non_stream" }); + return anthropicErrorResponse(400, "Invalid AgentRadar account pin", "invalid_request_error"); + } + // The native lane reads the pin from its config: exact account, no rotation, no selection commit. + const laneConfig = agentRadarAccount ? agentRadarPinnedConfig(config, agentRadarAccount) : config; const disabled = claudeInboundDisabled(config); if (disabled) { markProtocolBlocked(logCtx, { inbound: "messages", reasonCodes: ["surface-disabled"] }); @@ -653,6 +667,7 @@ async function handleClaudeMessagesWithBudget( // Model resolution reads this view; every other claudeCode setting keeps reading `config`. const cc = claudeCodeForIngress(config.claudeCode, ingress.claudeIntercept === true); let nativeComboSource = false; + let comboRandomSeed: string | undefined; let anthropicBody: unknown; let internalBody: Rec; @@ -749,7 +764,8 @@ async function handleClaudeMessagesWithBudget( // the envelope then keeps it by reference and charges no ingress copy. nativeComboSource = !effortOverride && isRec(messagesBody) && typeof messagesBody.model === "string" && resolveComboId(config, resolveInboundModel(messagesBody.model, cc)) !== null - && Object.keys(config.providers).some(name => resolveProtocolSettings(config, name).rollout.managedMessagesNative); + && (process.env.OPENCODEX_AGENTRADAR === "1" + || Object.keys(config.providers).some(name => resolveProtocolSettings(config, name).rollout.managedMessagesNative)); if (isRec(messagesBody) && (resolveProtocolSettings(config).unrepresentable === "reject" || nativeComboSource)) { envelope = createProtocolEnvelope({ inbound: "messages", body: messagesBody, translatorBudget }); } @@ -761,7 +777,7 @@ async function handleClaudeMessagesWithBudget( messagesFeatures = sourceEnvelope ? () => sourceEnvelope.features() : () => (scannedFeatures ??= featuresFromMessagesBody(messagesBody)); - if (!effortRow && !fastRow && isRec(anthropicBody) && wantsNativePassthrough(req, config, requestPolicy, anthropicBody.model, cc)) { + if (!agentRadarAccount && !effortRow && !fastRow && isRec(anthropicBody) && wantsNativePassthrough(req, config, requestPolicy, anthropicBody.model, cc)) { markProtocolEntry(logCtx, { inbound: "messages", lane: "native", features: messagesFeatures }); recordProtocolShadowPlan(logCtx, config, { inbound: "messages", model: requestedModel }); let projectedBody: Rec; @@ -827,6 +843,10 @@ async function handleClaudeMessagesWithBudget( }; delete anthropicBody.thinking; } + // AgentRadar: after the rewrite, so this turn and its count_tokens derive one random-combo seed. + if (process.env.OPENCODEX_AGENTRADAR === "1" && isRec(anthropicBody)) { + comboRandomSeed = (await import("./agentradar-count-tokens")).agentRadarComboSeed(config, cc, anthropicBody); + } // On the direct lane the core receives only the request settings; the conversation is // already decoded into the IR. Without a plan the translation below runs as before. directPlan = planMessagesAntigravityDirect(anthropicBody, cc, config, translatorBudget); @@ -935,7 +955,7 @@ async function handleClaudeMessagesWithBudget( }; let directLeaveError: unknown; try { - const route = routeModel(config, internalBody.model as string, directPlan ? directPlan.evidence : evidenceFromBody(internalBody)); + const route = routeModel(config, internalBody.model as string, directPlan ? directPlan.evidence : evidenceFromBody(internalBody), { comboRandomSeed }); // Settle the wire once so the sampling decision below reads the effective // adapter rather than the provider-wide default (#404). route.staticPolicy = captureRouteStaticPolicy( @@ -958,7 +978,10 @@ async function handleClaudeMessagesWithBudget( if (directEncodersApply(config, route)) { clientEncoder = { protocol: "messages", stream, model: clientFacingModelText(clientModelLabel), inputTokenFloor: claudeRequestTokenFloor() }; } - if (route.provider.adapter === "openai-responses") { + // AgentRadar hosted gateways keep the .27 rule: only the canonical ChatGPT backend, which 400s on + // them, loses the sampling fields; other Responses gateways receive the caller's own. + if (route.provider.adapter === "openai-responses" + && (process.env.OPENCODEX_AGENTRADAR !== "1" || isCanonicalOpenAiForwardProvider(route.provider))) { // max_output_tokens is the caller's spend cap: the adapter strips it only for the canonical // ChatGPT backend that rejects it (#5753), so other Responses gateways keep the cap. delete internalBody.temperature; @@ -970,7 +993,10 @@ async function handleClaudeMessagesWithBudget( // request-side estimate so the log's in:0 rows get a floor. NEVER set this for // accurate-usage adapters — the request-log merge is max(reported, estimate) and // would overwrite real usage (audit 133 R1#7). - if (route.provider.adapter === "cursor" || route.provider.adapter === "kiro") { + // A combo's preview route is not the child that serves: on a hosted gateway (.27) the floor waits + // for a physical estimated-usage child rather than inflating a real Anthropic child's usage. + if ((route.provider.adapter === "cursor" || route.provider.adapter === "kiro") + && !(route.combo && process.env.OPENCODEX_AGENTRADAR === "1")) { logCtx.usageLogInputTokens = claudeRequestTokenFloor(); } // Effort safety valve (devlog 136 B6, audit 139 R2#2): opus-shaped aliases make @@ -1029,7 +1055,7 @@ async function handleClaudeMessagesWithBudget( ...(settledRoute?.provider.authMode === "oauth" && settledInstance ? { oauthFailoverQuorum: anthropicRoutingFor(settledInstance).hasAnthropicFailoverQuorum() } : {}), }; - if (settledRoute && isRec(anthropicBody) + if (!agentRadarAccount && settledRoute && isRec(anthropicBody) && nativeMessagesDeclineReason(settledRoute, anthropicBody, config, nativeSelector) === "bridge-only-policy" && nativeMessagesDeclineReason(settledRoute, anthropicBody, config, { ...nativeSelector, webSearchBackendAvailable: false }) === undefined) { const { parseRequest } = await import("../responses/parser"); @@ -1109,9 +1135,12 @@ async function handleClaudeMessagesWithBudget( } } const nativeDecline = settledRoute && isRec(anthropicBody) - ? nativeMessagesDeclineReason(settledRoute, anthropicBody, config, nativeSelector) + ? nativeMessagesDeclineReason(settledRoute, anthropicBody, laneConfig, nativeSelector) : "rollout-disabled"; - const nativeMessagesRoute = settledRoute && nativeDecline === undefined ? settledRoute : undefined; + // A pin serves only its own Anthropic OAuth account; any other route refuses on the bridge. + const pinServes = !agentRadarAccount || (agentRadarAccount.provider === "claude" && settledRoute?.providerName === "anthropic" + && settledRoute.provider.authMode === "oauth" && !settledRoute.combo); + const nativeMessagesRoute = pinServes && settledRoute && nativeDecline === undefined ? settledRoute : undefined; if (nativeProjectionOnlyError && !nativeMessagesRoute) { if (logIds) addFinalRequestLog(logIds.requestId, logIds.start, logCtx, 400, { closeReason: "non_stream" }); return anthropicErrorResponse(400, isProxyError(nativeProjectionOnlyError) ? nativeProjectionOnlyError.message : "The proxy could not translate the Messages request.", "invalid_request_error"); @@ -1164,6 +1193,14 @@ async function handleClaudeMessagesWithBudget( if (logIds) addFinalRequestLog(logIds.requestId, logIds.start, logCtx, 400, { closeReason: "non_stream" }); return anthropicErrorResponse(400, "The selected route cannot carry thinking signatures or redacted_thinking blocks", "invalid_request_error"); } + // An Anthropic-native client tool or MCP server the bridge would drop silently: refuse before + // any send rather than serve a turn whose model cannot call what the caller declared (.27). + if (!nativeMessagesRoute && !directRoute && settledRoute && !settledRoute.combo && settledRoute.routeKind !== "policy" + && isRec(anthropicBody) && (hasNativeOnlyTool(anthropicBody) || Object.hasOwn(anthropicBody, "mcp_servers"))) { + logCtx.errorCode = "unsupported_feature"; + if (logIds) addFinalRequestLog(logIds.requestId, logIds.start, logCtx, 400, { closeReason: "non_stream" }); + return anthropicErrorResponse(400, EXACT_REPLAY_REQUIRED_MESSAGE, "invalid_request_error"); + } if (nativeMessagesRoute) { markProtocolEntry(logCtx, { inbound: "messages", lane: "native", features: messagesFeatures }); let nativeBody: Rec; @@ -1177,7 +1214,7 @@ async function handleClaudeMessagesWithBudget( } const { handleNativeMessages } = await import("./messages-native"); return await handleNativeMessages({ - req, config, logCtx, ...(logIds ? { logIds } : {}), admission: logIds?.admission, + req, config: laneConfig, logCtx, ...(logIds ? { logIds } : {}), admission: logIds?.admission, route: nativeMessagesRoute, body: nativeBody, requestedModel: clientFacingModelText(clientModelLabel), translatorBudget, selector: nativeSelector, modelScope: resolveAdmissionModelScope(config, logIds?.admission), // Compatibility identity is an opaque request-local handle, separate from credentials. @@ -1185,7 +1222,7 @@ async function handleClaudeMessagesWithBudget( callerAnthropicBeta: req.headers.get("anthropic-beta"), sessionKey: anthropicSessionKeyFromParts({ sessionIdHeader: req.headers.get("session_id")?.trim() || req.headers.get("x-claude-code-session-id"), - threadIdHeader: req.headers.get("thread_id"), + threadIdHeader: req.headers.get("thread_id") ?? req.headers.get("thread-id"), clientThreadId: conversationIdFromClaudeMetadata(isRec(nativeBody.metadata) ? nativeBody.metadata : undefined), promptCacheKey: typeof internalBody.prompt_cache_key === "string" ? internalBody.prompt_cache_key : null, promptCacheKeyIsSharedCohort: cacheKeySource === "system", @@ -1206,7 +1243,7 @@ async function handleClaudeMessagesWithBudget( // native replays have no caller ChatGPT credential. This enrichment is optional: // auth-context later rejects a real physical-main selection, while routed/pool // traffic continues without reading native credentials during a fence/recovery. - if (tryClaimNativeMainProfileForTurn(logIds?.turnAdmissionLease)) { + if (!agentRadarAccount && tryClaimNativeMainProfileForTurn(logIds?.turnAdmissionLease)) { const { getMainAccountToken } = await import("../codex/main-account"); const token = getMainAccountToken(); if (token) { @@ -1240,9 +1277,10 @@ async function handleClaudeMessagesWithBudget( ingress: req.headers, replay: headers, metadataSessionId: claudeNativeSessionId, admission: logIds?.admission, }); let internalReq: Request; + let bodyBytes: number | undefined; try { // The UTF-16 JSON string and the Request's UTF-8 body coexist until dispatch. - const bodyBytes = jsonUtf8Bytes(internalBody); + bodyBytes = jsonUtf8Bytes(internalBody); const reservation = translatorBudget.reserveTransient(3 * bodyBytes, { kind: "request_copies" }); try { internalReq = new Request("http://localhost/v1/responses", { @@ -1257,6 +1295,7 @@ async function handleClaudeMessagesWithBudget( translatorBudget.chargeRetained(bodyBytes, { kind: "request_copies" }); } catch (err) { if (!isTranslatorBudgetExceededError(err)) throw err; + logTranslationBufferLimit("request", { model: requestedModel, bytes: bodyBytes, limit: (err as { limitBytes?: number }).limitBytes }); if (logIds) addFinalRequestLog(logIds.requestId, logIds.start, logCtx, 413, { closeReason: "non_stream" }); return anthropicErrorResponse(413, "request translation buffer exceeded the safe limit", "request_too_large", "translation_buffer_limit"); } @@ -1267,22 +1306,27 @@ async function handleClaudeMessagesWithBudget( // frame, so tapping it records a bogus 502 with no usage/cache detail). const finalizeNativeLog = createFinalRequestLog(logIds, logCtx).finish; // The replay view is a new object; link it so the Advisor factory registered on `config` applies. + const replayConfig = buildClaudeReplayConfig(config); const upstream = await handleResponses(internalReq, linkAdvisorPlanConfig(config, { - ...buildClaudeReplayConfig(config), + ...(agentRadarAccount ? agentRadarPinnedConfig(replayConfig, agentRadarAccount) : replayConfig), // Operator config wins; otherwise the caller's explicit TTLs (all 1h, or a pinned 5m) carry // over, and TTL-less breakpoints take the destination default. cacheRetention: config.cacheRetention ?? callerCacheRetention(anthropicBody), }), logCtx, { + ...(agentRadarAccount ? { agentRadarAccount } : {}), + ...(comboRandomSeed ? { comboRandomSeed } : {}), ...(nativeComboSource && envelope && settledRoute?.combo ? { protocolSource: (await import("./messages-native")).createNativeMessagesComboSource({ - req, config, logCtx, envelope, requestedModel, translatorBudget, selector: nativeSelector, admission: logIds?.admission, + req, config, logCtx, envelope, requestedModel, translatorBudget, admission: logIds?.admission, + // Hosted combos replay a canonical child's server tools as sent, as .27 did, rather than the sidecar. + selector: process.env.OPENCODEX_AGENTRADAR === "1" ? { ...nativeSelector, webSearchBackendAvailable: false } : nativeSelector, echoModel: clientFacingModelText(clientModelLabel), modelScope: resolveAdmissionModelScope(config, logIds?.admission), callerAnthropicBeta: req.headers.get("anthropic-beta"), clientIdentity: captureAnthropicClientIdentity(req.headers), sessionKey: anthropicSessionKeyFromParts({ sessionIdHeader: req.headers.get("session_id")?.trim() || req.headers.get("x-claude-code-session-id"), - threadIdHeader: req.headers.get("thread_id"), + threadIdHeader: req.headers.get("thread_id") ?? req.headers.get("thread-id"), clientThreadId: logCtx.conversationId, promptCacheKey: typeof internalBody.prompt_cache_key === "string" ? internalBody.prompt_cache_key : null, promptCacheKeyIsSharedCohort: cacheKeySource === "system", @@ -1525,6 +1569,7 @@ async function handleClaudeMessagesWithBudget( message = responsesJsonToAnthropicMessage(json, clientFacingModelText(clientModelLabel), translatorBudget, blob => nativeReasoningTag(nativeReasoningMint.owner, blob)); } catch (err) { if (!isTranslatorBudgetExceededError(err)) throw err; + logTranslationBufferLimit("response", { model: requestedModel, limit: (err as { limitBytes?: number }).limitBytes }); return anthropicErrorResponse(413, "upstream translation buffer exceeded the safe limit", "request_too_large", "translation_buffer_limit"); } if ((message as Rec).type === "error") { @@ -1642,6 +1687,8 @@ async function handleClaudeCountTokensInner( requestPolicy: RequestPolicyView = config, ingress: ClaudeIngressOptions = {}, ): Promise { + // The same pin contract as /v1/messages: a malformed pin, or any pin with the flag off, is refused. + if (agentRadarAccountPin(req.headers) === null) return anthropicErrorResponse(400, "Invalid AgentRadar account pin", "invalid_request_error"); const disabled = claudeInboundDisabled(config); if (disabled) return disabled; const cc = claudeCodeForIngress(config.claudeCode, ingress.claudeIntercept === true); @@ -1693,11 +1740,17 @@ async function handleClaudeCountTokensInner( raw.model = model; } captureClaudeInbound("count_tokens", raw, resolveInboundModel(model, cc), req.headers.get("anthropic-beta") ?? undefined); - if (wantsNativePassthrough(req, config, requestPolicy, model, cc)) { + // As generation: a pinned count never takes the caller-credential passthrough. + if (!agentRadarAccountPin(req.headers) && wantsNativePassthrough(req, config, requestPolicy, model, cc)) { return await anthropicNativePassthrough(req, config, { model, provider: "anthropic-native", surface: "claude" }, undefined, raw, "/v1/messages/count_tokens"); } // A thread delta would undercount; refuse it exactly as the translated Messages path does. if (carriesMessageThread(raw)) return messageThreadUnsupportedResponse(); + if (process.env.OPENCODEX_AGENTRADAR === "1") { + const exact = await (await import("./agentradar-count-tokens")).agentRadarExactCountTokens( + req, config, cc, raw, countFastRow !== null, ingress.admission); + if (exact) return exact; + } // PF-08: an eligible managed-key route counts the body the native lane would send. const nativeCountBody = (await import("./messages-native")).nativeMessagesCountBody( config, cc, raw, { fastRow: countFastRow !== null }, captureAnthropicClientIdentity(req.headers), diff --git a/src/server/inbound-body-admission.ts b/src/server/inbound-body-admission.ts index bd2c5c7b9f5..074b109ffe7 100644 --- a/src/server/inbound-body-admission.ts +++ b/src/server/inbound-body-admission.ts @@ -3,6 +3,8 @@ import { MAX_DECOMPRESSED_BODY_BYTES, resolveInboundBodyLimitBytes, } from "./request-decompress"; +import { withAgentRadarServedAccount } from "../lib/agentradar-account-pin"; +import { logOversizedInbound } from "../lib/upstream-refusal-log"; import { withCors, type RequestPolicyView } from "./auth-cors"; import { codexCompatibleUrl } from "../codex/context-compat"; import { addFinalRequestLog, markLocalRequestLogRefusal } from "./request-log"; @@ -219,7 +221,16 @@ export function runAdmittedBodyWork( mapRefusal?: (response: Response) => Response, onAdmissionError?: (error: unknown) => Response, ): Promise { - return withRaisedInboundBodyAdmission(req, codexCompatibleUrl(req.url).pathname, configuredLimit, work, refused => { + const served = async () => { + const response = await work(); + return refusalLog ? withAgentRadarServedAccount(response, refusalLog.logCtx) : response; + }; + // The declared length only, before any body is read: a request near the cap shows up here. + const pathname = codexCompatibleUrl(req.url).pathname; + if (req.method === "POST" && CONFIGURABLE_JSON_BODY_ROUTES.has(pathname)) { + logOversizedInbound(pathname, Number(req.headers.get("content-length") ?? NaN)); + } + return withRaisedInboundBodyAdmission(req, pathname, configuredLimit, served, refused => { if (refusalLog) { markLocalRequestLogRefusal(refusalLog.logCtx, "server_busy"); refusalLog.logCtx.errorCode = "server_busy"; diff --git a/src/server/index/serve-options.ts b/src/server/index/serve-options.ts index 147b682ce0c..93afb30b807 100644 --- a/src/server/index/serve-options.ts +++ b/src/server/index/serve-options.ts @@ -275,6 +275,10 @@ export function createServeOptions(ctx: ServeOptionsContext) { // #8338: the kernel-observed peer, so the configured standalone dashboard proxy can be verified. else if (ingress === "public" && config.dashboardTrustedProxy !== undefined) markDashboardProxyPeer(req, requestServer.requestIP(req)?.address); const requestUrl = codexCompatibleUrl(req.url); + if ((req.headers.has("x-agentradar-account-id") || req.headers.has("x-agentradar-account-provider")) + && !["/v1/messages", "/v1/messages/count_tokens", "/v1/chat/completions"].includes(requestUrl.pathname)) { + return formatErrorResponse(400, "invalid_request_error", "AgentRadar account pin is only honoured on /v1/messages, /v1/messages/count_tokens and /v1/chat/completions"); + } const linkPolicyView = ingress === "hub-link" ? linkPolicy() : undefined; // The unauthenticated loopback listener (#1102) serves a fixed allowlist and nothing // else. Rejecting here, before any handler runs, is what keeps the surface from growing @@ -1513,7 +1517,7 @@ export function createServeOptions(ctx: ServeOptionsContext) { return withCors(anthropicErrorResponse(403, "cross-origin data-plane request blocked", "permission_error"), req, policy); } return runAdmittedHttpTurn(req, policy, async () => withCors( - await handleClaudeCountTokens(req, config, policy, { claudeIntercept: ingress === "claude-intercept" }), + await handleClaudeCountTokens(req, config, policy, { claudeIntercept: ingress === "claude-intercept", admission }), req, policy, )); diff --git a/src/server/inference/context.ts b/src/server/inference/context.ts index d0c972252d8..658d3dc1688 100644 --- a/src/server/inference/context.ts +++ b/src/server/inference/context.ts @@ -1,6 +1,6 @@ import { CODEX_TEXT_GUARDED_BUDGET_POLICY, createRequestExecutionBudget, type RequestExecutionBudget } from "../../lib/request-execution-budget"; import { TRANSIENT_RETRY_MAX_ATTEMPTS, type TransientSendBudget } from "../../lib/upstream-retry"; -import { GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST } from "../../oauth/generic-account-failover"; +import { oauthMaxAccountsPerRequest } from "../../oauth/generic-account-failover"; import type { RequestLogContext } from "../request-log"; import { attachRequestSpendTracker } from "../responses/request-spend"; @@ -29,7 +29,7 @@ export function createInferenceSendBudget( /** * Fund each account's normal transient ladder once, before the first physical send. The account * count is clamped here as well as at the caller, so no caller can raise the ceiling past - * GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST × TRANSIENT_RETRY_MAX_ATTEMPTS. + * oauthMaxAccountsPerRequest() × TRANSIENT_RETRY_MAX_ATTEMPTS. */ export function expandInferenceOAuthSendBudget(budget: TransientSendBudget | undefined, accounts: number): void { if (!budget) return; @@ -37,7 +37,7 @@ export function expandInferenceOAuthSendBudget(budget: TransientSendBudget | und if (!policy || budget.used !== 0 || ("physicalStarted" in budget && Number(budget.physicalStarted) > 0)) return; ingressPolicies.delete(budget); if (accounts < 2 || !Number.isSafeInteger(accounts)) return; - const sends = Math.min(accounts, GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST) * TRANSIENT_RETRY_MAX_ATTEMPTS; + const sends = Math.min(accounts, oauthMaxAccountsPerRequest()) * TRANSIENT_RETRY_MAX_ATTEMPTS; policy.baseSendAllowance = Math.max(policy.baseSendAllowance, sends); policy.maxTotalModelSends = Math.max(policy.maxTotalModelSends, sends); } diff --git a/src/server/management/oauth-account-routes.ts b/src/server/management/oauth-account-routes.ts index 41ff82cfb15..d30088c6f45 100644 --- a/src/server/management/oauth-account-routes.ts +++ b/src/server/management/oauth-account-routes.ts @@ -1,5 +1,8 @@ import { hasHumanDashboardSession } from "./consent-session"; -import { isAnthropicOAuthInstance, configuredAnthropicInstance, type AnthropicInstanceId } from "../../providers/anthropic-instance"; +import { isAnthropicOAuthInstance, configuredAnthropicInstance, isAnthropicInstanceId, type AnthropicInstanceId } from "../../providers/anthropic-instance"; +import { getTokenForAccountQuotaProbe } from "../../providers/quota/account-cache"; +import { anthropicSubscriptionLabel } from "../../providers/anthropic-subscription"; +import { readAnthropicPlan, type AnthropicPlanReading } from "../../oauth/anthropic-identity"; import { resolveAnthropicAccountPoolConfig } from "../../oauth/anthropic-pool-config"; import { persistAnthropicPoolPatch, writeAnthropicPoolSettings } from "./anthropic-pool-settings"; import { parseAnthropicModelRoutes, readAnthropicModelRoutes } from "../../oauth/anthropic-model-routes"; @@ -593,9 +596,22 @@ export async function handleOauthAccountRoutes(ctx: ManagementContext): Promise< : await fetchProviderAccountQuotas(provider, forceRefresh, quotaProvider, config); const byId = new Map(rows.map(row => [row.accountId, row])); const projected = projectAccounts(); + // A plan the credential has not stored yet is read live, with the token the probe just used; + // a sign-in that cannot be used says so instead of leaving the plan blank (AgentRadar 20720). + const plans = wantQuota && isAnthropicInstanceId(provider) + ? new Map(await Promise.all(projected.accounts.filter(account => account.plan == null).map(async account => [account.id, + await getTokenForAccountQuotaProbe(provider, account.id).then(readAnthropicPlan, + (): AnthropicPlanReading => ({ plan: null, planUnknownReason: "the sign-in needs renewing" }))] as const))) + : null; return jsonResponse({ ...(provider === "anthropic2" ? { provider } : {}), activeAccountId: projected.activeAccountId, - accounts: projected.accounts.map(account => { + accounts: projected.accounts.map(planless => { + const reading = plans?.get(planless.id); + const read = reading ? { ...planless, ...reading, ...(reading.plan ? { planLabel: anthropicSubscriptionLabel(reading.plan) } : {}) } : planless; + // AgentRadar 20720 keys on the hosted vocabulary; planLabel stays canonical. + const hostedPlan = process.env.OPENCODEX_AGENTRADAR === "1" && isAnthropicInstanceId(provider) + ? { max_20x: "ultra", max_5x: "max" }[read.plan ?? ""] : undefined; + const account = hostedPlan ? { ...read, plan: hostedPlan } : read; const row = byId.get(account.id); // #8397: a passive provider with no observation for this account yet says so. if (!row) return passiveQuota ? { ...account, quotaUnknownReason: "passive_not_observed" } : account; diff --git a/src/server/management/request-history-routes.ts b/src/server/management/request-history-routes.ts index 882331510b4..9e214107c65 100644 --- a/src/server/management/request-history-routes.ts +++ b/src/server/management/request-history-routes.ts @@ -21,7 +21,7 @@ import { requestLogEntryFromPersistedUsage } from "../request-log"; import { requestLogDto } from "./shared"; import { jsonResponse } from "../auth-cors"; import type { ManagementContext } from "./context"; -import type { PersistedUsageEntry } from "../../usage/log"; +import { physicalUsageAttempts, type PersistedUsageEntry } from "../../usage/log"; function parseQueryInt(raw: string | null): number | undefined | "invalid" { if (raw === null) return undefined; @@ -32,14 +32,8 @@ function parseQueryInt(raw: string | null): number | undefined | "invalid" { } function finalAttemptTarget(entry: PersistedUsageEntry): { provider: string; model: string } { - const attempts = entry.attempts; - if (Array.isArray(attempts) && attempts.length > 0) { - const last = attempts[attempts.length - 1]; - if (last && typeof last.provider === "string" && typeof last.model === "string") { - return { provider: last.provider, model: last.model }; - } - } - return { provider: entry.provider, model: entry.model }; + const last = physicalUsageAttempts(entry.attempts ?? []).at(-1); + return last ? { provider: last.provider, model: last.model } : { provider: entry.provider, model: entry.model }; } export async function handleRequestHistoryRoutes(ctx: ManagementContext): Promise { diff --git a/src/server/management/routing-analytics-routes.ts b/src/server/management/routing-analytics-routes.ts index e0e0d0ea88f..c2c727821cf 100644 --- a/src/server/management/routing-analytics-routes.ts +++ b/src/server/management/routing-analytics-routes.ts @@ -10,6 +10,7 @@ import { ANALYTICS_MAX_ROWS, computeRoutingAnalytics, } from "../../routing/analytics"; +import { USAGE_RANGES, USAGE_SURFACES, rangeWindow, type UsageRange, type UsageSurface } from "../../usage/summary"; import { jsonResponse } from "../auth-cors"; import type { ManagementContext } from "./context"; @@ -33,7 +34,16 @@ export async function handleRoutingAnalyticsRoutes(ctx: ManagementContext): Prom if (toParsed === "invalid") { return jsonResponse({ error: { code: "invalid_to", message: "to must be an integer timestamp" } }, 400, req, config); } - const from = fromParsed; + // `range` is the dashboard's calendar shorthand, resolved through the usage surface's own helper so + // both agree on where a local day starts. Explicit `from` wins; an unknown range is refused. + const rangeRaw = url.searchParams.get("range")?.trim(); + if (rangeRaw && !(USAGE_RANGES as readonly string[]).includes(rangeRaw)) { + return jsonResponse( + { error: { code: "invalid_range", message: `range must be one of ${USAGE_RANGES.join(", ")}` } }, + 400, req, config, + ); + } + const from = fromParsed ?? (rangeRaw ? rangeWindow(rangeRaw as UsageRange, Date.now()).since ?? undefined : undefined); const to = toParsed; if (from !== undefined && to !== undefined && from > to) { return jsonResponse({ error: { code: "invalid_range", message: "from must not be after to" } }, 400, req, config); @@ -66,7 +76,9 @@ export async function handleRoutingAnalyticsRoutes(ctx: ManagementContext): Prom provider: url.searchParams.get("provider")?.trim() || undefined, model: url.searchParams.get("model")?.trim() || undefined, profileId: url.searchParams.get("profileId")?.trim() || undefined, - surface: url.searchParams.get("surface")?.trim() || undefined, + surface: (USAGE_SURFACES as readonly string[]).includes(url.searchParams.get("surface")?.trim() ?? "") + ? url.searchParams.get("surface")!.trim() as UsageSurface + : undefined, from, to, }, { maxRows }); diff --git a/src/server/management/shared.ts b/src/server/management/shared.ts index 75ba5aff01a..94e09b9f15b 100644 --- a/src/server/management/shared.ts +++ b/src/server/management/shared.ts @@ -40,7 +40,7 @@ import { DEFAULT_PROVIDER_CONTEXT_CAP, globalContextCapValue, providerContextCap import { resolveCodexHomeDir } from "../../codex/home"; import { isKnownRequestFailureCause, isKnownRequestFailureStage, readUsageEntries } from "../../usage/log"; import { getUsageDebugLogEntries } from "../../usage/debug"; -import { cacheObservationFromUsage, parseRange, parseUsageSurface, summarizeUsage } from "../../usage/summary"; +import { cacheObservationFromUsage, normalizePhysicalEntry, parseRange, parseUsageSurface, summarizeUsage } from "../../usage/summary"; import { stripCodexRuntimeProviderFields } from "../../codex/auth-context"; import { getProviderRegistryEntry, providerMatchesRegistryTransport } from "../../providers/registry"; import { getDebugLogEntries } from "../../lib/debug-log-buffer"; @@ -58,7 +58,7 @@ import type { DesktopProfileModel } from "../../claude/desktop-profile"; import { drainAndShutdown } from "../lifecycle"; import { filterRequestLogs, getRequestLogEntries, type RequestLogEntry } from "../request-log"; import { estimateComboCost, estimateRequestCost, serviceTierContext, normalizeCostTokens, tokensPerSecond } from "../../usage/cost"; -import type { PersistedUsageAttempt } from "../../usage/log"; +import type { PersistedUsageAttempt, PersistedUsageEntry } from "../../usage/log"; import { usageModelPriceOptions } from "../../usage/model-identity"; import { isAllowedRequestOrigin, jsonResponse, providerManagementConfigError, publicProviderBaseUrl, safeConfigDTO } from "../auth-cors"; import { applySystemEnvToggle } from "../system-env"; @@ -111,7 +111,7 @@ export type MetricSource = Pick): TokPerSecondResult { - if (!entry.usage) return { kind: "unavailable", reason: "usage_missing" }; + if (!entry.usage || entry.usageStatus === "unreported") return { kind: "unavailable", reason: "usage_missing" }; if (entry.usageStatus === "unsupported") return { kind: "unavailable", reason: "usage_unsupported" }; const value = tokensPerSecond(entry.usage.outputTokens, entry.durationMs); if (value === null) { @@ -187,10 +187,11 @@ export function unavailableCostReason(entry: MetricSource): MetricUnavailableRea // Normalizer-first classification: the landed normalizer recovers legacy // cachedInputTokens=read+write rows via retry, so a raw read+write>input // pre-check would misclassify recoverable rows (020 audit blocker #2). - if (!entry.usage && !entry.attempts?.length) return "usage_missing"; + if (entry.attempts !== undefined) { + return entry.attempts.length > 0 ? "combo_attempt_unavailable" : "usage_missing"; + } + if (!entry.usage || entry.usageStatus === "unreported") return "usage_missing"; if (entry.usageStatus === "unsupported") return "usage_unsupported"; - if (entry.attempts?.length) return "combo_attempt_unavailable"; - if (!entry.usage) return "usage_missing"; if (!normalizeCostTokens(entry.usage)) { const effectiveRead = entry.usage.cacheReadInputTokens ?? entry.usage.cachedInputTokens ?? 0; const effectiveWrite = entry.usage.cacheCreationInputTokens ?? 0; @@ -204,7 +205,7 @@ export function unavailableCostReason(entry: MetricSource): MetricUnavailableRea /** Display-time cost estimate for one log entry (or its attempt list), including the reasons that qualify the estimate. */ export function costResult(entry: MetricSource): CostResult { const tier = serviceTierContext(entry); - const estimate = entry.attempts?.length + const estimate = entry.attempts !== undefined ? estimateComboCost(entry.attempts.map(attempt => ({ ...attempt, ...usageModelPriceOptions(entry, attempt) })), undefined, tier) : estimateRequestCost({ provider: entry.provider, model: entry.model, usage: entry.usage, usageStatus: entry.usageStatus, serviceTier: tier, ...usageModelPriceOptions(entry, entry) }); if (!estimate) return { kind: "unavailable", reason: unavailableCostReason(entry) }; @@ -252,16 +253,18 @@ export function requestLogDto( entry: RequestLogEntry, { includeDecodeRate = true }: { includeDecodeRate?: boolean } = {}, ): Record { + // Rate and cost read the physical-attempt projection, so hidden sends cannot inflate either. + const metricEntry = normalizePhysicalEntry(entry as PersistedUsageEntry) as RequestLogEntry; return { ...entry, ...(entry.attemptNormalizationDropped === true ? { attemptNormalizationDropped: true } : {}), ...resendVerdict(entry), displayMetrics: { - tokPerSecond: tokPerSecondResult(entry), + tokPerSecond: tokPerSecondResult(metricEntry), // The parent uses the REQUEST's own TTFT. A combo parent must not borrow an attempt's, // which would measure a window the parent never had. - ...(includeDecodeRate ? { decodeTokPerSecond: decodeTokPerSecondResult(entry) } : {}), - cost: costResult(entry), + ...(includeDecodeRate ? { decodeTokPerSecond: decodeTokPerSecondResult(metricEntry) } : {}), + cost: costResult(metricEntry), }, ...(entry.attempts?.length ? { diff --git a/src/server/messages-native-eligibility.ts b/src/server/messages-native-eligibility.ts index 79fc3fb56fe..0bc7f6330c4 100644 --- a/src/server/messages-native-eligibility.ts +++ b/src/server/messages-native-eligibility.ts @@ -5,11 +5,13 @@ * `count_tokens` and the protocol planner (`src/protocols/plan-snapshot.ts`) all ask here; * previews and counts remain conservative when runtime availability is unknown. */ +import { advisorServerToolDestination, isAdvisorServerToolType } from "../adapters/anthropic/advisor-tool"; import { anthropicBodyElidesBlockedSkill } from "../claude/inbound"; import { isClaudeWebSearchToolName } from "../claude/outbound"; import type { ProtocolReasonCode } from "../protocols/contract"; import { featuresFromMessagesBody } from "../protocols/features"; import { credentialDomainFor } from "../protocols/opaque-state"; +import { isCanonicalAnthropicMessagesUrl } from "../adapters/anthropic/passthrough"; import { resolveProtocolSettings } from "../protocols/settings"; import type { RouteResult } from "../router"; import type { OcxConfig } from "../types"; @@ -90,9 +92,18 @@ function bridgeOnlyPolicyApplies( ): boolean { if (resolvePinnedEffort(route, selector.routeSelector, config) !== undefined) return true; if (anthropicBodyElidesBlockedSkill(body, selector.claudeCode ?? config.claudeCode)) return true; + // An operator who keeps the Advisor off this provider is obeyed by the bridge; exact replay would send it. + if (hostedCanonical(route) && !advisorServerToolDestination(route.provider) && Array.isArray(body.tools) + && body.tools.some(tool => isRec(tool) && isAdvisorServerToolType(tool.type))) return true; return selector.webSearchBackendAvailable !== false && webSearchSidecarMayEngage(body, config); } +/** AgentRadar hosted gateways send canonical Anthropic routes natively whatever the rollout says (.27 source replay). */ +function hostedCanonical(route: RouteResult): boolean { + return process.env.OPENCODEX_AGENTRADAR === "1" && route.provider.adapter === "anthropic" + && isCanonicalAnthropicMessagesUrl(route.provider); +} + /** * The credential rule. A proxy-managed key is native since PF-08. An Anthropic OAuth account * is native only with `managedMessagesNativeOAuth` on (itself effective only with @@ -108,7 +119,7 @@ function credentialDeclineReason( const provider = route.provider; if (provider.authMode === undefined || provider.authMode === "key") return undefined; if (provider.authMode !== "oauth") return "auth-mode-not-native"; - if (!resolveProtocolSettings(config, route.providerName).rollout.managedMessagesNativeOAuth) return "auth-mode-not-native"; + if (!hostedCanonical(route) && !resolveProtocolSettings(config, route.providerName).rollout.managedMessagesNativeOAuth) return "auth-mode-not-native"; if (!configuredAnthropicInstance(config, route.providerName)) return "auth-mode-not-native"; if (!credentialDomainFor(provider)?.firstPartyAnthropic) return "auth-mode-not-native"; return undefined; @@ -134,7 +145,7 @@ export function nativeMessagesDeclineReason( config: OcxConfig, selector: NativeMessagesSelector = {}, ): NativeMessagesDeclineReason | undefined { - if (!resolveProtocolSettings(config, route.providerName).rollout.managedMessagesNative) return "rollout-disabled"; + if (!hostedCanonical(route) && !resolveProtocolSettings(config, route.providerName).rollout.managedMessagesNative) return "rollout-disabled"; const provider = route.provider; if (provider.adapter !== "anthropic") return "cross-wire-ir"; const credentialDecline = credentialDeclineReason(route, config, selector); diff --git a/src/server/messages-native-oauth.ts b/src/server/messages-native-oauth.ts index a0cce90e8e4..57ed29f962b 100644 --- a/src/server/messages-native-oauth.ts +++ b/src/server/messages-native-oauth.ts @@ -33,6 +33,8 @@ import { import type { TranslatorBudget } from "../lib/translator-budget"; import type { OcxConfig } from "../types"; import { relaySseWithPayloadRewrite } from "./sse-payload-rewrite"; +import { agentRadarPin } from "../lib/agentradar-account-pin"; +import { OAuthLoginRequiredError } from "../oauth"; const MAX_SELECTION_ATTEMPTS = 3; @@ -57,6 +59,8 @@ export interface NativeOAuthBinding { readonly model?: string; readonly config: OcxConfig; readonly manualSelectionGeneration: number; + /** A count resolves a usable credential without committing the active selection or affinity. */ + readonly readOnly?: true; } /** The selection moved or became ineligible while resolving. Maps to a 409 retry. */ @@ -114,6 +118,8 @@ export interface NativeOAuthBindingOptions { /** Selection captured before asynchronous refusal classification or throttle wait. */ expectedRecoverySelection?: Selection | null; expectedRecoveryRouteDecision?: AnthropicRouteDecision | null; + /** Select exactly as a dispatch would but commit nothing (AgentRadar exact count_tokens). */ + readOnly?: boolean; } /** Compatibility entrypoint: the original instance only. */ @@ -134,6 +140,28 @@ export async function resolveNativeOAuthBindingForInstance( }; const sessionKey = options.sessionKey ?? null; const model = options.model; + const pin = agentRadarPin(config); + if (pin) { + // A trusted AgentRadar pin serves exactly its account: no routing, no global selection commit. + if (pin.provider !== "claude" || instance !== "anthropic") throw new OAuthLoginRequiredError(instance); + const routeDecision = currentRoute(instance, config, model); + const usable = () => { + const row = getAccountCredentialWithStatus(instance, pin.accountId); + if (!row || row.paused || row.needsReauth + || !routeCandidates(routing.getEligibleAnthropicAccounts(Date.now(), model), routeDecision).includes(pin.accountId)) { + throw new OAuthLoginRequiredError(instance); + } + return row; + }; + usable(); + const snapshot = await routing.getAnthropicPoolAccessSnapshot(pin.accountId); + assertOwner(); + const binding: NativeOAuthBinding = { instance, routeTarget, selection: { accountId: pin.accountId, revision: "agentradar-request" }, + snapshot, providerAccountUuid: usable().credential.accountId, routeDecision, sessionKey, model, config, + manualSelectionGeneration: routing.captureAnthropicManualSelectionGeneration() }; + if (nativeOAuthBindingIsCurrent(binding)) return binding; + throw new NativeOAuthSelectionChangedError(); + } for (let attempt = 0; attempt < MAX_SELECTION_ATTEMPTS; attempt++) { assertOwner(); const routeDecision = currentRoute(instance, config, model); @@ -161,6 +189,10 @@ export async function resolveNativeOAuthBindingForInstance( assertOwner(); if (candidate.provider !== instance) throw new NativeOAuthSelectionChangedError(); if (!routeIsCurrent(instance, config, model, routeDecision)) continue; + if (options.readOnly) { + return { instance, routeTarget, selection, snapshot: candidate, routeDecision, sessionKey, model, config, manualSelectionGeneration, readOnly: true, + providerAccountUuid: getAccountCredentialWithStatus(instance, candidate.accountId)?.credential.accountId }; + } const committed = await commitOAuthAccountSelection(instance, candidate.accountId, { expectedSelection: selection, expectedCredentialGeneration: candidate.generation, @@ -195,15 +227,18 @@ export function nativeOAuthBindingIsCurrent(binding: NativeOAuthBinding): boolea if (nativeOAuthRouteTarget(instance, binding.config) !== binding.routeTarget) return false; } catch { return false; } if (binding.snapshot.provider !== instance) return false; - if (binding.manualSelectionGeneration !== routing.captureAnthropicManualSelectionGeneration() + const pin = agentRadarPin(binding.config); + // A pinned binding never followed the shared selection, so a manual switch does not revoke it. + if ((!pin && binding.manualSelectionGeneration !== routing.captureAnthropicManualSelectionGeneration()) || !routeIsCurrent(instance, binding.config, binding.model, binding.routeDecision)) return false; const selected = captureOAuthAccountSelection(instance); const row = getAccountCredentialWithStatus(instance, binding.snapshot.accountId); - const sessionRoute = routing.isAnthropicAccountPoolEnabled(binding.config) && binding.sessionKey + const sessionRoute = !pin && routing.isAnthropicAccountPoolEnabled(binding.config) && binding.sessionKey ? routing.resolveAnthropicAccountForSession(binding.sessionKey, binding.config, Date.now(), binding.routeDecision, binding.model) : null; // Another conversation may move the automatic active pointer without revoking this // session's affinity. Manual selection clears affinity and precedes it in the selector. - const selectionCurrent = sessionRoute?.reason === "affinity" + const selectionCurrent = pin ? binding.snapshot.accountId === pin.accountId + : binding.readOnly || (sessionRoute?.reason === "affinity" || sessionRoute?.reason === "hash") && sessionRoute.accountId === binding.snapshot.accountId || (selected?.accountId === binding.selection.accountId && selected?.revision === binding.selection.revision); return selectionCurrent diff --git a/src/server/messages-native-preflight.ts b/src/server/messages-native-preflight.ts index 524a609be4c..74b7102476f 100644 --- a/src/server/messages-native-preflight.ts +++ b/src/server/messages-native-preflight.ts @@ -8,14 +8,19 @@ import { sseFieldValue } from "../lib/sse-decoder"; import type { TranslatorBudget } from "../lib/translator-budget"; import { anthropicUsageToOcx } from "./messages-shared"; import type { RequestLogContext } from "./request-log"; -import { BoundedSseFrameBuffer, MAX_CLIENT_SSE_FRAME_BYTES, SseFrameCountLimitError, SseFrameTooLargeError } from "./sse-frame-buffer"; +import { BoundedSseFrameBuffer, joinSseFrameBytes, MAX_CLIENT_SSE_FRAME_BYTES, SseFrameCountLimitError, SseFrameTooLargeError } from "./sse-frame-buffer"; +import { markVerbatimComboFailure } from "../bridge/errors"; +import { markClientWire } from "./inference/client-wire"; import { UNDECLARED_TOOL_CALL_ERROR_CODE } from "./responses-undeclared-tool-guard"; type Rec = Record; const isRec = (value: unknown): value is Rec => value !== null && typeof value === "object" && !Array.isArray(value); /** Capture replay evidence before any relay can split a physical upstream read into frames. */ -export function observeNativeMessagesReplayBoundary(body: ReadableStream, logCtx: RequestLogContext, replay: { committed: boolean; pending: boolean }): ReadableStream { +export function observeNativeMessagesReplayBoundary( + body: ReadableStream, logCtx: RequestLogContext, replay: { committed: boolean; pending: boolean }, + hosted = process.env.OPENCODEX_AGENTRADAR === "1", +): ReadableStream { const reader = body.getReader(); const inspector = new BoundedSseFrameBuffer(); const decoder = new TextDecoder(); @@ -28,6 +33,7 @@ export function observeNativeMessagesReplayBoundary(body: ReadableStream 0) && type !== "ping") logCtx.streamErrorFollowed = true; + // Hosted strict relay stops at a malformed frame: neither it nor anything after it reaches the client. + if (hosted && dataFields.length > 0 && !isRec(payload)) { malformed = true; break; } if (type === "error" && isRec(payload) && isRec(payload.error)) errorSeen = true; else if ((event !== null || dataFields.length > 0) && type !== "ping" && !(type === "message_start" && isRec(payload) && isRec(payload.message) && Array.isArray(payload.message.content) && payload.message.content.length === 0)) { replay.committed = true; finished = true; } } - replay.pending = inspector.pendingBytes > 0; + replay.pending = !malformed && inspector.pendingBytes > 0; // An unfinished event can still acquire a different name or payload on the next read. if (errorSeen && inspector.pendingBytes > 0) logCtx.streamErrorFollowed = true; - if (errorSeen || finished) { active = false; inspector.dispose(); } + if (errorSeen || finished || malformed) { active = false; inspector.dispose(); } } catch (error) { if (!(error instanceof SseFrameTooLargeError || error instanceof SseFrameCountLimitError)) throw error; logCtx.streamErrorFollowed = true; @@ -65,6 +73,8 @@ export function observeNativeMessagesReplayBoundary(body: ReadableStream { if (!response.ok || !response.body || !response.headers.get("content-type")?.toLowerCase().includes("text/event-stream")) { return { kind: "accepted", response }; @@ -106,11 +116,13 @@ export async function preflightNativeMessagesResponse( // Otherwise a structured moderation verdict on the whole event is a request-level refusal, // whatever status the terminal mapping infers for its code; judge it like the HTTP reader. const moderation = framedAllowsModeration && isProviderModerationRefusal(400, wholeEvent, undefined, true); - const mapped = httpStatusFromTerminalError({ type: typeof error.type === "string" ? error.type : undefined, + // Hosted gateways record Anthropic's own overloaded status (529), as .27 did. + const mapped = hosted && error.type === "overloaded_error" ? 529 : httpStatusFromTerminalError({ type: typeof error.type === "string" ? error.type : undefined, code: typeof error.code === "string" ? error.code : undefined, message: typeof error.message === "string" ? error.message : undefined }) ?? 502; const status = moderation && mapped !== 400 && mapped !== 422 ? 400 : mapped; - if (logCtx.terminalSource !== "synthetic" && logCtx.errorCode !== UNDECLARED_TOOL_CALL_ERROR_CODE + // Hosted .27 contract: a relay-made pre-output failure (malformed frame, reset) also fails over. + if ((logCtx.terminalSource !== "synthetic" || process.env.OPENCODEX_AGENTRADAR === "1") && logCtx.errorCode !== UNDECLARED_TOOL_CALL_ERROR_CODE && comboFailureDecision(status, typeof error.message === "string" ? error.message : "", { code: typeof error.code === "string" ? error.code : undefined, rootCode: typeof payload.code === "string" ? payload.code : undefined, @@ -161,11 +173,16 @@ export async function preflightNativeMessagesResponse( release(); const headers = new Headers(response.headers); headers.set("content-type", "application/json"); headers.delete("content-length"); headers.delete("content-encoding"); + const failedBytes = chunks.length === 1 ? chunks[0]! : joinSseFrameBytes(chunks); const failed = Response.json({ error: { ...(typeof failure.type === "string" ? { type: failure.type } : {}), ...(typeof failure.code === "string" ? { code: failure.code } : {}), message, } }, { status: failureStatus, headers }); + // AgentRadar hosted gateways (.27): if this ends the combo, the client gets Anthropic's own stream bytes. + if (hosted) { + markVerbatimComboFailure(failed, () => markClientWire(new Response(failedBytes.slice(), { status: response.status, headers: response.headers }), "messages")); + } // The rebuilt envelope drops the evidence (root code, innererror) the verdict was judged on. return { kind: "failed", response: failureEvidence === undefined ? failed : attachComboFailureEvidence(failed, failureEvidence) }; } diff --git a/src/server/messages-native.ts b/src/server/messages-native.ts index 757045272f0..d519efc34e2 100644 --- a/src/server/messages-native.ts +++ b/src/server/messages-native.ts @@ -7,7 +7,7 @@ import { attachComboFailureEvidence, isProxyError, markProxyError } from "../lib import { captureAccountAutoPauseAttempt, wasAccountAutoPaused } from "../oauth/account-auto-pause"; import { createPhysicalSendReporter, reportDispatchSends, type RequestExecutionBudget, type SingleUseDispatchPermit } from "../lib/request-execution-budget"; import { createInferenceSendBudget, expandInferenceOAuthSendBudget } from "./inference/context"; -import { GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST } from "../oauth/generic-account-failover"; +import { oauthMaxAccountsPerRequest } from "../oauth/generic-account-failover"; /** * Managed native Messages lane (PF-08, behind `protocols.rollout.managedMessagesNative`). * @@ -39,16 +39,18 @@ import { ANTHROPIC_POOL_MAX_FAILOVERS_PER_REQUEST } from "../oauth/anthropic-rou import { enforceAnthropicImageLimits } from "../adapters/anthropic-image-guard"; import { normalizeAnthropicImages } from "../adapters/anthropic-image-normalize"; import { formatAnthropicErrorBody, resolveAnthropicMessagesUrl } from "../adapters/anthropic"; +import { anthropicPromptCacheTtlMs } from "../adapters/anthropic/cache-retention"; import { anthropicMessagesNativeWireBody, buildAnthropicMessagesPassthroughRequest, + isCanonicalAnthropicMessagesUrl, type AnthropicMessagesPassthroughRequest, } from "../adapters/anthropic/passthrough"; import { effortFromOutputConfig, nativeAnthropicProjection, resolveInboundModel } from "../claude/inbound"; import { anthropicErrorBody, claudeOverflowSsePayload, claudePromptTooLongMessage, collectAnthropicMessage, isContextOverflowText, isThroughputLimitText } from "../claude/outbound"; import type { AdmissionLease } from "../lib/admission"; -import { readBoundedResponseBody } from "../lib/bounded-body"; -import { classifyError } from "../lib/errors"; +import { readBoundedResponseBody, readBoundedResponseBytes } from "../lib/bounded-body"; +import { classifyError, httpStatusFromTerminalError } from "../lib/errors"; import { relaySseWithPayloadRewrite } from "./sse-payload-rewrite"; import { redactSecretString } from "../lib/redact"; import { formatClientRetryAfter, resolveClientRetryAfter } from "../lib/retry-after"; @@ -89,7 +91,7 @@ import { import { stampApiKeyAccountLabel } from "../providers/label"; import { affinityLaneFromRequest } from "./request-log-conversation"; import { publicOAuthErrorPresentation, OAuthAccountPausedError, OAuthLoginRequiredError, publicOAuthAuthenticationErrorMessage } from "../oauth"; -import { AnthropicAccountCooldownError, formatAnthropicProviderForLog, anthropicRoutingFor } from "../oauth/anthropic-routing"; +import { AnthropicAccountCooldownError, anthropicRoutingFor, stampAnthropicAccountForLog } from "../oauth/anthropic-routing"; import { configuredAnthropicInstance } from "../providers/anthropic-instance"; import { resolveProtocolSettings } from "../protocols/settings"; import { addProtocolEntryReason, markProtocolBlocked } from "../protocols/trace"; @@ -136,12 +138,15 @@ import { recordKeyWireAttemptUsage, type RequestLogContext, } from "./request-log"; +import { noteAgentRadarAccountDispatch } from "../lib/agentradar-account-pin"; import { fetchWithHeaderTimeout, physicalSendObserverInit, providerFetch, safeHostLabel, sendWithConnectionPolicy } from "./responses/fetch-helpers"; import { isLocalUpstream } from "../lib/local-upstream"; import { resolveConnectTimeoutMs, resolveJsonBodyReadTimeoutMs } from "../stall-timeout"; import { anthropicBillingPhysicalFetch } from "../providers/anthropic-billing-ownership"; import { rotateAnthropicBillingKeyOnResponse } from "../providers/anthropic-billing-recovery"; import { linkAbortSignal } from "./responses/core-lifetime"; +import { markVerbatimComboFailure } from "../bridge/errors"; +import { markClientWire } from "./inference/client-wire"; import { attachRequestSpendTracker } from "./responses/request-spend"; import { carryWorkflowRefusal, workflowRefusalResponse, workflowSendCeilingRefusalResponse } from "./workflow-refusal"; import { sseFieldValue } from "../lib/sse-decoder"; @@ -363,6 +368,30 @@ function messageAsSse(message: Rec): string { return frames.join(""); } +/** A complete Messages result, as the hosted .27 contract validated it before relaying the bytes. */ +function isAnthropicMessageResponse(value: unknown): value is Rec { + return isRec(value) && typeof value.id === "string" && value.type === "message" && value.role === "assistant" + && Array.isArray(value.content) && value.content.every(isRec) && typeof value.model === "string" + && (typeof value.stop_reason === "string" || value.stop_reason === null) + // Looser than .27 by design: an absent stop_sequence is not evidence of a cut body. + && (typeof value.stop_sequence === "string" || value.stop_sequence === null || value.stop_sequence === undefined) + && isRec(value.usage) && [value.usage.input_tokens, value.usage.output_tokens] + .every(tokens => typeof tokens === "number" && Number.isInteger(tokens) && tokens >= 0); +} + +/** + * The upstream headers a hosted relay carries, as .27 did: Content-Type and a bounded Retry-After. + * Organization ids, request ids and pool rate-limit state stay with the gateway. + */ +function relayedHeaders(headers: Headers): Headers { + const out = new Headers(); + const contentType = headers.get("content-type"); + if (contentType) out.set("content-type", contentType); + const retryAfter = formatClientRetryAfter(headers.get("retry-after")); + if (retryAfter !== undefined) out.set("retry-after", retryAfter); + return out; +} + /** Reject a body the Messages API cannot carry before anything is sent. */ async function prepareNativeBody(body: Rec, signal: AbortSignal): Promise { if (!Array.isArray(body.messages)) return; @@ -564,14 +593,14 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp } let activeProvider: OcxProviderConfig = oauthBinding ? oauthProvider(oauthBinding) : route.provider; stampApiKeyAccountLabel(logCtx, route.providerName, activeProvider); - if (oauthBinding) logCtx.provider = formatAnthropicProviderForLog(route.providerName, oauthBinding.snapshot.accountId, config); + if (oauthBinding) stampAnthropicAccountForLog(logCtx, route.providerName, oauthBinding.snapshot.accountId, config); const spendTracker = options.sendBudget ? undefined : attachRequestSpendTracker(req, logCtx); const physicalBudget = options.sendBudget ?? createInferenceSendBudget(req, logCtx); // Freeze funding before any refusal cools an account; selection still checks live eligibility. const fundedAccounts = oauthBinding ? Math.min(new Set([ oauthBinding.snapshot.accountId, ...anthropicRoutingFor(oauthBinding.instance).getEligibleAnthropicAccounts(Date.now(), route.modelId), - ]).size, GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST) : 0; + ]).size, oauthMaxAccountsPerRequest()) : 0; const oauthFailoverLimit = Math.max(ANTHROPIC_POOL_MAX_FAILOVERS_PER_REQUEST, fundedAccounts - 1); if (oauthBinding) expandInferenceOAuthSendBudget(physicalBudget, fundedAccounts); // Observe-only spend is already booked by the tracker; count billing sends without booking again. @@ -593,6 +622,9 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp // credential, header or stripped block from an earlier build reaches the next one, and a build // for another credential domain decides opaque state from the full source again. const rejectUnrepresentable = resolveProtocolSettings(config).unrepresentable === "reject"; + // AgentRadar hosted gateways keep the .27 source-replay contract for canonical Anthropic only: + // caller bytes out, upstream bytes back. Other Messages-compatible destinations keep upstream's lane. + const hosted = process.env.OPENCODEX_AGENTRADAR === "1" && isCanonicalAnthropicMessagesUrl(route.provider); const buildActiveRequest = () => { recordAttemptCredentialSource(attempt, route.providerName, activeProvider, "anthropic"); // Exact combo replay preserves TTL-less breakpoints; an explicit operator TTL still wins. @@ -602,6 +634,7 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp providerAccountUuid: oauthBinding?.providerAccountUuid, clientIdentity: options.clientIdentity, sessionKey: oauthBinding?.sessionKey ?? options.sessionKey, + exactSource: hosted, }); if (built.strippedOpaqueState && rejectUnrepresentable) throw new NativeOpaqueStateRefusal(); if (built.droppedBetas) { @@ -609,6 +642,10 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp logDroppedAnthropicBetas({ droppedNames: built.droppedBetaNames, droppedUnrecognized: built.droppedUnrecognizedBetas }); } if (built.strippedOpaqueState) addProtocolEntryReason(logCtx, "opaque-state-stripped"); + // The wire's effective cache lifetime, as the adapter stamps it, for cache-break attribution. + delete attempt.promptCacheTtlMs; + const promptCacheTtlMs = anthropicPromptCacheTtlMs(built.wireBody); + if (promptCacheTtlMs !== undefined) attempt.promptCacheTtlMs = promptCacheTtlMs; return built; }; const rebuildFor = (provider: OcxProviderConfig) => { @@ -673,7 +710,7 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp record?.(); }; /** Send the native Messages request under the transient retry policy, re-validating the selected OAuth account or API key at the dispatch boundary. */ - const send = async (recovery?: "rate-limit-429" | "oauth-401" | "oauth-account-403" | "key-429" | "key-401"): Promise => { + const send = async (recovery?: "rate-limit-429" | "anthropic-oauth-429" | "oauth-401" | "oauth-account-403" | "key-429" | "key-401" | "image-413"): Promise => { const remaining = remainingTransientSends(); if (oauthBinding && !oauthPhysicalSendAvailable()) throw new SendBudgetExhaustedError(); if (options.sendBudget && remaining <= 0) throw new SendBudgetExhaustedError(); @@ -714,7 +751,7 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp throw new NativeOAuthSelectionChangedError(); } rebuildFor(oauthProvider(oauthBinding)); - logCtx.provider = formatAnthropicProviderForLog(route.providerName, oauthBinding.snapshot.accountId, config); + stampAnthropicAccountForLog(logCtx, route.providerName, oauthBinding.snapshot.accountId, config); } } else { for (let attempt = 0; !providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider); attempt++) { @@ -791,7 +828,7 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp localUpstream = isLocalUpstream(wire.url); dispatched = await fetchWithHeaderTimeout( wire.url, - applyUpstreamRecoveryInit({ ...init, method: "POST", headers, body: wire.body, ...physicalSendObserverInit(observeAutoPause, wire.url) }, transportRecovery), + applyUpstreamRecoveryInit({ ...init, method: "POST", headers, body: wire.body, ...physicalSendObserverInit(observeAutoPause, wire.url, snapshot ? () => noteAgentRadarAccountDispatch(logCtx, snapshot.accountId, "native", options.sessionKey) : undefined) }, transportRecovery), init.signal ?? upstream.signal, resolveConnectTimeoutMs(config.connectTimeoutMs, { localUpstream }), requestedStream, physicalFetch, @@ -847,6 +884,15 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp let response: Response; try { response = await send(); + // A 413 on a turn carrying images is rebuilt once with every image one tier smaller, as the + // bridge's adapter retries (imageTierBias); only the images change. + if (response.status === 413 && Array.isArray(body.messages) && JSON.stringify(body.messages).includes('"type":"image"')) { + discard(response); + await normalizeAnthropicImages(body.messages, { tierBias: 1, abortSignal: req.signal }); + enforceAnthropicImageLimits(body.messages); + rebuildFor(activeProvider); + response = await send("image-413"); + } let retryPolicy: ReturnType | undefined; let retries = 0; let billingRecoveryActive = false; @@ -936,7 +982,7 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp const oauthRetryKey = {}; const triedAccountIds = new Set(); let oauthFailovers = 0; - while (oauthBinding && (response.status === 429 || response.status === 403 || response.status === 401)) { + while (oauthBinding && [429, 403, 401, 400].includes(response.status)) { const sendingBinding = oauthBinding; const expectedRecoverySelection = sendingBinding.selection; triedAccountIds.add(sendingBinding.snapshot.accountId); @@ -954,7 +1000,9 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp canRetry: transientSendAvailable() && oauthFailovers < oauthFailoverLimit && oauthPhysicalSendAvailable(), }); if (!nextAccountId) break; - const recovery = response.status === 401 ? "oauth-401" : response.status === 403 ? "oauth-account-403" : "rate-limit-429"; + // Hosted gateways label a pool 429 hop as the bridge's OAuth rotation does (.27, adapter-dispatch). + const recovery = response.status === 401 ? "oauth-401" : response.status === 403 ? "oauth-account-403" + : hosted ? "anthropic-oauth-429" : "rate-limit-429"; discard(response); oauthBinding = await resolveNativeOAuthBindingForInstance(nativeInstance!, config, { routeTarget, sessionKey: options.sessionKey, model: route.modelId, candidateAccountId: nextAccountId, @@ -962,7 +1010,7 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp }); oauthFailovers++; rebuildFor(oauthProvider(oauthBinding)); - logCtx.provider = formatAnthropicProviderForLog(route.providerName, oauthBinding.snapshot.accountId, config); + stampAnthropicAccountForLog(logCtx, route.providerName, oauthBinding.snapshot.accountId, config); response = await send(recovery); } } catch (error) { @@ -1018,6 +1066,26 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp logCtx.errorCode = "usage_limit_reached"; return retainUpstreamMessagesRequestId(localOAuthFailure(poolUsageLimit)!, response.headers); } + if (!response.ok && hosted && !isReplayRefusalResponse(response)) { + // Hosted .27 contract: the client (and a combo, which judges and may return it) gets Anthropic's + // own error status, headers and bytes. + let read: Awaited>; + try { read = await readBoundedResponseBytes(response, { signal: upstream.signal, maxBytes: MAX_NATIVE_MESSAGES_JSON_BYTES }); } + catch (error) { + cleanupAbort(); upstream.abort(); + if (req.signal.aborted) return fail(499, "Client cancelled request", "api_error"); + return failEvidence(502, error instanceof Error ? error.message : String(error), "api_error", "transport_failed"); + } + cleanupAbort(); + if (read.oversized) return fail(502, "upstream response exceeded the safe limit", "api_error", "translation_buffer_limit"); + const text = new TextDecoder().decode(read.bytes.subarray(0, MAX_NATIVE_MESSAGES_ERROR_BYTES)); + finishLog(response.status, text); + const init = { status: response.status, statusText: response.statusText, headers: relayedHeaders(response.headers) }; + const relayed = attachComboFailureEvidence(new Response(read.bytes, init), text); + // A combo rebuilds a consumed child failure; this keeps Anthropic's bytes if it ends the ladder. + return options.sendBudget + ? markVerbatimComboFailure(relayed, () => markClientWire(new Response(read.bytes.slice(), init), "messages")) : relayed; + } if (!response.ok) { let bodyText = ""; try { @@ -1033,21 +1101,29 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp } const contentType = response.headers.get("content-type")?.toLowerCase() ?? ""; + // Hosted .27 contract: each caller transport gets its own; the other is an invalid answer. + if (hosted && !requestedStream && !contentType.includes("application/json")) { + discard(response); + cleanupAbort(); + return fail(502, "canonical Anthropic response was not valid JSON", "api_error"); + } if (contentType.includes("text/event-stream") && response.body) { // `upstream` follows req.signal and is also what shutdown and turn release abort, so both // the client leaving and this lane's own abort read as a cancel, not as a failed stream. const bodyGuard = { ...resolvePassthroughBodyGuard(config, upstream.signal), ...(activeProvider.anthropicEofTolerance === true ? { eofTolerant: true } : {}), + // Only a canonical Anthropic relay keeps the .27 strict contract; other destinations keep upstream's. + strictRelay: hosted, }; const replay = { committed: false, pending: false }; - const raw = options.sendBudget ? observeNativeMessagesReplayBoundary(response.body, logCtx, replay) : response.body; + const raw = options.sendBudget ? observeNativeMessagesReplayBoundary(response.body, logCtx, replay, hosted) : response.body; const observed = logIds && onFirstOutput ? observeFirstChunk(raw, onFirstOutput) : raw; const renamed = activeRequest.oauthToolNames ? restoreOAuthToolNamesInSse(observed, activeRequest.oauthToolNames, translatorBudget) : observed; const frames = new MessagesSseFrameContext(); - const echoed = echoRequestedModel(renamed, options.echoModel ?? requestedModel, frames); + const echoed = hosted ? renamed : echoRequestedModel(renamed, options.echoModel ?? requestedModel, frames); let toolRefusal: string | undefined; const source = clientDeclaredEmptyToolCatalog ? guardNativeMessagesEmptyToolCatalog(echoed, translatorBudget, message => { @@ -1069,19 +1145,20 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp releaseStreamTurn(); if (toolRefusal || meta.closeReason !== "terminal") upstream.abort(); } - }, bodyGuard, frames, nativeInstance ? clientFilter : undefined); + }, bodyGuard, frames, nativeInstance && !hosted ? clientFilter : undefined); // A configured Messages provider words an oversized-input refusal its own way; Claude Code // compacts only on Anthropic's wording (devlog/_plan/261009_claude_1m_default/010). The // rewrite buffers whole frames, so it sits after the tap: stall detection keeps timing raw // upstream bytes. Real Anthropic pools refuse pre-stream in that wording and keep one relay. - const clientStream = nativeInstance ? relayed : relaySseWithPayloadRewrite(relayed, claudeOverflowSsePayload, translatorBudget, messagesSseFrameHooks(frames), clientFilter); + const clientStream = nativeInstance || hosted ? relayed : relaySseWithPayloadRewrite(relayed, claudeOverflowSsePayload, translatorBudget, messagesSseFrameHooks(frames), clientFilter); let delivered = retainUpstreamMessagesRequestId(new Response(clientStream, { status: 200, - headers: { "Content-Type": "text/event-stream; charset=utf-8", "Cache-Control": "no-cache", Connection: "keep-alive" }, + headers: hosted ? { "Content-Type": response.headers.get("content-type") ?? "text/event-stream", "Cache-Control": "no-cache", Connection: "keep-alive" } + : { "Content-Type": "text/event-stream; charset=utf-8", "Cache-Control": "no-cache", Connection: "keep-alive" }, }), response.headers); if (options.sendBudget) { try { - const preflight = await preflightNativeMessagesResponse(delivered, logCtx, translatorBudget, replay); + const preflight = await preflightNativeMessagesResponse(delivered, logCtx, translatorBudget, replay, hosted); if (preflight.kind === "failed") { bindUsage(logCtx.usage); return preflight.response; } delivered = new Response(watchFirstOutput(preflight.response.body!), { status: preflight.response.status, headers: preflight.response.headers }); } catch (error) { @@ -1103,7 +1180,7 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp const collectorVerdictSink: MessagesFailureVerdictSink = {}; try { if (options.sendBudget) { - const preflight = await preflightNativeMessagesResponse(new Response(tapped, { headers: response.headers }), logCtx, translatorBudget, replay); + const preflight = await preflightNativeMessagesResponse(new Response(tapped, { headers: response.headers }), logCtx, translatorBudget, replay, hosted); if (preflight.kind === "failed") { cleanupAbort(); upstream.abort(); bindUsage(logCtx.usage); finishLog(preflight.response.status, logCtx.upstreamError); @@ -1176,31 +1253,61 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp } } + // Hosted .27 contract: a streaming caller is answered in SSE or not at all; a JSON answer must be + // exact UTF-8 and a complete Messages result (or an Anthropic error, relayed with its status). + const invalidJson = "canonical Anthropic response was not valid JSON"; + if (hosted && requestedStream) { + discard(response); + cleanupAbort(); + return fail(502, "canonical Anthropic returned JSON for a streaming request", "api_error"); + } let read; try { const jsonBodyReadMs = resolveJsonBodyReadTimeoutMs(config.stallTimeoutSec, { localUpstream }, config.connectTimeoutMs); read = await readBoundedResponseBody(response, { signal: upstream.signal, - maxBytes: MAX_NATIVE_MESSAGES_JSON_BYTES, + maxBytes: hosted ? resolvePassthroughBodyGuard(config, upstream.signal).maxBytes || MAX_NATIVE_MESSAGES_JSON_BYTES : MAX_NATIVE_MESSAGES_JSON_BYTES, totalTimeoutMs: localUpstream ? 0 : jsonBodyReadMs, inactivityTimeoutMs: jsonBodyReadMs, + fatalUtf8: hosted, }); } catch (error) { cleanupAbort(); upstream.abort(); if (req.signal.aborted) return fail(499, "Client cancelled request", "api_error"); + if (hosted) return fail(502, invalidJson, "api_error"); return failEvidence(502, error instanceof Error ? error.message : String(error), "api_error", "transport_failed"); } cleanupAbort(); if (read.oversized) { upstream.abort(); + if (hosted) return fail(502, "canonical Anthropic response exceeded the configured body limit", "api_error"); return fail(502, "upstream response exceeded the safe limit", "api_error", "translation_buffer_limit"); } + if (hosted && read.truncated) return fail(502, invalidJson, "api_error"); let parsedMessage: unknown; try { parsedMessage = JSON.parse(read.text); } catch { - return fail(502, "upstream returned malformed Messages JSON", "api_error"); + return fail(502, hosted ? invalidJson : "upstream returned malformed Messages JSON", "api_error"); + } + if (hosted && isRec(parsedMessage) && parsedMessage.type === "error" && isRec(parsedMessage.error)) { + const error = parsedMessage.error; + // Anthropic's own HTTP status for overloaded_error is 529. + const status = error.type === "overloaded_error" ? 529 : httpStatusFromTerminalError({ + type: typeof error.type === "string" ? error.type : undefined, + code: error.code === null || typeof error.code === "string" ? error.code : undefined, + message: typeof error.message === "string" ? error.message : undefined, + }); + finishLog(status, typeof error.message === "string" ? error.message : undefined, { closeReason: "terminal", terminalStatus: "failed" }); + const init = { status, ...(status === response.status ? { statusText: response.statusText } : {}), headers: relayedHeaders(response.headers) }; + const relayed = attachComboFailureEvidence(new Response(read.text, init), read.text); + if (committed) markResponseNonReplayable(relayed); + return options.sendBudget + ? markVerbatimComboFailure(relayed, () => markClientWire(new Response(read.text, init), "messages")) : relayed; + } + if (hosted && !isAnthropicMessageResponse(parsedMessage)) { + return fail(502, "canonical Anthropic returned an invalid JSON response", "api_error"); } if (!isRec(parsedMessage) || parsedMessage.type !== "message") { return fail(502, "upstream response was not a Messages result", "api_error"); @@ -1223,9 +1330,9 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp } throw error; } - // The client's selector, not the wire id, as the translated lane answers. - message.model = options.echoModel ?? requestedModel; - const serialized = JSON.stringify(message); + // The client's selector, not the wire id, as the translated lane answers; hosted relays upstream bytes. + if (!hosted) message.model = options.echoModel ?? requestedModel; + const serialized = hosted && message === parsedMessage ? read.text : JSON.stringify(message); finishLog(200); if (requestedStream) { return retainUpstreamMessagesRequestId(new Response(messageAsSse(message), { @@ -1233,7 +1340,8 @@ async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemp headers: { "Content-Type": "text/event-stream; charset=utf-8", "Cache-Control": "no-cache" }, }), response.headers); } - return retainUpstreamMessagesRequestId(new Response(serialized, { status: 200, headers: { "Content-Type": "application/json" } }), response.headers); + return retainUpstreamMessagesRequestId(new Response(serialized, { status: 200, + headers: hosted ? relayedHeaders(response.headers) : { "Content-Type": "application/json" } }), response.headers); } /** A folded stream error that refuses an oversized input (same gate as the SSE rewrite). */ diff --git a/src/server/messages-shared.ts b/src/server/messages-shared.ts index f8b9d567c2b..535b84425c4 100644 --- a/src/server/messages-shared.ts +++ b/src/server/messages-shared.ts @@ -26,6 +26,7 @@ import { } from "../lib/claude-request-projection"; import { redactSecretString } from "../lib/redact"; import { sseFieldValue } from "../lib/sse-decoder"; +import { BoundedSseFrameBuffer, EMPTY_BYTES, joinSseFrameBytes } from "./sse-frame-buffer"; import { estimateTokens } from "../lib/token-estimate"; import { isTranslatorBudgetExceededError } from "../lib/translator-budget"; import { httpStatusFromTerminalError } from "../lib/errors"; @@ -89,6 +90,8 @@ export interface PassthroughBodyGuard { maxBytes: number; /** Client request signal for deterministic cancel classification. */ reqSignal?: AbortSignal; + /** The .27 strict relay (validated frames, first terminal ends it). Production callers set it; tests default to the flag. */ + strictRelay?: boolean; /** * The routed provider opted into anthropicEofTolerance: a content-only clean EOF (no * message_stop, no stop_reason, nothing cut) stays a completed turn, as it always was. @@ -97,7 +100,7 @@ export interface PassthroughBodyGuard { } export type PassthroughCloseReason = "terminal" | "client_cancel" | "body_stall" | "body_overflow"; -export type PassthroughFinalizeMeta = { closeReason: PassthroughCloseReason; terminalStatus?: "failed" | "incomplete" }; +export type PassthroughFinalizeMeta = { closeReason: PassthroughCloseReason; terminalStatus?: "completed" | "failed" | "incomplete" }; /** * Tap an Anthropic-vocabulary SSE stream for the request log (usage + terminal), @@ -148,6 +151,8 @@ export function tapAnthropicSseForLog( // message_stop or an upstream error event: the turn's own terminal already went through. let terminalSeen = false; let errorSeen = false; + // The first terminal ends the relay: nothing after message_stop or a root error reaches the client. + let terminalWasError = false; // stop_reason of the latest message event the client received as a complete SSE event. Any // later message event (resumed content) clears it; pings and comments do not. let lastDeliveredStop: string | null = null; @@ -180,6 +185,7 @@ export function tapAnthropicSseForLog( usageAcc = { ...usageAcc, ...data.usage }; } if (named(event, data, "message_stop") || named(event, data, "error")) { + if (!terminalSeen) terminalWasError = data.type === "error"; terminalSeen = true; if (data.type === "error") { errorSeen = true; @@ -270,6 +276,13 @@ export function tapAnthropicSseForLog( const recordUsage = () => { logCtx.usage = anthropicUsageToOcx(Object.keys(usageAcc).length > 0 ? usageAcc : undefined); }; + // AgentRadar hosted gateways keep the .27 strict relay (validated frames up to the first terminal); + // everywhere else the relay keeps upstream's byte-for-byte contract. + const strict = guard?.strictRelay ?? process.env.OPENCODEX_AGENTRADAR === "1"; + // A root error terminal is a failed turn, however cleanly it was delivered. + const finalizeTerminal = () => !strict ? finalize(200, { closeReason: "terminal" }) + : terminalWasError ? finalize(502, { closeReason: "terminal", terminalStatus: "failed" }) + : finalize(200, { closeReason: "terminal", terminalStatus: "completed" }); const closeWithErrorFrame = (errType: string, message: string, origin: MessagesSseLocalOrigin) => { const payload = JSON.stringify({ type: "error", error: { type: errType, message } }); try { @@ -292,7 +305,7 @@ export function tapAnthropicSseForLog( // The turn already ended (message_stop or an upstream error event): an upstream that then // idles or keeps sending did not cut it short. Same rule as the read-error branch, // including the restored blank line for a terminal found only in the tail. - finalize(200, { closeReason: "terminal" }); + finalizeTerminal(); try { if (terminalInTail) enqueue(encoder.encode("\n\n")); close(); @@ -303,7 +316,8 @@ export function tapAnthropicSseForLog( // row with no terminalStatus also lost its failure diagnostics in usage.jsonl. recordLocalFailureDetail(logCtx, message, closeReason === "body_stall" ? { code: "body_stall", value: guard?.stallMs ?? 0 } : { code: "body_overflow", value: guard?.maxBytes ?? 0 }); - finalize(502, { terminalStatus: "incomplete", closeReason }); + // The .27 hosted contract records a stall as a gateway timeout. + finalize(strict && closeReason === "body_stall" ? 504 : 502, { terminalStatus: strict ? "failed" : "incomplete", closeReason }); closeWithErrorFrame(errType, message, closeReason === "body_stall" ? "stall" : "cap"); } reader.cancel(new DOMException(message, closeReason === "body_stall" ? "TimeoutError" : "QuotaExceededError")).catch(() => {}); @@ -341,6 +355,38 @@ export function tapAnthropicSseForLog( return () => signal.removeEventListener("abort", onClientAbort); })(); + // Relay one validated SSE frame per pull. A frame that is not UTF-8 JSON-object data (or is the + // Responses `[DONE]` sentinel) never reaches the client: the turn fails before it, as a reset does. + const framer = new BoundedSseFrameBuffer(); + const strictText = new TextDecoder("utf-8", { fatal: true }); + const MALFORMED = new Uint8Array(0); + let pendingFrames: Uint8Array[] = []; + let pendingIndex = 0; + const validFrame = (block: Uint8Array): boolean => { + let text: string; + try { text = strictText.decode(block); } catch { return false; } + const data = text.split(/\r\n|\r|\n/).map(line => sseFieldValue(line, "data")).filter(value => value !== null); + if (data.length === 0) return true; + try { return isRec(JSON.parse(data.join("\n"))); } catch { return false; } + }; + const failProtocol = () => { + if (settled) return; + settled = true; + idle.cancel(); + detachAbort(); + pendingFrames = []; + framer.dispose(); + recordUsage(); + const message = "anthropic passthrough protocol error"; + logCtx.transportPhase = "mid_stream"; + logCtx.terminalSource = "synthetic"; + recordLocalFailureDetail(logCtx, message, { code: "stream_failed" }); + if (logCtx.activeAttempt) logCtx.activeAttempt.streamAborted = true; + finalize(502, { terminalStatus: "failed", closeReason: "terminal" }); + closeWithErrorFrame("api_error", message, "read"); + reader.cancel(new Error(message)).catch(() => {}); + }; + return new ReadableStream({ start(controller) { tapController = controller; @@ -349,11 +395,37 @@ export function tapAnthropicSseForLog( if (settled) return; try { for (;;) { + const frame = pendingFrames[pendingIndex++]; + if (frame !== undefined) { + if (frame === MALFORMED) { failProtocol(); return; } + inspect(frame); + if (terminalSeen) { + settled = true; + idle.cancel(); + detachAbort(); + pendingFrames = []; + framer.dispose(); + recordUsage(); + finalizeTerminal(); + enqueue(frame); + close(); + reader.cancel().catch(() => {}); + return; + } + if (enqueue(frame) > 0) return; + continue; + } idle.reset(); const { done, value } = await reader.read(); idle.pause(); if (settled) return; // stall/overflow/abort won the race while we awaited if (done) { + const tail = strict ? framer.finish() : EMPTY_BYTES; + if (tail.byteLength > 0) { + if (!validFrame(tail)) { failProtocol(); return; } + inspect(tail); + enqueue(tail); + } // Bun can resolve a cancelled read as done before the abort listener runs: a client // that left is a cancel, not a cut upstream. if (guard?.reqSignal?.aborted) { @@ -379,7 +451,7 @@ export function tapAnthropicSseForLog( ? null : "anthropic passthrough upstream stream ended before message_stop"; if (failure === null) { - finalize(200, { closeReason: "terminal" }); + finalizeTerminal(); if (terminalInTail) enqueue(encoder.encode("\n\n")); close(); return; @@ -405,8 +477,26 @@ export function tapAnthropicSseForLog( return; } } - inspect(value); - if (enqueue(value) > 0) return; + if (!strict) { + inspect(value); + if (enqueue(value) > 0) return; + continue; + } + // A dedicated producer frame (a local error presentation) keeps its identity through the + // re-framing, so the client filter still recognizes it; anything else fails closed. + const origin = frames?.take(value); + const atBoundary = framer.pendingBytes === 0; + let split; + try { split = framer.feed(value); } catch { failProtocol(); return; } + pendingFrames = []; + pendingIndex = 0; + for (const { block, delimiter, continuation } of split) { + pendingFrames.push(continuation || validFrame(block) ? joinSseFrameBytes([block, delimiter]) : MALFORMED); + } + if (origin) { + if (!atBoundary || framer.pendingBytes !== 0 || pendingFrames.length !== 1 || pendingFrames[0] === MALFORMED) { failProtocol(); return; } + frames!.register(pendingFrames[0]!, origin); + } } } catch (err) { if (settled) return; @@ -433,7 +523,7 @@ export function tapAnthropicSseForLog( if (terminalSeen) { // Only the transport trailer was lost; the client already has the turn's terminal. // The Responses relay likewise reports a read error only without a seen terminal. - finalize(200, { closeReason: "terminal" }); + finalizeTerminal(); try { // An SSE parser drops an event that EOF cuts off before its blank line, so restore // the delimiter when the terminal was only found in that unterminated tail. diff --git a/src/server/request-log-attempt-recovery.ts b/src/server/request-log-attempt-recovery.ts index 400f0e6426d..600e19c52ba 100644 --- a/src/server/request-log-attempt-recovery.ts +++ b/src/server/request-log-attempt-recovery.ts @@ -1,4 +1,13 @@ -import type { AttemptRecoveryWithheld, PersistedUsageAttempt } from "../usage/log"; +import type { AttemptRecoveryKind, AttemptRecoveryWithheld, PersistedUsageAttempt } from "../usage/log"; + +/** Credential recovery starts a new attempt even when the account's opaque identity is unchanged. */ +export function credentialRecoveryStatus(recovery?: AttemptRecoveryKind): 401 | 402 | 403 | 429 | undefined { + if (recovery === "oauth-401" || recovery === "key-401") return 401; + if (recovery === "oauth-account-402") return 402; + if (recovery === "oauth-account-403") return 403; + if (recovery === "key-429" || recovery === "anthropic-oauth-429" || recovery === "oauth-account-429") return 429; + return undefined; +} /** * Record that a recovery this attempt was eligible for did not happen. diff --git a/src/server/request-log-attempt-usage.ts b/src/server/request-log-attempt-usage.ts index 3109c59f1c2..b872b2681ac 100644 --- a/src/server/request-log-attempt-usage.ts +++ b/src/server/request-log-attempt-usage.ts @@ -1,4 +1,4 @@ -import { usageTotalTokens, type PersistedUsageAttempt, type UsageStatus } from "../usage/log"; +import { physicalUsageAttempts, usageTotalTokens, type PersistedUsageAttempt, type UsageStatus } from "../usage/log"; import type { OcxUsage } from "../types"; interface FinalizedUsageResult { @@ -8,8 +8,10 @@ interface FinalizedUsageResult { } export function aggregateAttemptUsage( - attempts: readonly PersistedUsageAttempt[], + allAttempts: readonly PersistedUsageAttempt[], ): FinalizedUsageResult { + // Only real upstream sends count: locally answered, unsent and duplicate-ordinal rows spent nothing. + const attempts = physicalUsageAttempts(allAttempts); // A usage-less attempt that never reached upstream spent nothing (summary.ts `spentNothing`), // so it cannot leave the row's usage unknown. const measured = attempts.filter(attempt => attempt.usage @@ -39,11 +41,21 @@ export function aggregateAttemptUsage( )); return present.length > 0 ? present.reduce((sum, value) => sum + value, 0) : undefined; }; - const cachedInputTokens = sumOptional("cachedInputTokens"); - // Readers take `cacheReadInputTokens ?? cachedInputTokens` (usage/cost.ts), so once any attempt - // reports the read field, an attempt that only reports `cachedInputTokens` must count in it too. - const cacheReadInputTokens = sumOptional("cacheReadInputTokens") === undefined ? undefined + // Readers take `cacheReadInputTokens ?? cachedInputTokens` (usage/cost.ts), so each attempt's read + // counts once, under whichever field it reported, and both aggregate fields carry the same sum. + const cacheReadInputTokens = sumOptional("cacheReadInputTokens") === undefined && sumOptional("cachedInputTokens") === undefined + ? undefined : usages.reduce((sum, usage) => sum + (usage.cacheReadInputTokens ?? usage.cachedInputTokens ?? 0), 0); + const cachedInputTokens = cacheReadInputTokens; + // A context checkpoint is absolute (the latest wins); server-tool counters add. + const contextTotalTokens = usages.findLast(usage => typeof usage.contextTotalTokens === "number" + && Number.isFinite(usage.contextTotalTokens) && usage.contextTotalTokens >= 0)?.contextTotalTokens; + const anthropicServerToolUse: Record = {}; + for (const usage of usages) { + for (const [name, value] of Object.entries(usage.anthropicServerToolUse ?? {})) { + if (Number.isFinite(value) && value >= 0) anthropicServerToolUse[name] = (anthropicServerToolUse[name] ?? 0) + value; + } + } const cacheCreationInputTokens = sumOptional("cacheCreationInputTokens"); const cacheCreation1hInputTokens = sumOptional("cacheCreation1hInputTokens"); const reasoningOutputTokens = sumOptional("reasoningOutputTokens"); @@ -62,8 +74,68 @@ export function aggregateAttemptUsage( ...(cacheCreation1hInputTokens !== undefined ? { cacheCreation1hInputTokens } : {}), ...(reasoningOutputTokens !== undefined ? { reasoningOutputTokens } : {}), ...(providerCredits !== undefined ? { providerCredits } : {}), + ...(contextTotalTokens !== undefined ? { contextTotalTokens } : {}), + ...(Object.keys(anthropicServerToolUse).length > 0 ? { anthropicServerToolUse } : {}), ...(status === "estimated" ? { estimated: true } : {}), ...(usages.every(usage => usage.upstreamReported) ? { upstreamReported: true as const } : {}), }; return { usage: aggregate, status, totalTokens }; } + +/** Responses (`input_tokens`) or Chat Completions (`prompt_tokens`) wire usage as an OcxUsage. */ +export function usageFromResponsesPayload(usage: unknown): OcxUsage | undefined { + if (!usage || typeof usage !== "object") return undefined; + const raw = usage as { + input_tokens?: unknown; + output_tokens?: unknown; + input_tokens_details?: { cached_tokens?: unknown; cache_write_tokens?: unknown; cache_write_1h_tokens?: unknown }; + output_tokens_details?: { reasoning_tokens?: unknown }; + total_tokens?: unknown; + prompt_tokens?: unknown; + completion_tokens?: unknown; + prompt_tokens_details?: { cached_tokens?: unknown; cache_write_tokens?: unknown }; + completion_tokens_details?: { reasoning_tokens?: unknown }; + }; + if (typeof raw.input_tokens === "number" && typeof raw.output_tokens === "number") { + return { + inputTokens: raw.input_tokens, + outputTokens: raw.output_tokens, + ...(typeof raw.total_tokens === "number" ? { totalTokens: raw.total_tokens } : {}), + ...(typeof raw.input_tokens_details?.cached_tokens === "number" + ? { + cachedInputTokens: raw.input_tokens_details.cached_tokens, + cacheReadInputTokens: raw.input_tokens_details.cached_tokens, + } + : {}), + ...(typeof raw.input_tokens_details?.cache_write_tokens === "number" + ? { cacheCreationInputTokens: raw.input_tokens_details.cache_write_tokens } + : {}), + ...(typeof raw.input_tokens_details?.cache_write_1h_tokens === "number" + ? { cacheCreation1hInputTokens: raw.input_tokens_details.cache_write_1h_tokens } + : {}), + ...(typeof raw.output_tokens_details?.reasoning_tokens === "number" + ? { reasoningOutputTokens: raw.output_tokens_details.reasoning_tokens } + : {}), + }; + } + if (typeof raw.prompt_tokens === "number" && typeof raw.completion_tokens === "number") { + return { + inputTokens: raw.prompt_tokens, + outputTokens: raw.completion_tokens, + ...(typeof raw.total_tokens === "number" ? { totalTokens: raw.total_tokens } : {}), + ...(typeof raw.prompt_tokens_details?.cached_tokens === "number" + ? { + cachedInputTokens: raw.prompt_tokens_details.cached_tokens, + cacheReadInputTokens: raw.prompt_tokens_details.cached_tokens, + } + : {}), + ...(typeof raw.prompt_tokens_details?.cache_write_tokens === "number" + ? { cacheCreationInputTokens: raw.prompt_tokens_details.cache_write_tokens } + : {}), + ...(typeof raw.completion_tokens_details?.reasoning_tokens === "number" + ? { reasoningOutputTokens: raw.completion_tokens_details.reasoning_tokens } + : {}), + }; + } + return undefined; +} diff --git a/src/server/request-log.ts b/src/server/request-log.ts index 9dd16451996..8a84727a8f9 100644 --- a/src/server/request-log.ts +++ b/src/server/request-log.ts @@ -3,12 +3,13 @@ import { randomBytes } from "node:crypto"; import { normalizeRecordedClient, type UsageClient } from "../usage/client"; import { anthropicAccountTagField, stampApiKeyAccountLabel, usesApiKeyAccount } from "../providers/label"; import { finalizeProviderCacheAffinity } from "./request-log-cache-affinity"; -import { aggregateAttemptUsage } from "./request-log-attempt-usage"; -export { aggregateAttemptUsage } from "./request-log-attempt-usage"; +import { aggregateAttemptUsage, usageFromResponsesPayload } from "./request-log-attempt-usage"; +export { aggregateAttemptUsage, usageFromResponsesPayload } from "./request-log-attempt-usage"; import { KEY_ACCOUNT_LOG_LABEL_RE } from "../codex/account-label"; import { attemptAccountChanged, sealRequestAttemptIdentity } from "./request-log-account-rotation"; export { sealRequestAttemptIdentity }; +import { credentialRecoveryStatus } from "./request-log-attempt-recovery"; export { noteAttemptRecoveryWithheld } from "./request-log-attempt-recovery"; import { bareVerdictRowErrorCode, captureUpstreamTerminalDiagnostics, guardedRowCopy, linkLogField, notifyLinkTurnFinal, upstreamDiagnosticLogFields, type LinkRequestLogCarrier, type UpstreamErrorDiagnostics, type UpstreamErrorPayload } from "./request-log-terminal-status"; export { noteUpstreamRequestId } from "./request-log-terminal-status"; @@ -36,6 +37,7 @@ import type { RequestSpendSettlement } from "./responses/request-spend"; import type { AdapterTierMetadata } from "../providers/fastwire"; import { UPSTREAM_RESET_REPLAY_REFUSED_CODE } from "../lib/upstream-retry"; import { redactSecretString, sanitizeLogMetadataString } from "../lib/redact"; +import { logUsageLine } from "../lib/usage-line-log"; import type { UpstreamCondition } from "../lib/upstream-condition"; import { boundedServiceTier, errorDetailFor, projectErrorDetail } from "../usage/error-detail"; import { incompleteReasonDetail, upstreamErrorIsLocal, type FailureDetail } from "./request-log-failure-detail"; @@ -60,6 +62,7 @@ import { normalizeRequestFailureAttribution, normalizeRequestSpend, normalizedAttempts, + physicalUsageAttempts, readRecentUsageEntries, modelIdentityLogFields, recordObservedServedModel, usageForFinalLog, @@ -187,6 +190,10 @@ export interface RequestLogContext extends UpstreamErrorDiagnostics, LinkRequest localTerminalReason?: string; /** Stable non-PII Codex Pool account identity for durable usage attribution. */ accountLogLabel?: string; + /** Set by the combo ladder when this logical request moved to another target. */ + comboTargetAdvanced?: true; + /** Root usage spans several physical sends (a sidecar total), so it is never one attempt's usage. */ + usageIsRequestAggregate?: boolean; requestedModel?: string; /** User-facing alias selector when routing resolved one; native model remains `model`. */ requestedAlias?: string; @@ -366,6 +373,7 @@ export interface RequestLogEntry extends UpstreamErrorDiagnostics, LinkRequestLo accountLogLabel?: string; /** Anthropic pool account of `provider` (#8383): its p-label, beside the unchanged `accountLogLabel`. */ accountTag?: `p${string}`; + comboTargetAdvanced?: true; /** Best-effort chat/session correlation for Logs grouping (#330). */ conversationId?: string; requestedModel?: string; @@ -540,6 +548,7 @@ export function requestLogEntryFromPersistedUsage(entry: PersistedUsageEntry): R ? { accountLogLabel: entry.accountLogLabel } : {}), ...(isAnthropicAccountTag(entry.accountTag) ? { accountTag: entry.accountTag } : {}), + ...(entry.comboTargetAdvanced === true ? { comboTargetAdvanced: true } : {}), ...(entry.requestedModel ? { requestedModel: entry.requestedModel } : {}), ...(entry.requestedAlias ? { requestedAlias: entry.requestedAlias } : {}), ...(entry.shadowCallRewrittenFrom @@ -558,7 +567,7 @@ export function requestLogEntryFromPersistedUsage(entry: PersistedUsageEntry): R ? { modelSupportsServiceTier: entry.modelSupportsServiceTier } : {}), ...(entry.responseServiceTier ? { responseServiceTier: entry.responseServiceTier } : {}), - ...(entry.tierOutcome ? { tierOutcome: entry.tierOutcome } : {}), + ...tierOutcomeField(finalPhysicalTierOutcome(entry.attempts, entry.tierOutcome)), ...modelIdentityLogFields(entry), status: entry.status, durationMs: entry.durationMs, @@ -716,6 +725,8 @@ export function addRequestLog(entry: RequestLogEntry) { else if (retained !== entry) delete retained.jevDecision; entry = retained; retainRequestLogEntry(entry); + // AgentRadar `[usage]` line: one per finished request record, never text or secrets. + logUsageLine(entry); for (const observer of requestLogObserversForTests) { try { observer(entry); } catch { /* test observation must never fail request logging */ } } @@ -750,6 +761,7 @@ export function addRequestLog(entry: RequestLogEntry) { ? { accountLogLabel: entry.accountLogLabel } : {}), ...(isAnthropicAccountTag(entry.accountTag) ? { accountTag: entry.accountTag } : {}), + ...(entry.comboTargetAdvanced === true ? { comboTargetAdvanced: true } : {}), ...(entry.conversationId ? { conversationId: entry.conversationId } : {}), ...(entry.resolvedModel ? { resolvedModel: entry.resolvedModel } : {}), ...(entry.servedModel ? { servedModel: entry.servedModel } : {}), @@ -867,6 +879,10 @@ export function recordAdapterReasoning( delete attempt.effectiveEffort; delete attempt.reasoningWireField; delete attempt.reasoningWireValue; + delete attempt.promptCacheTtlMs; + if (request.promptCacheTtlMs === 300_000 || request.promptCacheTtlMs === 3_600_000) { + attempt.promptCacheTtlMs = request.promptCacheTtlMs; + } } recordAttemptRequestedEffort(logCtx); @@ -1065,63 +1081,6 @@ export function applyResponseLogMetadata(logCtx: RequestLogContext, payload: unk } } -export function usageFromResponsesPayload(usage: unknown): OcxUsage | undefined { - if (!usage || typeof usage !== "object") return undefined; - const raw = usage as { - input_tokens?: unknown; - output_tokens?: unknown; - input_tokens_details?: { cached_tokens?: unknown; cache_write_tokens?: unknown; cache_write_1h_tokens?: unknown }; - output_tokens_details?: { reasoning_tokens?: unknown }; - total_tokens?: unknown; - prompt_tokens?: unknown; - completion_tokens?: unknown; - prompt_tokens_details?: { cached_tokens?: unknown; cache_write_tokens?: unknown }; - completion_tokens_details?: { reasoning_tokens?: unknown }; - }; - if (typeof raw.input_tokens === "number" && typeof raw.output_tokens === "number") { - return { - inputTokens: raw.input_tokens, - outputTokens: raw.output_tokens, - ...(typeof raw.total_tokens === "number" ? { totalTokens: raw.total_tokens } : {}), - ...(typeof raw.input_tokens_details?.cached_tokens === "number" - ? { - cachedInputTokens: raw.input_tokens_details.cached_tokens, - cacheReadInputTokens: raw.input_tokens_details.cached_tokens, - } - : {}), - ...(typeof raw.input_tokens_details?.cache_write_tokens === "number" - ? { cacheCreationInputTokens: raw.input_tokens_details.cache_write_tokens } - : {}), - ...(typeof raw.input_tokens_details?.cache_write_1h_tokens === "number" - ? { cacheCreation1hInputTokens: raw.input_tokens_details.cache_write_1h_tokens } - : {}), - ...(typeof raw.output_tokens_details?.reasoning_tokens === "number" - ? { reasoningOutputTokens: raw.output_tokens_details.reasoning_tokens } - : {}), - }; - } - if (typeof raw.prompt_tokens === "number" && typeof raw.completion_tokens === "number") { - return { - inputTokens: raw.prompt_tokens, - outputTokens: raw.completion_tokens, - ...(typeof raw.total_tokens === "number" ? { totalTokens: raw.total_tokens } : {}), - ...(typeof raw.prompt_tokens_details?.cached_tokens === "number" - ? { - cachedInputTokens: raw.prompt_tokens_details.cached_tokens, - cacheReadInputTokens: raw.prompt_tokens_details.cached_tokens, - } - : {}), - ...(typeof raw.prompt_tokens_details?.cache_write_tokens === "number" - ? { cacheCreationInputTokens: raw.prompt_tokens_details.cache_write_tokens } - : {}), - ...(typeof raw.completion_tokens_details?.reasoning_tokens === "number" - ? { reasoningOutputTokens: raw.completion_tokens_details.reasoning_tokens } - : {}), - }; - } - return undefined; -} - /** * Mark a refusal this proxy synthesized locally. Sets origin to `synthetic` and a * distinct local reason so the request log cannot be read as an upstream overload. @@ -1441,8 +1400,8 @@ export function recordKeyWireAttemptUsage(logCtx: RequestLogContext, usage: OcxU const current = { ...usage }; attempt.usage = baseline ? aggregateAttemptUsage([ - { ...attempt, usage: baseline, usageStatus: baseline.estimated ? "estimated" : "reported" }, - { ...attempt, usage: current, usageStatus: current.estimated ? "estimated" : "reported" }, + usageSnapshotAttempt(attempt, 1, baseline), + usageSnapshotAttempt(attempt, 2, current), ]).usage : current; logCtx.usage = attempt.usage; @@ -1510,7 +1469,9 @@ function writeFinalRequestLog( logCtx.activeAttempt, attemptStatus ?? effectiveStatus, Date.now() - (logCtx.activeAttemptStartedAt ?? start), - keyUsageOwners.has(logCtx.activeAttempt) + // Request usage describes this one send only when it is not a key-owned snapshot or a + // multi-send aggregate; otherwise the attempt keeps only its own usage. + keyUsageOwners.has(logCtx.activeAttempt) || logCtx.usageIsRequestAggregate ? logCtx.activeAttempt.usage : logCtx.usage, ); @@ -1562,12 +1523,12 @@ function writeFinalRequestLog( // A key or account rotation splits the attempt and clears `logCtx.usage`, so a multi-attempt // request totals from its attempts, combo or not. One attempt keeps the request-level usage // (and its input estimate); a locally answered row sent nothing and keeps its own. - const aggregate = isCombo || ((attempts?.length ?? 0) > 1 && logCtx.localTerminalReason === undefined) + // An explicit attempt list with no physical send keeps no stale request usage either. + const aggregate = isCombo || (attempts !== undefined && logCtx.localTerminalReason === undefined + && (attempts.length > 1 || physicalUsageAttempts(attempts).length === 0)) ? aggregateAttemptUsage(attempts ?? []) : null; - const loggedUsage = aggregate?.usage ?? existing.usage; - const usageStatus = aggregate?.status ?? existing.status; - const totalTokens = aggregate?.totalTokens ?? existing.totalTokens; + const { usage: loggedUsage, status: usageStatus, totalTokens } = aggregate ?? existing; const spend = requestSpendRecord(logCtx, attempts); const durationMs = Date.now() - start; logCtx.requestMetricsRecorder?.recordFinalRequest({ @@ -1640,6 +1601,7 @@ function writeFinalRequestLog( ? { accountLogLabel: logCtx.accountLogLabel } : {}), ...anthropicAccountTagField(isCombo ? undefined : logCtx.provider), + ...(logCtx.comboTargetAdvanced === true ? { comboTargetAdvanced: true } : {}), ...(logCtx.conversationId ? { conversationId: logCtx.conversationId } : {}), ...(logCtx.requestedModel ? { requestedModel: logCtx.requestedModel } : {}), ...(logCtx.requestedAlias ? { requestedAlias: logCtx.requestedAlias } : {}), @@ -1655,9 +1617,7 @@ function writeFinalRequestLog( ...(logCtx.configuredSpeedLabel ? { configuredSpeedLabel: logCtx.configuredSpeedLabel } : {}), ...(logCtx.modelSupportsServiceTier !== undefined ? { modelSupportsServiceTier: logCtx.modelSupportsServiceTier } : {}), ...(logCtx.responseServiceTier ? { responseServiceTier: logCtx.responseServiceTier } : {}), - ...((attempts?.at(-1)?.tierOutcome ?? logCtx.tierOutcome) - ? { tierOutcome: attempts?.at(-1)?.tierOutcome ?? { ...logCtx.tierOutcome! } } - : {}), + ...tierOutcomeField(finalPhysicalTierOutcome(attempts, logCtx.tierOutcome)), ...modelIdentityLogFields(logCtx), status: effectiveStatus, durationMs, @@ -1837,8 +1797,7 @@ export function recordKeyAttemptUsage(logCtx: RequestLogContext, usage: OcxUsage const attempt = logCtx.activeAttempt; if (!attempt || !usage) return; attempt.usage = attempt.usage - ? aggregateAttemptUsage([{ ...attempt, usageStatus: attempt.usage.estimated ? "estimated" : "reported" }, - { ...attempt, usage, usageStatus: usage.estimated ? "estimated" : "reported" }]).usage + ? aggregateAttemptUsage([usageSnapshotAttempt(attempt, 1, attempt.usage), usageSnapshotAttempt(attempt, 2, usage)]).usage : { ...usage }; logCtx.usage = attempt.usage; } @@ -1858,14 +1817,14 @@ export function noteProviderAttemptSend( stampApiKeyAccountLabel(logCtx, providerName, provider); const next = logCtx.accountLogLabel; if (attempt && usesApiKeyAccount(provider)) keyUsageOwners.add(attempt); - if (attempt && attempt.sendCount > 0 && (attemptAccountChanged(previous, next, attempt.provider, logCtx.provider) + const recoveryStatus = credentialRecoveryStatus(recovery); + if (attempt && attempt.sendCount > 0 && (recoveryStatus !== undefined || attemptAccountChanged(previous, next, attempt.provider, logCtx.provider) || resolved && (attempt.model !== resolved.model || attempt.adapter !== resolved.adapter))) { // An input estimate is not evidence that a failed send used that many tokens. delete attempt.inputTokenEstimate; // An account-scoped recovery kind names the status the sealed account answered. finishRequestAttempt(attempt, attempt.status >= 100 ? attempt.status - : recovery === "key-401" || recovery === "oauth-401" ? 401 : recovery === "oauth-account-403" ? 403 - : recovery === "oauth-account-402" ? 402 : recovery?.includes("429") ? 429 : 502, + : recoveryStatus ?? (recovery === "rate-limit-429" ? 429 : 502), Date.now() - (logCtx.activeAttemptStartedAt ?? Date.now()), attempt.usage); // This attempt is being sealed because a NAMED recovery rejected it, so the recovery kind // is direct evidence here rather than an inference from history. Without this the sealed @@ -1882,9 +1841,11 @@ export function noteProviderAttemptSend( else attempts.push(completed, attempt); const fresh = beginRequestAttempt(completed.ordinal + 1, providerName, completed.model, completed.adapter); // Effort/tier metadata describes the request and is captured before the physical send. - for (const key of ["requestedEffort", "effectiveEffort", "reasoningWireField", "reasoningWireValue", "tierOutcome"] as const) { + for (const key of ["requestedEffort", "effectiveEffort", "reasoningWireField", "reasoningWireValue"] as const) { if (completed[key] !== undefined) Object.assign(fresh, { [key]: completed[key] }); } + // The response observer owns this live object; the sealed row keeps its snapshot. + if (attempt.tierOutcome) fresh.tierOutcome = attempt.tierOutcome; for (const key of Object.keys(attempt)) delete (attempt as unknown as Record)[key]; Object.assign(attempt, fresh); delete logCtx.usage; @@ -1932,8 +1893,16 @@ export function noteAttemptSend( attempt: PersistedUsageAttempt | undefined, inputTokenEstimate: number | undefined, recovery?: AttemptRecoveryKind, + now = performance.timeOrigin + performance.now(), ): void { if (!attempt) return; + // A new send reopens a finalized attempt; its earlier terminal no longer describes it. + if (finalizedAttempts.delete(attempt)) { + attempt.status = 0; + attempt.durationMs = 0; + delete attempt.errorCode; + } + if (attempt.sendCount === 0 && Number.isFinite(now) && now >= 0) attempt.firstSendAt = now; attempt.sendCount += 1; if (typeof inputTokenEstimate === "number" && Number.isFinite(inputTokenEstimate) @@ -1945,17 +1914,23 @@ export function noteAttemptSend( contextWindowForModel(attempt.adapter, attempt.model), ); } - if (recovery && !attempt.recoveryKinds.includes(recovery)) { - attempt.recoveryKinds.push(recovery); + if (recovery) { + attempt.recoveryCount = (attempt.recoveryCount ?? 0) + 1; + if (!attempt.recoveryKinds.includes(recovery)) attempt.recoveryKinds.push(recovery); } } +const finalizedAttempts = new WeakSet(); + export function finishRequestAttempt( attempt: PersistedUsageAttempt, status: number, durationMs: number, usage?: OcxUsage, ): PersistedUsageAttempt { + // Reopened only by `noteAttemptSend`: a later terminal belongs to the next send and must not + // relabel tokens this attempt already accounted. + if (finalizedAttempts.has(attempt)) return attempt; const finalized = finalizedUsage( attempt.adapter, usage ?? attempt.usage, @@ -1973,9 +1948,29 @@ export function finishRequestAttempt( const errorCode = requestLogErrorCode(status); if (errorCode) attempt.errorCode = errorCode; else delete attempt.errorCode; + finalizedAttempts.add(attempt); return attempt; } +/** The last real upstream attempt's tier outcome, else the request's. */ +function finalPhysicalTierOutcome( + attempts: readonly PersistedUsageAttempt[] | undefined, + fallback: AttemptTierOutcome | undefined, +): AttemptTierOutcome | undefined { + return physicalUsageAttempts(attempts ?? []).findLast(attempt => attempt.tierOutcome)?.tierOutcome ?? fallback; +} + +function tierOutcomeField(outcome: AttemptTierOutcome | undefined): { tierOutcome?: AttemptTierOutcome } { + return outcome ? { tierOutcome: { ...outcome } } : {}; +} + +/** One wire snapshot of an attempt, shaped as its own physical send so the aggregate counts it once. */ +function usageSnapshotAttempt(attempt: PersistedUsageAttempt, ordinal: number, usage: OcxUsage): PersistedUsageAttempt { + const snapshot: PersistedUsageAttempt = { ...attempt, ordinal, sendCount: 1, usage, usageStatus: usage.estimated ? "estimated" : "reported" }; + delete snapshot.locallyAnswered; + return snapshot; +} + export function getRequestLogEntries(): RequestLogEntry[] { return requestLog; } /** Test-only process-state reset for isolated integration harnesses. */ diff --git a/src/server/responses/adapter-continuation.ts b/src/server/responses/adapter-continuation.ts index cacadd3c3af..a0bb6f4d3d7 100644 --- a/src/server/responses/adapter-continuation.ts +++ b/src/server/responses/adapter-continuation.ts @@ -45,6 +45,7 @@ import { bindRouteReasoningReplayScope } from "./core-replay"; import { anthropicRoutingFor, formatAnthropicProviderForLog, + stampAnthropicAccountForLog, } from "../../oauth/anthropic-routing"; import { hasEligibleGenericOAuthFailoverTarget, @@ -199,6 +200,11 @@ export function createAdapterContinuations( // Ordinary terminal continuations have already delivered output; empty retries opt in. allowAccountRefusal = false, ): AsyncGenerator { + // AgentRadar gateways (no combo behind the pool) keep the hosted contract: a streamed re-ask may + // still move off a refused account, opening a new attempt, rather than failing a visible turn. + // Only .27's set rotates there: 429, a quota 400 and an org-disabled 403. + const hostedReask = !allowAccountRefusal && process.env.OPENCODEX_AGENTRADAR === "1"; + allowAccountRefusal ||= hostedReask; let response: Response | undefined; let refusalPendingReplay: Response | undefined; let replaySetupFailed = false; @@ -467,7 +473,7 @@ export function createAdapterContinuations( } } if ( - (response.status === 429 || response.status === 403 || response.status === 401) + [429, 403, 401, 400].includes(response.status) && anthropicInstance && transportState.anthropicPoolAccountId && !replaySetupFailed && !isNonReplayableResponse(response) @@ -481,7 +487,7 @@ export function createAdapterContinuations( config, accountId: transportState.anthropicPoolAccountId, sessionKey: anthropicSessionKey, model: route.modelId, requestKey: transportState, decision: transportState.anthropicRouteDecision, currentDecision: currentAnthropicRouteDecision, signal: upstream.signal, canRetry: hop.allowed && transportState.anthropicPoolFailovers < transportState.anthropicFailoverLimit, - allowAccountRefusal, allow429Recovery: allowAccountRefusal, + allowAccountRefusal, allow429Recovery: allowAccountRefusal, orgDisabledRefusalOnly: hostedReask, }); } catch (error) { hop.permit?.release(); throw error; } if (!nextAccountId) hop.permit?.release(); @@ -494,7 +500,7 @@ export function createAdapterContinuations( transportState.anthropicPoolFailovers += 1; route.provider = { ...route.provider, apiKey: admitted.accessToken }; invalidateSameTargetRequest(); - logCtx.provider = formatAnthropicProviderForLog(anthropicInstance, admitted.accountId, config); + stampAnthropicAccountForLog(logCtx, anthropicInstance, admitted.accountId, config); transportState.activeAdapter = resolveSelectionAdapter( resolveWireProtocolOverride(route.providerName, route.modelId, route.provider, inboundWire, route.staticPolicy), config.cacheRetention, diff --git a/src/server/responses/adapter-dispatch.ts b/src/server/responses/adapter-dispatch.ts index 6cf51fc51d9..b61bb3921d8 100644 --- a/src/server/responses/adapter-dispatch.ts +++ b/src/server/responses/adapter-dispatch.ts @@ -84,6 +84,7 @@ import { AnthropicAccountCooldownError, anthropicRoutingFor, formatAnthropicProviderForLog, + stampAnthropicAccountForLog, } from "../../oauth/anthropic-routing"; import { isGenericOAuthFailoverEnabled, @@ -1368,7 +1369,7 @@ export async function prepareAdapterExchange( // Anthropic OAuth: recover a rate limit or proven account entitlement refusal // before output, within the shared request and account rotation limits. while ( - (upstreamResponse.status === 429 || upstreamResponse.status === 403 || upstreamResponse.status === 401) + [429, 403, 401, 400].includes(upstreamResponse.status) && anthropicInstance && transportState.anthropicPoolAccountId ) { @@ -1405,7 +1406,7 @@ export async function prepareAdapterExchange( transportState.anthropicPoolFailovers += 1; route.provider = { ...route.provider, apiKey: admitted.accessToken }; invalidateSameTargetRequest(); - logCtx.provider = formatAnthropicProviderForLog(anthropicInstance, admitted.accountId, config); + stampAnthropicAccountForLog(logCtx, anthropicInstance, admitted.accountId, config); transportState.activeAdapter = resolveSelectionAdapter( resolveWireProtocolOverride(route.providerName, route.modelId, route.provider, inboundWire, route.staticPolicy), config.cacheRetention, diff --git a/src/server/responses/codex-main-401-log.ts b/src/server/responses/codex-main-401-log.ts new file mode 100644 index 00000000000..653c0475057 --- /dev/null +++ b/src/server/responses/codex-main-401-log.ts @@ -0,0 +1,32 @@ +import type { CodexAuthContext } from "../../codex/auth-context"; +import type { RouteResult } from "../../router"; +import type { OcxConfig } from "../../types"; +import type { DataPlaneAdmission } from "../auth-cors"; +import { hasForwardableCodexBearer } from "../auth-cors"; +import { usesCodexForwardPoolAuth } from "./core-codex-account"; + +/** + * Why a 401 did NOT reach the native-main refresh. Without this the skip is silent and looks like a + * refresh that ran and was refused. A caller-owned `main` context carries no account id, so `kind` + * and the resolver's own inputs are the only record of its branch. Booleans, enums and the pinned + * account id (already in config) only; never a token. + */ +export function logCodexMain401Skip( + req: Request, config: OcxConfig, route: RouteResult, authCtx: CodexAuthContext, + admission: Pick | undefined, +): void { + const forwardPoolAuth = usesCodexForwardPoolAuth(authCtx, route.provider); + // A forward pool context reaches the native-main refresh; only the other branches are skips. + if ((authCtx.kind === "main-pool" || authCtx.kind === "pool") && forwardPoolAuth) return; + console.warn( + "[codex] 401 upstream without native-main refresh:" + + ` kind=${authCtx.kind}` + + ` forwardableBearer=${hasForwardableCodexBearer(req.headers, config) ? "y" : "n"}` + + ` callerBearer=${req.headers.has("authorization") ? "y" : "n"}` + + ` admission=${admission?.source ?? "none"}` + + ` pinned=${config.activeCodexAccountPinned ?? "none"}` + + ` forwardPoolAuth=${forwardPoolAuth ? "y" : "n"}` + + ` adapter=${route.provider.adapter ?? "none"} authMode=${route.provider.authMode ?? "none"}` + + ` accountMode=${route.codexAccountMode ?? "none"}`, + ); +} diff --git a/src/server/responses/combo-stream-preflight.ts b/src/server/responses/combo-stream-preflight.ts index d5d60b9b293..5c6a05c64df 100644 --- a/src/server/responses/combo-stream-preflight.ts +++ b/src/server/responses/combo-stream-preflight.ts @@ -594,7 +594,13 @@ function failedTerminalResponse( status: (logCtx.terminalHttpStatus !== 400 && logCtx.terminalHttpStatus !== 422 && isProviderModerationRefusal(400, JSON.stringify(terminalPayload), undefined, true)) ? 400 - : logCtx.terminalHttpStatus ?? bareErrorStatus(terminalPayload) ?? 502, + : logCtx.terminalHttpStatus ?? bareErrorStatus(terminalPayload) + // AgentRadar hosted keeps .27's classified status (a zero-output stall is 504, not 502). + ?? (process.env.OPENCODEX_AGENTRADAR === "1" ? httpStatusFromTerminalError({ + type: typeof error.type === "string" ? error.type : undefined, + code: error.code === null || typeof error.code === "string" ? error.code : undefined, + message: typeof error.message === "string" ? error.message : undefined, + }) : 502), headers, }); } diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index 9313160b68b..488ff05cc8c 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -1731,7 +1731,9 @@ export async function handleResponsesCompact( const bodyInferredQuota = !replayRefused && !buffered.ok && !explicitQuotaStatus - && isRateLimitOrQuotaFailureMessage(bufferedErrorText); + // Read the body with its real status: a 400/422 that only echoes client values + // ("rate limit", "insufficient_quota") is a request verdict, not quota evidence. + && isRateLimitOrQuotaFailureMessage(`Provider error ${buffered.status}: ${bufferedErrorText}`); const quotaFailure = explicitQuotaStatus || bodyInferredQuota; // Record pool health only after the body is fully delivered (or definitively failed). // A premature 200 would clear soft-avoid while the client still sees a buffer 502. diff --git a/src/server/responses/core-auth.ts b/src/server/responses/core-auth.ts index 1c654644e41..a824e19aefa 100644 --- a/src/server/responses/core-auth.ts +++ b/src/server/responses/core-auth.ts @@ -251,6 +251,11 @@ export async function resolveResponsesCodexAuth( credentialDomainWasRewritten, config, ); + if (options.agentRadarAccount) { + authInputHeaders = new Headers(authInputHeaders); + authInputHeaders.delete("authorization"); + authInputHeaders.delete("chatgpt-account-id"); + } // A caller-auth transport that is not canonical OpenAI (keyless Cursor) consumes the // caller's Authorization as its own upstream token. Keep that contract only for a clean // single bearer with NO ChatGPT-domain marker. A bearer marked for the ChatGPT domain — diff --git a/src/server/responses/core-codex-account.ts b/src/server/responses/core-codex-account.ts index eb2c0338c6c..cbac05104ca 100644 --- a/src/server/responses/core-codex-account.ts +++ b/src/server/responses/core-codex-account.ts @@ -1017,10 +1017,10 @@ export async function retryCodexPoolOnAlternateAccount( return reserveWorkflowSend(options.workflowRootId, workflowPolicy); }; let retrySendCount = 0; - // The gated-model 400 has no roster kind; it stays unattributed as before. - const moveRecovery: AttemptRecoveryKind | undefined = outcomeStatus === 429 ? "oauth-account-429" + // The gated-model 400 retry (another account, or the confirmed one in place) says why a second send exists. + const moveRecovery: AttemptRecoveryKind = outcomeStatus === 429 ? "oauth-account-429" : outcomeStatus === 402 ? "oauth-account-402" - : isTransientUpstreamStatus(outcomeStatus) ? "transient-5xx" : undefined; + : isTransientUpstreamStatus(outcomeStatus) ? "transient-5xx" : "codex-account-retry"; let upstreamResponse = firstResponse; try { while (true) { diff --git a/src/server/responses/core-combo-failure.ts b/src/server/responses/core-combo-failure.ts index 2fae14e4e8e..da8e6800297 100644 --- a/src/server/responses/core-combo-failure.ts +++ b/src/server/responses/core-combo-failure.ts @@ -33,7 +33,7 @@ import { import { normalizeUpstreamErrorText } from "./core-errors"; import { resolveClientRetryAfter } from "../../lib/retry-after"; import { formatErrorResponse } from "../../bridge"; -import { upstreamCodeForComboFailure } from "../../bridge/errors"; +import { carryVerbatimComboFailure, upstreamCodeForComboFailure } from "../../bridge/errors"; import { OLLAMA_LATE_ATTRIBUTION_LIMIT_MESSAGE } from "../../adapters/ollama-native"; import { carryReplayRefusal, isNonReplayableResponse, isReplayRefusalResponse, markResponseNonReplayable, UPSTREAM_RESET_REPLAY_REFUSED_CODE } from "../../lib/upstream-retry"; import { MODEL_NOT_ALLOWED_FOR_KEY } from "../admission-model-scope"; @@ -184,6 +184,7 @@ export async function consumeComboFailure( ...(clientRetryAfter !== undefined ? { retryAfter: clientRetryAfter } : {}), }, ); + carryVerbatimComboFailure(response, failureResponse); if (nonReplayable) markResponseNonReplayable(failureResponse); carryProxyError(response, failureResponse); carryWorkflowRefusal(response, failureResponse); diff --git a/src/server/responses/core-combo-native.ts b/src/server/responses/core-combo-native.ts index 090b1f96a92..a2925ba5644 100644 --- a/src/server/responses/core-combo-native.ts +++ b/src/server/responses/core-combo-native.ts @@ -35,7 +35,9 @@ import { } from "../request-log"; import { captureRouteStaticPolicy, routeConcreteModel } from "../../router"; import { comboDefaultEffort, concreteComboRequestBody, getCombo, type JevDecision } from "../../combos"; +import { parseComboModelId } from "../../combos/identifiers"; import { supportedLadderFor } from "../effort-policy"; +import { claudeFamilyVersion, usesAdaptiveThinking } from "../../adapters/anthropic-model-contract"; import { applyDroidReasoningDefault } from "../droid-reasoning-default"; import { resolveWireProtocolOverride } from "../adapter-resolve"; import { resolveOpenCodeGoTransport } from "../../providers/opencode-go-transport"; @@ -48,7 +50,7 @@ import { credentialDomainFor } from "../../protocols/opaque-state"; import { chatCompletionsErrorResponse } from "../../chat/outbound"; import { OUTPUT_CONFIG_EFFORTS } from "../../chat/inbound"; import { anthropicErrorResponse } from "../../claude/outbound"; -import { hasAnthropicOnlyServerTool, hasNativeOnlyTool } from "../../claude/inbound-content-options"; +import { EXACT_REPLAY_REQUIRED_MESSAGE, hasAnthropicOnlyServerTool, hasNativeOnlyTool } from "../../claude/inbound-content-options"; import { nativeMessagesInlineToolNames } from "../messages-native-tool-guard"; const isRecord = (value: unknown): value is Record => @@ -68,6 +70,32 @@ function withdrawsTopLevelTool(body: Record): boolean { && typeof block.tool.name === "string" && declared.has(block.tool.name))); } +/** Families that accept `thinking: {type: "adaptive"}` (.27 matrix; usesAdaptiveThinking is the narrower "requires"). */ +const ACCEPTS_ADAPTIVE: Record = { sonnet: [4, 6], opus: [4, 6], fable: [0, 0], haiku: [5, 0] }; + +/** + * AgentRadar hosted gateways (.27): a combo never hands a model-changing exact replay to a target model + * that would refuse what the caller sent (dated typed tools, an adaptive or budget thinking shape), so + * the ladder moves on instead of stopping on that target's 400. + */ +export function hostedReplayIncompatible(body: Record, targetModel: string): boolean { + if (process.env.OPENCODEX_AGENTRADAR !== "1" || body.model === targetModel) return false; + // .27 exempted combo aliases from the typed-tool check only: comparing combo/ with a concrete + // target never matches, which rejected every WebFetch and web_search turn on a combo. + if (parseComboModelId(typeof body.model === "string" ? body.model : "") === null + && Array.isArray(body.tools) && body.tools.some(tool => isRecord(tool) && Object.hasOwn(tool, "type"))) return true; + const thinking = isRecord(body.thinking) ? body.thinking.type : undefined; + if (thinking === "enabled") return usesAdaptiveThinking(targetModel); + if (thinking !== "adaptive") return false; + const parsed = claudeFamilyVersion(targetModel); + const minimum = parsed && ACCEPTS_ADAPTIVE[parsed.family]; + return !minimum || parsed.major < minimum[0] || (parsed.major === minimum[0] && parsed.minor < minimum[1]); +} + +function hasWebFetchTool(body: Record): boolean { + return Array.isArray(body.tools) && body.tools.some(tool => isRecord(tool) && typeof tool.type === "string" && tool.type.startsWith("web_fetch_")); +} + function hasUnnamedInlineToolDefinition(body: Record): boolean { return Array.isArray(body.messages) && body.messages.some(message => isRecord(message) && message.role === "system" && Array.isArray(message.content) && message.content.some(block => isRecord(block) && block.type === "tool_addition" @@ -219,7 +247,7 @@ export function createComboProtocolLanes(input: { // has no translated equivalent either; fail closed like a top-level native-only tool. const declaresNativeOnlyTool = (body: Record) => hasNativeOnlyTool(body) || nativeMessagesInlineToolNames(body).size > 0 || hasUnnamedInlineToolDefinition(body) - || withdrawsTopLevelTool(body); + || withdrawsTopLevelTool(body) || Object.hasOwn(body, "mcp_servers"); const judge = (target: ComboTarget): CandidateVerdict => { const key = selector(target); const cached = verdicts.get(key); @@ -235,6 +263,7 @@ export function createComboProtocolLanes(input: { if (targetRoute) { const settled = settle(target, targetRoute); const native = nativeEligible(target, settled); + if (native && inbound === "messages" && hostedReplayIncompatible(envelope.readBody(), settled.modelId)) verdict.nativeOnly = true; const upstream = native ? inbound : upstreamWireForAdapter((settled ?? targetRoute).provider.adapter); const checked = checkRepresentable({ inbound, @@ -254,7 +283,9 @@ export function createComboProtocolLanes(input: { if (!native && inbound === "messages") { const provider = (settled ?? targetRoute).provider; if ((nativeOnlyTool ??= declaresNativeOnlyTool(envelope.readBody())) - || ((anthropicOnlyTool ??= hasAnthropicOnlyServerTool(envelope.readBody())) + || ((anthropicOnlyTool ??= process.env.OPENCODEX_AGENTRADAR === "1" + // .27 kept a restricted web_search's translated fallback; only web_fetch is out-of-body only. + ? hasWebFetchTool(envelope.readBody()) : hasAnthropicOnlyServerTool(envelope.readBody())) && !(provider.adapter === "anthropic" && credentialDomainFor(provider)?.firstPartyAnthropic))) { verdict.nativeOnly = true; } @@ -270,13 +301,18 @@ export function createComboProtocolLanes(input: { return verdict.nativeOnly !== true && (!reject || verdict.skip === undefined); }, refusal: () => { - if (!reject) return undefined; const enabled = targets.filter(target => { const provider = config.providers[target.provider]; return provider !== undefined && provider.disabled !== true; }); if (enabled.length === 0) return undefined; const skipped = enabled.map(judge); + // AgentRadar hosted gateways (.27): a source no candidate can carry is the request's own 400, with no send. + if (process.env.OPENCODEX_AGENTRADAR === "1" && inbound === "messages" && skipped.every(verdict => verdict.nativeOnly)) { + logCtx.errorCode = "unsupported_feature"; + return markClientWire(anthropicErrorResponse(400, EXACT_REPLAY_REQUIRED_MESSAGE, "invalid_request_error", "unsupported_feature"), inbound); + } + if (!reject) return undefined; if (skipped.some(verdict => verdict.skip === undefined)) return undefined; const features = [...new Set(skipped.flatMap(verdict => verdict.skip ?? []))]; // The same refusal the ingress gives a single route: 400 in the Chat shape, feature keys diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index 36ff0927e64..9dcee8c3a6e 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -64,6 +64,7 @@ import { type JevDecision, } from "../../combos"; import { formatErrorResponse } from "../../bridge"; +import { hasVerbatimComboFailure, verbatimComboFailure } from "../../bridge/errors"; import { SEND_BUDGET_EXHAUSTED_CODE } from "../../lib/errors"; import { expandPreviousResponseInput, @@ -695,6 +696,7 @@ export async function executeComboResponses( eligible: target => (meteredAllowed || !target.metered) && (pickOptions.eligible?.(target) ?? true), waitForCooldownMs: combo.waitForCooldownMs, abortSignal: options.abortSignal, + randomSeed: options.comboRandomSeed, }); // An in-flight tool loop stays on the target that issued its calls while that target is still // selectable (#7298). Override routes never record a lane target, so they never pin either. @@ -919,7 +921,8 @@ export async function executeComboResponses( } // One immutable combo selection trace, before any child dispatch; child // adoption below must never replace it with a concrete child route trace. - const decision = comboRouteDecisionTrace(config, comboId, pick, responseModel); + // A target this request can never run on is traced `request-incompatible`, not a `not-selected` runner-up. + const decision = comboRouteDecisionTrace(config, comboId, pick, responseModel, undefined, targetEligible); const selectedReason = options.memoryModelPhase ? memoryModelRouteReason(options.memoryModelPhase) : toolLoopPin ? COMBO_TOOL_LOOP_PIN_REASON : jevDecision?.pinned ? JEV_SESSION_PIN_ROUTE_REASON : undefined; @@ -975,7 +978,12 @@ export async function executeComboResponses( // Snapshotted before dispatch: a 401/403 inside this ladder cools its own provider, and that // fresh cooldown is not a reason the request found no target. const quotaCooldownSnapshot = snapshotComboQuotaCooldowns(config, comboId); + // AgentRadar hosted: the client gets a canonical Anthropic child's own error; decisions read the formatted one. const exhaustedFailure = (targetsExhausted = false): Response => { + const failure = rawExhaustedFailure(targetsExhausted); + return verbatimComboFailure(failure) ?? failure; + }; + const rawExhaustedFailure = (targetsExhausted = false): Response => { if (contextFailure) { adoptFailedChildLog(contextFailure.childLog); return contextFailure.response; @@ -1403,6 +1411,8 @@ export async function executeComboResponses( activeAttemptStartedAt: segmentStartedAt(), resolvedModel: childLog.resolvedModel ?? childLog.model, }); + // The row's account is the serving attempt's, never one left over from before the ladder. + if (childLog.accountLogLabel === undefined) delete logCtx.accountLogLabel; options.onCodexAuthContextResolved?.(resolvedAuth); options.setTerminalOutcomeRecorder?.(terminalRecorder); callbackGate.commit(); @@ -1510,6 +1520,7 @@ export async function executeComboResponses( const wantsStream = (rawBody as { stream?: unknown } | null)?.stream === true; // Local byte admission has its own diagnostic; do not relabel it as an upstream refusal. const classifyOverflow = failure.response.status === 413 && !lateAttributionRefusal + && !hasVerbatimComboFailure(failure.response) && (wantsStream || (failure.upstreamCode !== "outbound_body_too_large" && failure.upstreamCode !== "translation_buffer_limit")); lastFailureClassifiesOverflow = classifyOverflow; @@ -1543,7 +1554,7 @@ export async function executeComboResponses( ? streamingContextOverflowResponse(responseModel, options.translatorBudget) : jsonContextOverflowResponse(); } - return lastFailure; + return verbatimComboFailure(lastFailure) ?? lastFailure; } if (failureDecision === "stop") { adoptFailedChildLog(childLog); @@ -1552,7 +1563,7 @@ export async function executeComboResponses( ? streamingContextOverflowResponse(responseModel, options.translatorBudget) : jsonContextOverflowResponse(); } - return lastFailure; + return verbatimComboFailure(lastFailure) ?? lastFailure; } // Retain a confirmed target context refusal when later retryable failures exhaust the ladder. // Hard stops above still return their own verdict, and every attempt keeps its own usage. @@ -1685,6 +1696,9 @@ export async function executeComboResponses( } // Waiting or recovery may have observed cancellation after the check above. if (options.abortSignal?.aborted) return clientCancelledResponse(); + } else { + // One logical request moved to another target; the persisted row and routing analytics read it. + logCtx.comboTargetAdvanced = true; } } finally { // External receipts make release a no-op after real work. runTurn owns its asynchronous diff --git a/src/server/responses/core-options.ts b/src/server/responses/core-options.ts index a6cd0dcc860..38e795d9366 100644 --- a/src/server/responses/core-options.ts +++ b/src/server/responses/core-options.ts @@ -113,6 +113,8 @@ export interface DirectRequestOption { } export interface HandleResponsesOptions { + /** Trusted AgentRadar bridge authority; never changes the global account selection. */ + agentRadarAccount?: import("../../lib/agentradar-account-pin").AgentRadarAccountPin; /** Original caller label, carried through synthetic children without route substitution. */ clientModelLabel?: ClientFacingModelLabel; /** Original client envelope label when a combo was selected by a synthetic rewrite. */ @@ -226,6 +228,8 @@ export interface HandleResponsesOptions { droidDefaultEffort?: string; /** The Chat or Messages source a combo child may send natively; set only by that ingress. */ protocolSource?: import("./core-combo-native").ComboProtocolSource; + /** AgentRadar: a random combo picks from this request correlation seed, as its count_tokens did. */ + comboRandomSeed?: string; /** Internal transport identity for route-scoped upstream compatibility policy. */ inboundTransport?: "websocket"; /** diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index ca0fcc784f9..9a40a3aee81 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -32,6 +32,7 @@ import { type ProviderEgressBinding, } from "../../lib/provider-egress"; import { redactSecretString } from "../../lib/redact"; +import { observeUpstreamRefusal } from "../../lib/upstream-refusal-log"; import { outboundCredentials } from "../../lib/outbound-credential-registry"; import { compressUpstreamRequestBody, isCanonicalCodexResponsesUrl } from "./request-compression"; import { redeemOutboundTurnState, TURN_STATE_BINDING_INIT, turnStateBinding } from "../turn-state-handles"; @@ -351,9 +352,14 @@ export function providerFetch( // `provider.fetch ?? execute`, and `execute` is this wrapper. It still carries the binding, so // an override that simply calls it gets the route decided rather than dropped; an override // that decided for itself has already marked the init and this pass defers to that decision. + // Every provider HTTP answer passes here with its final outgoing body: a 4xx gets one + // refusal-shape log line (AgentRadar-gated); the response itself is returned untouched. const dispatch = markEgressTransparentExecutor(Object.assign( - (input: Parameters[0], init?: RequestInit) => - sendWithConnectionPolicy(transport, input, init, egressBinding), + async (input: Parameters[0], init?: RequestInit) => { + const response = await sendWithConnectionPolicy(transport, input, init, egressBinding); + observeUpstreamRefusal(response, { provider: options.providerName, model: options.modelId, body: init?.body }); + return response; + }, { preconnect }, ) as typeof globalThis.fetch); inheritPassiveKeyProof(dispatch, transport); diff --git a/src/server/responses/passthrough-dispatch.ts b/src/server/responses/passthrough-dispatch.ts index 98ae4684846..f65c5322885 100644 --- a/src/server/responses/passthrough-dispatch.ts +++ b/src/server/responses/passthrough-dispatch.ts @@ -1,3 +1,5 @@ +import { noteAgentRadarAccountDispatch } from "../../lib/agentradar-account-pin"; +import { logCodexMain401Skip } from "./codex-main-401-log"; import { clientFacingModelText } from "../../lib/error-destination"; import { providerAffinityGenerationIsCurrent } from "../../providers/api-key-selection"; import { markProxyError } from "../../lib/client-error-body"; @@ -770,7 +772,8 @@ export async function preparePassthroughExchange( parsed, currentRequest: () => request, }); - const recordPhysicalSend = replayGuard.sendHandler; + const recordPhysicalSend = (next?: () => void) => replayGuard.sendHandler(() => { next?.(); + if (admissionState.authCtx.kind === "pool") noteAgentRadarAccountDispatch(logCtx, admissionState.authCtx.accountId); }); /** * The operator's replacement grant for THIS logical request. * @@ -1207,7 +1210,7 @@ export async function preparePassthroughExchange( if ("failed" in recovered) return recovered.failed; upstreamResponse = recovered; continue passthroughRecovery; } - + if (upstreamResponse.status === 401 && !wasPermanentAccountRefusal(upstreamResponse) && codex401ReplayKind === null) logCodexMain401Skip(req, config, route, admissionState.authCtx, options.admission); if ( upstreamResponse.status === 401 && !wasPermanentAccountRefusal(upstreamResponse) && (admissionState.authCtx.kind === "main-pool" || admissionState.authCtx.kind === "pool") @@ -1284,13 +1287,14 @@ export async function preparePassthroughExchange( const replayGuard = isCanonicalOpenAiForwardProvider(route.provider) ? createCodexAuthDispatchGuard(admissionState.authCtx, options.codexAuthPolicy ?? config, route.modelId, options.admission, options.visionDescribeTerminal === true) : undefined; let replayNotified = false; + let replayRecovery: AttemptRecoveryKind | undefined; const oauthReplayExecutor = providerFetch(route.provider, options.codexWsRuntimeIdentity, { httpOnly: canonicalBufferedJson, nativeControl: nativeResponseControlEligible(route.provider, options.nativeControl) && options.inboundTransport === "websocket" && !options.comboAttempt && responseEffects.plaintextV2AgentMessageToolNames.size === 0 ? options.nativeControl : undefined, dispatchOverride: oauthDispatch(request, parsed, "passthrough"), - onPhysicalDispatch: recordPhysicalSend(), + onPhysicalDispatch: recordPhysicalSend(() => transportState.noteRoutedAttemptSend(passthroughEstimate, replayRecovery ?? "oauth-401")), providerName: route.providerName, modelId: route.modelId, onContinuationDispatch, @@ -1311,7 +1315,7 @@ export async function preparePassthroughExchange( // have run it. upstreamResponse = await fetchWithTransientRetry( recovery => { - transportState.noteRoutedAttemptSend(passthroughEstimate, recovery ?? "oauth-401"); + replayRecovery = recovery; return fetchWithHeaderTimeout( request.url, applyUpstreamRecoveryInit({ diff --git a/src/server/responses/request-prepare.ts b/src/server/responses/request-prepare.ts index e45d88d5b49..4f4ad93c473 100644 --- a/src/server/responses/request-prepare.ts +++ b/src/server/responses/request-prepare.ts @@ -754,6 +754,15 @@ export async function prepareResponsesRequest( // captured where it becomes the request's route. A declined policy target // must not lend its eligibility or "policy" route kind to the request. if (captureRequestPolicy) capturePolicyRequestRoute(policyScope, candidate); + const pin = options.agentRadarAccount; + if (pin) { + const allowed = !candidate.combo && (pin.provider === "codex" + ? candidate.providerName === OPENAI_CODEX_PROVIDER_ID && candidate.codexAccountMode !== undefined && pin.accountId !== "__main__" + : candidate.provider.authMode === "oauth" + && candidate.providerName === (pin.provider === "claude" ? "anthropic" : pin.provider === "grok" ? "xai" : "kimi")); + if (!allowed) throw new Error("AgentRadar pinned account cannot serve this route"); + if (pin.provider === "codex") Object.assign(candidate, { codexAccountMode: "pool", codexAccountId: pin.accountId, codexAccountNamespace: undefined }); + } candidate.staticPolicy = captureRouteStaticPolicy( candidate.providerName, candidate.modelId, diff --git a/src/server/responses/request-sidecar-auth.ts b/src/server/responses/request-sidecar-auth.ts index 195e7768bcf..8dd1a8fa156 100644 --- a/src/server/responses/request-sidecar-auth.ts +++ b/src/server/responses/request-sidecar-auth.ts @@ -1,3 +1,4 @@ +import { formatErrorResponse } from "../../bridge"; import { CodexInstallationIdUnavailableError } from "../../codex/installation-id"; import type { ResponsesRequestContext } from "./core-options"; import type { PreparedResponsesRequest } from "./request-prepare"; @@ -65,6 +66,7 @@ export async function prepareResponsesSidecarAuth( ? shouldResolveOpenAiWebSearchSidecar(config, parsed, isPassthrough) : (shouldResolveOpenAiWebSearchSidecar(config, parsed, isPassthrough) || shouldResolveOpenAiPassthroughWebSearchBridge(route.provider, parsed, isPassthrough))); + if (options.agentRadarAccount && (needsOpenAiVision || needsOpenAiSearch)) return formatErrorResponse(400, "invalid_request_error", "AgentRadar pinned requests cannot use helper models"); if (needsOpenAiVision || needsOpenAiSearch) { try { const candidates = listOpenAiForwardSidecarCandidates(config); @@ -135,6 +137,7 @@ export async function prepareResponsesSidecarAuth( admission: options.admission, codexAuthPolicy: options.codexAuthPolicy, providerName: route.providerName, advisorAdmissionPolicy: options.advisorAdmissionPolicy, }); + if (options.agentRadarAccount && visionPlan) return formatErrorResponse(400, "invalid_request_error", "AgentRadar pinned requests cannot use helper models"); const recordSidecarOutcome = openAiSidecar?.recordOutcome; if (visionPlan) { try { diff --git a/src/server/responses/request-transport.ts b/src/server/responses/request-transport.ts index 5361a53cce0..34cc5e197a7 100644 --- a/src/server/responses/request-transport.ts +++ b/src/server/responses/request-transport.ts @@ -1,3 +1,4 @@ +import { noteAgentRadarAccountDispatch } from "../../lib/agentradar-account-pin"; import { publicOAuthErrorPresentation } from "../../oauth"; import { observeProviderFailureEvent, projectLocalModelRefusalLog } from "../request-log-failure-detail"; import { markProxyError } from "../../lib/client-error-body"; @@ -39,6 +40,7 @@ import { ANTHROPIC_POOL_MAX_FAILOVERS_PER_REQUEST, AnthropicAccountCooldownError, formatAnthropicProviderForLog, + stampAnthropicAccountForLog, } from "../../oauth/anthropic-routing"; import { OAuthAccountPausedError, @@ -51,7 +53,7 @@ import { UnsupportedOAuthProviderError, } from "../../oauth"; import { - GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST, + oauthMaxAccountsPerRequest, GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST, antigravityModelRefusal, eligibleFailoverAccounts, @@ -149,6 +151,11 @@ export async function prepareResponsesTransport( ) { const { config, logCtx, options, req } = requestContext; const { route, parsed, inboundWire, translatorBudget } = requestState; + const pin = options.agentRadarAccount; + const pinnedSnapshot = (): Promise => anthropicRouting + ? anthropicRouting.getAnthropicPoolAccessSnapshot(pin!.accountId) + : getValidAccessSnapshotForAccount(route.providerName, pin!.accountId, { requireUsableAccount: true }); + let anthropicSelectionReason: AnthropicAccountSelectionReason | undefined; const deferPhysicalAttribution = route.providerName === "github-copilot" && route.provider.authMode !== "forward"; let copilotCandidate: { model: string; adapter: string; accountLogLabel?: string } | undefined; bindResponsesSpendTarget(logCtx); @@ -229,7 +236,7 @@ export async function prepareResponsesTransport( let genericFailoverAccountId: string | null = null; let genericFailovers = 0; let oauthSelection = route.provider.authMode === "oauth" - ? captureOAuthAccountSelection(route.providerName) : null; + ? pin ? { accountId: pin.accountId, revision: "agentradar-request" } : captureOAuthAccountSelection(route.providerName) : null; const anthropicManualSelectionGeneration = anthropicRouting?.captureAnthropicManualSelectionGeneration(); let servingOAuthSnapshot: OAuthAccessSnapshot | undefined; // These owners also serve early passthrough and sidecar sends. A dispatch-time @@ -254,6 +261,20 @@ export async function prepareResponsesTransport( proactive = false, anthropicReason?: AnthropicAccountSelectionReason, ): Promise => { + if (pin) { + requireAnthropicAdmission(); + if (candidate.provider !== route.providerName || candidate.accountId !== pin.accountId) return null; + const row = getAccountCredentialWithStatus(route.providerName, pin.accountId); + if (!row || row.paused || row.needsReauth) throw new OAuthLoginRequiredError(route.providerName); + if (row.credential.expires <= Date.now() || credentialGeneration(row.credential) !== candidate.generation) candidate = await pinnedSnapshot(); + if (anthropicInstance) { + requireAnthropicAdmission(); + anthropicRouteDecision = currentAnthropicRouteDecision(); + if (!routeCandidates(anthropicRouting!.getEligibleAnthropicAccounts(Date.now(), route.modelId), anthropicRouteDecision).includes(pin.accountId)) throw new OAuthLoginRequiredError(route.providerName); + } + servingOAuthSnapshot = candidate; + return candidate; + } const maxSelectionAttempts = 3; for (let attempt = 0; attempt < maxSelectionAttempts; attempt++) { requireAnthropicAdmission(); @@ -336,7 +357,7 @@ export async function prepareResponsesTransport( // Keyed on the account, not the selection revision: a selection that moved away and back // (A -> B -> A) before the 401 landed still serves the rejected credential, and skipping // the refresh would replay it and spend the one recovery attempt. - const candidate = (current?.accountId === sent.accountId + const candidate = (pin || current?.accountId === sent.accountId || (genericAffinityMode && current?.revision === oauthSelection?.revision && current?.accountId === oauthSelection?.accountId && servingOAuthSnapshot?.accountId === sent.accountId)) ? await forceRefreshOAuthAccessSnapshot(sent) : sent; @@ -565,7 +586,7 @@ export async function prepareResponsesTransport( if (!deferPhysicalAttribution) stampOAuthAccountLabel(logCtx, route.providerName, route.provider, snapshot.accountId); if (anthropicInstance !== undefined) { anthropicPoolAccountId = snapshot.accountId; - logCtx.provider = formatAnthropicProviderForLog(anthropicInstance!, snapshot.accountId, config); + stampAnthropicAccountForLog(logCtx, anthropicInstance!, snapshot.accountId, config); } else if (!deferPhysicalAttribution) { genericFailoverAccountId = snapshot.accountId; } @@ -621,10 +642,10 @@ export async function prepareResponsesTransport( } const selected = captureOAuthAccountSelection(route.providerName); const row = getAccountCredentialWithStatus(route.providerName, binding.snapshot.accountId); - return selected?.accountId === binding.selection.accountId && selected?.revision === binding.selection.revision + return (pin ? binding.snapshot.accountId === pin.accountId : selected?.accountId === binding.selection.accountId && selected?.revision === binding.selection.revision) && !!row && !row.paused && !row.needsReauth && row.credential.expires > Date.now() && (anthropicInstance === undefined || !anthropicRouting!.getAnthropicAccountHealthSnapshot(binding.snapshot.accountId) && !anthropicRatePolicy!.anthropicRatePauseUntil(binding.snapshot.accountId) && !anthropicQuota!.anthropicFamilyRejected(binding.snapshot.accountId, route.modelId)) - && (!genericAffinityMode + && (pin || !genericAffinityMode || genericSessionAccountEligible(config, route.providerName, binding.snapshot.accountId, route.modelId, Date.now(), true)) && credentialGeneration(row.credential) === binding.snapshot.generation; }; @@ -782,16 +803,17 @@ export async function prepareResponsesTransport( // Pacing may outlive admission. Preserve local pause/cooldown reasons even when // a strict route rejects the old proposal before another bearer can be resolved. let candidate = servingOAuthSnapshot; - if (anthropicInstance === undefined && candidate && getAccountSet(route.providerName)?.accounts.find(row => row.id === candidate!.accountId)?.paused) { + if (pin) candidate = await pinnedSnapshot(); + if (!pin && anthropicInstance === undefined && candidate && getAccountSet(route.providerName)?.accounts.find(row => row.id === candidate!.accountId)?.paused) { oauthSelection = captureOAuthAccountSelection(route.providerName); candidate = oauthSelection ? await getValidAccessSnapshotForAccount(route.providerName, oauthSelection.accountId, { requireUsableAccount: true }) : undefined; } - if (anthropicInstance !== undefined) { + if (!pin && anthropicInstance !== undefined) { oauthSelection = captureOAuthAccountSelection(route.providerName); const accountId = await anthropicRouting!.resolveAnthropicDispatchAccountId(config, anthropicSessionKey, anthropicRouteDecision, route.modelId); candidate = await anthropicRouting!.getAnthropicPoolAccessSnapshot(accountId); } - if (genericAffinityMode + if (!pin && genericAffinityMode && (!candidate || !genericSessionAccountEligible(config, route.providerName, candidate.accountId, route.modelId, Date.now(), true))) { const accountId = preferredGenericSessionAccount(config, route.providerName, genericSessionKey, route.modelId) ?? captureOAuthAccountSelection(route.providerName)?.accountId; @@ -986,7 +1008,11 @@ export async function prepareResponsesTransport( passiveKeyPhysicalFetch(distributedKeyPhysicalFetch(anthropicBillingPhysicalFetch(fetchImpl, route.providerName, route.provider), route.providerName, route.provider, target), config, route.providerName, route.provider), input, { ...physicalInit, redirect: "manual", - ...physicalSendObserverInit(observeAutoPause, destination instanceof Request ? destination.url : String(destination), deferredCopilotNote ? commitKeyAttemptSend : undefined), + ...physicalSendObserverInit(observeAutoPause, destination instanceof Request ? destination.url : String(destination), () => { + if (deferredCopilotNote) commitKeyAttemptSend(); + if (binding?.kind === "oauth") noteAgentRadarAccountDispatch(logCtx, binding.snapshot.accountId, + anthropicInstance ? pin ? "pin" : anthropicSelectionReason ?? "active" : undefined, anthropicSessionKey); + }), ...(hasVertexRequestAuthority(wireRequest) ? physicalSendGuardInit((finalTarget, finalHeaders) => { if (!selectionIsCurrent(requestBindings.get(wireRequest)) || !vertexRequestAuthorityIsCurrent(wireRequest, finalTarget, finalHeaders)) throw new Error("Vertex credential authority changed before dispatch"); }) : {}) }, @@ -1086,7 +1112,19 @@ export async function prepareResponsesTransport( ? JSON.stringify([resolveContextPrincipal(req, config, options.admission) ?? "loopback", oauthSessionKey]) : null; if (route.provider.authMode === "oauth") { try { - if (anthropicInstance !== undefined && anthropicRouting!.isAnthropicAccountPoolEnabled(config)) { + if (pin) { + const admitted = await commitResolvedOAuthSelection(await pinnedSnapshot()); + if (!admitted) return markProxyError(formatErrorResponse(401, "authentication_error", "Pinned OAuth account is unavailable")); + replayOAuthCredentialSnapshot = { accountId: admitted.accountId, generation: admitted.generation }; + if (isOAuth401ReplayProvider) sentOAuthSnapshot = admitted; + route.provider = { ...route.provider, apiKey: admitted.accessToken }; + stampOAuthAccountLabel(logCtx, route.providerName, route.provider, admitted.accountId); + if (anthropicInstance) { + anthropicPoolAccountId = admitted.accountId; + stampAnthropicAccountForLog(logCtx, anthropicInstance, admitted.accountId, config); + } else genericFailoverAccountId = admitted.accountId; + if (hasPassiveAccountQuota(route.providerName)) passiveQuotaWriterGeneration = captureConfigGeneration(); + } else if (anthropicInstance !== undefined && anthropicRouting!.isAnthropicAccountPoolEnabled(config)) { const routeResult = resolveAnthropicModelRouteForInstance(anthropicInstance, config, route.modelId); if (routeResult.error) return markProxyError(formatErrorResponse(400, "invalid_request_error", "Invalid Anthropic model routes")); anthropicRouteDecision = routeResult.decision; @@ -1106,9 +1144,10 @@ export async function prepareResponsesTransport( const admitted = await commitResolvedOAuthSelection(await anthropicRouting!.getAnthropicPoolAccessSnapshot(selection.accountId), true, selection.reason); if (!admitted) return markProxyError(formatErrorResponse(409, "conflict_error", "OAuth account selection changed; retry the request")); anthropicPoolAccountId = admitted.accountId; + anthropicSelectionReason = selection.reason; if (anthropicRouteDecision) console.info(`[${anthropicInstance}-pool] route:#${anthropicRouteDecision.position} ${selection.reason}`); route.provider = { ...route.provider, apiKey: admitted.accessToken }; - logCtx.provider = formatAnthropicProviderForLog(anthropicInstance!, admitted.accountId, config); + stampAnthropicAccountForLog(logCtx, anthropicInstance!, admitted.accountId, config); } else { // Prefer the account with known headroom BEFORE the first attempt. Rotation alone // only reacts to a 429, so a turn could open on an account a previous probe already @@ -1424,13 +1463,13 @@ export async function prepareResponsesTransport( fundingAccountId, ...eligibleFailoverAccounts(route.providerName, Date.now(), classifyModelFamilyForQuota(route.providerName, route.modelId)), ]).size : 0; - const fundedAccounts = Math.min(genericRosterSize, GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST); + const fundedAccounts = Math.min(genericRosterSize, oauthMaxAccountsPerRequest()); const genericFailoverLimit = Math.max(GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST, fundedAccounts - 1); const anthropicRosterSize = anthropicRouting && anthropicPoolAccountId ? new Set([ anthropicPoolAccountId, ...anthropicRouting.getEligibleAnthropicAccounts(Date.now(), route.modelId), ]).size : 0; - const anthropicFundedAccounts = Math.min(anthropicRosterSize, GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST); + const anthropicFundedAccounts = Math.min(anthropicRosterSize, oauthMaxAccountsPerRequest()); const anthropicFailoverLimit = Math.max(ANTHROPIC_POOL_MAX_FAILOVERS_PER_REQUEST, anthropicFundedAccounts - 1); if (fundingAccountId || anthropicPoolAccountId) expandInferenceOAuthSendBudget(options.sendBudget, Math.max(fundedAccounts, anthropicFundedAccounts)); diff --git a/src/server/responses/sidecar-execution.ts b/src/server/responses/sidecar-execution.ts index 370b682c297..11dcae463d3 100644 --- a/src/server/responses/sidecar-execution.ts +++ b/src/server/responses/sidecar-execution.ts @@ -48,6 +48,7 @@ import { emptyCompletionRetryEnabled } from "./empty-completion-guard"; import { anthropicRoutingFor, formatAnthropicProviderForLog, + stampAnthropicAccountForLog, } from "../../oauth/anthropic-routing"; import { resolveWireProtocolOverride } from "../adapter-resolve"; import { bindRouteReasoningReplayScope, adapterNeedsForcedContinuation, adapterNeedsToolCallContinuation } from "./core-replay"; @@ -193,6 +194,7 @@ export async function executeResponsesSidecars( // request-scoped state only; preflight keeps fixed policy separate from user-role advice // before the worker is dispatched. Registration seam: the core path sees only // `parsed._advisorGuard`; only an activated factory can load the optional implementation. + if (options.agentRadarAccount && config.advisor?.enabled) return formatErrorResponse(400, "invalid_request_error", "AgentRadar pinned requests cannot use helper models"); const advisorPlan = options.advisorInternal === true ? null : await createRegisteredAdvisorPlan({ @@ -239,6 +241,9 @@ export async function executeResponsesSidecars( // Replayed search pairs call the synthetic web_search tool only the sidecar declares (#7269). if (!wsPlan) stripReplayedSearchPairs(parsed); const vidPlan = !routedCompaction ? await planVideoBridge(config, parsed, route.provider) : undefined; + if (options.agentRadarAccount && ((wsPlan && !(options.agentRadarAccount.provider === "claude" && wsPlan.backend === "anthropic")) || imgPlan || vidPlan)) { + return formatErrorResponse(400, "invalid_request_error", "AgentRadar pinned requests cannot use helper models"); + } // Hosted loops rebuild per iteration from spread copies; they share one per-message effort turn (#6895). if (wsPlan || imgPlan || vidPlan) shareAnthropicPerMessageEffortTurn(parsed); const canRunWebSearch = !!wsPlan && !transportState.adapter.runTurn; @@ -273,7 +278,7 @@ export async function executeResponsesSidecars( ): Promise<{ adapter: ProviderAdapter; recoveryKind: AttemptRecoveryKind } | null> => { const antigravityValidationResponse = canRunWebSearch && route.providerName === "google-antigravity" && route.provider.authMode === "oauth" && originalResponse?.status === 403; - if (route.providerName !== "kiro" && !(anthropicInstance && (originalResponse?.status === 403 || originalResponse?.status === 401)) + if (route.providerName !== "kiro" && !(anthropicInstance && [403, 401, 400].includes(originalResponse?.status ?? 0)) && !antigravityValidationResponse && originalResponse && originalResponse.status !== 429) return null; let antigravityVerification = false; if (antigravityValidationResponse) { @@ -417,13 +422,16 @@ export async function executeResponsesSidecars( "auth-recovery", `${route.providerName}|${route.modelId}|sidecar-anthropic-429`, ); - const nextAccountId = await rotateAnthropicAccountOnResponseForInstance(anthropicInstance, - originalResponse ?? new Response(null, { status: 429, headers: responseHeaders ?? (retryAfter ? { "retry-after": retryAfter } : undefined) }), { - config, accountId: transportState.anthropicPoolAccountId, sessionKey: anthropicSessionKey, - model: route.modelId, requestKey: transportState, decision: transportState.anthropicRouteDecision, currentDecision: currentAnthropicRouteDecision, signal: options.abortSignal, - allow429Recovery: !sidecarHasCommittedOutput(), allowAccountRefusal: !sidecarHasCommittedOutput(), - canRetry: hop.allowed && transportState.anthropicPoolFailovers < transportState.anthropicFailoverLimit, - }); + let nextAccountId: string | null; + try { + nextAccountId = await rotateAnthropicAccountOnResponseForInstance(anthropicInstance, + originalResponse ?? new Response(null, { status: 429, headers: responseHeaders ?? (retryAfter ? { "retry-after": retryAfter } : undefined) }), { + config, accountId: transportState.anthropicPoolAccountId, sessionKey: anthropicSessionKey, + model: route.modelId, requestKey: transportState, decision: transportState.anthropicRouteDecision, currentDecision: currentAnthropicRouteDecision, signal: options.abortSignal, + allow429Recovery: !sidecarHasCommittedOutput(), allowAccountRefusal: !sidecarHasCommittedOutput(), + canRetry: hop.allowed && transportState.anthropicPoolFailovers < transportState.anthropicFailoverLimit, + }); + } catch (error) { hop.permit?.release(); throw error; } if (!nextAccountId) { hop.permit?.release(); return null; @@ -439,7 +447,7 @@ export async function executeResponsesSidecars( transportState.anthropicPoolAccountId = admitted.accountId; transportState.anthropicPoolFailovers += 1; route.provider = { ...route.provider, apiKey: admitted.accessToken }; - logCtx.provider = formatAnthropicProviderForLog(anthropicInstance, admitted.accountId, config); + stampAnthropicAccountForLog(logCtx, anthropicInstance, admitted.accountId, config); } catch { hop.permit?.release(); return null; @@ -596,6 +604,12 @@ export async function executeResponsesSidecars( connectTimeoutMs: config.connectTimeoutMs ?? 200_000, stallTimeoutSec: config.stallTimeoutSec, proxyErrorContext: logCtx, waitForRequestSlot: imageProviderFetch.waitForPacing, + // A runTurn adapter sends inside the turn lease the loop reserved; its own sends are turn-scoped. + runTurnFetch: pacingSlot => providerFetch(route.provider, options.codexWsRuntimeIdentity, { + providerName: route.providerName, modelId: route.modelId, + pacingSlotAcquired: true, pacingSlot, turnScopedPacing: true, + ...sidecarBudget.fetchOptions(transportState.adapter), + }), fetchImpl: imageProviderFetch.unpacedFetch ?? imageProviderFetch, fetchForRequest: (request, iterParsed, adapter, recovery) => { const fetch = providerFetch(route.provider, options.codexWsRuntimeIdentity, { diff --git a/src/server/responses/terminal-guard.ts b/src/server/responses/terminal-guard.ts index 44c02706b97..ca0d2c99f1e 100644 --- a/src/server/responses/terminal-guard.ts +++ b/src/server/responses/terminal-guard.ts @@ -209,6 +209,12 @@ export function mergeUsage(first: OcxUsage | undefined, second: OcxUsage | undef inputTokens, outputTokens, totalTokens: inputTokens + outputTokens, + // The context window the final round saw is the larger one, never their sum (images/loop.ts). + ...(first.contextTotalTokens !== undefined || second.contextTotalTokens !== undefined + ? { contextTotalTokens: Math.max(first.contextTotalTokens ?? 0, second.contextTotalTokens ?? 0) } : {}), + ...(first.anthropicServerToolUse || second.anthropicServerToolUse ? { anthropicServerToolUse: Object.fromEntries( + [...new Set([...Object.keys(first.anthropicServerToolUse ?? {}), ...Object.keys(second.anthropicServerToolUse ?? {})])] + .map(name => [name, (first.anthropicServerToolUse?.[name] ?? 0) + (second.anthropicServerToolUse?.[name] ?? 0)])) } : {}), ...(cachedInputTokens !== undefined ? { cachedInputTokens } : {}), ...(cacheReadInputTokens !== undefined ? { cacheReadInputTokens } : {}), ...(cacheCreationInputTokens !== undefined ? { cacheCreationInputTokens } : {}), diff --git a/src/sidecar/anthropic-binding.ts b/src/sidecar/anthropic-binding.ts index 83ba792830a..28fb3193a30 100644 --- a/src/sidecar/anthropic-binding.ts +++ b/src/sidecar/anthropic-binding.ts @@ -1,3 +1,4 @@ +import { agentRadarPin } from "../lib/agentradar-account-pin"; import type { OcxConfig } from "../types"; import type { AnthropicInstanceId } from "../providers/anthropic-instance-id"; import { configuredAnthropicInstance } from "../providers/anthropic-instance"; @@ -36,7 +37,9 @@ export async function resolveAnthropicHelperSnapshot(config: OcxConfig, instance for (let attempt = 0; attempt < 3; attempt++) { const route = resolveAnthropicModelRouteForInstance(instance, config, model); if (route.error) throw new AnthropicHelperUnavailableError(instance); - const selection = routing.resolveAnthropicAccountForSession(null, config, Date.now(), route.decision, model); + const pin = agentRadarPin(config); + if (pin && (pin.provider !== "claude" || instance !== "anthropic")) throw new AnthropicHelperUnavailableError(instance); + const selection = pin ? { accountId: pin.accountId } : routing.resolveAnthropicAccountForSession(null, config, Date.now(), route.decision, model); if (!selection.accountId) throw new AnthropicHelperUnavailableError(instance); const snapshot = await routing.getAnthropicPoolAccessSnapshot(selection.accountId); assertConfigured(); @@ -56,6 +59,8 @@ export async function fetchAnthropicHelper( if (instance !== "anthropic" && instance !== "anthropic2") throw new Error("Invalid Anthropic helper instance"); const row = config.providers[instance]; const live = getAccountCredentialWithStatus(instance, snapshot.accountId); + const pin = agentRadarPin(config); + if (pin && (pin.provider !== "claude" || instance !== "anthropic" || snapshot.accountId !== pin.accountId)) throw new AnthropicHelperUnavailableError(instance); if (!row || row.disabled || row.authMode !== "oauth" || row.adapter !== "anthropic" || configuredAnthropicInstance(config, instance) !== instance || row.baseUrl !== capturedTarget || url !== anthropicHelperMessagesUrl(capturedTarget) || !live || live.paused || live.needsReauth diff --git a/src/types/request.ts b/src/types/request.ts index d516cb095b2..5d76efd805e 100644 --- a/src/types/request.ts +++ b/src/types/request.ts @@ -605,6 +605,8 @@ export interface OcxUsage { * `contextTotalTokens - outputTokens` so output is never added to an absolute checkpoint twice. */ contextTotalTokens?: number; + /** Anthropic server-tool request counters (`usage.server_tool_use`), keyed by counter name. */ + anthropicServerToolUse?: Record; totalTokens?: number; cachedInputTokens?: number; cacheReadInputTokens?: number; diff --git a/src/usage/log.ts b/src/usage/log.ts index ed3ae76f6fe..f7dc2a6383b 100644 --- a/src/usage/log.ts +++ b/src/usage/log.ts @@ -17,6 +17,8 @@ import { type PersistedJevDecisionV1, } from "./jev-stats"; import { normalizeAttemptDeliverySummary } from "./attempt-delivery"; +import { normalizeServerToolUsage, physicalAttemptTelemetry } from "./physical-attempts"; +export { normalizeServerToolUsage, physicalUsageAttempts } from "./physical-attempts"; import { isRequestCloseReason, isRequestTerminalStatus, @@ -162,6 +164,10 @@ export interface PersistedUsageAttempt { /** TTFT relative to THIS attempt's start (WP4); unset for non-streaming/tool-only. */ firstOutputMs?: number; sendCount: number; + /** Physical-send telemetry: first send time (ms), sent prompt-cache TTL (5m|1h), recovery occurrences. */ + firstSendAt?: number; + promptCacheTtlMs?: number; + recoveryCount?: number; recoveryKinds: AttemptRecoveryKind[]; /** * Recoveries this attempt was eligible for and did not make. Absent on ordinary attempts so @@ -336,6 +342,8 @@ export interface PersistedUsageEntry extends LinkRequestLogCarrier { accountLogLabel?: CodexUsageAccountLogLabel; /** Anthropic pool account of the row's provider (#8383); never read by account summaries. */ accountTag?: `p${string}`; + /** A combo moved this logical request to another target (routing analytics `comboFailover*`). */ + comboTargetAdvanced?: true; /** Best-effort chat/session correlation for Logs grouping (#330). */ conversationId?: string; resolvedModel?: string; @@ -623,6 +631,7 @@ export function usageStatusForFinalLog(usage: OcxUsage | undefined): UsageStatus function normalizeUsageValue(usage: OcxUsage | undefined): OcxUsage | undefined { if (!usage) return undefined; + const anthropicServerToolUse = normalizeServerToolUsage(usage.anthropicServerToolUse); return { inputTokens: usage.inputTokens, outputTokens: usage.outputTokens, @@ -640,6 +649,7 @@ function normalizeUsageValue(usage: OcxUsage | undefined): OcxUsage | undefined ...(typeof usage.cacheCreation1hInputTokens === "number" ? { cacheCreation1hInputTokens: usage.cacheCreation1hInputTokens } : {}), ...(typeof usage.reasoningOutputTokens === "number" ? { reasoningOutputTokens: usage.reasoningOutputTokens } : {}), ...(isNonNegativeFiniteNumber(usage.providerCredits) ? { providerCredits: usage.providerCredits } : {}), + ...(anthropicServerToolUse ? { anthropicServerToolUse } : {}), ...(usage.estimated ? { estimated: true } : {}), }; } @@ -767,14 +777,7 @@ function normalizeUsageAttempt(raw: unknown): PersistedUsageAttempt | null { || !USAGE_STATUSES.has(attempt.usageStatus as UsageStatus)) { return null; } - if ("inputTokenEstimate" in attempt - && !isNonNegativeFiniteNumber(attempt.inputTokenEstimate)) return null; - if ("firstOutputMs" in attempt - && !isNonNegativeFiniteNumber(attempt.firstOutputMs)) return null; - if ("totalTokens" in attempt - && !isNonNegativeFiniteNumber(attempt.totalTokens)) return null; const usage = "usage" in attempt ? normalizeAttemptUsage(attempt.usage) : undefined; - if ("usage" in attempt && usage === null) return null; const tierOutcome = "tierOutcome" in attempt ? normalizeAttemptTierOutcome(attempt.tierOutcome) : undefined; @@ -817,6 +820,7 @@ function normalizeUsageAttempt(raw: unknown): PersistedUsageAttempt | null { ? { firstOutputMs: attempt.firstOutputMs } : {}), sendCount: attempt.sendCount as number, + ...physicalAttemptTelemetry(attempt), recoveryKinds, ...(recoveryWithheld.length ? { recoveryWithheld } : {}), usageStatus: attempt.usageStatus as UsageStatus, @@ -1094,6 +1098,7 @@ function normalizeUsageEntry(entry: PersistedUsageEntry): PersistedUsageEntry { ? { accountLogLabel: entry.accountLogLabel } : {}), ...(isAnthropicAccountTag(entry.accountTag) ? { accountTag: entry.accountTag } : {}), + ...(entry.comboTargetAdvanced === true ? { comboTargetAdvanced: true } : {}), ...(typeof entry.conversationId === "string" && entry.conversationId.trim() ? { conversationId: entry.conversationId.trim().slice(0, 128) } : {}), diff --git a/src/usage/physical-attempts.ts b/src/usage/physical-attempts.ts new file mode 100644 index 00000000000..bdd46870bde --- /dev/null +++ b/src/usage/physical-attempts.ts @@ -0,0 +1,54 @@ +/** Physical-attempt identity and its optional telemetry, shared by the ledger reader and every report. */ + +const isCount = (value: unknown): value is number => typeof value === "number" && Number.isFinite(value) && value >= 0; + +/** One row per real upstream attempt: locally answered, unsent and duplicate-ordinal attempts are not. */ +export function physicalUsageAttempts< + T extends { ordinal?: number; sendCount?: number; locallyAnswered?: boolean }, +>(attempts: readonly T[]): T[] { + if (!Array.isArray(attempts)) return []; + const seen = new Set(); + return attempts.filter(attempt => { + if (!attempt || typeof attempt !== "object" || Array.isArray(attempt)) return false; + if (attempt.locallyAnswered === true || attempt.sendCount === 0) return false; + if (!Number.isInteger(attempt.ordinal)) return true; + if (seen.has(attempt.ordinal!)) return false; + seen.add(attempt.ordinal!); + return true; + }); +} + +/** + * The valid physical-send telemetry of a persisted attempt. Identity decides whether an attempt is + * trustworthy; a malformed optional field costs only that field, since dropping the attempt would + * delete a physical send from every report that sums them. + */ +export function physicalAttemptTelemetry(attempt: Record): { + firstSendAt?: number; promptCacheTtlMs?: number; recoveryCount?: number; +} { + return { + ...(isCount(attempt.firstSendAt) ? { firstSendAt: attempt.firstSendAt } : {}), + ...(attempt.promptCacheTtlMs === 300_000 || attempt.promptCacheTtlMs === 3_600_000 + ? { promptCacheTtlMs: attempt.promptCacheTtlMs } + : {}), + ...(Number.isInteger(attempt.recoveryCount) && isCount(attempt.recoveryCount) ? { recoveryCount: attempt.recoveryCount } : {}), + }; +} + +const MAX_SERVER_TOOL_USAGE_ENTRIES = 64; +const MAX_SERVER_TOOL_USAGE_KEY_LENGTH = 128; +const UNSAFE_SERVER_TOOL_USAGE_KEYS = new Set(["__proto__", "prototype", "constructor"]); + +/** Bounded Anthropic server-tool counters (`web_search_requests`, ...): plain object, finite counts. */ +export function normalizeServerToolUsage(value: unknown): Record | undefined { + if (!value || typeof value !== "object" || Array.isArray(value)) return undefined; + const prototype = Object.getPrototypeOf(value); + if (prototype !== Object.prototype && prototype !== null) return undefined; + const entries: [string, number][] = []; + for (const [key, count] of Object.entries(value)) { + if (entries.length >= MAX_SERVER_TOOL_USAGE_ENTRIES) break; + if (!key || key.length > MAX_SERVER_TOOL_USAGE_KEY_LENGTH || UNSAFE_SERVER_TOOL_USAGE_KEYS.has(key) || !isCount(count)) continue; + entries.push([key, count]); + } + return entries.length > 0 ? Object.fromEntries(entries) : undefined; +} diff --git a/src/usage/summary.ts b/src/usage/summary.ts index 4884c59f294..6164e9ee925 100644 --- a/src/usage/summary.ts +++ b/src/usage/summary.ts @@ -8,11 +8,13 @@ import { isUnresolvedRequestedModel, usageModelPriceOptions } from "./model-iden import { classifyCacheTelemetryProvenance, isCodexUsageAccountLogLabel, + normalizeServerToolUsage, + physicalUsageAttempts, type CacheTelemetryProvenance, type PersistedUsageEntry, type UsageStatus, } from "./log"; -import { type AttemptCostEstimate, type CostEstimate, estimateAttemptCost, estimateRequestCost, isMeasuredStatus, serviceTierContext, type ServiceTierContext, tokensPerSecond } from "./cost"; +import { type AttemptCostEstimate, type CostEstimate, estimateAttemptCost, estimateRequestCost, isMeasuredStatus, normalizeCostTokens, serviceTierContext, type ServiceTierContext, tokensPerSecond } from "./cost"; /** * Canonical range members. The warm-up loop in the management usage route @@ -261,16 +263,14 @@ export function cacheTokensFromUsage(usage?: PersistedUsageEntry["usage"]): { hasCacheTelemetry: boolean; } { if (!usage) return { read: undefined, creation: undefined, hasCacheTelemetry: false }; - const creation = usage.cacheCreationInputTokens; - const read = typeof usage.cacheReadInputTokens === "number" - ? usage.cacheReadInputTokens - : typeof usage.cachedInputTokens === "number" && typeof creation === "number" - ? Math.max(0, usage.cachedInputTokens - creation) - : usage.cachedInputTokens; const hasCacheTelemetry = typeof usage.cachedInputTokens === "number" || typeof usage.cacheReadInputTokens === "number" || typeof usage.cacheCreationInputTokens === "number"; - return { read, creation, hasCacheTelemetry }; + // The pricing reader's canonical-first decomposition: an implicit read is not reduced by writes, + // and contradictory telemetry (read + write above input under every reading) stays unavailable. + const normalized = hasCacheTelemetry ? normalizeCostTokens(usage) : null; + if (!normalized) return { read: undefined, creation: undefined, hasCacheTelemetry: false }; + return { read: normalized.cacheRead, creation: normalized.cacheWrite, hasCacheTelemetry: true }; } /** @@ -305,7 +305,7 @@ export function calculateCacheHitRate( export function computeEntryCost(entry: PersistedUsageEntry): EntryCostInfo { const tier = serviceTierContext(entry); - if (entry.attempts?.length) { + if (entry.attempts !== undefined) { const attemptEstimates = entry.attempts.map(attempt => estimateAttemptCost({ ...attempt, ...usageModelPriceOptions(entry, attempt) }, undefined, tier) ); @@ -492,7 +492,10 @@ function usageModelKey(providerKey: string, model: string): string { } export function usageAttributions(entry: PersistedUsageEntry): UsageAttribution[] { - if (!entry.attempts?.length) { + // `=== undefined`, not `!length`: after `normalizePhysicalEntry` an EMPTY list means no attempt + // reached upstream and the row's own totals were stripped, so falling back to the row would + // invent a request out of stale parent usage. Absent attempts is the older, different case. + if (entry.attempts === undefined) { return [{ requestId: entry.requestId, provider: entry.provider, @@ -526,7 +529,7 @@ export function usageAttributions(entry: PersistedUsageEntry): UsageAttribution[ }); } -function projectedComboUsage( +export function projectedComboUsage( attempts: readonly NonNullable[number][], ): { usage?: PersistedUsageEntry["usage"]; totalTokens?: number } { let inputTokens = 0; @@ -543,10 +546,20 @@ function projectedComboUsage( let totalTokens = 0; let hasTotalTokens = false; let estimated = false; + let contextTotalTokens: number | undefined; + const anthropicServerToolUse: Record = {}; for (const attempt of attempts) { if (attempt.usage) { hasUsage = true; + // A context checkpoint is absolute, so the latest wins; server-tool counters add. + if (typeof attempt.usage.contextTotalTokens === "number" + && Number.isFinite(attempt.usage.contextTotalTokens) && attempt.usage.contextTotalTokens >= 0) { + contextTotalTokens = attempt.usage.contextTotalTokens; + } + for (const [name, value] of Object.entries(normalizeServerToolUsage(attempt.usage.anthropicServerToolUse) ?? {})) { + anthropicServerToolUse[name] = (anthropicServerToolUse[name] ?? 0) + value; + } inputTokens += attempt.usage.inputTokens; outputTokens += attempt.usage.outputTokens; const cache = cacheTokensFromUsage(attempt.usage); @@ -574,13 +587,16 @@ function projectedComboUsage( } if (!hasUsage && !hasTotalTokens) return {}; + const boundedServerToolUse = normalizeServerToolUsage(anthropicServerToolUse); const usage = hasUsage ? { inputTokens, outputTokens, + ...(contextTotalTokens !== undefined ? { contextTotalTokens } : {}), ...(hasCacheTelemetry ? { cachedInputTokens, cacheReadInputTokens, cacheCreationInputTokens } : {}), ...(hasCacheWrite1h ? { cacheCreation1hInputTokens } : {}), ...(hasReasoningTelemetry ? { reasoningOutputTokens } : {}), + ...(boundedServerToolUse ? { anthropicServerToolUse: boundedServerToolUse } : {}), ...(estimated ? { estimated: true } : {}), } : undefined; @@ -603,6 +619,8 @@ function addTokens( totals: UsageSummaryTotals, entry: Pick, ): void { + // A stored total without usage detail is still tokens spent. + totals.totalTokens += usageDisplayTotalTokens(entry.usage, entry.totalTokens) ?? 0; if (!entry.usage) return; totals.inputTokens += entry.usage.inputTokens; totals.outputTokens += entry.usage.outputTokens; @@ -613,7 +631,6 @@ function addTokens( } if (typeof creation === "number") totals.cacheCreationInputTokens += creation; if (typeof entry.usage.reasoningOutputTokens === "number") totals.reasoningOutputTokens += entry.usage.reasoningOutputTokens; - totals.totalTokens += usageDisplayTotalTokens(entry.usage, entry.totalTokens) ?? 0; } /** @@ -831,6 +848,29 @@ function requestStatusFact(status: UsageStatus): number { return REQUEST_UNREPORTED; } +/** + * The row as its real upstream sends: locally answered, unsent and duplicate-ordinal attempts are + * dropped, legacy root usage with no per-attempt usage lands on the final physical attempt, and the + * row's totals and status are recomputed from those attempts. Rows without `attempts` are unchanged. + */ +export function normalizePhysicalEntry(entry: PersistedUsageEntry): PersistedUsageEntry { + if (entry.attempts === undefined) return entry; + const attempts = physicalUsageAttempts(entry.attempts).map(attempt => ({ ...attempt })); + const finalAttempt = attempts.at(-1); + if (finalAttempt && entry.usage && !attempts.some(attempt => attempt.usage !== undefined)) { + finalAttempt.usage = entry.usage; + finalAttempt.usageStatus = entry.usageStatus; + if (entry.totalTokens !== undefined) finalAttempt.totalTokens = entry.totalTokens; + } + const { usage: _usage, totalTokens: _totalTokens, ...rest } = entry; + return { + ...rest, + attempts, + usageStatus: statusFromRequestFacts(attempts.reduce((facts, attempt) => facts | requestStatusFact(attempt.usageStatus), 0)), + ...projectedComboUsage(attempts), + }; +} + function statusFromRequestFacts(facts: number): UsageStatus { const statuses = facts & REQUEST_STATUS_MASK; if (statuses === REQUEST_UNSUPPORTED) return "unsupported"; @@ -1091,7 +1131,7 @@ function projectedEntryForFilter( ): { entry: PersistedUsageEntry; comboOverlap: boolean } | null { if (filter.apiKeyId !== null && entry.apiKeyId !== filter.apiKeyId) return null; if (filter.sourceFilter && !matchesUsageSource(entry, filter.sourceFilter, filter.configuredApiKeyIdSet)) return null; - if (!entry.attempts?.length) { + if (entry.attempts === undefined) { const identity = usageModelIdentity(entry.provider, entry.model, entry.resolvedModel); return filterMatchesAttribution(filter, entry.provider, identity.model) ? { entry, comboOverlap: false } @@ -1454,6 +1494,7 @@ class StreamingUsageSummaryAccumulator implements UsageSummaryAccumulator { ): void { if (attribution.hasUnresolvedRequestedModel) breakdown.hasUnresolvedRequestedModel = true; breakdown.attemptCount += 1; + breakdown.summaryTotalTokens += usageDisplayTotalTokens(attribution.usage, attribution.totalTokens) ?? 0; if (attribution.usage) { breakdown.inputTokens += attribution.usage.inputTokens; breakdown.outputTokens += attribution.usage.outputTokens; @@ -1475,7 +1516,6 @@ class StreamingUsageSummaryAccumulator implements UsageSummaryAccumulator { } if (typeof read === "number") breakdown.cacheReadInputTokens += read; if (typeof creation === "number") breakdown.cacheCreationInputTokens += creation; - breakdown.summaryTotalTokens += usageDisplayTotalTokens(attribution.usage, attribution.totalTokens) ?? 0; } breakdown.dayTotalTokens += usageDisplayTotalTokens(attribution.usage, attribution.totalTokens) ?? 0; if (estimate) breakdown.estimatedCostUsd = (breakdown.estimatedCostUsd ?? 0) + estimate.cost.total; @@ -1580,6 +1620,7 @@ class StreamingUsageSummaryAccumulator implements UsageSummaryAccumulator { this.estimatedRetainedBytes += ESTIMATED_BREAKDOWN_BYTES + label.length * 2; } account.attemptCount += 1; + account.totalTokens += usageDisplayTotalTokens(attribution.usage, attribution.totalTokens) ?? 0; if (!attribution.usage || !isMeasuredStatus(attribution.usageStatus)) { account.unmeteredAttempts += 1; return label; @@ -1595,7 +1636,6 @@ class StreamingUsageSummaryAccumulator implements UsageSummaryAccumulator { if (typeof attribution.usage.reasoningOutputTokens === "number") { account.reasoningOutputTokens += attribution.usage.reasoningOutputTokens; } - account.totalTokens += usageDisplayTotalTokens(attribution.usage, attribution.totalTokens) ?? 0; if (estimate) { account.pricedAttempts += 1; account.estimatedCostUsd = (account.estimatedCostUsd ?? 0) + estimate.cost.total; @@ -1620,7 +1660,8 @@ class StreamingUsageSummaryAccumulator implements UsageSummaryAccumulator { if (!this.filter?.sourceFilter) this.noteSnapshot(sourceEntry); if (this.window && (!Number.isFinite(sourceEntry.timestamp) || sourceEntry.timestamp < this.window.since || sourceEntry.timestamp > this.window.until)) return; - const projected = this.filter ? projectedEntryForFilter(sourceEntry, this.filter) : { entry: sourceEntry, comboOverlap: false }; + const physicalEntry = normalizePhysicalEntry(sourceEntry); + const projected = this.filter ? projectedEntryForFilter(physicalEntry, this.filter) : { entry: physicalEntry, comboOverlap: false }; if (!projected) return; if (this.filter?.sourceFilter) this.noteSnapshot(sourceEntry); this.comboOverlap ||= projected.comboOverlap; diff --git a/src/usage/telemetry-contract.ts b/src/usage/telemetry-contract.ts index e6056ab1dcf..93a9838d3d9 100644 --- a/src/usage/telemetry-contract.ts +++ b/src/usage/telemetry-contract.ts @@ -33,6 +33,7 @@ export const ATTEMPT_RECOVERY_KIND_ROSTER = Object.freeze([ "rate-limit-429", "anthropic-oauth-429", "oauth-account-429", + "codex-account-retry", "oauth-account-402", "image-413", "console-go-upload-retry", diff --git a/src/web-search/loop.ts b/src/web-search/loop.ts index 9ae4bdf2a0e..9b63c6b96e0 100644 --- a/src/web-search/loop.ts +++ b/src/web-search/loop.ts @@ -791,7 +791,7 @@ export async function runWithWebSearch(deps: WebSearchLoopDeps): Promise call.kind === "advisor")) yield { type: "assistant_boundary" }; } catch (e) { - // The bridge reports no usage for either error; the iterations already sent still count. - if (hiddenUsage) deps.onUsage?.(hiddenUsage); if (isTranslatorBudgetExceededError(e)) { + // The bridge reports an overflow terminal's own usage, so the legs already sent move + // onto it: the client sees them and onUsage counts them exactly once. + const usage = hiddenUsage; + hiddenUsage = undefined; yield { type: "error", status: 502, errorType: "upstream_error", code: e.code, message: "upstream translation buffer exceeded the safe limit", + ...(usage ? { usage } : {}), }; } else { + // The bridge reports no usage for these errors; the iterations already sent still count. + if (hiddenUsage) deps.onUsage?.(hiddenUsage); if (e instanceof SendBudgetExhaustedError) yield { type: "error", status: 429, errorType: "rate_limit_error", code: e.code, message: e.message, retryable: false }; // A later iteration's overflow ends the turn as the main path's terminal diff --git a/structure/data-planes/protocol-paths.md b/structure/data-planes/protocol-paths.md index 7dbdf06afc1..2dd67f345c6 100644 --- a/structure/data-planes/protocol-paths.md +++ b/structure/data-planes/protocol-paths.md @@ -376,7 +376,7 @@ probe uses the bridge's credential-free affinity headers, existing plan and auth short-lived selection lease; uncertain admission keeps the bridge. Without a runnable plan search alone does not decline native Messages. `count_tokens` and the planner lack runtime facts and remain conservative. -Failover combos reuse this lane for eligible concrete Anthropic children, including key-auth compatible endpoints, through the existing `core-combo-native.ts` source/dispatcher seam. OAuth remains first-party only. Only strict failover without a default effort qualifies; policy/JEV/other strategies and bridge-only rewrites retain the bridge. The shared executor inherits the combo's attempt, target budget, prepaid permit and gated callbacks, opening no second tracker/final row. Key source system/messages/tools/cache markers retain the native allowlist, image/tool-id safety; explicit operator TTL wins, otherwise TTL-less combo breakpoints stay unchanged. OAuth keeps its preamble/metadata/tool transformations. Counts/previews still decline logical combos without picking accounts. A source declaring a tool the translated body drops with no out-of-body lane (`hasNativeOnlyTool` in `src/claude/inbound-content-options.ts`: `bash_*`, `text_editor_*`, ...; unrestricted `web_search_*` stays portable, while `web_fetch_*` and a domain- or location-restricted `web_search_*` stay portable only to a first-party Anthropic bridge child; a typed `advisor_*` declaration rides that lane too but is optional, so it never makes a child unpickable) makes every translated child unpickable under any `unrepresentable` policy, so a refused native child surfaces its own failure instead of continuing without the tool; non-combo translated routes are unchanged. [Commit boundaries](../transports/responses-combo-boundaries.md) own failover and preserve upstream read evidence before OAuth framing; `tests/claude-integration/messages-native-combo.test.ts` covers the boundary and mocked prefix preservation, not live cache-hit metrics. +Failover combos reuse this lane for eligible concrete Anthropic children, including key-auth compatible endpoints, through the existing `core-combo-native.ts` source/dispatcher seam. OAuth remains first-party only. Only strict failover without a default effort qualifies; policy/JEV/other strategies and bridge-only rewrites retain the bridge. The shared executor inherits the combo's attempt, target budget, prepaid permit and gated callbacks, opening no second tracker/final row. Key source system/messages/tools/cache markers retain the native allowlist, image/tool-id safety; explicit operator TTL wins, otherwise TTL-less combo breakpoints stay unchanged. OAuth keeps its preamble/metadata/tool transformations. Counts/previews still decline logical combos without picking accounts. A source declaring a tool the translated body drops with no out-of-body lane (`hasNativeOnlyTool` in `src/claude/inbound-content-options.ts`: `bash_*`, `text_editor_*`, ...; unrestricted `web_search_*` stays portable, while `web_fetch_*` and a domain- or location-restricted `web_search_*` stay portable only to a first-party Anthropic bridge child; a typed `advisor_*` declaration rides that lane too but is optional, so it never makes a child unpickable) makes every translated child unpickable under any `unrepresentable` policy, so a refused native child surfaces its own failure instead of continuing without the tool. A non-combo route that stays on the bridge refuses such a tool, or `mcp_servers`, with a local 400 before any send (`src/server/claude-messages.ts`) instead of dropping it. [Commit boundaries](../transports/responses-combo-boundaries.md) own failover and preserve upstream read evidence before OAuth framing; `tests/claude-integration/messages-native-combo.test.ts` covers the boundary and mocked prefix preservation, not live cache-hit metrics. AgentRadar hosted contract (`OPENCODEX_AGENTRADAR=1`, the .27 source-replay behaviour carried onto this lane): a route whose Messages URL is exactly `https://api.anthropic.com/v1/messages` takes the lane whatever the rollout switches say, and a trusted account pin may take it (Anthropic OAuth, non-combo): `resolveNativeOAuthBindingForInstance` then serves exactly the pinned account, never commits the shared selection, and the pinned config makes refusal rotation return null. `buildAnthropicMessagesPassthroughRequest({ exactSource })` sends the caller's body as sent with the wire model (plus the OAuth identity line only when no identity block exists) and every caller beta (caller, operator, OAuth order): no allowlist, TTL default, account metadata or tool-name rewrite. The lane relays upstream bytes and headers without model echo or error filtering, answers a streaming caller only from SSE and a buffered caller only from strict-UTF-8 complete Messages JSON (a root JSON error keeps its bytes with the mapped status), relays a single route's HTTP errors verbatim, and `tapAnthropicSseForLog` relays one validated frame per pull up to the first terminal (root error 502 failed, malformed frame fails before relay, stall 504). `messages-native-preflight.ts` lets a combo child fail over on such relay-made pre-output failures. Off the flag every rule above is upstream's. Covered by `tests/claude-integration/agentradar-hosted-fidelity.test.ts` and `tests/claude-integration/agentradar-hosted-tap.test.ts`. On every native turn a 413 for a request carrying images is rebuilt once with each image one tier smaller (recovery `image-413`), as the bridge adapter retries. Hosted combos also follow .27: a native candidate whose model differs from the caller's is unpickable when the source carries typed tools or a thinking shape the target model refuses (`hostedReplayIncompatible` in `core-combo-native.ts`), `mcp_servers` makes translated children unpickable, a ladder with no carrying candidate answers a 400 with no send, combo children replay server tools natively (no web-search sidecar), Anthropic's `overloaded_error` keeps status 529, and a canonical child's final failure is registered with `markVerbatimComboFailure` (`src/bridge/errors.ts`) so the combo returns its exact bytes. Hosted gateways default Anthropic cache retention to 5 minutes (`defaultAnthropicCacheRetention`). With the switch on, a declined route re-marks its bridge entry with the decline reason (after any `effort-row` / `fast-row`), so the trace says why the bridge was taken; with it off nothing is diff --git a/structure/ops/docs-and-release.md b/structure/ops/docs-and-release.md index e6695097076..404a8017692 100644 --- a/structure/ops/docs-and-release.md +++ b/structure/ops/docs-and-release.md @@ -86,7 +86,7 @@ Manual navigation is defined in `docs-site/astro.config.mjs`. When adding a publ sidebar and either add localized copies or intentionally accept Starlight fallback behavior. Use slugs for local pages, including Platform Support and OS Keychain, so navigation preserves the selected locale. `tests/ci-workflows/docs-spanish-localization.test.ts` guards canonical Spanish coverage, literal provider-key references, protected remote examples, foreground quickstart alternatives, separate-terminal and Bash/stdin pairing recipes, Pi reasoning metadata, and the data/management credential and admission-header distinctions across shipped locale copies, including empty-header bearer fallback. `tests/ci-workflows/docs-remote-hub-claims.test.ts` validates Docker initialization against the image seed and configuration schema; `tests/ci-workflows/docs-bun-source-requirement.test.ts` checks current bundle claims against the package pin. Provider preset totals are recounted from the current registry when a preset lands. The -documented split is 108 total: 87 key-based, 17 OAuth, three local, and one default +documented split is 109 total: 88 key-based, 17 OAuth, three local, and one default ChatGPT-forward preset. The English and Spanish API-key catalog pages, every other locale's provider guide, and every locale's quickstart carry the same counts. Section 3 of the provider guide lives on its own page, `docs-site/src/content/docs/guides/api-key-catalog.md` (with its Spanish copy), so the guide stays under the file-size ratchet's 2,000-line default; the guide keeps a section 3 stub that links every moved heading anchor, and other locales keep section 3 in their provider guide until they add a translated catalog page. diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index 1f5de472a41..3c48349660b 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -92,7 +92,7 @@ Kimi Coding's Chat, API-key, and optional Responses presets consume the same mod The Responses preset shares the `kimi` OAuth account and Coding endpoint, keeps Chat as the featured default, and enables adjacent tool-result repair on its Responses wire. Its metadata alias is generated from the registry; sharing authentication does not implicitly share a usage-price namespace. The separate `moonshot` API preset defaults to `kimi-k3.1-pro`; `src/providers/model-rename-migration.ts` moves discontinued K2.5 selections on official destinations to it, drops stale no-reasoning classification, and refreshes exact old 256K/empty reasoning seeds to 1M and `low`/`high`/`max` with the API `none` omission map. Explicit successor metadata and Custom destinations keep precedence. Roster repair accepts the literal pre-K3.1 seed and its post-rename form, with metadata additions measured against the original seed. K2.6 and K2.7 Code/highspeed stay seeded for existing API users. -Kimi quota in `src/providers/quota/vendor-probes-key.ts` also reads monthly `usages` at the root, under `data`, or under `quota` (including `data.quota`): `limit_month_total` / `monthTotal` precedes the `limit_month_code` / `monthCode` fallback. Containers are tried in root, `data`, `quota`, then `data.quota` order until a monthly row parses; empty or unusable earlier containers do not mask later data, and monthly lookup retains the original envelope even when legacy windows unwrap `data`. Rows prefer limit/used or remaining arithmetic, then fractional `used_ratio` / `usedRatio` in `[0, 1]`, then direct percentages; out-of-range ratios fall through and reset times remain optional. Missing monthly data stays absent, and legacy 5-hour, weekly and total subscription-credit windows stay separate. The probe calls `/usages` on the configured inference origin only when it is exactly `https://api.kimi.com/coding/v1`, or `https://api.kimi.ai/coding/v1` for key auth (#8332), so the bearer never reaches a host it was not already sent to; account-login tokens stay on `api.kimi.com`. The parser is shared by account-login and API-key probes; credential and destination guards are unchanged. +Kimi quota in `src/providers/quota/vendor-probes-key.ts` also reads monthly `usages` at the root, under `data`, or under `quota` (including `data.quota`): `limit_month_total` / `monthTotal` precedes the `limit_month_code` / `monthCode` fallback. Containers are tried in root, `data`, `quota`, then `data.quota` order until a monthly row parses; empty or unusable earlier containers do not mask later data, and monthly lookup retains the original envelope even when legacy windows unwrap `data`. Rows prefer limit/used or remaining arithmetic, then fractional `used_ratio` / `usedRatio` in `[0, 1]`, then direct percentages; out-of-range ratios fall through and reset times remain optional. Missing monthly data stays absent, and legacy 5-hour, weekly and total subscription-credit windows stay separate. The same container order reads `usages.limit_5h` and `usages.limit_7d` (Kimi Code's own client shape); a parsed one replaces the legacy 5-hour or weekly window, which can be an unused placeholder beside it, except that legacy counters above zero still win over a zero ratio (#8332). Total subscription credits are unaffected. The probe calls `/usages` on the configured inference origin only when it is exactly `https://api.kimi.com/coding/v1`, or `https://api.kimi.ai/coding/v1` for key auth (#8332), so the bearer never reaches a host it was not already sent to; account-login tokens stay on `api.kimi.com`. The parser is shared by account-login and API-key probes; credential and destination guards are unchanged. OrcaRouter key exchange uses the shared raw-byte reader before returning a durable key. Its 64 KiB response ceiling, single 30-second header/body deadline, and cancellation behavior follow diff --git a/structure/providers/anthropic-account-pool.md b/structure/providers/anthropic-account-pool.md index 7df859fdd44..edce91c4b96 100644 --- a/structure/providers/anthropic-account-pool.md +++ b/structure/providers/anthropic-account-pool.md @@ -44,7 +44,7 @@ kinds as fallback. Safe future tier strings retain their original spelling; malf or absent tiers project null and never come from quota percentages. Same-account refreshes retain an unobserved plan; an observed invalid tier clears it. Refresh gives the profile read a 2-second deadline because a refresh can sit on a request path; -a missed read leaves the stored label in place. +a missed read leaves the stored label in place. `GET /api/oauth/accounts?quota=1` fills a plan no credential has stored yet through `readAnthropicPlan` (per-bearer cache, an hour on success, five minutes on failure) and otherwise names a fixed `planUnknownReason` (`tests/oauth/oauth-accounts-api.test.ts`). An account replacement cannot inherit another account's label. `src/providers/anthropic-subscription.ts` projects optional management `planLabel`: Pro, Max 5x, Ultra (both `ultra` and `max_20x`), or the original unfamiliar validated tier. Dashboard OAuth rows prefer a user alias, then the @@ -401,6 +401,7 @@ quota recovery `anthropic-oauth-429`, as the native Messages path already did. A streamed terminal continuation after assistant output and search after live output or a published search call cannot rotate on 403. Non-streaming buffered continuations may recover. No eligible replacement preserves the upstream 403; generic 401 handling is unchanged. +A 400 counts only as Anthropic's `invalid_request_error` envelope whose message starts with the spent-extra-usage or spend-limit wording (`readAnthropicExtraUsageRefusal` in `src/providers/anthropic-billing-refusal.ts`); it cools as spent quota (Retry-After, rejected-window reset, else ten minutes) and rotates like a shared-quota 429, recorded as `anthropic-oauth-429` (`tests/adapters/anthropic/anthropic-pool-extra-usage-400.test.ts`, `tests/routing/router-combo-400-quota-echo.test.ts`). Reactive recovery also works with proactive pooling disabled when multiple accounts are stored. Regression coverage: `tests/adapters/anthropic/anthropic-quota-dispatch.test.ts`. @@ -485,7 +486,7 @@ Regression coverage: `tests/adapters/anthropic/anthropic-pool-usage-limit.test.t ## Family weekly admission -`src/providers/quota/anthropic-family-headers.ts` attributes 7d_oi only to fixture-confirmed +A unified 429 whose representative claim is the requested family's own `seven_day_` limit is family quota too (`anthropicClaimedFamily`, `tests/adapters/anthropic/anthropic-family-claim-429.test.ts`). `src/providers/quota/anthropic-family-headers.ts` attributes 7d_oi only to fixture-confirmed Fable 5 models. It preserves independent shared and model windows, including rejection-only family evidence only on HTTP 429, without advancing the usage-probe clock. Other response statuses retain soft utilization without creating a hard family refusal. diff --git a/structure/providers/generic-account-pool.md b/structure/providers/generic-account-pool.md index 5ccd5df8701..42542cfa039 100644 --- a/structure/providers/generic-account-pool.md +++ b/structure/providers/generic-account-pool.md @@ -47,3 +47,26 @@ account has an eligible sibling before cooling the whole target. This probe reco account's cooldown but passes `commit: false`, so it only peeks at a round-robin pool. An unsent sibling does not consume a sticky turn. Dispatching callers retain the default committed pick. `tests/oauth/generic-oauth-combo-account-cooldown.test.ts` covers this distinction. + +## AgentRadar hosted account contract + +`src/lib/agentradar-account-pin.ts` is active only with `OPENCODEX_AGENTRADAR=1`. Its two pin headers +are accepted only on `/v1/messages`, `/v1/messages/count_tokens` and `/v1/chat/completions`; elsewhere, malformed, or with the flag +off they are refused before dispatch. A pin bypasses native managed lanes and caller credentials, +uses only its own provider's stored account, never commits the operator selection, and records refusal +evidence without selecting another account. The Codex pin uses the explicit-account selector and never +names the native main account. Helpers that cannot run on the pin refuse; Claude web search is bound +to the pinned primary-pool account. The served-account response header and hashed `[ar-acct]` line are +written only after a physical dispatch. Under round-robin, a keyed AgentRadar Anthropic session takes +the rendezvous-hash winner over eligible accounts before startup baseline or restored affinity; ordinary +installs keep the durable-affinity order above. A refused hashed session fails over to its hash runner-up, so every process agrees on it (`tests/adapters/anthropic/anthropic-agentradar-pool-parity.test.ts`). A streamed terminal-guard re-ask on such a gateway may also move off a refused (429/403) account, opening a new attempt, where ordinary installs never rotate after visible output (`tests/server/terminal-guard-agentradar-rotation.test.ts`). There a 429 with no unified status cools its account (Retry-After, else the default) and rotates, and an allowed unified status fails over at once without a same-account wait (`tests/adapters/anthropic/anthropic-agentradar-429.test.ts`). `stampAnthropicAccountForLog` also writes the serving account's `p` into `accountLogLabel`, and `?quota=1` answers `plan` in the hosted vocabulary (`ultra`, `max`) while `planLabel` stays canonical. An organization-disabled refusal (`oauth_org_not_allowed`, or the structured `permission_error` with that exact wording) rotates as usual and also marks the account for reauthentication, since waiting out a cooldown cannot bring it back. On AgentRadar gateways a cached seven-day reading at 100% with a future reset is also a hard hold (selection, affinity, Retry-After and usage-limit answers), and a rejected seven-day refusal persists that reading, so a fresh process cannot seat a session on a spent account (`tests/adapters/anthropic/anthropic-weekly-spent-gate.test.ts`). It reads only cached evidence; no lazy usage probe runs on the request path. `tests/server/agentradar-account-pin-e2e.test.ts` +runs these cases against real gateway processes and a loopback supplier. + +On these gateways `/v1/messages/count_tokens` for a canonical Anthropic target returns Anthropic's +exact count rather than the local estimate (`src/server/agentradar-count-tokens.ts`). The count sends the exact source to +`https://api.anthropic.com/v1/messages/count_tokens` under generation's combo eligibility, image policy, +cooldowns and key scope. A random combo picks from one request-correlation seed shared by the count and its turn. +A pinned count uses only its pinned account and refuses without a send when that account is unusable. +An unpinned count selects an account as a dispatch would and commits nothing. Counts are neither billed nor +request-logged. A noncanonical target, a declined native route and a pinned combo keep the estimate +(`tests/claude-integration/agentradar-count-tokens.test.ts`). diff --git a/structure/providers/xai-grok.md b/structure/providers/xai-grok.md index 74ea3d14ab7..837d202bd65 100644 --- a/structure/providers/xai-grok.md +++ b/structure/providers/xai-grok.md @@ -12,6 +12,12 @@ Native steering follows [the shared WebSocket contract](../transports/streaming- xAI uses the same shared credential and delivery policies through the Responses [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. +The `xai-apikey` registry row is the `xai` row's key-auth lane under its own provider identity, for +hosted deployments that store an xAI API key under that id. `xaiApiKeyEntry` in +`src/providers/registry/entries-core.ts` derives it from the `xai` row at load, applying +`keyAuthServiceTier` and dropping the OAuth-only `oauthId`, `allowKeyAuthOverride` and `featured` +fields, so a capability fix to `xai` reaches it without a second copy. Grok OAuth stays on `xai`. + One Responses capability is seeded for xAI alone: `requiresPairedResponsesToolResults`, which answers a replayed tool call whose output never arrived. It is deliberately not the same flag as `requiresAdjacentResponsesToolResults`, which xAI also carries and shares with the Kimi presets. diff --git a/structure/runtime.md b/structure/runtime.md index 645a72717c1..8c92ee6b51d 100644 --- a/structure/runtime.md +++ b/structure/runtime.md @@ -397,7 +397,7 @@ A routed turn often ends on a client tool call after the sidecar already handed ### Loop usage accounting -Every iteration re-sends the full context, so `src/web-search/loop.ts` adds the usage of each iteration whose terminal the client never sees (search-calling passes and the discarded empty answer) to what `onUsage` reports for the request log, including on an in-stream or translator-budget error; token and cache counts add, while `contextTotalTokens` and `rawUsage` come only from the final terminal, never an earlier iteration's. The client-facing terminal keeps the last iteration's usage, because Codex and Claude Code size the context window from it (`tests/web-search/web-search-loop-iteration-usage.test.ts`). A pass that ends in an adapter `error` event was billed too: `parseStreamWithProgress` hands that event's usage to `onErrorUsage` before failing, so `src/web-search/loop.ts` and `src/images/loop.ts` (which already puts its hidden iterations' usage on the failed terminal) add it to the usage they report, while the turn still fails with the same error (`tests/web-search/web-search-loop-error-usage.test.ts`, `tests/images/loop.test.ts`). +Every iteration re-sends the full context, so `src/web-search/loop.ts` adds the usage of each iteration whose terminal the client never sees (search-calling passes and the discarded empty answer) to what `onUsage` reports for the request log, including on an in-stream or translator-budget error; token and cache counts add, while `contextTotalTokens` and `rawUsage` come only from the final terminal, never an earlier iteration's. The client-facing terminal keeps the last iteration's usage, because Codex and Claude Code size the context window from it (`tests/web-search/web-search-loop-iteration-usage.test.ts`). A pass that ends in an adapter `error` event was billed too: `parseStreamWithProgress` hands that event's usage to `onErrorUsage` before failing, so `src/web-search/loop.ts` and `src/images/loop.ts` (which already puts its hidden iterations' usage on the failed terminal) add it to the usage they report, while the turn still fails with the same error (`tests/web-search/web-search-loop-error-usage.test.ts`, `tests/images/loop.test.ts`). A translator-budget overflow puts the hidden iterations' usage on its own `response.failed` terminal in both loops, and the bridge reports it through `onUsage` exactly once (`tests/web-search/web-search.test.ts`, `tests/images/loop.test.ts`). A 429 rotation inside either loop splits the request into one attempt row per account, so the hidden usage already gathered belongs to the account the loop is leaving. Right after a rotation each loop offers it through `onRotatedUsage`; `src/server/responses/sidecar-execution.ts` accepts only when the next send will open a new attempt (`attemptAccountChanged`), binds it to the still-open attempt, and the loop drops it from what `onUsage` later reports, so the surviving attempt carries only its own rounds and nothing is counted twice. The image loop still puts the whole turn on the client terminal (`tests/web-search/web-search-rotation-usage-attribution.test.ts`, `tests/images/loop.test.ts`). diff --git a/structure/transports/byte-accounting.md b/structure/transports/byte-accounting.md index 2e0c18aae73..c8b066fcd00 100644 --- a/structure/transports/byte-accounting.md +++ b/structure/transports/byte-accounting.md @@ -550,3 +550,13 @@ counts each message that reached it, and ends the flow on close; `sendTextFrame` `src/server/ws-bridge.ts` counts each message Bun took for a link socket (a dropped send is not counted). A local socket has no flow, and link sockets exist only while the link WebSocket's switch and release gate are on. + +## Operator log lines and physical-attempt analytics + +`src/lib/usage-line-log.ts`: one `[usage]` line per final request record (`addRequestLog` in `src/server/request-log.ts`), only when `OPENCODEX_AGENTRADAR=1`: `status`, `provider` (pooled account suffix dropped), `model`, `in`, `out`, `cached`, `key` (8 hex of SHA-256 over the API key record id) and `ms`, `-` when unknown. One finalized record, not proof of one physical dispatch; ids sanitized, quoted, 80 characters; never text, body, header, secret, account label or email. `tests/lib/call-usage-line-log.test.ts`. + +`src/lib/upstream-refusal-log.ts`, also only when `OPENCODEX_AGENTRADAR=1`: one `[upstream-refusal]` line per upstream 4xx except 401/429 at the physical send with status, provider/model ids, envelope `type`/`code`/`param` and outgoing body bytes. The envelope is read from a clone (64 KiB, 2 s) off the request path; `message` is never read. A field is logged verbatim only as a short identifier without a 20+ character alphanumeric run, else as `#` plus 8 hex of SHA-256 over its redacted value. `logTranslationBufferLimit` and `logOversizedInbound` log the proxy's own 413 and a >5 MiB declared inbound size with closed fields. `tests/lib/upstream-refusal-log.test.ts`. + +Physical attempts: `physicalUsageAttempts` (`src/usage/physical-attempts.ts`, re-exported by `src/usage/log.ts`) keeps only real upstream sends (not locally answered, `sendCount > 0`, first of each ordinal). `aggregateAttemptUsage`, `normalizePhysicalEntry` (usage summary and `/api/logs` `displayMetrics`), the request-history ingest columns, route explainability's final target, health scoring (`src/routing/health.ts`, ordered by row then attempt) and Lab passive production evidence (`src/lab/query/passive-production.ts`) and the final row's tier outcome read only those; an explicit empty or all-nonphysical list never falls back to stale root usage, and legacy root usage with no attempt usage prices on the final physical attempt. A malformed optional attempt field drops only that field. Usage carries bounded `anthropicServerToolUse` counters (summed) and the latest `contextTotalTokens`. Covered by `tests/usage/usage-log-physical.test.ts`, `tests/usage/usage-summary-physical.test.ts`, `tests/usage/request-log-physical.test.ts` and `tests/server/management-api-logs-metrics-physical.test.ts`. + +`GET /api/routing-analytics` (`src/routing/analytics.ts`, bounded `withRequestHistoryRead`) adds physical-attempt telemetry: real sends only (locally answered, unsent and duplicate-ordinal attempts dropped), `physicalSends`, `repeatedSendAttempts`, `recoveryAttempts`/`recoveryEvents` (`recoveryCount`, else `recoveryKinds`), `comboFailoverRequests` (`comboTargetAdvanced` rows), input cache shares, `physicalUsageCoverage` (`usageCoverage` is its ratio; unreported and unsupported never count as measured) and `physicalBreakdown`. An Anthropic route folds to `anthropic` with one account: `accountTag`, else `p` `accountLogLabel`, else the provider suffix. `redAlerts` (newest `ANALYTICS_MAX_RED_ALERTS`, `redAlertsPartial` when capped or truncated) hold ids and numbers only: `recovery`, `repeated-send`, `combo-failover`, and per Anthropic conversation route in `firstSendAt` order `consecutive-high-raw-input`, `high-cache-write-after-warmup`, `low-cache-read-share-after-warmup`, `falling-cache-read`, with warm state expiring after `promptCacheTtlMs` (default 5 minutes). `range` reuses the usage calendar window; `surface` takes the usage groups. `tests/routing/routing-analytics-physical.test.ts`. diff --git a/structure/transports/inventory.md b/structure/transports/inventory.md index a2bbc17f914..c561c939be7 100644 --- a/structure/transports/inventory.md +++ b/structure/transports/inventory.md @@ -124,6 +124,10 @@ opaque metadata, normal tool passthrough, coalesced-tail accounting, and consume both execution paths. Final client tool output also follows the [explicit empty client-catalog boundary](responses-wire-shapes.md#declared-tool-membership-by-inbound-wire); internal media tool execution rejects explicitly unfinished tool ends and never fulfills media or re-asks after an incomplete, failed or truncated terminal. +Each `runTurn` iteration hands the slot it reserved to the caller's `runTurnFetch`, which becomes +the adapter's `providerFetch`: a Cursor turn's RunSSE and BidiAppend sends stay inside that one +lease and wait only on start intervals, as on the normal `runTurn` lane, and the turn records one +attempt. ## Bounded response ingestion and OrcaRouter login diff --git a/structure/transports/responses-combo-boundaries.md b/structure/transports/responses-combo-boundaries.md index 1b9fe6dbe86..92d1d0c15ac 100644 --- a/structure/transports/responses-combo-boundaries.md +++ b/structure/transports/responses-combo-boundaries.md @@ -351,3 +351,10 @@ ordinary failover by supplying the same header. The wire body remains `rate_limi The message says no additional provider request was sent when the current request already has physical sends; an admission refusal before its first send retains the no-provider wording. Coverage: `tests/responses/chat-native-spend.test.ts` and `tests/responses/responses-spend-contract-c.test.ts`. + +On an AgentRadar gateway (`OPENCODEX_AGENTRADAR=1`) a 429/402 target cooldown lifts early once the target's account +pool reports an eligible account again (`setComboPoolEligibilityProbe` in `src/combos/failover.ts`, registered by the +pool owner), and `coolComboTarget` caps server-stated delays at ten minutes as well as local fallbacks: a `Retry-After` +describes the account that answered, while the key is the shared target. Elsewhere a non-pool 429 parks the target in +full and server delays run to the one-day ceiling. Hosted gateways also trust the anchored classifier's 400 quota verdict +(`upstreamCodeForComboFailure` in `src/bridge/errors.ts`) and an `err_free_prompt_cap` field on a 400/422. diff --git a/structure/transports/responses-spend.md b/structure/transports/responses-spend.md index e1dfa962581..180c70a4fa0 100644 --- a/structure/transports/responses-spend.md +++ b/structure/transports/responses-spend.md @@ -144,6 +144,10 @@ cover up to `TRANSIENT_RETRY_MAX_ATTEMPTS` sends per eligible account (currently whatever the roster size. A single eligible account keeps the existing base ceiling of three and total ceiling of four. Explicit caller ceilings and combo-derived scopes keep their existing limits. +On an AgentRadar gateway (`OPENCODEX_AGENTRADAR=1`) `oauthMaxAccountsPerRequest()` in +`src/oauth/generic-account-failover.ts` raises that account cap to sixteen (48 default sends), so a +pooled request with no combo behind it reaches a healthy account after up to fifteen refusals; explicit +ceilings still intersect it (`tests/adapters/anthropic/anthropic-funded-roster.test.ts`). A helper recovery's prepaid hop remains part of the full leg attempt allowance. An adapter-owned pending hop reconciles the actual target synchronously: a regional endpoint change refunds the old diff --git a/structure/transports/responses-wire-shapes.md b/structure/transports/responses-wire-shapes.md index 4a0028c5e5e..3d47ce2cdbf 100644 --- a/structure/transports/responses-wire-shapes.md +++ b/structure/transports/responses-wire-shapes.md @@ -190,7 +190,7 @@ one line, with nothing in the response saying so. This also replaces kiro's form `invalid_request_error`, which told the client to start a new session and therefore skipped the recovery Codex performs on `previous_response_not_found`. Retention is the other half: local continuation state is held for `RESPONSE_TTL_MS` (24 hours), long enough that an ordinary idle gap -resumes by expansion rather than by asking the client to replay. Routed custom-tool lowering requires the same recovery when a delta +resumes by expansion rather than by asking the client to replay. `OPENCODEX_RESPONSE_STATE_TTL_MS`, `OPENCODEX_RESPONSE_STATE_MAX_BYTES` and `OPENCODEX_RESPONSE_STATE_MAX_SPILL_BYTES` may only lower retention, the resident cap and the spill cap (above default, below 1 or non-numeric is ignored). Routed custom-tool lowering requires the same recovery when a delta custom result has no local call, because its original wire type cannot be established and guessing it would send an unmatched result upstream. The check resolves the selected wire protocol and the request's own tool declarations after final route selection, so stateful destinations keep their diff --git a/structure/usage-client-attribution.md b/structure/usage-client-attribution.md index 8824856849c..00acf317d30 100644 --- a/structure/usage-client-attribution.md +++ b/structure/usage-client-attribution.md @@ -195,6 +195,21 @@ newly scoped by that rule. `src/server/hub-usage.ts` binds a client-filtered dat key and emits a safe scope echo; `src/remote/hub-usage.ts` projects it without key IDs, and `src/client/hub-client.ts` rejects a missing or mismatched echo. +## Credential recovery attempts + +`noteProviderAttemptSend` in `src/server/request-log.ts` closes a sent attempt before a +credential recovery even when the provider and opaque account label are unchanged, such as a +same-account OAuth refresh after 401. `credentialRecoveryStatus` in +`src/server/request-log-attempt-recovery.ts` maps the closed credential-recovery roster to its +401, 402, 403 or 429 refusal status; an already recorded upstream status takes precedence. +The existing in-place split preserves the active object, keeps the refused send's usage on its +sealed row and records the replacement send on a fresh row. The sealed tier outcome is a snapshot; +the replacement retains the live adapter observer's outcome so response-tier confirmation updates +only the replacement. A recovery before the first send creates no extra row. Same-account waits and transport resends remain on the existing attempt. +`tests/usage/key-attribution.test.ts` covers the split and non-split kinds; the loopback server +case in `tests/server/server-google-antigravity-oauth-401-replay.test.ts` records separate 401 +and 200 rows for a same-account refresh. + ## Anthropic pool account tag Anthropic pool attempts carry no `accountLogLabel`; the pool folds the account into the logged diff --git a/tests/adapters/adapter-usage.test.ts b/tests/adapters/adapter-usage.test.ts index 96e1aca0352..56fd4daf8a7 100644 --- a/tests/adapters/adapter-usage.test.ts +++ b/tests/adapters/adapter-usage.test.ts @@ -150,6 +150,70 @@ describe("adapter reasoning and usage details", () => { expect(body.messages[0].content).toBe("hi"); }); + test("Anthropic API-key requests mark system prompt as cacheable (AgentRadar hosted, .27 5-minute default)", async () => { + const prior = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + try { + const adapter = createAnthropicAdapter({ ...provider, adapter: "anthropic", baseUrl: "https://api.anthropic.com" }); + const request = await adapter.buildRequest({ + modelId: "claude-opus-4-1", + context: { + systemPrompt: ["stable project instructions"], + messages: [{ role: "user", content: "hi" }], + }, + stream: true, + options: {}, + }); + const body = JSON.parse(request.body) as { system: unknown; messages: Array<{ content: unknown }>; cache_control?: unknown }; + + // Native Anthropic gets top-level automatic caching plus stable explicit breakpoints. + expect(body.cache_control).toEqual({ type: "ephemeral" }); + expect(request.promptCacheTtlMs).toBe(5 * 60_000); + expect(body.system).toEqual([{ + type: "text", + text: "stable project instructions", + cache_control: { type: "ephemeral" }, + }]); + // The moving final user block is handled by top-level automatic caching. + expect(body.messages[0].content).toBe("hi"); + } finally { + if (prior === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = prior; + } + }); + + test("Anthropic long cache retention records its one-hour effective TTL", async () => { + const adapter = createAnthropicAdapter( + { ...provider, adapter: "anthropic", baseUrl: "https://api.anthropic.com" }, + "long", + ); + const request = await adapter.buildRequest({ + modelId: "claude-opus-4-1", + context: { + systemPrompt: ["stable project instructions"], + messages: [{ role: "user", content: "hi" }], + }, + stream: true, + options: {}, + }); + + expect(request.promptCacheTtlMs).toBe(60 * 60_000); + }); + + test("Anthropic short or uncached retention records five minutes or nothing", async () => { + const build = (retention: "short" | "none") => createAnthropicAdapter( + { ...provider, adapter: "anthropic", baseUrl: "https://api.anthropic.com" }, + retention, + ).buildRequest({ + modelId: "claude-opus-4-1", + context: { systemPrompt: ["stable"], messages: [{ role: "user", content: "hi" }] }, + stream: true, + options: {}, + }); + + expect((await build("short")).promptCacheTtlMs).toBe(5 * 60_000); + expect((await build("none")).promptCacheTtlMs).toBeUndefined(); + }); + test("Anthropic OAuth requests keep Claude identity first and cache user system prompt", async () => { const adapter = createAnthropicAdapter({ ...provider, diff --git a/tests/adapters/anthropic/anthropic-account-pool.test.ts b/tests/adapters/anthropic/anthropic-account-pool.test.ts index 9e17ede2d17..859dce3c4f1 100644 --- a/tests/adapters/anthropic/anthropic-account-pool.test.ts +++ b/tests/adapters/anthropic/anthropic-account-pool.test.ts @@ -627,13 +627,14 @@ describe("anthropic account pool quota window scoring", () => { expect(resolveAnthropicAccountForSession("weekly-unknown-last", cfg(true, 80, { quotaWindow: "weekly" })).accountId).toBe(aId); }); - test("weekly mode ranks known 100% before unknown weekly usage", async () => { + // .27 parity: a spent account outranking one that may have room was inverted. + test("weekly mode ranks unknown weekly usage before a known 100%", async () => { const { aId, bId } = await seedTwoAccounts(); const updatedAt = Date.now(); setCachedProviderAccountQuotaForTests("anthropic", aId, { fiveHourPercent: 90, weeklyPercent: 100, updatedAt }); setCachedProviderAccountQuotaForTests("anthropic", bId, { fiveHourPercent: 0, updatedAt }); - expect(resolveAnthropicAccountForSession("weekly-known-100-before-unknown", cfg(true, 80, { quotaWindow: "weekly" })).accountId).toBe(aId); + expect(resolveAnthropicAccountForSession("weekly-known-100-before-unknown", cfg(true, 80, { quotaWindow: "weekly" })).accountId).toBe(bId); }); test("weekly tie breaks by lower five-hour usage", async () => { diff --git a/tests/adapters/anthropic/anthropic-agentradar-429.test.ts b/tests/adapters/anthropic/anthropic-agentradar-429.test.ts new file mode 100644 index 00000000000..b040677ff90 --- /dev/null +++ b/tests/adapters/anthropic/anthropic-agentradar-429.test.ts @@ -0,0 +1,66 @@ +// Parity with v2.65.0-vadymhimself.27 anthropic-ratelimit-headers: on an AgentRadar gateway a 429 with no +// unified status cools its account and rotates; an allowed unified status fails over without cooling. +// Driven through the refusal path a request takes (rotateAnthropicAccountOnResponse); .27's log text is not ported. +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { clearPoolRotationState } from "../../../src/codex/pool-rotation"; +import { bindAnthropicRefusalCredential, rotateAnthropicAccountOnResponse } from "../../../src/oauth/anthropic-account-refusal"; +import { clearAnthropicAccountPoolState, getAnthropicAccountHealthSnapshot, getAnthropicPoolAccessSnapshot, getEligibleAnthropicAccounts } from "../../../src/oauth/anthropic-routing"; +import { getAccountSet, saveCredential } from "../../../src/oauth/store"; +import { clearAccountQuotaCache } from "../../../src/providers/quota"; +import type { OcxConfig } from "../../../src/types"; +import { removeTreeWithRetry } from "../../helpers/remove-tree"; + +const originalHome = process.env.OPENCODEX_HOME; +const originalFlag = process.env.OPENCODEX_AGENTRADAR; +let home: string; +const config = { providers: { anthropic: { adapter: "anthropic", authMode: "oauth" } }, anthropicAccountPool: { enabled: true } } as unknown as OcxConfig; +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "ocx-ar-429-")); process.env.OPENCODEX_HOME = home; process.env.OPENCODEX_AGENTRADAR = "1"; + clearAnthropicAccountPoolState(); clearPoolRotationState(); clearAccountQuotaCache("anthropic"); +}); +afterEach(() => { + clearAnthropicAccountPoolState(); clearPoolRotationState(); clearAccountQuotaCache("anthropic"); + if (originalFlag === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = originalFlag; + if (originalHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = originalHome; + removeTreeWithRetry(home); +}); +async function seed() { + for (const i of [0, 1]) await saveCredential("anthropic", { access: `access-${i}`, refresh: `refresh-${i}`, expires: Date.now() + 3_600_000, accountId: `uuid-${i}` }); + return getAccountSet("anthropic")!.accounts.map(account => account.id); +} +async function refuse(accountId: string, headers: Record, model = "claude-opus-5") { + const response = new Response(null, { status: 429, headers }); + bindAnthropicRefusalCredential(response, await getAnthropicPoolAccessSnapshot(accountId)); + return rotateAnthropicAccountOnResponse(response, { config, accountId, model, canRetry: true, requestKey: {} }); +} + +test("an allowed unified status fails the request over without cooling either account, while a header-less 429 still cools", async () => { + const ids = await seed(); + const allowed = { "anthropic-ratelimit-unified-status": "allowed", "anthropic-ratelimit-unified-5h-status": "allowed", "anthropic-ratelimit-unified-7d-status": "allowed" }; + expect(await refuse(ids[0]!, allowed)).toBe(ids[1]!); + expect(ids.map(id => getAnthropicAccountHealthSnapshot(id))).toEqual([null, null]); + clearAnthropicAccountPoolState(); + const start = Date.now(); + expect(await refuse(ids[0]!, {})).toBe(ids[1]!); + expect(getAnthropicAccountHealthSnapshot(ids[0]!)?.cooldownUntil).toBeGreaterThanOrEqual(start + 60_000); +}); + +test("a five-hour claim on an Opus request still cools the whole account", async () => { + const ids = await seed(); + const start = Date.now(); + expect(await refuse(ids[0]!, { "retry-after": "120", "anthropic-ratelimit-unified-representative-claim": "five_hour" })).toBe(ids[1]!); + const until = getAnthropicAccountHealthSnapshot(ids[0]!)?.cooldownUntil ?? 0; + expect(until).toBeGreaterThanOrEqual(start + 120_000); + expect(until).toBeLessThanOrEqual(Date.now() + 120_000); + expect(getEligibleAnthropicAccounts(Date.now(), "claude-sonnet-5-5")).toEqual([ids[1]!]); +}); + +test("ordinary installs keep upstream's request-local header-less 429", async () => { + delete process.env.OPENCODEX_AGENTRADAR; + const ids = await seed(); + await refuse(ids[0]!, {}); + expect(getAnthropicAccountHealthSnapshot(ids[0]!)).toBeNull(); +}); diff --git a/tests/adapters/anthropic/anthropic-agentradar-pool-parity.test.ts b/tests/adapters/anthropic/anthropic-agentradar-pool-parity.test.ts new file mode 100644 index 00000000000..8916139204a --- /dev/null +++ b/tests/adapters/anthropic/anthropic-agentradar-pool-parity.test.ts @@ -0,0 +1,322 @@ +// Ported from v2.65.0-vadymhimself.27 anthropic-account-pool.test.ts: AgentRadar hash placement and the +// cached seven-day gate. Assertions are unchanged; only imports and the gateway flag are adapted. +import { rotateAnthropicAccountOn429 } from "../../helpers/anthropic-shared-quota"; +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { clearPoolRotationState, notePoolRotationFailure, POOL_KEY_ANTHROPIC } from "../../../src/codex/pool-rotation"; +import { + anthropicQuotaWindow, + anthropicSessionKeyFromParts, + bindAnthropicSessionAffinity, + clearAnthropicAccountCooldown, + clearAnthropicAccountPoolState, + formatAnthropicProviderForLog, + getEligibleAnthropicAccounts, + isAnthropicAccountPoolEnabled, + normalizeAccountPoolQuotaWindow, + parseAccountPoolQuotaWindow, + resolveAnthropicAccountForSession, + getAnthropicPoolRetryAfterSeconds, + resetAnthropicRoutingForManualSelection, + getAnthropicPoolAccessSnapshot, + promoteAnthropicActiveAccount, + anthropicSessionAffinitySizeForTests, +} from "../../../src/oauth/anthropic-routing"; +import { + captureOAuthAccountSelection, + credentialGeneration, + getAccountCredential, + getAccountSet, + markAccountNeedsReauth, + saveCredential, + saveAccountCredential, + setActiveAccount, +} from "../../../src/oauth/store"; +import { subscribeAccountSelections } from "../../../src/lib/account-selection-events"; +import { clearAccountQuotaCache, setCachedProviderAccountQuotaForTests } from "../../../src/providers/quota"; +import type { OcxAccountPoolQuotaWindow, OcxAccountPoolRotationStrategy, OcxConfig } from "../../../src/types"; +import { removeTreeWithRetry } from "../../helpers/remove-tree"; + +const originalHome = process.env.OPENCODEX_HOME; +const originalAgentRadar = process.env.OPENCODEX_AGENTRADAR; +let home: string; + +async function admitAnthropic(sessionKey: string, config: OcxConfig) { + const expected = captureOAuthAccountSelection("anthropic"); + const choice = resolveAnthropicAccountForSession(sessionKey, config); + if (choice.accountId) { + const snapshot = await getAnthropicPoolAccessSnapshot(choice.accountId); + expect(await promoteAnthropicActiveAccount(choice.accountId, expected, { + config, sessionKey, reason: choice.reason, expectedCredentialGeneration: snapshot.generation, + })).not.toBeNull(); + } + return choice; +} + +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "ocx-anthropic-pool-")); + process.env.OPENCODEX_HOME = home; + process.env.OPENCODEX_AGENTRADAR = "1"; + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + clearAccountQuotaCache("anthropic"); +}); + +afterEach(() => { + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + clearAccountQuotaCache("anthropic"); + if (originalHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = originalHome; + if (originalAgentRadar === undefined) delete process.env.OPENCODEX_AGENTRADAR; + else process.env.OPENCODEX_AGENTRADAR = originalAgentRadar; + removeTreeWithRetry(home); +}); + +async function seedTwoAccounts() { + await saveCredential("anthropic", { + access: "access-a", + refresh: "refresh-a", + expires: Date.now() + 3_600_000, + accountId: "uuid-aaaa", + email: "a@example.test", + }); + await saveCredential("anthropic", { + access: "access-b", + refresh: "refresh-b", + expires: Date.now() + 3_600_000, + accountId: "uuid-bbbb", + email: "b@example.test", + }); + // saveCredential activates the newly appended account (B). Pin A as active for predictable tests. + const { getAccountSet } = await import("../../../src/oauth/store"); + const set = getAccountSet("anthropic")!; + const a = set.accounts.find(acc => acc.credential.accountId === "uuid-aaaa")!; + const b = set.accounts.find(acc => acc.credential.accountId === "uuid-bbbb")!; + await setActiveAccount("anthropic", a.id); + // Ordinary policy cases start after the initial stored selection has been admitted. + // Restart cases explicitly clear runtime state below to exercise first admission again. + await admitAnthropic("", cfg(false)); + return { aId: a.id, bId: b.id }; +} + +function cfg( + enabled: boolean, + threshold = 80, + pool: { + strategy?: OcxAccountPoolRotationStrategy; + stickyLimit?: number; + quotaWindow?: OcxAccountPoolQuotaWindow; + } = {}, +): OcxConfig { + return { + port: 0, + defaultProvider: "anthropic", + providers: { + anthropic: { adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "oauth" }, + }, + anthropicAccountPool: { + enabled, + autoSwitchThreshold: threshold, + ...pool, + }, + } as OcxConfig; +} + +async function seedThreeAccounts() { + await saveCredential("anthropic", { + access: "access-a", + refresh: "refresh-a", + expires: Date.now() + 3_600_000, + accountId: "uuid-aaaa", + email: "a@example.test", + }); + await saveCredential("anthropic", { + access: "access-b", + refresh: "refresh-b", + expires: Date.now() + 3_600_000, + accountId: "uuid-bbbb", + email: "b@example.test", + }); + await saveCredential("anthropic", { + access: "access-c", + refresh: "refresh-c", + expires: Date.now() + 3_600_000, + accountId: "uuid-cccc", + email: "c@example.test", + }); + const { getAccountSet } = await import("../../../src/oauth/store"); + const set = getAccountSet("anthropic")!; + const a = set.accounts.find(acc => acc.credential.accountId === "uuid-aaaa")!; + const b = set.accounts.find(acc => acc.credential.accountId === "uuid-bbbb")!; + const c = set.accounts.find(acc => acc.credential.accountId === "uuid-cccc")!; + await setActiveAccount("anthropic", a.id); + await admitAnthropic("", cfg(false)); + return { aId: a.id, bId: b.id, cId: c.id }; +} + + +describe("AgentRadar hash placement", () => { + test("unbound strategy pick does not promote active before token validation", async () => { + const { aId, bId, cId } = await seedThreeAccounts(); + setCachedProviderAccountQuotaForTests("anthropic", aId, { fiveHourPercent: 10 }); + setCachedProviderAccountQuotaForTests("anthropic", bId, { fiveHourPercent: 10 }); + setCachedProviderAccountQuotaForTests("anthropic", cId, { fiveHourPercent: 10 }); + await setActiveAccount("anthropic", aId); + const config = cfg(true, 80, { strategy: "round-robin", stickyLimit: 1 }); + + const before = getAccountSet("anthropic")!.activeAccountId; + const picks = Array.from({ length: 3 }, (_, i) => resolveAnthropicAccountForSession(`promo-${i}`, config)); + expect(picks.every(p => p.accountId)).toBe(true); + expect(getAccountSet("anthropic")!.activeAccountId).toBe(before); + }); + test("429 failover never returns null while any eligible account remains", async () => { + const { aId, bId } = await seedTwoAccounts(); + // B is known spent for the week with its reset still ahead, so it is not eligible (was: still returned). + setCachedProviderAccountQuotaForTests("anthropic", bId, { + fiveHourPercent: 100, + weeklyPercent: 100, + weeklyResetAt: Date.now() + 2 * 86_400_000, + updatedAt: Date.now(), + }); + + expect(rotateAnthropicAccountOn429(cfg(true, 80, { quotaWindow: "weekly" }), aId, "30", "weekly-never-null")).toBeNull(); + }); + test("round-robin keyed sessions take a stable hash seat; a 429 unbinds only the hitting session", async () => { + await seedThreeAccounts(); + const config = cfg(true, 80, { strategy: "round-robin" }); + const seats = () => Array.from({ length: 60 }, (_, i) => resolveAnthropicAccountForSession(`s${i}`, config).accountId); + + const first = seats(); + expect(new Set(first).size).toBe(3); + // A fresh process: empty affinity, cooldowns and round-robin ring (its stored selection admitted first). + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + await admitAnthropic("", cfg(false)); + expect(seats()).toEqual(first); + + const seat = first[0]!; + const bystander = `s${first.indexOf(seat, 1)}`; + bindAnthropicSessionAffinity("s0", seat); + bindAnthropicSessionAffinity(bystander, seat); + const next = rotateAnthropicAccountOn429(config, seat, "60", "s0"); + // .27 kept bystander bindings (size 1); upstream's durable affinity invalidates every binding on a + // spent account. Hash placement precedes affinity, so the observable seats below are unchanged. + expect(next).toBeTruthy(); + expect(next).not.toBe(seat); + // The cooled hash seat fails over to the session's runner-up, and the bystander moves off it too. + expect(resolveAnthropicAccountForSession("s0", config)).toEqual({ accountId: next, reason: "hash" }); + expect(resolveAnthropicAccountForSession(bystander, config).accountId).not.toBe(seat); + }); + + test("a hashed session moves off a spent seat to its runner-up and returns on recovery", async () => { + await seedThreeAccounts(); + const config = cfg(true, 80, { strategy: "round-robin" }); + const seat = resolveAnthropicAccountForSession("s0", config).accountId!; + bindAnthropicSessionAffinity("s0", seat); + + // Five-hour window spent: the session fails over to its hash runner-up and stays there. + const spent = new Headers({ + "anthropic-ratelimit-unified-status": "rejected", + "anthropic-ratelimit-unified-5h-status": "rejected", + "anthropic-ratelimit-unified-5h-reset": String(Math.ceil(Date.now() / 1000) + 3600), + }); + const next = rotateAnthropicAccountOn429(config, seat, null, "s0", Date.now(), spent)!; + expect(next).toBeTruthy(); + expect(next).not.toBe(seat); + bindAnthropicSessionAffinity("s0", next); + expect(resolveAnthropicAccountForSession("s0", config)).toEqual({ accountId: next, reason: "affinity" }); + // A fresh process does not know the seat is spent: its first 429 lands on the same runner-up. + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + await admitAnthropic("", cfg(false)); + expect(resolveAnthropicAccountForSession("s0", config).accountId).toBe(seat); + expect(rotateAnthropicAccountOn429(config, seat, null, "s0", Date.now(), spent)).toBe(next); + bindAnthropicSessionAffinity("s0", next); + // Recovery: this warm process returns the session to its seat, as a fresh one would. + clearAnthropicAccountCooldown(seat); + expect(resolveAnthropicAccountForSession("s0", config)).toEqual({ accountId: seat, reason: "hash" }); + }); + + test("the store's startup selection does not seat a keyed round-robin session", async () => { + const { aId } = await seedThreeAccounts(); + const config = cfg(true, 80, { strategy: "round-robin" }); + // A fresh process whose stored selection (with a revision) is A: a key hashed elsewhere keeps its seat. + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + expect(getAccountSet("anthropic")!.selectionRevision).toBeDefined(); + let key = ""; + for (let i = 0; !key && i < 50; i++) { + clearAnthropicAccountPoolState(); + const pick = resolveAnthropicAccountForSession(`fresh-${i}`, cfg(true, 80, { strategy: "quota" })); + expect(pick).toEqual({ accountId: aId, reason: "manual" }); + clearAnthropicAccountPoolState(); + const seat = resolveAnthropicAccountForSession(`fresh-${i}`, config); + if (seat.accountId !== aId) { + expect(seat.reason).toBe("hash"); + key = `fresh-${i}`; + } + } + expect(key).not.toBe(""); + // A pick made in this process still wins its one dispatch. + resetAnthropicRoutingForManualSelection(aId); + expect(resolveAnthropicAccountForSession(key, config)).toEqual({ accountId: aId, reason: "manual" }); + }); + +}); + +describe("weekly-spent accounts across fresh instances", () => { + const rrConfig = () => cfg(true, 80, { strategy: "round-robin" }); + const dayAhead = () => Date.now() + 86_400_000; + /** What a new instance looks like: no cooldowns, affinity or ring, the stored selection admitted first. */ + async function freshInstance() { + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + await admitAnthropic("", cfg(false)); + } + + test("a fresh instance never picks an account whose weekly window is spent (hash, affinity, active)", async () => { + const { aId, bId } = await seedTwoAccounts(); + // Admit the stored selection first: the gate (rightly) refuses to promote an account already known spent. + await freshInstance(); + setCachedProviderAccountQuotaForTests("anthropic", aId, { weeklyPercent: 100, weeklyResetAt: dayAhead(), updatedAt: Date.now() }); + setCachedProviderAccountQuotaForTests("anthropic", bId, { weeklyPercent: 20, weeklyResetAt: dayAhead(), updatedAt: Date.now() }); + + // Hash seats: without the gate about half of these keys land on A. + for (let i = 0; i < 40; i++) expect(resolveAnthropicAccountForSession(`s${i}`, rrConfig()).accountId).toBe(bId); + // Affinity bound to A (as a previous request in this process could have) is dropped, not honoured. + bindAnthropicSessionAffinity("bound", aId); + expect(resolveAnthropicAccountForSession("bound", cfg(true, 80)).accountId).toBe(bId); + // The stored active account is A: an unkeyed request must not stay on it. + expect(getAccountSet("anthropic")!.activeAccountId).toBe(aId); + expect(resolveAnthropicAccountForSession(null, rrConfig()).accountId).toBe(bId); + }); + + test("a seven-day 429 with the only other account known spent surfaces the 429 with the earliest weekly reset", async () => { + const { aId, bId } = await seedTwoAccounts(); + const now = Date.now(); + const aReset = now + 2 * 86_400_000; + setCachedProviderAccountQuotaForTests("anthropic", aId, { weeklyPercent: 100, weeklyResetAt: aReset, updatedAt: now }); + const bResetSec = Math.ceil(now / 1000) + 3_600; + const rejected = new Headers({ + "anthropic-ratelimit-unified-status": "rejected", + "anthropic-ratelimit-unified-7d-status": "rejected", + "anthropic-ratelimit-unified-7d-reset": String(bResetSec), + }); + + expect(rotateAnthropicAccountOn429(rrConfig(), bId, null, "k", now, rejected)).toBeNull(); + expect(getEligibleAnthropicAccounts(now)).toEqual([]); + expect(getAnthropicPoolRetryAfterSeconds(now)).toBe(Math.ceil((bResetSec * 1000 - now) / 1000)); + }); + + test("Retry-After follows the earliest weekly reset when no cooldown is recorded", async () => { + const { aId, bId } = await seedTwoAccounts(); + const now = Date.now(); + setCachedProviderAccountQuotaForTests("anthropic", aId, { weeklyPercent: 100, weeklyResetAt: now + 5_000_000, updatedAt: now }); + setCachedProviderAccountQuotaForTests("anthropic", bId, { weeklyPercent: 100, weeklyResetAt: now + 9_000_000, updatedAt: now }); + expect(getAnthropicPoolRetryAfterSeconds(now)).toBe(5_000); + expect(resolveAnthropicAccountForSession("k", rrConfig(), now)).toEqual({ accountId: null, reason: "all-cooled" }); + }); +}); diff --git a/tests/adapters/anthropic/anthropic-family-claim-429.test.ts b/tests/adapters/anthropic/anthropic-family-claim-429.test.ts new file mode 100644 index 00000000000..75759fcc817 --- /dev/null +++ b/tests/adapters/anthropic/anthropic-family-claim-429.test.ts @@ -0,0 +1,60 @@ +// Parity with v2.65.0-vadymhimself.27 anthropic-ratelimit-headers "a model-scoped weekly 429 cools only that +// model family": a unified refusal whose representative claim is seven_day_ spends that family alone. +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { clearPoolRotationState } from "../../../src/codex/pool-rotation"; +import { bindAnthropicRefusalCredential, rotateAnthropicAccountOnResponse } from "../../../src/oauth/anthropic-account-refusal"; +import { + anthropicPoolCooldownError, clearAnthropicAccountPoolState, getAnthropicAccountHealthSnapshot, + getAnthropicPoolAccessSnapshot, getEligibleAnthropicAccounts, resolveAnthropicAccountForSession, +} from "../../../src/oauth/anthropic-routing"; +import { getAccountSet, saveCredential } from "../../../src/oauth/store"; +import { clearAccountQuotaCache } from "../../../src/providers/quota"; +import type { OcxConfig } from "../../../src/types"; +import { removeTreeWithRetry } from "../../helpers/remove-tree"; + +const originalHome = process.env.OPENCODEX_HOME; +let home: string; +const config = { providers: { anthropic: { adapter: "anthropic", authMode: "oauth" } }, anthropicAccountPool: { enabled: true } } as unknown as OcxConfig; +const opusWeekly = (reset: number) => new Headers({ + "anthropic-ratelimit-unified-status": "rejected", "anthropic-ratelimit-unified-representative-claim": "seven_day_opus", + "anthropic-ratelimit-unified-reset": String(reset), "anthropic-ratelimit-unified-5h-status": "allowed", "anthropic-ratelimit-unified-7d-status": "allowed", +}); +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "ocx-family-claim-")); process.env.OPENCODEX_HOME = home; + clearAnthropicAccountPoolState(); clearPoolRotationState(); clearAccountQuotaCache("anthropic"); +}); +afterEach(() => { + clearAnthropicAccountPoolState(); clearPoolRotationState(); clearAccountQuotaCache("anthropic"); + if (originalHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = originalHome; + removeTreeWithRetry(home); +}); +async function refuse(accountId: string, headers: Headers, model: string) { + const response = new Response(null, { status: 429, headers }); + bindAnthropicRefusalCredential(response, await getAnthropicPoolAccessSnapshot(accountId)); + return rotateAnthropicAccountOnResponse(response, { config, accountId, model, canRetry: true, requestKey: {} }); +} + +test("Opus cools per account, Sonnet keeps the account, and an all-Opus-spent pool answers a usage limit", async () => { + for (const i of [0, 1]) await saveCredential("anthropic", { access: `access-${i}`, refresh: `refresh-${i}`, expires: Date.now() + 3_600_000, accountId: `uuid-${i}` }); + const ids = getAccountSet("anthropic")!.accounts.map(account => account.id); + const reset = Math.floor(Date.now() / 1000) + 3 * 86_400; + expect(await refuse(ids[0]!, opusWeekly(reset), "claude-opus-5")).toBe(ids[1]!); + expect(getAnthropicAccountHealthSnapshot(ids[0]!)).toBeNull(); + expect(getEligibleAnthropicAccounts(Date.now(), "claude-opus-5")).toEqual([ids[1]!]); + expect(getEligibleAnthropicAccounts(Date.now(), "claude-sonnet-5-5")).toEqual(ids); + + await refuse(ids[1]!, opusWeekly(reset), "claude-opus-5"); + expect(resolveAnthropicAccountForSession("s3", config, Date.now(), null, "claude-opus-5").reason).toBe("all-cooled"); + expect(anthropicPoolCooldownError(Date.now(), null, "claude-opus-5").usageLimit).toEqual({ resetsAt: reset * 1000 }); + expect(resolveAnthropicAccountForSession("s4", config, Date.now(), null, "claude-sonnet-5-5").accountId).not.toBeNull(); +}); + +test("the claim counts only for its own family: a Sonnet request on an Opus claim is shared quota", async () => { + for (const i of [0, 1]) await saveCredential("anthropic", { access: `access-${i}`, refresh: `refresh-${i}`, expires: Date.now() + 3_600_000, accountId: `uuid-${i}` }); + const ids = getAccountSet("anthropic")!.accounts.map(account => account.id); + expect(await refuse(ids[0]!, opusWeekly(Math.floor(Date.now() / 1000) + 3_600), "claude-sonnet-5-5")).toBe(ids[1]!); + expect(getAnthropicAccountHealthSnapshot(ids[0]!)).not.toBeNull(); +}); diff --git a/tests/adapters/anthropic/anthropic-funded-roster.test.ts b/tests/adapters/anthropic/anthropic-funded-roster.test.ts index ebd2df94e83..e4a7a6769ef 100644 --- a/tests/adapters/anthropic/anthropic-funded-roster.test.ts +++ b/tests/adapters/anthropic/anthropic-funded-roster.test.ts @@ -23,7 +23,7 @@ const model = "claude-sonnet-4-6"; function answer(stream = false, text = "The answer is complete."): Response { const message = { id: "msg_roster", type: "message", role: "assistant", model, - content: [{ type: "text", text }], stop_reason: "end_turn", usage: { input_tokens: 8, output_tokens: 6 } }; + content: [{ type: "text", text }], stop_reason: "end_turn", stop_sequence: null, usage: { input_tokens: 8, output_tokens: 6 } }; if (!stream) return Response.json(message); const frames = [ { type: "message_start", message: { ...message, content: [], stop_reason: null } }, @@ -148,6 +148,33 @@ for (const native of [false, true]) { expect(new Set(sends).size).toBe(6); expect(fixture.routing.getEligibleAnthropicAccounts()).toHaveLength(2); }); + test(`native=${native}: an AgentRadar gateway funds sixteen accounts, so seven refusals reach a healthy eighth`, async () => { + const previous = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + try { + await seed(8); + reply = (index, body) => index <= 7 ? refusal() : answer(body.stream === true); + const { response, text } = await post(native); + expect(response.status).toBe(200); + expect(text).toContain("The answer is complete."); + expect(sends).toEqual(Array.from({ length: 8 }, (_, index) => `Bearer synthetic-roster-${index}`)); + } finally { if (previous === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previous; } + }); + test(`native=${native}: the hosted roster stays inside an explicit physical ceiling`, async () => { + const previous = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + try { + await seed(8); + const budget = createRequestExecutionBudget({ maxTotalModelSends: 3, baseSendAllowance: 3, + finalRecoveryAllowance: 0, maxAlternateTargetSends: 0, maxTargetTransitions: 0 }); + if (native) budgetSpy = spyOn(inference, "createInferenceSendBudget").mockReturnValue(budget); + else options.sendBudget = budget; + reply = () => refusal(); + const { response } = await post(native); + expect(response.status).toBe(429); + expect(sends).toHaveLength(3); + } finally { if (previous === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previous; } + }); test(`native=${native}: accounts added mid-turn cannot expand initial four-account snapshot`, async () => { await seed(4); reply = async index => { if (index === 1) await seed(4, 4); return refusal(); }; diff --git a/tests/adapters/anthropic/anthropic-pool-extra-usage-400.test.ts b/tests/adapters/anthropic/anthropic-pool-extra-usage-400.test.ts new file mode 100644 index 00000000000..45cab846a70 --- /dev/null +++ b/tests/adapters/anthropic/anthropic-pool-extra-usage-400.test.ts @@ -0,0 +1,200 @@ +// Ported from v2.65.0-vadymhimself.27 (45776369ce): Anthropic answers a spent plan whose extra +// usage is also spent with HTTP 400, not 429; one spent pool account must not fail the request. +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../../src/config"; +import { clearPoolRotationState } from "../../../src/codex/pool-rotation"; +import { clearAnthropicAccountPoolState, getAnthropicAccountHealthSnapshot, rotateAnthropicAccountOnRefusal } from "../../../src/oauth/anthropic-routing"; +import { readAnthropicExtraUsageRefusal } from "../../../src/providers/anthropic-billing-refusal"; +import { getAccountSet, saveCredential, setActiveAccount } from "../../../src/oauth/store"; +import { startServer } from "../../../src/server"; +import { clearRequestLogsForTests, getRequestLogEntries } from "../../../src/server/request-log"; +import type { OcxConfig } from "../../../src/types"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "../../helpers/isolated-codex-home"; +import { removeTreeWithRetry } from "../../helpers/remove-tree"; +import { fixturePath } from "../../helpers/repo-root"; + +const OUT_OF_EXTRA_USAGE = "You're out of extra usage. Add more at claude.ai/settings/usage and keep going."; +// A recorded gateway log line cut at 200 chars; only the closing quote and braces the cut removed are added. +const RECORDED_THIRD_PARTY_LINE = readFileSync(fixturePath("anthropic-extra-usage-thirdparty-400.txt"), "utf8"); +const RECORDED_THIRD_PARTY_BODY = `${RECORDED_THIRD_PARTY_LINE.slice("Provider error 400: ".length)}"}}`; +const ROLE_ORDER = "messages: roles must alternate between \"user\" and \"assistant\", but found multiple \"user\" roles in a row"; +const envelope = (type: string, message: string) => JSON.stringify({ type: "error", error: { type, message } }); +const json400 = (body: string) => new Response(body, { status: 400, headers: { "content-type": "application/json" } }); +const rotates = (body: string) => readAnthropicExtraUsageRefusal(json400(body)); + +function anthropicTextSse(text: string): Response { + return new Response([ + 'event: message_start\ndata: {"type":"message_start","message":{"id":"msg_test","type":"message","role":"assistant","model":"claude-opus-5","content":[],"stop_reason":null,"usage":{"input_tokens":3,"output_tokens":0}}}\n\n', + `event: content_block_start\ndata: ${JSON.stringify({ type: "content_block_start", index: 0, content_block: { type: "text", text: "" } })}\n\n`, + `event: content_block_delta\ndata: ${JSON.stringify({ type: "content_block_delta", index: 0, delta: { type: "text_delta", text } })}\n\n`, + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":0}\n\n', + `event: message_delta\ndata: ${JSON.stringify({ type: "message_delta", delta: { stop_reason: "end_turn" }, usage: { output_tokens: 1 } })}\n\n`, + 'event: message_stop\ndata: {"type":"message_stop"}\n\n', + ].join(""), { headers: { "content-type": "text/event-stream" } }); +} + +let testDir = ""; +let previousHome: string | undefined; +let isolatedCodexHome: IsolatedCodexHome | null = null; +const originalFetch = globalThis.fetch; +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + isolatedCodexHome = installIsolatedCodexHome("ocx-pool-400-"); + testDir = mkdtempSync(join(tmpdir(), "ocx-pool-400-")); + process.env.OPENCODEX_HOME = testDir; + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + clearRequestLogsForTests(); +}); +afterEach(() => { + globalThis.fetch = originalFetch; + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + isolatedCodexHome?.restore(); + isolatedCodexHome = null; + if (testDir) removeTreeWithRetry(testDir); +}); + +describe("extra-usage 400 evidence", () => { + test("the recorded fixture is Anthropic's envelope, kept as recorded", () => { + expect(RECORDED_THIRD_PARTY_LINE.startsWith("Provider error 400: {\"type\":\"error\"")).toBe(true); + expect(JSON.parse(RECORDED_THIRD_PARTY_BODY)).toEqual({ type: "error", error: { type: "invalid_request_error", + message: "Third-party apps now draw from your extra usage, not your plan limits. Add more at claude.ai/settings/usage and k" } }); + }); + + test("reading the evidence leaves the whole 400 body for the caller", async () => { + const response = json400(envelope("invalid_request_error", ROLE_ORDER)); + expect(await readAnthropicExtraUsageRefusal(response)).toBe(false); + expect(response.status).toBe(400); + expect(await response.text()).toContain("roles must alternate"); + }); + + test("both wordings rotate only inside Anthropic's envelope", async () => { + expect(await rotates(RECORDED_THIRD_PARTY_BODY)).toBe(true); + expect(await rotates(envelope("invalid_request_error", OUT_OF_EXTRA_USAGE))).toBe(true); + expect(await rotates(envelope("invalid_request_error", " You’re out of extra usage."))).toBe(true); + expect(await rotates(envelope("invalid_request_error", "YOU'RE OUT OF EXTRA USAGE"))).toBe(true); + expect(await rotates(OUT_OF_EXTRA_USAGE)).toBe(false); + expect(await rotates(RECORDED_THIRD_PARTY_LINE)).toBe(false); + }); + + test("echoed client values, mid-message wording, wrong envelopes and other 400s never rotate", async () => { + for (const value of ["toolu_insufficient_quota", "toolu_out_of_extra_usage", "You're out of extra usage", "Third-party apps now draw from your extra usage"]) { + expect(await rotates(envelope("invalid_request_error", `tool_use ids were found without tool_result blocks: ${value}`))).toBe(false); + } + for (const message of [`messages.0.content.0.text: ${OUT_OF_EXTRA_USAGE}`, "Invalid model: you're out of extra usage", "x You're out of extra usage", ROLE_ORDER, + "The prompt says: Third-party apps now draw from your extra usage, not your plan limits"]) { + expect(await rotates(envelope("invalid_request_error", message))).toBe(false); + } + for (const body of [ + envelope("rate_limit_error", OUT_OF_EXTRA_USAGE), envelope("api_error", OUT_OF_EXTRA_USAGE), + JSON.stringify({ error: { type: "invalid_request_error", message: OUT_OF_EXTRA_USAGE } }), + JSON.stringify({ type: "invalid_request_error", error: { type: "invalid_request_error", message: OUT_OF_EXTRA_USAGE } }), + JSON.stringify({ type: "error", message: OUT_OF_EXTRA_USAGE }), + JSON.stringify({ type: "error", error: { message: OUT_OF_EXTRA_USAGE } }), + JSON.stringify({ type: "error", error: { type: "invalid_request_error", message: [OUT_OF_EXTRA_USAGE] } }), + JSON.stringify([{ type: "error", error: { type: "invalid_request_error", message: OUT_OF_EXTRA_USAGE } }]), "", + ]) expect(await rotates(body)).toBe(false); + expect(await readAnthropicExtraUsageRefusal(new Response(envelope("invalid_request_error", OUT_OF_EXTRA_USAGE), { status: 500, headers: { "content-type": "application/json" } }))).toBe(false); + }); +}); + +describe("extra-usage 400 cooldown", () => { + const config = { port: 0, defaultProvider: "anthropic", + providers: { anthropic: { adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "oauth" } }, + anthropicAccountPool: { enabled: true, autoSwitchThreshold: 100 } } as OcxConfig; + async function seedTwoAccounts() { + for (const id of ["a", "b"]) await saveCredential("anthropic", { access: `access-${id}`, refresh: `refresh-${id}`, expires: Date.now() + 3_600_000, accountId: `uuid-${id}` }); + const set = getAccountSet("anthropic")!; + const aId = set.accounts.find(account => account.credential.accountId === "uuid-a")!.id; + await setActiveAccount("anthropic", aId); + return { aId, bId: set.accounts.find(account => account.credential.accountId === "uuid-b")!.id }; + } + + test("without a reset it cools for ten minutes even while unified status reads allowed", async () => { + const { aId, bId } = await seedTwoAccounts(); + const now = 1_800_000_000_000; + const allowed = new Headers({ "anthropic-ratelimit-unified-status": "allowed" }); + expect(rotateAnthropicAccountOnRefusal(config, aId, 400, null, "s", now, allowed, null, "claude-opus-5")).toBe(bId); + expect(getAnthropicAccountHealthSnapshot(aId, now)?.cooldownUntil).toBe(now + 10 * 60_000); + }); + + test("a rejected window's reset sets the deadline", async () => { + const { aId } = await seedTwoAccounts(); + const now = 1_800_000_000_000; + const resetAt = now / 1000 + 3 * 3600; + rotateAnthropicAccountOnRefusal(config, aId, 400, null, "s", now, new Headers({ + "anthropic-ratelimit-unified-status": "rejected", "anthropic-ratelimit-unified-5h-status": "rejected", + "anthropic-ratelimit-unified-5h-reset": String(resetAt) }), null, "claude-opus-5"); + expect(getAnthropicAccountHealthSnapshot(aId, now)?.cooldownUntil).toBe(resetAt * 1000); + }); +}); + +describe("routed /v1/messages through the Anthropic pool", () => { + async function send(first: () => Response, providerExtra: object = {}, later: () => Response = () => anthropicTextSse("rotated")) { + const authorizations: Array = []; + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const request = new Request(input, init); + if (request.url !== "https://api.anthropic.com/v1/messages") throw new Error("unexpected fixture egress"); + authorizations.push(request.headers.get("authorization")); + return authorizations.length === 1 ? first() : later(); + }) as typeof fetch; + writeFileSync(join(testDir, "auth.json"), JSON.stringify({ anthropic: { activeAccountId: "account-a", accounts: ["a", "b"].map(id => ({ + id: `account-${id}`, credential: { access: `synthetic-access-${id}`, refresh: `synthetic-refresh-${id}`, expires: 9999999999999, accountId: `synthetic-account-${id}` }, + })) } }), { mode: 0o600 }); + // Hosted-gateway shape: no combo, the pool alone. + saveConfig({ port: 0, defaultProvider: "anthropic", + providers: { anthropic: { adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "oauth", ...providerExtra } }, + anthropicAccountPool: { enabled: true, autoSwitchThreshold: 100 } } as OcxConfig); + const server = startServer(0); + try { + const response = await originalFetch(new URL("/v1/messages", server.url), { method: "POST", + headers: { "content-type": "application/json", "x-api-key": "placeholder" }, + body: JSON.stringify({ model: "claude-opus-5", max_tokens: 64, stream: true, messages: [{ role: "user", content: [{ type: "text", text: "hi" }] }] }) }); + return { response, text: await response.text(), authorizations }; + } finally { await server.stop(true); } + } + + test("an out-of-extra-usage 400 on account A retries on account B and cools A", async () => { + const { response, text, authorizations } = await send(() => json400(envelope("invalid_request_error", OUT_OF_EXTRA_USAGE))); + expect(response.status, text).toBe(200); + expect(text).toContain("rotated"); + expect(authorizations).toEqual(["Bearer synthetic-access-a", "Bearer synthetic-access-b"]); + const entry = getRequestLogEntries().findLast(row => row.surface === "claude"); + expect(entry?.attempts).toHaveLength(2); + // Native Messages labels its pool quota hop as it labels a 429 hop; AgentRadar hosted gateways keep + // .27's label from the Responses bridge, anthropic-oauth-429. + const hopLabel = process.env.OPENCODEX_AGENTRADAR === "1" ? "anthropic-oauth-429" : "rate-limit-429"; + expect(entry?.attempts?.[1]).toMatchObject({ status: 200, recoveryKinds: [hopLabel] }); + expect(getAnthropicAccountHealthSnapshot("account-a")?.cooldownUntil).toBeGreaterThan(Date.now() + 9 * 60_000); + }); + + test("the recorded third-party-apps 400 rotates too", async () => { + const { response, authorizations } = await send(() => json400(RECORDED_THIRD_PARTY_BODY)); + expect(response.status).toBe(200); + expect(authorizations).toEqual(["Bearer synthetic-access-a", "Bearer synthetic-access-b"]); + }); + + test("a malformed-request or echoing 400 reaches the caller unchanged", async () => { + for (const message of [ROLE_ORDER, "messages.1: `tool_use` ids were found without `tool_result` blocks immediately after: toolu_insufficient_quota"]) { + clearAnthropicAccountPoolState(); + const { response, text, authorizations } = await send(() => json400(envelope("invalid_request_error", message))); + expect(response.status).toBe(400); + expect(text).not.toContain("rotated"); + expect(authorizations).toEqual(["Bearer synthetic-access-a"]); + expect(getAnthropicAccountHealthSnapshot("account-a")).toBeNull(); + } + }); + + test("when every account is spent the ladder stops at the roster and returns the 400", async () => { + const spent = () => json400(envelope("invalid_request_error", OUT_OF_EXTRA_USAGE)); + const { response, authorizations } = await send(spent, {}, spent); + expect(response.status).toBe(400); + expect(authorizations).toEqual(["Bearer synthetic-access-a", "Bearer synthetic-access-b"]); + }); +}); diff --git a/tests/adapters/anthropic/anthropic-pool-usage-limit.test.ts b/tests/adapters/anthropic/anthropic-pool-usage-limit.test.ts index 466904c6600..0810accbaf4 100644 --- a/tests/adapters/anthropic/anthropic-pool-usage-limit.test.ts +++ b/tests/adapters/anthropic/anthropic-pool-usage-limit.test.ts @@ -255,6 +255,27 @@ test.each(["transient throttle", "entitlement 403"] as const)("one %s among quot expect(sent).toHaveLength(0); }); +test("AgentRadar hosted: a headerless 429 cools and rotates but never reports a usage limit", async () => { + const previous = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + try { + const ids = await seed(2); + const config = configFor(true); + const now = Date.now(); + holdOnQuota(config, ids[0]!, 20 * 60, now); + // Hosted .27: no unified status still cools the account (shared-quota), but is no proof of spent quota. + expect(recordAnthropicAccount429(config, ids[1]!, "30", now, new Headers())).toBe(true); + const response = await post(config); + const error = await errorOf(response); + expect(response.status).toBe(429); + expect(error).toMatchObject({ type: "rate_limit_error", code: "rate_limit_exceeded" }); + expect(error).not.toHaveProperty("resets_at"); + expect(sent).toHaveLength(0); + } finally { + if (previous === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previous; + } +}); + test("the last account refusing on spent quota answers the usage limit at once", async () => { const ids = await seed(2); const config = configFor(true, () => spent("5h", 1800)); diff --git a/tests/adapters/anthropic/anthropic-revoked-token-continuation.test.ts b/tests/adapters/anthropic/anthropic-revoked-token-continuation.test.ts index 45eafc3465a..7a1f2b30941 100644 --- a/tests/adapters/anthropic/anthropic-revoked-token-continuation.test.ts +++ b/tests/adapters/anthropic/anthropic-revoked-token-continuation.test.ts @@ -124,4 +124,19 @@ for (const instance of instances) { ]); expect(store.getAccountCredentialWithStatus(instance, fixture.ids[0])?.needsReauth).toBe(false); }); + test(instance + ": AgentRadar hosted re-ask after output still never rotates on a revoked 401 (.27 set)", async () => { + const previous = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + try { + reply = (_instance, index) => index === 1 ? answerStream("I will modify the file now.") : revoked(); + const response = await post(instance, {}, { stream: true, input: "Please modify the file now", + tools: [{ type: "function", name: "read_file", description: "read a file", parameters: { type: "object" } }] }); + expect(await response.text()).toContain("I will modify the file now."); + expect(sends.map(send => send.token)).toEqual([ + "Bearer synthetic-" + instance + "-1-access", "Bearer synthetic-" + instance + "-1-access", + ]); + } finally { + if (previous === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previous; + } + }); } diff --git a/tests/adapters/anthropic/anthropic-sidecar-account-failover.test.ts b/tests/adapters/anthropic/anthropic-sidecar-account-failover.test.ts index 4eb75c30559..823d1ceaf6f 100644 --- a/tests/adapters/anthropic/anthropic-sidecar-account-failover.test.ts +++ b/tests/adapters/anthropic/anthropic-sidecar-account-failover.test.ts @@ -4,13 +4,13 @@ * recover there: `anthropicAccountPool.enabled: false` declines PROACTIVE routing, not the * reactive retry that runs only after upstream has already refused the request. */ -import { afterAll, afterEach, beforeAll, beforeEach, expect, mock, test } from "bun:test"; +import { afterAll, afterEach, beforeAll, beforeEach, expect, mock, spyOn, test } from "bun:test"; import { mkdtempSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import type { AdapterRequest, IncomingMeta, ProviderAdapter } from "../../../src/adapters/base"; import type { AttemptRecoveryKind } from "../../../src/usage/log"; -import { clearAnthropicAccountPoolState } from "../../../src/oauth/anthropic-routing"; +import { clearAnthropicAccountPoolState, getAnthropicAccountHealthSnapshot } from "../../../src/oauth/anthropic-routing"; import { clearGenericFailoverHealth } from "../../../src/oauth/generic-account-failover"; import { getAccountSet, saveCredential, setActiveAccount } from "../../../src/oauth/store"; import { clearAccountQuotaCache, getCachedProviderAccountQuota, resetProviderQuotaReconcileStateForTests } from "../../../src/providers/quota"; @@ -94,7 +94,7 @@ beforeAll(async () => { const rotated = await args.on429?.(response.headers.get("retry-after"), response.headers, args.parsed, response); if (!rotated) return response; await response.body?.cancel(); - expect(rotated.recoveryKind).toBe("anthropic-oauth-429"); + expect(rotated.recoveryKind).toBe(response.status === 403 ? "oauth-account-403" : "anthropic-oauth-429"); adapter = rotated.adapter; } throw new Error("Anthropic sidecar exceeded bounded fixture roster"); @@ -135,6 +135,49 @@ afterAll(() => { restoreModuleMocks?.(); }); +test.each([new Error("Synthetic rotation failure"), new DOMException("Synthetic cancellation", "AbortError")])( + "Anthropic sidecar refunds unused hop and rethrows %s", async error => { + sidecarMode = true; + for (let index = 0; index < 2; index++) await saveCredential("anthropic", { + access: `anthropic-access-${index}`, refresh: `anthropic-refresh-${index}`, + expires: Date.now() + 3_600_000, accountId: `anthropic-account-${index}`, + }); + await setActiveAccount("anthropic", getAccountSet("anthropic")!.accounts[0]!.id); + const { createRequestExecutionBudget } = await import("../../../src/lib/request-execution-budget"); + const sendBudget = createRequestExecutionBudget({ maxTotalModelSends: 2, baseSendAllowance: 1, + finalRecoveryAllowance: 1, maxAlternateTargetSends: 0, maxTargetTransitions: 0 }); + const refusal = await import("../../../src/oauth/anthropic-account-refusal"); + const rotation = spyOn(refusal, "rotateAnthropicAccountOnResponseForInstance").mockImplementation(async () => { + expect(sendBudget.used).toBe(2); + expect(sendBudget.reserveSpent).toBe(true); + throw error; + }); + const config = { + port: 0, defaultProvider: "anthropic", + anthropicAccountPool: { enabled: false, strategy: "round-robin" }, + providers: { anthropic: { + adapter: "test-anthropic-sidecar", baseUrl: "https://anthropic-sidecar.test/v1", + authMode: "oauth", models: ["model"], + fetch: (async (_input, init) => { + observedKeys.push(new Headers(init?.headers).get("authorization") ?? ""); + return new Response("rate limited", { status: 429 }); + }) as typeof fetch, + } }, + } as unknown as OcxConfig; + try { + await expect(handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "anthropic/model", input: "search", stream: true, + tools: [{ type: "web_search" }] }), + }), config, { model: "", provider: "" }, { sendBudget })).rejects.toBe(error); + expect(rotation).toHaveBeenCalledTimes(1); + expect(observedKeys).toEqual(["Bearer anthropic-access-0"]); + expect(sendBudget.used).toBe(1); + expect(sendBudget.reserveSpent).toBe(false); + } finally { rotation.mockRestore(); } + }, +); + test.each([[2, false], [6, false], [8, false], [6, true]] as const)("Anthropic sidecar snapshots %i accounts, spent budget=%s", async (count, spent) => { sidecarMode = true; for (let index = 0; index < count; index += 1) { @@ -205,3 +248,115 @@ test.each([[2, false], [6, false], [8, false], [6, true]] as const)("Anthropic s expect(getCachedProviderAccountQuota("anthropic", ids[0]!)).toMatchObject({ fiveHourPercent: 100, weeklyPercent: 61 }); if (count !== 8 && !spent) expect(getCachedProviderAccountQuota("anthropic", ids[expectedSends - 1]!)).toMatchObject({ fiveHourPercent: 23, weeklyPercent: 47 }); }); + +// Parity with v2.65.0-vadymhimself.27 "Anthropic web-search sidecar rotates on 403 credential denial". +// Upstream routes the 403 through the same physical-send-bound hook as a 429, so the refusal is a +// real send the pool owns rather than a synthetic response handed to a separate callback. +test("Anthropic web-search sidecar rotates on 403 credential denial", async () => { + sidecarMode = true; + for (let index = 0; index < 2; index += 1) { + await saveCredential("anthropic", { + access: `anthropic-access-${index}`, refresh: `anthropic-refresh-${index}`, + expires: Date.now() + 3_600_000, accountId: `anthropic-account-${index}`, + }); + } + const ids = getAccountSet("anthropic")!.accounts.map(account => account.id); + await setActiveAccount("anthropic", ids[0]!); + const config = { + port: 0, defaultProvider: "anthropic", + anthropicAccountPool: { enabled: false, strategy: "round-robin" }, + providers: { anthropic: { adapter: "test-anthropic-sidecar", baseUrl: "https://anthropic-sidecar.test/v1", authMode: "oauth", models: ["model"], + fetch: (async (_input, init) => { + observedKeys.push(new Headers(init?.headers).get("authorization") ?? ""); + return observedKeys.length === 1 ? Response.json({ type: "error", error: { type: "oauth_org_not_allowed", + message: "Your organization has disabled Claude subscription access for Claude Code. Use an Anthropic API key instead, or ask your admin to enable access." } }, { status: 403 }) + : new Response("sidecar-ok"); + }) as typeof fetch } }, + } as unknown as OcxConfig; + const request = () => new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "anthropic/model", input: "search", stream: true, tools: [{ type: "web_search" }] }), + }); + const response = await handleResponses(request(), config, { model: "", provider: "" }); + expect(response.status).toBe(200); + expect(await response.text()).toBe("sidecar-ok"); + expect(observedKeys).toEqual(["Bearer anthropic-access-0", "Bearer anthropic-access-1"]); + // Ordinary installs cool the entitlement refusal, so a re-enabled organization serves again without a login. + expect(getAnthropicAccountHealthSnapshot(ids[0]!)?.cooldownUntil).toBeGreaterThan(Date.now() + 9 * 60_000); + expect(getAccountSet("anthropic")!.accounts.find(account => account.id === ids[0])?.needsReauth).not.toBe(true); + +}); + +// Upstream routes the 403 through the same physical-send-bound hook as a 429, so the refusal is a +// real send the pool owns rather than a synthetic response handed to a separate callback. +test("Anthropic web-search sidecar rotates on 403 credential denial (AgentRadar: org-disabled needs reauth)", async () => { + const previous = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + try { + sidecarMode = true; + for (let index = 0; index < 2; index += 1) { + await saveCredential("anthropic", { + access: `anthropic-access-${index}`, refresh: `anthropic-refresh-${index}`, + expires: Date.now() + 3_600_000, accountId: `anthropic-account-${index}`, + }); + } + const ids = getAccountSet("anthropic")!.accounts.map(account => account.id); + await setActiveAccount("anthropic", ids[0]!); + const config = { + port: 0, defaultProvider: "anthropic", + anthropicAccountPool: { enabled: false, strategy: "round-robin" }, + providers: { anthropic: { adapter: "test-anthropic-sidecar", baseUrl: "https://anthropic-sidecar.test/v1", authMode: "oauth", models: ["model"], + fetch: (async (_input, init) => { + observedKeys.push(new Headers(init?.headers).get("authorization") ?? ""); + return observedKeys.length === 1 ? Response.json({ type: "error", error: { type: "oauth_org_not_allowed", + message: "Your organization has disabled Claude subscription access for Claude Code. Use an Anthropic API key instead, or ask your admin to enable access." } }, { status: 403 }) + : new Response("sidecar-ok"); + }) as typeof fetch } }, + } as unknown as OcxConfig; + const request = () => new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "anthropic/model", input: "search", stream: true, tools: [{ type: "web_search" }] }), + }); + const response = await handleResponses(request(), config, { model: "", provider: "" }); + expect(response.status).toBe(200); + expect(await response.text()).toBe("sidecar-ok"); + expect(observedKeys).toEqual(["Bearer anthropic-access-0", "Bearer anthropic-access-1"]); + // .27 contract on gateways: the org-disabled account rotates and is marked for reauthentication. + expect(getAccountSet("anthropic")!.accounts.find(account => account.id === ids[0])?.needsReauth).toBe(true); + } finally { if (previous === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previous; } + +}); + +// Parity with v2.65.0-vadymhimself.27 "Anthropic sidecar dispatch seam records A429 and B200 when proactive +// pooling is disabled": its 429 carries Retry-After and utilization but no unified status, which upstream +// classifies as a transient throttle; a 30s wait exceeds the inline bound, so the sidecar detours once. +test("Anthropic sidecar dispatch seam records A429 and B200 when proactive pooling is disabled", async () => { + sidecarMode = true; + for (let index = 0; index < 2; index += 1) { + await saveCredential("anthropic", { access: `anthropic-access-${index}`, refresh: `anthropic-refresh-${index}`, + expires: Date.now() + 3_600_000, accountId: `anthropic-account-${index}` }); + } + const ids = getAccountSet("anthropic")!.accounts.map(account => account.id); + await setActiveAccount("anthropic", ids[0]!); + const config = { + port: 0, defaultProvider: "anthropic", anthropicAccountPool: { enabled: false, strategy: "round-robin" }, + providers: { anthropic: { adapter: "test-anthropic-sidecar", baseUrl: "https://anthropic-sidecar.test/v1", authMode: "oauth", models: ["model"], + fetch: (async (_input, init) => { + observedKeys.push(new Headers(init?.headers).get("authorization") ?? ""); + if (observedKeys.length === 1) return new Response("rate limited", { status: 429, headers: { + "retry-after": "30", "anthropic-ratelimit-unified-5h-utilization": "1", "anthropic-ratelimit-unified-7d-utilization": "0.61" } }); + expect(observedKeys).toHaveLength(2); + expect(getCachedProviderAccountQuota("anthropic", ids[0]!)).toMatchObject({ fiveHourPercent: 100, weeklyPercent: 61 }); + return new Response("sidecar-ok", { headers: { "anthropic-ratelimit-unified-5h-utilization": "0.23", "anthropic-ratelimit-unified-7d-utilization": "0.47" } }); + }) as typeof fetch } }, + } as unknown as OcxConfig; + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "anthropic/model", input: "search", stream: true, tools: [{ type: "web_search" }] }), + }), config, { model: "", provider: "" }); + expect(response.status).toBe(200); + expect(await response.text()).toBe("sidecar-ok"); + expect(observedKeys).toEqual(["Bearer anthropic-access-0", "Bearer anthropic-access-1"]); + expect(getCachedProviderAccountQuota("anthropic", ids[0]!)).toMatchObject({ fiveHourPercent: 100, weeklyPercent: 61 }); + expect(getCachedProviderAccountQuota("anthropic", ids[1]!)).toMatchObject({ fiveHourPercent: 23, weeklyPercent: 47 }); +}); diff --git a/tests/adapters/anthropic/anthropic-weekly-spent-gate.test.ts b/tests/adapters/anthropic/anthropic-weekly-spent-gate.test.ts new file mode 100644 index 00000000000..9eef0cc811b --- /dev/null +++ b/tests/adapters/anthropic/anthropic-weekly-spent-gate.test.ts @@ -0,0 +1,87 @@ +// Ported from v2.65.0-vadymhimself.27 (c49bb29146), cached gate only: an AgentRadar process with an +// empty cooldown map must not seat a session on an account whose seven-day window is known spent. +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { clearPoolRotationState } from "../../../src/codex/pool-rotation"; +import { + anthropicPoolCooldownError, bindAnthropicSessionAffinity, clearAnthropicAccountPoolState, + getEligibleAnthropicAccounts, recordAnthropicAccountRefusal, resolveAnthropicAccountForSession, +} from "../../../src/oauth/anthropic-routing"; +import { getAccountSet, saveCredential, setActiveAccount } from "../../../src/oauth/store"; +import { clearAccountQuotaCache, getCachedProviderAccountQuota, setCachedProviderAccountQuotaForTests } from "../../../src/providers/quota"; +import type { OcxConfig } from "../../../src/types"; +import { removeTreeWithRetry } from "../../helpers/remove-tree"; + +const originalHome = process.env.OPENCODEX_HOME; +const originalFlag = process.env.OPENCODEX_AGENTRADAR; +let home: string; +const config = (strategy: "round-robin" | "quota" = "round-robin") => ({ + providers: { anthropic: { adapter: "anthropic", authMode: "oauth" } }, + anthropicAccountPool: { enabled: true, strategy, autoSwitchThreshold: 80 }, +}) as unknown as OcxConfig; +const dayAhead = () => Date.now() + 86_400_000; + +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "ocx-weekly-gate-")); + process.env.OPENCODEX_HOME = home; + process.env.OPENCODEX_AGENTRADAR = "1"; + clearAnthropicAccountPoolState(); clearPoolRotationState(); clearAccountQuotaCache("anthropic"); +}); +afterEach(() => { + clearAnthropicAccountPoolState(); clearPoolRotationState(); clearAccountQuotaCache("anthropic"); + if (originalFlag === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = originalFlag; + if (originalHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = originalHome; + removeTreeWithRetry(home); +}); + +async function seed() { + for (const id of ["a", "b"]) await saveCredential("anthropic", { access: `access-${id}`, refresh: `refresh-${id}`, expires: Date.now() + 3_600_000, accountId: `uuid-${id}` }); + const accounts = getAccountSet("anthropic")!.accounts; + const aId = accounts.find(account => account.credential.accountId === "uuid-a")!.id; + const bId = accounts.find(account => account.credential.accountId === "uuid-b")!.id; + await setActiveAccount("anthropic", aId); + return { aId, bId }; +} + +test("hash seats, restored affinity and the stored active account all skip a weekly-spent account", async () => { + const { aId, bId } = await seed(); + setCachedProviderAccountQuotaForTests("anthropic", aId, { weeklyPercent: 100, weeklyResetAt: dayAhead(), updatedAt: Date.now() }); + setCachedProviderAccountQuotaForTests("anthropic", bId, { weeklyPercent: 20, weeklyResetAt: dayAhead(), updatedAt: Date.now() }); + for (let i = 0; i < 40; i++) expect(resolveAnthropicAccountForSession(`s${i}`, config()).accountId).toBe(bId); + bindAnthropicSessionAffinity("bound", aId); + expect(resolveAnthropicAccountForSession("bound", config("quota")).accountId).toBe(bId); + expect(resolveAnthropicAccountForSession(null, config()).accountId).toBe(bId); +}); + +test("an all-spent pool refuses locally with the earliest weekly reset instead of sending", async () => { + const { aId, bId } = await seed(); + const aReset = dayAhead() + 86_400_000, bReset = dayAhead(); + setCachedProviderAccountQuotaForTests("anthropic", aId, { weeklyPercent: 100, weeklyResetAt: aReset, updatedAt: Date.now() }); + setCachedProviderAccountQuotaForTests("anthropic", bId, { weeklyPercent: 100, weeklyResetAt: bReset, updatedAt: Date.now() }); + expect(getEligibleAnthropicAccounts()).toEqual([]); + expect(resolveAnthropicAccountForSession("s", config())).toMatchObject({ accountId: null, reason: "all-cooled" }); + expect(anthropicPoolCooldownError().usageLimit).toEqual({ resetsAt: bReset }); +}); + +test("a reading without a future reset fails open, and ordinary installs keep the soft preference", async () => { + const { aId, bId } = await seed(); + setCachedProviderAccountQuotaForTests("anthropic", aId, { weeklyPercent: 100, updatedAt: Date.now() }); + setCachedProviderAccountQuotaForTests("anthropic", bId, { weeklyPercent: 100, weeklyResetAt: Date.now() - 1_000, updatedAt: Date.now() }); + expect(getEligibleAnthropicAccounts().sort()).toEqual([aId, bId].sort()); + setCachedProviderAccountQuotaForTests("anthropic", bId, { weeklyPercent: 100, weeklyResetAt: dayAhead(), updatedAt: Date.now() }); + delete process.env.OPENCODEX_AGENTRADAR; + expect(getEligibleAnthropicAccounts().sort()).toEqual([aId, bId].sort()); +}); + +test("a rejected seven-day refusal persists the spent window, so a fresh process skips the account", async () => { + const { aId, bId } = await seed(); + const resetSeconds = Math.ceil(dayAhead() / 1000); + expect(recordAnthropicAccountRefusal(config(), aId, 429, null, Date.now(), new Headers({ + "anthropic-ratelimit-unified-status": "rejected", "anthropic-ratelimit-unified-7d-status": "rejected", + "anthropic-ratelimit-unified-7d-reset": String(resetSeconds) }))).toBe(true); + expect(getCachedProviderAccountQuota("anthropic", aId)).toMatchObject({ weeklyPercent: 100, weeklyResetAt: resetSeconds * 1000 }); + clearAnthropicAccountPoolState(); // the per-process cooldown is gone; the quota cache remains + expect(getEligibleAnthropicAccounts()).toEqual([bId]); +}); diff --git a/tests/adapters/key-failover.test.ts b/tests/adapters/key-failover.test.ts index f5a21b503fb..bda23681317 100644 --- a/tests/adapters/key-failover.test.ts +++ b/tests/adapters/key-failover.test.ts @@ -210,6 +210,19 @@ describe("rotateKeyOn429", () => { expect(getKeyCooldownUntil("p", "k1", now)).toBe(now + 10 * 60_000); }); + test("AgentRadar hosted: caps absurd Retry-After at the max cooldown", () => { + const previous = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + try { + const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3() }); + const now = 1_000_000; + rotateKeyOn429(config, "p", "86400", now); + expect(getKeyCooldownUntil("p", "k1", now)).toBe(now + 60_000); + } finally { + if (previous === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previous; + } + }); + test("skips keys already in cooldown and wraps around the pool", () => { const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3() }); const now = 1_000_000; diff --git a/tests/claude-integration/agentradar-count-tokens.test.ts b/tests/claude-integration/agentradar-count-tokens.test.ts new file mode 100644 index 00000000000..cd96091615e --- /dev/null +++ b/tests/claude-integration/agentradar-count-tokens.test.ts @@ -0,0 +1,611 @@ +// AgentRadar hosted gateways (.27): count_tokens for a canonical Anthropic target is Anthropic's own +// exact count. Ports the .27 claude-messages-endpoint count_tokens cases, plus the pin fix: a pinned +// request counts on its pinned account only, and an unpinned count commits no account selection. +import { afterEach, beforeEach, expect, spyOn } from "bun:test"; +import { readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { installEndpointHooks, test, startServer, testDir, originalFetch } from "../helpers/claude-endpoint-lifetime"; +import { clearComboTargetCooldowns, coolComboTarget } from "../../src/combos"; +import { saveConfig } from "../../src/config"; +import { handleClaudeCountTokens, handleClaudeMessages } from "../../src/server/claude-messages"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { getRequestLogEntries } from "../../src/server/request-log"; +import type { OcxConfig } from "../../src/types"; + +installEndpointHooks(); +const previousHosted = process.env.OPENCODEX_AGENTRADAR; +beforeEach(() => { process.env.OPENCODEX_AGENTRADAR = "1"; }); +afterEach(() => { + if (previousHosted === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previousHosted; +}); + +function routedAnthropicNativeToolSource(): Record { + return { + model: "claude-haiku-4-5", + max_tokens: 128, + stream: true, + messages: [{ role: "user", content: [{ type: "text", text: "use the computer" }] }], + tools: [{ + type: "computer_20250124", + name: "computer", + display_width_px: 1024, + display_height_px: 768, + input_schema: { type: "object" }, + }], + context_management: { source_sentinel: true }, + }; +} + +function keyConfig(extra: Partial, providers: Record = { target: "selected-key" }): OcxConfig { + return { + port: 0, + defaultProvider: Object.keys(providers)[0], + providers: Object.fromEntries(Object.entries(providers).map(([name, apiKey]) => [name, { + adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "key", apiKey, + }])), + ...extra, + } as OcxConfig; +} + +function countRequest(body: unknown, headers: Record = {}, signal?: AbortSignal): Request { + return new Request("http://localhost/v1/messages/count_tokens", { + method: "POST", + headers: { "content-type": "application/json", ...headers }, + body: JSON.stringify(body), + ...(signal ? { signal } : {}), + }); +} + +function writeAccounts(activeAccountId: string, ids: string[]): void { + writeFileSync(join(testDir, "auth.json"), JSON.stringify({ + anthropic: { + activeAccountId, + accounts: ids.map(id => ({ id, credential: { access: `token-${id}`, refresh: "test-refresh-token", expires: 9999999999999 } })), + }, + }), { mode: 0o600 }); +} + +const OAUTH_POOL_CONFIG = { + port: 0, + defaultProvider: "anthropic", + providers: { anthropic: { adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "oauth" } }, + anthropicAccountPool: { enabled: true, autoSwitchThreshold: 100 }, + claudeCode: { modelMap: { "claude-opus-5": "anthropic/claude-opus-5" } }, +} as OcxConfig; + +test("mapped combo count_tokens uses exact canonical Anthropic counting", async () => { + const captured: Array<{ url: string; headers: Headers; body: Record; keepalive?: boolean }> = []; + const upstreamBody = JSON.stringify({ type: "error", error: { type: "rate_limit_error", message: "count limit" } }); + globalThis.fetch = async (input, init) => { + const request = new Request(input, init); + if (request.url !== "https://api.anthropic.com/v1/messages/count_tokens") throw new Error(`unexpected egress ${request.url}`); + captured.push({ url: request.url, headers: request.headers, body: await request.json() as Record, keepalive: init?.keepalive }); + return new Response(upstreamBody, { status: 429, headers: { "content-type": "application/json", "retry-after": "17" } }); + }; + writeAccounts("count-account", ["count-account"]); + saveConfig({ + port: 0, + defaultProvider: "anthropic", + providers: { + anthropic: { + adapter: "anthropic", + baseUrl: "https://api.anthropic.com", + authMode: "oauth", + headers: { "anthropic-beta": "provider-beta,source-beta" }, + }, + }, + anthropicAccountPool: { enabled: true, autoSwitchThreshold: 100 }, + combos: { waterfall: { strategy: "failover", targets: [{ provider: "anthropic", model: "claude-opus-5" }] } }, + claudeCode: { modelMap: { "claude-opus-5": "combo/waterfall" } }, + } as OcxConfig); + const source = { + model: "claude-opus-5", + system: [{ type: "text", text: "You are Claude Code, Anthropic's official CLI for Claude.", cache_control: { type: "ephemeral", ttl: "1h" } }], + messages: [{ role: "user", content: [{ type: "text", text: "count this" }] }], + tools: [{ name: "Read", description: "read", input_schema: { type: "object", properties: {} } }], + metadata: { user_id: "count-session" }, + }; + const server = startServer(0); + try { + const response = await originalFetch(new URL("/v1/messages/count_tokens", server.url), { + method: "POST", + headers: { + "content-type": "application/json", + authorization: "Bearer caller-admission", + "x-api-key": "caller-admission", + "anthropic-version": "2023-06-01", + "anthropic-beta": "source-beta,duplicate-beta,source-beta", + "user-agent": "source-user-agent", + "x-arbitrary-source-header": "must-not-forward", + }, + body: JSON.stringify(source), + }); + expect(response.status).toBe(429); + expect(await response.text()).toBe(upstreamBody); + expect(response.headers.get("retry-after")).toBe("17"); + expect(captured).toHaveLength(1); + expect(captured[0]!.body).toEqual(source); + expect(captured[0]!.headers.get("authorization")).toBe("Bearer token-count-account"); + expect(captured[0]!.headers.get("x-api-key")).toBeNull(); + expect(captured[0]!.headers.get("anthropic-version")).toBe("2023-06-01"); + expect(captured[0]!.headers.get("anthropic-beta")).toBe( + "source-beta,duplicate-beta,provider-beta,claude-code-20250219,oauth-2025-04-20", + ); + expect(captured[0]!.headers.get("user-agent")).not.toBe("source-user-agent"); + expect(captured[0]!.headers.get("x-arbitrary-source-header")).toBeNull(); + expect(captured[0]!.headers.get("connection")).toBe("close"); + expect(captured[0]!.keepalive).toBe(false); + expect(getRequestLogEntries()).toHaveLength(0); + } finally { + await server.stop(true); + } +}); + +test("count_tokens applies generation source eligibility before picking a combo target", async () => { + const captured: Record[] = []; + globalThis.fetch = async (input, init) => { + const request = new Request(input, init); + if (request.url !== "https://api.anthropic.com/v1/messages/count_tokens") throw new Error(`unexpected egress ${request.url}`); + captured.push(await request.json() as Record); + return Response.json({ input_tokens: 123 }); + }; + const config = { + port: 0, + defaultProvider: "target", + providers: { + xai: { adapter: "openai-chat", baseUrl: "https://api.x.ai/v1", authMode: "key", apiKey: "xai-test-key" }, + target: { adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "key", apiKey: "selected-key" }, + }, + combos: { + "count-source-waterfall": { + strategy: "failover", + targets: [ + { provider: "xai", model: "grok-4-1-fast" }, + { provider: "target", model: "claude-opus-5" }, + { provider: "target", model: "claude-haiku-4-5" }, + ], + }, + }, + claudeCode: { modelMap: { "claude-haiku-4-5": "combo/count-source-waterfall" } }, + } as OcxConfig; + + const response = await handleClaudeCountTokens(countRequest(routedAnthropicNativeToolSource()), config); + + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ input_tokens: 123 }); + expect(captured).toEqual([{ ...structuredClone(routedAnthropicNativeToolSource()), model: "claude-haiku-4-5" }]); + expect(getRequestLogEntries()).toHaveLength(0); +}); + +test("count_tokens and following generation share a random combo target", async () => { + // Direct physical dispatch needs the writer lease before the shared spend ledger. + const releaseSpendHome = acquireOwnedSpendHome(); + try { + const captured: Array<{ path: string; model: string }> = []; + globalThis.fetch = async (input, init) => { + const request = new Request(input, init); + const body = await request.json() as { model: string }; + const path = new URL(request.url).pathname; + captured.push({ path, model: body.model }); + if (path === "/v1/messages/count_tokens") { + return Response.json({ input_tokens: 123 }); + } + if (path !== "/v1/messages") throw new Error(`unexpected egress ${request.url}`); + return new Response([ + `event: message_start\ndata: ${JSON.stringify({ type: "message_start", message: { id: "msg_random", type: "message", role: "assistant", model: body.model, content: [], stop_reason: null, usage: { input_tokens: 3, output_tokens: 0 } } })}\n\n`, + 'event: content_block_start\ndata: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}\n\n', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"ok"}}\n\n', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":0}\n\n', + 'event: message_delta\ndata: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":1}}\n\n', + 'event: message_stop\ndata: {"type":"message_stop"}\n\n', + ].join(""), { headers: { "content-type": "text/event-stream" } }); + }; + const config = { + port: 0, + defaultProvider: "first", + providers: { + first: { + adapter: "anthropic", + baseUrl: "https://api.anthropic.com", + authMode: "key", + apiKey: "first-key", + }, + second: { + adapter: "anthropic", + baseUrl: "https://api.anthropic.com", + authMode: "key", + apiKey: "second-key", + }, + }, + combos: { + "count-random": { + strategy: "random", + targets: [ + { provider: "first", model: "claude-opus-5" }, + { provider: "second", model: "claude-haiku-4-5" }, + ], + }, + }, + claudeCode: { modelMap: { "claude-opus-5": "combo/count-random" } }, + } as OcxConfig; + const source = { + model: "claude-opus-5", + metadata: { user_id: "random-count-generation-session" }, + messages: [{ role: "user", content: "same request" }], + }; + const originalRandom = Math.random; + const draws = [0, 0.99, 0.99]; + Math.random = () => draws.shift() ?? 0.99; + try { + const count = await handleClaudeCountTokens(new Request("http://localhost/v1/messages/count_tokens", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(source), + }), config); + expect(count.status).toBe(200); + expect(await count.json()).toEqual({ input_tokens: 123 }); + + const generation = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ ...source, max_tokens: 1024, stream: true }), + }), config, {}); + expect(generation.status).toBe(200); + expect(await generation.text()).toContain("message_stop"); + expect(captured.map(entry => entry.path)).toEqual([ + "/v1/messages/count_tokens", + "/v1/messages", + ]); + expect(captured[1]!.model).toBe(captured[0]!.model); + + const changedSource = { + ...source, + metadata: { user_id: "random-count-generation-cooldown" }, + messages: [{ role: "user", content: "eligibility changes" }], + }; + const changedCount = await handleClaudeCountTokens(new Request("http://localhost/v1/messages/count_tokens", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(changedSource), + }), config); + expect(changedCount.status).toBe(200); + await changedCount.text(); + const countedModel = captured[2]!.model; + const cooledProvider = countedModel === "claude-opus-5" ? "first" : "second"; + coolComboTarget("count-random", { provider: cooledProvider, model: countedModel }, { cooldownMs: 60_000 }); + try { + const changedGeneration = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ ...changedSource, max_tokens: 1024, stream: true }), + }), config, {}); + expect(changedGeneration.status).toBe(200); + await changedGeneration.text(); + expect(captured[3]!.model).not.toBe(countedModel); + } finally { + clearComboTargetCooldowns("count-random"); + } + } finally { + Math.random = originalRandom; + } + } finally { + releaseSpendHome(); + } +}); + +test("count_tokens enforces combo image policy before dispatch", async () => { + let sends = 0; + globalThis.fetch = async () => { + sends++; + return Response.json({ input_tokens: 123 }); + }; + const config = keyConfig({ + combos: { "count-no-images": { strategy: "failover", imageInput: "disabled", targets: [{ provider: "target", model: "claude-opus-5" }] } }, + claudeCode: { modelMap: { "claude-opus-5": "combo/count-no-images" } }, + } as Partial); + + const response = await handleClaudeCountTokens(countRequest({ + model: "claude-opus-5", + messages: [{ + role: "user", + content: [{ + type: "image", + source: { + type: "base64", + media_type: "image/png", + data: "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==", + }, + }], + }], + }), config); + + expect(response.status).toBe(400); + expect(await response.text()).toContain("does not accept image input"); + expect(sends).toBe(0); + expect(getRequestLogEntries()).toHaveLength(0); +}); + +test("count_tokens skips cooled combo targets like generation", async () => { + const capturedModels: string[] = []; + globalThis.fetch = async (input, init) => { + const request = new Request(input, init); + if (request.url !== "https://api.anthropic.com/v1/messages/count_tokens") throw new Error(`unexpected egress ${request.url}`); + capturedModels.push((await request.json() as { model: string }).model); + return Response.json({ input_tokens: 321 }); + }; + const config = keyConfig({ + combos: { + "count-cooldown": { + strategy: "failover", + targets: [{ provider: "first", model: "claude-opus-5" }, { provider: "backup", model: "claude-haiku-4-5" }], + }, + }, + claudeCode: { modelMap: { "claude-haiku-4-5": "combo/count-cooldown" } }, + } as Partial, { first: "first-key", backup: "backup-key" }); + coolComboTarget("count-cooldown", { provider: "first", model: "claude-opus-5" }, { cooldownMs: 60_000 }); + try { + const response = await handleClaudeCountTokens(countRequest({ + model: "claude-haiku-4-5", + messages: [{ role: "user", content: "count this" }], + }), config); + + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ input_tokens: 321 }); + expect(capturedModels).toEqual(["claude-haiku-4-5"]); + expect(getRequestLogEntries()).toHaveLength(0); + } finally { + clearComboTargetCooldowns("count-cooldown"); + } +}); + +test("mapped canonical count_tokens rejects translator overflow before dispatch", async () => { + let sends = 0; + globalThis.fetch = async () => { + sends++; + throw new Error("unexpected oversized count send"); + }; + writeAccounts("overflow-account", ["overflow-account"]); + saveConfig(OAUTH_POOL_CONFIG); + const server = startServer(0); + try { + const response = await originalFetch(new URL("/v1/messages/count_tokens", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "claude-opus-5", messages: [{ role: "user", content: "x".repeat(16 * 1024 * 1024) }] }), + }); + expect(response.status).toBe(413); + expect(await response.text()).toMatch(/translation_buffer_limit/); + expect(sends).toBe(0); + } finally { + await server.stop(true); + } +}); + +test("mapped canonical count_tokens cancels its physical request", async () => { + let resolveStarted!: () => void; + const started = new Promise(resolve => { resolveStarted = resolve; }); + let physicalSignal: AbortSignal | null = null; + globalThis.fetch = async (_input, init) => { + const signal = init?.signal; + physicalSignal = signal ?? null; + resolveStarted(); + return await new Promise((_resolve, reject) => { + if (!signal) return reject(new Error("missing physical abort signal")); + signal.addEventListener("abort", () => reject(signal.reason), { once: true }); + }); + }; + const config = keyConfig({ claudeCode: { modelMap: { "claude-opus-5": "target/claude-opus-5" } } } as Partial); + const client = new AbortController(); + const pending = handleClaudeCountTokens(countRequest({ + model: "claude-opus-5", + messages: [{ role: "user", content: "count this" }], + }, {}, client.signal), config); + await started; + client.abort(new DOMException("client closed", "AbortError")); + expect((await pending).status).toBe(499); + expect((physicalSignal as AbortSignal | null)?.aborted).toBe(true); + expect(getRequestLogEntries()).toHaveLength(0); +}); + +test("mapped noncanonical count_tokens remains local", async () => { + let sends = 0; + globalThis.fetch = async () => { + sends++; + throw new Error("unexpected noncanonical count send"); + }; + const config = { + port: 0, + defaultProvider: "target", + providers: { target: { adapter: "anthropic", baseUrl: "https://compatible.example", authMode: "key", apiKey: "selected-key" } }, + claudeCode: { modelMap: { "claude-opus-5": "target/claude-opus-5" } }, + } as OcxConfig; + const response = await handleClaudeCountTokens(countRequest({ + model: "claude-opus-5", + messages: [{ role: "user", content: "count locally" }], + }), config); + expect(response.status).toBe(200); + expect((await response.json() as { input_tokens: number }).input_tokens).toBeGreaterThan(0); + expect(sends).toBe(0); +}); + +test("incompatible canonical count_tokens fails before dispatch", async () => { + let sends = 0; + globalThis.fetch = async () => { + sends++; + throw new Error("unexpected incompatible count send"); + }; + const config = keyConfig({ claudeCode: { modelMap: { "claude-haiku-4-5": "target/claude-opus-5" } } } as Partial); + const response = await handleClaudeCountTokens(countRequest(routedAnthropicNativeToolSource()), config); + expect(response.status).toBe(400); + expect(await response.text()).toMatch(/replayed exactly/); + expect(sends).toBe(0); + expect(getRequestLogEntries()).toHaveLength(0); +}); + +test("a pinned count_tokens counts on the pinned account only and commits nothing", async () => { + const auths: Array = []; + globalThis.fetch = async (input, init) => { + const request = new Request(input, init); + if (request.url !== "https://api.anthropic.com/v1/messages/count_tokens") throw new Error(`unexpected egress ${request.url}`); + auths.push(request.headers.get("authorization")); + return Response.json({ input_tokens: 77 }); + }; + writeAccounts("acct-a", ["acct-a", "acct-b"]); + saveConfig(OAUTH_POOL_CONFIG); + const server = startServer(0); + try { + const response = await originalFetch(new URL("/v1/messages/count_tokens", server.url), { + method: "POST", + headers: { "content-type": "application/json", "x-agentradar-account-provider": "claude", "x-agentradar-account-id": "acct-b" }, + body: JSON.stringify({ model: "claude-opus-5", messages: [{ role: "user", content: "count on b" }] }), + }); + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ input_tokens: 77 }); + expect(auths).toEqual(["Bearer token-acct-b"]); + expect(JSON.parse(readFileSync(join(testDir, "auth.json"), "utf8")).anthropic.activeAccountId).toBe("acct-a"); + } finally { + await server.stop(true); + } +}); + +test("a pinned count_tokens on an unknown account fails closed with no send", async () => { + let sends = 0; + globalThis.fetch = async () => { + sends++; + return Response.json({ input_tokens: 1 }); + }; + writeAccounts("acct-a", ["acct-a", "acct-b"]); + saveConfig(OAUTH_POOL_CONFIG); + const server = startServer(0); + try { + const response = await originalFetch(new URL("/v1/messages/count_tokens", server.url), { + method: "POST", + headers: { "content-type": "application/json", "x-agentradar-account-provider": "claude", "x-agentradar-account-id": "acct-missing" }, + body: JSON.stringify({ model: "claude-opus-5", messages: [{ role: "user", content: "count" }] }), + }); + expect(response.status).toBe(401); + expect(sends).toBe(0); + } finally { + await server.stop(true); + } +}); + +test("an unpinned pool count_tokens selects like a dispatch without committing the selection", async () => { + const auths: Array = []; + globalThis.fetch = async (input, init) => { + auths.push(new Request(input, init).headers.get("authorization")); + return Response.json({ input_tokens: 5 }); + }; + writeAccounts("acct-a", ["acct-a", "acct-b"]); + saveConfig(OAUTH_POOL_CONFIG); + const server = startServer(0); + try { + const response = await originalFetch(new URL("/v1/messages/count_tokens", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "claude-opus-5", messages: [{ role: "user", content: "count" }] }), + }); + expect(response.status).toBe(200); + expect(auths).toHaveLength(1); + expect(["Bearer token-acct-a", "Bearer token-acct-b"]).toContain(auths[0]!); + expect(JSON.parse(readFileSync(join(testDir, "auth.json"), "utf8")).anthropic.activeAccountId).toBe("acct-a"); + } finally { + await server.stop(true); + } +}); + +test("off the hosted flag count_tokens keeps the local estimate", async () => { + delete process.env.OPENCODEX_AGENTRADAR; + let sends = 0; + globalThis.fetch = async () => { + sends++; + return Response.json({ input_tokens: 1 }); + }; + const config = keyConfig({ claudeCode: { modelMap: { "claude-opus-5": "target/claude-opus-5" } } } as Partial); + const response = await handleClaudeCountTokens(countRequest({ model: "claude-opus-5", messages: [{ role: "user", content: "estimate" }] }), config); + expect(response.status).toBe(200); + expect((await response.json() as { input_tokens: number }).input_tokens).toBeGreaterThan(0); + expect(sends).toBe(0); +}); + +test("off the hosted flag a count_tokens pin header is refused, as on /v1/messages", async () => { + delete process.env.OPENCODEX_AGENTRADAR; + let sends = 0; + globalThis.fetch = async () => { + sends++; + return Response.json({ input_tokens: 1 }); + }; + const config = keyConfig({ claudeCode: { modelMap: { "claude-opus-5": "target/claude-opus-5" } } } as Partial); + const response = await handleClaudeCountTokens(countRequest({ model: "claude-opus-5", messages: [{ role: "user", content: "pinned" }] }, + { "x-agentradar-account-provider": "claude", "x-agentradar-account-id": "acct-a" }), config); + expect(response.status).toBe(400); + expect(await response.text()).toContain("Invalid AgentRadar account pin"); + expect(sends).toBe(0); +}); + +for (const change of ["pause", "remove", "replace", "cancel"] as const) { + test(`a pinned count revalidates its credential after pacing: ${change}`, async () => { + const pacing = await import("../../src/providers/request-pacing"); + const store = await import("../../src/oauth/store"); + writeAccounts("acct-a", ["acct-a", "acct-b"]); + saveConfig(OAUTH_POOL_CONFIG); + let before = store.captureOAuthAccountSelection("anthropic"); + const auths: Array = []; + globalThis.fetch = async (input, init) => { + auths.push(new Request(input, init).headers.get("authorization")); + return Response.json({ input_tokens: 123 }); + }; + let enter!: () => void, resume!: () => void; + const entered = new Promise(resolve => { enter = resolve; }); + const released = new Promise(resolve => { resume = resolve; }); + const realWait = pacing.waitForProviderRequestSlot; + let waits = 0; + const wait = spyOn(pacing, "waitForProviderRequestSlot").mockImplementation(async (...args) => { + if (waits++ === 0) { enter(); await released; } + return realWait(...args); + }); + const client = new AbortController(); + const pending = handleClaudeCountTokens(countRequest({ model: "claude-opus-5", messages: [{ role: "user", content: "count" }] }, + { "x-agentradar-account-provider": "claude", "x-agentradar-account-id": "acct-b" }, client.signal), OAUTH_POOL_CONFIG); + try { + await entered; + if (change === "pause") await store.setAccountPaused("anthropic", "acct-b", true); + if (change === "remove") await store.removeAccount("anthropic", "acct-b"); + if (change === "replace") await store.mergeAccountCredential("anthropic", "acct-b", { + access: "token-b-replaced", refresh: "test-refresh-replaced", expires: 9999999999999, + }); + if (change === "cancel") client.abort(new DOMException("count cancelled while queued", "AbortError")); + before = store.captureOAuthAccountSelection("anthropic"); + } finally { resume(); } + try { + const response = await pending; + await response.text(); + if (change === "replace") { + expect(response.status).toBe(200); + expect(auths).toEqual(["Bearer token-b-replaced"]); + } else { + expect(response.status).toBe(change === "cancel" ? 499 : 502); + expect(auths).toEqual([]); + } + expect(store.captureOAuthAccountSelection("anthropic")).toEqual(before); + } finally { wait.mockRestore(); } + }); +} + +for (const [retryAfter, expected] of [ + ["17", "17"], + ["99999999999999999999", "691200"], + ["9".repeat(129), null], + ["malformed_retry_marker", null], +] as const) { + test(`exact count bounds upstream Retry-After: ${retryAfter.slice(0, 24)}`, async () => { + globalThis.fetch = async () => Response.json({ error: { type: "rate_limit_error", message: "count limit" } }, { + status: 429, headers: { "retry-after": retryAfter }, + }); + const config = keyConfig({ claudeCode: { modelMap: { "claude-opus-5": "target/claude-opus-5" } } } as Partial); + const response = await handleClaudeCountTokens(countRequest({ model: "claude-opus-5", messages: [{ role: "user", content: "count" }] }), config); + expect(response.status).toBe(429); + expect(response.headers.get("retry-after")).toBe(expected); + await response.text(); + }); +} diff --git a/tests/claude-integration/agentradar-hosted-fidelity.test.ts b/tests/claude-integration/agentradar-hosted-fidelity.test.ts new file mode 100644 index 00000000000..4b09e67c5d9 --- /dev/null +++ b/tests/claude-integration/agentradar-hosted-fidelity.test.ts @@ -0,0 +1,285 @@ +/** + * AgentRadar hosted contract (v2.65.0-vadymhimself.27 source replay): a canonical Anthropic route + * receives the caller's Messages body as sent and the client receives Anthropic's bytes as sent, + * pinned or not, whatever the rollout switches say. A translated hop drops thinking text, server + * blocks, citations, pause_turn and stop_sequence; a TTL rewrite changes cache-write billing. + */ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../src/config"; +import { clearAnthropicAccountPoolState, forgetAnthropicFailoverQuorum } from "../../src/oauth/anthropic-routing"; +import { getAccountSet, saveCredential, setActiveAccount } from "../../src/oauth/store"; +import { handleClaudeMessages } from "../../src/server/claude-messages"; +import type { RequestLogContext } from "../../src/server/request-log"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const SSE = [ + 'event:message_start\ndata:{"type":"message_start","message":{"id":"msg_x","type":"message","role":"assistant","model":"claude-opus-5","content":[],"stop_reason":null,"usage":{"input_tokens":3,"cache_creation_input_tokens":5,"cache_read_input_tokens":7,"output_tokens":0}}}\n\n', + 'event: content_block_start\ndata: {"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":""}}\n\n', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"t"}}\n\n', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"signature_delta","signature":"SIG"}}\n\n', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":0}\n\n', + 'event: content_block_start\ndata: {"type":"content_block_start","index":1,"content_block":{"type":"redacted_thinking","data":"RED"}}\n\n', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":1}\n\n', + 'event: content_block_start\ndata: {"type":"content_block_start","index":2,"content_block":{"type":"server_tool_use","id":"srvtoolu_x","name":"code_execution","input":{"code":"1"}}}\n\n', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":2}\n\n', + 'event: content_block_start\ndata: {"type":"content_block_start","index":3,"content_block":{"type":"code_execution_tool_result","tool_use_id":"srvtoolu_x","content":{"type":"code_execution_result","stdout":"1","stderr":"","return_code":0,"content":[]}}}\n\n', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":3}\n\n', + 'event: content_block_start\ndata: {"type":"content_block_start","index":4,"content_block":{"type":"text","text":""}}\n\n', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":4,"delta":{"type":"text_delta","text":"héllo"}}\n\n', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":4,"delta":{"type":"citations_delta","citation":{"type":"char_location","cited_text":"c","document_index":0,"start_char_index":0,"end_char_index":1}}}\n\n', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":4}\n\n', + 'event: message_delta\ndata: {"type":"message_delta","delta":{"stop_reason":"pause_turn","stop_sequence":null},"usage":{"output_tokens":2}}\n\n', + 'event: message_stop\ndata: {"type":"message_stop"}\n\n', +].join(""); +const JSON_BODY = ' {"id" : "msg_j", "type" : "message", "role" : "assistant", "model" : "claude-opus-5",\n' + + ' "content" : [{"type":"text","text":"héllo","citations":[{"type":"char_location","cited_text":"c","document_index":0,"start_char_index":0,"end_char_index":1}]}],\n' + + ' "stop_reason" : "stop_sequence", "stop_sequence" : "END", "usage" : {"input_tokens":2,"output_tokens":1}}\n'; + +const ERROR_BODY = ' {"type":"error","error":{"type":"rate_limit_error","message":"slow down"},"request_id":"req_x"}\n'; + +const SOURCE = { + model: "anthropic/claude-opus-5", + max_tokens: 2048, + system: [ + { type: "text", text: "You are Claude Code, Anthropic's official CLI for Claude." }, + { type: "text", text: "policy", cache_control: { type: "ephemeral" } }, + ], + thinking: { type: "adaptive" }, + context_management: { edits: [{ type: "clear_tool_uses_20250919" }] }, + tools: [ + { type: "code_execution_20250825", name: "code_execution" }, + { name: "lookup", description: "d", input_schema: { type: "object", properties: {} }, cache_control: { type: "ephemeral" } }, + ], + metadata: { user_id: "fixture-user" }, + messages: [{ role: "user", content: [{ type: "text", text: "hi", cache_control: { type: "ephemeral" } }] }], +}; +const BETA = "context-management-2025-06-27,code-execution-2025-08-25"; + +let home = ""; +let sent: Array<{ headers: Headers; body: Record }> = []; +let status = 200; +let previousHome: string | undefined; +let previousHosted: string | undefined; +let releaseSpendHome: (() => void) | undefined; + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + previousHosted = process.env.OPENCODEX_AGENTRADAR; + home = mkdtempSync(join(tmpdir(), "ocx-ar-fidelity-")); + process.env.OPENCODEX_HOME = home; + process.env.OPENCODEX_AGENTRADAR = "1"; + sent = []; + status = 200; + clearAnthropicAccountPoolState(); + forgetAnthropicFailoverQuorum(); + releaseSpendHome = acquireOwnedSpendHome(); +}); + +afterEach(() => { + releaseSpendHome?.(); + clearAnthropicAccountPoolState(); + forgetAnthropicFailoverQuorum(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previousHome; + if (previousHosted === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previousHosted; + removeTreeWithRetry(home); +}); + +function config(authMode: "key" | "oauth"): OcxConfig { + const fetch = (async (_input: Parameters[0], init?: RequestInit) => { + const body = JSON.parse(String(init?.body)) as Record; + sent.push({ headers: new Headers(init?.headers), body }); + if (status !== 200) return new Response(ERROR_BODY, { status, headers: { + "content-type": "application/json", "retry-after": "17", + "anthropic-organization-id": "org-fixture", "anthropic-ratelimit-unified-5h-utilization": "0.97", "request-id": "req_fixture", + } }); + return body.stream === true + ? new Response(SSE, { headers: { "content-type": "text/event-stream" } }) + : new Response(JSON_BODY, { headers: { "content-type": "application/json" } }); + }) as typeof globalThis.fetch; + const provider = { + adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode, models: ["claude-opus-5"], fetch, + ...(authMode === "key" ? { apiKey: "fixture-key" } : {}), + } as OcxProviderConfig; + // No protocols.rollout: the hosted flag alone selects the native lane. + // A key row needs its own name: the registry owns the `anthropic` row's OAuth auth mode. + const name = authMode === "key" ? "anthropic-key" : "anthropic"; + const result = { port: 0, defaultProvider: name, providers: { [name]: provider } } as OcxConfig; + saveConfig(result); + return result; +} + +async function send(cfg: OcxConfig, stream: boolean, headers: Record = {}) { + const logCtx: RequestLogContext = { model: "", provider: "" }; + const response = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", + headers: { "content-type": "application/json", "anthropic-beta": BETA, "x-api-key": "fixture-admission", ...headers }, + body: JSON.stringify({ ...SOURCE, model: `${cfg.defaultProvider}/claude-opus-5`, stream }), + }), cfg, logCtx); + return { response, text: await response.text(), logCtx }; +} + +async function seedTwo(): Promise { + for (const index of [0, 1]) { + await saveCredential("anthropic", { + access: `fixture-access-${index}`, refresh: `fixture-refresh-${index}`, expires: Date.now() + 3_600_000, + accountId: `fixture-account-${index}`, source: "oauth", + }); + } + const ids = getAccountSet("anthropic")!.accounts.map(account => account.id); + await setActiveAccount("anthropic", ids[1]!); + forgetAnthropicFailoverQuorum(); + return ids; +} + +function expectExactSource(body: Record, stream: boolean): void { + // Caller body unchanged but for the wire model: no TTL injection, context_management kept. + expect(body).toEqual({ ...SOURCE, stream, model: "claude-opus-5" }); + expect(JSON.stringify(body)).not.toContain('"ttl"'); +} + +for (const stream of [true, false]) { + test(`hosted unpinned key route relays caller and Anthropic bytes unchanged (stream=${stream})`, async () => { + const { response, text, logCtx } = await send(config("key"), stream); + expect(response.status).toBe(200); + expect(text).toBe(stream ? SSE : JSON_BODY); + // TTL-less breakpoints sent as is: the attempt records the 5-minute lifetime they really have. + expect(logCtx.activeAttempt?.promptCacheTtlMs).toBe(300_000); + expect(sent).toHaveLength(1); + expectExactSource(sent[0]!.body, stream); + expect(sent[0]!.headers.get("anthropic-beta")).toContain("context-management-2025-06-27"); + }); + + test(`hosted pinned OAuth turn serves exactly its account with exact bytes (stream=${stream})`, async () => { + const ids = await seedTwo(); + const { response, text } = await send(config("oauth"), stream, + { "x-agentradar-account-provider": "claude", "x-agentradar-account-id": ids[0]! }); + expect(response.status).toBe(200); + expect(text).toBe(stream ? SSE : JSON_BODY); + expect(sent).toHaveLength(1); + expect(sent[0]!.headers.get("authorization")).toBe("Bearer fixture-access-0"); + expectExactSource(sent[0]!.body, stream); + expect(getAccountSet("anthropic")!.activeAccountId).toBe(ids[1]); + }); +} + +test("a hosted pinned 429 is relayed verbatim from the pinned account without rotating", async () => { + const ids = await seedTwo(); + status = 429; + const { response, text } = await send(config("oauth"), true, + { "x-agentradar-account-provider": "claude", "x-agentradar-account-id": ids[0]! }); + expect(response.status).toBe(429); + expect(text).toBe(ERROR_BODY); + expect(response.headers.get("retry-after")).toBe("17"); + // As .27: only Content-Type and Retry-After travel; the pooled account's org and quota state do not. + for (const name of ["anthropic-organization-id", "anthropic-ratelimit-unified-5h-utilization", "request-id"]) { + expect(response.headers.get(name)).toBeNull(); + } + expect(sent.map(row => row.headers.get("authorization"))).toEqual(["Bearer fixture-access-0"]); + expect(getAccountSet("anthropic")!.activeAccountId).toBe(ids[1]); +}); + +test("a 413 on an image turn is rebuilt once with only the images smaller (.27 image-413)", async () => { + const onePx = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=="; + const png = await new Bun.Image(Buffer.from(onePx, "base64")).resize(1500, 1000).png().toBuffer(); + const image = { type: "image", source: { type: "base64", media_type: "image/png", data: Buffer.from(png).toString("base64") }, + cache_control: { type: "ephemeral", ttl: "1h" } }; + const bodies: Array> = []; + const cfg = config("key"); + (cfg.providers["anthropic-key"] as { fetch: typeof fetch }).fetch = (async (_input: unknown, init?: RequestInit) => { + bodies.push(JSON.parse(String(init?.body))); + return bodies.length === 1 + ? Response.json({ type: "error", error: { type: "request_too_large", message: "too large" } }, { status: 413 }) + : new Response(JSON_BODY, { headers: { "content-type": "application/json" } }); + }) as typeof fetch; + const response = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", + headers: { "content-type": "application/json", "x-api-key": "fixture-admission" }, + body: JSON.stringify({ model: "anthropic-key/claude-opus-5", max_tokens: 128, stream: false, + messages: [{ role: "user", content: [image, { type: "text", text: "look" }] }], context_management: { edits: [] } }), + }), cfg, { model: "", provider: "" }); + expect(response.status).toBe(200); + expect(await response.text()).toBe(JSON_BODY); + expect(bodies).toHaveLength(2); + expect(bodies[0]!.messages[0].content[0]).toEqual(image); + const resized = bodies[1]!.messages[0].content[0]; + expect(resized.source.media_type).toBe("image/jpeg"); + expect(resized.cache_control).toEqual(image.cache_control); + const expected = structuredClone(bodies[0]!); + expected.messages[0].content[0] = resized; + expect(bodies[1]).toEqual(expected); +}); + +test("a hosted combo skips a model-changing target for typed tools and replays exactly on the matching one", async () => { + const cfg = config("key"); + const provider = cfg.providers["anthropic-key"]!; + provider.models = ["claude-opus-5", "claude-haiku-4-5"]; + cfg.combos = { waterfall: { strategy: "failover", targets: [ + { provider: "anthropic-key", model: "claude-opus-5" }, { provider: "anthropic-key", model: "claude-haiku-4-5" }, + ] } }; + cfg.claudeCode = { modelMap: { "claude-haiku-4-5": "combo/waterfall" } }; + saveConfig(cfg); + const response = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", + headers: { "content-type": "application/json", "anthropic-beta": BETA, "x-api-key": "fixture-admission" }, + body: JSON.stringify({ ...SOURCE, model: "claude-haiku-4-5", stream: true }), + }), cfg, { model: "", provider: "" }); + expect(response.status).toBe(200); + expect(await response.text()).toBe(SSE); + expect(sent.map(row => row.body.model)).toEqual(["claude-haiku-4-5"]); + expect(sent[0]!.body).toEqual({ ...SOURCE, model: "claude-haiku-4-5", stream: true }); +}); + +test("a hosted combo named directly by its alias carries typed tools to its first canonical target", async () => { + const cfg = config("key"); + cfg.combos = { waterfall: { strategy: "failover", targets: [{ provider: "anthropic-key", model: "claude-opus-5" }] } }; + saveConfig(cfg); + const response = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", + headers: { "content-type": "application/json", "anthropic-beta": BETA, "x-api-key": "fixture-admission" }, + body: JSON.stringify({ ...SOURCE, model: "combo/waterfall", stream: true }), + }), cfg, { model: "", provider: "" }); + // .27 exempted combo aliases: the caller chose no model to preserve, so typed tools are no refusal. + expect(response.status).toBe(200); + await response.text(); + expect(sent.map(row => row.body.model)).toEqual(["claude-opus-5"]); +}); + +test("a hosted combo never sends typed web_fetch to an Anthropic-compatible third party", async () => { + const cfg = config("key"); + cfg.providers["anthropic-key"]!.baseUrl = "https://compat.example.com"; + cfg.combos = { waterfall: { strategy: "failover", targets: [{ provider: "anthropic-key", model: "claude-opus-5" }] } }; + saveConfig(cfg); + const response = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/waterfall", max_tokens: 64, stream: true, + messages: [{ role: "user", content: "fetch this" }], + tools: [{ type: "web_search_20250305", name: "web_search" }, { type: "web_fetch_20250910", name: "web_fetch" }] }), + }), cfg, { model: "", provider: "" }); + expect(response.status).toBe(400); + await response.text(); + expect(sent).toHaveLength(0); +}); + +test("a hosted combo no target can carry refuses MCP servers with a 400 and no send", async () => { + const cfg = config("key"); + cfg.providers.xai = { adapter: "openai-chat", baseUrl: "https://api.x.ai/v1", authMode: "key", apiKey: "k", + fetch: (async () => { throw new Error("unexpected send"); }) as typeof fetch } as OcxProviderConfig; + cfg.combos = { exact: { strategy: "failover", targets: [{ provider: "xai", model: "grok-4-1-fast" }] } }; + cfg.claudeCode = { modelMap: { "claude-haiku-4-5": "combo/exact" } }; + saveConfig(cfg); + const response = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", + headers: { "content-type": "application/json", "x-api-key": "fixture-admission" }, + body: JSON.stringify({ model: "claude-haiku-4-5", max_tokens: 64, messages: [{ role: "user", content: "hi" }], + mcp_servers: [{ type: "url", name: "docs", url: "https://mcp.example.test" }] }), + }), cfg, { model: "", provider: "" }); + expect(response.status).toBe(400); + expect(await response.text()).toContain("require exact canonical Anthropic Messages replay"); + expect(sent).toHaveLength(0); +}); diff --git a/tests/claude-integration/agentradar-hosted-tap.test.ts b/tests/claude-integration/agentradar-hosted-tap.test.ts new file mode 100644 index 00000000000..51cf8d26dd5 --- /dev/null +++ b/tests/claude-integration/agentradar-hosted-tap.test.ts @@ -0,0 +1,294 @@ +/** + * AgentRadar hosted contract for the Anthropic Messages relay tap, ported from + * v2.65.0-vadymhimself.27 tests/claude-integration/claude-messages-endpoint.test.ts (A0b-A0f). + * Under OPENCODEX_AGENTRADAR=1 the tap relays one validated frame per pull, stops at the first + * terminal, records a root error as failed, and fails a malformed frame before it reaches the + * client. Adapted assertions are marked "upstream:"; everything else is the .27 text. + */ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import type { RequestLogContext } from "../../src/server/request-log"; +import { tapAnthropicSseForLog } from "../../src/server/claude-messages"; + +let previousHosted: string | undefined; +beforeEach(() => { + previousHosted = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; +}); +afterEach(() => { + if (previousHosted === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previousHosted; +}); + +const sseEncoder = new TextEncoder(); + +function spyFinalize() { + const calls: Array<{ status: number; closeReason: string }> = []; + return { + calls, + finalize: (status: number, meta: { closeReason: string }) => calls.push({ status, closeReason: meta.closeReason }), + }; +} +function freshLogCtx(): RequestLogContext { + return { model: "claude-test", provider: "anthropic-native" }; +} + +const MESSAGE_START_FRAME = 'event: message_start\ndata: {"type":"message_start","message":{"usage":{"input_tokens":3}}}\n\n'; + +const MESSAGE_STOP_FRAME = 'event:message_stop\ndata:{"type":"message_stop"}\n\n'; +const ROOT_ERROR_FRAME = 'event:error\ndata:{"type":"error","error":{"type":"overloaded_error","message":"busy"}}\n\n'; +test("A0b: truncated Anthropic SSE preserves prior bytes, records inclusive usage, and appends adapter_eof", async () => { + const prefix = [ + 'event: message_start\ndata: {"type":"message_start","message":{"usage":{"input_tokens":3,"cache_creation_input_tokens":5,"cache_read_input_tokens":7}}}\n\n', + 'event: content_block_start\ndata: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}\n\n', + ].join(""); + const upstream = new ReadableStream({ + start(controller) { + controller.enqueue(sseEncoder.encode(prefix)); + controller.close(); + }, + }); + const ctx = freshLogCtx(); + const calls: Array<{ status: number; meta: unknown }> = []; + const tap = tapAnthropicSseForLog( + upstream, + ctx, + (status, meta) => calls.push({ status, meta }), + { stallMs: 5_000, maxBytes: 0 }, + // upstream: the tap's fifth parameter is the frame context; first output is recorded by the lane. + ); + const text = await new Response(tap).text(); + + expect(text.startsWith(prefix)).toBe(true); + // upstream: its own EOF sentence, logged as a failed turn rather than an incomplete one. + expect(text).toContain("anthropic passthrough upstream stream ended before message_stop"); + expect(ctx.usage).toEqual({ + inputTokens: 15, + outputTokens: 0, + cachedInputTokens: 7, + cacheReadInputTokens: 7, + cacheCreationInputTokens: 5, + }); + expect(calls).toEqual([{ + status: 502, + meta: { closeReason: "terminal", terminalStatus: "failed" }, + }]); +}); + +test("A0c: Anthropic root error is relayed unchanged and recorded failed", async () => { + const frame = 'event:error\ndata:{"type":"error","error":{"type":"overloaded_error","message":"busy"}}\n\n'; + const upstream = new ReadableStream({ + start(controller) { + controller.enqueue(sseEncoder.encode(frame)); + controller.close(); + }, + }); + const calls: Array<{ status: number; meta: unknown }> = []; + const body = await new Response(tapAnthropicSseForLog( + upstream, + freshLogCtx(), + (status, meta) => calls.push({ status, meta }), + { stallMs: 5_000, maxBytes: 0 }, + )).text(); + + expect(body).toBe(frame); + expect(calls).toEqual([{ + status: 502, + meta: { closeReason: "terminal", terminalStatus: "failed" }, + }]); +}); + +test.each([ + { + name: "message_stop then error in one read", + chunks: [MESSAGE_STOP_FRAME + ROOT_ERROR_FRAME], + expectedBody: MESSAGE_STOP_FRAME, + expectedStatus: 200, + expectedTerminal: "completed", + }, + { + name: "message_stop then error across reads", + chunks: [MESSAGE_STOP_FRAME, ROOT_ERROR_FRAME], + expectedBody: MESSAGE_STOP_FRAME, + expectedStatus: 200, + expectedTerminal: "completed", + }, + { + name: "error then message_stop in one read", + chunks: [ROOT_ERROR_FRAME + MESSAGE_STOP_FRAME], + expectedBody: ROOT_ERROR_FRAME, + expectedStatus: 502, + expectedTerminal: "failed", + }, +])("A0c terminal boundary: $name", async ({ + chunks, + expectedBody, + expectedStatus, + expectedTerminal, +}) => { + const pending = [...chunks]; + const upstream = new ReadableStream({ + pull(controller) { + const chunk = pending.shift(); + if (chunk !== undefined) controller.enqueue(sseEncoder.encode(chunk)); + else controller.close(); + }, + }); + const calls: Array<{ status: number; meta: unknown }> = []; + const body = await new Response(tapAnthropicSseForLog( + upstream, + freshLogCtx(), + (status, meta) => calls.push({ status, meta }), + { stallMs: 5_000, maxBytes: 0 }, + )).text(); + + expect(body).toBe(expectedBody); + expect(calls).toEqual([{ + status: expectedStatus, + meta: { closeReason: "terminal", terminalStatus: expectedTerminal }, + }]); +}); + +test("A0d: one large upstream read relays at most one validated frame per pull", async () => { + const frame = `data: {"type":"content_block_delta","padding":"${"x".repeat(2_048)}"}\n\n`; + const chunk = sseEncoder.encode(frame.repeat(2_049) + MESSAGE_STOP_FRAME); + expect(chunk.byteLength).toBeGreaterThan(4 * 1024 * 1024); + const upstream = new ReadableStream({ + start(controller) { + controller.enqueue(chunk); + controller.close(); + }, + }); + const { calls, finalize } = spyFinalize(); + const reader = tapAnthropicSseForLog( + upstream, + freshLogCtx(), + finalize, + { stallMs: 5_000, maxBytes: 0 }, + ).getReader(); + + try { + const first = await reader.read(); + expect(first.done).toBe(false); + expect(new TextDecoder().decode(first.value)).toBe(frame); + expect(calls).toEqual([]); + } finally { + await reader.cancel(); + } +}); + +test("A0e: Anthropic reader rejection appends one failed error terminal", async () => { + let pull = 0; + const upstream = new ReadableStream({ + pull(controller) { + if (pull++ === 0) controller.enqueue(sseEncoder.encode(MESSAGE_START_FRAME)); + else controller.error(new Error("wire broke")); + }, + }); + const calls: Array<{ status: number; meta: unknown }> = []; + const body = await new Response(tapAnthropicSseForLog( + upstream, + freshLogCtx(), + (status, meta) => calls.push({ status, meta }), + { stallMs: 5_000, maxBytes: 0 }, + )).text(); + + expect(body.startsWith(MESSAGE_START_FRAME)).toBe(true); + // upstream: its own reset sentence. + expect(body).toContain("anthropic passthrough upstream stream failed: wire broke"); + expect((body.match(/event: error/g) ?? [])).toHaveLength(1); + expect(calls).toEqual([{ + status: 502, + meta: { closeReason: "terminal", terminalStatus: "failed" }, + }]); +}); + +test("A0f: malformed Anthropic SSE records fail before relay or later terminal", async () => { + const invalidUtf8 = Uint8Array.from([ + ...sseEncoder.encode("data: \"bad_utf8_"), + 0xc3, + 0x28, + ...sseEncoder.encode("\"\n\n"), + ]); + const cases = [ + { name: "invalid UTF-8", frame: invalidUtf8, marker: "bad_utf8_" }, + { name: "malformed JSON", frame: sseEncoder.encode("data: {bad_json_marker\n\n"), marker: "bad_json_marker" }, + { name: "scalar JSON", frame: sseEncoder.encode('data: "bad_scalar_marker"\n\n'), marker: "bad_scalar_marker" }, + { name: "array JSON", frame: sseEncoder.encode('data: ["bad_array_marker"]\n\n'), marker: "bad_array_marker" }, + { name: "Responses sentinel", frame: sseEncoder.encode("data: [DONE]\n\n"), marker: "[DONE]" }, + ]; + + for (const testCase of cases) { + let pulled = false; + let cancels = 0; + const chunk = Uint8Array.from([ + ...sseEncoder.encode(MESSAGE_START_FRAME), + ...testCase.frame, + ...sseEncoder.encode(MESSAGE_STOP_FRAME), + ]); + const upstream = new ReadableStream({ + pull(controller) { + if (pulled) return; + pulled = true; + controller.enqueue(chunk); + }, + cancel() { cancels += 1; }, + }, { highWaterMark: 0 }); + const calls: Array<{ status: number; meta: unknown }> = []; + const body = await new Response(tapAnthropicSseForLog( + upstream, + freshLogCtx(), + (status, meta) => calls.push({ status, meta }), + { stallMs: 5_000, maxBytes: 0 }, + )).text(); + + expect(body.startsWith(MESSAGE_START_FRAME), testCase.name).toBe(true); + expect(body, testCase.name).not.toContain(testCase.marker); + expect(body, testCase.name).not.toContain("message_stop"); + expect(body.match(/event: error/g) ?? [], testCase.name).toHaveLength(1); + expect(cancels, testCase.name).toBe(1); + expect(calls, testCase.name).toEqual([{ + status: 502, + meta: { closeReason: "terminal", terminalStatus: "failed" }, + }]); + } +}); + +test("A0g: a local error presentation keeps its identity through the strict re-framing", async () => { + const { MessagesSseFrameContext } = await import("../../src/lib/messages-sse-error-presentation"); + const { createMessagesSseErrorFilter } = await import("../../src/server/messages-sse-error-filter"); + const { SSE_SENTINEL, sseFrame } = await import("../helpers/messages-sse-cases"); + const frames = new MessagesSseFrameContext(); + // An upstream frame echoing the caller's prompt or a key, marked for replacement by the client filter. + const bytes = new TextEncoder().encode(sseFrame("message_start", { type: "message_start", message: { model: "fixed", usage: {} }, request_id: SSE_SENTINEL })); + frames.register(bytes, { echoRenamedError: true }); + const source = new ReadableStream({ start(controller) { controller.enqueue(bytes); controller.close(); } }); + const text = await new Response(tapAnthropicSseForLog(source, {} as RequestLogContext, () => {}, { stallMs: 0, maxBytes: 0 }, + frames, createMessagesSseErrorFilter(frames))).text(); + expect(text).not.toContain(SSE_SENTINEL); + expect(text.startsWith('event: message_start\ndata: {"type":"error"')).toBe(true); +}); + +for (const [name, tail] of [ + ["malformed JSON", sseEncoder.encode('data: {bad_tail_marker')], + ["non-object JSON", sseEncoder.encode('data: "bad_tail_marker"')], + ["invalid UTF-8", new Uint8Array([...sseEncoder.encode('data: {"text":"bad_tail_marker'), 0xff, ...sseEncoder.encode('"}')])], +] as const) { + test(`strict EOF tail rejects ${name} before relay`, async () => { + const calls: Array<{ status: number; meta: unknown }> = []; + const upstream = new ReadableStream({ start(controller) { controller.enqueue(tail); controller.close(); } }); + const text = await new Response(tapAnthropicSseForLog(upstream, {} as RequestLogContext, + (status, meta) => calls.push({ status, meta }), { stallMs: 0, maxBytes: 0 })).text(); + expect(text).not.toContain("bad_tail_marker"); + expect(text).toContain("anthropic passthrough protocol error"); + expect(calls).toEqual([{ status: 502, meta: { terminalStatus: "failed", closeReason: "terminal" } }]); + }); +} + +test("strict EOF tail completes a valid unterminated terminal", async () => { + const terminal = 'event: message_stop\ndata: {"type":"message_stop"}'; + const calls: number[] = []; + const upstream = new ReadableStream({ start(controller) { controller.enqueue(sseEncoder.encode(terminal)); controller.close(); } }); + const text = await new Response(tapAnthropicSseForLog(upstream, {} as RequestLogContext, + status => calls.push(status), { stallMs: 0, maxBytes: 0 })).text(); + expect(text).toBe(terminal + "\n\n"); + expect(calls).toEqual([200]); +}); diff --git a/tests/claude-integration/claude-inbound.test.ts b/tests/claude-integration/claude-inbound.test.ts index 8030ea4d99b..ba74a8c3162 100644 --- a/tests/claude-integration/claude-inbound.test.ts +++ b/tests/claude-integration/claude-inbound.test.ts @@ -652,6 +652,35 @@ describe("prompt cache key provenance (devlog 130 B3)", () => { .toBe(createHash("sha256").update(raw).digest("hex")); }); + test("every call of one Claude Code session shares one key, whatever else its user_id carries", () => { + const key = (meta: object) => anthropicToResponsesTranslation({ + model: "m", max_tokens: 1, messages, metadata: { user_id: JSON.stringify(meta) }, + }).body.prompt_cache_key; + const base = { device_id: "d".repeat(64), account_uuid: "acct", session_id: "sess-1" }; + // A side call with a different agent token (`tk`) is the same session. + expect(key({ ...base, tk: "agent-a" })).toBe(key({ ...base, tk: "agent-b" })); + expect(key({ ...base, tk: "agent-a" })).toBe(key(base)); + // A fork keys on its parent, so it lands where the parent's cache is. + expect(key({ ...base, session_id: "fork-1", parent_session_id: "sess-1" })).toBe(key(base)); + expect(key({ ...base, session_id: "sess-2" })).not.toBe(key(base)); + }); + + test("an AgentRadar hosted gateway keys a Claude Code session on its (parent) session id alone (.27)", () => { + const prior = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + try { + const key = (meta: object) => anthropicToResponsesTranslation({ + model: "m", max_tokens: 1, messages, metadata: { user_id: JSON.stringify(meta) }, + }).body.prompt_cache_key; + const sessionKey = createHash("sha256").update("s".repeat(36)).digest("hex").slice(0, 32); + expect(key({ device_id: "d".repeat(64), account_uuid: "", session_id: "s".repeat(36) })).toBe(sessionKey); + expect(key({ device_id: "e".repeat(64), account_uuid: "acct", session_id: "s".repeat(36), tk: "t" })).toBe(sessionKey); + expect(key({ device_id: "d".repeat(64), session_id: "fork", parent_session_id: "s".repeat(36) })).toBe(sessionKey); + } finally { + if (prior === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = prior; + } + }); + test("metadata.user_id boundary: exactly 64 chars forwarded, 65 chars hashed", () => { const atLimit = "u".repeat(64); const overLimit = "u".repeat(65); diff --git a/tests/claude-integration/messages-error-contracts.test.ts b/tests/claude-integration/messages-error-contracts.test.ts index b62e7316a62..70b47f9a870 100644 --- a/tests/claude-integration/messages-error-contracts.test.ts +++ b/tests/claude-integration/messages-error-contracts.test.ts @@ -184,9 +184,11 @@ const envelopeWriterRows = [ ["src/claude/outbound.ts", 6, "shared baseline helpers, original verdict and incomplete/unfinished presentation construction"], ["src/protocols/encoders/messages.ts", 4, "original verdict construction and fixed local collection failures"], ["src/server/messages-native.ts", 5, "local/evidence legacy formatters, original combo and fold envelopes, owned workflow refusal projection"], - ["src/server/claude-messages.ts", 30, "mapped local ingress/bypass constructors, original combo and defensive JSON presentation envelopes"], + ["src/server/claude-messages.ts", 33, "mapped local ingress/bypass constructors (incl. the AgentRadar pin on messages and count_tokens, and native-tool exact-replay refusals), original combo and defensive JSON presentation envelopes"], + ["src/server/agentradar-count-tokens.ts", 9, "AgentRadar exact count_tokens local refusals (image policy, replay, overflow, cancel/timeout/transport) before or around its one count send"], ["src/server/index/serve-options.ts", 4, "fixed endpoint auth/origin bypasses"], ["src/server/messages-native-tool-guard.ts", 1, "deferred local guard terminal"], + ["src/server/responses/core-combo-native.ts", 1, "AgentRadar hosted exact-replay refusal before any combo send"], ] as const; test("W1/W3 envelope writers are covered by explicit dispositions", async () => { const actual: Record = {}; @@ -340,10 +342,22 @@ const nativeFailRows = [ "call": "if (sendError instanceof Error && isProxyError(sendError)) return fail(502, sendError.message, \"api_error\");", "disposition": "bounded typed local exception or producer-marked dispatch diagnostic" }, + { + "call": "if (req.signal.aborted) return fail(499, \"Client cancelled request\", \"api_error\");", + "disposition": "fixed local literal at the mapped native producer" + }, + { + "call": "if (read.oversized) return fail(502, \"upstream response exceeded the safe limit\", \"api_error\", \"translation_buffer_limit\");", + "disposition": "hosted .27 relay contract: fixed local literal before any client output" + }, { "call": "return fail(499, \"Client cancelled request\", \"api_error\");", "disposition": "fixed local literal at the mapped native producer" }, + { + "call": "return fail(502, \"canonical Anthropic response was not valid JSON\", \"api_error\");", + "disposition": "hosted .27 relay contract: fixed local literal before any client output" + }, { "call": "if (req.signal.aborted) return fail(499, \"Client cancelled request\", \"api_error\");", "disposition": "fixed local literal at the mapped native producer" @@ -376,17 +390,37 @@ const nativeFailRows = [ "call": "return fail(413, \"upstream translation buffer exceeded the safe limit\", \"request_too_large\", \"translation_buffer_limit\");", "disposition": "fixed local literal at the mapped native producer" }, + { + "call": "return fail(502, \"canonical Anthropic returned JSON for a streaming request\", \"api_error\");", + "disposition": "hosted .27 relay contract: fixed local literal before any client output" + }, { "call": "if (req.signal.aborted) return fail(499, \"Client cancelled request\", \"api_error\");", "disposition": "fixed local literal at the mapped native producer" }, + { + "call": "if (hosted) return fail(502, invalidJson, \"api_error\");", + "disposition": "hosted .27 relay contract: fixed local literal before any client output" + }, + { + "call": "if (hosted) return fail(502, \"canonical Anthropic response exceeded the configured body limit\", \"api_error\");", + "disposition": "hosted .27 relay contract: fixed local literal before any client output" + }, { "call": "return fail(502, \"upstream response exceeded the safe limit\", \"api_error\", \"translation_buffer_limit\");", "disposition": "fixed local literal at the mapped native producer" }, { - "call": "return fail(502, \"upstream returned malformed Messages JSON\", \"api_error\");", - "disposition": "fixed local literal at the mapped native producer" + "call": "if (hosted && read.truncated) return fail(502, invalidJson, \"api_error\");", + "disposition": "hosted .27 relay contract: fixed local literal before any client output" + }, + { + "call": "return fail(502, hosted ? invalidJson : \"upstream returned malformed Messages JSON\", \"api_error\");", + "disposition": "hosted .27 relay contract: fixed local literal before any client output" + }, + { + "call": "return fail(502, \"canonical Anthropic returned an invalid JSON response\", \"api_error\");", + "disposition": "hosted .27 relay contract: fixed local literal before any client output" }, { "call": "return fail(502, \"upstream response was not a Messages result\", \"api_error\");", diff --git a/tests/cli/cli-provider.test.ts b/tests/cli/cli-provider.test.ts index a7bbe3338b7..adf0880e4c6 100644 --- a/tests/cli/cli-provider.test.ts +++ b/tests/cli/cli-provider.test.ts @@ -268,7 +268,8 @@ describe("ocx provider", () => { // `--headless` is the OAuth code-display mode. Codex has none; its headless path is the // device-code login, so the CLI points there instead of forwarding the flag. - test.each([["--headless", "codex"], ["codex", "--headless"], ["--headless", "openai"]])( + // `--code` is the flag's earlier name and must stay its alias rather than be silently ignored. + test.each([["--headless", "codex"], ["codex", "--headless"], ["--headless", "openai"], ["codex", "--code"], ["--code", "openai"]])( "login %s %s refuses code mode for Codex and points at --device", (a, b) => { const { dir } = freshConfig({}); diff --git a/tests/cli/route-explainability.test.ts b/tests/cli/route-explainability.test.ts index 203bfeb0564..2953c2b8eb6 100644 --- a/tests/cli/route-explainability.test.ts +++ b/tests/cli/route-explainability.test.ts @@ -186,6 +186,44 @@ describe("route explainability (RI-09)", () => { expect(body.summary).toMatchObject({ finalProvider: "b", finalModel: "m2" }); }); + test("final target ignores trailing local, unsent, and duplicate attempts", async () => { + appendUsageEntry({ + requestId: "physical-final", + timestamp: 1_700_000_000_001, + provider: "root", + model: "root-model", + status: 200, + durationMs: 900, + usageStatus: "reported", + attempts: [ + { ordinal: 1, provider: "a", model: "m1", adapter: "openai-chat", status: 503, durationMs: 100, sendCount: 1, recoveryKinds: [], usageStatus: "unreported" }, + { ordinal: 2, provider: "b", model: "m2", adapter: "openai-chat", status: 200, durationMs: 700, sendCount: 1, recoveryKinds: [], usageStatus: "reported" }, + { ordinal: 3, provider: "local", model: "local-model", adapter: "openai-chat", status: 200, durationMs: 1, sendCount: 1, locallyAnswered: true, recoveryKinds: [], usageStatus: "reported" }, + { ordinal: 4, provider: "unsent", model: "unsent-model", adapter: "openai-chat", status: 503, durationMs: 1, sendCount: 0, recoveryKinds: [], usageStatus: "unreported" }, + { ordinal: 2, provider: "duplicate", model: "duplicate-model", adapter: "openai-chat", status: 200, durationMs: 1, sendCount: 1, recoveryKinds: [], usageStatus: "reported" }, + ], + }); + appendUsageEntry({ + requestId: "no-physical-target", + timestamp: 1_700_000_000_002, + provider: "root", + model: "root-model", + status: 200, + durationMs: 1, + usageStatus: "reported", + attempts: [ + { ordinal: 1, provider: "local", model: "local-model", adapter: "openai-chat", status: 200, durationMs: 1, sendCount: 1, locallyAnswered: true, recoveryKinds: [], usageStatus: "reported" }, + ], + }); + + const physical = await apiGet("/api/request-history/physical-final/route-decision", config()); + const physicalBody = await physical.json() as { summary?: { finalProvider?: string; finalModel?: string } }; + expect(physicalBody.summary).toMatchObject({ finalProvider: "b", finalModel: "m2" }); + const root = await apiGet("/api/request-history/no-physical-target/route-decision", config()); + const rootBody = await root.json() as { summary?: { finalProvider?: string; finalModel?: string } }; + expect(rootBody.summary).toMatchObject({ finalProvider: "root", finalModel: "root-model" }); + }); + test("dry-run without candidate evidence assembles canonical evidence", async () => { const cfg = config(); const req = new ManagementRequest("http://localhost/api/routing-profiles/dry-run", { diff --git a/tests/codex-integration/codex-reset-credit-auto-redeem.test.ts b/tests/codex-integration/codex-reset-credit-auto-redeem.test.ts index b69dbd97f63..db1f83af857 100644 --- a/tests/codex-integration/codex-reset-credit-auto-redeem.test.ts +++ b/tests/codex-integration/codex-reset-credit-auto-redeem.test.ts @@ -545,6 +545,24 @@ describe("reset-credit auto-redeemer runtime (#822)", () => { expect(entries[0].updatedAt).toBe(start + 20 * MIN); }); + test("a settled entry is kept until its credit expires, even past the one-week age", async () => { + const DAY = 24 * 60 * MIN; + const journalFile = join(dir, "j.json"); + const seeded = (grantedAt: string, expiresAt: string) => ({ + accountKey: "seeded", grantedAt, expiresAt, redeemRequestId: `id-${grantedAt}`, state: "settled", updatedAt: T0 - 8 * DAY, + }); + writeFileSync(journalFile, JSON.stringify({ version: 1, entries: [ + seeded("2026-08-01T00:00:00Z", new Date(T0 + 30 * DAY).toISOString()), // old update, credit not expired yet + seeded("2026-08-02T00:00:00Z", new Date(T0 - 1 * DAY).toISOString()), // old update, credit expired + seeded("2026-08-03T00:00:00Z", "not-a-date"), // malformed expiry falls back to the week rule + ] })); + const h = harness({ journalFile, credits: () => [credit(30)] }); + h.setNow(T0 + 20 * MIN); + expect((await h.redeemer.tick()).kind).toBe("dispatched"); + const entries = JSON.parse(readFileSync(journalFile, "utf8")).entries as Array<{ grantedAt: string }>; + expect(entries.map(e => e.grantedAt).sort()).toEqual(["2026-08-01T00:00:00Z", "2026-09-01T00:00:00Z"]); + }); + test("a persistent reservation write failure uses the idle retry interval", async () => { const journalFile = join(dir, "journal-directory"); mkdirSync(journalFile); diff --git a/tests/codex-integration/combo-pool-cooldown-blast-radius.test.ts b/tests/codex-integration/combo-pool-cooldown-blast-radius.test.ts new file mode 100644 index 00000000000..4654873ce52 --- /dev/null +++ b/tests/codex-integration/combo-pool-cooldown-blast-radius.test.ts @@ -0,0 +1,53 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { clearComboTargetCooldowns, coolComboTarget, isComboTargetInCooldown } from "../../src/combos/failover"; + +/** + * Regression for the 2026-09-28 pool-cooldown outage. + * + * A combo target key is `comboId` + provider/model; every account in that provider's pool shares + * it. One account answered `Retry-After: 318747` (88.5h, weekly limit spent) and the gateway wrote + * that straight onto the shared key, clamped only by the 24h server-delay ceiling. Four healthy + * accounts became unreachable and the gateway returned 503 with zero upstream sends for four hours. + * + * A `Retry-After` is authoritative about the account that answered it, not about the target. Until + * the key carries the account, the ceiling is what bounds the blast radius. + */ +const combo = "pool"; +const target = { provider: "anthropic", model: "claude-opus-5-5" }; +const TEN_MIN = 10 * 60_000; + +// AgentRadar hosted gateways only; ordinary installs keep upstream's one-day server-delay ceiling. +const previousHosted = process.env.OPENCODEX_AGENTRADAR; +beforeEach(() => { process.env.OPENCODEX_AGENTRADAR = "1"; }); +afterEach(() => { + clearComboTargetCooldowns(); + if (previousHosted === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previousHosted; +}); + +describe("a pooled target's cooldown is bounded", () => { + test("an 88-hour Retry-After cannot hold the target for more than ten minutes", () => { + const now = 1_000; + coolComboTarget(combo, target, { now, status: 429, message: "rate limited", retryAfter: "318747" }); + expect(isComboTargetInCooldown(combo, target, now + TEN_MIN - 1)).toBe(true); + expect(isComboTargetInCooldown(combo, target, now + TEN_MIN)).toBe(false); + }); + + test("a 24-hour Retry-After is bounded the same way", () => { + const now = 1_000; + coolComboTarget(combo, target, { now, status: 429, message: "rate limited", retryAfter: "86400" }); + expect(isComboTargetInCooldown(combo, target, now + TEN_MIN)).toBe(false); + }); + + test("a short server delay is still honoured exactly, not rounded up", () => { + const now = 1_000; + coolComboTarget(combo, target, { now, status: 429, message: "slow down", retryAfter: "30" }); + expect(isComboTargetInCooldown(combo, target, now + 30_000 - 1)).toBe(true); + expect(isComboTargetInCooldown(combo, target, now + 30_000)).toBe(false); + }); + + test("an immediate Retry-After: 0 stays immediate", () => { + const now = 1_000; + coolComboTarget(combo, target, { now, status: 429, message: "retry now", retryAfter: "0" }); + expect(isComboTargetInCooldown(combo, target, now + 1)).toBe(false); + }); +}); diff --git a/tests/fixtures/anthropic-extra-usage-thirdparty-400.txt b/tests/fixtures/anthropic-extra-usage-thirdparty-400.txt new file mode 100644 index 00000000000..2b121a6ba90 --- /dev/null +++ b/tests/fixtures/anthropic-extra-usage-thirdparty-400.txt @@ -0,0 +1 @@ +Provider error 400: {"type":"error","error":{"type":"invalid_request_error","message":"Third-party apps now draw from your extra usage, not your plan limits. Add more at claude.ai/settings/usage and k \ No newline at end of file diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 991fe84ca26..0ed80f2d2bb 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1,4 +1,35 @@ { "responses-combo-unavailable-cause.test.ts": "responses", "responses-native-sse-stall.test.ts": "responses", "client-restart-sync-fence.test.ts": "clients", "service-scheduler-restart-respawn-refusal.test.ts": "service", "route-to-home-sync-review.test.ts": "routing", "route-intent-connect-post-commit.test.ts": "clients", "route-intent-file-originals.test.ts": "clients", "route-to-child-defaults.test.ts": "routing", "route-to-home-live-flags.test.ts": "routing", "route-to-home-native-truth.test.ts": "routing", "route-to-home-decisions.test.ts": "routing", "route-to-ephemeral.test.ts": "server", "link-pinned-sync-overwrite.test.ts": "clients", "cli-sync-fanout-ephemeral.test.ts": "cli", "route-intent-home-consent.test.ts": "clients", "management-route-intent-native-errors.test.ts": "server", "route-intent-offline-mode-switch.test.ts": "clients", "route-intent-parity-withdrawal.test.ts": "clients", "service-scheduler-restart-cleanup.test.ts": "service", "route-to-retirement-review.test.ts": "server", "route-to-policy-review.test.ts": "server", "management-route-intent-native-review.test.ts": "server", "route-to-home-off-review.test.ts": "routing", "codex-journal-provider-reserialization.test.ts": "codex-integration", "route-intent-parity-fallback.test.ts": "clients", "route-intent-native-originals.test.ts": "clients", "route-intent-native-review.test.ts": "clients", "route-intent-connected-capabilities.test.ts": "clients", "route-intent-connected-review.test.ts": "clients", "route-intent-connect-recovery.test.ts": "clients", "client-disconnect-field-snapshot.test.ts": "clients", "cli-route-to-confirmation.test.ts": "cli", "route-intent-runtime-stop.test.ts": "clients", "route-intent-codex-connect-disabled.test.ts": "clients", "route-intent-codex-transitions.test.ts": "clients", "route-to-live-settings.test.ts": "server", "route-intent-claude-surface.test.ts": "clients", "route-intent-local-receipt.test.ts": "clients", "route-intent-scheduler.test.ts": "clients", "route-intent-watch-client.test.ts": "clients", "route-intent-watch-boundary.test.ts": "server", "route-intent-watch-waiters.test.ts": "server", "home-route-intent-upgrade.test.ts": "clients", "route-intent-overwrite.test.ts": "clients", "codex-disconnect-owned-recovery.test.ts": "codex-integration", "service-restart-registration-guard.test.ts": "service", "client-disconnect-config-recovery.test.ts": "clients", "route-intent-aside.test.ts": "clients", "route-intent-connected.test.ts": "clients", "cli-link-route-intent.test.ts": "cli", "cli-route-to-aliases.test.ts": "cli", "cli-route-to.test.ts": "cli", "client-disconnect-edited-codex.test.ts": "clients", "codex-disconnect-fields.test.ts": "codex-integration", "link-home-route-intent.test.ts": "clients", "link-route-intent.test.ts": "server", "route-intent-desktop.test.ts": "clients", "route-intent-materializer.test.ts": "clients", "route-intent-parity.test.ts": "clients", "route-intent-reporter.test.ts": "clients", "route-intent-state.test.ts": "clients", "route-intent.test.ts": "clients", "route-to-routes.test.ts": "server", "route-to-sync-child.test.ts": "routing", "service-disconnected-client-restart.test.ts": "service", "responses-forward-client-headers-live.test.ts": "responses", "pool-retry-client-metadata.test.ts": "responses", "responses-websocket-workflow-ceiling.test.ts": "responses", "responses-workflow-send-ceiling.test.ts": "responses", "kimi-monthly-quota.test.ts": "providers", "sync-fanout.test.ts": "cli", "sync-command-fanout.test.ts": "cli", "link-pinned-sync-argv.test.ts": "clients", "sync-link-status.test.ts": "cli", "sync-fanout-process.test.ts": "cli", "codex-installation-acl.test.ts": "codex-integration", "codex-installation-id.test.ts": "codex-integration", "codex-installation-wire.test.ts": "codex-integration", "codex-installation-failover.test.ts": "responses", "lmstudio-system-policy.test.ts": "config", "openai-chat-single-system.test.ts": "adapters/openai", "update-reclaim-authorizer.test.ts": "update", "update-capability-policy.test.ts": "update", "update-service-home.test.ts": "update", "update-service-home-policy.test.ts": "update", "update-home-recovery.test.ts": "update", "update-recovery-authority.test.ts": "update", "update-recovery-boundaries.test.ts": "update", "update-home-launcher.test.ts": "update", "update-home-worker.test.ts": "update", "update-mutation-boundaries.test.ts": "update", "update-mutation-guard.test.ts": "update", "update-guarded-ops.test.ts": "update", "update-effect-authority.test.ts": "update", "update-windows-wrapper-authority.test.ts": "update", "update-mutation-scanner.test.ts": "update", "refreshed-upstream-deadlines.test.ts": "responses", "local-upstream-deadlines.test.ts": "responses", "service-desktop-recovery.test.ts": "service", "provider-input-modalities.test.ts": "providers", "turn-membership.test.ts": "responses", "turn-membership-ingress.test.ts": "responses", "workbuddy-quota.test.ts": "providers", "workbuddy-login.test.ts": "oauth", "workbuddy-adapter.test.ts": "adapters", "workbuddy-provider.test.ts": "providers", "link-tunnel-history.test.ts": "clients", "link-readiness.test.ts": "clients", "link-tunnel-status.test.ts": "clients", "link-diagnostics.test.ts": "clients", "combo-description.test.ts": "routing", "combo-description-catalog.test.ts": "codex-integration", "catalog-auto-review-override.test.ts": "codex-integration", "management-auto-review-settings.test.ts": "server", "management-auto-review-auth.test.ts": "server", "prefer-model-alias-ids.test.ts": "server", "responses-mimo-envelope.test.ts": "responses", "responses-call-id-linkage.test.ts": "responses", "responses-phase-inference.test.ts": "responses", "cli-message-mcp-resume.test.ts": "cli", "messaging-mcp-resume.test.ts": "codex-integration", "link-diagnostics-read.test.ts": "clients", "cli-link-diagnostics.test.ts": "cli", "link-diagnostics-route.test.ts": "server", "provider-passive-wire-model.test.ts": "providers", "passive-key-observation.test.ts": "providers", "responses-passive-key-limits.test.ts": "responses", "passive-key-limits.test.ts": "providers", "terminal-safe-text.test.ts": "lib", + "combo-pool-cooldown-blast-radius.test.ts": "codex-integration", + "agentradar-count-tokens.test.ts": "claude-integration", + "usage-anthropic-account-label.test.ts": "usage", + "server-combo-failover-advance.test.ts": "server", + "combo-pool-stale-cooldown.test.ts": "routing", + "quota-explain.test.ts": "providers", + "agentradar-account-pin-native-e2e.test.ts": "server", + "agentradar-hosted-fidelity.test.ts": "claude-integration", + "agentradar-hosted-tap.test.ts": "claude-integration", + "image-bridge-runturn-pacing.test.ts": "server", + "anthropic-agentradar-429.test.ts": "adapters/anthropic", + "server-combo-local-pacing-cooldown.test.ts": "server", + "web-search-overflow-usage.test.ts": "web-search", + "call-usage-line-log.test.ts": "lib", + "upstream-refusal-log.test.ts": "lib", + "routing-analytics-physical.test.ts": "routing", + "usage-log-physical.test.ts": "usage", + "usage-summary-physical.test.ts": "usage", + "request-log-physical.test.ts": "usage", + "management-api-logs-metrics-physical.test.ts": "server", + "responses-state-env-bounds.test.ts": "responses", + "router-combo-400-quota-echo.test.ts": "routing", + "responses-compact-400-quota-echo.test.ts": "responses", + "provider-registry-xai-apikey.test.ts": "providers", + "anthropic-agentradar-pool-parity.test.ts": "adapters/anthropic", + "anthropic-family-claim-429.test.ts": "adapters/anthropic", + "anthropic-pool-extra-usage-400.test.ts": "adapters/anthropic", + "anthropic-weekly-spent-gate.test.ts": "adapters/anthropic", + "agentradar-account-pin-parity-e2e.test.ts": "server", + "terminal-guard-agentradar-rotation.test.ts": "server", + "agentradar-account-pin-e2e.test.ts": "server", "responses-estimated-usage.test.ts": "responses", "codex-installation-live-content.test.ts": "codex-integration", "codex-installation-invalid-account.test.ts": "codex-integration", diff --git a/tests/helpers/responses-core-source.ts b/tests/helpers/responses-core-source.ts index 98b253558b6..667150baaf2 100644 --- a/tests/helpers/responses-core-source.ts +++ b/tests/helpers/responses-core-source.ts @@ -32,6 +32,8 @@ export const RESPONSES_CORE_MODULES = [ "core-errors.ts", "core-opaque-recovery.ts", "core-codex-account.ts", + // Why a 401 skipped the native-main refresh (a log line), reached from passthrough-dispatch.ts. + "codex-main-401-log.ts", "core-combo-failure.ts", "core-auth.ts", "core-normalize.ts", diff --git a/tests/images/loop.test.ts b/tests/images/loop.test.ts index 846bfad1637..7ec83c2cd4d 100644 --- a/tests/images/loop.test.ts +++ b/tests/images/loop.test.ts @@ -3,14 +3,21 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { randomUUID } from "node:crypto"; import type { ProviderAdapter, IncomingMeta } from "../../src/adapters/base"; -import type { AdapterEvent, OcxParsedRequest } from "../../src/types"; +import type { AdapterEvent, OcxParsedRequest, OcxProviderConfig } from "../../src/types"; import type { AttemptRecoveryKind } from "../../src/usage/log"; import type { ImageBridgePlan, ImageCallResult } from "../../src/images/types"; import type { ImageBridgeDeps } from "../../src/images/loop"; import { createTestTranslatorBudget } from "../helpers/translator-budget"; import { PROVIDER_INPUT_TOO_LARGE_MESSAGE } from "../../src/server/responses/context-overflow"; import { markResponseNonReplayable } from "../../src/lib/upstream-retry"; -import { RequestPacingQueueOverloadError } from "../../src/providers/request-pacing"; +import { + providerRequestPacingStatus, + RequestPacingQueueOverloadError, + resetProviderRequestPacingForTest, + waitForProviderRequestSlot, + type ProviderRequestSlot, +} from "../../src/providers/request-pacing"; +import { providerFetch } from "../../src/server/responses/fetch-helpers"; import { bridgeToResponsesSSE as realBridgeToResponsesSSE } from "../../src/bridge"; import { parseStreamWithProgress, WebSearchConsumerInactivityError, type ParseStreamWithProgressOptions } from "../../src/web-search/progress-stream"; import { TRANSLATOR_MAX_CALL_ARGUMENT_BYTES, TRANSLATOR_MAX_TURN_BYTES, translatorLiveBudgetCountForTests } from "../../src/lib/translator-budget"; @@ -455,6 +462,39 @@ describe("runWithImageBridge", () => { expect(sse).not.toContain("event: response.completed"); }); + test("usage-bearing translator overflow reaches request and wire without image dispatch", async () => { + // The real collector folds the overflow event's usage (onErrorUsage) before it throws. + useRealProgressStream = true; + const aggregate: unknown[] = []; + streamQueue = [[ + { type: "tool_call_start", id: "call_1", name: "image_gen" }, + { type: "tool_call_delta", arguments: '{"prompt":"a cat"}' }, + { type: "tool_call_end" }, + { + type: "error", + status: 502, + errorType: "upstream_error", + code: "translation_buffer_limit", + message: "upstream translation buffer exceeded the safe limit", + usage: { inputTokens: 7, outputTokens: 1 }, + }, + ]]; + const response = await runWithImageBridge({ + parsed: makeParsed(), + adapter: mockAdapter, + plan, + onUsage: usage => aggregate.push(usage), + }); + const sse = await response.text(); + + expect(sse).toContain("event: response.failed"); + expect(sse).toContain('"code":"translation_buffer_limit"'); + expect(sse).toContain('"usage":{"input_tokens":7,"output_tokens":1,"total_tokens":8'); + expect(sse).not.toContain("event: response.completed"); + expect(aggregate).toEqual([{ inputTokens: 7, outputTokens: 1 }]); + expect(fulfillCallCount).toBe(0); + }); + test("no image tool call → passthrough text + done", async () => { const sse = await runAndGetSSE([ [{ type: "text_delta", text: "hello world" }, { type: "done" }], @@ -1221,6 +1261,54 @@ describe("runWithImageBridge — runTurn adapter", () => { expect(sse).toContain("hello from runTurn"); }); + test("runTurn adapter paces every physical dispatch inside its turn lease", async () => { + // A capped provider: the turn's own lease must not block its BidiAppend sends. + let transportSends = 0; + let recordedSends = 0; + let turnSlot: ProviderRequestSlot | undefined; + const configured: OcxProviderConfig = { + adapter: "cursor", baseUrl: "https://cursor.test", requestPacing: { enabled: true, maxConcurrentRequests: 1 }, + fetch: Object.assign(async () => { + transportSends += 1; + return new Response("ok"); + }, { preconnect() {} }) as typeof globalThis.fetch, + }; + const dispatchingAdapter: ProviderAdapter = { + ...mockAdapter, + runTurn: async (_parsed, incoming, emit) => { + if (!incoming.providerFetch) throw new Error("missing provider fetch"); + for (const url of ["https://cursor.test/run-sse", "https://cursor.test/bidi-append", "https://cursor.test/bidi-append"]) { + await (await incoming.providerFetch(url, { signal: AbortSignal.timeout(1_000) })).text(); + } + expect(providerRequestPacingStatus("cursor", configured).inFlight).toBe(1); + emit({ type: "text_delta", text: "sent" }); + emit({ type: "done" }); + }, + }; + try { + const response = await runWithImageBridge({ + parsed: makeParsed(), + adapter: dispatchingAdapter, + plan, + waitForRequestSlot: async signal => (turnSlot = await waitForProviderRequestSlot("cursor", configured, "model-a", signal)), + runTurnFetch: pacingSlot => providerFetch(configured, undefined, { + providerName: "cursor", modelId: "model-a", pacingSlotAcquired: true, pacingSlot, turnScopedPacing: true, + }), + onAttemptSend: () => { recordedSends += 1; }, + }); + + expect(await response.text()).toContain("sent"); + expect(transportSends).toBe(3); + // One attempt per runTurn, as on the normal runTurn lane; the lease ends with the turn. + expect(recordedSends).toBe(1); + expect(turnSlot?.released).toBe(true); + expect(providerRequestPacingStatus("cursor", configured).inFlight).toBe(0); + } finally { + turnSlot?.release(); + resetProviderRequestPacingForTest(); + } + }); + test("runTurn adapter → error event surfaces as upstream failure", async () => { runTurnEventQueue = [ [{ type: "error", message: "cursor blew up" }], diff --git a/tests/lab/lab-passive-production-evidence.test.ts b/tests/lab/lab-passive-production-evidence.test.ts index 471aba9505b..fdb084e7116 100644 --- a/tests/lab/lab-passive-production-evidence.test.ts +++ b/tests/lab/lab-passive-production-evidence.test.ts @@ -256,6 +256,48 @@ describe("CL-09 bounded passive production projection", () => { expect(derivePassiveProductionSignals([entry], subjectId).signals).toHaveLength(1); expect(derivePassiveProductionSignals([], subjectId).signals).toHaveLength(0); }); + + test("projects only physical attempts and keeps the first physical duplicate ordinal", () => { + const subjectId = "8".repeat(64); + const entry = usageEntryWithAttempt({ labRouteSubjectId: subjectId }); + const attempt = entry.attempts![0]!; + entry.attempts = [ + { ...attempt, sendCount: 0, locallyAnswered: true }, + { ...attempt, status: 502, errorCode: "upstream_error" }, + { ...attempt, status: 200 }, + { ...attempt, ordinal: 2, status: 503, sendCount: 0 }, + { ...attempt, ordinal: 3, status: 200 }, + ]; + + const result = derivePassiveProductionSignals([entry], subjectId); + + expect(result.signals.map(signal => [signal.attemptOrdinal, signal.outcome])).toEqual([ + [3, "success"], + [1, "route_error"], + ]); + expect(result.summary.recentProductionAttempts).toBe(2); + }); + test("classifies cancellation against the final physical attempt", () => { + const subjectId = "9".repeat(64); + const entry = usageEntryWithAttempt({ + labRouteSubjectId: subjectId, + status: 499, + errorCode: "upstream_error", + }); + entry.status = 499; + entry.closeReason = "client_cancel"; + entry.attempts!.push({ + ...entry.attempts![0]!, + ordinal: 2, + sendCount: 0, + locallyAnswered: true, + }); + + const result = derivePassiveProductionSignals([entry], subjectId); + + expect(result.signals).toHaveLength(1); + expect(result.signals[0]?.outcome).toBe("client_cancel"); + }); }); describe("CL-09 no-feedback architecture guards", () => { diff --git a/tests/lib/call-usage-line-log.test.ts b/tests/lib/call-usage-line-log.test.ts new file mode 100644 index 00000000000..d3ebc119281 --- /dev/null +++ b/tests/lib/call-usage-line-log.test.ts @@ -0,0 +1,159 @@ +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; +import { createHash } from "node:crypto"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { hashedKeyId, logUsageLine, usageLogLine, USAGE_LOG_ID_MAX_CHARS } from "../../src/lib/usage-line-log"; +import { addFinalRequestLog, clearRequestLogsForTests, type RequestLogContext } from "../../src/server/request-log"; +import { resetUsageReadCacheForTests } from "../../src/usage/log"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +// Shaped as test sentinels so the privacy scan knows they are fake. +const PROMPT_TEXT = "DISTINCTIVE-PROMPT-TEXT-7731"; +const API_KEY_SECRET = "sk-test-9942distinctive"; +const AUTH_HEADER = "Bearer access-token-5531distinctive"; +const KEY_ID = "key-record-1"; + +const LINE_RE = /^\[usage\] status=\d+ provider="[^"]*" model="[^"]*" in=(?:\d+|-) out=(?:\d+|-) cached=(?:\d+|-) key=(?:[0-9a-f]{8}|-) ms=(?:\d+|-)$/; + +let infos: string[]; +let restoreEnv: () => void; +let restoreSpy: () => void; + +beforeEach(() => { + infos = []; + const spy = spyOn(console, "info").mockImplementation((...args: unknown[]) => { infos.push(args.map(String).join(" ")); }); + restoreSpy = () => spy.mockRestore(); + const previous = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + restoreEnv = () => { + if (previous === undefined) delete process.env.OPENCODEX_AGENTRADAR; + else process.env.OPENCODEX_AGENTRADAR = previous; + }; +}); +afterEach(() => { restoreSpy(); restoreEnv(); }); + +const usageLines = () => infos.filter(line => line.startsWith("[usage]")); + +describe("usage line", () => { + test("carries status, ids, counts, hashed key id and duration", () => { + logUsageLine({ + status: 200, provider: "anthropic", model: "claude-test-1", + usage: { inputTokens: 1200, outputTokens: 340, cacheReadInputTokens: 900 }, + apiKeyId: KEY_ID, durationMs: 2150, + }); + const expectedHash = createHash("sha256").update(KEY_ID).digest("hex").slice(0, 8); + expect(usageLines()).toEqual([ + `[usage] status=200 provider="anthropic" model="claude-test-1" in=1200 out=340 cached=900 key=${expectedHash} ms=2150`, + ]); + expect(LINE_RE.test(usageLines()[0]!)).toBe(true); + }); + + test("cached falls back to cachedInputTokens and the key hash is stable per id", () => { + expect(usageLogLine({ status: 200, usage: { inputTokens: 5, outputTokens: 1, cachedInputTokens: 4 }, apiKeyId: KEY_ID })).toContain("cached=4"); + expect(hashedKeyId(KEY_ID)).toBe(hashedKeyId(KEY_ID)); + expect(hashedKeyId(KEY_ID)).not.toBe(hashedKeyId("key-record-2")); + }); + + test("missing counts, key and duration log as a dash", () => { + logUsageLine({ status: 502 }); + expect(usageLines()).toEqual(['[usage] status=502 provider="-" model="-" in=- out=- cached=- key=- ms=-']); + expect(hashedKeyId(undefined)).toBe("-"); + expect(hashedKeyId("")).toBe("-"); + const odd = usageLogLine({ status: 200, usage: { inputTokens: Number.NaN, outputTokens: -3 } }); + expect(odd).toContain("in=- out=- cached=-"); + }); + + test("ids with newlines or 200 characters stay on one bounded line", () => { + logUsageLine({ + status: 200, provider: "prov\r\nider\n[usage] status=200 forged", model: `m${"x".repeat(200)}\nsecond line`, + usage: { inputTokens: 1, outputTokens: 1 }, + }); + const [line] = usageLines(); + expect(line).toBeDefined(); + expect(line).not.toMatch(/[\r\n]/); + expect(infos).toHaveLength(1); + const model = /model="([^"]*)"/.exec(line!)![1]!; + expect(model.length).toBeLessThanOrEqual(USAGE_LOG_ID_MAX_CHARS); + expect(LINE_RE.test(line!)).toBe(true); + }); + + test("a pooled provider label drops the account suffix", () => { + expect(usageLogLine({ status: 200, provider: "anthropic-p2e22d0" })).toContain('provider="anthropic"'); + expect(usageLogLine({ status: 200, provider: "xai-oabcdef" })).toContain('provider="xai"'); + expect(usageLogLine({ status: 200, provider: `acme-k${"a".repeat(32)}` })).toContain('provider="acme"'); + expect(usageLogLine({ status: 200, provider: "opencode-go" })).toContain('provider="opencode-go"'); + }); + + test("a key-shaped token in a provider label is redacted, not logged", () => { + logUsageLine({ status: 200, provider: `relay-${API_KEY_SECRET}`, model: "gpt-test", apiKeyId: KEY_ID }); + const line = usageLines()[0]!; + expect(line).not.toContain(API_KEY_SECRET); + expect(LINE_RE.test(line)).toBe(true); + }); + + test("is silent unless the AgentRadar mode is on", () => { + delete process.env.OPENCODEX_AGENTRADAR; + logUsageLine({ status: 200, provider: "anthropic", model: "claude-test-1" }); + process.env.OPENCODEX_AGENTRADAR = "0"; + logUsageLine({ status: 200, provider: "anthropic", model: "claude-test-1" }); + expect(infos).toEqual([]); + }); +}); + +describe("usage line from the final request record", () => { + test("every finished call emits one line, and no text, secret or header reaches it", () => { + const home = mkdtempSync(join(tmpdir(), "ocx-usage-line-")); + const previousHome = process.env.OPENCODEX_HOME; + process.env.OPENCODEX_HOME = home; + clearRequestLogsForTests(); + resetUsageReadCacheForTests(); + try { + const ctx: RequestLogContext = { + model: "claude-test-1", + provider: "anthropic-p2e22d0", + apiKeyId: KEY_ID, + admissionKind: "configured", + usage: { inputTokens: 800, outputTokens: 120, cacheReadInputTokens: 600 }, + // Fields that must never be read into the line. + conversationId: PROMPT_TEXT, + upstreamError: `${PROMPT_TEXT} ${API_KEY_SECRET} ${AUTH_HEADER}`, + accountLogLabel: "p2e22d0", + }; + addFinalRequestLog("ocx-usage-line-1", Date.now() - 50, ctx, 200, { closeReason: "non_stream" }); + const [line] = usageLines(); + expect(usageLines()).toHaveLength(1); + expect(LINE_RE.test(line!)).toBe(true); + expect(line).toContain('provider="anthropic" model="claude-test-1"'); + expect(line).toContain(`key=${hashedKeyId(KEY_ID)}`); + for (const secret of [PROMPT_TEXT, API_KEY_SECRET, AUTH_HEADER, "Bearer", "p2e22d0", KEY_ID]) { + expect(line).not.toContain(secret); + } + } finally { + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + clearRequestLogsForTests(); + resetUsageReadCacheForTests(); + removeTreeWithRetry(home); + } + }); + + test("a call with no usage and no configured key still logs one line with dashes", () => { + const home = mkdtempSync(join(tmpdir(), "ocx-usage-line-")); + const previousHome = process.env.OPENCODEX_HOME; + process.env.OPENCODEX_HOME = home; + clearRequestLogsForTests(); + resetUsageReadCacheForTests(); + try { + addFinalRequestLog("ocx-usage-line-2", Date.now(), { model: "gpt-test", provider: "openai", admissionKind: "loopback" }, 499, { closeReason: "client_cancel" }); + expect(usageLines()).toHaveLength(1); + expect(usageLines()[0]).toMatch(/^\[usage\] status=499 provider="openai" model="gpt-test" in=- out=- cached=- key=- ms=\d+$/); + } finally { + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + clearRequestLogsForTests(); + resetUsageReadCacheForTests(); + removeTreeWithRetry(home); + } + }); +}); diff --git a/tests/lib/upstream-refusal-log.test.ts b/tests/lib/upstream-refusal-log.test.ts new file mode 100644 index 00000000000..7f2691f467c --- /dev/null +++ b/tests/lib/upstream-refusal-log.test.ts @@ -0,0 +1,258 @@ +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; +import { createHash } from "node:crypto"; +import { + INBOUND_SIZE_LOG_THRESHOLD_BYTES, + logOversizedInbound, + logTranslationBufferLimit, + logUpstreamRefusal, + outgoingBodyBytes, + refusalEnvelopeFields, +} from "../../src/lib/upstream-refusal-log"; +import { providerFetch } from "../../src/server/responses/fetch-helpers"; +import type { OcxProviderConfig } from "../../src/types"; + +// Provider answers copied byte for byte, never tidied: +// - Anthropic "Claude API errors" page (platform.claude.com/docs/en/api/errors), "Prefill not +// supported" and "Error shapes" examples, read 2026-11-15 (the request id is the page's own). +// - OpenAI error-codes guide, "400 - Invalid service_tier argument", in OpenAI's standard envelope. +// - OpenCode Go 400 for `reasoning_effort: max`, recorded in devlog/_plan/260912_reasoning_metadata. +// - A staging gateway log line for an Anthropic 400, cut at 200 characters: a body that is not JSON. +const anthropicPrefill400 = `{"type":"error","error":{"type":"invalid_request_error","message":"This model does not support assistant message prefill. The conversation must end with a user message."}}`; +const anthropicNotFound404 = `{"type":"error","error":{"type":"not_found_error","message":"The requested resource could not be found."},"request_id":"req_011CSHoEeqs5C35K2UUqR7Fy"}`; +const openaiServiceTier400 = `{"error":{"message":"Invalid \`service_tier\` argument: The requested service tier is not allowed for this project.","type":"invalid_request_error","param":"service_tier","code":null}}`; +const opencodeGoEffort400 = `{"param":"reasoning.effort","type":"invalid_request_error","message":"Error from provider (Console Go): Upstream request failed: [invalid_request_error] reasoning_effort max requires an active Muse Code subscription for model muse-spark-1.3-contributor."}`; +const cutNonJson400 = `Provider error 400: {"type":"error","error":{"type":"invalid_request_error","message":"Third-party apps now draw from your extra usage, not your plan limits. Add more at claude.ai/settings/usage and k`; + +const PROMPT_TEXT = "DISTINCTIVE-PROMPT-TEXT-7731"; +// Shaped as a test sentinel so the privacy scan knows it is fake. +const API_KEY = "sk-test-9942distinctive"; + +let warnings: string[]; +let restore: () => void; + +beforeEach(() => { + warnings = []; + const spy = spyOn(console, "warn").mockImplementation((...args: unknown[]) => { warnings.push(args.map(String).join(" ")); }); + const previous = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + restore = () => { + spy.mockRestore(); + if (previous === undefined) delete process.env.OPENCODEX_AGENTRADAR; + else process.env.OPENCODEX_AGENTRADAR = previous; + }; +}); +afterEach(() => restore()); + +function refusalLines(): string[] { + return warnings.filter(line => line.startsWith("[upstream-refusal]")); +} + +async function waitForRefusalLine(): Promise { + for (let i = 0; i < 100 && refusalLines().length === 0; i++) await Bun.sleep(10); + expect(refusalLines()).toHaveLength(1); + return refusalLines()[0]!; +} + +/** One model call through the same wrapper every adapter and route uses, answered by `answer`. */ +async function callThroughProviderFetch(answer: () => Response, opts: { model?: string; provider?: string } = {}) { + const requestBody = JSON.stringify({ model: "ignored", messages: [{ role: "user", content: PROMPT_TEXT }] }); + const fetcher = providerFetch( + { fetch: async () => answer() } as unknown as OcxProviderConfig, + undefined, + { providerName: opts.provider ?? "test-provider", modelId: opts.model ?? "test-model" }, + ); + const response = await fetcher("https://upstream.example.test/v1/messages", { + method: "POST", + headers: { authorization: `Bearer ${API_KEY}`, "content-type": "application/json" }, + body: requestBody, + }); + return { response, requestBody }; +} + +const json = (body: string, status: number) => () => + new Response(body, { status, headers: { "content-type": "application/json" } }); + +describe("upstream refusal log line", () => { + test("a documented Anthropic 400 logs status, ids, type and size, and none of the message text", async () => { + const { response, requestBody } = await callThroughProviderFetch(json(anthropicPrefill400, 400)); + const line = await waitForRefusalLine(); + expect(line).toBe( + `[upstream-refusal] status=400 provider="test-provider" model="test-model" type="invalid_request_error"` + + ` bytes=${Buffer.byteLength(requestBody)}`, + ); + // Must-fail guard: the provider's message is free text that echoes client content. + expect(line).not.toContain("prefill"); + expect(line).not.toContain("conversation must end"); + expect(line).not.toContain(PROMPT_TEXT); + expect(line).not.toContain(API_KEY); + expect(line).not.toContain("authorization"); + // The caller still gets the whole answer. + expect(response.status).toBe(400); + expect(await response.text()).toBe(anthropicPrefill400); + }); + + test("a recorded 400 that names a parameter logs the parameter, not the message", async () => { + const { requestBody } = await callThroughProviderFetch(json(opencodeGoEffort400, 400)); + const line = await waitForRefusalLine(); + expect(line).toContain(" status=400 "); + expect(line).toContain(`type="invalid_request_error"`); + expect(line).toContain(`param="reasoning.effort"`); + expect(line).toContain(` bytes=${Buffer.byteLength(requestBody)}`); + // No `code` in this envelope, so none is logged. + expect(line).not.toContain("code="); + expect(line).not.toContain("Muse Code subscription"); + expect(line).not.toContain("Console Go"); + expect(line).not.toContain("muse-spark"); + }); + + test("a documented OpenAI 400 logs the parameter it names, not the message", async () => { + await callThroughProviderFetch(json(openaiServiceTier400, 400)); + const line = await waitForRefusalLine(); + expect(line).toContain(`type="invalid_request_error" param="service_tier"`); + // `code` is null in this envelope, so none is logged. + expect(line).not.toContain("code="); + expect(line).not.toContain("not allowed for this project"); + }); + + test("envelope fields with a newline, a 200-character value or a key stay on one bounded line", async () => { + const hostile = JSON.stringify({ + error: { type: "invalid_request_error\n[upstream-refusal] status=200", code: "c".repeat(200), param: API_KEY, message: PROMPT_TEXT }, + }); + await callThroughProviderFetch(json(hostile, 400)); + const line = await waitForRefusalLine(); + // One line: no newline survives, and a forged marker can only sit inside a quoted value. + expect(line).not.toContain("\n"); + expect(line.startsWith("[upstream-refusal] status=400 ")).toBe(true); + for (const field of ["type", "code", "param"]) { + const value = new RegExp(`${field}="([^"]*)"`).exec(line)?.[1]; + expect(value).toBeDefined(); + expect(value!.length).toBeLessThanOrEqual(80); + } + expect(line).not.toContain(API_KEY); + expect(line).not.toContain(PROMPT_TEXT); + }); + + test("an envelope field that is not a short identifier is logged only as a hash", async () => { + const echoed = JSON.stringify({ error: { type: "invalid_request_error", code: `${PROMPT_TEXT} please`, param: "req_011CSHoEeqs5C35K2UUqR7Fy" } }); + await callThroughProviderFetch(json(echoed, 400)); + const line = await waitForRefusalLine(); + const hash = (text: string) => `#${createHash("sha256").update(text).digest("hex").slice(0, 8)}`; + expect(line).toContain(`type="invalid_request_error" code="${hash(`${PROMPT_TEXT} please`)}" param="${hash("req_011CSHoEeqs5C35K2UUqR7Fy")}"`); + expect(line).not.toContain("DISTINCTIVE"); + expect(line).not.toContain("011CSHoE"); + expect(refusalEnvelopeFields(JSON.stringify({ error: { code: 400, param: "tools[3].function.name" } }))) + .toEqual({ code: "400", param: "tools[3].function.name" }); + }); + + test("a documented 404 logs its type and never the request id that the body carries", async () => { + await callThroughProviderFetch(json(anthropicNotFound404, 404)); + const line = await waitForRefusalLine(); + expect(line).toContain(" status=404 "); + expect(line).toContain(`type="not_found_error"`); + expect(line).not.toContain("req_011CSHoEeqs5C35K2UUqR7Fy"); + expect(line).not.toContain("could not be found"); + }); + + test("a body that is not JSON logs status, ids and size only", async () => { + const { requestBody } = await callThroughProviderFetch(json(cutNonJson400, 400)); + const line = await waitForRefusalLine(); + expect(line).toBe( + `[upstream-refusal] status=400 provider="test-provider" model="test-model" bytes=${Buffer.byteLength(requestBody)}`, + ); + expect(line).not.toContain("type="); + expect(line).not.toContain("extra usage"); + }); + + test("an empty refusal body still logs the line", async () => { + await callThroughProviderFetch(() => new Response(null, { status: 413 })); + const line = await waitForRefusalLine(); + expect(line).toContain(" status=413 "); + expect(line).not.toContain("type="); + }); + + test("success, rotation statuses and server errors write no refusal line", async () => { + await callThroughProviderFetch(json(anthropicPrefill400, 200)); + await callThroughProviderFetch(json(anthropicPrefill400, 401)); + await callThroughProviderFetch(json(anthropicPrefill400, 429)); + await callThroughProviderFetch(json(anthropicPrefill400, 503)); + await Bun.sleep(50); + expect(refusalLines()).toEqual([]); + }); + + test("provider and model ids are redacted, bounded to 80 characters and cannot break the line", async () => { + await callThroughProviderFetch(json(anthropicPrefill400, 400), { + provider: `provider\nforged-line`, + model: `${"m".repeat(200)}\n[upstream-refusal] status=200`, + }); + const line = await waitForRefusalLine(); + expect(line).not.toContain("\n"); + expect(line.match(/\[upstream-refusal\]/g)).toHaveLength(1); + const model = /model="([^"]*)"/.exec(line)?.[1] ?? ""; + expect(model.length).toBeLessThanOrEqual(80); + }); + + test("an unnamed provider and model log as dashes", async () => { + const fetcher = providerFetch({ fetch: async () => json(anthropicPrefill400, 400)() } as unknown as OcxProviderConfig); + await fetcher("https://upstream.example.test/v1/messages", { method: "POST", body: "{}" }); + const line = await waitForRefusalLine(); + expect(line).toContain(`provider="-" model="-"`); + }); +}); + +test("is silent unless the AgentRadar mode is on", async () => { + delete process.env.OPENCODEX_AGENTRADAR; + await callThroughProviderFetch(json(anthropicPrefill400, 400)); + logTranslationBufferLimit("request", { model: "m", limit: 1 }); + logOversizedInbound("/v1/messages", INBOUND_SIZE_LOG_THRESHOLD_BYTES + 1); + await Bun.sleep(50); + expect(warnings).toEqual([]); +}); + +describe("envelope parsing", () => { + test("reads only type, code and param from the error object", () => { + expect(refusalEnvelopeFields(anthropicPrefill400)).toEqual({ type: "invalid_request_error" }); + // A flat envelope: the fields sit at the top level, with no `error` object. + expect(refusalEnvelopeFields(opencodeGoEffort400)).toEqual({ type: "invalid_request_error", param: "reasoning.effort" }); + expect(refusalEnvelopeFields(cutNonJson400)).toEqual({}); + expect(refusalEnvelopeFields("")).toEqual({}); + expect(refusalEnvelopeFields("null")).toEqual({}); + expect(refusalEnvelopeFields(`[]`)).toEqual({}); + }); + + test("measures string, binary and blob bodies and leaves streams unmeasured", () => { + expect(outgoingBodyBytes("héllo")).toBe(6); + expect(outgoingBodyBytes(new Uint8Array(7))).toBe(7); + expect(outgoingBodyBytes(new ArrayBuffer(9))).toBe(9); + expect(outgoingBodyBytes(new Blob(["abcd"]))).toBe(4); + expect(outgoingBodyBytes(new ReadableStream())).toBeUndefined(); + expect(outgoingBodyBytes(undefined)).toBeUndefined(); + }); + + test("the line writer never prints an envelope field that was not given", () => { + logUpstreamRefusal({ status: 400, provider: "p", model: "m" }); + expect(refusalLines()).toEqual([`[upstream-refusal] status=400 provider="p" model="m" bytes=-`]); + }); +}); + +describe("the fork's own size refusals", () => { + test("a translation buffer refusal logs its size and cap, with no text", () => { + logTranslationBufferLimit("request", { model: "test-model", bytes: 40_000_000, limit: 33_554_432 }); + expect(refusalLines()).toEqual([ + `[upstream-refusal] status=413 provider="-" model="test-model" code="translation_buffer_limit"` + + ` where=request bytes=40000000 limit=33554432`, + ]); + }); + + test("a translation buffer refusal without a known size logs a dash", () => { + logTranslationBufferLimit("response", { model: "test-model", limit: 33_554_432 }); + expect(refusalLines()[0]).toContain(" where=response bytes=- limit=33554432"); + }); + + test("an inbound request over 5 MB is logged with its path and size, a smaller one is not", () => { + logOversizedInbound("/v1/messages", INBOUND_SIZE_LOG_THRESHOLD_BYTES); + logOversizedInbound("/v1/messages", Number.NaN); + expect(warnings).toEqual([]); + logOversizedInbound("/v1/messages", INBOUND_SIZE_LOG_THRESHOLD_BYTES + 1); + expect(warnings).toEqual([`[inbound-size] path="/v1/messages" bytes=${INBOUND_SIZE_LOG_THRESHOLD_BYTES + 1}`]); + }); +}); diff --git a/tests/oauth/oauth-account-attribution.test.ts b/tests/oauth/oauth-account-attribution.test.ts index 05fed3917b0..b602f43a4e3 100644 --- a/tests/oauth/oauth-account-attribution.test.ts +++ b/tests/oauth/oauth-account-attribution.test.ts @@ -8,6 +8,7 @@ import { clearGenericFailoverHealth } from "../../src/oauth/generic-account-fail import * as accountFailover from "../../src/oauth/generic-account-failover"; import * as oauth from "../../src/oauth"; import { stampOAuthAccountLabel } from "../../src/providers/label"; +import { formatAnthropicProviderForLog } from "../../src/oauth/anthropic-routing"; import { isCodexUsageAccountLogLabel, isCodexPoolAccountLogLabel } from "../../src/usage/log"; import type { PersistedUsageEntry } from "../../src/usage/log"; import { summarizeUsage } from "../../src/usage/summary"; @@ -143,6 +144,28 @@ describe("stampOAuthAccountLabel", () => { } }); + test("on an AgentRadar gateway, stamps anthropic with the same p-digest its account-qualified provider carries", () => { + // The hosted ledger needs the serving account on every Anthropic row, including the plain + // active-account path that never rewrites the provider; it must join the + // `anthropic-p` rows of the same account. Self-hosted installs keep the skip above. + const prior = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + try { + for (const name of ["anthropic", "anthropic2"]) { + const ctx: { accountLogLabel?: string } = {}; + stampOAuthAccountLabel(ctx, name, oauth, "acct-1"); + expect(ctx.accountLogLabel).toMatch(/^p[a-f0-9]{6}$/); + expect(formatAnthropicProviderForLog(name, "acct-1")).toBe(`${name}-${ctx.accountLogLabel}`); + } + const ctx: { accountLogLabel?: string } = {}; + stampOAuthAccountLabel(ctx, "openai", oauth, "acct-1"); + expect(ctx.accountLogLabel).toBeUndefined(); + } finally { + if (prior === undefined) delete process.env.OPENCODEX_AGENTRADAR; + else process.env.OPENCODEX_AGENTRADAR = prior; + } + }); + test("skips a suffixed openai provider name too", () => { // `openai-pabc123` bases to `openai`, so a suffix must not smuggle it past the check. const ctx: { accountLogLabel?: string } = {}; diff --git a/tests/oauth/oauth-accounts-api.test.ts b/tests/oauth/oauth-accounts-api.test.ts index eed095a6a40..e881fb2ea20 100644 --- a/tests/oauth/oauth-accounts-api.test.ts +++ b/tests/oauth/oauth-accounts-api.test.ts @@ -459,6 +459,56 @@ describe("multiauth accounts API", () => { } }); + // .27 parity (1f93f9141c, AgentRadar 20720): ?quota=1 fills a plan the credential has not stored yet. + test("GET ?quota=1 reads each Anthropic account's plan from its profile, or says why it can't", async () => { + const account = (id: string, access: string) => ({ id, credential: { access, refresh: `r-${id}`, expires: 9999999999999, email: `${id}@example.com`, accountId: `acct-${id}` } }); + writeFileSync(join(testDir, "auth.json"), JSON.stringify({ anthropic: { activeAccountId: "ultra", accounts: [ + account("ultra", "plan-t-ultra"), account("max", "plan-t-max"), account("pro", "plan-t-pro"), account("down", "plan-t-down"), account("odd", "plan-t-odd"), + ] } }), { mode: 0o600 }); + const profiles: Record = { + "plan-t-ultra": { organization: { organization_type: "claude_max", rate_limit_tier: "default_claude_max_20x" } }, + "plan-t-max": { organization: { organization_type: "claude_max", rate_limit_tier: "default_claude_max_5x" } }, + "plan-t-pro": { organization: { organization_type: "claude_pro" } }, + "plan-t-odd": { organization: { organization_type: "Max!" } }, + }; + const asked: string[] = []; + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const raw = input instanceof Request ? input.url : String(input); + if (!raw.startsWith("https://api.anthropic.com/")) return originalFetch(input, init); + if (raw !== "https://api.anthropic.com/api/oauth/profile") return new Response("{}", { status: 500 }); + const token = new Headers(init?.headers).get("authorization")!.replace("Bearer ", ""); + asked.push(token); + return token in profiles ? Response.json(profiles[token]) : new Response("{}", { status: 503 }); + }) as typeof fetch; + + const previousAgentRadar = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + const server = startServer(0); + try { + const read = async () => { + const res = await fetch(new URL("/api/oauth/accounts?provider=anthropic"a=1", server.url)); + expect(res.status).toBe(200); + const body = await res.json() as { accounts: Array<{ id: string; plan: string | null; planUnknownReason?: string }> }; + return Object.fromEntries(body.accounts.map(a => [a.id, [a.plan, a.planUnknownReason ?? null]])); + }; + expect(await read()).toEqual({ + // Ordinary installs answer the canonical max_20x / max_5x; this gateway keeps .27's words. + ultra: ["ultra", null], max: ["max", null], pro: ["pro", null], + down: [null, "the plan lookup failed (HTTP 503)"], odd: [null, "Anthropic did not report a plan"], + }); + expect(asked.sort()).toEqual(["plan-t-down", "plan-t-max", "plan-t-odd", "plan-t-pro", "plan-t-ultra"]); + // Cached per token: a second read asks upstream nothing. + await read(); + expect(asked).toHaveLength(5); + // The plain list stays a local read with an explicit null. + const plain = await (await fetch(new URL("/api/oauth/accounts?provider=anthropic", server.url))).json() as { accounts: Array<{ plan: unknown }> }; + expect(plain.accounts.map(a => a.plan)).toEqual([null, null, null, null, null]); + } finally { + if (previousAgentRadar === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previousAgentRadar; + await server.stop(true); + } + }); + test("PUT active switches; unknown account 404; unknown provider 400", async () => { const server = startServer(0); try { diff --git a/tests/oauth/oauth-callback-server.test.ts b/tests/oauth/oauth-callback-server.test.ts index feeb4301895..8bb23fa1b6f 100644 --- a/tests/oauth/oauth-callback-server.test.ts +++ b/tests/oauth/oauth-callback-server.test.ts @@ -1,4 +1,5 @@ import { describe, expect, test } from "bun:test"; +import { AntigravityOAuthFlow } from "../../src/oauth/google-antigravity"; import { OAuthCallbackFlow, type OAuthCallbackFlowOptions } from "../../src/oauth/callback-server"; import type { OAuthController, OAuthCredentials } from "../../src/oauth/types"; @@ -89,6 +90,12 @@ describe("OAuth callback server defaults", () => { } }); + test("Google Antigravity sign-in waits 20 minutes for the pasted redirect URL, not the 5-minute default", () => { + // A person without a local browser copies the redirect URL from the address bar and pastes it back by hand. + const flow = new AntigravityOAuthFlow({ onAuth: () => {} } as unknown as OAuthController); + expect(flow.timeoutMs).toBe(20 * 60 * 1000); + }); + test("manual-input loop rejects a code#state paste with a foreign suffix and re-prompts", async () => { let prompts = 0; const flow = new ManualFallbackFlow( diff --git a/tests/providers/api-key-selection-cooldown.test.ts b/tests/providers/api-key-selection-cooldown.test.ts index d1f04eaf55f..2b489867d90 100644 --- a/tests/providers/api-key-selection-cooldown.test.ts +++ b/tests/providers/api-key-selection-cooldown.test.ts @@ -156,3 +156,23 @@ describe("distributed selection cooldown validation", () => { expect(providerApiKeySelectionIsCurrent(config, NAME, route)).toBe(false); }); }); + +test("an AgentRadar gateway caps a key's Retry-After cooldown at 60s; a 401 hold and self-hosted keep 10 minutes", () => { + const prior = process.env.OPENCODEX_AGENTRADAR; + const now = Date.now(); + try { + for (const [flag, retryCap] of [[undefined, 600_000], ["1", 60_000]] as const) { + if (flag === undefined) delete process.env.OPENCODEX_AGENTRADAR; + else process.env.OPENCODEX_AGENTRADAR = flag; + clearKeyCooldowns(); + const config = fixture(); + cool(config, binding(config), now, "3600"); + expect(getKeyCooldownUntil(NAME, "a")).toBe(now + retryCap); + expect(recordDistributedKeyFailure(config, NAME, binding(config, 1)._apiKeyAttempt, 401, now)).toBe(true); + expect(getKeyCooldownUntil(NAME, "b")).toBe(now + 600_000); + } + } finally { + if (prior === undefined) delete process.env.OPENCODEX_AGENTRADAR; + else process.env.OPENCODEX_AGENTRADAR = prior; + } +}); diff --git a/tests/providers/kimi-monthly-quota.test.ts b/tests/providers/kimi-monthly-quota.test.ts index 9189c57cca3..fc09a9ed98e 100644 --- a/tests/providers/kimi-monthly-quota.test.ts +++ b/tests/providers/kimi-monthly-quota.test.ts @@ -512,6 +512,88 @@ describe("Kimi monthly quota", () => { }); }); +describe("Kimi usages.limit_5h / limit_7d windows", () => { + // Kimi's own client test payload, fields copied as written and serialized as JSON: MoonshotAI/kimi-code + // packages/oauth/test/managed-usage.test.ts "parses the full payload" (read 2026-11-15); the vendor + // parser is packages/oauth/src/managed-usage.ts. Documented shape, not a live capture; no key or id. + const CLIENT_TEST_USAGES_FULL = '{"goods_version":2,"usages":{"limit_5h":{"used_ratio":0.3,"reset_time":"2026-09-11T18:00:00Z"},"limit_7d":{"used_ratio":0.2,"reset_time":"2026-09-17T00:00:00Z"},"limit_month_total":{"used_ratio":0.4,"reset_time":"2026-10-01T00:00:00Z"},"limit_month_code":{"used_ratio":0.25,"reset_time":"2026-10-01T00:00:00Z"}}}'; + const kimiOnlyConfig = (): OcxConfig => ({ + defaultProvider: "kimi", + providers: { kimi: { adapter: "openai-chat", authMode: "oauth", baseUrl: "https://api.kimi.com/coding/v1" } }, + } as OcxConfig); + + // Ported verbatim from the hosted fork (aa80674b33). + test("Kimi quota reads Kimi Code's documented usages shape: 5-hour, weekly and monthly pools", async () => { + await saveCredential("kimi", { access: "kimi-access-secret", refresh: "kimi-refresh-secret", expires: Date.now() + 3600_000 }); + // Kimi's own client test payload (fixtures/external/kimi/README.md). + const usages = CLIENT_TEST_USAGES_FULL; + globalThis.fetch = (async (input: RequestInfo | URL) => { + if (String(input) !== "https://api.kimi.com/coding/v1/usages") return new Response("not found", { status: 404 }); + return new Response(usages, { status: 200 }); + }) as typeof fetch; + + const result = await fetchProviderQuotaReports(kimiOnlyConfig(), true); + + expect(result.reports[0]?.quota).toEqual({ + fiveHourPercent: 30, + fiveHourResetAt: Date.parse("2026-09-11T18:00:00Z"), + weeklyPercent: 20, + weeklyResetAt: Date.parse("2026-09-17T00:00:00Z"), + monthlyPercent: 40, + monthlyResetAt: Date.parse("2026-10-01T00:00:00Z"), + updatedAt: expect.any(Number), + }); + }); + + test("Kimi quota usages shape: when usages is present, legacy limits and totalQuota blocks are not read", async () => { + await saveCredential("kimi", { access: "kimi-access-secret", refresh: "kimi-refresh-secret", expires: Date.now() + 3600_000 }); + // Edge input, not a provider sample: the documented usages plus legacy blocks that must be ignored. + const documented = JSON.parse(CLIENT_TEST_USAGES_FULL); + globalThis.fetch = (async () => new Response(JSON.stringify({ + ...documented, + limits: [{ window: { duration: 300, timeUnit: "TIME_UNIT_MINUTE" }, detail: { limit: "100", remaining: "1" } }], + totalQuota: { limit: "100", remaining: "1" }, + }), { status: 200 })) as typeof fetch; + + const result = await fetchProviderQuotaReports(kimiOnlyConfig(), true); + + expect(result.reports[0]?.quota?.fiveHourPercent).toBe(30); + expect(result.reports[0]?.quota?.monthlyPercent).toBe(40); + }); + + // Edge input, not a provider sample: ratio forms the parser must accept or skip. + test("Kimi quota usages shape: a 0 ratio is 0%, limit_7d is weekly, a missing ratio is skipped", async () => { + await saveCredential("kimi", { access: "kimi-access-secret", refresh: "kimi-refresh-secret", expires: Date.now() + 3600_000 }); + globalThis.fetch = (async () => new Response(JSON.stringify({ + usages: { + limit_5h: { used_ratio: 0, reset_time: "2026-11-06T07:46:00Z" }, + limit_7d: { used_ratio: "0.5", reset_time: "2026-11-10T00:00:00Z" }, + limit_month_total: { reset_time: "2026-12-06T03:22:00Z" }, + }, + }), { status: 200 })) as typeof fetch; + + const result = await fetchProviderQuotaReports(kimiOnlyConfig(), true); + + expect(result.reports[0]?.quota).toEqual({ + fiveHourPercent: 0, + fiveHourResetAt: Date.parse("2026-11-06T07:46:00Z"), + weeklyPercent: 50, + weeklyResetAt: Date.parse("2026-11-10T00:00:00Z"), + updatedAt: expect.any(Number), + }); + }); + + test("a positive usages window wins over an unused legacy placeholder", async () => { + mockUsage({ + ...JSON.parse(CLIENT_TEST_USAGES_FULL), + usage: { limit: "100", remaining: "100" }, + limits: [{ window: { duration: 300, timeUnit: "TIME_UNIT_MINUTE" }, detail: { limit: "100", remaining: "100" } }], + }); + const { reports } = await fetchProviderQuotaReports(config(), true); + expect(reports[0]?.quota).toMatchObject({ fiveHourPercent: 30, weeklyPercent: 20, monthlyPercent: 40 }); + }); +}); + describe("Kimi regional (kimi.ai) Coding Plan quota (#8332)", () => { const regional = "https://api.kimi.ai/coding/v1"; diff --git a/tests/providers/provider-registry-parity.test.ts b/tests/providers/provider-registry-parity.test.ts index 3a8cdab945c..7333c7fead7 100644 --- a/tests/providers/provider-registry-parity.test.ts +++ b/tests/providers/provider-registry-parity.test.ts @@ -1561,6 +1561,7 @@ describe("provider registry parity", () => { test("jawcode metadata aliases are derived from the registry", () => { expect(deriveJawcodeAliases()).toEqual({ xai: "xai", + "xai-apikey": "xai", anthropic: "anthropic", anthropic2: "anthropic", "anthropic-apikey": "anthropic", diff --git a/tests/providers/provider-registry-xai-apikey.test.ts b/tests/providers/provider-registry-xai-apikey.test.ts new file mode 100644 index 00000000000..ff8375f4d0c --- /dev/null +++ b/tests/providers/provider-registry-xai-apikey.test.ts @@ -0,0 +1,23 @@ +import { expect, test } from "bun:test"; +import { resolveMetadataProvider } from "../../src/generated/model-metadata"; +import { KEY_LOGIN_PROVIDERS } from "../../src/oauth/key-providers"; +import { PROVIDER_REGISTRY } from "../../src/providers/registry"; + +test("xai-apikey is a separate key provider carrying the xai row's key-auth capabilities", () => { + // Hosted gateways regenerate their key-provider list from this registry and store the xAI key + // under this id; a missing row would drop the configured Grok key. + const xai = PROVIDER_REGISTRY.find(entry => entry.id === "xai")!; + const key = PROVIDER_REGISTRY.find(entry => entry.id === "xai-apikey")!; + expect(key).toMatchObject({ + authKind: "key", adapter: "openai-chat", baseUrl: "https://api.x.ai/v1", jawcodeBundle: "xai", + supportsServiceTier: true, chatServiceTier: true, + supportsOpenAiWebSearchToolFields: false, supportsResponsesCustomTools: false, parallelToolCalls: true, liveModels: true, + }); + expect(key.deprecated).toBeUndefined(); + for (const field of ["allowKeyAuthOverride", "keyAuthServiceTier", "oauthId"] as const) expect(key[field]).toBeUndefined(); + const { id: _a, label: _b, authKind: _c, allowKeyAuthOverride: _d, keyAuthServiceTier: _e, oauthId: _f, featured: _g, note: _h, + supportsServiceTier: _i, chatServiceTier: _j, dashboardUrl: _k, ...shared } = xai; + expect(key).toMatchObject(shared); + expect(KEY_LOGIN_PROVIDERS["xai-apikey"]).toMatchObject({ adapter: "openai-chat", baseUrl: "https://api.x.ai/v1" }); + expect(resolveMetadataProvider("xai-apikey")).toBe("xai"); +}); diff --git a/tests/providers/quota-explain.test.ts b/tests/providers/quota-explain.test.ts new file mode 100644 index 00000000000..5ab31d50ecc --- /dev/null +++ b/tests/providers/quota-explain.test.ts @@ -0,0 +1,51 @@ +import { describe, expect, test } from "bun:test"; +import { explainQuotaRefusal, explainPoolQuotaRefusal } from "../../src/providers/quota-explain"; +import type { ProviderQuota } from "../../src/providers/quota-types"; + +const now = 1_800_000_000_000; +const q = (extra: Record): ProviderQuota => + ({ updatedAt: now, ...extra }) as unknown as ProviderQuota; + +describe("quota refusal explanation", () => { + test("names the spent model-family window instead of leaving a generic rate limit", () => { + const note = explainQuotaRefusal( + q({ customWindows: [{ label: "Fable", scope: "model", percent: 100, resetAt: now + 6 * 3_600_000 }] }), + "claude-fable-5-1", now, + ); + expect(note).toContain("Fable window at 100%"); + expect(note).toContain("resets in 6h"); + }); + + test("a model-scoped window says nothing about a different family", () => { + // The whole point of the producer-set scope flag: Fable being spent is not evidence about + // Opus, and saying so would send the operator after a quota problem they do not have. + expect(explainQuotaRefusal( + q({ customWindows: [{ label: "Fable", scope: "model", percent: 100, resetAt: now + 3_600_000 }] }), + "claude-opus-5", now, + )).toBeUndefined(); + }); + + test("a provider-wide window applies to every model", () => { + expect(explainQuotaRefusal( + q({ customWindows: [{ label: "Prepaid credits", percent: 100, resetAt: now + 3_600_000 }] }), + "claude-opus-5", now, + )).toContain("Prepaid credits at 100%"); + }); + + test("stays silent when nothing is exhausted, so throttling still reads as throttling", () => { + expect(explainQuotaRefusal(q({ weeklyPercent: 40 }), "claude-fable-5-1", now)).toBeUndefined(); + }); + + test("a window whose reset already passed is not reported as spent", () => { + expect(explainQuotaRefusal(q({ weeklyPercent: 100, weeklyResetAt: now - 1 }), "claude-opus-5", now)) + .toBeUndefined(); + }); + + test("a pool with no cached reading explains nothing rather than guessing", () => { + // Partial evidence is the dangerous case: an account the cache cannot speak for might have + // headroom, so claiming a pool-wide outage from it would be a confident wrong answer. + expect(explainPoolQuotaRefusal("anthropic", ["never-cached"], "claude-fable-5-1", now)) + .toBeUndefined(); + expect(explainPoolQuotaRefusal("anthropic", [], "claude-fable-5-1", now)).toBeUndefined(); + }); +}); diff --git a/tests/providers/xai/grok-47-fast-model-wire.test.ts b/tests/providers/xai/grok-47-fast-model-wire.test.ts index 4c565be9b4a..d73ec71a957 100644 --- a/tests/providers/xai/grok-47-fast-model-wire.test.ts +++ b/tests/providers/xai/grok-47-fast-model-wire.test.ts @@ -176,18 +176,20 @@ function assertFastSend(send: CapturedSend): void { expect(Object.hasOwn(send.body, "service_tier")).toBe(false); } -function assertFastReceipts(receiptModel = LOGICAL_MODEL): void { +function assertFastReceipts(receiptModel = LOGICAL_MODEL, attemptCount = 1): void { const log = getRequestLogEntries().at(-1); const usage = readUsageEntries().at(-1); for (const receipt of [log, usage]) { expect(receipt?.model).toBe(receiptModel); expect(receipt?.wireModel).toBe(FAST_MODEL); - expect(receipt?.attempts).toHaveLength(1); - expect(receipt?.attempts?.[0]?.model).toBe(LOGICAL_MODEL); - expect(receipt?.attempts?.[0]?.tierOutcome).toMatchObject({ - wireKind: "model-variant", wireValue: FAST_MODEL, fastOutcome: "applied", - confirmation: "assumed", responseTierAuthoritative: false, - }); + expect(receipt?.attempts).toHaveLength(attemptCount); + for (const attempt of receipt?.attempts ?? []) { + expect(attempt.model).toBe(LOGICAL_MODEL); + expect(attempt.tierOutcome).toMatchObject({ + wireKind: "model-variant", wireValue: FAST_MODEL, fastOutcome: "applied", + confirmation: "assumed", responseTierAuthoritative: false, + }); + } } } @@ -360,8 +362,9 @@ describe("Grok 4.7 Fast serialized upstream model", () => { "Bearer fake-xai-old-access", "Bearer fake-xai-new-access", ]); expect(fixture.counts.refresh).toBe(1); - assertFastReceipts(); - expect(readUsageEntries().at(-1)?.attempts?.[0]?.sendCount).toBe(2); + assertFastReceipts(LOGICAL_MODEL, 2); + expect(readUsageEntries().at(-1)?.attempts?.map(attempt => ({ status: attempt.status, sendCount: attempt.sendCount }))) + .toEqual([{ status: 401, sendCount: 1 }, { status: 200, sendCount: 1 }]); }); test("WebSocket response.create sends build-fast without a tier and preserves logical receipts", async () => { diff --git a/tests/responses/responses-compact-400-quota-echo.test.ts b/tests/responses/responses-compact-400-quota-echo.test.ts new file mode 100644 index 00000000000..062f9954147 --- /dev/null +++ b/tests/responses/responses-compact-400-quota-echo.test.ts @@ -0,0 +1,117 @@ +/** + * Compact reads a non-ok body for quota evidence so a quota refusal behind a generic status still + * cools the pool account. It used to read the body with no status, so a 400 or 422 that only + * echoed client values ("rate limit", "usage limit", "insufficient_quota") was recorded as a 429 + * and cooled a healthy account. The body is now read with its real status. + */ +import { afterEach, describe, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveCodexAccountCredential } from "../../src/codex/account-store"; +import { clearAccountQuota, updateAccountQuota } from "../../src/codex/auth-api"; +import { clearCodexUpstreamHealth, isCodexAccountInCooldown, isCodexAccountSoftAvoided } from "../../src/codex/routing"; +import { clearUpstreamHostHealth } from "../../src/codex/upstream-host-health"; +import { handleResponsesCompact } from "../../src/server/responses"; +import type { OcxConfig } from "../../src/types"; +import { baseCompactionBody, compactionRequest, twoAccountPoolConfig } from "../helpers/compaction-routing-fixtures"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const originalFetch = globalThis.fetch; + +// REAL CAPTURE, byte-identical: a staging gateway log line for an Anthropic HTTP 400 on House +// inference (2026-11-11; AgentRadarLabs/agentradar-paperclip fixtures/external/ +// anthropic-extra-usage-thirdparty-400.txt, blob 2b121a6ba9). The logger cut it at 200 chars; the +// bytes after the prefix are Anthropic's raw envelope, and only the closing quote and braces the cut +// removed are added below. It holds no account, org or request id. +// No recorded validation 400 that echoes client values exists yet, so the echo bodies are this +// recorded envelope with only `error.message` replaced; envelope keys, `type` and `error.type` stay as recorded. +const RECORDED_LINE = + "Provider error 400: {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"Third-party apps now draw from your extra usage, not your plan limits. Add more at claude.ai/settings/usage and k"; +const RECORDED_BODY = `${RECORDED_LINE.slice("Provider error 400: ".length)}"}}`; +function recordedEnvelopeWithMessage(message: string): string { + const recorded = JSON.parse(RECORDED_BODY) as { error: Record }; + return JSON.stringify({ ...recorded, error: { ...recorded.error, message } }); +} +const ECHO = recordedEnvelopeWithMessage( + "input.0.content.0.text: unexpected value: rate limit, too many requests, usage limit reached, toolu_insufficient_quota", +); + +async function withPoolEnv(name: string, run: (config: OcxConfig) => Promise): Promise { + const testDir = mkdtempSync(join(tmpdir(), name)); + const previousOpencodexHome = process.env.OPENCODEX_HOME; + const previousCodexHome = process.env.CODEX_HOME; + process.env.OPENCODEX_HOME = testDir; + process.env.CODEX_HOME = testDir; + const releaseSpendHome = acquireOwnedSpendHome(); + clearCodexUpstreamHealth(); + clearUpstreamHostHealth(); + clearAccountQuota(); + for (const id of ["pool-a", "pool-b"]) { + saveCodexAccountCredential(id, { + accessToken: `${id}-access-token`, + refreshToken: `${id}-refresh-token`, + expiresAt: Date.now() + 300_000, + chatgptAccountId: id === "pool-a" ? "pool_acc_a" : "pool_acc_b", + }); + updateAccountQuota(id, id === "pool-a" ? 10 : 20); + } + try { + await run(twoAccountPoolConfig()); + } finally { + globalThis.fetch = originalFetch; + clearCodexUpstreamHealth(); + clearUpstreamHostHealth(); + clearAccountQuota(); + releaseSpendHome(); + removeTreeWithRetry(testDir); + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; + } +} + +async function compactAgainst(status: number, body: string, config: OcxConfig) { + let sends = 0; + globalThis.fetch = (async () => { + sends += 1; + return new Response(body, { status, headers: { "content-type": "application/json" } }); + }) as unknown as typeof fetch; + const res = await handleResponsesCompact(compactionRequest(baseCompactionBody({})), config, { model: "", provider: "" }); + const accountMarked = isCodexAccountInCooldown("pool-a") || isCodexAccountSoftAvoided("pool-a"); + return { res, sends, accountMarked }; +} + +afterEach(() => { globalThis.fetch = originalFetch; }); + +describe("compact reads a non-ok body for quota with its real status", () => { + for (const status of [400, 422]) { + test(`MUST-FAIL: an echoing ${status} does not cool the pool account`, async () => { + await withPoolEnv(`ocx-compact-echo-${status}-`, async config => { + const result = await compactAgainst(status, ECHO, config); + expect(result.res.status).toBe(status); + expect(result.sends).toBe(1); + expect(result.accountMarked).toBe(false); + }); + }); + } + + test("a 400 whose error.code field is insufficient_quota still cools the account", async () => { + await withPoolEnv("ocx-compact-field-400-", async config => { + // OpenAI/Codex quota shape: the verdict is the `code` field, not the message words. + const body = JSON.stringify({ error: { message: "quota", type: "invalid_request_error", code: "insufficient_quota" } }); + const result = await compactAgainst(400, body, config); + expect(result.accountMarked).toBe(true); + }); + }); + + test("a quota body behind a 5xx still cools the account", async () => { + await withPoolEnv("ocx-compact-5xx-quota-", async config => { + const body = JSON.stringify({ error: { message: "The usage limit has been reached", type: "usage_limit_reached" } }); + const result = await compactAgainst(502, body, config); + expect(result.accountMarked).toBe(true); + }); + }); +}); diff --git a/tests/responses/responses-fetch-helpers-boundary.test.ts b/tests/responses/responses-fetch-helpers-boundary.test.ts index 9de711c03ef..094ccf6cba3 100644 --- a/tests/responses/responses-fetch-helpers-boundary.test.ts +++ b/tests/responses/responses-fetch-helpers-boundary.test.ts @@ -221,6 +221,8 @@ describe("Responses fetch-helper import boundary", () => { "../../lib/proxy-env", "../../lib/redact", "../../lib/upstream-http-version", + // AgentRadar-gated 4xx refusal-shape line; runtime imports are node:crypto and ./redact only. + "../../lib/upstream-refusal-log", // Import-free plugin rewrite slot (src/plugins/upstream-hooks.ts). "../../plugins/upstream-hooks", "../../providers/request-pacing", diff --git a/tests/responses/responses-native-main-refresh.test.ts b/tests/responses/responses-native-main-refresh.test.ts index 11870ccff41..f8c2fff5a72 100644 --- a/tests/responses/responses-native-main-refresh.test.ts +++ b/tests/responses/responses-native-main-refresh.test.ts @@ -393,17 +393,76 @@ describe("native main 401 refresh and replay", () => { } }); + test("a main-context 401 names the branch that produced the context", async () => { + // The skip line is the only place a `main` context is identifiable: it carries no account id, + // so the request log has no account label for it and every branch looks the same afterwards. + // Direct with an admission bearer is the one `main` branch reachable from a test, and pinning + // it here is what keeps `mainReason` plumbed from the resolver to the log. + const payload = Buffer.from(JSON.stringify({ exp: Math.floor(Date.now() / 1000) + 86_400 })).toString("base64url"); + writeFileSync(join(home, "auth.json"), JSON.stringify({ + tokens: { access_token: `header.${payload}.signature`, refresh_token: "refresh-grant", account_id: "account-main" }, + })); + globalThis.fetch = (async (input: RequestInfo | URL) => { + const url = new URL(input instanceof Request ? input.url : String(input)); + if (!url.pathname.endsWith("/responses")) { + return Response.json({ rate_limit: { primary_window: { used_percent: 10 } } }); + } + return Response.json({ error: { message: "invalidated", type: "invalid_request_error", code: "token_invalidated" } }, { status: 401 }); + }) as typeof fetch; + + const cfg = config(); + cfg.hostname = "0.0.0.0"; + cfg.providers.openai!.codexAccountMode = "direct"; + cfg.apiKeys = [{ + id: "direct-test", name: "direct-test", key: "ocx_data_direct_ingress", + createdAt: "2026-09-14T00:00:00.000Z", + }]; + const req = request("/v1/responses"); + req.headers.set("authorization", "Bearer ocx_data_direct_ingress"); + const admission = resolveResponsesApiAuth(req, cfg); + expect(admission?.source).toBe("bearer"); + + const lines: string[] = []; + const warnSpy = spyOn(console, "warn").mockImplementation((...args: unknown[]) => { + lines.push(args.map(String).join(" ")); + }); + const turn = tryAdmitTurn(); + try { + const response = await handleResponses(req, cfg, { model: "", provider: "" } as RequestLogContext, { + admission: admission!, turnAdmissionLease: turn!, + }); + await response.text(); + } finally { + warnSpy.mockRestore(); + turn?.release(); + } + const skip = lines.find(line => line.startsWith("[codex] 401 upstream without native-main refresh")); + expect(skip).toBeDefined(); + expect(skip).toContain("kind=main"); + expect(skip).toContain("accountMode=direct"); + // Direct substitution forwards our stored credential, so the caller's own bearer is an + // admission secret and not a forwardable Codex one: the pairing that tells a never-resolved + // context apart from one that legitimately belongs to the caller. + expect(skip).toContain("forwardableBearer=n"); + expect(skip).toContain("admission=bearer"); + }); + test("Responses refreshes and performs exactly one physical replay", async () => { const harness = install401ThenRefreshHarness(); + const logCtx = { model: "", provider: "" } as RequestLogContext; const response = await handleResponses( request("/v1/responses"), config(), - { model: "", provider: "" } as RequestLogContext, + logCtx, ); expect(response.status).toBe(200); expect(harness.sends).toEqual(["Bearer rejected-access", "Bearer refreshed-access"]); expect(harness.refreshes).toEqual(["refresh-grant"]); + expect(logCtx.attempts).toMatchObject([ + { status: 401, sendCount: 1 }, + { sendCount: 1, recoveryKinds: ["oauth-401"] }, + ]); expect(JSON.parse(readFileSync(join(home, "auth.json"), "utf8")).tokens.refresh_token) .toBe("rotated-refresh"); }); diff --git a/tests/responses/responses-pool-401-refresh.test.ts b/tests/responses/responses-pool-401-refresh.test.ts index 8ec98b0bdca..7c9ab8cd6d3 100644 --- a/tests/responses/responses-pool-401-refresh.test.ts +++ b/tests/responses/responses-pool-401-refresh.test.ts @@ -667,16 +667,21 @@ describe("ordinary pool 401 refresh and replay (#2887)", () => { test("Responses refreshes a time-valid stored credential once and replays the same account", async () => { const harness = installHarness(); + const logCtx = { model: "", provider: "" } as RequestLogContext; const response = await handleResponses( request("/v1/responses"), config(), - { model: "", provider: "" } as RequestLogContext, + logCtx, ); // The defect surfaced as a 401 reaching the client with no refresh attempted. expect(response.status).toBe(200); expect(harness.refreshes).toEqual(["refresh-grant"]); expect(harness.sends).toEqual(["Bearer rejected-access", "Bearer refreshed-access"]); + expect(logCtx.attempts).toMatchObject([ + { status: 401, sendCount: 1 }, + { sendCount: 1, recoveryKinds: ["oauth-401"] }, + ]); // Quarantine is the other half of the report: the account must stay usable. expect(isAccountNeedsReauth(ACCOUNT_ID)).toBe(false); expect(readStoredGeneration()).toBe(4); @@ -1465,12 +1470,13 @@ describe("ordinary pool 401 refresh and replay (#2887)", () => { }); let dispatchSignals = 0; + const logCtx = { model: "", provider: "" } as RequestLogContext; let response: Response; try { response = await handleResponses( request("/v1/responses"), cfg, - { model: "", provider: "" } as RequestLogContext, + logCtx, { onStoredPool401ReplayDispatched: () => { dispatchSignals += 1; } }, ); } finally { @@ -1484,6 +1490,9 @@ describe("ordinary pool 401 refresh and replay (#2887)", () => { // downstream fallback is unreachable here for a reason that has nothing to do with this fix. expect(dispatchSignals).toBe(0); expect(harness.sends.filter(send => send === "Bearer refreshed-access")).toEqual([]); + // .27 parity (fa05c0c8b0): a replay the pacing queue refused books no send. + expect(logCtx.attempts).toHaveLength(1); + expect(logCtx.attempts?.[0]).toMatchObject({ sendCount: 1 }); // The refresh did happen — this is the post-refresh replay being rejected, not an earlier stop. expect(harness.refreshes).toEqual(["refresh-grant"]); expect(response.status).toBe(429); diff --git a/tests/responses/responses-state-env-bounds.test.ts b/tests/responses/responses-state-env-bounds.test.ts new file mode 100644 index 00000000000..ddb1f724890 --- /dev/null +++ b/tests/responses/responses-state-env-bounds.test.ts @@ -0,0 +1,160 @@ +/** + * Environment tightening of the continuation store's bounds. + * + * The built-in resident cap (64 MiB, counted as serialized bytes) and the 24 h + * retention suit a long-lived host. A small instance needs both lower, without a + * rebuild: OPENCODEX_RESPONSE_STATE_MAX_BYTES and OPENCODEX_RESPONSE_STATE_TTL_MS + * may lower them, never raise them. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + clearResponseStateForTests, + clearResponseStateMemoryForTests, + getSpilledResponseBytesForTests, + getStoredResponseBytesForTests, + rememberResponseState, + responseStateMetrics, + setResponseStateByteCapForTests, + sweepExpiredResponseStates, +} from "../../src/responses/state"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const CAP_ENV = "OPENCODEX_RESPONSE_STATE_MAX_BYTES"; +const TTL_ENV = "OPENCODEX_RESPONSE_STATE_TTL_MS"; +const SPILL_ENV = "OPENCODEX_RESPONSE_STATE_MAX_SPILL_BYTES"; + +function assistantItem(turn: number, chars = 1024) { + return { + id: `msg_env_${turn}`, + type: "message", + role: "assistant", + content: [{ type: "output_text", text: `a${turn}:` + "y".repeat(chars) }], + }; +} + +function rememberForced(id: string, input: unknown[], output: unknown[]): void { + rememberResponseState( + { model: "test/model", input, store: false }, + { id, status: "completed", output }, + undefined, + { force: true }, + ); +} + +describe("responses-state environment bounds", () => { + let home: string; + const priorHome = process.env["OPENCODEX_HOME"]; + const priorCap = process.env[CAP_ENV]; + const priorTtl = process.env[TTL_ENV]; + const priorSpill = process.env[SPILL_ENV]; + + beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "ocx-state-env-")); + process.env["OPENCODEX_HOME"] = home; + delete process.env[CAP_ENV]; + delete process.env[TTL_ENV]; + delete process.env[SPILL_ENV]; + clearResponseStateMemoryForTests(); + }); + + afterEach(() => { + setResponseStateByteCapForTests(null); + clearResponseStateForTests(); + removeTreeWithRetry(home); + if (priorHome === undefined) delete process.env["OPENCODEX_HOME"]; + else process.env["OPENCODEX_HOME"] = priorHome; + if (priorCap === undefined) delete process.env[CAP_ENV]; + else process.env[CAP_ENV] = priorCap; + if (priorTtl === undefined) delete process.env[TTL_ENV]; + else process.env[TTL_ENV] = priorTtl; + if (priorSpill === undefined) delete process.env[SPILL_ENV]; + else process.env[SPILL_ENV] = priorSpill; + }); + + test("the resident byte cap can be lowered from the environment", () => { + process.env[CAP_ENV] = String(256 * 1024); + for (let index = 0; index < 8; index += 1) { + rememberForced( + `resp_solo_${index}`, + [{ role: "user", content: `solo-${index}:` + "z".repeat(64 * 1024) }], + [assistantItem(index)], + ); + } + expect(getStoredResponseBytesForTests()).toBeLessThanOrEqual(256 * 1024); + expect(responseStateMetrics().spillStubCount).toBeGreaterThan(0); + }); + + test("retention can be shortened from the environment", () => { + process.env[TTL_ENV] = String(60_000); + rememberForced("resp_ttl", [{ role: "user", content: "hello" }], [assistantItem(0)]); + expect(sweepExpiredResponseStates(Date.now() + 30_000)).toBe(0); + expect(sweepExpiredResponseStates(Date.now() + 61_000)).toBe(1); + }); + + test("unlinked full-history turns stay under a lowered byte cap", () => { + // A client that resends its whole history every turn, with no previous_response_id, + // under forced storage: each turn is a fresh full copy under a fresh id, parsed anew. + const cap = 512 * 1024; + process.env[CAP_ENV] = String(cap); + const history: unknown[] = []; + let unboundedTotal = 0; + for (let turn = 0; turn < 40; turn += 1) { + history.push({ role: "user", content: `u${turn}:` + "x".repeat(4 * 1024) }); + const output = [assistantItem(turn, 4 * 1024)]; + const input = JSON.parse(JSON.stringify(history)) as unknown[]; + unboundedTotal += JSON.stringify(input).length + JSON.stringify(output).length; + rememberForced(`resp_full_${turn}`, input, output); + history.push(output[0]); + expect(getStoredResponseBytesForTests()).toBeLessThanOrEqual(cap); + } + // The traffic itself is far past the cap, so the bound is doing the work. + expect(unboundedTotal).toBeGreaterThan(4 * cap); + const metrics = responseStateMetrics(); + expect(metrics.count).toBe(40); + expect(metrics.residentCount).toBeGreaterThan(0); + expect(metrics.spillStubCount).toBeGreaterThan(0); + }); + + test("the disk spill cap can be lowered from the environment, so a small temp disk is not filled", () => { + // RAM cap low enough that every full-history turn spills; disk cap far below what the turns write. + const spillCap = 128 * 1024; + process.env[CAP_ENV] = String(32 * 1024); + process.env[SPILL_ENV] = String(spillCap); + const history: unknown[] = []; + let written = 0; + let peakSpilled = 0; + for (let turn = 0; turn < 30; turn += 1) { + history.push({ role: "user", content: `u${turn}:` + "x".repeat(4 * 1024) }); + const output = [assistantItem(turn, 4 * 1024)]; + written += JSON.stringify(history).length + JSON.stringify(output).length; + rememberForced(`resp_disk_${turn}`, JSON.parse(JSON.stringify(history)) as unknown[], output); + history.push(output[0]); + peakSpilled = Math.max(peakSpilled, getSpilledResponseBytesForTests()); + expect(getSpilledResponseBytesForTests()).toBeLessThanOrEqual(spillCap); + } + expect(written).toBeGreaterThan(4 * spillCap); + // Turns did spill, so the disk cap (not an idle spill path) is what held the bound. + expect(peakSpilled).toBeGreaterThan(0); + expect(getSpilledResponseBytesForTests()).toBeLessThanOrEqual(spillCap); + }); + + test("a fraction below 1 is ignored rather than flooring to 0 and emptying the store", () => { + process.env[CAP_ENV] = "0.5"; + process.env[TTL_ENV] = "0.5"; + rememberForced("resp_fraction", [{ role: "user", content: "hello" }], [assistantItem(0)]); + expect(responseStateMetrics().count).toBe(1); + expect(getStoredResponseBytesForTests()).toBeGreaterThan(0); + expect(sweepExpiredResponseStates(Date.now() + 1_000)).toBe(0); + }); + + test("a value above the built-in cap is ignored rather than raising it", () => { + process.env[CAP_ENV] = String(Number.MAX_SAFE_INTEGER); + process.env[TTL_ENV] = String(Number.MAX_SAFE_INTEGER); + rememberForced("resp_raise", [{ role: "user", content: "hello" }], [assistantItem(0)]); + // Still expires at the built-in 24 h retention. + expect(sweepExpiredResponseStates(Date.now() + 24 * 60 * 60 * 1_000 + 1)).toBe(1); + }); +}); diff --git a/tests/routing/combo-failure-transport-parity.test.ts b/tests/routing/combo-failure-transport-parity.test.ts index 4118c9e660b..79be082f0ba 100644 --- a/tests/routing/combo-failure-transport-parity.test.ts +++ b/tests/routing/combo-failure-transport-parity.test.ts @@ -1,4 +1,6 @@ import { describe, expect, test } from "bun:test"; +import { formatErrorResponse } from "../../src/bridge"; +import { markVerbatimComboFailure, verbatimComboFailure } from "../../src/bridge/errors"; import { comboFailureCooldownScope, comboFailureDecision, isProviderModerationRefusal } from "../../src/combos/failover"; import { consumeComboFailure } from "../../src/server/responses/core-combo-failure"; import { preflightComboStreamResponse } from "../../src/server/responses/combo-stream-preflight"; @@ -503,3 +505,15 @@ describe("every prose field beside a coded verdict must describe the moderation expect(performance.now() - started).toBeLessThan(250); }); }); + +test("AgentRadar hosted: a verbatim child failure is classified by its formatted status, presented by its own bytes", async () => { + // An SSE pre-output error arrives as HTTP 200; deciding the hop on that status would end the ladder. + const formatted = formatErrorResponse(529, "overloaded_error", "Overloaded"); + markVerbatimComboFailure(formatted, () => new Response("event: error\ndata: {}\n\n", { status: 200 })); + const failure = await consumeComboFailure(formatted); + expect(failure.response.status).toBe(529); + expect(comboFailureDecision(failure.response.status, failure.classificationText, { code: failure.upstreamCode })).toBe("hop"); + const presented = verbatimComboFailure(failure.response); + expect(presented?.status).toBe(200); + expect(await presented!.text()).toBe("event: error\ndata: {}\n\n"); +}); diff --git a/tests/routing/combo-pool-stale-cooldown.test.ts b/tests/routing/combo-pool-stale-cooldown.test.ts new file mode 100644 index 00000000000..4e64b7261cc --- /dev/null +++ b/tests/routing/combo-pool-stale-cooldown.test.ts @@ -0,0 +1,48 @@ +// Ported from v2.65.0-vadymhimself.27 tests/routing/combo-pool-stale-cooldown.test.ts (4269f45a42); the +// third .27 case exercised the fork-only source-replay gate, which the native lane replaces. +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { + clearComboTargetCooldowns, + coolComboTarget, + isComboTargetInCooldown, + setComboPoolEligibilityProbe, +} from "../../src/combos/failover"; + +// upstream: the lift is an AgentRadar hosted-gateway rule; off the flag upstream parks a non-pool 429 in full. +const previousHosted = process.env.OPENCODEX_AGENTRADAR; +beforeEach(() => { process.env.OPENCODEX_AGENTRADAR = "1"; }); +afterEach(() => { + setComboPoolEligibilityProbe(undefined); + clearComboTargetCooldowns(); + if (previousHosted === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previousHosted; +}); + +const target = { provider: "anthropic", model: "claude-opus-5-5" }; + +test("a quota cooldown on a pooled target lifts once the pool has an eligible account", () => { + coolComboTarget("w", target, { retryAfter: "300", status: 429 }); + setComboPoolEligibilityProbe(() => false); + expect(isComboTargetInCooldown("w", target)).toBe(true); + setComboPoolEligibilityProbe(provider => provider === "anthropic"); + expect(isComboTargetInCooldown("w", target)).toBe(false); +}); + +test("a non-quota cooldown is not lifted by pool eligibility", () => { + coolComboTarget("w", target, { retryAfter: "300", status: 502 }); + setComboPoolEligibilityProbe(() => true); + expect(isComboTargetInCooldown("w", target)).toBe(true); +}); + +test("an AgentRadar hosted gateway caps server cooldowns at ten minutes too (d3ea59ccb9)", () => { + const now = Date.parse("2026-07-18T00:00:00.000Z"); + coolComboTarget("hosted-weekly-retry", target, { now, retryAfter: "318747" }); + expect(isComboTargetInCooldown("hosted-weekly-retry", target, now + 600_000 - 1)).toBe(true); + expect(isComboTargetInCooldown("hosted-weekly-retry", target, now + 600_000)).toBe(false); +}); + +test("off the hosted flag a quota cooldown is not lifted by pool eligibility", () => { + delete process.env.OPENCODEX_AGENTRADAR; + coolComboTarget("w", target, { retryAfter: "300", status: 429 }); + setComboPoolEligibilityProbe(() => true); + expect(isComboTargetInCooldown("w", target)).toBe(true); +}); diff --git a/tests/routing/combo-stream-preflight.test.ts b/tests/routing/combo-stream-preflight.test.ts index 72f353a1020..882864bab19 100644 --- a/tests/routing/combo-stream-preflight.test.ts +++ b/tests/routing/combo-stream-preflight.test.ts @@ -234,6 +234,41 @@ describe("combo stream preflight", () => { } }); + test("AgentRadar hosted: converts zero-output transport incompletes into retryable HTTP failures", async () => { + const previous = process.env.OPENCODEX_AGENTRADAR; + process.env.OPENCODEX_AGENTRADAR = "1"; + try { + const cases = [ + ["adapter_eof", "Upstream stream ended unexpectedly without a terminal event", 502], + ["missing_terminal_event", "Upstream incomplete", 502], + ["upstream_stall_timeout", "Upstream stalled", 504], + ] as const; + for (const [reason, message, status] of cases) { + const result = await preflightComboStreamResponse(sse( + { type: "response.created", response: { id: "r1", status: "in_progress" } }, + { + type: "response.incomplete", + response: { + id: "r1", + status: "incomplete", + incomplete_details: { reason }, + usage: { input_tokens: 11, output_tokens: 0, total_tokens: 11 }, + }, + }, + ), { model: "m1", provider: "a" }); + + expect(result.kind).toBe("failed"); + expect(result.response.status).toBe(status); + const body = await result.response.json(); + expect(body.error).toMatchObject({ type: "upstream_error", code: "upstream_server_error" }); + expect(body.error.message).toContain(message); + expect(body.response.usage).toMatchObject({ input_tokens: 11, output_tokens: 0 }); + } + } finally { + if (previous === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previous; + } + }); + test("converts a zero-output incomplete without a stated reason into a retryable failure", async () => { for (const response of [ { id: "r1", status: "incomplete" }, diff --git a/tests/routing/router-combo-400-quota-echo.test.ts b/tests/routing/router-combo-400-quota-echo.test.ts new file mode 100644 index 00000000000..678be9e2c8b --- /dev/null +++ b/tests/routing/router-combo-400-quota-echo.test.ts @@ -0,0 +1,459 @@ +/** + * A 400 is quota only by its fields, never by quota words anywhere in its text (Mars L1 on #25). + * + * Anthropic's validation 400s echo client values (tool ids, content) into `error.message`. + * classifyError used to match quota phrases anywhere in a 400, so one crafted request read as a + * spent account: `insufficient_quota` is a provider-scoped combo failure, which cools every target + * of that provider for ten minutes. These tests drive the same steps the combo takes on an adapter + * 400: the dispatch's client error, the combo's read of it, the hop decision and the cooldown. + * + * Ported verbatim from the hosted fork (f4303eb19c) except four combo cases that current upstream + * deliberately decides elsewhere (provider-cooldown hop on a 400 quota envelope, free-tier field hop, + * and the fork-only pool-rotation module); see the parity ledger for their disposition. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { formatErrorResponse } from "../../src/bridge"; +import { adapterFailureFromEvent } from "../../src/bridge/internal"; +import { advanceComboAfterFailure, clearComboTargetCooldowns, isComboTargetInCooldown, pickComboTarget } from "../../src/combos"; +import { comboFailureCooldownScope, comboFailureDecision } from "../../src/combos/failover"; +import { + isModelHealthBlocked, + recordSubagentQuotaFailureForThreadSpawn, + resetSubagentModelFallbackStateForTests, +} from "../../src/codex/subagent-model-fallback"; +import { classifyError, inferHttpStatusFromAdapterMessage, isRateLimitOrQuotaFailureMessage } from "../../src/lib/errors"; +import { readAnthropicExtraUsageRefusal } from "../../src/providers/anthropic-billing-refusal"; +import { consumeComboFailure } from "../../src/server/responses/core-combo-failure"; +import { normalizeUpstreamErrorText } from "../../src/server/responses/core-errors"; +import type { OcxConfig } from "../../src/types"; + +// The recorded third-party refusal (fixtures/external/anthropic/README.md): a gateway log line cut +// at 200 chars. The bytes after the log prefix are Anthropic's raw envelope; only the closing quote +// and braces the cut removed are added. +const RECORDED_THIRD_PARTY_LINE = + "Provider error 400: {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"Third-party apps now draw from your extra usage, not your plan limits. Add more at claude.ai/settings/usage and k"; +const RECORDED_THIRD_PARTY_BODY = `${RECORDED_THIRD_PARTY_LINE.slice("Provider error 400: ".length)}"}}`; + +/** + * NO REAL ECHO SAMPLE: no recorded Anthropic validation 400 exists in fixtures/external/ (or + * anywhere in the workspace) yet. Until one is logged, the echo payload is the RECORDED envelope + * above with only `error.message` replaced by the client-echoing validation text. Envelope keys, + * `type` and `error.type` stay exactly as recorded. + */ +function recordedEnvelopeWithMessage(message: string): string { + const recorded = JSON.parse(RECORDED_THIRD_PARTY_BODY) as { error: Record }; + return JSON.stringify({ ...recorded, error: { ...recorded.error, message } }); +} + +// Client-chosen values echoed back: a tool_use id, and user text quoted in a content error. +const ECHOED_TOOL_ID = recordedEnvelopeWithMessage( + "messages.1: `tool_use` ids were found without `tool_result` blocks immediately after: toolu_insufficient_quota. " + + "Each `tool_use` block must have a corresponding `tool_result` block in the next message.", +); +const ECHOED_CONTENT = recordedEnvelopeWithMessage( + "messages.3.content.0.text: unexpected value: You're out of extra usage. Add more at claude.ai/settings/usage and keep going.", +); +const ECHOED_THIRD_PARTY = recordedEnvelopeWithMessage( + "messages.3.content.0.text: Third-party apps now draw from your extra usage, not your plan limits. rate limit exceeded", +); + +function config(): OcxConfig { + return { + port: 0, + defaultProvider: "anthropic", + providers: { + anthropic: { adapter: "anthropic", baseUrl: "https://api.anthropic.com", apiKey: "ka", models: ["opus", "sonnet"] }, + b: { adapter: "openai-chat", baseUrl: "https://b.example/v1", apiKey: "kb", models: ["m2"] }, + }, + combos: { + free: { + strategy: "failover", + targets: [ + { provider: "anthropic", model: "opus" }, + { provider: "anthropic", model: "sonnet" }, + { provider: "b", model: "m2" }, + ], + }, + }, + } as OcxConfig; +} + +/** The combo's handling of one adapter 400, as core-combo runs it. */ +async function comboOn400(upstreamBody: string, now: number) { + // adapter-dispatch: the client error for an upstream 400. + const normalized = normalizeUpstreamErrorText(upstreamBody, "unknown error"); + return comboOnChild(formatErrorResponse(400, "upstream_error", `Provider error 400: ${normalized.safeText}`), now); +} + +/** The combo's handling of an adapter terminal error event that carries only its message text. */ +async function comboOnAdapterMessage(message: string, now: number) { + const { httpStatus, error } = adapterFailureFromEvent({ type: "error", message }); + return comboOnChild(formatErrorResponse(httpStatus, error.type, error.message, { code: error.code }), now); +} + +async function comboOnChild(child: Response, now: number) { + const childStatus = child.status; + const childError = (await child.clone().json() as { error: { type: string; code: string | null } }).error; + // core-combo: read the child failure, decide, and cool only when it hops. + const failure = await consumeComboFailure(child, undefined, now); + const decision = comboFailureDecision(failure.response.status, failure.classificationText, { code: failure.upstreamCode }); + const scope = comboFailureCooldownScope(failure.response.status, failure.classificationText, { code: failure.upstreamCode }); + const cfg = config(); + const pick = pickComboTarget(cfg, "free", { now })!; + expect(pick.target).toMatchObject({ provider: "anthropic", model: "opus" }); + if (decision === "hop") { + advanceComboAfterFailure(cfg, pick, { + now, + cooldownScope: scope, + retryAfter: failure.retryAfter, + status: failure.response.status, + code: failure.upstreamCode, + message: failure.classificationText, + }); + } + const cooled = (model: string, at: number) => isComboTargetInCooldown("free", { provider: "anthropic", model }, at); + return { childStatus, childError, decision, scope, cooled }; +} + +beforeEach(() => clearComboTargetCooldowns()); +afterEach(() => clearComboTargetCooldowns()); + +describe("HTTP 400 quota classification is anchored to the provider's fields", () => { + test("the recorded third-party refusal is still insufficient_quota, raw and wrapped", () => { + for (const message of [RECORDED_THIRD_PARTY_BODY, `Provider error 400: ${RECORDED_THIRD_PARTY_BODY}`]) { + expect(classifyError(400, "upstream_error", message)).toMatchObject({ type: "insufficient_quota", code: "insufficient_quota" }); + } + // The cut log line itself is not a whole envelope: truncated JSON is never quota. + expect(classifyError(400, "upstream_error", RECORDED_THIRD_PARTY_LINE).code).toBe("invalid_request_error"); + }); + + test("MUST-FAIL: an echoed client value with quota or rate words is a plain invalid request", () => { + for (const body of [ECHOED_TOOL_ID, ECHOED_CONTENT, ECHOED_THIRD_PARTY]) { + for (const message of [body, `Provider error 400: ${body}`]) { + expect(classifyError(400, "upstream_error", message)).toMatchObject({ + type: "invalid_request_error", + code: "invalid_request_error", + }); + } + } + }); + + test("OpenAI/Codex insufficient_quota counts as a field, not as words in the message", () => { + const field = JSON.stringify({ error: { message: "You exceeded your current quota.", type: "insufficient_quota", code: "insufficient_quota" } }); + expect(classifyError(400, "upstream_error", `Provider error 400: ${field}`).code).toBe("insufficient_quota"); + const echoed = JSON.stringify({ + error: { message: "Invalid 'tools[0].name': 'insufficient_quota' exceeded your current quota", type: "invalid_request_error", code: "invalid_value" }, + }); + expect(classifyError(400, "upstream_error", `Provider error 400: ${echoed}`).code).toBe("invalid_request_error"); + }); + + test("429 keeps its substring classification", () => { + expect(classifyError(429, "upstream_error", `Provider error 429: ${ECHOED_TOOL_ID}`).code).toBe("insufficient_quota"); + expect(classifyError(429, "upstream_error", "Rate limit reached for model").code).toBe("rate_limit_exceeded"); + }); +}); + +describe("combo cooldown on an Anthropic 400", () => { + for (const [name, body] of [ + ["an echoed toolu_insufficient_quota tool id", ECHOED_TOOL_ID], + ["echoed 'out of extra usage' user content", ECHOED_CONTENT], + ["echoed third-party wording and rate words", ECHOED_THIRD_PARTY], + ] as const) { + test(`MUST-FAIL: ${name} stops and records no combo cooldown`, async () => { + const now = 2_000_000; + const result = await comboOn400(body, now); + expect(result.childError).toMatchObject({ type: "invalid_request_error", code: "invalid_request_error" }); + expect(result.decision).toBe("stop"); + expect(result.scope).not.toBe("provider"); + expect(result.cooled("opus", now + 1)).toBe(false); + expect(result.cooled("sonnet", now + 1)).toBe(false); + }); + } +}); + +// The web-search/images loops and native chat report an upstream failure as adapter text, +// `Provider error : `, and the status is inferred back from that text. +const ECHOED_RATE_WORDS = recordedEnvelopeWithMessage( + "messages.2.content.0.text: unexpected value: Rate limit reached, too many requests, throttling", +); + +describe("an explicit Provider error status is not re-inferred from echoed words", () => { + test("MUST-FAIL: a 400 echoing rate-limit words stays 400", () => { + for (const message of [ + `Provider error 400: ${ECHOED_RATE_WORDS}`, + // formatAnthropicErrorBody's `type: message` summary, as the loops write it. + "Provider error 400: invalid_request_error: messages.2.content.0.text: rate limit, too many requests", + ]) { + expect(inferHttpStatusFromAdapterMessage(message)).toBe(400); + } + }); + + test("an explicit 429 stays 429, and an explicit bare status is kept", () => { + expect(inferHttpStatusFromAdapterMessage("Provider error 429: Number of request tokens has exceeded your rate limit")).toBe(429); + expect(inferHttpStatusFromAdapterMessage("Provider error 429")).toBe(429); + expect(inferHttpStatusFromAdapterMessage("Provider error 503: invalid upstream answer")).toBe(503); + }); + + test("a message without the prefix keeps keyword inference", () => { + expect(inferHttpStatusFromAdapterMessage("Rate limit reached for requests")).toBe(429); + expect(inferHttpStatusFromAdapterMessage("too many requests, invalid model")).toBe(429); + expect(inferHttpStatusFromAdapterMessage("model not found")).toBe(400); + expect(inferHttpStatusFromAdapterMessage("upstream request timed out")).toBe(504); + // Not this proxy's exact prefix: inferred as before. + expect(inferHttpStatusFromAdapterMessage("provider error 400: rate limit")).toBe(429); + expect(inferHttpStatusFromAdapterMessage("Upstream said Provider error 400: rate limit")).toBe(429); + }); + + test("MUST-FAIL: the echoing 400 adapter error stops the combo with no cooldown", async () => { + const now = 3_000_000; + const result = await comboOnAdapterMessage(`Provider error 400: ${ECHOED_RATE_WORDS}`, now); + expect(result.childStatus).toBe(400); + expect(result.childError.code).toBe("invalid_request_error"); + expect(result.decision).toBe("stop"); + expect(result.cooled("opus", now + 1)).toBe(false); + expect(result.cooled("sonnet", now + 1)).toBe(false); + }); + + test("an explicit 429 adapter error still hops and cools its target", async () => { + const now = 4_000_000; + const result = await comboOnAdapterMessage("Provider error 429: Rate limit reached for requests", now); + expect(result.childStatus).toBe(429); + expect(result.childError.code).toBe("rate_limit_exceeded"); + expect(result.decision).toBe("hop"); + expect(result.cooled("opus", now + 1)).toBe(true); + }); +}); + +// adapter-dispatch reports a failed thread-spawn turn to subagent model health as +// `Provider error : ` (a 429 or 402 as the bare number). +describe("subagent model health reads an explicit Provider error status", () => { + const SPAWN = new Headers({ "x-openai-subagent": "collab_spawn" }); + const MODEL = "kimi/k3"; + const subagentConfig = (): OcxConfig => ({ + port: 0, + defaultProvider: "kimi", + providers: { kimi: { adapter: "openai-chat", apiKey: "test", baseUrl: "https://example.invalid", models: ["k3"] } }, + } as OcxConfig); + const blockedAfter = (message: string | number): boolean => { + resetSubagentModelFallbackStateForTests(); + const now = 5_000_000; + recordSubagentQuotaFailureForThreadSpawn(SPAWN, MODEL, message, subagentConfig(), undefined, now); + return isModelHealthBlocked(MODEL, subagentConfig(), undefined, now + 1); + }; + afterEach(() => resetSubagentModelFallbackStateForTests()); + + const ECHOED_USAGE_LIMIT = recordedEnvelopeWithMessage( + "messages.0.content.0.text: unexpected value: usage limit reached, rate limit, too many requests", + ); + + test("MUST-FAIL: an echoed 400 does not mark the model unhealthy", () => { + for (const body of [ECHOED_RATE_WORDS, ECHOED_TOOL_ID, ECHOED_CONTENT, ECHOED_USAGE_LIMIT]) { + const message = `Provider error 400: ${body}`; + expect(isRateLimitOrQuotaFailureMessage(message)).toBe(false); + expect(blockedAfter(message)).toBe(false); + } + }); + + test("the recorded extra-usage 400 and an explicit 429 still mark it unhealthy", () => { + expect(blockedAfter(`Provider error 400: ${RECORDED_THIRD_PARTY_BODY}`)).toBe(true); + expect(blockedAfter("Provider error 429: Rate limit reached for requests")).toBe(true); + expect(blockedAfter("Provider error 429")).toBe(true); + expect(blockedAfter(429)).toBe(true); + }); + + test("messages without the prefix behave as before", () => { + expect(isRateLimitOrQuotaFailureMessage("Weekly usage limit reached")).toBe(true); + expect(isRateLimitOrQuotaFailureMessage("Too Many Requests")).toBe(true); + expect(isRateLimitOrQuotaFailureMessage(`messages.0: toolu_insufficient_quota`)).toBe(true); + expect(isRateLimitOrQuotaFailureMessage("connection refused")).toBe(false); + // A 5xx keeps the usage-limit cue: Codex reports a spent window as 502. + expect(isRateLimitOrQuotaFailureMessage("Provider error 502: usage limit reached")).toBe(true); + }); +}); + +// Anthropic's spend-limit 400 (https://platform.claude.com/docs/en/api/errors and +// https://platform.claude.com/docs/en/api/rate-limits#setting-your-own-spend-limit): HTTP 400 +// `invalid_request_error` whose message STARTS with "You have reached your specified API usage +// limits" (or "...specified workspace API usage limits"). NO RECORDED SAMPLE: the body is the +// recorded envelope above with only `error.message` replaced by the documented opening. +const SPEND_LIMIT = recordedEnvelopeWithMessage( + "You have reached your specified API usage limits. You will regain access on 2026-12-01 at 00:00 UTC.", +); +const WORKSPACE_SPEND_LIMIT = recordedEnvelopeWithMessage( + "You have reached your specified workspace API usage limits. You will regain access on 2026-12-01 at 00:00 UTC.", +); +const ECHOED_SPEND_LIMIT = recordedEnvelopeWithMessage( + "messages.0.content.0.text: unexpected value: You have reached your specified API usage limits", +); + +describe("Anthropic spend-limit 400", () => { + test("is insufficient_quota", () => { + // The fork's combo-hop half of this case is decided by upstream failover; see the ledger. + for (const body of [SPEND_LIMIT, WORKSPACE_SPEND_LIMIT]) { + expect(classifyError(400, "upstream_error", `Provider error 400: ${body}`).code).toBe("insufficient_quota"); + } + }); + + test("the pool rotates on it too, through the same matcher", async () => { + const refusal = (body: string) => readAnthropicExtraUsageRefusal( + new Response(body, { status: 400, headers: { "content-type": "application/json" } })); + expect(await refusal(SPEND_LIMIT)).toBe(true); + expect(await refusal(WORKSPACE_SPEND_LIMIT)).toBe(true); + expect(await refusal(ECHOED_SPEND_LIMIT)).toBe(false); + }); + + test("echoed anywhere but the start it is a plain invalid request", () => { + expect(classifyError(400, "upstream_error", `Provider error 400: ${ECHOED_SPEND_LIMIT}`).code).toBe("invalid_request_error"); + }); +}); + +// Volcengine Ark (https://www.volcengine.com/docs/82379/1299023): HTTP 400, `error.code` +// "InvalidSubscription", "Coding Plan not subscribed or expired". NO RECORDED SAMPLE: code, type +// and message are the documented table row, in Ark's OpenAI-style `error` object. +const VOLCENGINE_INVALID_SUBSCRIPTION = JSON.stringify({ + error: { + code: "InvalidSubscription", + message: "Your account (2100000000) does not have a valid coding plan subscription, or your subscription has expired.", + param: "", + type: "Forbidden", + }, +}); + +describe("Volcengine InvalidSubscription 400", () => { + test("the error.code field is insufficient_quota", () => { + expect(classifyError(400, "upstream_error", `Provider error 400: ${VOLCENGINE_INVALID_SUBSCRIPTION}`).code).toBe("insufficient_quota"); + }); + + test("guard: the word in a message, or another field, is not", () => { + for (const body of [ + JSON.stringify({ error: { code: "InvalidParameter", message: "InvalidSubscription", param: "model", type: "BadRequest" } }), + JSON.stringify({ error: { code: "InvalidParameter", message: "x", param: "InvalidSubscription", type: "BadRequest" } }), + JSON.stringify({ code: "InvalidSubscription", error: { code: "InvalidParameter", message: "x" } }), + ]) { + expect(classifyError(400, "upstream_error", `Provider error 400: ${body}`).code).toBe("invalid_request_error"); + } + }); +}); + +describe("HTTP 422 gets the same field-only rule as 400", () => { + const echoed = recordedEnvelopeWithMessage( + "tools.0.name: unexpected value: insufficient_quota, rate limit, too many requests, usage limit reached", + ); + + test("MUST-FAIL: words in a 422 are no quota or rate verdict", () => { + for (const message of [echoed, `Provider error 422: ${echoed}`]) { + const classified = classifyError(422, "upstream_error", message); + expect(classified.code).not.toBe("insufficient_quota"); + expect(classified.code).not.toBe("rate_limit_exceeded"); + } + expect(isRateLimitOrQuotaFailureMessage(`Provider error 422: ${echoed}`)).toBe(false); + expect(inferHttpStatusFromAdapterMessage(`Provider error 422: ${echoed}`)).toBe(422); + }); + + test("MUST-FAIL: an echoing 422 stops the combo with no cooldown", async () => { + const now = 7_000_000; + const result = await comboOnChild( + formatErrorResponse(422, "upstream_error", `Provider error 422: ${echoed}`), + now, + ); + expect(result.decision).toBe("stop"); + expect(result.cooled("opus", now + 1)).toBe(false); + expect(result.cooled("sonnet", now + 1)).toBe(false); + }); + + test("a 422 field verdict and the existing non-quota 422 behaviour stay", () => { + const field = JSON.stringify({ error: { message: "quota", type: "insufficient_quota", code: "insufficient_quota" } }); + expect(classifyError(422, "upstream_error", `Provider error 422: ${field}`).code).toBe("insufficient_quota"); + expect(comboFailureDecision(422, "invalid_api_key")).toBe("hop"); + expect(comboFailureDecision(422, "unprocessable entity")).toBe("stop"); + }); +}); + +describe("the free-tier prompt cap on a 400 or 422 counts only as a field", () => { + const echoed = recordedEnvelopeWithMessage( + "messages.0.content.0.text: unexpected value: err_free_prompt_cap, longer than the free tier allows for a single request", + ); + + test("MUST-FAIL: echoed err_free_prompt_cap / free-tier words do not hop or cool the provider", () => { + for (const status of [400, 422]) { + for (const message of [echoed, `Provider error ${status}: ${echoed}`]) { + expect(comboFailureDecision(status, message)).toBe("stop"); + expect(comboFailureCooldownScope(status, message)).not.toBe("provider"); + } + } + }); + +}); + +// .27 combo cases, kept on AgentRadar hosted gateways (OPENCODEX_AGENTRADAR=1): the anchored classifier's quota +// verdict is combo evidence there, and the free-tier cap counts on a 400/422 as a field. Echo negatives hold too. +describe("AgentRadar hosted: .27 combo hops on field-anchored 400 quota verdicts", () => { + const previousHosted = process.env.OPENCODEX_AGENTRADAR; + beforeEach(() => { process.env.OPENCODEX_AGENTRADAR = "1"; }); + afterEach(() => { + if (previousHosted === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previousHosted; + }); + + test("the recorded refusal hops and cools the whole provider for ten minutes", async () => { + const now = 1_000_000; + const result = await comboOn400(RECORDED_THIRD_PARTY_BODY, now); + expect(result.childError).toMatchObject({ type: "insufficient_quota", code: "insufficient_quota" }); + expect(result.decision).toBe("hop"); + expect(result.scope).toBe("provider"); + expect(result.cooled("opus", now + 1)).toBe(true); + expect(result.cooled("sonnet", now + 10 * 60_000 - 1)).toBe(true); + expect(isComboTargetInCooldown("free", { provider: "b", model: "m2" }, now + 1)).toBe(false); + }); + + test("Anthropic spend-limit 400 is insufficient_quota and cools the provider like the extra-usage refusal", async () => { + for (const body of [SPEND_LIMIT, WORKSPACE_SPEND_LIMIT]) { + expect(classifyError(400, "upstream_error", `Provider error 400: ${body}`).code).toBe("insufficient_quota"); + clearComboTargetCooldowns(); + const now = 6_000_000; + const result = await comboOn400(body, now); + expect(result.decision).toBe("hop"); + expect(result.scope).toBe("provider"); + expect(result.cooled("sonnet", now + 1)).toBe(true); + } + }); + + test("MUST-FAIL: unanchored adapter prose still mints no quota evidence", async () => { + for (const status of [400, 422]) { + const failure = await consumeComboFailure(formatErrorResponse(status, "upstream_error", "Invalid value: quota exhausted")); + expect(failure.upstreamCode).not.toBe("insufficient_quota"); + expect(comboFailureDecision(status, failure.classificationText, { code: failure.upstreamCode })).toBe("stop"); + } + }); + + test("MUST-FAIL: echoed quota words still stop with no cooldown", async () => { + for (const body of [ECHOED_TOOL_ID, ECHOED_CONTENT, ECHOED_THIRD_PARTY, ECHOED_SPEND_LIMIT]) { + clearComboTargetCooldowns(); + const result = await comboOn400(body, 7_000_000); + expect(result.decision).toBe("stop"); + expect(result.cooled("opus", 7_000_001)).toBe(false); + } + }); + + test("the free-tier field still hops; echoed words do not; other statuses keep the word match", () => { + const echoed = recordedEnvelopeWithMessage( + "messages.0.content.0.text: unexpected value: err_free_prompt_cap, longer than the free tier allows for a single request", + ); + for (const status of [400, 422]) { + for (const message of [echoed, `Provider error ${status}: ${echoed}`]) { + expect(comboFailureDecision(status, message)).toBe("stop"); + expect(comboFailureCooldownScope(status, message)).not.toBe("provider"); + } + } + // The shape the combo end-to-end test sends for the free-tier refusal (`metadata.reason`). + const field = JSON.stringify({ error: { + message: "This prompt is longer than the free tier allows for a single request.", + type: "invalid_request_error", + metadata: { reason: "err_free_prompt_cap" }, + } }); + expect(comboFailureDecision(400, field)).toBe("hop"); + expect(comboFailureCooldownScope(400, field)).toBe("none"); + expect(comboFailureCooldownScope(422, JSON.stringify({ error: { code: "err_free_prompt_cap", message: "x" } }))).toBe("provider"); + expect(comboFailureCooldownScope(429, "err_free_prompt_cap")).toBe("provider"); + expect(comboFailureDecision(413, JSON.stringify({ detail: "err_free_prompt_cap: prompt exceeds this tier" }))).toBe("hop"); + }); +}); diff --git a/tests/routing/routing-analytics-physical.test.ts b/tests/routing/routing-analytics-physical.test.ts new file mode 100644 index 00000000000..255433b4911 --- /dev/null +++ b/tests/routing/routing-analytics-physical.test.ts @@ -0,0 +1,1239 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { appendFileSync, mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { handleManagementAPI } from "../../src/server/management-api"; +import { ManagementRequest } from "../helpers/management-auth"; +import { + appendUsageEntry, + resetUsageReadCacheForTests, + usageLogPath, + type PersistedUsageAttempt, + type PersistedUsageEntry, +} from "../../src/usage/log"; +import { + closeRequestHistoryIndex, + openRequestHistoryIndex, + requestHistoryDb, +} from "../../src/routing/history/indexer"; +import { computeRoutingAnalytics } from "../../src/routing/analytics"; +import { addFinalRequestLog, beginRequestAttempt, clearRequestLogsForTests, noteAttemptSend } from "../../src/server/request-log"; +import { estimateComboCost } from "../../src/usage/cost"; +import type { OcxConfig } from "../../src/types"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +let testDir = ""; +let previousHome: string | undefined; + +function entry( + requestId: string, + overrides: Partial & { timestamp: number; status: number; durationMs: number }, +): PersistedUsageEntry { + return { + requestId, + provider: "a", + model: "m1", + usageStatus: "reported", + ...overrides, + }; +} + +function attempt(overrides: Partial = {}): PersistedUsageAttempt { + return { + ordinal: 1, + provider: "a", + model: "m1", + adapter: "openai-chat", + status: 200, + durationMs: 10, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported", + ...overrides, + }; +} + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + testDir = mkdtempSync(join(tmpdir(), "ocx-analytics-")); + process.env.OPENCODEX_HOME = testDir; + resetUsageReadCacheForTests(); + closeRequestHistoryIndex(); +}); + +afterEach(() => { + closeRequestHistoryIndex(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + if (testDir) removeTreeWithRetry(testDir); +}); + +function config(): OcxConfig { + return { + port: 10100, + defaultProvider: "a", + providers: { a: { adapter: "openai-chat", baseUrl: "https://a.example/v1", apiKey: "ka", models: ["m1"] } }, + }; +} + +describe("routing analytics physical attempts and red alerts", () => { + test("preserves indexed legacy fallback metadata when row JSON has no attempts", async () => { + appendUsageEntry(entry("legacy", { timestamp: 1, status: 200, durationMs: 100 })); + await openRequestHistoryIndex(); + requestHistoryDb().query( + "UPDATE requests SET attempt_count = 3, fallback = 1 WHERE request_id = ?", + ).run("legacy"); + + const result = await computeRoutingAnalytics({}); + expect(result.fallbackRate).toBe(1); + expect(result.totalAttempts).toBe(3); + expect(result.averageAttemptsPerRequest).toBe(3); + expect(result.physicalAttempts).toBe(1); + }); + + test("prices combo requests from their physical attempts", async () => { + const attempts = [ + attempt({ + provider: "openai-apikey", + model: "gpt-5.6-sol", + usage: { inputTokens: 1_000, outputTokens: 100 }, + }), + attempt({ + ordinal: 2, + provider: "openai-apikey", + model: "gpt-5.6-sol", + usage: { inputTokens: 2_000, outputTokens: 200 }, + }), + ]; + const expected = estimateComboCost(attempts); + expect(expected).not.toBeNull(); + appendUsageEntry(entry("combo-cost", { + timestamp: 1, + provider: "combo", + model: "policy/fast", + status: 200, + durationMs: 100, + attempts, + })); + + const result = await computeRoutingAnalytics({}); + expect(result.estimatedCostUsdTotalSuccessful).toBeCloseTo(expected!.cost.total); + expect(result.estimatedCostUsdPerSuccessfulRequest).toBeCloseTo(expected!.cost.total); + expect(result.priceCoverage).toBe(1); + }); + + test("prices legacy root usage on its final physical attempt", async () => { + const usage = { inputTokens: 1_000, outputTokens: 100 }; + const physical = attempt({ + provider: "openai-apikey", + model: "gpt-5.6-sol", + usageStatus: "unreported", + }); + const expected = estimateComboCost([{ + ...physical, + usage, + usageStatus: "reported", + }]); + expect(expected).not.toBeNull(); + appendUsageEntry(entry("legacy-root-cost", { + timestamp: 1, + provider: "combo", + model: "policy/fast", + status: 200, + durationMs: 100, + usage, + attempts: [physical], + })); + + const result = await computeRoutingAnalytics({}); + expect(result.estimatedCostUsdTotalSuccessful).toBeCloseTo(expected!.cost.total); + expect(result.estimatedCostUsdPerSuccessfulRequest).toBeCloseTo(expected!.cost.total); + expect(result.priceCoverage).toBe(1); + }); + + test("combo pricing ignores duplicate, local, and unsent attempts", async () => { + const physical = attempt({ + provider: "openai-apikey", + model: "gpt-5.6-sol", + usage: { inputTokens: 1_000, outputTokens: 100 }, + }); + const expected = estimateComboCost([physical]); + expect(expected).not.toBeNull(); + appendUsageEntry(entry("combo-physical-cost", { + timestamp: 1, + provider: "combo", + model: "policy/fast", + status: 200, + durationMs: 100, + attempts: [ + physical, + { ...physical }, + attempt({ + ordinal: 2, + provider: "openai-apikey", + model: "gpt-5.6-sol", + locallyAnswered: true, + usage: { inputTokens: 10_000, outputTokens: 1_000 }, + }), + attempt({ + ordinal: 3, + provider: "openai-apikey", + model: "gpt-5.6-sol", + sendCount: 0, + usage: { inputTokens: 20_000, outputTokens: 2_000 }, + }), + ], + })); + + const result = await computeRoutingAnalytics({}); + expect(result.estimatedCostUsdTotalSuccessful).toBeCloseTo(expected!.cost.total); + expect(result.estimatedCostUsdPerSuccessfulRequest).toBeCloseTo(expected!.cost.total); + expect(result.priceCoverage).toBe(1); + }); + + test("expands valid physical attempts and reports quota telemetry without root double-counting", async () => { + appendUsageEntry(entry("r1", { + timestamp: 0, + provider: "root-a", + model: "root-model", + status: 200, + durationMs: 100, + attempts: [ + attempt({ + provider: "p1", + model: "m1", + usage: { inputTokens: 100, outputTokens: 1, cacheReadInputTokens: 60, cacheCreationInputTokens: 10 }, + }), + attempt({ + ordinal: 2, + provider: "p2", + model: "m2", + sendCount: 2, + recoveryKinds: ["connection-reset", "transient-5xx"], + usageStatus: "estimated", + usage: { inputTokens: 200, outputTokens: 1, cachedInputTokens: 150, cacheCreationInputTokens: 50, estimated: true }, + }), + ], + })); + const rawRows = [ + { + ...entry("r2", { timestamp: 3_600_000, provider: "root-b", model: "root-model", status: 200, durationMs: 100 }), + attempts: [ + attempt({ provider: "p3", model: "m3", usageStatus: "unsupported" }), + { ordinal: 2, provider: "bad", model: "bad" }, + ], + }, + { + ...entry("r3", { + timestamp: 7_200_000, + provider: "root-c", + model: "root-model", + status: 200, + durationMs: 100, + usage: { inputTokens: 50, outputTokens: 1, cachedInputTokens: 20 }, + }), + attempts: { ordinal: 1, provider: "bad", model: "bad" }, + }, + entry("r4", { + timestamp: 10_800_000, + provider: "root-d", + model: "root-model", + status: 200, + durationMs: 100, + usageStatus: "unreported", + }), + ]; + appendFileSync(usageLogPath(), `${rawRows.map(row => JSON.stringify(row)).join("\n")}\n`); + + const result = await computeRoutingAnalytics({}); + expect(result.totalRequests).toBe(4); + expect(result.totalAttempts).toBe(5); + expect(result.physicalAttempts).toBe(5); + expect(result.physicalSends).toBe(6); + expect(result.repeatedSendAttempts).toBe(1); + expect(result.recoveryAttempts).toBe(1); + expect(result.recoveryEvents).toBe(2); + expect(result.recoveryRate).toBe(0.2); + expect(result.fallbackRate).toBe(0.25); + expect(result.comboFailoverRequests).toBe(0); + expect(result.comboFailoverRate).toBe(0); + expect(result.requestRatePerHour).toBeCloseTo(4 / 3); + expect(result.attemptUsage).toEqual({ + inclusiveInputTokens: 350, + rawInputTokens: 60, + cacheReadInputTokens: 230, + cacheWriteInputTokens: 60, + rawInputShare: 60 / 350, + cacheReadShare: 230 / 350, + cacheWriteShare: 60 / 350, + }); + expect(result.physicalUsageCoverage).toEqual({ + totalAttempts: 5, + measuredAttempts: 3, + reportedAttempts: 2, + estimatedAttempts: 1, + unreportedAttempts: 1, + unsupportedAttempts: 1, + ratio: 0.6, + supportedRatio: 0.75, + }); + expect(result.usageCoverage).toBe(0.6); + expect(result.physicalBreakdown.map(row => `${row.provider}/${row.model}`).sort()).toEqual([ + "p1/m1", "p2/m2", "p3/m3", "root-c/root-model", "root-d/root-model", + ]); + expect(result.physicalBreakdown.find(row => row.provider === "root-b")).toBeUndefined(); + expect(result.physicalBreakdown.find(row => row.provider === "root-c")?.attemptUsage) + .toMatchObject({ inclusiveInputTokens: 50, cacheReadInputTokens: 20 }); + expect(result.physicalBreakdown.find(row => row.provider === "p1")).toMatchObject({ + comboFailoverRequests: 0, + comboFailoverRate: 0, + }); + + const rootFiltered = await computeRoutingAnalytics({ provider: "root-a" }); + expect(rootFiltered.totalRequests).toBe(1); + expect(rootFiltered.physicalBreakdown.map(row => row.provider).sort()).toEqual(["p1", "p2"]); + }); + + test("uses one requested window for comparable overall and route request rates", async () => { + appendUsageEntry(entry("r1", { timestamp: 1, status: 200, durationMs: 10 })); + appendUsageEntry(entry("r2", { timestamp: 1_001, status: 200, durationMs: 10 })); + + const result = await computeRoutingAnalytics({ from: 0, to: 30 * 24 * 3_600_000 }); + expect(result.requestRatePerHour).toBeCloseTo(2 / (30 * 24)); + expect(result.physicalBreakdown[0]?.requestRatePerHour).toBeCloseTo(2 / (30 * 24)); + }); + + test("does not add aggregate root usage when any attempt reports valid usage", async () => { + appendUsageEntry(entry("root-fallback", { + timestamp: 1, + status: 200, + durationMs: 10, + usage: { + inputTokens: 250, + outputTokens: 1, + cachedInputTokens: 10, + cacheReadInputTokens: 200, + cacheCreationInputTokens: 20, + }, + attempts: [ + attempt({ + accountLogLabel: "oaaaaaa", + usage: { inputTokens: 100, outputTokens: 1, cachedInputTokens: 60 }, + }), + attempt({ + ordinal: 2, + accountLogLabel: "obbbbbb", + usageStatus: "unreported", + }), + ], + })); + + const result = await computeRoutingAnalytics({}); + expect(result.attemptUsage).toEqual({ + inclusiveInputTokens: 100, + rawInputTokens: 40, + cacheReadInputTokens: 60, + cacheWriteInputTokens: 0, + rawInputShare: 0.4, + cacheReadShare: 0.6, + cacheWriteShare: 0, + }); + expect(result.physicalUsageCoverage.measuredAttempts).toBe(1); + expect(result.physicalBreakdown.map(row => row.accountRef).sort()).toEqual([ + "oaaaaaa", + "obbbbbb", + ]); + expect(result.physicalBreakdown.find(row => row.accountRef === "obbbbbb")?.attemptUsage) + .toMatchObject({ inclusiveInputTokens: 0, cacheReadInputTokens: 0 }); + }); + + test("does not attribute root usage when physical attempt usage is present but invalid", async () => { + appendUsageEntry(entry("invalid-attempt-usage", { + timestamp: 1, + status: 200, + durationMs: 10, + usage: { inputTokens: 250, outputTokens: 1, cacheReadInputTokens: 200 }, + attempts: [ + attempt({ + provider: "first", + usage: { + inputTokens: 100, + outputTokens: 1, + cacheReadInputTokens: 80, + cacheCreationInputTokens: 30, + }, + }), + attempt({ + ordinal: 2, + provider: "final", + usage: { + inputTokens: 150, + outputTokens: 1, + cacheReadInputTokens: 120, + cacheCreationInputTokens: 40, + }, + }), + ], + })); + + const result = await computeRoutingAnalytics({}); + expect(result.attemptUsage.inclusiveInputTokens).toBe(0); + expect(result.physicalUsageCoverage).toMatchObject({ + totalAttempts: 2, + measuredAttempts: 0, + unreportedAttempts: 2, + }); + expect(result.physicalBreakdown.find(row => row.provider === "final")?.attemptUsage.inclusiveInputTokens).toBe(0); + }); + + test("attributes a legacy root account to the final unlabeled physical attempt", async () => { + appendUsageEntry(entry("combo-final", { + timestamp: 1, + provider: "combo", + model: "requested-model", + accountLogLabel: "oaaaaaa", + status: 200, + durationMs: 10, + attempts: [ + attempt({ provider: "other", model: "m-other" }), + attempt({ ordinal: 2, provider: "final", model: "m-final" }), + ], + })); + appendUsageEntry(entry("same-route", { + timestamp: 2, + provider: "same", + model: "m-same", + accountLogLabel: "obbbbbb", + status: 200, + durationMs: 10, + attempts: [ + attempt({ provider: "same", model: "m-same" }), + attempt({ ordinal: 2, provider: "same", model: "m-same" }), + ], + })); + + const result = await computeRoutingAnalytics({}); + expect(result.physicalBreakdown.find(row => row.provider === "final")?.accountRef).toBe("oaaaaaa"); + expect(result.physicalBreakdown.find(row => row.provider === "other")?.accountRef).toBeUndefined(); + expect(result.physicalBreakdown.filter(row => row.provider === "same") + .map(row => row.accountRef ?? "none").sort()).toEqual(["none", "obbbbbb"]); + }); + + test("counts recovery occurrences and excludes duplicate and locally answered attempts", async () => { + const physical = attempt({ + sendCount: 4, + recoveryKinds: ["connection-reset"], + recoveryCount: 3, + }); + const row = { + ...entry("attempt-normalization", { timestamp: 1, status: 200, durationMs: 10 }), + attempts: [ + physical, + { ...physical, provider: "duplicate" }, + attempt({ ordinal: 2, provider: "local", sendCount: 0, locallyAnswered: true }), + attempt({ ordinal: 3, provider: "unsent", sendCount: 0, usageStatus: "unreported" }), + ], + }; + appendFileSync(usageLogPath(), `${JSON.stringify(row)}\n`); + + const result = await computeRoutingAnalytics({}); + expect(result.physicalAttempts).toBe(1); + expect(result.physicalSends).toBe(4); + expect(result.recoveryAttempts).toBe(1); + expect(result.recoveryEvents).toBe(3); + expect(result.physicalBreakdown.map(route => route.provider)).toEqual(["a"]); + expect(result.redAlerts.find(alert => alert.kind === "recovery")?.value).toBe(3); + }); + + test("treats explicit all-invalid and all-filtered attempt arrays as zero physical attempts", async () => { + const rows = [ + { + ...entry("invalid-attempt-array", { + timestamp: 1, + status: 200, + durationMs: 10, + usage: { inputTokens: 100, outputTokens: 10, cacheReadInputTokens: 40 }, + }), + attempts: [null, { ordinal: "bad" }], + }, + { + ...entry("filtered-attempt-array", { + timestamp: 2, + status: 200, + durationMs: 10, + usage: { inputTokens: 200, outputTokens: 20, cacheReadInputTokens: 80 }, + }), + attempts: [ + attempt({ locallyAnswered: true }), + attempt({ ordinal: 2, sendCount: 0 }), + ], + }, + ]; + appendFileSync(usageLogPath(), `${rows.map(row => JSON.stringify(row)).join("\n")}\n`); + + const result = await computeRoutingAnalytics({}); + expect(result).toMatchObject({ + totalRequests: 2, + totalAttempts: 0, + physicalAttempts: 0, + attemptUsage: { + inclusiveInputTokens: 0, + rawInputTokens: 0, + cacheReadInputTokens: 0, + cacheWriteInputTokens: 0, + }, + physicalBreakdown: [], + }); + }); + + test("keeps unsupported usage out of measured coverage", async () => { + appendUsageEntry(entry("unsupported", { + timestamp: 1, + status: 200, + durationMs: 10, + attempts: [attempt({ + usageStatus: "unsupported", + usage: { inputTokens: 100, outputTokens: 1 }, + })], + })); + + const result = await computeRoutingAnalytics({}); + expect(result.attemptUsage).toMatchObject({ + inclusiveInputTokens: 0, + rawInputTokens: 0, + cacheReadInputTokens: 0, + cacheWriteInputTokens: 0, + }); + expect(result.physicalBreakdown[0]?.attemptUsage.inclusiveInputTokens).toBe(0); + expect(result.physicalUsageCoverage).toMatchObject({ + totalAttempts: 1, + measuredAttempts: 0, + reportedAttempts: 0, + unsupportedAttempts: 1, + ratio: 0, + supportedRatio: null, + }); + }); + + test("keeps unreported usage out of measured coverage even when counters are present", async () => { + appendUsageEntry(entry("unreported", { + timestamp: 1, + status: 200, + durationMs: 10, + attempts: [attempt({ + usageStatus: "unreported", + usage: { inputTokens: 100, outputTokens: 1 }, + })], + })); + + const result = await computeRoutingAnalytics({}); + expect(result.attemptUsage.inclusiveInputTokens).toBe(0); + expect(result.physicalUsageCoverage).toMatchObject({ + totalAttempts: 1, + measuredAttempts: 0, + reportedAttempts: 0, + unreportedAttempts: 1, + ratio: 0, + }); + }); + + test("recovers combined read and write tokens from legacy cache totals", async () => { + appendUsageEntry(entry("legacy-cache", { + timestamp: 1, + status: 200, + durationMs: 10, + attempts: [attempt({ + usage: { + inputTokens: 100, + outputTokens: 1, + cachedInputTokens: 90, + cacheCreationInputTokens: 20, + }, + })], + })); + + const result = await computeRoutingAnalytics({}); + expect(result.attemptUsage).toMatchObject({ + inclusiveInputTokens: 100, + rawInputTokens: 10, + cacheReadInputTokens: 70, + cacheWriteInputTokens: 20, + }); + expect(result.physicalUsageCoverage.measuredAttempts).toBe(1); + }); + + test("rejects impossible cache decomposition from measured coverage", async () => { + appendUsageEntry(entry("invalid-cache", { + timestamp: 1, + status: 200, + durationMs: 10, + attempts: [attempt({ + usageStatus: "reported", + usage: { + inputTokens: 100, + outputTokens: 1, + cacheReadInputTokens: 80, + cacheCreationInputTokens: 30, + }, + })], + })); + + const result = await computeRoutingAnalytics({}); + expect(result.attemptUsage.inclusiveInputTokens).toBe(0); + expect(result.physicalUsageCoverage).toMatchObject({ + totalAttempts: 1, + measuredAttempts: 0, + reportedAttempts: 0, + unreportedAttempts: 1, + ratio: 0, + }); + }); + + test("classifies only marked structural combo rows as combo failover", async () => { + appendUsageEntry(entry("policy", { + timestamp: 1, + status: 200, + durationMs: 10, + provider: "policy", + attempts: [attempt(), attempt({ ordinal: 2, provider: "b" })], + })); + appendUsageEntry(entry("combo-account-rotation", { + timestamp: 2, + status: 200, + durationMs: 10, + provider: "combo", + attempts: [attempt(), attempt({ ordinal: 2, provider: "b" })], + })); + appendUsageEntry(entry("combo", { + timestamp: 3, + status: 200, + durationMs: 10, + provider: "combo", + comboTargetAdvanced: true, + attempts: [attempt(), attempt({ ordinal: 2, provider: "b" })], + })); + + const result = await computeRoutingAnalytics({}); + expect(result.fallbackRate).toBe(1); + expect(result.comboFailoverRequests).toBe(1); + expect(result.redAlerts.filter(alert => alert.kind === "combo-failover").map(alert => alert.requestId)) + .toEqual(["combo"]); + }); + + test("uses canonical Codex and Claude surface groups", async () => { + appendUsageEntry(entry("codex", { timestamp: 1, status: 200, durationMs: 10 })); + appendUsageEntry(entry("claude", { timestamp: 2, status: 200, durationMs: 10, surface: "claude" })); + appendUsageEntry(entry("desktop", { timestamp: 3, status: 200, durationMs: 10, surface: "claude-desktop" })); + appendUsageEntry(entry("grok", { timestamp: 4, status: 200, durationMs: 10, surface: "grok" })); + + expect((await computeRoutingAnalytics({ surface: "all" })).totalRequests).toBe(4); + expect((await computeRoutingAnalytics({ surface: "codex" })).totalRequests).toBe(1); + expect((await computeRoutingAnalytics({ surface: "claude" })).totalRequests).toBe(2); + expect((await computeRoutingAnalytics({ surface: "grok" })).totalRequests).toBe(1); + }); + + test("routing analytics API uses canonical surface groups", async () => { + appendUsageEntry(entry("codex", { timestamp: 1, status: 200, durationMs: 10 })); + appendUsageEntry(entry("claude", { timestamp: 2, status: 200, durationMs: 10, surface: "claude" })); + appendUsageEntry(entry("desktop", { timestamp: 3, status: 200, durationMs: 10, surface: "claude-desktop" })); + appendUsageEntry(entry("grok", { timestamp: 4, status: 200, durationMs: 10, surface: "grok" })); + + for (const [surface, expected] of [["codex", 1], ["claude", 2], ["grok", 1], ["unknown", 4]] as const) { + const req = new ManagementRequest(`http://localhost/api/routing-analytics?surface=${surface}`, { method: "GET" }); + const response = await handleManagementAPI(req, new URL(req.url), config(), { refreshCodexCatalog: async () => {} }); + expect(response).not.toBeNull(); + expect(response!.status).toBe(200); + const body = await response!.json() as { totalRequests?: number }; + expect(body.totalRequests).toBe(expected); + } + }); + + test("isolates alert sequences by physical account and request fallback", async () => { + const add = ( + requestId: string, + timestamp: number, + accountLogLabel: "oaaaaaa" | "obbbbbb", + conversationId?: string, + ) => appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + ...(conversationId ? { conversationId } : {}), + attempts: [attempt({ + provider: "anthropic-paaaaaa", + model: "claude-opus-5", + accountLogLabel, + usage: { inputTokens: 120_000 + timestamp, outputTokens: 1 }, + })], + })); + add("a-first", 1, "oaaaaaa", "conversation"); + add("b-first", 2, "obbbbbb", "conversation"); + add("a-second", 3, "oaaaaaa", "conversation"); + add("isolated-1", 4, "oaaaaaa"); + add("isolated-2", 5, "oaaaaaa"); + + const result = await computeRoutingAnalytics({}); + expect(result.redAlerts.filter(alert => alert.kind === "consecutive-high-raw-input")) + .toEqual([expect.objectContaining({ requestId: "a-second", accountRef: "oaaaaaa" })]); + expect(result.physicalBreakdown.map(route => route.accountRef).sort()).toEqual([ + "oaaaaaa", + "obbbbbb", + ]); + expect(result.sequentialRoutes).toBe(4); + expect(result.warmedRoutes).toBe(0); + }); + + test("an Anthropic account named by both provider and label is one route, counted once", async () => { + // The ledger names a served Anthropic account twice -- `anthropic-p` and the same + // `p` label -- and the plain active account only through the label. Every shape below + // is account X except the combo's serving attempt (account Y). + const x = "paaaaaa"; + const y = "pbbbbbb"; + const claude = (overrides: Partial) => + attempt({ adapter: "anthropic", model: "claude-opus-5", ...overrides }); + // Written before the label existed: the provider suffix is the only account. + appendUsageEntry(entry("legacy-pooled", { timestamp: 1, status: 200, durationMs: 10, + provider: `anthropic-${x}`, model: "claude-opus-5", + attempts: [claude({ provider: `anthropic-${x}` })] })); + appendUsageEntry(entry("pooled", { timestamp: 2, status: 200, durationMs: 10, + provider: `anthropic-${x}`, model: "claude-opus-5", accountLogLabel: x, + attempts: [claude({ provider: `anthropic-${x}`, accountLogLabel: x })] })); + appendUsageEntry(entry("plain-active", { timestamp: 3, status: 200, durationMs: 10, + provider: "anthropic", model: "claude-opus-5", accountLogLabel: x, + attempts: [claude({ provider: "anthropic", accountLogLabel: x })] })); + // A combo row: the row label is the serving attempt's, and it must not be counted again. + appendUsageEntry(entry("combo", { timestamp: 4, status: 200, durationMs: 10, + provider: "combo", model: "combo/claude", accountLogLabel: y, + attempts: [ + claude({ provider: `anthropic-${x}`, accountLogLabel: x, status: 429 }), + claude({ ordinal: 2, provider: `anthropic-${y}`, accountLogLabel: y }), + ] })); + + const result = await computeRoutingAnalytics({}); + const routes = result.physicalBreakdown + .map(route => ({ provider: route.provider, accountRef: route.accountRef, + requests: route.requests, physicalAttempts: route.physicalAttempts })) + .sort((a, b) => (a.accountRef ?? "").localeCompare(b.accountRef ?? "")); + expect(routes).toEqual([ + { provider: "anthropic", accountRef: x, requests: 4, physicalAttempts: 4 }, + { provider: "anthropic", accountRef: y, requests: 1, physicalAttempts: 1 }, + ]); + // Five physical attempts in, five attributed out: nothing dropped, nothing doubled. + expect(result.physicalBreakdown.reduce((sum, route) => sum + route.physicalAttempts, 0)).toBe(5); + }); + + test("folds the frozen upstream accountTag into the same Anthropic route as the provider suffix", async () => { + const x = "paaaaaa"; + const claude = (overrides: Partial) => + attempt({ adapter: "anthropic", model: "claude-opus-5", ...overrides }); + appendUsageEntry(entry("tagged", { timestamp: 1, status: 200, durationMs: 10, + provider: `anthropic-${x}`, model: "claude-opus-5", accountTag: x, + attempts: [claude({ provider: `anthropic-${x}`, accountTag: x })] })); + appendUsageEntry(entry("suffix-only", { timestamp: 2, status: 200, durationMs: 10, + provider: `anthropic-${x}`, model: "claude-opus-5", + attempts: [claude({ provider: `anthropic-${x}` })] })); + + const result = await computeRoutingAnalytics({}); + expect(result.physicalBreakdown.map(route => ({ provider: route.provider, accountRef: route.accountRef, + requests: route.requests }))).toEqual([{ provider: "anthropic", accountRef: x, requests: 2 }]); + }); + + test("the final request record persists send time, recovery count and combo advance", async () => { + clearRequestLogsForTests(); + const first = beginRequestAttempt(1, "a", "m1", "openai-chat"); + noteAttemptSend(first, undefined); + noteAttemptSend(first, undefined, "connection-reset"); + noteAttemptSend(first, undefined, "connection-reset"); + expect(first.firstSendAt).toBeGreaterThan(0); + addFinalRequestLog("combo-advanced", Date.now(), { + model: "m1", provider: "a", attempts: [first], activeAttempt: first, + comboTargetAdvanced: true, usage: { inputTokens: 5, outputTokens: 1 }, + }, 200, { closeReason: "non_stream" }); + + const result = await computeRoutingAnalytics({}); + expect(result).toMatchObject({ comboFailoverRequests: 1, recoveryAttempts: 1, recoveryEvents: 2, physicalSends: 3 }); + expect(result.redAlerts.map(alert => alert.kind).sort()).toEqual(["combo-failover", "recovery", "repeated-send"]); + clearRequestLogsForTests(); + }); + + test("emits route-sequential red alerts in timestamp and request-id order", async () => { + const addAttemptRow = ( + requestId: string, + timestamp: number, + usage: PersistedUsageAttempt["usage"], + overrides: Partial = {}, + ) => appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + conversationId: "conversation", + attempts: [attempt({ + provider: "anthropic-paaaaaa", + model: "claude-opus-5", + usage, + })], + ...overrides, + })); + + addAttemptRow("later", 2, { inputTokens: 120_000, outputTokens: 1 }); + addAttemptRow("earlier", 1, { inputTokens: 110_000, outputTokens: 1 }); + addAttemptRow("warm", 3, { inputTokens: 1_000, outputTokens: 1, cacheReadInputTokens: 950 }); + addAttemptRow("degrade", 4, { inputTokens: 12_000, outputTokens: 1, cacheReadInputTokens: 9_000, cacheCreationInputTokens: 2_000 }); + addAttemptRow("bad-cache", 5, { inputTokens: 20_000, outputTokens: 1, cacheReadInputTokens: 8_000, cacheCreationInputTokens: 11_000 }); + addAttemptRow("missing-1", 6, { inputTokens: 130_000, outputTokens: 1 }, { conversationId: undefined }); + addAttemptRow("missing-2", 7, { inputTokens: 140_000, outputTokens: 1 }, { conversationId: undefined }); + appendUsageEntry(entry("operational", { + timestamp: 8, + status: 200, + durationMs: 10, + attempts: [attempt({ sendCount: 2, recoveryKinds: ["connection-reset"] })], + })); + + const result = await computeRoutingAnalytics({}); + const alertsFor = (requestId: string) => result.redAlerts + .filter(alert => alert.requestId === requestId) + .map(alert => alert.kind); + expect(alertsFor("later")).toEqual(["consecutive-high-raw-input"]); + expect(alertsFor("bad-cache").sort()).toEqual([ + "falling-cache-read", + "low-cache-read-share-after-warmup", + ]); + expect(alertsFor("missing-2")).toEqual([]); + expect(alertsFor("operational").sort()).toEqual(["recovery", "repeated-send"]); + expect(result.sequentialRoutes).toBe(3); + expect(result.warmedRoutes).toBe(1); + expect(result.redAlertsPartial).toBe(false); + }); + + test("cache creation alone warms an Anthropic route", async () => { + const add = (requestId: string, timestamp: number, usage: PersistedUsageAttempt["usage"]) => + appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + conversationId: "conversation", + attempts: [attempt({ + provider: "anthropic-paaaaaa", + model: "claude-opus-5", + usage, + })], + })); + + add("write-warm", 1, { + inputTokens: 1_000, + outputTokens: 1, + cacheCreationInputTokens: 950, + }); + add("bad-cache", 2, { + inputTokens: 20_000, + outputTokens: 1, + cacheReadInputTokens: 8_000, + cacheCreationInputTokens: 11_000, + }); + + const result = await computeRoutingAnalytics({}); + const alerts = result.redAlerts + .filter(alert => alert.requestId === "bad-cache") + .map(alert => alert.kind) + .sort(); + expect(alerts).toEqual([]); + expect(result.warmedRoutes).toBe(1); + }); + + test("does not mistake a fully reused prefix plus a large tool result for cache loss", async () => { + const add = (requestId: string, timestamp: number, usage: PersistedUsageAttempt["usage"]) => + appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + conversationId: "growing-conversation", + attempts: [attempt({ + provider: "anthropic-paaaaaa", + model: "claude-opus-5", + usage, + })], + })); + + add("warm-prefix", 1, { + inputTokens: 4_539, + outputTokens: 1, + cacheCreationInputTokens: 4_537, + }); + add("large-tool-result", 2, { + inputTokens: 25_657, + outputTokens: 1, + cacheReadInputTokens: 4_537, + cacheCreationInputTokens: 21_118, + }); + + const alerts = (await computeRoutingAnalytics({})).redAlerts + .filter(alert => alert.requestId === "large-tool-result"); + expect(alerts).toEqual([]); + }); + + test("detects a full-prefix rewrite despite legitimate prompt growth", async () => { + const add = (requestId: string, timestamp: number, usage: PersistedUsageAttempt["usage"]) => + appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + conversationId: "rewritten-conversation", + attempts: [attempt({ + provider: "anthropic-paaaaaa", + model: "claude-opus-5", + usage, + })], + })); + + add("initial-write", 1, { + inputTokens: 931_882, + outputTokens: 1, + cacheCreationInputTokens: 931_880, + }); + add("full-rewrite", 2, { + inputTokens: 966_539, + outputTokens: 1, + cacheCreationInputTokens: 966_537, + }); + + const alerts = (await computeRoutingAnalytics({})).redAlerts + .filter(alert => alert.requestId === "full-rewrite") + .map(alert => alert.kind) + .sort(); + expect(alerts).toEqual([ + "high-cache-write-after-warmup", + "low-cache-read-share-after-warmup", + ]); + }); + + test("keeps one caller-side lineage rewrite isolated when its successor reuses cache", async () => { + const add = (requestId: string, timestamp: number, usage: PersistedUsageAttempt["usage"]) => + appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + conversationId: "source-lineage-change", + attempts: [attempt({ + provider: "anthropic-paaaaaa", + model: "claude-opus-5", + usage, + })], + })); + + add("warmed-lineage", 1, { + inputTokens: 28_395, + outputTokens: 1, + cacheReadInputTokens: 27_927, + cacheCreationInputTokens: 466, + }); + add("changed-toolset", 2, { + inputTokens: 28_110, + outputTokens: 1, + cacheCreationInputTokens: 28_108, + }); + add("stable-new-lineage", 3, { + inputTokens: 28_508, + outputTokens: 1, + cacheReadInputTokens: 28_108, + cacheCreationInputTokens: 398, + }); + + const alerts = (await computeRoutingAnalytics({})).redAlerts; + expect(alerts + .filter(alert => alert.requestId === "changed-toolset") + .map(alert => alert.kind) + .sort()).toEqual([ + "falling-cache-read", + "high-cache-write-after-warmup", + "low-cache-read-share-after-warmup", + ]); + expect(alerts.filter(alert => alert.requestId === "stable-new-lineage")).toEqual([]); + }); + + test("keeps interleaved cache lineages independent within one Claude conversation", async () => { + const add = (requestId: string, timestamp: number, usage: PersistedUsageAttempt["usage"]) => + appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + conversationId: "shared-parent-conversation", + attempts: [attempt({ + provider: "anthropic-paaaaaa", + model: "claude-opus-5", + usage, + })], + })); + + add("lineage-a", 1, { + inputTokens: 100_502, + outputTokens: 1, + cacheReadInputTokens: 100_000, + cacheCreationInputTokens: 500, + }); + add("lineage-b", 2, { + inputTokens: 700_002, + outputTokens: 1, + cacheCreationInputTokens: 700_000, + }); + add("lineage-a-resumed", 3, { + inputTokens: 100_402, + outputTokens: 1, + cacheReadInputTokens: 100_000, + cacheCreationInputTokens: 400, + }); + add("lineage-b-resumed", 4, { + inputTokens: 700_502, + outputTokens: 1, + cacheReadInputTokens: 700_000, + cacheCreationInputTokens: 500, + }); + + const alerts = (await computeRoutingAnalytics({})).redAlerts; + expect(alerts.filter(alert => alert.requestId === "lineage-a-resumed")).toEqual([]); + expect(alerts.filter(alert => alert.requestId === "lineage-b-resumed")).toEqual([]); + }); + + test("expires Anthropic warm-up state after the prompt-cache TTL", async () => { + const add = ( + requestId: string, + timestamp: number, + usage: PersistedUsageAttempt["usage"], + conversationId = "conversation", + ) => appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + conversationId, + attempts: [attempt({ + provider: "anthropic-paaaaaa", + model: "claude-opus-5", + usage, + ...({ promptCacheTtlMs: 5 * 60_000 } as Partial), + })], + })); + + add("warm", 1, { + inputTokens: 800_229, + outputTokens: 1, + cacheReadInputTokens: 799_819, + cacheCreationInputTokens: 408, + }); + add("post-idle-rewrite", 568_201, { + inputTokens: 808_955, + outputTokens: 1, + cacheCreationInputTokens: 808_953, + }); + add("reuse-rewritten-prefix", 568_202, { + inputTokens: 810_595, + outputTokens: 1, + cacheReadInputTokens: 808_953, + cacheCreationInputTokens: 1_640, + }); + add("raw-before-idle", 1, { inputTokens: 120_001, outputTokens: 1 }, "raw-conversation"); + add("raw-after-idle", 568_201, { inputTokens: 120_002, outputTokens: 1 }, "raw-conversation"); + + const result = await computeRoutingAnalytics({}); + expect(result.redAlerts).toEqual([]); + expect(result.sequentialRoutes).toBe(2); + expect(result.warmedRoutes).toBe(1); + }); + + test("keeps a one-hour Anthropic route warm beyond the default five-minute TTL", async () => { + const add = ( + requestId: string, + timestamp: number, + usage: PersistedUsageAttempt["usage"], + ) => appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + conversationId: "long-cache-conversation", + attempts: [attempt({ + provider: "anthropic-paaaaaa", + model: "claude-opus-5", + usage, + ...({ promptCacheTtlMs: 60 * 60_000 } as Partial), + })], + })); + + add("warm-long-cache", 1, { + inputTokens: 800_229, + outputTokens: 1, + cacheReadInputTokens: 799_819, + cacheCreationInputTokens: 408, + }); + add("leak-before-long-cache-expiry", 568_201, { + inputTokens: 808_955, + outputTokens: 1, + cacheCreationInputTokens: 808_953, + }); + + const alerts = (await computeRoutingAnalytics({})).redAlerts + .filter(alert => alert.requestId === "leak-before-long-cache-expiry") + .map(alert => alert.kind); + expect(alerts).toContain("high-cache-write-after-warmup"); + }); + + test("orders sequential alerts by first physical send rather than request completion", async () => { + const add = ( + requestId: string, + timestamp: number, + firstSendAt: number, + inputTokens: number, + ) => appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + conversationId: "overlapping-conversation", + attempts: [attempt({ + provider: "anthropic-paaaaaa", + model: "claude-opus-5", + usage: { inputTokens, outputTokens: 1 }, + ...({ firstSendAt } as Partial), + })], + })); + + add("sent-first-finished-last", 1_000, 100, 110_000); + add("sent-second-finished-first", 500, 200, 120_000); + + const alerts = (await computeRoutingAnalytics({})).redAlerts + .filter(alert => alert.kind === "consecutive-high-raw-input"); + expect(alerts).toEqual([ + expect.objectContaining({ requestId: "sent-second-finished-first", previousValue: 110_000 }), + ]); + }); + + test("does not apply Anthropic cache thresholds to OpenAI routes", async () => { + const add = ( + requestId: string, + timestamp: number, + usage: PersistedUsageAttempt["usage"], + overrides: Partial = {}, + ) => appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + conversationId: "conversation", + attempts: [attempt({ provider: "openai", model: "gpt-5.6-terra", usage, ...overrides })], + })); + + add("raw-1", 1, { inputTokens: 110_000, outputTokens: 1 }); + add("raw-2", 2, { inputTokens: 120_000, outputTokens: 1 }); + add("warm", 3, { inputTokens: 10_000, outputTokens: 1, cacheReadInputTokens: 9_500 }); + add("low-read", 4, { + inputTokens: 20_000, + outputTokens: 1, + cacheReadInputTokens: 1_000, + cacheCreationInputTokens: 11_000, + }, { sendCount: 2, recoveryKinds: ["connection-reset"] }); + + const result = await computeRoutingAnalytics({}); + expect(result.redAlerts.map(alert => alert.kind).sort()).toEqual(["recovery", "repeated-send"]); + expect(result.sequentialRoutes).toBe(0); + expect(result.warmedRoutes).toBe(0); + }); + + test("applies Anthropic cache thresholds to custom-named Anthropic adapters", async () => { + for (const [requestId, timestamp] of [["raw-1", 1], ["raw-2", 2]] as const) { + appendUsageEntry(entry(requestId, { + timestamp, + status: 200, + durationMs: 10, + conversationId: "conversation", + attempts: [attempt({ + provider: "custom-name", + model: "claude-opus-5", + adapter: "anthropic", + usage: { inputTokens: 110_000 + timestamp, outputTokens: 1 }, + })], + })); + } + + const result = await computeRoutingAnalytics({}); + expect(result.redAlerts).toEqual([ + expect.objectContaining({ + kind: "consecutive-high-raw-input", + requestId: "raw-2", + provider: "custom-name", + }), + ]); + expect(result.sequentialRoutes).toBe(1); + }); + + test("bounds red alerts and marks capped history partial", async () => { + for (let index = 0; index < 105; index++) { + appendUsageEntry(entry(`r${String(index).padStart(3, "0")}`, { + timestamp: index, + status: 200, + durationMs: 10, + attempts: [attempt({ recoveryKinds: ["connection-reset"] })], + })); + } + + const bounded = await computeRoutingAnalytics({}); + expect(bounded.redAlerts).toHaveLength(100); + expect(bounded.redAlertsPartial).toBe(true); + const capped = await computeRoutingAnalytics({}, { maxRows: 10 }); + expect(capped.historyTruncated).toBe(true); + expect(capped.redAlertsPartial).toBe(true); + }); + + test("ignores malformed attempts while preserving explicit 429 classification", async () => { + appendUsageEntry(entry("baseline", { timestamp: 1, status: 200, durationMs: 10 })); + const historicalRows = [ + { + ...entry("missing-recovery-kinds", { timestamp: 2, status: 503, durationMs: 20 }), + attempts: [{ ordinal: 1 }], + }, + { + ...entry("malformed-attempts", { timestamp: 3, status: 503, durationMs: 30 }), + attempts: { recoveryKinds: ["rate-limit-429"] }, + }, + { + ...entry("malformed-recovery-kinds", { timestamp: 4, status: 503, durationMs: 40 }), + attempts: [null, { recoveryKinds: "rate-limit-429" }, { recoveryKinds: [null, 42, "unknown"] }], + }, + { + ...entry("malformed-429", { timestamp: 5, status: 429, durationMs: 50 }), + attempts: { recoveryKinds: ["unknown"] }, + }, + ]; + appendFileSync(usageLogPath(), `${historicalRows.map(row => JSON.stringify(row)).join("\n")}\n`); + + const result = await computeRoutingAnalytics({}); + expect(result.totalRequests).toBe(5); + expect(result.cooldownTriggeringFailures).toBe(1); + }); + + test("routing analytics API applies usage calendar ranges in the server timezone", async () => { + const start = new Date(); + start.setHours(0, 0, 0, 0); + start.setDate(start.getDate() - 6); + appendUsageEntry(entry("outside", { timestamp: start.getTime() - 1, status: 200, durationMs: 10 })); + appendUsageEntry(entry("inside", { timestamp: start.getTime(), status: 200, durationMs: 10 })); + + const req = new ManagementRequest("http://localhost/api/routing-analytics?range=7d", { method: "GET" }); + const response = await handleManagementAPI(req, new URL(req.url), config(), { refreshCodexCatalog: async () => {} }); + expect(response?.status).toBe(200); + const body = await response!.json() as { totalRequests?: number }; + expect(body.totalRequests).toBe(1); + }); + + test("routing analytics API returns 400 for invalid from/to/range/limit", async () => { + const cases = [ + { query: "from=abc", code: "invalid_from" }, + { query: "to=xyz", code: "invalid_to" }, + { query: "from=10&to=5", code: "invalid_range" }, + { query: "range=quarter", code: "invalid_range" }, + { query: "limit=0", code: "invalid_limit" }, + ] as const; + for (const { query, code } of cases) { + const req = new ManagementRequest(`http://localhost/api/routing-analytics?${query}`, { method: "GET" }); + const response = await handleManagementAPI(req, new URL(req.url), config(), { refreshCodexCatalog: async () => {} }); + expect(response).not.toBeNull(); + expect(response!.status).toBe(400); + const body = await response!.json() as { error?: { code?: string } }; + expect(body.error?.code).toBe(code); + } + }); +}); diff --git a/tests/server/agentradar-account-pin-e2e.test.ts b/tests/server/agentradar-account-pin-e2e.test.ts new file mode 100644 index 00000000000..35cf1cdba34 --- /dev/null +++ b/tests/server/agentradar-account-pin-e2e.test.ts @@ -0,0 +1,159 @@ +import { afterAll, expect, test } from "bun:test"; +import { createHash } from "node:crypto"; +import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { repoPath } from "../helpers/repo-root"; +import { agentRadarAccountPin, noteAgentRadarAccountDispatch, withAgentRadarServedAccount } from "../../src/lib/agentradar-account-pin"; + +const seen: { auth: string | null; internal: string[]; search: boolean }[] = []; +let refusal = 0; +let hold: (() => void) | undefined; +let entered: (() => void) | undefined; +function stream(events: [string, object][]) { + return new Response(events.map(([type, body]) => `event: ${type}\ndata: ${JSON.stringify({ type, ...body })}\n\n`).join(""), + { headers: { "content-type": "text/event-stream" } }); +} +const fake = Bun.serve({ hostname: "127.0.0.1", port: 0, async fetch(req) { + if (req.method !== "POST") return Response.json({ data: [{ id: "claude-stub", type: "model" }, { id: "gpt-5.5", type: "model" }] }); + if (new URL(req.url).pathname.includes("codex/responses")) { + seen.push({ auth: req.headers.get("authorization"), internal: [...req.headers.keys()].filter(key => key.startsWith("x-agentradar-")), search: false }); + const item = { type: "message", id: "msg_codex", role: "assistant", status: "completed", content: [{ type: "output_text", text: "ok", annotations: [] }] }; + return stream([["response.created", { response: { id: "resp_codex", object: "response", status: "in_progress", output: [] } }], + ["response.output_text.delta", { item_id: item.id, output_index: 0, content_index: 0, delta: "ok" }], + ["response.output_item.done", { output_index: 0, item }], + ["response.completed", { response: { id: "resp_codex", object: "response", status: "completed", output: [item], usage: { input_tokens: 1, output_tokens: 1, total_tokens: 2 } } }]]); + } + const body = await req.json() as { tools?: { type?: string; name?: string }[]; messages?: unknown[] }; + const search = body.tools?.some(tool => tool.type === "web_search_20250305") ?? false; + seen.push({ auth: req.headers.get("authorization"), internal: [...req.headers.keys()].filter(key => key.startsWith("x-agentradar-")), search }); + if (entered) { const notify = entered; entered = undefined; notify(); await new Promise(resolve => { hold = resolve; }); } + if (refusal === 400) return Response.json({ type: "error", error: { type: "invalid_request_error", message: "You're out of extra usage. Add more at claude.ai/settings/usage and keep going." } }, { status: 400 }); + if (refusal) return Response.json({ type: "error", error: refusal === 403 + ? { type: "permission_error", message: "Your organization has disabled Claude subscription access for Claude Code. Use an Anthropic API key instead, or ask your admin to enable access." } + : { type: "rate_limit_error", message: "fixture quota refusal" } }, { status: refusal, headers: refusal === 429 + ? { "retry-after": "30", "anthropic-ratelimit-unified-status": "rejected", "anthropic-ratelimit-unified-5h-status": "rejected" } : {} }); + if (!search && body.tools?.some(tool => tool.name?.endsWith("web_search")) && !JSON.stringify(body.messages).includes("tool_result")) return stream([ + ["message_start", { message: { id: "msg_search", type: "message", role: "assistant", model: "claude-stub", content: [], usage: { input_tokens: 1, output_tokens: 0 } } }], + ["content_block_start", { index: 0, content_block: { type: "tool_use", id: "toolu_search", name: body.tools.find(tool => tool.name?.endsWith("web_search"))!.name, input: {} } }], + ["content_block_delta", { index: 0, delta: { type: "input_json_delta", partial_json: JSON.stringify({ query: "fixture query" }) } }], + ["content_block_stop", { index: 0 }], ["message_delta", { delta: { stop_reason: "tool_use" }, usage: { output_tokens: 1 } }], ["message_stop", {}], + ]); + return stream([ + ["message_start", { message: { id: "msg_fixture", type: "message", role: "assistant", model: "claude-stub", content: [], usage: { input_tokens: 1, output_tokens: 0 } } }], + ["content_block_start", { index: 0, content_block: { type: "text", text: "" } }], + ["content_block_delta", { index: 0, delta: { type: "text_delta", text: "ok" } }], ["content_block_stop", { index: 0 }], + ["message_delta", { delta: { stop_reason: "end_turn" }, usage: { output_tokens: 1 } }], ["message_stop", {}], + ]); +} }); +const base = `http://127.0.0.1:${fake.port}`; +const homes: string[] = []; +const processes: Bun.Subprocess[] = []; +const pin = (provider = "claude", id = "A") => ({ "x-agentradar-account-provider": provider, "x-agentradar-account-id": id }); +async function gateway(enabled = true, extra: object = {}) { + const home = mkdtempSync(join(tmpdir(), "ocx-ar-port-")); homes.push(home); + mkdirSync(join(home, ".codex")); + writeFileSync(join(home, "codex-accounts.json"), JSON.stringify(Object.fromEntries(["acct1", "acct2"].map(id => [id, { credential: { accessToken: `fixture-codex-${id}`, refreshToken: `fixture-refresh-${id}`, expiresAt: Date.now() + 3_600_000, chatgptAccountId: `fixture-chatgpt-${id}` }, generation: 0 }]))), { mode: 0o600 }); + writeFileSync(join(home, "auth.json"), JSON.stringify({ anthropic: { activeAccountId: "B", selectionRevision: "fixture-selection", accounts: ["A", "B", "C"].map(id => ({ id, + loginId: `fixture-login-${id}`, credential: { access: `fixture-${id}`, refresh: `fixture-refresh-${id}`, expires: Date.now() + 3_600_000 }, ...(id === "C" ? { needsReauth: true } : {}) })) } }), { mode: 0o600 }); + writeFileSync(join(home, "config.json"), JSON.stringify({ hostname: "127.0.0.1", clientIntegrations: { codex: false }, defaultProvider: "anthropic", + anthropicAccountPool: { enabled: true, strategy: "round-robin", autoSwitchThreshold: 0 }, + providers: { anthropic: { adapter: "anthropic", baseUrl: base, authMode: "oauth", allowPrivateNetwork: true, models: ["claude-stub", "claude-blind"], + modelCapabilities: { "claude-blind": { inputModalities: ["text"] } } }, other: { adapter: "openai-chat", baseUrl: base, apiKey: "fixture-other", models: ["other-stub"], allowPrivateNetwork: true } }, + ...extra }), { mode: 0o600 }); + // The canonical Codex host maps to the fake supplier; every other non-loopback request fails locally. + writeFileSync(join(home, "network.ts"), `const original = globalThis.fetch; globalThis.fetch = ((input, init) => { const u = new URL(input instanceof Request ? input.url : String(input)); if (u.hostname === "chatgpt.com") return original(new Request(${JSON.stringify(base)} + u.pathname + u.search, new Request(input, init))); if (u.hostname !== "127.0.0.1" && u.hostname !== "localhost") throw new Error("non-loopback fixture request refused"); return original(input, init); }) as typeof fetch;`); + const probe = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => new Response() }); const port = probe.port; probe.stop(true); + const proc = Bun.spawn([process.execPath, "--preload", join(home, "network.ts"), repoPath("src/cli/index.ts"), "start", "--port", String(port)], { + cwd: repoPath(), env: { ...process.env, HOME: home, OPENCODEX_HOME: home, OPENCODEX_AGENTRADAR: enabled ? "1" : "0" }, stdout: "ignore", stderr: "pipe", + }); processes.push(proc); + let log = ""; + void (async () => { for await (const chunk of proc.stderr as ReadableStream) log += new TextDecoder().decode(chunk); })(); + const url = `http://127.0.0.1:${port}`; + let error: unknown; + for (let i = 0; i < 200; i++) { + try { if ((await fetch(`${url}/healthz`)).ok) return { url, home, proc, log: () => log }; } catch (caught) { error = caught; } + if (proc.exitCode !== null) break; + await Bun.sleep(50); + } + throw new Error(`fixture gateway failed: ${String(error)} ${log}`); +} +type Gateway = Awaited>; +async function call(gw: Gateway, headers: Record = {}, body: object = {}, path = "/v1/messages") { + const start = seen.length; + const response = await fetch(`${gw.url}${path}`, { method: "POST", headers: { "content-type": "application/json", "anthropic-version": "2023-06-01", ...headers }, + body: JSON.stringify(path === "/v1/chat/completions" ? { model: "anthropic/claude-stub", messages: [{ role: "user", content: "hi" }], ...body } + : { model: "claude-stub", max_tokens: 16, messages: [{ role: "user", content: "hi" }], ...body }) }); + return { status: response.status, served: response.headers.get("x-agentradar-served-account-id"), text: await response.text(), sends: seen.slice(start) }; +} +afterAll(async () => { + hold?.(); + for (const proc of processes) proc.kill(); + await Promise.all(processes.map(proc => proc.exited)); + fake.stop(true); + for (const home of homes) removeTreeWithRetry(home); +}); + +test("AgentRadar parsing and dispatch-only attribution stay opt-in", () => { + const previous = process.env.OPENCODEX_AGENTRADAR; + try { + delete process.env.OPENCODEX_AGENTRADAR; + expect(agentRadarAccountPin(new Headers(pin()))).toBeNull(); + expect(agentRadarAccountPin(new Headers())).toBeUndefined(); + process.env.OPENCODEX_AGENTRADAR = "1"; + expect(agentRadarAccountPin(new Headers(pin()))).toEqual({ provider: "claude", accountId: "A" }); + for (const headers of [pin("other"), pin("claude", "has space"), { "x-agentradar-account-id": "A" }]) expect(agentRadarAccountPin(new Headers(headers))).toBeNull(); + const context = {}; + expect(withAgentRadarServedAccount(new Response(), context).headers.has("x-agentradar-served-account-id")).toBe(false); + noteAgentRadarAccountDispatch(context, "A"); + expect(withAgentRadarServedAccount(new Response(), context).headers.get("x-agentradar-served-account-id")).toBe("A"); + } finally { if (previous === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previous; } +}); + +test("real gateway pins both ingresses, refuses escapes and helpers, and never rotates a refused pin", async () => { + const gw = await gateway(true, { visionSidecar: { backend: "routed", model: "anthropic/claude-stub" } }); + for (const path of ["/v1/messages", "/v1/chat/completions"]) { + const result = await call(gw, { ...pin(), "x-api-key": "sk-ant-fixture-caller" }, {}, path); + expect(result.status, result.text).toBe(200); expect(result.served).toBe("A"); + expect(result.sends.map(send => send.auth)).toEqual(["Bearer fixture-A"]); + expect(result.sends.every(send => send.internal.length === 0)).toBe(true); + } + for (const headers of [pin("claude", "missing"), pin("claude", "C"), pin("grok"), pin("codex", "__main__"), pin("claude", "has space")]) { + const result = await call(gw, headers); expect(result.status).toBeGreaterThanOrEqual(400); expect(result.sends).toHaveLength(0); expect(result.served).toBeNull(); + } + for (const body of [{ model: "other/other-stub" }, { model: "claude-blind", messages: [{ role: "user", content: [{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==" } }] }] }]) { + const result = await call(gw, pin(), body); expect(result.status).toBeGreaterThanOrEqual(400); expect(result.sends).toHaveLength(0); expect(result.served).toBeNull(); + } + const codexGateway = await gateway(true, { activeCodexAccountId: "acct2", + codexAccounts: ["acct1", "acct2"].map(id => ({ id, email: `${id}@example.invalid`, isMain: false, chatgptAccountId: `fixture-chatgpt-${id}` })), + providers: { anthropic: { adapter: "anthropic", baseUrl: base, authMode: "oauth", allowPrivateNetwork: true, models: ["claude-stub"] }, + openai: { adapter: "openai-responses", baseUrl: "https://chatgpt.com/backend-api/codex", authMode: "forward", codexAccountMode: "pool" } } }); + const codex = await call(codexGateway,{ ...pin("codex", "acct1"), authorization: "Bearer caller-must-not-forward", "chatgpt-account-id": "caller-account" }, { model: "openai/gpt-5.5" }, "/v1/chat/completions"); + expect(codex.status, `${codex.text}\n${codexGateway.log()}`).toBe(200); expect(codex.served).toBe("acct1"); + expect(codex.sends.map(send => send.auth)).toEqual(["Bearer fixture-codex-acct1"]); + const unsupported = await call(gw, pin(), { model: "anthropic/claude-stub", input: "hi" }, "/v1/responses"); + expect(unsupported.status).toBe(400); expect(unsupported.sends).toHaveLength(0); + const search = await call(gw, pin(), { tools: [{ type: "web_search_20250305", name: "web_search", max_uses: 1 }] }); + expect(search.status, search.text).toBe(200); expect(search.sends.some(send => send.search)).toBe(true); + expect(new Set(search.sends.map(send => send.auth))).toEqual(new Set(["Bearer fixture-A"])); + for (const status of [400, 403, 429]) { + const isolated = await gateway(); refusal = status; + try { + const result = await call(isolated, pin()); expect(result.status, result.text).toBe(status); + expect(result.sends.map(send => send.auth)).toEqual(["Bearer fixture-A"]); expect(result.served).toBe("A"); + } finally { refusal = 0; } + } + const disabled = await gateway(false); + const refused = await call(disabled, pin()); expect(refused.status).toBe(400); expect(refused.sends).toHaveLength(0); +}, 45_000); + +test("fresh and warm gateways agree on keyed hash seat independently of baseline and persisted affinity", async () => { + const first = await gateway(); const second = await gateway(); + const session = "11111111-1111-4111-8111-111111111111"; + const expected = ["A", "B"].sort((a, b) => createHash("sha256").update(`${createHash("sha256").update(session).digest("hex").slice(0, 32)}\0${b}`).digest("hex").localeCompare(createHash("sha256").update(`${createHash("sha256").update(session).digest("hex").slice(0, 32)}\0${a}`).digest("hex")))[0]; + for (const gw of [first, second, first]) { + const result = await call(gw, {}, { metadata: { user_id: JSON.stringify({ session_id: session, tk: crypto.randomUUID() }) } }); + expect(result.status, result.text).toBe(200); expect(result.served).toBe(expected); + } +}, 30_000); diff --git a/tests/server/agentradar-account-pin-native-e2e.test.ts b/tests/server/agentradar-account-pin-native-e2e.test.ts new file mode 100644 index 00000000000..86e5358ddfc --- /dev/null +++ b/tests/server/agentradar-account-pin-native-e2e.test.ts @@ -0,0 +1,349 @@ +// Variant of agentradar-account-pin-parity-e2e.test.ts with Anthropic on its real host, as AgentRadar +// runs it: the fake-net preload redirects api.anthropic.com to the loopback provider, so routing +// takes the managed native Messages lane. Assertions are identical to the parity file. +// Ported from v2.65.0-vadymhimself.27 tests/server/agentradar-account-pin-e2e.test.ts (itself from AgentRadar +// images/ocx/pin.check.ts and failover.check.ts). Adaptations are marked "upstream:" inline. +// Originally: pin.check.ts and failover.check.ts (the moved pin-patch rows): a real +// OCX process serving fake credentials against a loopback fake provider. No real tokens, no real provider. +import { afterAll, expect, test } from "bun:test"; +import assert from "node:assert"; +import { createHash } from "node:crypto"; +import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { agentRadarAccountPin } from "../../src/lib/agentradar-account-pin"; + +const CLI = join(import.meta.dir, "../../src/cli/index.ts"); +const sha = (v: string) => createHash("sha256").update(v).digest("hex"); + +// --- fake provider: records every request OCX makes; 429s the tokens in `limited`, 403s (credential denial) those in `denied`. +type Seen = { path: string; auth: string | null; account: string | null; internal: string[]; search?: boolean; image?: boolean }; +const seen: Seen[] = []; +const limited = new Set(); +const denied = new Set(); +const enc = new TextEncoder(); +const sse = (events: [string, object][]) => new Response(new ReadableStream({ + start(c) { for (const [type, data] of events) c.enqueue(enc.encode(`event: ${type}\ndata: ${JSON.stringify({ type, ...data })}\n\n`)); c.close(); }, +}), { headers: { "content-type": "text/event-stream" } }); +const rateLimited = (retryAfter: string) => Response.json({ type: "error", error: { type: "rate_limit_error", message: "slow down" } }, { status: 429, headers: { "retry-after": retryAfter } }); +let retryAfter = "30"; +const fake = Bun.serve({ + port: 0, + async fetch(req) { + const path = new URL(req.url).pathname; + const auth = req.headers.get("authorization"); + const body = await req.text(); + const tools: any[] = path === "/v1/messages" ? JSON.parse(body).tools ?? [] : []; + // The helper's own call carries Anthropic's native search tool; the turn it serves carries OCX's web_search function. + const search = tools.some(t => t.type === "web_search_20250305"); + if (path === "/v1/messages" || path === "/backend-api/codex/responses" || path.startsWith("/xai/") || path.startsWith("/kimi/")) { + seen.push({ path, auth, account: req.headers.get("chatgpt-account-id"), internal: [...req.headers.keys()].filter(k => k.startsWith("x-agentradar-")), search, image: body.includes('"type":"image"') }); + } + if (path === "/v1/messages") { + if (auth && limited.has(auth)) return rateLimited(retryAfter); + if (auth && denied.has(auth)) return Response.json({ type: "error", error: { type: "oauth_org_not_allowed", + message: "Your organization has disabled Claude subscription access for Claude Code. Use an Anthropic API key instead, or ask your admin to enable access." } }, { status: 403 }); + if (!search && tools.some(t => t.name === "web_search") && !body.includes('"tool_result"')) return sse([ + ["message_start", { message: { id: "msg_tool", type: "message", role: "assistant", model: "claude-stub", content: [], stop_reason: null, usage: { input_tokens: 1, output_tokens: 0 } } }], + ["content_block_start", { index: 0, content_block: { type: "tool_use", id: "toolu_fake", name: "web_search", input: {} } }], + ["content_block_delta", { index: 0, delta: { type: "input_json_delta", partial_json: JSON.stringify({ query: "agentradar release notes" }) } }], + ["content_block_stop", { index: 0 }], + ["message_delta", { delta: { stop_reason: "tool_use", stop_sequence: null }, usage: { output_tokens: 1 } }], + ["message_stop", {}], + ]); + return sse([ + ["message_start", { message: { id: "msg_fake", type: "message", role: "assistant", model: "claude-stub", content: [], stop_reason: null, usage: { input_tokens: 1, output_tokens: 0 } } }], + ["content_block_start", { index: 0, content_block: { type: "text", text: "" } }], + ["content_block_delta", { index: 0, delta: { type: "text_delta", text: "ok" } }], + ["content_block_stop", { index: 0 }], + ["message_delta", { delta: { stop_reason: "end_turn", stop_sequence: null }, usage: { output_tokens: 1 } }], + ["message_stop", {}], + ]); + } + if (path.startsWith("/xai/") || path.startsWith("/kimi/")) return rateLimited(retryAfter); + if (path === "/backend-api/codex/responses") { + const item = { type: "message", id: "msg_fake", role: "assistant", status: "completed", content: [{ type: "output_text", text: "ok", annotations: [] }] }; + const response = { id: "resp_fake", object: "response", model: "gpt-5.5", status: "in_progress", output: [] }; + return sse([ + ["response.created", { response }], + ["response.output_text.delta", { item_id: item.id, output_index: 0, content_index: 0, delta: "ok" }], + ["response.output_item.done", { output_index: 0, item }], + ["response.completed", { response: { ...response, status: "completed", output: [item], usage: { input_tokens: 1, output_tokens: 1, total_tokens: 2 } } }], + ]); + } + return Response.json({ error: "not found" }, { status: 404 }); + }, +}); +const FAKE = `http://127.0.0.1:${fake.port}`; +const cred = (access: string) => ({ access, refresh: `r-${access}`, expires: 9999999999999, email: `${access}@example.invalid` }); +const codexCred = (n: number) => ({ credential: { accessToken: `cat${n}`, refreshToken: `crt${n}`, expiresAt: 9999999999999, chatgptAccountId: `cga${n}` }, generation: 0 }); +const homes: string[] = []; + +// One OCX process on its own home. Every non-loopback fetch (the Codex, Grok and Kimi backends) goes to the fake. +async function startOcx(auth: object, config: object) { + const home = mkdtempSync(join(tmpdir(), "ocx-ar-pin-native-")); + homes.push(home); + mkdirSync(`${home}/.codex`, { mode: 0o700 }); + writeFileSync(`${home}/auth.json`, JSON.stringify(auth), { mode: 0o600 }); + writeFileSync(`${home}/codex-accounts.json`, JSON.stringify({ acct1: codexCred(1), acct2: codexCred(2) }), { mode: 0o600 }); + const probe = Bun.serve({ port: 0, fetch: () => new Response() }); + const port = probe.port; + probe.stop(true); + writeFileSync(`${home}/config.json`, JSON.stringify({ port, hostname: "127.0.0.1", ...config }), { mode: 0o600 }); + writeFileSync(`${home}/fake-net.ts`, `const real = globalThis.fetch; +globalThis.fetch = ((input: any, init?: any) => { + const u = new URL(input instanceof Request ? input.url : String(input)); + if (u.hostname === "127.0.0.1" || u.hostname === "localhost") return real(input, init); + const base = ${JSON.stringify(FAKE)} + (u.hostname === "api.x.ai" || u.hostname === "cli-chat-proxy.grok.com" ? "/xai" : u.hostname === "api.kimi.com" ? "/kimi" : ""); + return real(new Request(base + u.pathname + u.search, new Request(input, init))); +}) as typeof fetch; +`); + const admin = `ocx_admin_${crypto.randomUUID()}${crypto.randomUUID()}`; + const proc = Bun.spawn([process.execPath, "--preload", `${home}/fake-net.ts`, CLI, "start", "--port", String(port)], { + env: { ...process.env, OPENCODEX_HOME: home, HOME: home, OPENCODEX_ADMIN_AUTH_TOKEN: admin, OPENCODEX_AGENTRADAR: "1" }, stdout: "ignore", stderr: "pipe", + }); + const ocx = { proc, home, admin, base: `http://127.0.0.1:${port}`, log: "" }; + (async () => { for await (const chunk of proc.stderr) ocx.log += new TextDecoder().decode(chunk); })(); + let lastError: unknown; + for (let i = 0; ; i++) { + try { if ((await fetch(`${ocx.base}/healthz`)).status < 500) break; } catch (error) { lastError = error; } + if (i > 300) throw new Error(`OCX did not start (${String(lastError)}): ${ocx.log}`); + await Bun.sleep(100); + } + return ocx; +} +type Ocx = Awaited>; + +const call = async (ocx: Ocx, path: string, body: object, headers: Record = {}) => { + const from = seen.length; + const res = await fetch(`${ocx.base}${path}`, { method: "POST", headers: { "content-type": "application/json", ...headers }, body: JSON.stringify(body) }); + const text = await res.text(); + return { status: res.status, text, served: res.headers.get("x-agentradar-served-account-id"), upstream: seen.slice(from) }; +}; +const pin = (provider: string, id: string) => ({ "x-agentradar-account-provider": provider, "x-agentradar-account-id": id }); +const msg = (ocx: Ocx, headers: Record = {}, model = "claude-stub") => call(ocx, "/v1/messages", + { model, max_tokens: 16, stream: true, messages: [{ role: "user", content: "hi" }] }, + { "anthropic-version": "2023-06-01", "x-api-key": "unused", ...headers }); +const chat = (ocx: Ocx, model: string, headers: Record = {}) => call(ocx, "/v1/chat/completions", + { model, messages: [{ role: "user", content: "hi" }] }, headers); +const tokens = (r: { upstream: Seen[] }, prefix: string) => r.upstream.filter(s => s.path.startsWith(prefix)).map(s => s.auth); +const inference = (r: { upstream: Seen[] }) => r.upstream; +const accounts = async (ocx: Ocx, provider: string) => (await fetch(`${ocx.base}/api/oauth/accounts?provider=${provider}`, { headers: { authorization: `Bearer ${ocx.admin}` } })).json() as Promise; + +afterAll(() => { + fake.stop(true); + for (const home of homes) removeTreeWithRetry(home); +}); + +test("outside an AgentRadar gateway a pin is refused, never served unpinned", () => { + const prior = process.env.OPENCODEX_AGENTRADAR; + const headers = new Headers({ "x-agentradar-account-provider": "claude", "x-agentradar-account-id": "A" }); + try { + delete process.env.OPENCODEX_AGENTRADAR; + expect(agentRadarAccountPin(headers)).toBeNull(); + expect(agentRadarAccountPin(new Headers())).toBeUndefined(); + process.env.OPENCODEX_AGENTRADAR = "1"; + expect(agentRadarAccountPin(headers)).toEqual({ provider: "claude", accountId: "A" }); + } finally { + if (prior === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = prior; + } +}); + +test("an AgentRadar account pin serves exactly its own stored account (pin.check)", async () => { + limited.clear(); denied.clear(); retryAfter = "30"; + const ocx = await startOcx({ + anthropic: { activeAccountId: "B", accounts: [ + { id: "A", credential: cred("atA") }, { id: "B", credential: cred("atB") }, { id: "C", credential: cred("atC"), needsReauth: true }, + { id: "D", credential: cred("atD") }, { id: "E", credential: cred("atE") }] }, + xai: { activeAccountId: "gB", accounts: ["gA", "gB", "gC"].map(id => ({ id, credential: cred(`t${id}`) })) }, + kimi: { activeAccountId: "kB", accounts: ["kA", "kB"].map(id => ({ id, credential: cred(`t${id}`) })) }, + }, { + defaultProvider: "anthropic", activeCodexAccountId: "acct2", + clientIntegrations: { codex: false }, + // claude-blind cannot see images, so an image turn plans a vision describe through a helper model. + visionSidecar: { backend: "routed", model: "anthropic/claude-stub" }, + providers: { + anthropic: { adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "oauth", allowPrivateNetwork: true, models: ["claude-stub", "claude-blind"], modelCapabilities: { "claude-blind": { inputModalities: ["text"] } } }, + xai: { adapter: "openai-chat", baseUrl: `${FAKE}/xai`, authMode: "oauth", allowPrivateNetwork: true, models: ["grok-stub"] }, + kimi: { adapter: "openai-chat", baseUrl: `${FAKE}/kimi`, authMode: "oauth", allowPrivateNetwork: true, models: ["kimi-stub"] }, + openai: { adapter: "openai-responses", baseUrl: "https://chatgpt.com/backend-api/codex", authMode: "forward", codexAccountMode: "pool" }, + }, + codexAccounts: [1, 2].map(n => ({ id: `acct${n}`, email: `c${n}@example.invalid`, isMain: false, chatgptAccountId: `cga${n}` })), + }); + // upstream: the pool logs an entitlement failover as "account unavailable (403); failing over". + const rotations = () => ocx.log.split("account unavailable (403); failing over").length - 1; + const failovers429 = () => ocx.log.split("; failing over to").length - 1; + try { + // Claude: unpinned control serves the active account; a pin serves exactly its own, both ways, and names it. + let r = await msg(ocx); + assert.equal(r.status, 200, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atB"]); + assert.equal(r.served, "B"); + r = await msg(ocx, pin("claude", "A")); + assert.equal(r.status, 200, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atA"]); + assert.equal(r.served, "A"); + // The managed native Messages lane served the pin: only its send boundary logs r=native (the + // translated bridge logs r=pin), so a silent fallback to translation fails here. + for (let i = 0; !ocx.log.includes(`[ar-acct] a=${sha("A").slice(0, 8)} r=native`) && i < 50; i++) await Bun.sleep(20); + assert.ok(ocx.log.includes(`[ar-acct] a=${sha("A").slice(0, 8)} r=native`), ocx.log); + assert.ok(!ocx.log.includes(`[ar-acct] a=${sha("A").slice(0, 8)} r=pin`), ocx.log); + r = await msg(ocx, pin("claude", "B")); + assert.equal(r.status, 200, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atB"]); + // A caller Anthropic key would take native passthrough; a pin never does. + r = await msg(ocx, { ...pin("claude", "A"), "x-api-key": "sk-ant-fake" }); + assert.equal(r.status, 200, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atA"]); + // Disconnected or missing pinned account: refused, never served by another account. + for (const id of ["C", "missing"]) { + r = await msg(ocx, pin("claude", id)); + assert.equal(r.status, 401, r.text); assert.deepEqual(inference(r), []); + assert.equal(r.served, null, "an answer no account served names none"); + } + // Malformed pins are refused, never served unpinned. + for (const h of [pin("gemini", "A"), { "x-agentradar-account-id": "A" }, pin("claude", "has space")]) { + r = await msg(ocx, h); + assert.equal(r.status, 400, r.text); assert.deepEqual(inference(r), []); + assert.equal(r.served, null); + } + // A pin cannot be spent on another provider's route. + for (const h of [pin("grok", "gA"), pin("kimi", "kA"), pin("codex", "acct1")]) { + r = await msg(ocx, h); + assert.equal(r.status, 404, r.text); assert.deepEqual(inference(r), []); + } + // A helper model runs outside the pin (the loopback describe carries no pin): refused, nothing sent. + const png = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=="; + const blindImage = { model: "claude-blind", max_tokens: 16, stream: true, messages: [{ role: "user", content: [ + { type: "image", source: { type: "base64", media_type: "image/png", data: png } }, { type: "text", text: "what is this" }] }] }; + // Control: unpinned, the same turn really does plan a describe, so the refusal below is not vacuous. + r = await call(ocx, "/v1/messages", blindImage, { "anthropic-version": "2023-06-01", "x-api-key": "unused" }); + assert.equal(r.status, 200, r.text); + assert.ok(inference(r).length >= 2 && !inference(r).at(-1)!.image, `no vision describe planned: ${JSON.stringify(r.upstream)}`); + r = await call(ocx, "/v1/messages", blindImage, { "anthropic-version": "2023-06-01", "x-api-key": "unused", ...pin("claude", "A") }); + assert.equal(r.status, 400, `${r.text}\n${JSON.stringify(r.upstream)}`); assert.deepEqual(inference(r), []); + // Surfaces that do not honour a pin refuse it instead of serving the active account. + r = await call(ocx, "/v1/responses", { model: "anthropic/claude-stub", input: "hi" }, pin("claude", "A")); + assert.equal(r.status, 400, r.text); assert.match(r.text, /only honoured on/); assert.deepEqual(inference(r), []); + // Claude Code's web search, as it sends it: the search helper runs on the pinned account too. + r = await call(ocx, "/v1/messages", { model: "claude-stub", max_tokens: 64, stream: true, + tools: [{ type: "web_search_20250305", name: "web_search", max_uses: 8 }], + messages: [{ role: "user", content: "Search the web for the latest agentradar release notes" }] }, + { "anthropic-version": "2023-06-01", "anthropic-beta": "claude-code-20250219", "x-api-key": "unused", ...pin("claude", "A") }); + assert.equal(r.status, 200, r.text); + assert.ok(r.upstream.some(s => s.search), `no search helper call: ${JSON.stringify(r.upstream)}`); + assert.ok(r.upstream.filter(s => !s.search).length >= 2, `turn did not resume after the search: ${JSON.stringify(r.upstream)}`); + assert.deepEqual([...new Set(tokens(r, "/v1/messages"))], ["Bearer atA"], JSON.stringify(r.upstream)); + // Grok and Codex have no helper that can run on their pin: still refused. + r = await call(ocx, "/v1/chat/completions", { model: "xai/grok-stub", messages: [{ role: "user", content: "hi" }], tools: [{ type: "web_search" }] }, pin("grok", "gA")); + assert.equal(r.status, 400, r.text); assert.match(r.text, /cannot use helper models/); assert.deepEqual(inference(r), []); + assert.equal((await accounts(ocx, "anthropic")).activeAccountId, "B", "pins never move the active account"); + + // Claude 429 on the pinned account: returned as is, cooldown recorded, no switch to B. + limited.add("Bearer atA"); + r = await msg(ocx, pin("claude", "A")); + assert.equal(r.status, 429, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atA"]); + // The rotator's own pin check: without it the caller picks another account and cancels this 429's body. + // upstream: Messages error-detail normalization rewrites the client message, so the 429 is proven by + // status, the single pinned send and the zero failover count instead of the provider wording. + assert.match(r.text, /rate_limit_error/); + await Bun.sleep(200); + assert.equal(failovers429(), 0, ocx.log); + const claude = await accounts(ocx, "anthropic"); + assert.equal(claude.activeAccountId, "B"); + assert.equal(claude.accounts.find((a: any) => a.id === "A")?.health?.status, "cooldown", JSON.stringify(claude)); + r = await msg(ocx); + assert.equal(r.status, 200, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atB"]); + + // Claude credential denial on the pinned account: returned as is, never moved to E; the unpinned control does fail over. + denied.add("Bearer atD"); + r = await msg(ocx, pin("claude", "D")); + assert.equal(r.status, 403, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atD"]); + assert.equal((await accounts(ocx, "anthropic")).activeAccountId, "B"); + denied.add("Bearer atB"); + r = await msg(ocx); + assert.equal(r.status, 200, r.text); + const denialControl = tokens(r, "/v1/messages"); + assert.equal(denialControl[0], "Bearer atB"); assert.ok(denialControl.length > 1, `stock denial control did not rotate: ${denialControl}`); + // The pinned 403's rotation stopped at the reauth mark: only the control's failover was logged. + for (let i = 0; rotations() < 1 && i < 50; i++) await Bun.sleep(20); + await Bun.sleep(200); + assert.equal(rotations(), 1, ocx.log); + + // Grok (every call 429s): the pinned account alone, active unchanged; the unpinned control does rotate. + r = await chat(ocx, "xai/grok-stub", pin("grok", "gA")); + assert.equal(r.status, 429, `${r.text}\n${JSON.stringify(r.upstream)}`); assert.deepEqual(tokens(r, "/xai/"), ["Bearer tgA"]); + assert.equal((await accounts(ocx, "xai")).activeAccountId, "gB"); + r = await chat(ocx, "xai/grok-stub"); + assert.equal(r.status, 429, r.text); + const control = tokens(r, "/xai/"); + assert.equal(control[0], "Bearer tgB"); assert.ok(control.length > 1, `stock rotation control did not rotate: ${control}`); + + // Kimi (every call 429s): the pinned account alone, and the active account stays. + r = await chat(ocx, "kimi/kimi-stub", pin("kimi", "kA")); + assert.equal(r.status, 429, `${r.text}\n${JSON.stringify(r.upstream)}`); assert.deepEqual(tokens(r, "/kimi/"), ["Bearer tkA"]); + assert.equal((await accounts(ocx, "kimi")).activeAccountId, "kB"); + r = await chat(ocx, "xai/grok-stub", pin("kimi", "kA")); + assert.equal(r.status, 404, r.text); assert.deepEqual(inference(r), []); + + // Codex: OCX's exact-account route serves the pinned account; the active account is untouched. + for (const n of [1, 2]) { + r = await chat(ocx, "openai/gpt-5.5", pin("codex", `acct${n}`)); + assert.equal(r.status, 200, r.text); + assert.deepEqual(r.upstream.filter(s => s.path === "/backend-api/codex/responses").map(s => [s.auth, s.account]), [[`Bearer cat${n}`, `cga${n}`]]); + assert.equal(r.served, `acct${n}`, "a pinned chat answer names its account"); + } + r = await chat(ocx, "openai/gpt-5.5", pin("codex", "missing")); + assert.ok(r.status >= 400, r.text); assert.deepEqual(inference(r), []); + const active = await (await fetch(`${ocx.base}/api/codex-auth/active`, { headers: { authorization: `Bearer ${ocx.admin}` } })).json() as any; + assert.equal(active.activeCodexAccountId, "acct2", JSON.stringify(active)); + + // The internal headers never reached any upstream request. + assert.deepEqual(seen.filter(s => s.internal.length), []); + } finally { + ocx.proc.kill(); + } +}, 120_000); + +test("pooled turns log and answer their served account hashed; one request walks every spent account (failover.check)", async () => { + limited.clear(); denied.clear(); retryAfter = "318747"; + const ids = ["A", "B", "C", "D", "E", "F", "G", "H"]; + // The hosted gateway's pool defaults (AgentRadar images/ocx/pool-defaults.mjs). + const ocx = await startOcx({ + anthropic: { activeAccountId: "A", accounts: ids.map(id => ({ id, credential: cred(`at${id}`) })) }, + xai: { activeAccountId: "gA", accounts: ids.map(id => ({ id: `g${id}`, credential: cred(`tg${id}`) })) }, + }, { + defaultProvider: "anthropic", activeCodexAccountId: "acct1", + anthropicAccountPool: { enabled: true, strategy: "round-robin" }, accountPoolStrategy: "round-robin", clientIntegrations: { codex: false }, + providers: { + anthropic: { adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "oauth", allowPrivateNetwork: true, models: ["claude-stub"] }, + xai: { adapter: "openai-chat", baseUrl: `${FAKE}/xai`, authMode: "oauth", allowPrivateNetwork: true, models: ["grok-stub"] }, + }, + }); + try { + // CACHE-ACCOUNT-AFFINITY: each turn logs its served account and session hashed, and answers the account's id. + const turns = [await msg(ocx, { "thread-id": "s1" }), await msg(ocx, { "thread-id": "s2" })]; + for (const t of turns) { + assert.equal(t.status, 200, t.text); + const servedId = ids.find(id => t.upstream[0]!.auth === `Bearer at${id}`)!; + expect(t.served).toBe(servedId); + } + for (let i = 0; (ocx.log.match(/\[ar-acct\]/g) ?? []).length < 2 && i < 50; i++) await Bun.sleep(20); + const logged = [...ocx.log.matchAll(/\[ar-acct\] a=([0-9a-f]{8}) r=(\S+) s=([0-9a-f]{8})\n/g)]; + expect(logged.map(m => m[1])).toEqual(turns.map(t => sha(t.served!).slice(0, 8))); + expect(logged.map(m => m[3])).toEqual([sha("s1").slice(0, 8), sha("s2").slice(0, 8)]); + expect(ocx.log).not.toContain("atA"); + + // GR2 10-09: stock OCX gave up after 3 rotations. With every account answering 429, one Claude request + // asks each account once; Grok's generic failover is bounded by OCX's shared request budget (5 sends). + for (const id of ids) limited.add(`Bearer at${id}`).add(`Bearer tg${id}`); + const claude = await msg(ocx, { "thread-id": "all-spent" }); + assert.equal(claude.status, 429, claude.text); + expect([...tokens(claude, "/v1/messages")].sort()).toEqual(ids.map(id => `Bearer at${id}`)); + const grok = await msg(ocx, { "thread-id": "grok-all-spent" }, "xai/grok-stub"); + assert.equal(grok.status, 429, grok.text); + // upstream: the hosted funded roster (oauthMaxAccountsPerRequest, 16) covers Grok's generic failover too, so + // one request asks each spent account once (.27 stopped at its 5-send request budget). Never a repeat. + expect(tokens(grok, "/xai/")).toHaveLength(8); + expect(new Set(tokens(grok, "/xai/")).size).toBe(8); + } finally { + ocx.proc.kill(); + } +}, 120_000); diff --git a/tests/server/agentradar-account-pin-parity-e2e.test.ts b/tests/server/agentradar-account-pin-parity-e2e.test.ts new file mode 100644 index 00000000000..d2a5ef00bd9 --- /dev/null +++ b/tests/server/agentradar-account-pin-parity-e2e.test.ts @@ -0,0 +1,341 @@ +// Ported from v2.65.0-vadymhimself.27 tests/server/agentradar-account-pin-e2e.test.ts (itself from AgentRadar +// images/ocx/pin.check.ts and failover.check.ts). Adaptations are marked "upstream:" inline. +// Originally: pin.check.ts and failover.check.ts (the moved pin-patch rows): a real +// OCX process serving fake credentials against a loopback fake provider. No real tokens, no real provider. +import { afterAll, expect, test } from "bun:test"; +import assert from "node:assert"; +import { createHash } from "node:crypto"; +import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { agentRadarAccountPin } from "../../src/lib/agentradar-account-pin"; + +const CLI = join(import.meta.dir, "../../src/cli/index.ts"); +const sha = (v: string) => createHash("sha256").update(v).digest("hex"); + +// --- fake provider: records every request OCX makes; 429s the tokens in `limited`, 403s (credential denial) those in `denied`. +type Seen = { path: string; auth: string | null; account: string | null; internal: string[]; search?: boolean; image?: boolean }; +const seen: Seen[] = []; +const limited = new Set(); +const denied = new Set(); +const enc = new TextEncoder(); +const sse = (events: [string, object][]) => new Response(new ReadableStream({ + start(c) { for (const [type, data] of events) c.enqueue(enc.encode(`event: ${type}\ndata: ${JSON.stringify({ type, ...data })}\n\n`)); c.close(); }, +}), { headers: { "content-type": "text/event-stream" } }); +const rateLimited = (retryAfter: string) => Response.json({ type: "error", error: { type: "rate_limit_error", message: "slow down" } }, { status: 429, headers: { "retry-after": retryAfter } }); +let retryAfter = "30"; +const fake = Bun.serve({ + port: 0, + async fetch(req) { + const path = new URL(req.url).pathname; + const auth = req.headers.get("authorization"); + const body = await req.text(); + const tools: any[] = path === "/v1/messages" ? JSON.parse(body).tools ?? [] : []; + // The helper's own call carries Anthropic's native search tool; the turn it serves carries OCX's web_search function. + const search = tools.some(t => t.type === "web_search_20250305"); + if (path === "/v1/messages" || path === "/backend-api/codex/responses" || path.startsWith("/xai/") || path.startsWith("/kimi/")) { + seen.push({ path, auth, account: req.headers.get("chatgpt-account-id"), internal: [...req.headers.keys()].filter(k => k.startsWith("x-agentradar-")), search, image: body.includes('"type":"image"') }); + } + if (path === "/v1/messages") { + if (auth && limited.has(auth)) return rateLimited(retryAfter); + if (auth && denied.has(auth)) return Response.json({ type: "error", error: { type: "oauth_org_not_allowed", + message: "Your organization has disabled Claude subscription access for Claude Code. Use an Anthropic API key instead, or ask your admin to enable access." } }, { status: 403 }); + if (!search && tools.some(t => t.name === "web_search") && !body.includes('"tool_result"')) return sse([ + ["message_start", { message: { id: "msg_tool", type: "message", role: "assistant", model: "claude-stub", content: [], stop_reason: null, usage: { input_tokens: 1, output_tokens: 0 } } }], + ["content_block_start", { index: 0, content_block: { type: "tool_use", id: "toolu_fake", name: "web_search", input: {} } }], + ["content_block_delta", { index: 0, delta: { type: "input_json_delta", partial_json: JSON.stringify({ query: "agentradar release notes" }) } }], + ["content_block_stop", { index: 0 }], + ["message_delta", { delta: { stop_reason: "tool_use", stop_sequence: null }, usage: { output_tokens: 1 } }], + ["message_stop", {}], + ]); + return sse([ + ["message_start", { message: { id: "msg_fake", type: "message", role: "assistant", model: "claude-stub", content: [], stop_reason: null, usage: { input_tokens: 1, output_tokens: 0 } } }], + ["content_block_start", { index: 0, content_block: { type: "text", text: "" } }], + ["content_block_delta", { index: 0, delta: { type: "text_delta", text: "ok" } }], + ["content_block_stop", { index: 0 }], + ["message_delta", { delta: { stop_reason: "end_turn", stop_sequence: null }, usage: { output_tokens: 1 } }], + ["message_stop", {}], + ]); + } + if (path.startsWith("/xai/") || path.startsWith("/kimi/")) return rateLimited(retryAfter); + if (path === "/backend-api/codex/responses") { + const item = { type: "message", id: "msg_fake", role: "assistant", status: "completed", content: [{ type: "output_text", text: "ok", annotations: [] }] }; + const response = { id: "resp_fake", object: "response", model: "gpt-5.5", status: "in_progress", output: [] }; + return sse([ + ["response.created", { response }], + ["response.output_text.delta", { item_id: item.id, output_index: 0, content_index: 0, delta: "ok" }], + ["response.output_item.done", { output_index: 0, item }], + ["response.completed", { response: { ...response, status: "completed", output: [item], usage: { input_tokens: 1, output_tokens: 1, total_tokens: 2 } } }], + ]); + } + return Response.json({ error: "not found" }, { status: 404 }); + }, +}); +const FAKE = `http://127.0.0.1:${fake.port}`; +const cred = (access: string) => ({ access, refresh: `r-${access}`, expires: 9999999999999, email: `${access}@example.invalid` }); +const codexCred = (n: number) => ({ credential: { accessToken: `cat${n}`, refreshToken: `crt${n}`, expiresAt: 9999999999999, chatgptAccountId: `cga${n}` }, generation: 0 }); +const homes: string[] = []; + +// One OCX process on its own home. Every non-loopback fetch (the Codex, Grok and Kimi backends) goes to the fake. +async function startOcx(auth: object, config: object) { + const home = mkdtempSync(join(tmpdir(), "ocx-ar-pin-")); + homes.push(home); + mkdirSync(`${home}/.codex`, { mode: 0o700 }); + writeFileSync(`${home}/auth.json`, JSON.stringify(auth), { mode: 0o600 }); + writeFileSync(`${home}/codex-accounts.json`, JSON.stringify({ acct1: codexCred(1), acct2: codexCred(2) }), { mode: 0o600 }); + const probe = Bun.serve({ port: 0, fetch: () => new Response() }); + const port = probe.port; + probe.stop(true); + writeFileSync(`${home}/config.json`, JSON.stringify({ port, hostname: "127.0.0.1", ...config }), { mode: 0o600 }); + writeFileSync(`${home}/fake-net.ts`, `const real = globalThis.fetch; +globalThis.fetch = ((input: any, init?: any) => { + const u = new URL(input instanceof Request ? input.url : String(input)); + if (u.hostname === "127.0.0.1" || u.hostname === "localhost") return real(input, init); + const base = ${JSON.stringify(FAKE)} + (u.hostname === "api.x.ai" || u.hostname === "cli-chat-proxy.grok.com" ? "/xai" : u.hostname === "api.kimi.com" ? "/kimi" : ""); + return real(new Request(base + u.pathname + u.search, new Request(input, init))); +}) as typeof fetch; +`); + const admin = `ocx_admin_${crypto.randomUUID()}${crypto.randomUUID()}`; + const proc = Bun.spawn([process.execPath, "--preload", `${home}/fake-net.ts`, CLI, "start", "--port", String(port)], { + env: { ...process.env, OPENCODEX_HOME: home, HOME: home, OPENCODEX_ADMIN_AUTH_TOKEN: admin, OPENCODEX_AGENTRADAR: "1" }, stdout: "ignore", stderr: "pipe", + }); + const ocx = { proc, home, admin, base: `http://127.0.0.1:${port}`, log: "" }; + (async () => { for await (const chunk of proc.stderr) ocx.log += new TextDecoder().decode(chunk); })(); + let lastError: unknown; + for (let i = 0; ; i++) { + try { if ((await fetch(`${ocx.base}/healthz`)).status < 500) break; } catch (error) { lastError = error; } + if (i > 300) throw new Error(`OCX did not start (${String(lastError)}): ${ocx.log}`); + await Bun.sleep(100); + } + return ocx; +} +type Ocx = Awaited>; + +const call = async (ocx: Ocx, path: string, body: object, headers: Record = {}) => { + const from = seen.length; + const res = await fetch(`${ocx.base}${path}`, { method: "POST", headers: { "content-type": "application/json", ...headers }, body: JSON.stringify(body) }); + const text = await res.text(); + return { status: res.status, text, served: res.headers.get("x-agentradar-served-account-id"), upstream: seen.slice(from) }; +}; +const pin = (provider: string, id: string) => ({ "x-agentradar-account-provider": provider, "x-agentradar-account-id": id }); +const msg = (ocx: Ocx, headers: Record = {}, model = "claude-stub") => call(ocx, "/v1/messages", + { model, max_tokens: 16, stream: true, messages: [{ role: "user", content: "hi" }] }, + { "anthropic-version": "2023-06-01", "x-api-key": "unused", ...headers }); +const chat = (ocx: Ocx, model: string, headers: Record = {}) => call(ocx, "/v1/chat/completions", + { model, messages: [{ role: "user", content: "hi" }] }, headers); +const tokens = (r: { upstream: Seen[] }, prefix: string) => r.upstream.filter(s => s.path.startsWith(prefix)).map(s => s.auth); +const inference = (r: { upstream: Seen[] }) => r.upstream; +const accounts = async (ocx: Ocx, provider: string) => (await fetch(`${ocx.base}/api/oauth/accounts?provider=${provider}`, { headers: { authorization: `Bearer ${ocx.admin}` } })).json() as Promise; + +afterAll(() => { + fake.stop(true); + for (const home of homes) removeTreeWithRetry(home); +}); + +test("outside an AgentRadar gateway a pin is refused, never served unpinned", () => { + const prior = process.env.OPENCODEX_AGENTRADAR; + const headers = new Headers({ "x-agentradar-account-provider": "claude", "x-agentradar-account-id": "A" }); + try { + delete process.env.OPENCODEX_AGENTRADAR; + expect(agentRadarAccountPin(headers)).toBeNull(); + expect(agentRadarAccountPin(new Headers())).toBeUndefined(); + process.env.OPENCODEX_AGENTRADAR = "1"; + expect(agentRadarAccountPin(headers)).toEqual({ provider: "claude", accountId: "A" }); + } finally { + if (prior === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = prior; + } +}); + +test("an AgentRadar account pin serves exactly its own stored account (pin.check)", async () => { + limited.clear(); denied.clear(); retryAfter = "30"; + const ocx = await startOcx({ + anthropic: { activeAccountId: "B", accounts: [ + { id: "A", credential: cred("atA") }, { id: "B", credential: cred("atB") }, { id: "C", credential: cred("atC"), needsReauth: true }, + { id: "D", credential: cred("atD") }, { id: "E", credential: cred("atE") }] }, + xai: { activeAccountId: "gB", accounts: ["gA", "gB", "gC"].map(id => ({ id, credential: cred(`t${id}`) })) }, + kimi: { activeAccountId: "kB", accounts: ["kA", "kB"].map(id => ({ id, credential: cred(`t${id}`) })) }, + }, { + defaultProvider: "anthropic", activeCodexAccountId: "acct2", + clientIntegrations: { codex: false }, + // claude-blind cannot see images, so an image turn plans a vision describe through a helper model. + visionSidecar: { backend: "routed", model: "anthropic/claude-stub" }, + providers: { + anthropic: { adapter: "anthropic", baseUrl: FAKE, authMode: "oauth", allowPrivateNetwork: true, models: ["claude-stub", "claude-blind"], modelCapabilities: { "claude-blind": { inputModalities: ["text"] } } }, + xai: { adapter: "openai-chat", baseUrl: `${FAKE}/xai`, authMode: "oauth", allowPrivateNetwork: true, models: ["grok-stub"] }, + kimi: { adapter: "openai-chat", baseUrl: `${FAKE}/kimi`, authMode: "oauth", allowPrivateNetwork: true, models: ["kimi-stub"] }, + openai: { adapter: "openai-responses", baseUrl: "https://chatgpt.com/backend-api/codex", authMode: "forward", codexAccountMode: "pool" }, + }, + codexAccounts: [1, 2].map(n => ({ id: `acct${n}`, email: `c${n}@example.invalid`, isMain: false, chatgptAccountId: `cga${n}` })), + }); + // upstream: the pool logs an entitlement failover as "account unavailable (403); failing over". + const rotations = () => ocx.log.split("account unavailable (403); failing over").length - 1; + const failovers429 = () => ocx.log.split("; failing over to").length - 1; + try { + // Claude: unpinned control serves the active account; a pin serves exactly its own, both ways, and names it. + let r = await msg(ocx); + assert.equal(r.status, 200, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atB"]); + assert.equal(r.served, "B"); + r = await msg(ocx, pin("claude", "A")); + assert.equal(r.status, 200, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atA"]); + assert.equal(r.served, "A"); + r = await msg(ocx, pin("claude", "B")); + assert.equal(r.status, 200, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atB"]); + // A caller Anthropic key would take native passthrough; a pin never does. + r = await msg(ocx, { ...pin("claude", "A"), "x-api-key": "sk-ant-fake" }); + assert.equal(r.status, 200, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atA"]); + // Disconnected or missing pinned account: refused, never served by another account. + for (const id of ["C", "missing"]) { + r = await msg(ocx, pin("claude", id)); + assert.equal(r.status, 401, r.text); assert.deepEqual(inference(r), []); + assert.equal(r.served, null, "an answer no account served names none"); + } + // Malformed pins are refused, never served unpinned. + for (const h of [pin("gemini", "A"), { "x-agentradar-account-id": "A" }, pin("claude", "has space")]) { + r = await msg(ocx, h); + assert.equal(r.status, 400, r.text); assert.deepEqual(inference(r), []); + assert.equal(r.served, null); + } + // A pin cannot be spent on another provider's route. + for (const h of [pin("grok", "gA"), pin("kimi", "kA"), pin("codex", "acct1")]) { + r = await msg(ocx, h); + assert.equal(r.status, 404, r.text); assert.deepEqual(inference(r), []); + } + // A helper model runs outside the pin (the loopback describe carries no pin): refused, nothing sent. + const png = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=="; + const blindImage = { model: "claude-blind", max_tokens: 16, stream: true, messages: [{ role: "user", content: [ + { type: "image", source: { type: "base64", media_type: "image/png", data: png } }, { type: "text", text: "what is this" }] }] }; + // Control: unpinned, the same turn really does plan a describe, so the refusal below is not vacuous. + r = await call(ocx, "/v1/messages", blindImage, { "anthropic-version": "2023-06-01", "x-api-key": "unused" }); + assert.equal(r.status, 200, r.text); + assert.ok(inference(r).length >= 2 && !inference(r).at(-1)!.image, `no vision describe planned: ${JSON.stringify(r.upstream)}`); + r = await call(ocx, "/v1/messages", blindImage, { "anthropic-version": "2023-06-01", "x-api-key": "unused", ...pin("claude", "A") }); + assert.equal(r.status, 400, `${r.text}\n${JSON.stringify(r.upstream)}`); assert.deepEqual(inference(r), []); + // Surfaces that do not honour a pin refuse it instead of serving the active account. + r = await call(ocx, "/v1/responses", { model: "anthropic/claude-stub", input: "hi" }, pin("claude", "A")); + assert.equal(r.status, 400, r.text); assert.match(r.text, /only honoured on/); assert.deepEqual(inference(r), []); + // Claude Code's web search, as it sends it: the search helper runs on the pinned account too. + r = await call(ocx, "/v1/messages", { model: "claude-stub", max_tokens: 64, stream: true, + tools: [{ type: "web_search_20250305", name: "web_search", max_uses: 8 }], + messages: [{ role: "user", content: "Search the web for the latest agentradar release notes" }] }, + { "anthropic-version": "2023-06-01", "anthropic-beta": "claude-code-20250219", "x-api-key": "unused", ...pin("claude", "A") }); + assert.equal(r.status, 200, r.text); + assert.ok(r.upstream.some(s => s.search), `no search helper call: ${JSON.stringify(r.upstream)}`); + assert.ok(r.upstream.filter(s => !s.search).length >= 2, `turn did not resume after the search: ${JSON.stringify(r.upstream)}`); + assert.deepEqual([...new Set(tokens(r, "/v1/messages"))], ["Bearer atA"], JSON.stringify(r.upstream)); + // Grok and Codex have no helper that can run on their pin: still refused. + r = await call(ocx, "/v1/chat/completions", { model: "xai/grok-stub", messages: [{ role: "user", content: "hi" }], tools: [{ type: "web_search" }] }, pin("grok", "gA")); + assert.equal(r.status, 400, r.text); assert.match(r.text, /cannot use helper models/); assert.deepEqual(inference(r), []); + assert.equal((await accounts(ocx, "anthropic")).activeAccountId, "B", "pins never move the active account"); + + // Claude 429 on the pinned account: returned as is, cooldown recorded, no switch to B. + limited.add("Bearer atA"); + r = await msg(ocx, pin("claude", "A")); + assert.equal(r.status, 429, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atA"]); + // The rotator's own pin check: without it the caller picks another account and cancels this 429's body. + // upstream: Messages error-detail normalization rewrites the client message, so the 429 is proven by + // status, the single pinned send and the zero failover count instead of the provider wording. + assert.match(r.text, /rate_limit_error/); + await Bun.sleep(200); + assert.equal(failovers429(), 0, ocx.log); + const claude = await accounts(ocx, "anthropic"); + assert.equal(claude.activeAccountId, "B"); + assert.equal(claude.accounts.find((a: any) => a.id === "A")?.health?.status, "cooldown", JSON.stringify(claude)); + r = await msg(ocx); + assert.equal(r.status, 200, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atB"]); + + // Claude credential denial on the pinned account: returned as is, never moved to E; the unpinned control does fail over. + denied.add("Bearer atD"); + r = await msg(ocx, pin("claude", "D")); + assert.equal(r.status, 403, r.text); assert.deepEqual(tokens(r, "/v1/messages"), ["Bearer atD"]); + assert.equal((await accounts(ocx, "anthropic")).activeAccountId, "B"); + denied.add("Bearer atB"); + r = await msg(ocx); + assert.equal(r.status, 200, r.text); + const denialControl = tokens(r, "/v1/messages"); + assert.equal(denialControl[0], "Bearer atB"); assert.ok(denialControl.length > 1, `stock denial control did not rotate: ${denialControl}`); + // The pinned 403's rotation stopped at the reauth mark: only the control's failover was logged. + for (let i = 0; rotations() < 1 && i < 50; i++) await Bun.sleep(20); + await Bun.sleep(200); + assert.equal(rotations(), 1, ocx.log); + + // Grok (every call 429s): the pinned account alone, active unchanged; the unpinned control does rotate. + r = await chat(ocx, "xai/grok-stub", pin("grok", "gA")); + assert.equal(r.status, 429, `${r.text}\n${JSON.stringify(r.upstream)}`); assert.deepEqual(tokens(r, "/xai/"), ["Bearer tgA"]); + assert.equal((await accounts(ocx, "xai")).activeAccountId, "gB"); + r = await chat(ocx, "xai/grok-stub"); + assert.equal(r.status, 429, r.text); + const control = tokens(r, "/xai/"); + assert.equal(control[0], "Bearer tgB"); assert.ok(control.length > 1, `stock rotation control did not rotate: ${control}`); + + // Kimi (every call 429s): the pinned account alone, and the active account stays. + r = await chat(ocx, "kimi/kimi-stub", pin("kimi", "kA")); + assert.equal(r.status, 429, `${r.text}\n${JSON.stringify(r.upstream)}`); assert.deepEqual(tokens(r, "/kimi/"), ["Bearer tkA"]); + assert.equal((await accounts(ocx, "kimi")).activeAccountId, "kB"); + r = await chat(ocx, "xai/grok-stub", pin("kimi", "kA")); + assert.equal(r.status, 404, r.text); assert.deepEqual(inference(r), []); + + // Codex: OCX's exact-account route serves the pinned account; the active account is untouched. + for (const n of [1, 2]) { + r = await chat(ocx, "openai/gpt-5.5", pin("codex", `acct${n}`)); + assert.equal(r.status, 200, r.text); + assert.deepEqual(r.upstream.filter(s => s.path === "/backend-api/codex/responses").map(s => [s.auth, s.account]), [[`Bearer cat${n}`, `cga${n}`]]); + assert.equal(r.served, `acct${n}`, "a pinned chat answer names its account"); + } + r = await chat(ocx, "openai/gpt-5.5", pin("codex", "missing")); + assert.ok(r.status >= 400, r.text); assert.deepEqual(inference(r), []); + const active = await (await fetch(`${ocx.base}/api/codex-auth/active`, { headers: { authorization: `Bearer ${ocx.admin}` } })).json() as any; + assert.equal(active.activeCodexAccountId, "acct2", JSON.stringify(active)); + + // The internal headers never reached any upstream request. + assert.deepEqual(seen.filter(s => s.internal.length), []); + } finally { + ocx.proc.kill(); + } +}, 120_000); + +test("pooled turns log and answer their served account hashed; one request walks every spent account (failover.check)", async () => { + limited.clear(); denied.clear(); retryAfter = "318747"; + const ids = ["A", "B", "C", "D", "E", "F", "G", "H"]; + // The hosted gateway's pool defaults (AgentRadar images/ocx/pool-defaults.mjs). + const ocx = await startOcx({ + anthropic: { activeAccountId: "A", accounts: ids.map(id => ({ id, credential: cred(`at${id}`) })) }, + xai: { activeAccountId: "gA", accounts: ids.map(id => ({ id: `g${id}`, credential: cred(`tg${id}`) })) }, + }, { + defaultProvider: "anthropic", activeCodexAccountId: "acct1", + anthropicAccountPool: { enabled: true, strategy: "round-robin" }, accountPoolStrategy: "round-robin", clientIntegrations: { codex: false }, + providers: { + anthropic: { adapter: "anthropic", baseUrl: FAKE, authMode: "oauth", allowPrivateNetwork: true, models: ["claude-stub"] }, + xai: { adapter: "openai-chat", baseUrl: `${FAKE}/xai`, authMode: "oauth", allowPrivateNetwork: true, models: ["grok-stub"] }, + }, + }); + try { + // CACHE-ACCOUNT-AFFINITY: each turn logs its served account and session hashed, and answers the account's id. + const turns = [await msg(ocx, { "thread-id": "s1" }), await msg(ocx, { "thread-id": "s2" })]; + for (const t of turns) { + assert.equal(t.status, 200, t.text); + const servedId = ids.find(id => t.upstream[0]!.auth === `Bearer at${id}`)!; + expect(t.served).toBe(servedId); + } + for (let i = 0; (ocx.log.match(/\[ar-acct\]/g) ?? []).length < 2 && i < 50; i++) await Bun.sleep(20); + const logged = [...ocx.log.matchAll(/\[ar-acct\] a=([0-9a-f]{8}) r=(\S+) s=([0-9a-f]{8})\n/g)]; + expect(logged.map(m => m[1])).toEqual(turns.map(t => sha(t.served!).slice(0, 8))); + expect(logged.map(m => m[3])).toEqual([sha("s1").slice(0, 8), sha("s2").slice(0, 8)]); + expect(ocx.log).not.toContain("atA"); + + // GR2 10-09: stock OCX gave up after 3 rotations. With every account answering 429, one Claude request + // asks each account once; Grok's generic failover is bounded by OCX's shared request budget (5 sends). + for (const id of ids) limited.add(`Bearer at${id}`).add(`Bearer tg${id}`); + const claude = await msg(ocx, { "thread-id": "all-spent" }); + assert.equal(claude.status, 429, claude.text); + expect([...tokens(claude, "/v1/messages")].sort()).toEqual(ids.map(id => `Bearer at${id}`)); + const grok = await msg(ocx, { "thread-id": "grok-all-spent" }, "xai/grok-stub"); + assert.equal(grok.status, 429, grok.text); + // upstream: the hosted funded roster (oauthMaxAccountsPerRequest, 16) covers Grok's generic failover too, so + // one request asks each spent account once (.27 stopped at its 5-send request budget). Never a repeat. + expect(tokens(grok, "/xai/")).toHaveLength(8); + expect(new Set(tokens(grok, "/xai/")).size).toBe(8); + } finally { + ocx.proc.kill(); + } +}, 120_000); diff --git a/tests/server/health-scoring.test.ts b/tests/server/health-scoring.test.ts index d1458c1454a..fb0286bf130 100644 --- a/tests/server/health-scoring.test.ts +++ b/tests/server/health-scoring.test.ts @@ -1,5 +1,5 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test"; -import { mkdtempSync} from "node:fs"; +import { mkdtempSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { appendUsageEntry, resetUsageReadCacheForTests, type PersistedUsageEntry } from "../../src/usage/log"; @@ -247,6 +247,55 @@ describe("health-aware scoring (RI-06)", () => { expect(finalTarget.successRate).toBe(1); }); + test("latest same-route physical attempt ends the failure streak", () => { + appendUsageEntry({ + ...row("same-route-retry", 200, 1500, { + provider: "combo", + model: "waterfall", + }), + attempts: [ + { ordinal: 1, provider: "a", model: "m1", adapter: "openai-chat", status: 429, durationMs: 1000, sendCount: 1, recoveryKinds: [], usageStatus: "unreported" }, + { ordinal: 2, provider: "a", model: "m1", adapter: "openai-chat", status: 200, durationMs: 500, sendCount: 1, recoveryKinds: [], usageStatus: "reported" }, + ], + }); + + const evidence = healthEvidenceForCandidate({ provider: "a", model: "m1" }); + expect(evidence).toMatchObject({ sampleCount: 2, successRate: 0.5 }); + expect(evidence.failures).toBeUndefined(); + }); + test("single combo attempt contributes health to its physical target", () => { + appendUsageEntry({ + ...row("combo-single", 200, 1500, { + provider: "combo", + model: "waterfall", + }), + attempts: [ + { ordinal: 1, provider: "a", model: "m1", adapter: "openai-chat", status: 200, durationMs: 1500, sendCount: 1, recoveryKinds: [], usageStatus: "reported" }, + ], + }); + + expect(healthEvidenceForCandidate({ provider: "a", model: "m1" })).toMatchObject({ + sampleCount: 1, + successRate: 1, + }); + }); + test("historical attempt health counts only physical sends and unique ordinals", () => { + const entry = row("physical-attempts", 200, 1500, { + attempts: [ + { ordinal: 1, provider: "a", model: "m1", adapter: "openai-chat", status: 200, durationMs: 1, sendCount: 0, locallyAnswered: true, recoveryKinds: [], usageStatus: "unreported" }, + { ordinal: 1, provider: "a", model: "m1", adapter: "openai-chat", status: 503, durationMs: 4000, sendCount: 1, recoveryKinds: [], usageStatus: "unreported" }, + { ordinal: 1, provider: "a", model: "m1", adapter: "openai-chat", status: 200, durationMs: 2, sendCount: 1, recoveryKinds: [], usageStatus: "unreported" }, + { ordinal: 2, provider: "a", model: "m1", adapter: "openai-chat", status: 503, durationMs: 3, sendCount: 0, recoveryKinds: [], usageStatus: "unreported" }, + { ordinal: 3, provider: "a", model: "m1", adapter: "openai-chat", status: 200, durationMs: 1000, sendCount: 1, recoveryKinds: [], usageStatus: "reported" }, + ], + }); + writeFileSync(join(testDir, "usage.jsonl"), `${JSON.stringify(entry)}\n`); + + const evidence = healthEvidenceForCandidate({ provider: "a", model: "m1" }); + + expect(evidence.sampleCount).toBe(2); + expect(evidence.successRate).toBeCloseTo(0.5, 4); + }); test("a stream that dies after its 200 head is a failure for the attempt that dropped it", () => { for (let index = 0; index < 5; index++) appendUsageEntry(row(`ok-${index}`, 200, 800)); // b/m2 failed over to a/m1, whose 200 stream was then cut mid-flight. diff --git a/tests/server/image-bridge-runturn-pacing.test.ts b/tests/server/image-bridge-runturn-pacing.test.ts new file mode 100644 index 00000000000..40e7c1cab4c --- /dev/null +++ b/tests/server/image-bridge-runturn-pacing.test.ts @@ -0,0 +1,80 @@ +// .27 parity (tests/images/loop.test.ts "runTurn adapter paces and records every physical dispatch"), +// at the server seam: a forward-auth runTurn provider is not wrapped by request-transport's +// runSelectedTurn, so the image bridge's runTurnFetch is what paces its sends inside the turn lease. +import { afterAll, afterEach, beforeAll, beforeEach, expect, mock, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { ProviderAdapter } from "../../src/adapters/base"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest } from "../../src/providers/request-pacing"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const previousHome = process.env.OPENCODEX_HOME; +let home = ""; +let release: (() => void) | undefined; +let restore: (() => void) | undefined; +let handleResponses: typeof import("../../src/server/responses")["handleResponses"]; +let upstream: ReturnType; +let sends = 0; +let inFlightDuringTurn: number[] = []; +let config: OcxConfig; + +function runTurnAdapter(provider: OcxProviderConfig): ProviderAdapter { + return { + name: "openai-chat", + buildRequest: () => ({ url: provider.baseUrl, method: "POST", headers: {}, body: "{}" }), + async *parseStream() { yield { type: "done" as const }; }, + async runTurn(_parsed, incoming, emit) { + if (!incoming.providerFetch) throw new Error("missing provider fetch"); + for (let i = 0; i < 3; i++) { + await (await incoming.providerFetch(`${provider.baseUrl}/send-${i}`, { method: "POST", body: "{}" })).text(); + inFlightDuringTurn.push(providerRequestPacingStatus("rtpaced", config.providers.rtpaced!).inFlight); + } + emit({ type: "text_delta", text: "sent" }); + emit({ type: "done" }); + }, + } as ProviderAdapter; +} + +beforeAll(async () => { + upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => { sends += 1; return new Response("{}"); } }); + const actual = { ...await import("../../src/server/adapter-resolve") }; + restore = () => mock.module("../../src/server/adapter-resolve", () => actual); + mock.module("../../src/server/adapter-resolve", () => ({ + ...actual, + resolveAdapter: (provider: OcxProviderConfig, retention?: "none" | "short" | "long", name?: string) => + provider.adapter === "test-runturn-image" ? runTurnAdapter(provider) : actual.resolveAdapter(provider, retention, name), + })); + ({ handleResponses } = await import("../../src/server/responses")); +}); +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "ocx-image-runturn-")); process.env.OPENCODEX_HOME = home; + release = acquireOwnedSpendHome(); resetProviderRequestPacingForTest(); sends = 0; inFlightDuringTurn = []; + config = { + port: 0, defaultProvider: "rtpaced", images: { bridgeEnabled: true }, + providers: { + rtpaced: { adapter: "test-runturn-image" as OcxProviderConfig["adapter"], baseUrl: `${upstream.url.toString().replace(/\/$/, "")}/v1`, + authMode: "forward", allowPrivateNetwork: true, models: ["m"], requestPacing: { enabled: true, maxConcurrentRequests: 1 } }, + xai: { adapter: "openai-chat", baseUrl: "https://api.x.ai/v1", apiKey: "synthetic-xai-key", models: ["grok-imagine-image"] }, + }, + } as OcxConfig; +}); +afterEach(() => { + release?.(); release = undefined; resetProviderRequestPacingForTest(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previousHome; + removeTreeWithRetry(home); +}); +afterAll(async () => { mock.restore(); restore?.(); await upstream.stop(true); }); + +test("a forward-auth runTurn provider paces every image-bridge send inside its turn lease", async () => { + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json", authorization: "Bearer synthetic-caller" }, + body: JSON.stringify({ model: "rtpaced/m", input: "draw", stream: true, tools: [{ type: "image_generation" }] }), + }), config, { model: "", provider: "" }); + expect(await response.text()).toContain("sent"); + expect(sends).toBe(3); + expect(inFlightDuringTurn).toEqual([1, 1, 1]); + expect(providerRequestPacingStatus("rtpaced", config.providers.rtpaced!).inFlight).toBe(0); +}); diff --git a/tests/server/loopback-listener-admission.test.ts b/tests/server/loopback-listener-admission.test.ts index ea3075085f7..d4a743a1063 100644 --- a/tests/server/loopback-listener-admission.test.ts +++ b/tests/server/loopback-listener-admission.test.ts @@ -73,7 +73,7 @@ describe("loopback listener policy view", () => { expect(chatStart).toBeGreaterThan(messagesStart); expect(source.slice(countTokensStart, messagesStart)).toContain( // First-party bindings ride on the same call: only the intercept ingress sets the flag. - 'await handleClaudeCountTokens(req, config, policy, { claudeIntercept: ingress === "claude-intercept" })', + 'await handleClaudeCountTokens(req, config, policy, { claudeIntercept: ingress === "claude-intercept", admission })', ); expect(source.slice(messagesStart, chatStart)).toContain( 'await handleClaudeMessages(sessionReq, config, logCtx, { requestId, start, turnAdmissionLease, admission, advisorAdmissionPolicy }, policy, { claudeIntercept: ingress === "claude-intercept" })', diff --git a/tests/server/management-api-logs-metrics-physical.test.ts b/tests/server/management-api-logs-metrics-physical.test.ts new file mode 100644 index 00000000000..ccbaab523dd --- /dev/null +++ b/tests/server/management-api-logs-metrics-physical.test.ts @@ -0,0 +1,291 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { existsSync, mkdtempSync, readFileSync} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { handleManagementAPI } from "../../src/server/management-api"; +import { usageLogPath } from "../../src/usage/log"; +import { + addRequestLog, + clearRequestLogsForTests, + evictOldestRequestLogForBudget, + getRequestLogEntries, + type RequestLogEntry, +} from "../../src/server/request-log"; +import type { OcxConfig } from "../../src/types"; +import { buildRouteDecisionTrace } from "../../src/routing/trace"; +import { summarizeUsage } from "../../src/usage/summary"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { refreshUserCostOverlays } from "../../src/usage/user-cost-overlays"; + +interface LogPollEnvelope { + logs: Array>; + cursor: string; + reset: boolean; + generatedAt: number; + timeZone: string; + total: number; +} + +async function readLogPoll(query = "", cursor?: string): Promise { + const url = new URL(`http://localhost/api/logs?${query}`); + if (cursor) url.searchParams.set("cursor", cursor); + const before = Date.now(); + const response = await handleManagementAPI(new Request(url), url, config); + expect(response?.status).toBe(200); + const body = await response!.json() as LogPollEnvelope; + expect(body.generatedAt).toBeGreaterThanOrEqual(before); + expect(body.generatedAt).toBeLessThanOrEqual(Date.now()); + expect(body.timeZone).toBe(Intl.DateTimeFormat().resolvedOptions().timeZone); + expect(typeof body.cursor).toBe("string"); + return body; +} + +const config = { providers: [] } as unknown as OcxConfig; + +let testDir = ""; +let previousHome: string | undefined; + +beforeEach(() => { + // The request log is process-wide: start empty so the first case does not read a row an + // earlier file left behind (a one-process tests/server run handed it a Kiro entry). + clearRequestLogsForTests(); + // addRequestLog persists to usage.jsonl; without a scratch OPENCODEX_HOME a bare + // `bun test ` run from outside the repo (no bunfig preload) writes these + // fixture rows into the real ~/.opencodex log and poisons the GUI Usage page. + previousHome = process.env.OPENCODEX_HOME; + testDir = mkdtempSync(join(tmpdir(), "ocx-logs-metrics-")); + process.env.OPENCODEX_HOME = testDir; +}); + +afterEach(() => { + clearRequestLogsForTests(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + if (testDir) removeTreeWithRetry(testDir); +}); + +async function readLogs(): Promise>> { + const url = new URL("http://localhost/api/logs"); + const response = await handleManagementAPI(new Request(url), url, config); + expect(response?.status).toBe(200); + const body = await response!.json() as { logs?: Array>; timeZone?: string }; + expect(typeof body.timeZone).toBe("string"); + expect(body.timeZone!.length).toBeGreaterThan(0); + return body.logs ?? []; +} + +function baseEntry(overrides: Partial): RequestLogEntry { + return { + requestId: `req-${Math.random().toString(36).slice(2)}`, + timestamp: Date.now(), + model: "claude-3-haiku-20240307", + provider: "anthropic", + status: 200, + durationMs: 2000, + usageStatus: "reported", + ...overrides, + }; +} + +describe("GET /api/logs display metrics", () => { + + + + + + + + + test("unmeasured counters are unavailable for both metrics", async () => { + for (const usageStatus of ["unreported", "unsupported"] as const) { + addRequestLog(baseEntry({ + requestId: usageStatus, + usageStatus, + usage: { inputTokens: 100, outputTokens: 10 }, + })); + } + const logs = await readLogs(); + const byStatus = new Map(logs.map(row => [row.usageStatus, row])); + expect(byStatus.get("unreported")!.displayMetrics).toEqual({ + tokPerSecond: { kind: "unavailable", reason: "usage_missing" }, + decodeTokPerSecond: { kind: "unavailable", reason: "ttft_missing" }, + cost: { kind: "unavailable", reason: "usage_missing" }, + }); + expect(byStatus.get("unsupported")!.displayMetrics).toEqual({ + tokPerSecond: { kind: "unavailable", reason: "usage_unsupported" }, + decodeTokPerSecond: { kind: "unavailable", reason: "usage_unsupported" }, + cost: { kind: "unavailable", reason: "usage_unsupported" }, + }); + }); + + + + test("partial combo keeps measured attempt metrics without pricing unreported counters", async () => { + addRequestLog(baseEntry({ + provider: "combo", + model: "combo/my-combo", + usageStatus: "unreported", + usage: { inputTokens: 200, outputTokens: 20 }, + attempts: [ + { + ordinal: 1, + provider: "anthropic", + model: "claude-3-haiku-20240307", + adapter: "anthropic", + status: 429, + durationMs: 900, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported", + usage: { inputTokens: 100, outputTokens: 10 }, + }, + { + ordinal: 2, + provider: "anthropic", + model: "claude-3-haiku-20240307", + adapter: "anthropic", + status: 200, + durationMs: 1100, + sendCount: 1, + recoveryKinds: [], + usageStatus: "unreported", + usage: { inputTokens: 100, outputTokens: 10 }, + }, + ], + })); + const [dto] = await readLogs(); + expect(dto!.displayMetrics.tokPerSecond).toEqual({ kind: "unavailable", reason: "usage_missing" }); + expect(dto!.displayMetrics.cost).toEqual({ kind: "unavailable", reason: "combo_attempt_unavailable" }); + expect(dto!.attempts[0].displayMetrics.cost.kind).toBe("value"); + expect(dto!.attempts[0].displayMetrics.tokPerSecond.kind).toBe("value"); + expect(dto!.attempts[1].displayMetrics.cost).toEqual({ kind: "unavailable", reason: "usage_missing" }); + expect(dto!.attempts[1].displayMetrics.tokPerSecond).toEqual({ kind: "unavailable", reason: "usage_missing" }); + }); + + test("aggregate cost excludes local, unsent, and duplicate attempts", async () => { + const physical = { + ordinal: 1, + provider: "anthropic", + model: "claude-3-haiku-20240307", + adapter: "anthropic" as const, + status: 200, + durationMs: 900, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported" as const, + usage: { inputTokens: 100, outputTokens: 10 }, + }; + addRequestLog(baseEntry({ + provider: "combo", + model: "combo/my-combo", + usage: { inputTokens: 999, outputTokens: 999 }, + attempts: [ + physical, + { ...physical, ordinal: 2, provider: "local-only", locallyAnswered: true }, + { ...physical, ordinal: 3, provider: "unsent", sendCount: 0 }, + { ...physical, provider: "duplicate" }, + ], + })); + + const [dto] = await readLogs(); + expect(dto!.attempts).toHaveLength(4); + expect(dto!.displayMetrics.cost.kind).toBe("value"); + expect(dto!.displayMetrics.cost.estimate.cost.total) + .toBe(dto!.attempts[0].displayMetrics.cost.estimate.cost.total); + }); + + test("uses legacy root usage to price its final physical attempt", async () => { + addRequestLog(baseEntry({ + provider: "combo", + model: "combo/my-combo", + usage: { inputTokens: 100, outputTokens: 10 }, + attempts: [{ + ordinal: 1, + provider: "anthropic", + model: "claude-3-haiku-20240307", + adapter: "anthropic", + status: 200, + durationMs: 900, + sendCount: 1, + recoveryKinds: [], + usageStatus: "unreported", + }], + })); + + const [dto] = await readLogs(); + expect(dto!.displayMetrics.cost.kind).toBe("value"); + expect(dto!.displayMetrics.cost.estimate.attempts).toHaveLength(1); + }); + + test("explicit all-nonphysical attempts do not reuse stale root cost", async () => { + const nonphysical = { + ordinal: 1, + provider: "anthropic", + model: "claude-3-haiku-20240307", + adapter: "anthropic" as const, + status: 200, + durationMs: 900, + sendCount: 0, + recoveryKinds: [], + usageStatus: "reported" as const, + usage: { inputTokens: 100, outputTokens: 10 }, + }; + addRequestLog(baseEntry({ + usage: { inputTokens: 10_000, outputTokens: 1_000 }, + attempts: [nonphysical, { ...nonphysical, ordinal: 2, sendCount: 1, locallyAnswered: true }], + })); + + const [dto] = await readLogs(); + expect(dto!.displayMetrics.cost).toEqual({ kind: "unavailable", reason: "usage_missing" }); + }); + + + +}); +import { ManagementRequest as Request } from "../helpers/management-auth"; + + +describe("GET /api/logs snapshot polling", () => { + beforeEach(() => clearRequestLogsForTests()); + + + + +}); + +/** + * #4057: the account label was persisted on the row and on every attempt long before anything + * could read it back. `requestLogDto` carries it only because it spreads the entry — the sibling + * projection `requestLogEntryFromPersistedUsage` rebuilds field by field and warns in its own + * comment that a field missing there never reaches usage.jsonl. These assertions pin the served + * contract so a future field-by-field rewrite of the DTO cannot drop the label silently. + */ +describe("GET /api/logs account identity", () => { + beforeEach(() => clearRequestLogsForTests()); + +}); + +/** + * #4038 — Logs showed one rate that conflates first-token latency with delivery speed. + * `tokensPerSecond` never subtracted TTFT, and the MetricSource Pick did not even include + * `firstOutputMs`, so a decode-rate metric could not be computed at all. + * + * The history matters more than the arithmetic here. Contributor PR #4040 implemented this exact + * metric and was closed unmerged as an unreliable estimate: proxy TTFT is not the provider's + * generation window, and a small post-TTFT remainder makes the number explode. The issue stayed + * open, so the repository held both an acceptance criterion and a rejection of the same feature. + * + * MIN_DECODE_WINDOW_MS is what answers that rejection, and + * "a decode window under the floor yields no value" is the assertion that proves it. Everything + * else here is scaffolding around that one case. + */ +describe("estimated decode rate (#4038)", () => { + + + + + + + + +}); diff --git a/tests/server/route-decision-trace.test.ts b/tests/server/route-decision-trace.test.ts index 6918a9f77b9..7b054110458 100644 --- a/tests/server/route-decision-trace.test.ts +++ b/tests/server/route-decision-trace.test.ts @@ -141,6 +141,33 @@ describe("route decision traces (RI-01)", () => { } }); + test("combo route records request-level incompatibility in candidate evidence", () => { + const route = routeModel(baseConfig(), "combo/free", undefined, { + comboEligible: target => target.provider === "b", + }); + expect(route.providerName).toBe("b"); + expect(route.routeDecision!.candidates[0]).toMatchObject({ + provider: "a", + eligible: false, + exclusions: [{ code: "request-incompatible" }], + }); + }); + + test("combo route does not label operational ineligibility as request incompatibility", () => { + const options = { + comboEligible: (target: { provider: string }) => target.provider === "b", + comboRequestCompatible: () => true, + } as Parameters[3] & { + comboRequestCompatible: (target: { provider: string }) => boolean; + }; + const route = routeModel(baseConfig(), "combo/free", undefined, options); + const candidate = route.routeDecision!.candidates[0]!; + + expect(route.providerName).toBe("b"); + expect(candidate.eligible).toBe(false); + expect(candidate.exclusions.map(exclusion => exclusion.code)).not.toContain("request-incompatible"); + }); + test("default-provider fallback records a default-provider trace", () => { const route = routeModel(baseConfig(), "totally-unknown-model"); expect(route.providerName).toBe("a"); diff --git a/tests/server/server-auth.test.ts b/tests/server/server-auth.test.ts index 8aadb2aae3a..7983c3f9072 100644 --- a/tests/server/server-auth.test.ts +++ b/tests/server/server-auth.test.ts @@ -2335,6 +2335,16 @@ describe("server local API auth", () => { expect.objectContaining({ status: 400, authKind: "pool", credentialSubstituted: true }), expect.objectContaining({ status: 200, authKind: "pool", credentialSubstituted: true }), ]); + expect(getRequestLogEntries().at(-1)?.attempts).toMatchObject([ + { ordinal: 1, status: 400, sendCount: 1 }, + { + ordinal: 2, + status: 200, + sendCount: 1, + recoveryKinds: ["codex-account-retry"], + recoveryCount: 1, + }, + ]); } finally { await stopPoolRetryHarness(harness); } @@ -2519,6 +2529,13 @@ describe("server local API auth", () => { expect(response.status).toBe(200); expect((await response.json() as { id: string }).id).toBe("same-account-success"); expect(harness.dispatches).toEqual(["acct-pool-a", "acct-pool-a", "acct-pool-a"]); + expect(getRequestLogEntries().at(-1)?.attempts).toMatchObject([{ + ordinal: 1, + status: 200, + sendCount: 3, + recoveryKinds: ["codex-account-retry"], + recoveryCount: 2, + }]); } finally { await stopPoolRetryHarness(harness); } @@ -2564,6 +2581,13 @@ describe("server local API auth", () => { expect(response.status).toBe(400); expect(await response.text()).toBe(body); expect(harness.dispatches).toEqual(Array(8).fill("acct-pool-a")); + expect(getRequestLogEntries().at(-1)?.attempts).toMatchObject([{ + ordinal: 1, + status: 400, + sendCount: 8, + recoveryKinds: ["codex-account-retry"], + recoveryCount: 7, + }]); } finally { await stopPoolRetryHarness(harness); } diff --git a/tests/server/server-combo-failover-advance.test.ts b/tests/server/server-combo-failover-advance.test.ts new file mode 100644 index 00000000000..f90976f51fa --- /dev/null +++ b/tests/server/server-combo-failover-advance.test.ts @@ -0,0 +1,458 @@ +import { registerComboForcedEffortCases } from "../helpers/combo-forced-effort-cases"; +import { registerComboToolRoutingCases } from "../helpers/combo-tool-routing-cases"; +import { comboProviderFactory } from "../helpers/combo-provider"; +import { registerComboContextOverflowCases } from "../helpers/combo-context-overflow-cases"; +import { registerComboContextHeadroomCases } from "../helpers/combo-context-headroom-cases"; +import { sessionLaneIdFromRequest } from "../../src/server/request-log-conversation"; +import { afterAll, afterEach, beforeEach, describe, expect, mock, setDefaultTimeout, test } from "bun:test"; +import { logsFromApiBody } from "../helpers/logs-api"; +import { computeRoutingAnalytics } from "../../src/routing/analytics"; +import { managementFetch as fetch, ManagementRequest as Request } from "../helpers/management-auth"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + clearComboSelectionState, + clearComboTargetCooldowns, + isComboTargetInCooldown, +} from "../../src/combos"; +import { readConfigDiagnostics, saveConfig } from "../../src/config"; +import type { ProviderAdapter } from "../../src/adapters/base"; +import { handleManagementAPI } from "../../src/server/management-api"; +import { saveCredential } from "../../src/oauth/store"; +import { XAI_OAUTH_DISCOVERY_URL } from "../../src/oauth/xai"; +import { XAI_GROK_CLI_BASE_URL } from "../../src/providers/xai-transport"; +import type { AdapterEvent, OcxConfig, OcxProviderConfig, OcxProviderContinuationState } from "../../src/types"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; +import { clearRequestLogsForTests, hydrateRequestLogsFromDisk, httpStatusForRequestLogTerminal, inspectResponseLogSsePayload, type RequestLogContext } from "../../src/server/request-log"; +import { responseWithDeferredRequestLog } from "../../src/server/relay"; +import { readUsageEntries } from "../../src/usage/log"; +import { saveCodexAccountCredential } from "../../src/codex/account-store"; +import { + clearCodexUpstreamHealth, + formatCodexProviderForLog, + getCodexUpstreamHealth, + getCodexQuotaHealthSnapshot, + codexQuotaScopeForModel, +} from "../../src/codex/routing"; +import { clearAccountQuota, getAccountQuota } from "../../src/codex/quota"; +import { startServer } from "../../src/server"; +import { fakeChatGptJwt } from "../helpers/fake-chatgpt-jwt"; +import { catalogConvergenceFactory } from "../helpers/catalog-convergence"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { chatErrorStream, chatStream, chatSuccess, chatTruncatedZeroOutputStream, responsesSuccess } from "../helpers/combo-failover-upstream"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { heldResponse } from "../helpers/held-response"; +import { + clearResponseStateForTests, + flushResponseState, + responseStatePersistPendingForTests, +} from "../../src/responses/state"; +import { clearCursorThreadContinuityForTests } from "../../src/adapters/cursor/thread-continuity"; +import { COMPACT_PROMPT, encodeCompactionSummary } from "../../src/responses/compaction"; +import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import { consumeComboFailure, createChildPassthroughCallbackGate } from "../../src/server/responses/core"; +import { clearComboRecallForTests, recallComboForLane, reconcileComboRecall } from "../../src/server/responses/combo-session-recall"; +import { captureConfigGeneration } from "../../src/lib/state-store-sweeper"; + +// Full-suite Windows load: startServer + combo rename/delete management flows exceed the +// default 5s per-test budget (same flake class as 810fa115 / claude-management-api). +setDefaultTimeout(30_000); + +// `mock.module` outlives this file: Bun keeps both overrides below for every file that runs after +// this one in the same process. These are spread snapshots of the real modules, taken before them. +const actualResolver = { ...(await import("../../src/server/adapter-resolve")) }; +const actualResolveAdapter = actualResolver.resolveAdapter; +const actualRetry = { ...(await import("../../src/lib/upstream-retry")) }; +const actualFetchWithTransientRetry = actualRetry.fetchWithTransientRetry; +const { createCursorAdapter } = await import("../../src/adapters/cursor"); +import type { CursorTransportFactory } from "../../src/adapters/cursor/transport"; +let customRunTurn: NonNullable | undefined; +let customFetchResponse: NonNullable | undefined; +const provider = comboProviderFactory(() => customFetchResponse); +let customTransientResponse: (() => Promise) | undefined; +let customUsageEstimate: ((model: string) => number | undefined) | undefined; +let customCursorTransportFactory: CursorTransportFactory | undefined; + +mock.module("../../src/server/adapter-resolve", () => ({ + ...actualResolver, + resolveAdapter(provider: OcxProviderConfig, cacheRetention?: "none" | "short" | "long") { + if (provider.adapter === "cursor" && customCursorTransportFactory) { + // Real cursor adapter (adapter.name === "cursor") over a fake transport, so server-level + // tests can drive the genuine continuation/persistence policy without a live socket. + return createCursorAdapter(provider, { createTransport: customCursorTransportFactory }); + } + if ( + provider.adapter === "test-run-turn" + || provider.adapter === "test-kiro" + || provider.adapter === "test-owned" + ) { + const adapter: ProviderAdapter = { + name: provider.adapter === "test-kiro" ? "kiro" : provider.adapter, + buildRequest: () => ({ url: provider.baseUrl, method: "POST", headers: {}, body: "" }), + async *parseStream(): AsyncGenerator { + yield { type: "error", message: "test runTurn adapter does not use parseStream" }; + }, + async runTurn(parsed, incoming, emit) { + if (!customRunTurn) throw new Error("custom runTurn not installed"); + await customRunTurn(parsed, incoming, emit); + }, + }; + return adapter; + } + if (provider.adapter === "test-response") { + const base = actualResolveAdapter({ ...provider, adapter: "openai-chat" }, cacheRetention); + return { + ...base, + name: "test-response", + async buildRequest(parsed, options) { + const request = await base.buildRequest(parsed, options); + const estimate = customUsageEstimate?.(parsed.modelId); + return estimate === undefined + ? request + : { ...request, usageLog: { inputTokens: estimate } }; + }, + async fetchResponse(request, context) { + if (!customFetchResponse) throw new Error("custom fetchResponse not installed"); + return context!.executor!(request.url, { method: request.method, headers: request.headers, + body: request.body, signal: context?.abortSignal }); + }, + }; + } + return actualResolveAdapter(provider, cacheRetention); + }, +})); + +mock.module("../../src/lib/upstream-retry", () => ({ + ...actualRetry, + fetchWithTransientRetry( + ...args: Parameters + ): ReturnType { + if (customTransientResponse) return customTransientResponse(); + return actualFetchWithTransientRetry(...args); + }, +})); + +afterAll(() => { // Put the real modules back for every later file in the same process. + mock.module("../../src/server/adapter-resolve", () => actualResolver); + mock.module("../../src/lib/upstream-retry", () => actualRetry); +}); + +const { handleResponses } = await import("../../src/server/responses"); +const { handleResponsesCompact } = await import("../../src/server/responses/compact"); +type HandleOptions = NonNullable[3]>; + +const TOKEN_ENDPOINT = "https://auth.x.ai/oauth/token"; +const XAI_CHAT_ENDPOINT = `${XAI_GROK_CLI_BASE_URL}/chat/completions`; + +let testDir = ""; +let previousHome: string | undefined; +let previousCursorToken: string | undefined; +let isolatedCodexHome: IsolatedCodexHome | null = null; +let originalFetch: typeof fetch; +let originalNow: () => number; +const servers: Array> = []; + +// A case that calls a handler directly never takes the writer lease startServer takes, so its +// dispatch is refused. Taken at the dispatch helpers rather than file-wide: the cases that do +// start a real server already own it, and this only ever adds a reference on the same home. +let releaseSpendHome: (() => void) | undefined; +const takeSpendHome = (): void => { releaseSpendHome ??= acquireOwnedSpendHome(); }; +// Every turn these helpers hand back. A row that asserts only on a status leaves a +// transformed body unread, and cancelling those before the listeners stop is what keeps a +// reader from settling against the journal after its lease is gone. +const pendingTurns: Response[] = []; +function trackTurn(response: Response): Response { pendingTurns.push(response); return response; } + +beforeEach(() => { + originalFetch = globalThis.fetch; + originalNow = Date.now; + previousHome = process.env.OPENCODEX_HOME; + previousCursorToken = process.env.OPENCODEX_CURSOR_TEST_TOKEN; + delete process.env.OPENCODEX_CURSOR_TEST_TOKEN; + isolatedCodexHome = installIsolatedCodexHome("ocx-combo-030-codex-"); + testDir = mkdtempSync(join(tmpdir(), "ocx-combo-030-")); + process.env.OPENCODEX_HOME = testDir; + clearComboSelectionState(); + clearComboRecallForTests(); + clearComboTargetCooldowns(); + clearKeyCooldowns(); + clearCodexUpstreamHealth(); + customRunTurn = undefined; + customFetchResponse = undefined; + customTransientResponse = undefined; + customUsageEstimate = undefined; + customCursorTransportFactory = undefined; + clearRequestLogsForTests(); + clearResponseStateForTests(); + clearCursorThreadContinuityForTests(); +}); + +afterEach(async () => { + let responseStatePending = true; + let cancelFailure: unknown; + try { + // Rows that assert only on a status leave a transformed body unread. Cancelling those first + // means no reader is still attached when the listeners stop and the lease is given back. A + // cancel that throws fails the case rather than being swallowed, but not before every other + // turn and the listeners below have had their chance to close. + for (const turn of pendingTurns.splice(0)) { + if (turn.bodyUsed || !turn.body || turn.body.locked) continue; + try { await turn.body.cancel(); } + catch (error) { cancelFailure ??= error; } + } + for (const server of servers.splice(0)) await server.stop(true); + await flushResponseState(); + responseStatePending = responseStatePersistPendingForTests(); + } finally { + // After the listeners are stopped and the response state is flushed, both of which can + // still account against the journal, and before the home below is removed. + releaseSpendHome?.(); + releaseSpendHome = undefined; + clearResponseStateForTests(); + clearCursorThreadContinuityForTests(); + globalThis.fetch = originalFetch; + Date.now = originalNow; + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + if (previousCursorToken === undefined) delete process.env.OPENCODEX_CURSOR_TEST_TOKEN; + else process.env.OPENCODEX_CURSOR_TEST_TOKEN = previousCursorToken; + isolatedCodexHome?.restore(); + isolatedCodexHome = null; + if (testDir) removeTreeWithRetry(testDir); + clearComboSelectionState(); + clearComboRecallForTests(); + clearComboTargetCooldowns(); + clearKeyCooldowns(); + clearCodexUpstreamHealth(); + clearRequestLogsForTests(); + } + if (cancelFailure !== undefined) throw cancelFailure; + expect(responseStatePending).toBe(false); +}); + +function serve(handler: (request: Request) => Response | Promise) { + const server = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: handler }); + servers.push(server); + return server; +} + +function baseUrl(server: ReturnType): string { + return `${server.url.toString().replace(/\/$/, "")}/v1`; +} + +function comboConfig( + providers: OcxConfig["providers"], + targets = Object.keys(providers).map((name, index) => ({ provider: name, model: `m${index + 1}` })), + extra: Partial[string]> = {}, +): OcxConfig { + return { + port: 0, + defaultProvider: Object.keys(providers)[0]!, + providers, + combos: { free: { strategy: "failover", targets, ...extra } }, + }; +} + +async function post( + config: OcxConfig, + raw: Record = {}, + options: HandleOptions = {}, + headers: Record = {}, +): Promise { + takeSpendHome(); + return trackTurn(await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json", ...headers }, + body: JSON.stringify({ model: "combo/free", input: "hello", stream: false, ...raw }), + }), config, { model: "", provider: "" }, options)); +} + +let loggedRequestSequence = 0; + +async function postLogged( + config: OcxConfig, + raw: Record = {}, + options: HandleOptions = {}, + headers: Record = {}, +): Promise { + const logCtx: RequestLogContext = { model: "", provider: "" }; + const start = Date.now(); + takeSpendHome(); + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json", ...headers }, + body: JSON.stringify({ model: "combo/free", input: "hello", stream: false, ...raw }), + }), config, logCtx, options); + loggedRequestSequence += 1; + return trackTurn(responseWithDeferredRequestLog( + response, + `combo-test-${loggedRequestSequence}`, + start, + logCtx, + )); +} + +async function postModelLogged( + config: OcxConfig, + model: string, + raw: Record = {}, + options: HandleOptions = {}, + headers: Record = {}, +): Promise { + const logCtx: RequestLogContext = { model: "", provider: "" }; + const start = Date.now(); + takeSpendHome(); + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json", ...headers }, + body: JSON.stringify({ model, input: "hello", stream: false, ...raw }), + }), config, logCtx, options); + loggedRequestSequence += 1; + return trackTurn(responseWithDeferredRequestLog( + response, + `direct-test-${loggedRequestSequence}`, + start, + logCtx, + )); +} + +async function latestAttemptReceipts(config: OcxConfig) { + const response = await management(config, "GET", "/api/logs?tail=1"); + const logs = logsFromApiBody(await response!.json()); + const usage = readUsageEntries(); + return { log: logs[0]!, usage: usage.at(-1)! }; +} + +async function expectCancelledAttemptReceipt( + config: OcxConfig, + expected: { provider: string; model: string; adapter: string }, +): Promise { + const { log, usage } = await latestAttemptReceipts(config); + for (const receipt of [log, usage]) { + expect(receipt).toMatchObject({ + provider: "combo", + model: "combo/free", + attempts: [{ ...expected, status: 499 }], + }); + expect((receipt.attempts as unknown[])).toHaveLength(1); + } +} + +interface SseFrame { + event?: string; + data: Record; +} + +async function collectSse(response: Response): Promise { + const text = await response.text(); + return text.split("\n\n").flatMap(block => { + if (!block.trim()) return []; + let event: string | undefined; + let data = ""; + for (const line of block.split("\n")) { + if (line.startsWith("event:")) event = line.slice(6).trim(); + if (line.startsWith("data:")) data += line.slice(5).trim(); + } + if (!data || data === "[DONE]") return []; + try { + return [{ ...(event ? { event } : {}), data: JSON.parse(data) as Record }]; + } catch { + return []; + } + }); +} + +async function management( + config: OcxConfig, + method: string, + path: string, + body?: unknown, +): Promise { + const request = new Request(`http://localhost${path}`, { + method, + headers: body === undefined ? undefined : { "content-type": "application/json" }, + body: body === undefined ? undefined : JSON.stringify(body), + }); + return handleManagementAPI(request, new URL(request.url), config, { + createManagementConvergeCodex: catalogConvergenceFactory(), + }); +} + +function deferred(): { promise: Promise; resolve: () => void } { + let resolve!: () => void; + const promise = new Promise(done => { resolve = done; }); + return { promise, resolve }; +} + +async function within(promise: Promise, ms = 2_000): Promise { + let timer: ReturnType | undefined; + try { + return await Promise.race([ + promise, + new Promise((_resolve, reject) => { + timer = setTimeout(() => reject(new Error(`timed out after ${ms}ms`)), ms); + }), + ]); + } finally { + if (timer) clearTimeout(timer); + } +} + +function heldNativeTerminal(payload: Record) { + const release = deferred(); + const encoder = new TextEncoder(); + const upstream = serve(() => new Response(new ReadableStream({ + async start(controller) { + controller.enqueue(encoder.encode(`event: response.output_text.delta\ndata: ${JSON.stringify({ + type: "response.output_text.delta", item_id: "msg_late", output_index: 0, + content_index: 0, delta: "already visible", + })}\n\n`)); + await release.promise; + controller.enqueue(encoder.encode(`event: ${payload.type}\ndata: ${JSON.stringify(payload)}\n\n`)); + controller.close(); + }, + }), { headers: { "content-type": "text/event-stream" } })); + return { upstream, release: release.resolve }; +} + +// The .27 combo-advance contract: a combo that moved targets persists `comboTargetAdvanced`, which +// survives hydration, /api/logs and routing analytics. Split from server-combo-failover-e2e (size cap). +describe("server combo failover 030 activation matrix", () => { + test("persists one logical A503 to B200 request with ordered physical usage", async () => { + const a = serve(() => Response.json({ + error: { message: "overloaded" }, + usage: { input_tokens: 7, output_tokens: 1, total_tokens: 8 }, + }, { status: 503 })); + const b = serve(() => chatSuccess("logged backup", "m2")); + const config = comboConfig({ + a: provider("openai-chat", baseUrl(a), "key-a"), + b: provider("openai-chat", baseUrl(b), "key-b"), + }); + const response = await postLogged(config); + expect(response.status).toBe(200); + await response.text(); + const { log, usage } = await latestAttemptReceipts(config); + + for (const receipt of [log, usage]) { + expect(receipt).toMatchObject({ + provider: "combo", + model: "combo/free", + requestedModel: "combo/free", + resolvedModel: "m2", + comboTargetAdvanced: true, + attempts: [ + { ordinal: 1, provider: "a", model: "m1", status: 503, usage: { inputTokens: 7, outputTokens: 1 } }, + { ordinal: 2, provider: "b", model: "m2", status: 200, usage: { inputTokens: 2, outputTokens: 1 } }, + ], + }); + } + + clearRequestLogsForTests(); + expect(hydrateRequestLogsFromDisk()).toBe(1); + const hydratedResponse = await management(config, "GET", "/api/logs?tail=1"); + const hydrated = logsFromApiBody(await hydratedResponse!.json()); + expect(hydrated[0]?.comboTargetAdvanced).toBe(true); + expect((await computeRoutingAnalytics({})).comboFailoverRequests).toBe(1); + }); +}); diff --git a/tests/server/server-combo-local-pacing-cooldown.test.ts b/tests/server/server-combo-local-pacing-cooldown.test.ts new file mode 100644 index 00000000000..2bc00f3eb6e --- /dev/null +++ b/tests/server/server-combo-local-pacing-cooldown.test.ts @@ -0,0 +1,64 @@ +// A local request-pacing refusal is this proxy's own queue answering, not provider quota: on an +// AgentRadar gateway it reaches the client as the local retryable 429, never cools the combo target, +// and sends nothing to the paced provider. +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { comboProviderFactory } from "../helpers/combo-provider"; +import { chatSuccess } from "../helpers/combo-failover-upstream"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { clearComboSelectionState, clearComboTargetCooldowns, isComboTargetInCooldown } from "../../src/combos"; +import { resetProviderRequestPacingForTest, setProviderRequestPacingLimitsForTest, waitForProviderRequestSlot, releaseProviderRequestSlot } from "../../src/providers/request-pacing"; +import type { OcxConfig } from "../../src/types"; + +const { handleResponses } = await import("../../src/server/responses"); +const provider = comboProviderFactory(() => undefined); +const previous = { home: process.env.OPENCODEX_HOME, flag: process.env.OPENCODEX_AGENTRADAR }; +const servers: Array> = []; +let home = ""; +let release: (() => void) | undefined; +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), "ocx-combo-pacing-")); + process.env.OPENCODEX_HOME = home; process.env.OPENCODEX_AGENTRADAR = "1"; + clearComboSelectionState(); clearComboTargetCooldowns(); resetProviderRequestPacingForTest(); + release = acquireOwnedSpendHome(); +}); +afterEach(async () => { + for (const server of servers.splice(0)) await server.stop(true); + release?.(); release = undefined; + resetProviderRequestPacingForTest(); clearComboSelectionState(); clearComboTargetCooldowns(); + if (previous.flag === undefined) delete process.env.OPENCODEX_AGENTRADAR; else process.env.OPENCODEX_AGENTRADAR = previous.flag; + if (previous.home === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previous.home; + removeTreeWithRetry(home); +}); + +test("a local pacing 429 is answered locally and never cools the combo target", async () => { + let aRequests = 0; + const a = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => { aRequests += 1; return chatSuccess("from a", "m1"); } }); + const b = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => chatSuccess("from b", "m2") }); + servers.push(a, b); + const url = (server: typeof a) => `${server.url.toString().replace(/\/$/, "")}/v1`; + const config = { port: 0, defaultProvider: "a", + providers: { + a: provider("openai-chat", url(a), "key-a", { requestPacing: { enabled: true, maxConcurrentRequests: 1 } }), + b: provider("openai-chat", url(b), "key-b"), + }, + combos: { free: { strategy: "failover", targets: [{ provider: "a", model: "m1" }, { provider: "b", model: "m2" }] } }, + } as OcxConfig; + // A holds its only slot and has no queue, so its admission refuses locally with 429 queue_full. + const held = await waitForProviderRequestSlot("a", config.providers.a!, "m1"); + setProviderRequestPacingLimitsForTest({ maxQueueDepth: 0 }); + try { + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/free", input: "hello", stream: false }), + }), config, { model: "", provider: "" }); + expect(response.status).toBe(429); + expect(response.headers.get("retry-after")).not.toBeNull(); + expect(await response.text()).toContain("request pacing queue for provider 'a' is full"); + expect(aRequests).toBe(0); + expect(isComboTargetInCooldown("free", { provider: "a", model: "m1" })).toBe(false); + } finally { releaseProviderRequestSlot(held); } +}); diff --git a/tests/server/server-google-antigravity-oauth-401-replay.test.ts b/tests/server/server-google-antigravity-oauth-401-replay.test.ts index 30cadba7666..91ca3fab3bb 100644 --- a/tests/server/server-google-antigravity-oauth-401-replay.test.ts +++ b/tests/server/server-google-antigravity-oauth-401-replay.test.ts @@ -968,6 +968,11 @@ describe("Google Antigravity OAuth upstream 401 replay", () => { expect(observed.chatAuth).toEqual(["Bearer rejected-access", "Bearer fresh-access"]); expect(observed.chatProjects).toEqual(["initial-project-id", "refreshed-project-id"]); expect(observed.counts.refresh).toBe(1); + const attempts = readUsageEntries().at(-1)?.attempts; + expect(attempts?.map(attempt => ({ status: attempt.status, sendCount: attempt.sendCount }))) + .toEqual([{ status: 401, sendCount: 1 }, { status: 200, sendCount: 1 }]); + expect(attempts?.[1].accountLogLabel).toBe(attempts?.[0].accountLogLabel); + expect(attempts?.[1].recoveryKinds).toContain("oauth-401"); } finally { await server.stop(true); } }); diff --git a/tests/server/server-xai-oauth-401-replay.test.ts b/tests/server/server-xai-oauth-401-replay.test.ts index 887415af39d..fb74177d473 100644 --- a/tests/server/server-xai-oauth-401-replay.test.ts +++ b/tests/server/server-xai-oauth-401-replay.test.ts @@ -210,10 +210,12 @@ describe("xAI OAuth Responses opt-in upstream 401 replay", () => { expect(json.output?.find(item => item.type === "message")?.content?.[0]?.text).toBe("ok after refresh"); expect(observed.counts.refresh).toBe(1); expect(observed.chatAuth).toEqual(["Bearer rejected-access", "Bearer fresh-access"]); - const attempt = readUsageEntries().at(-1)?.attempts?.[0]; - expect(attempt?.credentialSource).toBe("grok-oauth"); - expect(attempt?.sendCount).toBe(2); - expect(attempt?.totalTokens).toBe(5); + const attempts = readUsageEntries().at(-1)?.attempts; + expect(attempts?.map(attempt => ({ status: attempt.status, sendCount: attempt.sendCount, credentialSource: attempt.credentialSource }))) + .toEqual([{ status: 401, sendCount: 1, credentialSource: "grok-oauth" }, + { status: 200, sendCount: 1, credentialSource: "grok-oauth" }]); + expect(attempts?.[1].totalTokens).toBe(5); + expect(attempts?.[1].recoveryKinds).toContain("oauth-401"); const persisted = readFileSync(usageLogPath(), "utf8"); expect(persisted).not.toContain("rejected-access"); expect(persisted).not.toContain("fresh-access"); diff --git a/tests/server/terminal-guard-agentradar-rotation.test.ts b/tests/server/terminal-guard-agentradar-rotation.test.ts new file mode 100644 index 00000000000..6776f05bb02 --- /dev/null +++ b/tests/server/terminal-guard-agentradar-rotation.test.ts @@ -0,0 +1,341 @@ +// Ported from v2.65.0-vadymhimself.27 tests/server/terminal-guard-server.test.ts (8fb1bbbe79/e0b898adba): +// on an AgentRadar gateway a streamed terminal-guard re-ask moves off a refused Anthropic account. Assertions +// unchanged; the spend-ledger lease and the gateway flag are the only additions. +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { clearPoolRotationState } from "../../src/codex/pool-rotation"; +import { clearAnthropicAccountPoolState } from "../../src/oauth/anthropic-routing"; +import { saveConfig } from "../../src/config"; +import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import type { RequestLogContext } from "../../src/server/request-log"; +import { handleResponses } from "../../src/server/responses"; +import type { OcxConfig } from "../../src/types"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; + +let releaseInheritedSpendHome: (() => void) | undefined; +// Taken per inherited-home dispatch because one row below installs a different home. +const takeInheritedSpendHome = (): void => { releaseInheritedSpendHome = acquireOwnedSpendHome(); }; + +const config = { + port: 0, + defaultProvider: "claude-se", + providers: { + "claude-se": { + adapter: "anthropic", + baseUrl: "https://example.test", + apiKey: "sk-test", + }, + }, +} as unknown as OcxConfig; + +/** Build an Anthropic SSE response from raw frames. */ +function anthropicSse(body: string): Response { + return new Response(body, { status: 200, headers: { "content-type": "text/event-stream" } }); +} + +const firstTurn = [ + 'event: message_start\ndata: {"type":"message_start","message":{"usage":{"input_tokens":10,"output_tokens":1}}}\n\n', + 'event: content_block_start\ndata: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}\n\n', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"我接下来会修改相关文件。"}}\n\n', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":0}\n\n', + 'event: message_delta\ndata: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":2}}\n\n', + 'event: message_stop\ndata: {"type":"message_stop"}\n\n', +].join(""); + +const continuationTurn = [ + 'event: message_start\ndata: {"type":"message_start","message":{"usage":{"input_tokens":20,"output_tokens":1}}}\n\n', + 'event: content_block_start\ndata: {"type":"content_block_start","index":0,"content_block":{"type":"tool_use","id":"toolu_1","name":"exec_command","input":{}}}\n\n', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"{}"}}\n\n', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":0}\n\n', + 'event: message_delta\ndata: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":3}}\n\n', + 'event: message_stop\ndata: {"type":"message_stop"}\n\n', +].join(""); + +/** Build an OpenAI Chat Completions SSE response from raw frames. */ +function chatSse(body: string): Response { + return new Response(body, { status: 200, headers: { "content-type": "text/event-stream" } }); +} + +// A clean end-of-turn with only assistant text and no tool call — the suspicious +// no-tool completion the guard is meant to re-ask (mirrors firstTurn for openai-chat). +const chatFirstTurn = [ + 'data: {"choices":[{"delta":{"content":"我接下来会修改相关文件。"}}]}\n\n', + 'data: {"choices":[{"delta":{},"finish_reason":"stop"}]}\n\n', + "data: [DONE]\n\n", +].join(""); + +// The continuation turn emits the tool call the model should have produced. +const chatContinuationTurn = [ + 'data: {"choices":[{"delta":{"tool_calls":[{"index":0,"id":"call_1","type":"function","function":{"name":"exec_command","arguments":""}}]}}]}\n\n', + 'data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{}"}}]}}]}\n\n', + 'data: {"choices":[{"delta":{},"finish_reason":"tool_calls"}]}\n\n', + "data: [DONE]\n\n", +].join(""); + +function openAiChatConfig(terminalContinuationGuard?: boolean): OcxConfig { + return { + port: 0, + defaultProvider: "glm-gw", + providers: { + "glm-gw": { + adapter: "openai-chat", + baseUrl: "https://example.test/v1", + apiKey: "key", + defaultModel: "glm-5.2", + models: ["glm-5.2"], + ...(terminalContinuationGuard !== undefined ? { terminalContinuationGuard } : {}), + }, + }, + } as unknown as OcxConfig; +} + +describe("server terminal guard integration", () => { + let originalFetch: typeof fetch; + let calls: number; + let requestBodies: Record[]; + + const previousAgentRadar = process.env.OPENCODEX_AGENTRADAR; + beforeEach(() => { + process.env.OPENCODEX_AGENTRADAR = "1"; + originalFetch = globalThis.fetch; + calls = 0; + requestBodies = []; + globalThis.fetch = (async (_input, init) => { + calls += 1; + requestBodies.push(JSON.parse(String(init?.body ?? "{}")) as Record); + return anthropicSse(calls === 1 ? firstTurn : continuationTurn); + }) as typeof fetch; + }); + + afterEach(() => { + // Released first so a failed row cannot carry its writer lease into the next case. + releaseInheritedSpendHome?.(); + releaseInheritedSpendHome = undefined; + globalThis.fetch = originalFetch; + if (previousAgentRadar === undefined) delete process.env.OPENCODEX_AGENTRADAR; + else process.env.OPENCODEX_AGENTRADAR = previousAgentRadar; + }); + + + test("terminal-guard Anthropic account rotation opens a new physical attempt", async () => { + const home = mkdtempSync(join(tmpdir(), "ocx-terminal-anthropic-pool-")); + const previousHome = process.env.OPENCODEX_HOME; + process.env.OPENCODEX_HOME = home; + // PORT: upstream ledger requires the spend-journal writer lease for direct handler dispatch. + const releasePortSpendHome = acquireOwnedSpendHome(); + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + try { + writeFileSync(join(home, "auth.json"), JSON.stringify({ + anthropic: { + activeAccountId: "account-a", + accounts: [ + { + id: "account-a", + credential: { + access: "synthetic-access-a", + refresh: "synthetic-refresh-a", + expires: 9999999999999, + accountId: "synthetic-account-a", + }, + }, + { + id: "account-b", + credential: { + access: "synthetic-access-b", + refresh: "synthetic-refresh-b", + expires: 9999999999999, + accountId: "synthetic-account-b", + }, + }, + ], + }, + }), { mode: 0o600 }); + const poolConfig = { + port: 0, + defaultProvider: "anthropic", + providers: { + anthropic: { + adapter: "anthropic", + baseUrl: "https://example.test", + authMode: "oauth", + }, + }, + anthropicAccountPool: { enabled: true, autoSwitchThreshold: 100 }, + } as unknown as OcxConfig; + const authorizations: string[] = []; + globalThis.fetch = (async (_input, init) => { + authorizations.push(new Headers(init?.headers).get("authorization") ?? ""); + if (authorizations.length === 1) return anthropicSse(firstTurn); + if (authorizations.length === 2) { + return Response.json( + { type: "error", error: { type: "rate_limit_error", message: "rate limited" } }, + { status: 429, headers: { "retry-after": "30" } }, + ); + } + return anthropicSse(continuationTurn); + }) as typeof fetch; + const logCtx: RequestLogContext = { model: "", provider: "" }; + + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model: "anthropic/claude-opus-4.8", + input: "请检查这个问题并修复代码", + stream: true, + tools: [{ type: "function", name: "exec_command", description: "run a command", parameters: { type: "object" } }], + }), + }), poolConfig, logCtx); + + expect(await response.text()).toContain("response.completed"); + expect(authorizations).toEqual([ + "Bearer synthetic-access-a", + "Bearer synthetic-access-a", + "Bearer synthetic-access-b", + ]); + expect(logCtx.attempts).toHaveLength(2); + expect(logCtx.attempts?.[0]).toMatchObject({ status: 429, sendCount: 2 }); + expect(logCtx.attempts?.[1]).toMatchObject({ + sendCount: 1, + recoveryKinds: ["anthropic-oauth-429"], + }); + expect(logCtx.attempts?.[1]?.provider).not.toBe(logCtx.attempts?.[0]?.provider); + } finally { + releasePortSpendHome(); + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + rmSync(home, { recursive: true, force: true }); + } + }); + + test("Anthropic rotates on initial and terminal-guard continuation 403 denials", async () => { + const home = mkdtempSync(join(tmpdir(), "ocx-terminal-anthropic-denial-")); + const previousHome = process.env.OPENCODEX_HOME; + process.env.OPENCODEX_HOME = home; + // PORT: upstream ledger requires the spend-journal writer lease for direct handler dispatch. + const releasePortSpendHome = acquireOwnedSpendHome(); + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + try { + writeFileSync(join(home, "auth.json"), JSON.stringify({ + anthropic: { + activeAccountId: "account-a", + accounts: [ + { + id: "account-a", + credential: { + access: "synthetic-access-a", + refresh: "synthetic-refresh-a", + expires: 9999999999999, + accountId: "synthetic-account-a", + }, + }, + { + id: "account-b", + credential: { + access: "synthetic-access-b", + refresh: "synthetic-refresh-b", + expires: 9999999999999, + accountId: "synthetic-account-b", + }, + }, + { + id: "account-c", + credential: { + access: "synthetic-access-c", + refresh: "synthetic-refresh-c", + expires: 9999999999999, + accountId: "synthetic-account-c", + }, + }, + ], + }, + }), { mode: 0o600 }); + const poolConfig = { + port: 0, + defaultProvider: "anthropic", + providers: { + anthropic: { + adapter: "anthropic", + baseUrl: "https://example.test", + authMode: "oauth", + }, + }, + anthropicAccountPool: { enabled: true, autoSwitchThreshold: 100 }, + } as unknown as OcxConfig; + const sends: Array<{ authorization: string; body: string }> = []; + globalThis.fetch = (async (_input, init) => { + sends.push({ + authorization: new Headers(init?.headers).get("authorization") ?? "", + body: String(init?.body ?? ""), + }); + if (sends.length === 2) return anthropicSse(firstTurn); + if (sends.length === 1 || sends.length === 3) { + return Response.json({ + type: "error", + error: { + type: "oauth_org_not_allowed", + message: "Your organization has disabled Claude subscription access for Claude Code. Use an Anthropic API key instead, or ask your admin to enable access.", + }, + }, { status: 403 }); + } + return anthropicSse(continuationTurn); + }) as typeof fetch; + const logCtx: RequestLogContext = { model: "", provider: "" }; + + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model: "anthropic/claude-opus-4.8", + input: "请检查这个问题并修复代码", + stream: true, + tools: [{ type: "function", name: "exec_command", description: "run a command", parameters: { type: "object" } }], + }), + }), poolConfig, logCtx); + + const text = await response.text(); + expect(response.status).toBe(200); + expect(text).toContain("response.completed"); + expect(text).toContain("exec_command"); + expect(sends.map(send => send.authorization)).toEqual([ + "Bearer synthetic-access-a", + "Bearer synthetic-access-b", + "Bearer synthetic-access-b", + "Bearer synthetic-access-c", + ]); + expect(sends[1]!.body).toBe(sends[0]!.body); + expect(sends[3]!.body).toBe(sends[2]!.body); + expect(logCtx.attempts).toHaveLength(3); + expect(logCtx.attempts?.[0]).toMatchObject({ status: 403, sendCount: 1 }); + expect(logCtx.attempts?.[1]).toMatchObject({ + status: 403, + sendCount: 2, + recoveryKinds: [/* upstream roster name for .27 anthropic-oauth-403 */ "oauth-account-403"], + }); + expect(logCtx.attempts?.[2]).toMatchObject({ + sendCount: 1, + recoveryKinds: [/* upstream roster name for .27 anthropic-oauth-403 */ "oauth-account-403"], + }); + const auth = JSON.parse(readFileSync(join(home, "auth.json"), "utf8")) as { + anthropic: { accounts: Array<{ id: string; needsReauth?: boolean }> }; + }; + expect(auth.anthropic.accounts.find(account => account.id === "account-a")?.needsReauth).toBe(true); + expect(auth.anthropic.accounts.find(account => account.id === "account-b")?.needsReauth).toBe(true); + expect(auth.anthropic.accounts.find(account => account.id === "account-c")?.needsReauth).toBeUndefined(); + } finally { + releasePortSpendHome(); + clearAnthropicAccountPoolState(); + clearPoolRotationState(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + rmSync(home, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/server/terminal-guard.test.ts b/tests/server/terminal-guard.test.ts index 5af10dcc831..a7f69e27758 100644 --- a/tests/server/terminal-guard.test.ts +++ b/tests/server/terminal-guard.test.ts @@ -266,6 +266,58 @@ describe("terminal guard", () => { expect(actual.at(-1)).toMatchObject({ usage: { inputTokens: 30, outputTokens: 5, totalTokens: 35, providerCredits: 0.05 } }); }); + // .27 parity (8fb1bbbe79): the context window is the larger round, server-tool counters add up. + test("can guard a fetch-based adapter stream with a continuation callback: context and server-tool usage", async () => { + let continuations = 0; + const actual: AdapterEvent[] = []; + for await (const event of guardTerminalEventStream({ + parsed: parsed("请检查这个问题并修复代码"), + firstEvents: (async function* () { + yield { type: "text_delta", text: "我接下来会修改相关文件。" } as AdapterEvent; + yield { + type: "done", + usage: { + inputTokens: 10, + outputTokens: 2, + contextTotalTokens: 100, + anthropicServerToolUse: { web_search_requests: 1 }, + }, + } as AdapterEvent; + })(), + continuation: next => { + continuations += 1; + expect(next.context.messages.at(-1)).toMatchObject({ role: "developer" }); + return (async function* () { + yield { type: "tool_call_start", id: "call_1", name: "exec_command" } as AdapterEvent; + yield { type: "tool_call_end" } as AdapterEvent; + yield { + type: "done", + usage: { + inputTokens: 20, + outputTokens: 3, + contextTotalTokens: 140, + anthropicServerToolUse: { web_search_requests: 2, web_fetch_requests: 1 }, + }, + } as AdapterEvent; + })(); + }, + adapterName: "anthropic", + })) actual.push(event); + + expect(continuations).toBe(1); + expect(actual.filter(event => event.type === "done")).toHaveLength(1); + expect(actual.some(event => event.type === "assistant_boundary")).toBe(true); + expect(actual.at(-1)).toMatchObject({ + usage: { + inputTokens: 30, + outputTokens: 5, + totalTokens: 35, + contextTotalTokens: 140, + anthropicServerToolUse: { web_search_requests: 3, web_fetch_requests: 1 }, + }, + }); + }); + // A heartbeat is adapter liveness, not turn content. The openai-chat adapter emits one per // tool-call delta while it buffers, so retaining them here would let a single large argument diff --git a/tests/usage/key-attribution.test.ts b/tests/usage/key-attribution.test.ts index 092bd9f8a2f..bfcfd225d22 100644 --- a/tests/usage/key-attribution.test.ts +++ b/tests/usage/key-attribution.test.ts @@ -2,12 +2,13 @@ import { describe, expect, test } from "bun:test"; import { addFinalRequestLog, aggregateAttemptUsage, beginRequestAttempt, finishRequestAttempt, noteProviderAttemptSend, recordKeyAttemptFailure, recordKeyAttemptUsage, applyResponseLogMetadata, sealRequestAttemptIdentity, - inspectResponseLogSsePayload, type RequestLogContext, type RequestLogEntry, + inspectResponseLogSsePayload, recordAdapterTierMetadata, type RequestLogContext, type RequestLogEntry, } from "../../src/server/request-log"; import { readFileSync } from "node:fs"; import { repoPath } from "../helpers/repo-root"; import { normalizeUsageEntryForTest } from "../../src/usage/log"; import { formatAnthropicProviderForLog } from "../../src/oauth/anthropic-routing"; +import { createAdapterTierMetadata } from "../../src/providers/fastwire"; describe("key attempt accounting", () => { test("a combo parent copied before streaming rotation cannot overwrite the final key's usage", () => { @@ -42,10 +43,10 @@ describe("key attempt accounting", () => { ); // The entry-level row is the fallback for a request written before attempts existed, and it // is reachable only when there are none. - expect(attributions).toContain("if (!entry.attempts?.length) {"); + expect(attributions).toContain("if (entry.attempts === undefined) {"); // Everything after that early return maps the attempts; there is no branch that emits the // entry row alongside them. - const fallback = attributions.indexOf("if (!entry.attempts?.length) {"); + const fallback = attributions.indexOf("if (entry.attempts === undefined) {"); const perAttempt = attributions.indexOf("return entry.attempts.map(attempt =>", fallback); expect(fallback).toBeGreaterThan(-1); expect(perAttempt).toBeGreaterThan(fallback); @@ -256,6 +257,81 @@ describe("key attempt accounting", () => { noteProviderAttemptSend(ctx, "anthropic", oauth, undefined, recovery); expect(ctx.attempts?.[0]).toMatchObject({ ordinal: 1, status }); }); + test.each([ + ["oauth-401", 401], ["key-401", 401], ["oauth-account-402", 402], + ["oauth-account-403", 403], ["key-429", 429], ["anthropic-oauth-429", 429], ["oauth-account-429", 429], + ] as const)("same-account %s recovery keeps each physical attempt's outcome and usage", (recovery, status) => { + const oauth = { adapter: "openai-responses" as const, authMode: "oauth" as const, baseUrl: "https://example.test" }; + const active = beginRequestAttempt(1, "test", "model", "openai-responses"); + const ctx: RequestLogContext = { provider: "test", model: "model", accountLogLabel: "o111111", + activeAttempt: active, attempts: [active] }; + noteProviderAttemptSend(ctx, "test", oauth, 100); + recordKeyAttemptUsage(ctx, { inputTokens: 100, outputTokens: 10 }); + noteProviderAttemptSend(ctx, "test", oauth, 200, recovery); + expect(ctx.activeAttempt).toBe(active); + expect(ctx.attempts).toHaveLength(2); + expect(ctx.attempts?.[0]).toMatchObject({ ordinal: 1, status, sendCount: 1, accountLogLabel: "o111111", + usage: { inputTokens: 100, outputTokens: 10 } }); + expect(ctx.attempts?.[0].inputTokenEstimate).toBeUndefined(); + expect(active).toMatchObject({ ordinal: 2, status: 0, sendCount: 1, inputTokenEstimate: 200, + accountLogLabel: "o111111", recoveryKinds: [recovery] }); + expect(active.usage).toBeUndefined(); + recordKeyAttemptUsage(ctx, { inputTokens: 200, outputTokens: 20 }); + const rows: RequestLogEntry[] = []; + addFinalRequestLog("same-account-recovery", Date.now(), ctx, 200, undefined, row => rows.push(row)); + const persisted = normalizeUsageEntryForTest(JSON.parse(JSON.stringify(rows[0]))); + expect(persisted.attempts?.map(attempt => attempt.status)).toEqual([status, 200]); + expect(persisted.usage).toMatchObject({ inputTokens: 300, outputTokens: 30 }); + }); + test.each(["rate-limit-429", "transient-5xx", "connection-reset", "offline-resend", "opaque-blob-rejection"] as const)( + "same-account %s resend stays on its existing attempt", recovery => { + const oauth = { adapter: "openai-responses" as const, authMode: "oauth" as const, baseUrl: "https://example.test" }; + const active = beginRequestAttempt(1, "test", "model", "openai-responses"); + const ctx: RequestLogContext = { provider: "test", model: "model", activeAttempt: active, attempts: [active] }; + noteProviderAttemptSend(ctx, "test", oauth, undefined); + noteProviderAttemptSend(ctx, "test", oauth, undefined, recovery); + expect(ctx.attempts).toHaveLength(1); + expect(active).toMatchObject({ ordinal: 1, sendCount: 2, recoveryKinds: [recovery] }); + }); + test("same-account OAuth recovery keeps the live tier observer and snapshots the refused tier", () => { + const oauth = { adapter: "openai-responses" as const, authMode: "oauth" as const, baseUrl: "https://example.test" }; + const active = beginRequestAttempt(1, "test", "model", "openai-responses"); + const ctx: RequestLogContext = { provider: "test", model: "model", activeAttempt: active, attempts: [active] }; + const metadata = createAdapterTierMetadata({ capability: true, eligibility: "eligible", + fastWire: { kind: "service-tier", canonicalToWire: { priority: "priority" }, foreignCallerTiers: "verbatim" }, + demandDecision: "force-fast" }, { kind: "set", value: "priority" }, "service-tier", "priority")!; + recordAdapterTierMetadata(ctx, metadata); + noteProviderAttemptSend(ctx, "test", oauth, undefined); + noteProviderAttemptSend(ctx, "test", oauth, undefined, "oauth-401"); + expect(active.tierOutcome).toBe(metadata.outcome); + expect(ctx.attempts?.[0].tierOutcome).not.toBe(metadata.outcome); + metadata.observeResponseServiceTier("priority"); + const rows: RequestLogEntry[] = []; + addFinalRequestLog("same-account-tier-recovery", Date.now(), ctx, 200, undefined, row => rows.push(row)); + const persisted = normalizeUsageEntryForTest(JSON.parse(JSON.stringify(rows[0]))); + expect(persisted.attempts?.[0].tierOutcome).toMatchObject({ confirmation: "assumed", fastOutcome: "applied" }); + expect(persisted.attempts?.[0].tierOutcome?.responseServiceTier).toBeUndefined(); + expect(persisted.attempts?.[1].tierOutcome).toMatchObject({ confirmation: "confirmed", responseServiceTier: "priority" }); + }); + test("a same-key recovery preserves the observed refusal status", () => { + const key = { adapter: "openai-chat" as const, authMode: "key" as const, baseUrl: "https://example.test", + _apiKeyAttempt: { reference: "synthetic-same-key" } }; + const active = beginRequestAttempt(1, "test", "model", "openai-chat"); + const ctx: RequestLogContext = { provider: "test", model: "model", activeAttempt: active, attempts: [active] }; + noteProviderAttemptSend(ctx, "test", key, undefined); + active.status = 403; + noteProviderAttemptSend(ctx, "test", key, undefined, "key-401"); + expect(ctx.attempts?.map(attempt => attempt.status)).toEqual([403, 0]); + expect(active.accountLogLabel).toBe(ctx.attempts?.[0].accountLogLabel); + }); + test("credential recovery before the first send does not create a phantom attempt", () => { + const oauth = { adapter: "openai-responses" as const, authMode: "oauth" as const, baseUrl: "https://example.test" }; + const active = beginRequestAttempt(1, "test", "model", "openai-responses"); + const ctx: RequestLogContext = { provider: "test", model: "model", activeAttempt: active, attempts: [active] }; + noteProviderAttemptSend(ctx, "test", oauth, undefined, "oauth-401"); + expect(ctx.attempts).toHaveLength(1); + expect(active).toMatchObject({ ordinal: 1, sendCount: 1 }); + }); test("wire snapshots replace the current send against a pre-send baseline", () => { const active = beginRequestAttempt(1, "test", "model", "openai-chat"); const ctx: RequestLogContext = { provider: "test", model: "model", activeAttempt: active, attempts: [active] }; diff --git a/tests/usage/request-history-index.test.ts b/tests/usage/request-history-index.test.ts index e2c7918ab63..b9717ae3a8e 100644 --- a/tests/usage/request-history-index.test.ts +++ b/tests/usage/request-history-index.test.ts @@ -20,12 +20,14 @@ import { normalizeUsageEntryForTest, resetUsageReadCacheForTests, usageLogPath, + type PersistedUsageAttempt, type PersistedUsageEntry, } from "../../src/usage/log"; import { closeRequestHistoryIndex, queryRequestHistory, rebuildRequestHistoryIndex, + requestHistoryDb, requestHistoryRowById, sourceIdentityMatches, REQUEST_HISTORY_MAX_RECORD_BYTES, @@ -513,4 +515,55 @@ describe("request-history index (RI-02)", () => { expect(ids).toEqual(["req-5", "req-4", "req-3", "req-2", "req-1", "req-0"]); expect(second.hasMore).toBe(false); }); + + test("fallback and attempt count use only distinct physical attempts", async () => { + const attempt = (ordinal: number, overrides: Partial = {}): PersistedUsageAttempt => ({ + ordinal, + provider: "a", + model: "m1", + adapter: "openai-chat", + status: 200, + durationMs: 5, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported", + ...overrides, + }); + appendUsageEntry(entry("attempts-absent", 1000)); + appendUsageEntry(entry("attempts-empty", 1001, "a", "m1", { attempts: [] })); + appendUsageEntry(entry("attempts-nonphysical", 1002, "a", "m1", { + attempts: [attempt(1, { sendCount: 0 }), attempt(2, { locallyAnswered: true })], + })); + appendUsageEntry(entry("attempts-duplicate", 1003, "a", "m1", { + attempts: [attempt(1), attempt(1, { provider: "duplicate" })], + })); + appendFileSync(usageLogPath(), `${JSON.stringify(entry("attempts-fallback", 1004, "a", "m1", { + attempts: [ + attempt(1, { sendCount: 0 }), + attempt(1), + attempt(2), + attempt(2, { provider: "duplicate" }), + ], + }))}\n`, "utf-8"); + appendFileSync(usageLogPath(), `${JSON.stringify(entry("attempts-invalid", 1005, "a", "m1", { + attempts: [ + { ordinal: 1, provider: "a", model: "m1", status: 503, durationMs: 5 }, + { ordinal: 2, provider: "b", model: "m2", status: 200, durationMs: 5 }, + ] as PersistedUsageAttempt[], + }))}\n`, "utf-8"); + + const fallback = await queryRequestHistory({ fallback: true }, undefined, 10); + expect(fallback.rows.map(row => row.requestId)).toEqual(["attempts-fallback"]); + const counts = requestHistoryDb().query( + "SELECT request_id, attempt_count FROM requests ORDER BY request_id", + ).all() as Array<{ request_id: string; attempt_count: number }>; + expect(counts).toEqual([ + { request_id: "attempts-absent", attempt_count: 1 }, + { request_id: "attempts-duplicate", attempt_count: 1 }, + { request_id: "attempts-empty", attempt_count: 0 }, + { request_id: "attempts-fallback", attempt_count: 2 }, + { request_id: "attempts-invalid", attempt_count: 0 }, + { request_id: "attempts-nonphysical", attempt_count: 0 }, + ]); + }); }); diff --git a/tests/usage/request-log-estimate-cap.test.ts b/tests/usage/request-log-estimate-cap.test.ts index 80faa5ad3cd..68a6301bfee 100644 --- a/tests/usage/request-log-estimate-cap.test.ts +++ b/tests/usage/request-log-estimate-cap.test.ts @@ -161,10 +161,12 @@ describe("request log token-estimate context-window cap (codex-router PR #140)", test("an aggregate carries the upstream-reported mark only when every attempt has usage with it", () => { const marked = () => { const attempt = beginRequestAttempt(1, "kiro", "deepseek-3.2", "kiro"); + noteAttemptSend(attempt, undefined); finishRequestAttempt(attempt, 200, 10, { inputTokens: 10, outputTokens: 2, upstreamReported: true }); return attempt; }; const unmarked = beginRequestAttempt(2, "kiro", "deepseek-3.2", "kiro"); + noteAttemptSend(unmarked, undefined); finishRequestAttempt(unmarked, 200, 10, { inputTokens: 10, outputTokens: 2 }); expect(aggregateAttemptUsage([marked(), marked()]).usage?.upstreamReported).toBe(true); diff --git a/tests/usage/request-log-physical.test.ts b/tests/usage/request-log-physical.test.ts new file mode 100644 index 00000000000..6d5a9d8d83d --- /dev/null +++ b/tests/usage/request-log-physical.test.ts @@ -0,0 +1,491 @@ +import { describe, expect, spyOn, test } from "bun:test"; +import { + filterRequestLogs, + addFinalRequestLog, + httpStatusFromTerminalError, + nextRequestLogId, + responseWithDeferredRequestLog, + requestLogErrorCode, + requestLogSpeedLabel, + type RequestLogEntry, +} from "../../src/server"; +import { + aggregateAttemptUsage, + addRequestLog, + beginRequestAttempt, + clearRequestLogsForTests, + finishRequestAttempt, + getRequestLogEntries, + hydrateRequestLogsFromDisk, + noteAttemptSend, + noteProviderAttemptSend, + recordAdapterReasoning, + recordFirstOutput, + requestLogEntryFromPersistedUsage, + sealRequestAttemptIdentity, + recordAttemptCredentialSource, + inspectResponseLogSsePayload, + httpStatusForRequestLogTerminal, + type RequestLogContext, +} from "../../src/server/request-log"; +import { handleResponses } from "../../src/server/responses"; +import { bridgeToResponsesSSE } from "../../src/bridge"; +import type { AdapterEvent, OcxConfig, OcxProviderConfig, OcxUsage } from "../../src/types"; +import { + appendUsageEntry, + normalizeUsageEntryForTest, + readUsageEntries, + resetUsageReadCacheForTests, + type PersistedUsageEntry, +} from "../../src/usage/log"; +import { mkdtempSync} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { log } from "../helpers/request-log-entry"; +import { decodeRequestLogCursor, selectRequestLogPoll } from "../../src/server/request-log-cursor"; + +async function* replayAdapterEvents(events: AdapterEvent[]): AsyncGenerator { + for (const event of events) yield event; +} + +describe("request log metadata", () => { + + + + for (const error of [ + { type: "authentication_error", message: "Usage limit lookup requires renewed authentication" }, + { code: "invalid_api_key", message: "Usage limit unavailable for this credential" }, + ]) { + } + + + + + + + + + test("records only valid adapter prompt-cache TTL metadata on the active attempt", () => { + const attempt = beginRequestAttempt(1, "anthropic", "claude-opus-5", "anthropic"); + const logCtx: RequestLogContext = { + model: "claude-opus-5", + provider: "anthropic", + activeAttempt: attempt, + }; + const request = { + url: "https://api.anthropic.com/v1/messages", + method: "POST", + headers: {}, + body: "{}", + promptCacheTtlMs: 3_600_000, + }; + + recordAdapterReasoning(logCtx, request as never); + expect((attempt as unknown as { promptCacheTtlMs?: number }).promptCacheTtlMs).toBe(3_600_000); + + recordAdapterReasoning(logCtx, { ...request, promptCacheTtlMs: -1 } as never); + expect((attempt as unknown as { promptCacheTtlMs?: number }).promptCacheTtlMs).toBeUndefined(); + }); + + + + + + + + // The value is caller-controlled, so proving it lands is only half the contract: the + // persistence path must also be the SANITIZED one. A test that only ever writes a safe + // short slug passes identically whether `sanitizeLogMetadataString` is applied or not. + + // `addFinalRequestLog` is not the only ingress: `addRequestLog` is exported and callable + // directly. Sanitizing only on the disk projection left the in-memory ring — and therefore + // /api/logs — serving the raw value, which is the worst shape for a sanitization bug + // because the surface you would check is the clean one. + + test("captures first physical send time once across retries", () => { + const attempt = beginRequestAttempt(1, "anthropic", "claude-opus-5", "anthropic"); + + noteAttemptSend(attempt, undefined, undefined, 123.25); + noteAttemptSend(attempt, undefined, "transient-5xx", 456.5); + + expect(attempt.firstSendAt).toBe(123.25); + expect(attempt.sendCount).toBe(2); + }); + + + test("keeps the latest context checkpoint while adding server tool counters", () => { + const first = beginRequestAttempt(1, "a", "m1", "anthropic"); + noteAttemptSend(first, undefined); + finishRequestAttempt( + first, + 429, + 1, + { + inputTokens: 10, + outputTokens: 1, + contextTotalTokens: 90, + anthropicServerToolUse: { web_search_requests: 1 }, + }, + ); + const second = beginRequestAttempt(2, "a", "m1", "anthropic"); + noteAttemptSend(second, undefined); + finishRequestAttempt( + second, + 200, + 1, + { + inputTokens: 12, + outputTokens: 2, + contextTotalTokens: 80, + anthropicServerToolUse: { web_search_requests: 2, web_fetch_requests: 1 }, + }, + ); + + expect(aggregateAttemptUsage([first, second])).toMatchObject({ + usage: { + inputTokens: 22, + outputTokens: 3, + contextTotalTokens: 80, + anthropicServerToolUse: { web_search_requests: 3, web_fetch_requests: 1 }, + }, + }); + }); + + test("explicit attempts do not reuse stale root usage when no physical usage survives", () => { + const local = beginRequestAttempt(1, "local", "m1", "anthropic"); + local.locallyAnswered = true; + finishRequestAttempt(local, 200, 1, { inputTokens: 30, outputTokens: 3 }); + const unsent = finishRequestAttempt( + beginRequestAttempt(2, "unsent", "m2", "anthropic"), + 502, + 1, + { inputTokens: 40, outputTokens: 4 }, + ); + + for (const [requestId, attempts] of [ + ["explicit-empty", []], + ["explicit-nonphysical", [local, unsent]], + ] as const) { + const entries: RequestLogEntry[] = []; + addFinalRequestLog(requestId, Date.now(), { + model: "stale-model", + provider: "stale-provider", + usage: { inputTokens: 99, outputTokens: 9 }, + attempts: [...attempts], + }, 200, undefined, entry => entries.push(entry)); + + expect(entries[0]?.usageStatus).toBe("unreported"); + expect(entries[0]).not.toHaveProperty("usage"); + expect(entries[0]).not.toHaveProperty("totalTokens"); + } + }); + + test("final logging derives tier outcome from the last unique physical attempt", () => { + const first = beginRequestAttempt(1, "a", "m1", "anthropic"); + noteAttemptSend(first, undefined); + first.tierOutcome = { + wireValue: "first", + fastOutcome: "applied", + confirmation: "confirmed", + }; + finishRequestAttempt(first, 429, 1); + const final = beginRequestAttempt(2, "b", "m2", "anthropic"); + noteAttemptSend(final, undefined); + final.tierOutcome = { + wireValue: "final", + fastOutcome: "applied", + confirmation: "confirmed", + }; + finishRequestAttempt(final, 200, 1); + const duplicate = { + ...final, + tierOutcome: { + wireValue: "duplicate", + fastOutcome: "downgraded" as const, + confirmation: "downgraded" as const, + }, + }; + const local = beginRequestAttempt(3, "local", "m3", "anthropic"); + local.locallyAnswered = true; + local.tierOutcome = { + wireValue: "local", + fastOutcome: "applied", + confirmation: "confirmed", + }; + finishRequestAttempt(local, 200, 1); + const unsent = beginRequestAttempt(4, "unsent", "m4", "anthropic"); + unsent.tierOutcome = { + wireValue: "unsent", + fastOutcome: "applied", + confirmation: "confirmed", + }; + finishRequestAttempt(unsent, 502, 1); + const rootFallback = { + wireValue: "stale-root", + fastOutcome: "unknown" as const, + confirmation: "unknown" as const, + }; + const entries: RequestLogEntry[] = []; + + addFinalRequestLog("tier-physical", Date.now(), { + model: "m2", + provider: "b", + attempts: [first, final, duplicate, local, unsent], + tierOutcome: rootFallback, + }, 200, undefined, entry => entries.push(entry)); + + expect(entries[0]?.tierOutcome).toEqual(final.tierOutcome); + + const fallbackEntries: RequestLogEntry[] = []; + const physicalWithoutTier = beginRequestAttempt(1, "a", "m1", "anthropic"); + noteAttemptSend(physicalWithoutTier, undefined); + finishRequestAttempt(physicalWithoutTier, 200, 1); + addFinalRequestLog("tier-fallback", Date.now(), { + model: "m1", + provider: "a", + attempts: [physicalWithoutTier], + tierOutcome: rootFallback, + }, 200, undefined, entry => fallbackEntries.push(entry)); + expect(fallbackEntries[0]?.tierOutcome).toEqual(rootFallback); + }); + + test("hydration derives tier outcome from the last unique physical attempt", () => { + const first = beginRequestAttempt(1, "a", "m1", "anthropic"); + noteAttemptSend(first, undefined); + first.tierOutcome = { + wireValue: "first", + fastOutcome: "applied", + confirmation: "confirmed", + }; + finishRequestAttempt(first, 429, 1); + const final = beginRequestAttempt(2, "b", "m2", "anthropic"); + noteAttemptSend(final, undefined); + final.tierOutcome = { + wireValue: "final", + fastOutcome: "applied", + confirmation: "confirmed", + }; + finishRequestAttempt(final, 200, 1); + const duplicate = { + ...final, + tierOutcome: { + wireValue: "duplicate", + fastOutcome: "downgraded" as const, + confirmation: "downgraded" as const, + }, + }; + const trailingLocal = beginRequestAttempt(3, "local", "m3", "anthropic"); + trailingLocal.locallyAnswered = true; + trailingLocal.tierOutcome = { + wireValue: "local", + fastOutcome: "applied", + confirmation: "confirmed", + }; + finishRequestAttempt(trailingLocal, 200, 1); + + const projected = requestLogEntryFromPersistedUsage({ + requestId: "hydrated-tier", + timestamp: 1, + model: "m2", + provider: "b", + status: 200, + durationMs: 1, + usageStatus: "unreported", + attempts: [first, final, duplicate, trailingLocal], + tierOutcome: { + wireValue: "stale-root", + fastOutcome: "unknown", + confirmation: "unknown", + }, + }); + + expect(projected.tierOutcome).toEqual(final.tierOutcome); + }); + + test("aggregates only unique sent physical attempts", () => { + const physical = beginRequestAttempt(1, "a", "m1", "anthropic"); + noteAttemptSend(physical, undefined); + finishRequestAttempt(physical, 200, 1, { inputTokens: 10, outputTokens: 2 }); + const duplicate = { ...physical }; + const local = beginRequestAttempt(2, "local", "m2", "anthropic"); + local.locallyAnswered = true; + finishRequestAttempt(local, 200, 1, { inputTokens: 1_000, outputTokens: 100 }); + const unsent = finishRequestAttempt( + beginRequestAttempt(3, "unsent", "m3", "anthropic"), + 502, + 1, + { inputTokens: 2_000, outputTokens: 200 }, + ); + + expect(aggregateAttemptUsage([physical, duplicate, local, unsent])).toMatchObject({ + totalTokens: 12, + usage: { inputTokens: 10, outputTokens: 2, totalTokens: 12 }, + }); + }); + + test("finalized attempts reopen only when another send starts", () => { + const attempt = beginRequestAttempt(1, "a", "m1", "anthropic"); + noteAttemptSend(attempt, undefined); + finishRequestAttempt(attempt, 200, 10, { inputTokens: 10, outputTokens: 2 }); + + finishRequestAttempt(attempt, 502, 20, { inputTokens: 99, outputTokens: 9 }); + expect(attempt).toMatchObject({ + status: 200, + durationMs: 10, + usage: { inputTokens: 10, outputTokens: 2 }, + }); + + noteAttemptSend(attempt, undefined); + expect(attempt.status).toBe(0); + finishRequestAttempt(attempt, 502, 30, { inputTokens: 12, outputTokens: 3 }); + expect(attempt).toMatchObject({ + status: 502, + durationMs: 30, + sendCount: 2, + usage: { inputTokens: 12, outputTokens: 3 }, + }); + }); + + + + test("request aggregate is not assigned to a later attempt that reported no usage", () => { + const entries: RequestLogEntry[] = []; + const first = beginRequestAttempt(1, "a", "model-a", "anthropic"); + noteAttemptSend(first, undefined); + finishRequestAttempt(first, 429, 3, { inputTokens: 10, outputTokens: 4 }); + const final = beginRequestAttempt(2, "b", "model-b", "anthropic"); + noteAttemptSend(final, "oauth-account-429"); + const start = Date.now(); + const logCtx = { + model: "combo/free", + provider: "combo", + usage: { inputTokens: 10, outputTokens: 4 }, + usageIsRequestAggregate: true, + attempts: [first, final], + activeAttempt: final, + activeAttemptStartedAt: start, + } as RequestLogContext & { usageIsRequestAggregate: boolean }; + + addFinalRequestLog("aggregate-after-rotation", start, logCtx, 502, undefined, entry => entries.push(entry)); + + expect(entries[0]?.usage).toEqual({ inputTokens: 10, outputTokens: 4, totalTokens: 14 }); + expect(entries[0]?.attempts?.[1]?.usage).toBeUndefined(); + }); + + test("final non-combo logging aggregates every explicit physical attempt", () => { + const entries: RequestLogEntry[] = []; + const first = beginRequestAttempt(1, "a", "model-a", "openai-chat"); + noteAttemptSend(first, undefined); + finishRequestAttempt( + first, + 503, + 3, + { inputTokens: 4, outputTokens: 1, cacheReadInputTokens: 2 }, + ); + const final = beginRequestAttempt(2, "b", "model-b", "openai-chat"); + noteAttemptSend(final, undefined); + const start = Date.now(); + addFinalRequestLog("policy-parent", start, { + model: "policy/fast", + provider: "policy", + requestedModel: "policy/fast", + providerAdapter: "openai-chat", + usage: { inputTokens: 10, outputTokens: 2, cachedInputTokens: 7 }, + attempts: [first, final], + activeAttempt: final, + activeAttemptStartedAt: start, + }, 200, undefined, entry => entries.push(entry)); + + expect(entries).toHaveLength(1); + expect(entries[0]).toMatchObject({ + provider: "policy", + model: "policy/fast", + usageStatus: "reported", + usage: { + inputTokens: 14, + outputTokens: 3, + cachedInputTokens: 9, + cacheReadInputTokens: 9, + }, + attempts: [ + { provider: "a", usage: { inputTokens: 4, cacheReadInputTokens: 2 } }, + { provider: "b", usage: { inputTokens: 10, cachedInputTokens: 7 } }, + ], + }); + }); + + + + + + + + + + + + /** + * #2704: there was no `model` clause at all, so `?model=x` was accepted and silently + * ignored -- every row came back, and `ocx logs --model x` looked like it had filtered. + * The non-matching assertion is the one that matters: an unfiltered implementation passes + * the positive case for free. + */ + + /** + * #4057: the account label was already persisted on every row and every attempt, but nothing + * could select on it, so "which of my accounts served this?" could only be answered by + * grepping usage.jsonl. The non-matching assertion is the one that matters: an implementation + * that ignores `account` entirely passes the positive cases for free. + */ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +}); + +describe("request log restart hydrate", () => { + + + + +}); + + +describe("request log snapshot cursor", () => { + const epoch = "a".repeat(32); + const query = new URLSearchParams("limit=2000"); + const encode = (value: unknown) => Buffer.from(JSON.stringify(value)).toString("base64url"); + + + + + +}); diff --git a/tests/usage/request-log.test.ts b/tests/usage/request-log.test.ts index a87d350ff59..f358b560037 100644 --- a/tests/usage/request-log.test.ts +++ b/tests/usage/request-log.test.ts @@ -541,12 +541,9 @@ describe("request log metadata", () => { }); test("folds partial and unsupported attempt measurement honestly", () => { - const reported = finishRequestAttempt( - beginRequestAttempt(1, "a", "m1", "openai-chat"), - 200, - 1, - { inputTokens: 4, outputTokens: 1 }, - ); + const sentReported = beginRequestAttempt(1, "a", "m1", "openai-chat"); + noteAttemptSend(sentReported, undefined); + const reported = finishRequestAttempt(sentReported, 200, 1, { inputTokens: 4, outputTokens: 1 }); const sent = beginRequestAttempt(2, "b", "m2", "openai-chat"); noteAttemptSend(sent, undefined); const unreported = finishRequestAttempt(sent, 503, 1); @@ -579,6 +576,7 @@ describe("request log metadata", () => { wireValue: 0, }, }); + noteAttemptSend(a, undefined); finishRequestAttempt( a, 503, diff --git a/tests/usage/usage-anthropic-account-label.test.ts b/tests/usage/usage-anthropic-account-label.test.ts new file mode 100644 index 00000000000..2a31d609213 --- /dev/null +++ b/tests/usage/usage-anthropic-account-label.test.ts @@ -0,0 +1,192 @@ +// The usage ledger names the Anthropic subscription account that served each request, masked, on +// every path: plain active account (pool off, no pin), AgentRadar pin, pool pick, and the combo +// ladder that carries nearly all hosted traffic -- including a failover inside the combo where a +// different account serves. A real OCX process serves fake credentials against a loopback fake +// provider and the assertions read the `usage.jsonl` it writes. No real tokens, no real provider. +import { afterAll, expect, test } from "bun:test"; +import { createHash } from "node:crypto"; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { repoPath } from "../helpers/repo-root"; + +const CLI = repoPath("src/cli/index.ts"); +/** The contract, computed independently of src: `p` + the first 6 hex of sha256(account id). */ +const label = (accountId: string) => `p${createHash("sha256").update(accountId).digest("hex").slice(0, 6)}`; + +const ids = ["acct-alpha-7f3e", "acct-bravo-91c2", "acct-charlie-0d4b", "acct-delta-5a6e"]; +const token = (id: string) => `synthetic-access-${id}`; +const accountOf = (auth: string | null) => ids.find(id => auth === `Bearer ${token(id)}`); + +const sent: Array = []; +const limited = new Set(); +const enc = new TextEncoder(); +const sse = (events: [string, object][]) => new Response(new ReadableStream({ + start(c) { for (const [type, data] of events) c.enqueue(enc.encode(`event: ${type}\ndata: ${JSON.stringify({ type, ...data })}\n\n`)); c.close(); }, +}), { headers: { "content-type": "text/event-stream" } }); +const fake = Bun.serve({ + port: 0, + async fetch(req) { + if (new URL(req.url).pathname !== "/v1/messages") return Response.json({ error: "not found" }, { status: 404 }); + const streamed = (JSON.parse(await req.text()) as { stream?: unknown }).stream === true; + const auth = req.headers.get("authorization"); + sent.push(auth); + if (auth && limited.has(auth)) { + return Response.json({ type: "error", error: { type: "rate_limit_error", message: "slow down" } }, + { status: 429, headers: { "retry-after": "30" } }); + } + if (!streamed) { + return Response.json({ id: "msg_fake", type: "message", role: "assistant", model: "claude-stub", + content: [{ type: "text", text: "ok" }], stop_reason: "end_turn", stop_sequence: null, + usage: { input_tokens: 3, output_tokens: 1 } }); + } + return sse([ + ["message_start", { message: { id: "msg_fake", type: "message", role: "assistant", model: "claude-stub", content: [], stop_reason: null, usage: { input_tokens: 3, output_tokens: 0 } } }], + ["content_block_start", { index: 0, content_block: { type: "text", text: "" } }], + ["content_block_delta", { index: 0, delta: { type: "text_delta", text: "ok" } }], + ["content_block_stop", { index: 0 }], + ["message_delta", { delta: { stop_reason: "end_turn", stop_sequence: null }, usage: { output_tokens: 1 } }], + ["message_stop", {}], + ]); + }, +}); +const FAKE = `http://127.0.0.1:${fake.port}`; +const homes: string[] = []; + +async function startOcx(activeAccountId: string, config: object) { + const home = mkdtempSync(join(tmpdir(), "ocx-ledger-acct-")); + homes.push(home); + mkdirSync(`${home}/.codex`, { mode: 0o700 }); + writeFileSync(`${home}/auth.json`, JSON.stringify({ anthropic: { activeAccountId, accounts: ids.map(id => ({ + id, credential: { access: token(id), refresh: `refresh-${id}`, expires: 9999999999999, email: `${id}@example.invalid` }, + })) } }), { mode: 0o600 }); + const probe = Bun.serve({ port: 0, fetch: () => new Response() }); + const port = probe.port; + probe.stop(true); + writeFileSync(`${home}/config.json`, JSON.stringify({ + port, hostname: "127.0.0.1", defaultProvider: "anthropic", clientIntegrations: { codex: false }, + providers: { anthropic: { adapter: "anthropic", baseUrl: FAKE, authMode: "oauth", allowPrivateNetwork: true, models: ["claude-stub", "claude-alt"] } }, + combos: { claude: { strategy: "failover", targets: [ + { provider: "anthropic", model: "claude-stub" }, { provider: "anthropic", model: "claude-alt" }, + ] } }, + ...config, + }), { mode: 0o600 }); + const proc = Bun.spawn(["bun", CLI, "start", "--port", String(port)], { + env: { ...process.env, OPENCODEX_HOME: home, HOME: home, + OPENCODEX_ADMIN_AUTH_TOKEN: `ocx_admin_${crypto.randomUUID()}${crypto.randomUUID()}`, OPENCODEX_AGENTRADAR: "1" }, + stdout: "ignore", stderr: "pipe", + }); + const ocx = { proc, home, base: `http://127.0.0.1:${port}`, log: "" }; + (async () => { for await (const chunk of proc.stderr) ocx.log += new TextDecoder().decode(chunk); })(); + for (let i = 0; ; i++) { + try { if ((await fetch(`${ocx.base}/healthz`)).status < 500) break; } catch { /* not up yet */ } + if (i > 300) throw new Error(`OCX did not start: ${ocx.log}`); + await Bun.sleep(100); + } + return ocx; +} +type Ocx = Awaited>; + +const ledgerText = (ocx: Ocx) => existsSync(join(ocx.home, "usage.jsonl")) ? readFileSync(join(ocx.home, "usage.jsonl"), "utf8") : ""; +const ledger = (ocx: Ocx) => ledgerText(ocx).split("\n").filter(Boolean).map(line => JSON.parse(line)); + +/** One request; returns its HTTP status, the accounts it reached in order, and its ledger row. */ +async function send(ocx: Ocx, path: string, body: object, headers: Record = {}) { + const rowsBefore = ledger(ocx).length; + const from = sent.length; + const res = await fetch(`${ocx.base}${path}`, { method: "POST", + headers: { "content-type": "application/json", "anthropic-version": "2023-06-01", "x-api-key": "unused", ...headers }, + body: JSON.stringify(body) }); + const text = await res.text(); + let rows = ledger(ocx); + for (let i = 0; rows.length === rowsBefore && i < 100; i++) { await Bun.sleep(50); rows = ledger(ocx); } + expect(rows.length).toBe(rowsBefore + 1); + return { status: res.status, text, reached: sent.slice(from).map(accountOf), row: rows.at(-1) }; +} +const msg = (ocx: Ocx, headers: Record = {}) => send(ocx, "/v1/messages", + { model: "claude-stub", max_tokens: 16, stream: true, messages: [{ role: "user", content: "hi" }] }, headers); +const combo = (ocx: Ocx) => send(ocx, "/v1/responses", { model: "combo/claude", input: "hi", stream: false }); + +/** Every attempt is labelled with the account it was sent to, and the row with the one that served. */ +function expectServedBy(row: any, reached: Array) { + const served = reached.at(-1)!; + expect(served).toBeDefined(); + expect(row.accountLogLabel).toBe(label(served)); + expect(row.attempts.map((a: any) => a.accountLogLabel)).toEqual(reached.map(id => label(id!))); +} + +afterAll(() => { + fake.stop(true); + for (const home of homes) removeTreeWithRetry(home); +}); + +test("pool off: plain active, pinned and combo rows (with a failover inside the combo) name their serving account", async () => { + limited.clear(); + const [alpha, bravo] = [ids[0]!, ids[1]!]; + const ocx = await startOcx(bravo, {}); + try { + // Plain active account: the provider stays `anthropic` (unchanged), the label names the account. + const plain = await msg(ocx); + expect(plain.status).toBe(200); + expect(plain.reached).toEqual([bravo]); + expect(plain.row.provider).toBe("anthropic"); + expectServedBy(plain.row, plain.reached); + + // AgentRadar pin: the account-qualified provider keeps its form, and its suffix IS the label. + const pinned = await msg(ocx, { "x-agentradar-account-provider": "claude", "x-agentradar-account-id": alpha }); + expect(pinned.status).toBe(200); + expect(pinned.reached).toEqual([alpha]); + expect(pinned.row.provider).toBe(`anthropic-${label(alpha)}`); + expectServedBy(pinned.row, pinned.reached); + + // Combo, active account spent: the ladder fails over to another account, which serves. + limited.add(`Bearer ${token(bravo)}`); + const failover = await combo(ocx); + expect(failover.status).toBe(200); + expect(failover.reached[0]).toBe(bravo); + expect(failover.reached.length).toBeGreaterThan(1); + expect(failover.reached.at(-1)).not.toBe(bravo); + expect(failover.row.provider).toBe("combo"); + expectServedBy(failover.row, failover.reached); + + // Masked only: no raw account id, token or email reaches the ledger. + const text = ledgerText(ocx); + for (const id of ids) { + expect(text).not.toContain(id); + expect(text).not.toContain(token(id)); + } + expect(text).not.toContain("@example.invalid"); + } finally { + ocx.proc.kill(); + } +}, 120_000); + +test("pool on: pooled rows, an in-request pool failover and a pooled combo name their serving account", async () => { + limited.clear(); + const ocx = await startOcx(ids[0]!, { anthropicAccountPool: { enabled: true, strategy: "round-robin" } }); + try { + const pooled = await msg(ocx, { "thread-id": "ledger-s1" }); + expect(pooled.status).toBe(200); + expect(pooled.reached).toHaveLength(1); + expect(pooled.row.provider).toBe(`anthropic-${label(pooled.reached[0]!)}`); + expectServedBy(pooled.row, pooled.reached); + + // Every account but one spent: one request walks the pool, and the last account serves. + const survivor = ids[3]!; + for (const id of ids.slice(0, 3)) limited.add(`Bearer ${token(id)}`); + const walked = await msg(ocx, { "thread-id": "ledger-s2" }); + expect(walked.status).toBe(200); + expect(walked.reached.at(-1)).toBe(survivor); + expect(walked.row.provider).toBe(`anthropic-${label(survivor)}`); + expectServedBy(walked.row, walked.reached); + + const pooledCombo = await combo(ocx); + expect(pooledCombo.status).toBe(200); + expect(pooledCombo.reached.at(-1)).toBe(survivor); + expect(pooledCombo.row.provider).toBe("combo"); + expectServedBy(pooledCombo.row, pooledCombo.reached); + } finally { + ocx.proc.kill(); + } +}, 120_000); diff --git a/tests/usage/usage-client-attribution.test.ts b/tests/usage/usage-client-attribution.test.ts index e675b8c43da..b0581b1e26a 100644 --- a/tests/usage/usage-client-attribution.test.ts +++ b/tests/usage/usage-client-attribution.test.ts @@ -98,6 +98,11 @@ describe("usage client attribution", () => { accumulator.add({ timestamp: now, requestId: "mixed", provider: "combo", model: "combo", client: "codex_cli", status: 200, durationMs: 1, usageStatus: "unreported", attempts: [...attempts, removed] }); const result = accumulator.summarize("30d", now); + // No physical send reached openai, so the openai-filtered view has no request at all. + if (!attempts.some(attempt => attempt.sendCount > 0)) { + expect(result.summary.requests).toBe(0); + continue; + } const measured = expected === "reported" || expected === "estimated" ? 1 : 0; expect(result.summary[`${expected}Requests`]).toBe(1); expect(result.apps[0]?.measuredRequests).toBe(measured); diff --git a/tests/usage/usage-cost.test.ts b/tests/usage/usage-cost.test.ts index 0ecf16758f2..b1ee246cb47 100644 --- a/tests/usage/usage-cost.test.ts +++ b/tests/usage/usage-cost.test.ts @@ -508,6 +508,7 @@ describe("resolveMatchedPrice", () => { for (const expected of [ "openai-apikey/gpt-6.1-sol", "openai-apikey/gpt-6.1-terra", "openai/gpt-6.1-sol", "openai/gpt-6.1-terra", + "openai-apikey/gpt-6.2-astra", "openai/gpt-6.2-astra", "openai-apikey/gpt-6.2-sol", "openai/gpt-6.2-sol", "devin/gpt-6-1-sol", "devin/gpt-6-1-terra", "devin-cli/gpt-6-1-sol", "devin-cli/gpt-6-1-terra", "anthropic/claude-fable-5-1", diff --git a/tests/usage/usage-log-physical.test.ts b/tests/usage/usage-log-physical.test.ts new file mode 100644 index 00000000000..895e88496d1 --- /dev/null +++ b/tests/usage/usage-log-physical.test.ts @@ -0,0 +1,364 @@ +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; +import { STORE_BUDGET_MS } from "../helpers/test-budget"; +import { chmodSync, closeSync, existsSync, mkdtempSync, openSync, readFileSync, rmSync, statSync, truncateSync, writeFileSync, writeSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + appendUsageEntry, + currentUsageLogRevision, + encodePersistedRequestedModel, + normalizeUsageEntryForTest, + normalizeClaudeCompatibilityUsageLog, + normalizePersistedUsageRow, + physicalUsageAttempts, + readRecentUsageEntries, + readUsageEntries, + readUsageEntriesForManagement, + readUsageSnapshotForManagement, + resetUsageReadCacheForTests, + usageForFinalLog, + usageLogPath, + usageStatusForFinalLog, + usageTotalTokens, + usageReadCacheStatsForTests, + usageLogRevisionKey, + type PersistedUsageEntry, +} from "../../src/usage/log"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +let testDir = ""; +let previousHome: string | undefined; + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + testDir = mkdtempSync(join(tmpdir(), "ocx-usage-")); + process.env.OPENCODEX_HOME = testDir; + resetUsageReadCacheForTests(); +}); + +afterEach(() => { + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + if (testDir) removeTreeWithRetry(testDir); +}); + +describe("usage log", () => { + + + + + + + + + test("preserves only literal-true combo target advances", () => { + const entry: PersistedUsageEntry = { + requestId: "ocx-combo-target-advance", + timestamp: 1, + provider: "combo", + model: "combo/test", + status: 200, + durationMs: 1, + usageStatus: "unreported", + comboTargetAdvanced: true, + }; + + expect(normalizeUsageEntryForTest(entry).comboTargetAdvanced).toBe(true); + expect(normalizeUsageEntryForTest({ + ...entry, + comboTargetAdvanced: false, + } as unknown as PersistedUsageEntry).comboTargetAdvanced).toBeUndefined(); + }); + + + test("preserves only valid physical send time and prompt-cache TTL metadata", () => { + const base = { + requestId: "ocx-physical-order", + timestamp: 1, + provider: "anthropic-pabc123", + model: "claude-opus-5", + status: 200, + durationMs: 1, + usageStatus: "reported" as const, + attempts: [{ + ordinal: 1, + provider: "anthropic-pabc123", + model: "claude-opus-5", + adapter: "anthropic", + status: 200, + durationMs: 1, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported" as const, + firstSendAt: 123.25, + promptCacheTtlMs: 3_600_000, + }], + }; + + const normalized = normalizeUsageEntryForTest(base as unknown as PersistedUsageEntry); + expect(normalized.attempts?.[0]).toMatchObject({ + firstSendAt: 123.25, + promptCacheTtlMs: 3_600_000, + }); + + const rejected = normalizeUsageEntryForTest({ + ...base, + attempts: [{ + ...base.attempts[0], + firstSendAt: -1, + promptCacheTtlMs: Number.NaN, + }], + } as unknown as PersistedUsageEntry); + expect(rejected.attempts?.[0]?.firstSendAt).toBeUndefined(); + expect(rejected.attempts?.[0]?.promptCacheTtlMs).toBeUndefined(); + }); + + + + + + /** Build one minimal persisted-usage JSONL line for the given request id. */ + const persistedLine = (requestId: string) => JSON.stringify({ + requestId, + timestamp: 1, + provider: "openai", + model: "gpt-5.5", + status: 200, + durationMs: 1, + usageStatus: "reported", + usage: { inputTokens: 1, outputTokens: 1 }, + totalTokens: 2, + }); + + + + + + + + + test("persists server-tool counters on parent and attempt usage", () => { + appendUsageEntry({ + requestId: "ocx-server-tools", + timestamp: 1, + provider: "anthropic", + model: "claude-opus-5", + status: 200, + durationMs: 10, + usageStatus: "reported", + usage: { + inputTokens: 20, + outputTokens: 2, + anthropicServerToolUse: { web_search_requests: 3, web_fetch_requests: 1 }, + }, + attempts: [{ + ordinal: 1, + provider: "anthropic", + model: "claude-opus-5", + adapter: "anthropic", + status: 200, + durationMs: 10, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported", + usage: { + inputTokens: 20, + outputTokens: 2, + anthropicServerToolUse: { web_search_requests: 3, web_fetch_requests: 1 }, + }, + }], + }); + + expect(readUsageEntries()[0]).toMatchObject({ + usage: { + anthropicServerToolUse: { web_search_requests: 3, web_fetch_requests: 1 }, + }, + attempts: [{ + usage: { + anthropicServerToolUse: { web_search_requests: 3, web_fetch_requests: 1 }, + }, + }], + }); + }); + + test("bounds and sanitizes persisted server-tool counter maps", () => { + const counters = Object.fromEntries([ + ["web_search_requests", 2], + ["negative", -1], + ["not_finite", Number.POSITIVE_INFINITY], + ["__proto__", 1], + ["x".repeat(200), 1], + ...Array.from({ length: 200 }, (_, index) => [`tool_${index}`, index]), + ]); + const normalized = normalizeUsageEntryForTest({ + requestId: "ocx-server-tool-bounds", + timestamp: 1, + provider: "anthropic", + model: "claude-opus-5", + status: 200, + durationMs: 1, + usageStatus: "reported", + usage: { + inputTokens: 1, + outputTokens: 1, + anthropicServerToolUse: counters, + }, + }); + const persisted = normalized.usage?.anthropicServerToolUse; + + expect(persisted?.web_search_requests).toBe(2); + expect(persisted).not.toHaveProperty("negative"); + expect(persisted).not.toHaveProperty("not_finite"); + expect(Object.prototype.hasOwnProperty.call(persisted, "__proto__")).toBe(false); + expect(persisted).not.toHaveProperty("x".repeat(200)); + expect(Object.keys(persisted ?? {}).length).toBeLessThan(200); + }); + + + + + + + + test("keeps physical identity when optional attempt telemetry is malformed", () => { + const valid = { + ordinal: 1, + provider: "anthropic", + model: "claude-opus-5", + adapter: "anthropic", + status: 200, + durationMs: 1, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported", + usage: { inputTokens: 2, outputTokens: 1 }, + totalTokens: 3, + }; + const malformed: Array<[string, Record]> = [ + ["inputTokenEstimate", { ...valid, inputTokenEstimate: -1 }], + ["totalTokens", { ...valid, totalTokens: -1 }], + ["firstOutputMs", { ...valid, firstOutputMs: -1 }], + ["firstOutputMs", { ...valid, firstOutputMs: null }], + ["firstOutputMs", { ...valid, firstOutputMs: "3" }], + ["recoveryCount", { ...valid, recoveryCount: -1 }], + ["usage", { ...valid, usage: { inputTokens: "2", outputTokens: 1 } }], + ["usage", { ...valid, usage: { inputTokens: 2, outputTokens: "1" } }], + ]; + + for (const [field, rawAttempt] of malformed) { + const normalized = normalizeUsageEntryForTest({ + requestId: `optional-${field}`, + timestamp: 1, + provider: "combo", + model: "combo/free", + status: 200, + durationMs: 1, + usageStatus: "reported", + attempts: [rawAttempt as unknown as typeof valid], + }); + expect(normalized.attempts).toHaveLength(1); + expect(normalized.attempts?.[0]).toMatchObject({ + ordinal: 1, + provider: "anthropic", + model: "claude-opus-5", + status: 200, + }); + expect(normalized.attempts?.[0]).not.toHaveProperty(field); + } + }); + + test("retains nonphysical duplicate ordinals until physical filtering", () => { + const physical = { + ordinal: 1, + provider: "anthropic", + model: "claude-opus-5", + adapter: "anthropic", + status: 200, + durationMs: 2, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported" as const, + usage: { inputTokens: 10, outputTokens: 2 }, + totalTokens: 12, + }; + + for (const diagnostic of [ + { ...physical, provider: "local", locallyAnswered: true }, + { ...physical, provider: "unsent", sendCount: 0 }, + ]) { + const normalized = normalizeUsageEntryForTest({ + requestId: `duplicate-${diagnostic.provider}`, + timestamp: 1, + provider: "combo", + model: "combo/free", + status: 200, + durationMs: 3, + usageStatus: "reported", + attempts: [diagnostic, physical], + }); + + expect(normalized.attempts?.map(attempt => attempt.provider)).toEqual([ + diagnostic.provider, + physical.provider, + ]); + expect(physicalUsageAttempts(normalized.attempts ?? [])).toEqual([physical]); + } + }); + + + + + + + + + + + + // A turn the proxy answered locally issued no upstream request, so its zero counts are exact. + // The provider-wide estimated marking exists because the Kiro/Cursor adapters can only guess a + // real inference's usage; there is nothing to guess when there was no inference, and marking it + // estimated makes a no-send turn indistinguishable from one whose usage frame never arrived. + + + + + + test.skipIf(process.platform === "win32")( + "appendUsageEntry re-narrows externally widened permissions after the bounded cache window", + () => { + const now = Date.now(); + const clock = spyOn(Date, "now").mockReturnValue(now); + try { + appendUsageEntry({ + requestId: "ocx-permission-initial", + timestamp: now, + provider: "openai", + model: "gpt-4o", + status: 200, + durationMs: 10, + usageStatus: "unreported", + }); + chmodSync(testDir, 0o755); + chmodSync(usageLogPath(), 0o644); + + clock.mockReturnValue(now + 1_000); + appendUsageEntry({ + requestId: "ocx-permission-recheck", + timestamp: now + 1_000, + provider: "openai", + model: "gpt-4o", + status: 200, + durationMs: 11, + usageStatus: "unreported", + }); + + expect(statSync(testDir).mode & 0o777).toBe(0o700); + expect(statSync(usageLogPath()).mode & 0o777).toBe(0o600); + } finally { + clock.mockRestore(); + } + }, + ); + +}); diff --git a/tests/usage/usage-log.test.ts b/tests/usage/usage-log.test.ts index fc43c30931e..bb30d2aebd6 100644 --- a/tests/usage/usage-log.test.ts +++ b/tests/usage/usage-log.test.ts @@ -662,7 +662,7 @@ describe("usage log", () => { expect(value.attempts?.[0]?.locallyAnswered).toBe(true); }); - test("drops only malformed persisted attempts while preserving valid siblings", () => { + test("drops malformed attempt identity while preserving valid siblings", () => { const valid = (ordinal: number) => ({ ordinal, provider: ordinal === 1 ? "a" : "c", @@ -680,13 +680,6 @@ describe("usage log", () => { { ...valid(2), status: 99 }, { ...valid(2), status: 600 }, { ...valid(2), status: 200.5 }, - { ...valid(2), inputTokenEstimate: -1 }, - { ...valid(2), totalTokens: -1 }, - { ...valid(2), firstOutputMs: -1 }, - { ...valid(2), firstOutputMs: null }, - { ...valid(2), firstOutputMs: "3" }, - { ...valid(2), usage: { inputTokens: "2", outputTokens: 1 } }, - { ...valid(2), usage: { inputTokens: 2, outputTokens: "1" } }, ]; for (const middle of malformed) { writeFileSync(usageLogPath(), `${JSON.stringify({ diff --git a/tests/usage/usage-summary-physical.test.ts b/tests/usage/usage-summary-physical.test.ts new file mode 100644 index 00000000000..7191aab4afe --- /dev/null +++ b/tests/usage/usage-summary-physical.test.ts @@ -0,0 +1,633 @@ +import { describe, expect, test } from "bun:test"; +import { + MAX_USAGE_MODEL_BREAKDOWN_ROWS, + MAX_USAGE_DAY_BUCKETS, + USAGE_RANGES, + USAGE_SURFACES, + createUsageSummaryAccumulator, + cacheTokensFromUsage, + parseRange, + parseUsageSurface, + projectedComboUsage, + projectUsageSummary, + rangeWindow, + summarizeUsage, +} from "../../src/usage/summary"; +import type { PersistedUsageEntry } from "../../src/usage/log"; +import { computeEntryCost } from "../../src/usage/summary"; +import { buildRouteDecisionTrace } from "../../src/routing/trace"; +import { isUnresolvedRequestedModel } from "../../src/usage/model-identity"; + +const FIXED_NOW = Date.UTC(2026, 5, 28, 12, 0, 0); + +describe("custom usage windows", () => { + + const since = new Date(2026, 1, 10, 12, 0, 0, 123).getTime(); + const until = since + 3_600_000; + + + + + + +}); + +function entry(overrides: Partial & { ts: number }): PersistedUsageEntry { + const { ts, ...rest } = overrides; + return { + requestId: rest.requestId ?? `req-${ts}`, + timestamp: ts, + provider: rest.provider ?? "openai", + model: rest.model ?? "gpt-5.5", + status: rest.status ?? 200, + durationMs: rest.durationMs ?? 10, + usageStatus: rest.usageStatus ?? "unreported", + ...(rest.surface === "claude" ? { surface: rest.surface } : {}), + ...(rest.accountLogLabel !== undefined ? { accountLogLabel: rest.accountLogLabel } : {}), + ...(rest.resolvedModel !== undefined ? { resolvedModel: rest.resolvedModel } : {}), + ...(rest.usage ? { usage: rest.usage } : {}), + ...(rest.totalTokens !== undefined ? { totalTokens: rest.totalTokens } : {}), + ...(rest.attempts ? { attempts: rest.attempts } : {}), + ...(rest.apiKeyId !== undefined ? { apiKeyId: rest.apiKeyId } : {}), + ...(rest.routeDecision ? { routeDecision: rest.routeDecision } : {}), + }; +} + +describe("unresolved requested model attribution", () => { + function fallback(model: string, overrides: Partial = {}): PersistedUsageEntry { + return entry({ + ts: FIXED_NOW - 1, provider: "kimi", model, usageStatus: "reported", + usage: { inputTokens: 30, outputTokens: 8 }, totalTokens: 38, + routeDecision: buildRouteDecisionTrace({ + requestedModel: model, routeKind: "default-provider", + selected: { provider: "kimi", model, reason: "default-provider" }, + }), + ...overrides, + }); + } + + + + + + +}); + +describe("parseRange", () => { + +}); + +describe("today range (CLI cost query)", () => { + + +}); + +describe("day-level estimated cost", () => { + const at = Date.UTC(2026, 5, 28, 10, 0, 0); + + + + + + // Only the tail row carries cache telemetry, so its 100 input tokens are the whole denominator + // (`cacheObservedInputTokens`); the other rows folded into "other" never measured cache and + // are not miss evidence. A read ABOVE that denominator is contradictory telemetry, which + // `cacheTokensFromUsage` refuses outright, so the clamp is only reachable at exactly 1. +}); + + +describe("canonical range and surface constants", () => { +}); + +describe("projectUsageSummary", () => { + const at = Date.UTC(2026, 5, 28, 10, 0, 0); + const priced = { inputTokens: 1_000, outputTokens: 100 }; + + + + + test("projects additive usage with the latest context checkpoint and bounded server-tool sums", () => { + const attempts: NonNullable = [ + { + ordinal: 1, + provider: "anthropic", + model: "claude-opus-5", + adapter: "anthropic", + status: 502, + durationMs: 10, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported", + usage: { + inputTokens: 10, + outputTokens: 1, + contextTotalTokens: 100, + reasoningOutputTokens: 2, + anthropicServerToolUse: { + web_search_requests: 1, + invalid_negative: -1, + invalid_infinite: Number.POSITIVE_INFINITY, + }, + }, + totalTokens: 11, + }, + { + ordinal: 2, + provider: "anthropic", + model: "claude-opus-5", + adapter: "anthropic", + status: 502, + durationMs: 10, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported", + usage: { + inputTokens: 20, + outputTokens: 2, + anthropicServerToolUse: { web_search_requests: 2, web_fetch_requests: 1 }, + }, + totalTokens: 22, + }, + { + ordinal: 3, + provider: "anthropic", + model: "claude-opus-5", + adapter: "anthropic", + status: 200, + durationMs: 10, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported", + usage: { + inputTokens: 30, + outputTokens: 3, + contextTotalTokens: 80, + reasoningOutputTokens: 3, + anthropicServerToolUse: { web_search_requests: 3 }, + }, + totalTokens: 33, + }, + ]; + + expect(projectedComboUsage(attempts)).toEqual({ + usage: { + inputTokens: 60, + outputTokens: 6, + contextTotalTokens: 80, + reasoningOutputTokens: 5, + anthropicServerToolUse: { web_search_requests: 6, web_fetch_requests: 1 }, + }, + totalTokens: 66, + }); + }); + + + + + + +}); + +describe("parseUsageSurface", () => { + +}); + +describe("summarizeUsage", () => { + + + + + + + + + + test("canonical implicit cache reads are not reduced by cache writes", () => { + const sum = summarizeUsage([entry({ + ts: FIXED_NOW - 1000, + provider: "anthropic", + usageStatus: "reported", + usage: { + inputTokens: 160, + outputTokens: 10, + cachedInputTokens: 60, + cacheCreationInputTokens: 20, + }, + totalTokens: 170, + })], "30d", FIXED_NOW); + + expect(sum.summary).toMatchObject({ + inputTokens: 160, + outputTokens: 10, + cachedInputTokens: 60, + cacheReadInputTokens: 60, + cacheCreationInputTokens: 20, + totalTokens: 170, + }); + }); + + + + + + + + + + + test("preserves stored totals when physical attempt usage is unavailable", () => { + const sum = summarizeUsage([entry({ + ts: FIXED_NOW - 1, + requestId: "combo-total-only", + provider: "combo", + model: "combo/free", + usageStatus: "unreported", + attempts: [{ + ordinal: 1, + provider: "openai", + model: "gpt-5.5", + adapter: "openai-responses", + accountLogLabel: "main", + status: 200, + durationMs: 2, + sendCount: 1, + recoveryKinds: [], + usageStatus: "unreported", + totalTokens: 17, + }], + })], "30d", FIXED_NOW); + + expect(sum.summary.totalTokens).toBe(17); + expect(sum.days.find(day => day.requests === 1)?.totalTokens).toBe(17); + expect(sum.days.find(day => day.requests === 1)?.models[0]?.totalTokens).toBe(17); + expect(sum.models[0]?.totalTokens).toBe(17); + expect(sum.providers[0]?.totalTokens).toBe(17); + expect(sum.accounts[0]).toMatchObject({ + accountLogLabel: "main", + totalTokens: 17, + measuredAttempts: 0, + unmeteredAttempts: 1, + usageCoverageRatio: 0, + }); + }); + + test("filters duplicate, local, and unsent attempts before every summary aggregate", () => { + const physical = { + ordinal: 1, + provider: "openai", + model: "gpt-5.5", + adapter: "openai-chat", + status: 200, + durationMs: 2, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported" as const, + usage: { inputTokens: 10, outputTokens: 2 }, + totalTokens: 12, + }; + const combo = entry({ + ts: FIXED_NOW - 1, + requestId: "combo-physical-only", + provider: "combo", + model: "combo/free", + usageStatus: "reported", + usage: { inputTokens: 9_999, outputTokens: 999 }, + totalTokens: 10_998, + attempts: [ + physical, + { ...physical }, + { + ...physical, + ordinal: 2, + provider: "local", + model: "local-model", + locallyAnswered: true, + usage: { inputTokens: 1_000, outputTokens: 100 }, + totalTokens: 1_100, + }, + { + ...physical, + ordinal: 3, + provider: "unsent", + model: "unsent-model", + status: 502, + sendCount: 0, + usage: { inputTokens: 2_000, outputTokens: 200 }, + totalTokens: 2_200, + }, + ], + }); + + const sum = summarizeUsage([combo], "30d", FIXED_NOW); + expect(sum.summary).toMatchObject({ + requests: 1, + attemptCount: 1, + inputTokens: 10, + outputTokens: 2, + totalTokens: 12, + }); + expect(sum.providers).toHaveLength(1); + expect(sum.providers[0]).toMatchObject({ + provider: "openai", + requests: 1, + attemptCount: 1, + totalTokens: 12, + }); + expect(sum.models).toHaveLength(1); + expect(sum.models[0]).toMatchObject({ + provider: "openai", + model: "gpt-5.5", + requests: 1, + attemptCount: 1, + totalTokens: 12, + }); + }); + + test("attributes legacy root usage to final physical attempt when attempt usage is absent", () => { + const sum = summarizeUsage([entry({ + ts: FIXED_NOW - 1, + requestId: "combo-root-fallback", + provider: "combo", + model: "combo/free", + usageStatus: "reported", + usage: { + inputTokens: 100, + outputTokens: 5, + cacheReadInputTokens: 80, + cacheCreationInputTokens: 10, + }, + totalTokens: 105, + attempts: [{ + ordinal: 1, + provider: "anthropic", + model: "claude-opus-5", + adapter: "anthropic", + status: 200, + durationMs: 2, + sendCount: 1, + recoveryKinds: [], + usageStatus: "unreported", + }], + })], "30d", FIXED_NOW); + + expect(sum.summary).toMatchObject({ + requests: 1, + attemptCount: 1, + measuredRequests: 1, + inputTokens: 100, + outputTokens: 5, + cacheReadInputTokens: 80, + cacheCreationInputTokens: 10, + totalTokens: 105, + }); + expect(sum.providers).toEqual([ + expect.objectContaining({ + provider: "anthropic", + requests: 1, + attemptCount: 1, + totalTokens: 105, + }), + ]); + }); + + test("an explicit empty attempt list never falls back to stale parent usage", () => { + const sum = summarizeUsage([entry({ + ts: FIXED_NOW - 1, + requestId: "combo-empty", + provider: "combo", + model: "combo/free", + usageStatus: "reported", + usage: { inputTokens: 9_999, outputTokens: 999 }, + totalTokens: 10_998, + attempts: [], + })], "30d", FIXED_NOW); + + expect(sum.summary).toMatchObject({ + requests: 1, + attemptCount: 0, + measuredRequests: 0, + totalTokens: 0, + }); + }); + + test("an all-nonphysical attempt list never falls back to stale parent usage", () => { + const diagnostic = { + ordinal: 1, + provider: "local", + model: "local-model", + adapter: "anthropic", + status: 200, + durationMs: 1, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported" as const, + locallyAnswered: true, + usage: { inputTokens: 1_000, outputTokens: 100 }, + totalTokens: 1_100, + }; + const sum = summarizeUsage([entry({ + ts: FIXED_NOW - 1, + requestId: "combo-nonphysical-only", + provider: "combo", + model: "combo/free", + usageStatus: "reported", + usage: { inputTokens: 9_999, outputTokens: 999 }, + totalTokens: 10_998, + attempts: [diagnostic, { + ...diagnostic, + ordinal: 2, + provider: "unsent", + locallyAnswered: false, + sendCount: 0, + }], + })], "30d", FIXED_NOW); + + expect(sum.summary).toMatchObject({ + requests: 1, + attemptCount: 0, + measuredRequests: 0, + inputTokens: 0, + outputTokens: 0, + totalTokens: 0, + }); + expect(sum.providers).toEqual([]); + expect(sum.models).toEqual([]); + }); + + + + + + + + + test("lets unpriced attempts dominate priced attempts in the collapsed model row", () => { + const retained = Array.from({ length: MAX_USAGE_MODEL_BREAKDOWN_ROWS - 1 }, (_, index) => + [0, 1].map(request => entry({ + ts: FIXED_NOW - index * 2 - request, + requestId: `retained-${index}-${request}`, + provider: `provider-${index}`, + model: `model-${index}`, + usageStatus: "reported", + usage: { inputTokens: 1, outputTokens: 1 }, + }))).flat(); + const mixed = entry({ + ts: FIXED_NOW, + requestId: "mixed-overflow-pricing", + provider: "combo", + model: "combo/native", + usageStatus: "reported", + attempts: [ + { + ordinal: 1, + provider: "openai", + model: "gpt-5.5", + status: 200, + durationMs: 10, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported", + usage: { inputTokens: 100, outputTokens: 10 }, + }, + { + ordinal: 2, + provider: "openai", + model: "unpriced-overflow-model", + status: 200, + durationMs: 10, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported", + usage: { inputTokens: 100, outputTokens: 10 }, + }, + ], + }); + + const sum = summarizeUsage([...retained, mixed], "30d", FIXED_NOW); + expect(sum.models.at(-1)).toMatchObject({ + provider: "other", + model: "other", + requests: 1, + pricedRequests: 0, + unpricedRequests: 1, + priceCoverageRatio: 0, + }); + }); + + + + test.each([ + ["canonical", { inputTokens: 100, outputTokens: 20, cacheReadInputTokens: 150 }], + ["legacy", { inputTokens: 100, outputTokens: 20, cachedInputTokens: 150, cacheCreationInputTokens: 80 }], + ])("keeps contradictory %s cache telemetry unavailable", (_kind, usage) => { + expect(cacheTokensFromUsage(usage)).toEqual({ + read: undefined, + creation: undefined, + hasCacheTelemetry: false, + }); + + const sum = summarizeUsage([ + entry({ + ts: FIXED_NOW - 1000, + provider: "anthropic", + model: "claude-sonnet-5", + usageStatus: "reported", + usage, + }), + ], "30d", FIXED_NOW); + expect(sum.models[0]?.cacheHitRate).toBeNull(); + }); + + test("keeps same-route mixed physical pricing below full coverage", () => { + const sum = summarizeUsage([entry({ + ts: FIXED_NOW - 1000, + requestId: "same-route-mixed-pricing", + provider: "combo", + model: "combo/native", + usageStatus: "reported", + attempts: [ + { + ordinal: 1, + provider: "openai", + model: "gpt-5.5", + adapter: "openai-responses", + status: 502, + durationMs: 10, + sendCount: 1, + recoveryKinds: [], + usageStatus: "reported", + usage: { inputTokens: 100, outputTokens: 10 }, + }, + { + ordinal: 2, + provider: "openai", + model: "gpt-5.5", + adapter: "openai-responses", + status: 200, + durationMs: 20, + sendCount: 1, + recoveryKinds: [], + usageStatus: "unreported", + }, + ], + })], "30d", FIXED_NOW); + + expect(sum.models.find(model => model.model === "gpt-5.5")).toMatchObject({ + requests: 1, + pricedRequests: 0, + unpricedRequests: 1, + priceCoverageRatio: 0, + }); + expect(sum.providers.find(provider => provider.provider === "openai")).toMatchObject({ + requests: 1, + pricedRequests: 0, + unpricedRequests: 1, + priceCoverageRatio: 0, + }); + }); + + +}); + +describe("UsageSummaryAccumulator modes", () => { + const at = Date.UTC(2026, 5, 28, 10, 0, 0); + + + + + + + + +}); + +test("a refused attempt does not zero the cost of the attempt that succeeded", () => { + // Combo failover: target 1 is refused (a physical send, no usage to price), target 2 serves the + // turn. Treating the refusal as UNPRICEABLE rather than as a known zero reported the whole + // request at $0, so real money spent on target 2 disappeared from the daily total. + const entry = { + requestId: "combo-failover-cost", + timestamp: Date.now(), + provider: "combo", + model: "combo/waterfall", + status: 200, + durationMs: 900, + usageStatus: "reported", + attempts: [ + { + ordinal: 1, provider: "openai", model: "gpt-5.6", adapter: "openai-responses", + status: 429, durationMs: 40, sendCount: 1, recoveryKinds: [], usageStatus: "unreported", + }, + { + ordinal: 2, provider: "anthropic", model: "claude-opus-4-5", adapter: "anthropic", + status: 200, durationMs: 800, sendCount: 1, recoveryKinds: [], usageStatus: "reported", + usage: { inputTokens: 50_000, outputTokens: 1_000, totalTokens: 51_000 }, + }, + ], + } as unknown as PersistedUsageEntry; + + const cost = computeEntryCost(entry); + + expect(cost.isPriced).toBe(true); + expect(cost.costTotal).toBeGreaterThan(0); +}); diff --git a/tests/usage/usage-summary.test.ts b/tests/usage/usage-summary.test.ts index 50242799917..13691c88c70 100644 --- a/tests/usage/usage-summary.test.ts +++ b/tests/usage/usage-summary.test.ts @@ -420,9 +420,12 @@ describe("day-level estimated cost", () => { expect(day!.estimatedCostUsd).toBeCloseTo(modelSum, 10); }); + // A read above the observed input denominator is contradictory telemetry, which + // `cacheTokensFromUsage` refuses outright, so the clamp is only reachable at exactly 1. test.each([ [0, 0], - [150, 1], + [50, 0.5], + [100, 1], ])("overflow rows preserve cache reads and clamp cache hit rate (%d reads)", (cacheRead, expected) => { const total = MAX_USAGE_MODEL_BREAKDOWN_ROWS + 1; const entries = Array.from({ length: total }, (_, i) => entry({ @@ -1680,20 +1683,6 @@ describe("summarizeUsage", () => { expect(dayUnpriced?.cacheHitRate).toBeNull(); }); - test("clamps cache hit rate when cache reads exceed input tokens", () => { - const sum = summarizeUsage([ - entry({ - ts: FIXED_NOW - 1000, - provider: "anthropic", - model: "claude-sonnet-5", - usageStatus: "reported", - usage: { inputTokens: 100, outputTokens: 20, cacheReadInputTokens: 150 }, - }), - ], "30d", FIXED_NOW); - - expect(sum.models[0]?.cacheHitRate).toBe(1); - }); - test("attributes combo with mixed priced and unpriced attempts per attempt", () => { const combo = entry({ ts: FIXED_NOW - 1000, diff --git a/tests/web-search/web-search-overflow-usage.test.ts b/tests/web-search/web-search-overflow-usage.test.ts new file mode 100644 index 00000000000..35716ec8ee5 --- /dev/null +++ b/tests/web-search/web-search-overflow-usage.test.ts @@ -0,0 +1,65 @@ +import { expect, test } from "bun:test"; +import { parseRequest } from "../../src/responses/parser"; +import { runWithWebSearch } from "../../src/web-search/loop"; +import type { ProviderAdapter } from "../../src/adapters/base"; +import type { OcxProviderConfig } from "../../src/types"; +import { collectSse } from "../helpers/responses-conformance"; +import { createTestTranslatorBudget } from "../helpers/translator-budget"; + +const forwardProvider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://chatgpt.test/v1", + authMode: "forward", +}; + +test("usage-bearing translator overflow reaches request and wire without sidecar dispatch", async () => { + const aggregate: unknown[] = []; + const sidecarOutcomes: unknown[] = []; + const adapter: ProviderAdapter = { + name: "overflow", + buildRequest: () => ({ url: "https://routed.test/v1", method: "POST", headers: {}, body: "{}" }), + fetchResponse: async () => new Response("wire", { status: 200 }), + async *parseStream() { + yield { type: "tool_call_start", id: "ws1", name: "web_search" }; + yield { type: "tool_call_delta", arguments: '{"q":"docs"}' }; + yield { type: "tool_call_end" }; + yield { + type: "error", + status: 502, + errorType: "upstream_error", + code: "translation_buffer_limit", + message: "upstream translation buffer exceeded the safe limit", + usage: { inputTokens: 7, outputTokens: 1 }, + }; + }, + }; + const response = await runWithWebSearch({ + parsed: parseRequest({ model: "routed/model", input: "hi", stream: true, tools: [{ type: "web_search" }] }), + adapter, + forwardProvider, + hostedTool: { type: "web_search" }, + selectedForwardHeaders: new Headers({ authorization: "Bearer token" }), + settings: { model: "gpt-5.6-luna", reasoning: "low", timeoutMs: 30_000 }, + maxSearches: 1, + incomingMeta: { headers: new Headers(), translatorBudget: createTestTranslatorBudget() }, + onUsage: usage => aggregate.push(usage), + recordSidecarOutcome: outcome => sidecarOutcomes.push(outcome), + }); + + const frames = await collectSse(response.body!); + const failed = frames.filter(frame => frame.event === "response.failed"); + expect(failed).toHaveLength(1); + const failedResponse = failed[0]?.data.response as { + error?: { code?: string }; + usage?: Record; + }; + expect(failedResponse.error?.code).toBe("translation_buffer_limit"); + expect(failedResponse.usage).toMatchObject({ + input_tokens: 7, + output_tokens: 1, + total_tokens: 8, + }); + expect(frames.some(frame => frame.event === "response.completed")).toBe(false); + expect(aggregate).toEqual([{ inputTokens: 7, outputTokens: 1 }]); + expect(sidecarOutcomes).toEqual([]); +});