Skip to content

fix(antigravity): let a 429 hop's 5xx replacement take the one fault hop - #8839

Merged
lidge-jun merged 1 commit into
devfrom
fix/antigravity-429-then-5xx
Nov 16, 2026
Merged

lidge-jun merged 1 commit into
devfrom
fix/antigravity-429-then-5xx

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Addresses CodeRabbit review thread PRRT_kwDOS-0Gi863wcE0 on #8778.

With three eligible Antigravity accounts, account A answers 429 and the generic 429 account hop replays on B. If B's same-account transient ladder then ends in a replayable 502/503/504, the request failed with B's 5xx and never tried C, even though the one-hop 5xx sibling rotation was still unspent. The 429 loop re-entered recovery only for non-replayable responses, so the 5xx arm never saw B's response.

  • New src/server/responses/antigravity-fault-rotation.ts (isAntigravityFaultRotatable) holds the existing 5xx-rotation guard: Antigravity, 502/503/504, Google adapter, pool activated, not yet attempted, replayable, failover limit not reached. The 5xx arm in adapter-dispatch.ts uses it unchanged in meaning.
  • After the generic 429 account hop, recovery re-enters when that guard holds, so the one fault hop can take the request from B to C. antigravityFaultRotationAttempted still limits it to one hop per exchange, and the failover limit and send budget still apply.
  • Only the generic 429 loop changes. The Kiro loop near the same place cannot reach the Antigravity-only 5xx hop.
  • adapter-dispatch.ts shrinks from 1,993 to 1,991 lines.
  • structure/transports/responses.md gains an ownership row for the new module, and tests/helpers/responses-core-source.ts lists it in the module roster. The provider docs already state that a surviving 502/503/504 replays once on an eligible sibling; this change makes the 429-then-5xx path honour that, so the en/es docs need no edit.

This PR does not overlap #8835 (fix/antigravity-auth-failed-enum, the other half of the #8778 follow-ups). git merge-tree of the two heads is clean.

Verification

Regression test (focused, single file under /usr/bin/lockf -k /tmp/ocx-test.lock):

  • tests/server/server-google-antigravity-oauth-401-replay.test.ts → "a 429 hop whose sibling then answers 5xx takes the one fault hop to a third account". The script is A:429 ×3, B:503 ×3, C:200. It expects a 200 from C on project-c, A to keep its rate-limit cooldown, B to get the fault cooldown (both ineligible), no needs-reauth mark, and both oauth-account-429 and transient-5xx recovery kinds. On origin/dev 3ef14ad it fails: the sends stop after B's three 503s and C is never tried. With the fix the whole file passes, 65 pass / 0 fail.
  • Also green with the fix: responses-core-modules, server-google-antigravity-oauth-429-budget, inference-send-budget, server-generic-oauth-affinity, generic-oauth-failover, server-key-failover-e2e, responses-send-budget-errors, errors-adapter-failure, openrouter-quota-reset-cooldown-4024, oauth-account-quota-rank, server-antigravity-model-denial-recovery, server-antigravity-web-search-validation, provider-antigravity-quota-retry, file-size-ratchet, structure-ssot.
  • bun run typecheck, bun run structure:check, bun run privacy:scan: pass.

NOT RUN: full bun run test and test:changed locally. The host is under heavy load and this lane is limited to focused single-file runs. Broader coverage is left to CI (ci:full).

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Summary by CodeRabbit

  • Bug Fixes
    • Improved recovery when a request encounters temporary upstream server errors after switching accounts. Eligible requests can now continue to another account, increasing the chance of a successful response without incorrectly marking the affected accounts as unavailable. This also covers sequences involving repeated rate-limit responses followed by temporary server errors.

The generic 429 account-hop loop re-entered recovery only for non-replayable
responses, so a replayable 502/503/504 from the replacement account left the loop
and the one-hop 5xx sibling rotation never ran: A:429 -> B:503 failed without
trying C. The 5xx guard moves to antigravity-fault-rotation.ts and the 429 hop
re-enters recovery when it holds, still once per exchange.
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner November 16, 2026 13:33
@lidge-jun lidge-jun added the ci:full Run the four full Ubuntu test shards on this pull request label Nov 16, 2026
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The dispatcher now shares one predicate for Antigravity fault-rotation eligibility. After generic OAuth account failover, it continues recovery when the replacement response qualifies for fault rotation. A regression test covers a 429 response, sibling 503 retries, and success from a third account.

Changes: Antigravity response recovery

Layer / File(s) Summary
Centralize rotation eligibility
src/server/responses/antigravity-fault-rotation.ts, src/server/responses/adapter-dispatch.ts
The new predicate checks the provider, response status, adapter, pool state, rotation-attempt state, replayability, and failover limit. The dispatcher uses the predicate for held responses and replacement responses.
Continue recovery after account failover
src/server/responses/adapter-dispatch.ts, tests/server/server-google-antigravity-oauth-401-replay.test.ts, structure/transports/responses.md, tests/helpers/responses-core-source.ts
After generic OAuth account failover, recovery continues for non-replayable responses or responses eligible for fault rotation. The regression test checks repeated 429 responses on the first account, repeated 503 responses on its sibling, and success on a third account. The test also checks send order, credentials, recovery records, and account eligibility. The module documentation and test source inventory include the new predicate.

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Dispatch as adapter-dispatch
  participant First as First Antigravity account
  participant Sibling as Sibling account
  participant Third as Third account
  Dispatch->>First: Send request
  First-->>Dispatch: Return 429
  Dispatch->>Sibling: Retry after generic OAuth account failover
  Sibling-->>Dispatch: Return repeated 503 responses
  Dispatch->>Third: Rotate after eligible 503 response
  Third-->>Dispatch: Return success
Loading

Possibly related PRs

  • lidge-jun/opencodex#8778: Adds sibling failover and 502/503/504 rotation behavior that this change uses when continuing recovery after a 429 account hop.
Merge Risk: Minimal · up to e2a0b

Antigravity requests that hit a 429 on one account and then a 503 on its sibling can now recover on a third account. The change is small and bounded, and a regression test covers the new path. No merge-blocking risk was found.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the main change: a 5xx replacement response returned by a 429 account hop is allowed to use the one-hop Antigravity fault rotation. The raw summary confirms the new isAntigravityFaultR…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: a 5xx replacement response returned by a 429 account hop is allowed to use the one-hop Antigravity fault rotation. The raw summary confirms the new isAntigravityFaultRotatable guard, its use in adapter-dispatch.ts for re-entering recovery, and the regression test for the 429-then-503 sequence. The title is specific, uses the conventional fix(scope) form, and stays under typical length limits, although at 72 characters it is slightly long.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 3ef14ad and e2a0b00.

Files selected for processing: 5

  • src/server/responses/adapter-dispatch.ts
  • src/server/responses/antigravity-fault-rotation.ts
  • structure/transports/responses.md
  • tests/helpers/responses-core-source.ts
  • tests/server/server-google-antigravity-oauth-401-replay.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 1 remain after this review.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T13:36:12.124754Z e2a0b00 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Nov 16, 2026
@lidge-jun

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor
Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@lidge-jun

Copy link
Copy Markdown
Owner Author

Maintainer integration of #8839 at e2a0b00. It fixes CodeRabbit thread PRRT_kwDOS-0Gi863wcE0, left on #8778.

  • What it fixes: after a 429 hop, a replayable 502, 503 or 504 from the sibling now re-enters recovery whenever the one-hop 5xx rotation check passes, which still runs at most once per request. The check moved into antigravity-fault-rotation.ts, and adapter-dispatch.ts went from 1,993 to 1,991 lines.
  • Regression test: A:429 → B:503 → C:200 now succeeds through C, with the right cooldowns and no reauth mark. On dev the request stopped after B.
  • Known follow-up, not in scope: the verify-account 403 hop loop has the same shape and is untested. It is queued.
  • Review gate: Codex completed on this exact head with no unresolved findings, CodeRabbit reviewed this head, and no review thread is open. CI on this exact head: green, including enforce-target. assert-mergeable-review: OK. Simulated merge onto the dev of this moment: clean, and no file over its size cap. Dev state: batch lane=all 43890997984 (147b59f) failed only windows 11/14 (active-turn-disconnect timeout cascade). The same product code passed that shard and case in 43874258334 (3ef14ad to 147b59f changes only test(claude-cli): hold the reader until the pre-fix child exits so the frame is always cut #8840's test files), so it is an intermittent hosted-Windows failure, recorded on Windows hosted CI quarantine for 2.98.0 (overflow matrix, client-link-stop, CLI plan) #8262.

@lidge-jun
lidge-jun merged commit 18bb646 into dev Nov 16, 2026
40 of 41 checks passed
@lidge-jun
lidge-jun deleted the fix/antigravity-429-then-5xx branch November 16, 2026 15:27
lidge-jun added a commit that referenced this pull request Nov 16, 2026
… one fault hop (#8846)

The verify-account 403 hop loop re-entered recovery only for non-replayable
responses, so a replayable 502/503/504 from the replacement account left the
loop and the one-hop 5xx sibling rotation never ran: A:403 verify -> B:503
failed without trying C. The hop now asks isAntigravityFaultRotatable, the
same guard the 429 hop uses since #8839, still once per request.

Follow-up to #8839 (CodeRabbit thread PRRT_kwDOS-0Gi863wcE0 on #8778).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci:full Run the four full Ubuntu test shards on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant