Skip to content

fix(antigravity): let a verify-account hop's 5xx replacement take the one fault hop - #8846

Merged
lidge-jun merged 1 commit into
devfrom
fix/antigravity-verify-hop-5xx
Nov 16, 2026
Merged

lidge-jun merged 1 commit into
devfrom
fix/antigravity-verify-hop-5xx

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

On a Google Antigravity account pool, a verify-account 403 on account A rotates to sibling B. If B then returned a replayable 502, 503 or 504 that outlived its same-account retries, the request failed with that 5xx and never tried C. The verify-account hop loop re-entered recovery only for non-replayable responses, so the one-hop 5xx sibling rotation never saw B's failure. #8839 fixed the same gap for the 429 hop.

  • src/server/responses/adapter-dispatch.ts: the verify-account hop now also re-enters recovery when isAntigravityFaultRotatable (from antigravity-fault-rotation.ts) holds for the replacement. The one-hop guard still limits the fault rotation to once per request. Line count is unchanged at 1,991.
  • tests/server/server-google-antigravity-oauth-401-replay.test.ts: new case A:403 verify-account → B:503 ×3 → C:200. It asserts the send order, that C's project carries the answer, that A keeps its verify_account quarantine, that B is cooled but not marked, and that the attempt records both the oauth-account-403 and transient-5xx recovery kinds.
  • Docs: the guard's docstring, the antigravity-fault-rotation.ts row in structure/transports/responses.md, and the en and es reference/configuration/providers.md 5xx paragraph now say the replay also covers a 5xx from the sibling that a 429 or verify-account hop reached.

Follow-up to #8839 (CodeRabbit thread PRRT_kwDOS-0Gi863wcE0, left on #8778).

Verification

  • Regression on origin/dev (950b7d9) with only the test added: bun test tests/server/server-google-antigravity-oauth-401-replay.test.ts -t 'verify 403 hop whose sibling' failed. The sends were rejected-access, access-b ×3 and access-c was never tried.
  • With the fix: server-google-antigravity-oauth-401-replay, server-antigravity-verify-replay-scope, server-google-antigravity-oauth-429-budget, oauth/generic-oauth-failover, server-antigravity-web-search-validation: 179 pass, 0 fail.
  • responses-core-modules, responses-reasoning-effort-downgrade, file-size-ratchet: 160 pass, 0 fail. docs-spanish-localization, docs-link-targets (with the 401-replay file): 107 pass, 0 fail.
  • bun run typecheck, bun run structure:check, bun run privacy:scan: pass.
  • Tests ran under /usr/bin/lockf -k /tmp/ocx-test.lock. The full suite was not run locally (concurrent worktrees on this host); ci:full covers it.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Summary by CodeRabbit

  • New Features
    • Antigravity account failover can now continue to eligible sibling accounts after supported, structured 403 permission errors and exhausted 5xx retries, including after certain account switches.
    • Failed accounts are temporarily skipped during recovery: 60 seconds for supported 403 rejections and 15 seconds for upstream 502, 503, or 504 errors.
    • Web search requests using fetch can also fail over after supported 403 rejections.
  • Documentation
    • Updated the configuration guide to describe the expanded failover behavior and limits.

… one fault hop

The verify-account 403 hop loop re-entered recovery only for non-replayable
responses, so a replayable 502/503/504 from the replacement account left the
loop and the one-hop 5xx sibling rotation never ran: A:403 verify -> B:503
failed without trying C. The hop now asks isAntigravityFaultRotatable, the
same guard the 429 hop uses since #8839, still once per request.

Follow-up to #8839 (CodeRabbit thread PRRT_kwDOS-0Gi863wcE0 on #8778).
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner November 16, 2026 16:06
@lidge-jun lidge-jun added the ci:full Run the four full Ubuntu test shards on this pull request label Nov 16, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T16:08:11.850052Z 4a10f49 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Nov 16, 2026
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The Antigravity recovery loop now checks eligible upstream-fault responses after a verify-account 403 hop. A regression test covers the sequence, and the English and Spanish configuration documentation describes related failover behavior.

Changes: Antigravity replacement-response fault recovery

Layer / File(s) Summary
Replacement-response fault recovery
src/server/responses/adapter-dispatch.ts, src/server/responses/antigravity-fault-rotation.ts, tests/server/server-google-antigravity-oauth-401-replay.test.ts, structure/transports/responses.md, docs-site/src/content/docs/reference/configuration/providers.md, docs-site/src/content/docs/es/reference/configuration/providers.md
At adapter-dispatch.ts:1767, the recovery loop continues when a replacement response qualifies for fault rotation. The regression test at server-google-antigravity-oauth-401-replay.test.ts:822-847 checks a verify-account 403, the sibling account’s exhausted 5xx retries, and success on a third account. The test also checks quarantine, temporary ineligibility, and usage records. The code comment, ownership description, and English and Spanish documentation describe the updated recovery behavior.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Possibly related PRs

  • lidge-jun/opencodex#8778: Adds the sibling-account failover behavior that this change extends to replacement 5xx responses after a verify-account hop.
  • lidge-jun/opencodex#8835: Changes 403 classification used by the regression test’s verify-account failover case.

Suggested labels: bug

Merge Risk: Minimal · up to 4a10f

This change lets a replayable 502, 503 or 504 from a replacement account, reached after a verify-account 403, rotate to another eligible account once. A regression test and updated documentation accompany it. No concrete merge-blocking risk was identified.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the primary change: a verify-account hop's replayable 5xx replacement may now take the single allowed fault-rotation hop. The raw summary confirms the adapter-dispatch recovery loop no…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the primary change: a verify-account hop's replayable 5xx replacement may now take the single allowed fault-rotation hop. The raw summary confirms the adapter-dispatch recovery loop now continues for such replacement responses, and the regression test covers the A:403 verify-account, B:503, C:200 path. The title is specific and matches the changeset, and the docs and guard docstring updates are consistent with it.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 950b7d9 and 4a10f49.

Files selected for processing: 6

  • docs-site/src/content/docs/es/reference/configuration/providers.md
  • docs-site/src/content/docs/reference/configuration/providers.md
  • src/server/responses/adapter-dispatch.ts
  • src/server/responses/antigravity-fault-rotation.ts
  • structure/transports/responses.md
  • tests/server/server-google-antigravity-oauth-401-replay.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 5 remain after this review.

@lidge-jun

Copy link
Copy Markdown
Owner Author

Maintainer integration of #8846 at 4a10f49. It closes the follow-up gap noted on #8839: the verify-account 403 hop had the same shape as the 429 hop.

  • Fix: the change is one line in adapter-dispatch.ts. After the verify-account hop, recovery also re-enters when antigravityFaultRotatable passes (the fix(antigravity): let a 429 hop's 5xx replacement take the one fault hop #8839 check, still one 5xx rotation per request). The file stays at 1,991 lines.
  • Regression test: A:403 verify-account → B:503 → C. On dev, B was retried three times and C was never tried. Here C is reached, A keeps its verify_account quarantine, and B is cooled but not marked. The recovery kinds record oauth-account-403 and transient-5xx.
  • Docs: the en and es providers.md 5xx paragraph and the antigravity-fault-rotation.ts row in structure/transports/responses.md are updated.
  • Gate: CI is green on this exact head, including enforce-target. Codex completed on this head with no findings, CodeRabbit reviewed this head with no actionable comments, and no thread is open. assert-mergeable-review: OK. The simulated merge onto the dev of this moment is clean.

@lidge-jun
lidge-jun merged commit 4d74ace into dev Nov 16, 2026
39 of 41 checks passed
@lidge-jun
lidge-jun deleted the fix/antigravity-verify-hop-5xx branch November 16, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci:full Run the four full Ubuntu test shards on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant