Skip to content

feat(gui): context-window breakdown and prompt-cache meter on the live routing tab - #8731

Closed
andrew05060414 wants to merge 12 commits into
lidge-jun:devfrom
andrew05060414:feat/route-live-context
Closed

andrew05060414 wants to merge 12 commits into
lidge-jun:devfrom
andrew05060414:feat/route-live-context

Conversation

@andrew05060414

@andrew05060414 andrew05060414 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The problem. A long agent session fills the model's context window, but nothing in OpenCodex says what it is filled with: the agent's system prompt, tool definitions, skills, memory files like AGENTS.md/CLAUDE.md, files it read, other tool output, or the conversation itself. That is the first thing you want to know when a session gets slow, expensive or close to its limit.

What changes. Stacked on #8730 (this branch contains its commits; review the last four: the context-breakdown backend, the card, the docs bullet and one French label). Merge #8730 first. The Live routing tab gains a context-window card for the selected request, modeled on Magpie's context card (MIT, © yetone; ported code carries the notice):

  • A treemap of estimated tokens by category (system prompt, tools, skills, memory and AGENTS.md, files, tool results, each conversation turn) against the free space left in the serving model's window, with a fill percentage and a plain health word.
  • Estimates are scaled to the prompt size the upstream reported, when it reported one, and are labeled "estimate" otherwise. A Responses request with previous_response_id says the earlier turns live upstream.
  • A prompt-cache meter ("62% hit · 6.5K cached · 4K new"). An upstream that reported no cache numbers says so instead of showing 0%.

Backend

  • src/server/context-breakdown*.ts walks the client body once at the Responses, Messages and Chat Completions entry points and keeps only numbers: per-category estimated tokens, per-turn totals (at most 48, older turns summed), tool count, and a held flag. The classifier rules (skills markers, Contents of …CLAUDE.md, Codex # AGENTS.md instructions for …, <system-reminder> handling, file-read tool calls including shell cat/sed -n, images and documents) are ported from Magpie's Go classifier. No string from the body is retained; tests assert that no fixture text appears in the result.
  • Cost: linear in body size, no JSON.stringify of large values, no hashing, no timers. A synthetic ~6 MB body (3000 messages) measured in about 11 ms on a busy machine; ordinary turns are far smaller. First observer wins, so a Messages/Chat request replayed through the Responses path is measured once, as the client sent it.
  • The final in-memory request row carries contextBreakdown and contextWindow (the adapter's window, falling back to the bundled vendor metadata table). Neither reaches usage.jsonl.
  • request-log.ts grows by 1 line; each entry point by 2.

Screenshot

Local test instance, fake upstream, a synthetic Codex-style request (developer prompt, AGENTS.md, skills, a cat file read, a test-run tool result):

Live routing with context card

Verification

  • bun run typecheck, bun run structure:check, bun run privacy:scan: pass.
  • Focused tests: tests/usage/context-breakdown.test.ts (Responses, Messages and Chat classifier cases, held flag, turn counting, malformed bodies, no retained text), tests/usage/request-log-context-breakdown.test.ts, tests/lab/core-lab-boundary.test.ts, tests/ci-workflows/file-size-ratchet.test.ts, tests/test-layout.test.ts: pass.
  • GUI: cd gui && bun test tests/route-live* tests/fr-localization.test.ts tests/locale-parity.test.ts tests/models-workspace-panels.test.tsx tests/compatibility-lab.test.tsx tests/models-keep-native-v1-placement.test.ts (new route-live-context: calibration, treemap covers the box without overlap, unknown window, cache unknown vs 0, parser drops malformed data), bun run lint, bun run lint:i18n, tsc --noEmit, bun run build: pass.
  • Manual: the screenshot above, from the proxy running this branch.
  • Not run locally: the full bun run test suite. Several sessions were running suites on the same Windows machine, so full-suite timeouts there were load, not signal; and the test preload's git-config guard rejects GIT_CONFIG_GLOBAL=NUL with this machine's git 2.56, failing the first case of every file and a handful of test-layout-tooling cases on unmodified dev as well. Focused files ran with OCX_GIT_CONFIG_GUARD_CHECKOUT pointing at a directory without .git. A local GUI suite run showed no failures in files this PR touches beyond one French-copy case, which is fixed here. Full coverage is left to CI.

Readiness (2026-11-13)

  • Branch is 2 commit(s) behind dev (873ec9b), inside the gate's window; no rebase for review.
  • No Codex or CodeRabbit findings are open on this PR (CodeRabbit skipped the draft; there are no inline review comments).
  • Also merged into a local integration build with the other open PRs of this series and deployed on a real install. The combined build passed bun run typecheck, structure:check, skill:surface:check, privacy:scan, the layout and file-size guards and this PR's focused tests; after the deploy one real request each through Claude, GPT, Gemini, DeepSeek and Grok returned 200 and the error rate did not rise.
  • Still needs the maintainer-sponsored label: the hygiene gate flags src/oauth/generic-account-failover.ts and src/server/management/oauth-account-routes.ts (through feat(gui): live routing tab: see which account answered, why others were skipped, and where the time went #8730, which this PR is stacked on). Everything else is ready for review.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed. (guides/model-routing.md, context-window bullet)
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. (Sizes only; no prompt text is stored.)

Review readiness checklist

This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:

  • Required local validation passed; commands, results, and any full-suite exception are documented.

  • I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).

  • I resolved all correct Codex and CodeRabbit findings.

  • My PR is ready for review.

…he final request row

The final request log of a request served by a multi-account generic OAuth
provider now carries accountRoster: each account's hashed o-label and whether
it was available, paused, needing re-auth, cooling down, out of quota, or not
listing the model. It stays in memory for /api/logs and never reaches
usage.jsonl. /api/oauth/accounts returns the same logLabel per account so the
dashboard can name the account a roster label belongs to.
… long privacy redaction took

Adds upstreamSendMs (request start to the final attempt's start, which
includes earlier failed attempts) and an optional privacyRedactionMs to the
in-memory request row, so the live routing view can split a request's time
into measured phases without inventing DNS or TLS numbers.
Models gains a Live routing tab (#models/live) that polls /api/logs while
visible. For the latest or a pinned request it shows client -> OpenCodex ->
candidates, marking the account or target that answered, the ones that
failed with their status and reason, and the ones skipped with the reason the
route decision or the account pool recorded. An outcome sentence and a step
list explain the route. The request list groups by request, session or key
and shows account, model, effort, duration, time to first token, tokens,
cache-hit rate, tok/s and estimated cost; unknown values show a dash, never
zero. Combos and accounts keep their own pages and are linked.
The live routing view gains a time bar for the pinned or latest request:
privacy redaction, OpenCodex internal time, earlier failed attempts, the wait
for the upstream's first token, and streaming. Every segment comes from a
measured field; a request without timing says so instead of drawing a bar,
and no DNS or TLS split is shown because Bun's fetch does not expose one.
…nly in the live routing view

A combo row showed the target it landed on as what the client asked for; it
now shows the combo. An API-key account reads as its provider plus a short
fingerprint instead of a 33-character digest, and the explicit-provider,
alias, rule and native route reasons have plain sentences.
…zes only

At the Responses, Messages and Chat Completions entry points the client body
is walked once and its estimated tokens are split into system prompt, tools,
skills, memory files (CLAUDE.md, AGENTS.md), files, tool results and
per-turn conversation, following Magpie's classifier (MIT, yetone). Only
numbers are kept; no prompt text is retained. The final in-memory request row
carries the breakdown and the serving model's context window.
…e routing view

For the pinned or latest request the live routing view now draws a treemap of
the context window: system prompt, tools, skills, memory files, files, tool
results and each conversation turn, against the free space left in the
serving model's window. Estimates are scaled to the token count the upstream
reported when it reported one. A prompt-cache meter shows the hit rate, and
an upstream that did not report cache reads says so instead of showing 0%.
Layout follows Magpie's context card (MIT, yetone).
@coderabbitai

coderabbitai Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Deterministic hygiene checks failed.

  • unsponsored_surface — This changes an authentication, workflow, release-automation, or dependency surface. MAINTAINERS.md requires security review for these; ask a maintainer to apply maintainer-sponsored once they have reviewed it. Paths: src/oauth/generic-account-failover.ts, src/server/management/oauth-account-routes.ts.

@github-actions github-actions Bot added the enhancement New feature or request label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

⏳ DRAFT

  • hygiene: unsponsored_surface.

What to do

  • Fix unsponsored_surface — This changes an authentication, workflow, release-automation, or dependency surface. MAINTAINERS.md requires security review for these; ask a maintainer to apply maintainer-sponsored once they have reviewed it. Paths: src/oauth/generic-account-failover.ts, src/server/management/oauth-account-routes.ts.

Review readiness checklist

  • ✅ Required local validation passed; commands, results, and any full-suite exception are documented.
  • ✅ I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).
  • ✅ I resolved all correct Codex and CodeRabbit findings.
  • ✅ My PR is ready for review.

✅ 4/4 boxes ticked.

This pull request was already a draft. Its draft status will be preserved after every issue above is resolved.

Hygiene

⚠️ Deterministic hygiene checks failed.

  • unsponsored_surface — This changes an authentication, workflow, release-automation, or dependency surface. MAINTAINERS.md requires security review for these; ask a maintainer to apply maintainer-sponsored once they have reviewed it. Paths: src/oauth/generic-account-failover.ts, src/server/management/oauth-account-routes.ts.

@andrew05060414

Copy link
Copy Markdown
Contributor Author

Superseded by #8741, which carries this change together with the related live routing PRs as one commit each on current dev, with no stacking. The design notes in this description still apply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request intake: hygiene-blocked Deterministic PR hygiene checks failed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant