Repository navigation
feat(gui): context-window breakdown and prompt-cache meter on the live routing tab - #8731
Closed
andrew05060414 wants to merge 12 commits into
Closed
andrew05060414 wants to merge 12 commits into
andrew05060414 wants to merge 12 commits into
Conversation
…he final request row The final request log of a request served by a multi-account generic OAuth provider now carries accountRoster: each account's hashed o-label and whether it was available, paused, needing re-auth, cooling down, out of quota, or not listing the model. It stays in memory for /api/logs and never reaches usage.jsonl. /api/oauth/accounts returns the same logLabel per account so the dashboard can name the account a roster label belongs to.
… long privacy redaction took Adds upstreamSendMs (request start to the final attempt's start, which includes earlier failed attempts) and an optional privacyRedactionMs to the in-memory request row, so the live routing view can split a request's time into measured phases without inventing DNS or TLS numbers.
Models gains a Live routing tab (#models/live) that polls /api/logs while visible. For the latest or a pinned request it shows client -> OpenCodex -> candidates, marking the account or target that answered, the ones that failed with their status and reason, and the ones skipped with the reason the route decision or the account pool recorded. An outcome sentence and a step list explain the route. The request list groups by request, session or key and shows account, model, effort, duration, time to first token, tokens, cache-hit rate, tok/s and estimated cost; unknown values show a dash, never zero. Combos and accounts keep their own pages and are linked.
The live routing view gains a time bar for the pinned or latest request: privacy redaction, OpenCodex internal time, earlier failed attempts, the wait for the upstream's first token, and streaming. Every segment comes from a measured field; a request without timing says so instead of drawing a bar, and no DNS or TLS split is shown because Bun's fetch does not expose one.
…nly in the live routing view A combo row showed the target it landed on as what the client asked for; it now shows the combo. An API-key account reads as its provider plus a short fingerprint instead of a 33-character digest, and the explicit-provider, alias, rule and native route reasons have plain sentences.
…zes only At the Responses, Messages and Chat Completions entry points the client body is walked once and its estimated tokens are split into system prompt, tools, skills, memory files (CLAUDE.md, AGENTS.md), files, tool results and per-turn conversation, following Magpie's classifier (MIT, yetone). Only numbers are kept; no prompt text is retained. The final in-memory request row carries the breakdown and the serving model's context window.
…e routing view For the pinned or latest request the live routing view now draws a treemap of the context window: system prompt, tools, skills, memory files, files, tool results and each conversation turn, against the free space left in the serving model's window. Estimates are scaled to the token count the upstream reported when it reported one. A prompt-cache meter shows the hit rate, and an upstream that did not report cache reads says so instead of showing 0%. Layout follows Magpie's context card (MIT, yetone).
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
7 tasks done
Contributor
|
Contributor
⏳ DRAFT
What to do
Review readiness checklist
✅ 4/4 boxes ticked. This pull request was already a draft. Its draft status will be preserved after every issue above is resolved. Hygiene
|
3 of 7 tasks
Contributor
Author
|
Superseded by #8741, which carries this change together with the related live routing PRs as one commit each on current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The problem. A long agent session fills the model's context window, but nothing in OpenCodex says what it is filled with: the agent's system prompt, tool definitions, skills, memory files like
AGENTS.md/CLAUDE.md, files it read, other tool output, or the conversation itself. That is the first thing you want to know when a session gets slow, expensive or close to its limit.What changes. Stacked on #8730 (this branch contains its commits; review the last four: the context-breakdown backend, the card, the docs bullet and one French label). Merge #8730 first. The Live routing tab gains a context-window card for the selected request, modeled on Magpie's context card (MIT, © yetone; ported code carries the notice):
previous_response_idsays the earlier turns live upstream.Backend
src/server/context-breakdown*.tswalks the client body once at the Responses, Messages and Chat Completions entry points and keeps only numbers: per-category estimated tokens, per-turn totals (at most 48, older turns summed), tool count, and aheldflag. The classifier rules (skills markers,Contents of …CLAUDE.md, Codex# AGENTS.md instructions for …,<system-reminder>handling, file-read tool calls including shellcat/sed -n, images and documents) are ported from Magpie's Go classifier. No string from the body is retained; tests assert that no fixture text appears in the result.JSON.stringifyof large values, no hashing, no timers. A synthetic ~6 MB body (3000 messages) measured in about 11 ms on a busy machine; ordinary turns are far smaller. First observer wins, so a Messages/Chat request replayed through the Responses path is measured once, as the client sent it.contextBreakdownandcontextWindow(the adapter's window, falling back to the bundled vendor metadata table). Neither reachesusage.jsonl.request-log.tsgrows by 1 line; each entry point by 2.Screenshot
Local test instance, fake upstream, a synthetic Codex-style request (developer prompt, AGENTS.md, skills, a
catfile read, a test-run tool result):Verification
bun run typecheck,bun run structure:check,bun run privacy:scan: pass.tests/usage/context-breakdown.test.ts(Responses, Messages and Chat classifier cases, held flag, turn counting, malformed bodies, no retained text),tests/usage/request-log-context-breakdown.test.ts,tests/lab/core-lab-boundary.test.ts,tests/ci-workflows/file-size-ratchet.test.ts,tests/test-layout.test.ts: pass.cd gui && bun test tests/route-live* tests/fr-localization.test.ts tests/locale-parity.test.ts tests/models-workspace-panels.test.tsx tests/compatibility-lab.test.tsx tests/models-keep-native-v1-placement.test.ts(newroute-live-context: calibration, treemap covers the box without overlap, unknown window, cache unknown vs 0, parser drops malformed data),bun run lint,bun run lint:i18n,tsc --noEmit,bun run build: pass.bun run testsuite. Several sessions were running suites on the same Windows machine, so full-suite timeouts there were load, not signal; and the test preload's git-config guard rejectsGIT_CONFIG_GLOBAL=NULwith this machine's git 2.56, failing the first case of every file and a handful oftest-layout-toolingcases on unmodifieddevas well. Focused files ran withOCX_GIT_CONFIG_GUARD_CHECKOUTpointing at a directory without.git. A local GUI suite run showed no failures in files this PR touches beyond one French-copy case, which is fixed here. Full coverage is left to CI.Readiness (2026-11-13)
dev(873ec9b), inside the gate's window; no rebase for review.bun run typecheck,structure:check,skill:surface:check,privacy:scan, the layout and file-size guards and this PR's focused tests; after the deploy one real request each through Claude, GPT, Gemini, DeepSeek and Grok returned 200 and the error rate did not rise.maintainer-sponsoredlabel: the hygiene gate flagssrc/oauth/generic-account-failover.tsandsrc/server/management/oauth-account-routes.ts(through feat(gui): live routing tab: see which account answered, why others were skipped, and where the time went #8730, which this PR is stacked on). Everything else is ready for review.Checklist
guides/model-routing.md, context-window bullet)Review readiness checklist
This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:
Required local validation passed; commands, results, and any full-suite exception are documented.
I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).
I resolved all correct Codex and CodeRabbit findings.
My PR is ready for review.