Repository navigation
feat(models): infer unknown model capabilities by name, label their source, and probe what a relay accepts - #8674
Closed
andrew05060414 wants to merge 3 commits into
Closed
andrew05060414 wants to merge 3 commits into
andrew05060414 wants to merge 3 commits into
Conversation
…xis's source, and probe what a relay accepts GET /api/models routed rows gain capabilitySources (manual > probed > reported > inferred > unknown) and display-only inferredCapabilities from a same-name lookup over the vendor registry, bundled metadata and other providers' declarations. POST /api/providers/probe-capabilities sends a baseline, a 1x1 image and one request per effort through the completion diagnostic, asks before spending on a metered provider, and with apply writes the conclusive axes plus a sparse modelCapabilityProbes label.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Contributor
|
Contributor
⏳ DRAFT
What to do
Review readiness checklist
✅ 4/4 boxes ticked. This pull request was already a draft. Its draft status will be preserved after every issue above is resolved. Hygiene
|
andrew05060414
added a commit
to andrew05060414/opencodex
that referenced
this pull request
Nov 13, 2026
…alues with Adopt, live capability probe - Source tags read capabilitySources (manual / probed / reported / inferred / unknown) from /api/models, falling back to the manual flags on an older server. - A gap with an inferredCapabilities estimate shows the value and its basis, with Adopt writing it through PUT /api/model-settings. Estimates fill gaps only in the combined row, labelled. - Test it calls POST /api/providers/probe-capabilities with apply, confirms metered probes inline with the planned request count, and shows inapplicable reasons, 409, and an inconclusive baseline. Depends on lidge-jun#8674.
Closed
7 tasks done
andrew05060414
added a commit
to andrew05060414/opencodex
that referenced
this pull request
Nov 13, 2026
…alues with Adopt, live capability probe - Source tags read capabilitySources (manual / probed / reported / inferred / unknown) from /api/models, falling back to the manual flags on an older server. - A gap with an inferredCapabilities estimate shows the value and its basis, with Adopt writing it through PUT /api/model-settings. Estimates fill gaps only in the combined row, labelled. - Test it calls POST /api/providers/probe-capabilities with apply, confirms metered probes inline with the planned request count, and shows inapplicable reasons, 409, and an inconclusive baseline. Depends on lidge-jun#8674.
Closed
7 tasks done
…e route registry The capability probe handler was routed in provider-routes but never declared, so the registry reconciliation test failed.
Contributor
Author
|
Superseded by #8738, which carries this change together with the related model metadata and usage PRs as one commit each on current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why. A model served through an OpenAI-compatible relay usually arrives with an id-only
/modelsrow. Its context window, input modalities and reasoning ladder then read as unknown everywhere in the dashboard, even when the same model id is well known (claude-opus-5-5behind a relay is the Anthropic model). Operators have no way to see a sensible value, where it would come from, or to check what the relay actually accepts without hand-editing config.What.
src/providers/model-capability-inference.tslooks a model id up, per axis (context window, input modalities, reasoning efforts), in this order of authority: built-in original-vendor registry entries (multi-vendor hosts such as Vertex and Antigravity excluded), bundled vendor metadata, other configured providers' explicit declarations, bundled aggregator metadata. Ids match ignoring case, avendor/prefix, dot-versus-hyphen spelling and a[...]suffix. Two sources of the same authority that disagree give no answer, and the lookup does not fall through to a weaker tier, the rulevendor-context.tsalready applies to windows.GET /api/models. Routed rows gaincapabilitySources: { contextWindow, inputModalities, reasoningEfforts }, eachmanual>probed>reported>inferred>unknown, andinferredCapabilitieswith{ value, basis: { source, provider, modelId } }per axis (src/server/management/model-capability-sources.ts). The estimates are display data: they never route and never enter the Codex catalog on their own. The existing opt-ininferContextWindowstill controls whether an estimated window is applied (its rows are labeledinferred), and anything else is applied by saving it as a declaration. This keeps the Models GUI cannot define a fallback context window for metadata-poor third-party providers #1073 decision that an id-only relay row is not silently given a vendor's capacity: a relay can serve less than the vendor, and sending an image to a relay that drops it is worse than showing "unknown, estimate: image".modelCapabilities.contextWindow. A numeric per-model window override already exists (modelContextWindows, written byPUT /api/model-settings, validated, sparse, read by the catalog and by input admission). A second axis would be a second authority for one value, so this PR does not add one.POST /api/providers/probe-capabilities(src/server/management/provider-capability-probe.ts) reuses the explicit completion diagnostic (provider-completion-test.ts,openai-chat/openai-responseswires, 32-token cap, 20 s per request, no retries), which gains an optional bodyvariant(imageorreasoningEffort). It sends a plain baseline first; only if that completes are the variants sent, so an unrelated 400 is never read as "rejects images". Then one 1x1 PNG request, and one request per effort. API-key providers on non-local destinations are metered: withoutconfirmMetered: truenothing is sent and the response carriesrequiresConfirmationandplannedRequests. Withapply: true, conclusive results are written tomodelCapabilities[model].inputModalitiesandmodelReasoningEfforts[model]and labeled in a new sparse provider fieldmodelCapabilityProbes: Record<model, { probedAt, inputModalities?: true, reasoningEfforts?: true }>; a later manual edit of the axis through PATCH/api/providersorPUT /api/model-settingsdrops the label. Only status codes reach the response; upstream bodies are never returned or logged. The context window is not probed: a tiny request never reaches it.Known limits, also in the docs: "accepted" means "not refused" (a gateway that ignores
reasoning_effortreads as accepting every rung); the raw config editor (PUT /api/providers) and combo routes do not clear a probe label; a same-name provider POST does not carrymodelReasoningEffortstoday, so its probe label is pruned with it; a worst-case run is 8 sequential requests.Verification
bun run typecheck: exit 0.bun run structure:check,bun run privacy:scan: pass.tests/providers/model-capability-inference.test.ts(normalization, registry / provider-declaration tiers, conflict answers nothing, unknown model, source labels and precedence),tests/providers/provider-capability-probe.test.ts(metered gate sends nothing, not-applicable providers, body validation, chat and responses variant bodies, accepted / rejected 422 / timeout, baseline 400 or timeout blocks every variant, reasoning mix, 401 early stop, default ladder, apply writes and labels, inconclusive keeps the ladder, nothing conclusive saves nothing),tests/server/provider-capability-probe-provenance.test.ts(field validation, sparse GET/PATCH/POST round trips, label dropped on manual edit). All pass.cli-capability-provider-workflows,model-settings-management-api,management-provider-validation,provider-config-validation,provider-input-modalities,server/config,management-provider-hide-raw-reasoning,test-layout,file-size-ratchet,structure-ssot: pass. All runs used--timeout 60000on a shared Windows machine.GET /api/models, run together: 920 pass, 9 fail. Eight are 40 s timeouts in Link/Home sync tests (connect inherits non-native intent,cached Home intent cannot block local off, and siblings) on this loaded machine. The ninth,openai-provider-option-e2e"keeps Pool, Direct, and API ownership stable", fails only in that batch and passes when its file runs alone, so it is cross-file state in the batch, not this change. CI decides.bun run test:changedand the full suite were not run locally: the change reachesprovider-routes.tsandmodel-rows.ts, whose import graph covers most of the suite, and this shared machine times tests out under load. Coverage beyond the files above is left to CI.Refreshed on
dev86be880 (2026-11-13)Merged the current
devhead86be8805cinto the branch (merge commit, no history rewrite).bun run typecheck,bun run structure:check,bun run skill:surface:checkandbun run privacy:scanall exit 0. Focused run: the layout, file-size ratchet, repo-hygiene, management route registry, skill-ocx, structure-ssot and import-cycle guards plus every test file this PR adds or changes.POST /api/providers/probe-capabilitiesinsrc/server/management/route-registry.ts; the route-registry reconciliation now passes.devcheckout on this Windows host (test-layout-toolingrewriteSpecifier / move end to end ×5 and twostructure-ssotgit-index cases): their temp repositories run git withGIT_CONFIG_GLOBAL=NUL, which git 2.56 for Windows rejects withfatal: unable to access 'NUL'. None of them touches a file in this PR.bun run testandbun run test:changedwere not rerun for this refresh. This host runs several worktrees at once and an earliertest:changedhit 900 s with timeouts in unrelated modules, so full coverage is left to CI.maintainer-sponsoredlabel: the hygiene gate flagssrc/server/auth-cors.ts(one field-policy line). Everything else is ready for review.Checklist
Review readiness checklist
This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:
Required local validation passed; commands, results, and any full-suite exception are documented.
I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).
I resolved all correct Codex and CodeRabbit findings.
My PR is ready for review.