Skip to content

feat(models): infer unknown model capabilities by name, label their source, and probe what a relay accepts - #8674

Closed
andrew05060414 wants to merge 3 commits into
lidge-jun:devfrom
andrew05060414:feat/model-capability-inference
Closed

andrew05060414 wants to merge 3 commits into
lidge-jun:devfrom
andrew05060414:feat/model-capability-inference

Conversation

@andrew05060414

@andrew05060414 andrew05060414 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Why. A model served through an OpenAI-compatible relay usually arrives with an id-only /models row. Its context window, input modalities and reasoning ladder then read as unknown everywhere in the dashboard, even when the same model id is well known (claude-opus-5-5 behind a relay is the Anthropic model). Operators have no way to see a sensible value, where it would come from, or to check what the relay actually accepts without hand-editing config.

What.

  1. Same-name inference, labeled. New src/providers/model-capability-inference.ts looks a model id up, per axis (context window, input modalities, reasoning efforts), in this order of authority: built-in original-vendor registry entries (multi-vendor hosts such as Vertex and Antigravity excluded), bundled vendor metadata, other configured providers' explicit declarations, bundled aggregator metadata. Ids match ignoring case, a vendor/ prefix, dot-versus-hyphen spelling and a [...] suffix. Two sources of the same authority that disagree give no answer, and the lookup does not fall through to a weaker tier, the rule vendor-context.ts already applies to windows.
  2. Per-axis provenance on GET /api/models. Routed rows gain capabilitySources: { contextWindow, inputModalities, reasoningEfforts }, each manual > probed > reported > inferred > unknown, and inferredCapabilities with { value, basis: { source, provider, modelId } } per axis (src/server/management/model-capability-sources.ts). The estimates are display data: they never route and never enter the Codex catalog on their own. The existing opt-in inferContextWindow still controls whether an estimated window is applied (its rows are labeled inferred), and anything else is applied by saving it as a declaration. This keeps the Models GUI cannot define a fallback context window for metadata-poor third-party providers #1073 decision that an id-only relay row is not silently given a vendor's capacity: a relay can serve less than the vendor, and sending an image to a relay that drops it is worse than showing "unknown, estimate: image".
  3. No modelCapabilities.contextWindow. A numeric per-model window override already exists (modelContextWindows, written by PUT /api/model-settings, validated, sparse, read by the catalog and by input admission). A second axis would be a second authority for one value, so this PR does not add one.
  4. Capability probe. POST /api/providers/probe-capabilities (src/server/management/provider-capability-probe.ts) reuses the explicit completion diagnostic (provider-completion-test.ts, openai-chat / openai-responses wires, 32-token cap, 20 s per request, no retries), which gains an optional body variant (image or reasoningEffort). It sends a plain baseline first; only if that completes are the variants sent, so an unrelated 400 is never read as "rejects images". Then one 1x1 PNG request, and one request per effort. API-key providers on non-local destinations are metered: without confirmMetered: true nothing is sent and the response carries requiresConfirmation and plannedRequests. With apply: true, conclusive results are written to modelCapabilities[model].inputModalities and modelReasoningEfforts[model] and labeled in a new sparse provider field modelCapabilityProbes: Record<model, { probedAt, inputModalities?: true, reasoningEfforts?: true }>; a later manual edit of the axis through PATCH /api/providers or PUT /api/model-settings drops the label. Only status codes reach the response; upstream bodies are never returned or logged. The context window is not probed: a tiny request never reaches it.

Known limits, also in the docs: "accepted" means "not refused" (a gateway that ignores reasoning_effort reads as accepting every rung); the raw config editor (PUT /api/providers) and combo routes do not clear a probe label; a same-name provider POST does not carry modelReasoningEfforts today, so its probe label is pruned with it; a worst-case run is 8 sequential requests.

Verification

  • bun run typecheck: exit 0. bun run structure:check, bun run privacy:scan: pass.
  • New tests: tests/providers/model-capability-inference.test.ts (normalization, registry / provider-declaration tiers, conflict answers nothing, unknown model, source labels and precedence), tests/providers/provider-capability-probe.test.ts (metered gate sends nothing, not-applicable providers, body validation, chat and responses variant bodies, accepted / rejected 422 / timeout, baseline 400 or timeout blocks every variant, reasoning mix, 401 early stop, default ladder, apply writes and labels, inconclusive keeps the ladder, nothing conclusive saves nothing), tests/server/provider-capability-probe-provenance.test.ts (field validation, sparse GET/PATCH/POST round trips, label dropped on manual edit). All pass.
  • Existing: provider completion-test suites, cli-capability-provider-workflows, model-settings-management-api, management-provider-validation, provider-config-validation, provider-input-modalities, server/config, management-provider-hide-raw-reasoning, test-layout, file-size-ratchet, structure-ssot: pass. All runs used --timeout 60000 on a shared Windows machine.
  • The 30 test files that exercise GET /api/models, run together: 920 pass, 9 fail. Eight are 40 s timeouts in Link/Home sync tests (connect inherits non-native intent, cached Home intent cannot block local off, and siblings) on this loaded machine. The ninth, openai-provider-option-e2e "keeps Pool, Direct, and API ownership stable", fails only in that batch and passes when its file runs alone, so it is cross-file state in the batch, not this change. CI decides.
  • bun run test:changed and the full suite were not run locally: the change reaches provider-routes.ts and model-rows.ts, whose import graph covers most of the suite, and this shared machine times tests out under load. Coverage beyond the files above is left to CI.

Refreshed on dev 86be880 (2026-11-13)

Merged the current dev head 86be8805c into the branch (merge commit, no history rewrite).

  • bun run typecheck, bun run structure:check, bun run skill:surface:check and bun run privacy:scan all exit 0. Focused run: the layout, file-size ratchet, repo-hygiene, management route registry, skill-ocx, structure-ssot and import-cycle guards plus every test file this PR adds or changes.
  • 227 tests: 219 pass, 1 skip, 7 fail. This refresh also declares POST /api/providers/probe-capabilities in src/server/management/route-registry.ts; the route-registry reconciliation now passes.
  • The 7 failures are the same 7 cases that fail on an untouched dev checkout on this Windows host (test-layout-tooling rewriteSpecifier / move end to end ×5 and two structure-ssot git-index cases): their temp repositories run git with GIT_CONFIG_GLOBAL=NUL, which git 2.56 for Windows rejects with fatal: unable to access 'NUL'. None of them touches a file in this PR.
  • Full-suite exception: the full bun run test and bun run test:changed were not rerun for this refresh. This host runs several worktrees at once and an earlier test:changed hit 900 s with timeouts in unrelated modules, so full coverage is left to CI.
  • Still needs the maintainer-sponsored label: the hygiene gate flags src/server/auth-cors.ts (one field-policy line). Everything else is ready for review.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Review readiness checklist

This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:

  • Required local validation passed; commands, results, and any full-suite exception are documented.

  • I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).

  • I resolved all correct Codex and CodeRabbit findings.

  • My PR is ready for review.

…xis's source, and probe what a relay accepts

GET /api/models routed rows gain capabilitySources (manual > probed > reported >
inferred > unknown) and display-only inferredCapabilities from a same-name lookup
over the vendor registry, bundled metadata and other providers' declarations.
POST /api/providers/probe-capabilities sends a baseline, a 1x1 image and one
request per effort through the completion diagnostic, asks before spending on a
metered provider, and with apply writes the conclusive axes plus a sparse
modelCapabilityProbes label.
@coderabbitai

coderabbitai Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions github-actions Bot added the intake: hygiene-blocked Deterministic PR hygiene checks failed label Nov 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Deterministic hygiene checks failed.

  • unsponsored_surface — This changes an authentication, workflow, release-automation, or dependency surface. MAINTAINERS.md requires security review for these; ask a maintainer to apply maintainer-sponsored once they have reviewed it. Paths: src/server/auth-cors.ts.

@github-actions github-actions Bot added the enhancement New feature or request label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

⏳ DRAFT

  • hygiene: unsponsored_surface.

What to do

  • Fix unsponsored_surface — This changes an authentication, workflow, release-automation, or dependency surface. MAINTAINERS.md requires security review for these; ask a maintainer to apply maintainer-sponsored once they have reviewed it. Paths: src/server/auth-cors.ts.

Review readiness checklist

  • ✅ Required local validation passed; commands, results, and any full-suite exception are documented.
  • ✅ I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).
  • ✅ I resolved all correct Codex and CodeRabbit findings.
  • ✅ My PR is ready for review.

✅ 4/4 boxes ticked.

This pull request was already a draft. Its draft status will be preserved after every issue above is resolved.

Hygiene

⚠️ Deterministic hygiene checks failed.

  • unsponsored_surface — This changes an authentication, workflow, release-automation, or dependency surface. MAINTAINERS.md requires security review for these; ask a maintainer to apply maintainer-sponsored once they have reviewed it. Paths: src/server/auth-cors.ts.

andrew05060414 added a commit to andrew05060414/opencodex that referenced this pull request Nov 13, 2026
…alues with Adopt, live capability probe

- Source tags read capabilitySources (manual / probed / reported / inferred / unknown) from
  /api/models, falling back to the manual flags on an older server.
- A gap with an inferredCapabilities estimate shows the value and its basis, with Adopt writing it
  through PUT /api/model-settings. Estimates fill gaps only in the combined row, labelled.
- Test it calls POST /api/providers/probe-capabilities with apply, confirms metered probes inline
  with the planned request count, and shows inapplicable reasons, 409, and an inconclusive baseline.

Depends on lidge-jun#8674.
andrew05060414 added a commit to andrew05060414/opencodex that referenced this pull request Nov 13, 2026
…alues with Adopt, live capability probe

- Source tags read capabilitySources (manual / probed / reported / inferred / unknown) from
  /api/models, falling back to the manual flags on an older server.
- A gap with an inferredCapabilities estimate shows the value and its basis, with Adopt writing it
  through PUT /api/model-settings. Estimates fill gaps only in the combined row, labelled.
- Test it calls POST /api/providers/probe-capabilities with apply, confirms metered probes inline
  with the planned request count, and shows inapplicable reasons, 409, and an inconclusive baseline.

Depends on lidge-jun#8674.
…e route registry

The capability probe handler was routed in provider-routes but never declared, so the registry reconciliation test failed.
@andrew05060414

Copy link
Copy Markdown
Contributor Author

Superseded by #8738, which carries this change together with the related model metadata and usage PRs as one commit each on current dev, with no stacking. The design notes in this description still apply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request intake: hygiene-blocked Deterministic PR hygiene checks failed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant