Repository navigation
feat(pricing): fetch model prices from models.dev and OpenRouter, label every price's source - #8708
Closed
andrew05060414 wants to merge 6 commits into
Closed
andrew05060414 wants to merge 6 commits into
andrew05060414 wants to merge 6 commits into
Conversation
…xis's source, and probe what a relay accepts GET /api/models routed rows gain capabilitySources (manual > probed > reported > inferred > unknown) and display-only inferredCapabilities from a same-name lookup over the vendor registry, bundled metadata and other providers' declarations. POST /api/providers/probe-capabilities sends a baseline, a 1x1 image and one request per effort through the completion diagnostic, asks before spending on a metered provider, and with apply writes the conclusive axes plus a sparse modelCapabilityProbes label.
…el every price's source Automatic prices fill the two lowest tiers of price resolution (manual > official > bundled > models.dev > OpenRouter). Catalogs are cached in price-catalog-cache.json, refreshed daily and on a pricing miss (merging only the missing keys, 6h per-key cooldown, never on the request path). Catalog prices are verified-derived estimates carrying their feed URL and fetch time. Adds an opt-in providers.<name>.priceMultiplier for relays (never applied to manual prices), apiEquivalent labelling for subscription providers, a price field on /api/models rows, POST /api/pricing/refresh and PUT /api/providers/:name/price-multiplier. The dashboard shows a source chip per model and adds adopt-as-manual, refresh and multiplier controls. Also registers POST /api/providers/probe-capabilities, which the base commit left out of the route registry.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Contributor
|
Contributor
⏳ DRAFT
What to do
Review readiness checklist
✅ 4/4 boxes ticked. This pull request was already a draft. Its draft status will be preserved after every issue above is resolved. Hygiene
|
…e route registry The capability probe handler was routed in provider-routes but never declared, so the registry reconciliation test failed.
Contributor
Author
|
Superseded by #8738, which carries this change together with the related model metadata and usage PRs as one commit each on current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Many models have no price, so the dashboard shows an unknown cost for a large share of traffic. In one local install, the last 30 days included about 15k unpriced
deepseek-v4.1-flashattempts across four relays. There were alsogemini-4-argonon Vertex/Cursor/Zed,cn:-prefixed Kimi/GLM/MiniMax/Step ids, andzd/-prefixed relay ids. This PR fetches public prices automatically, labels where every price came from, and keeps a manually set price authoritative.Depends on #8674 (stacked on
feat/model-capability-inference; it reusescapabilityInferenceKey). Review only the top commit until #8674 lands.modelCosts> official (verified overrides, bundled table, expected overlay, model-level vendor fallback) > models.dev > OpenRouter > unknown.verified-derived, so every estimate using one is marked estimated. Each one carries the feed URL and fetch time.vendor/prefix, dots vs hyphens,[...]suffix), plus a leadingchannel:prefix (cn:,global:). If two rows in one feed claim the same key with different prices, the key is dropped.<configDir>/price-catalog-cache.json.GET /api/modelsandGET /api/usage).redirect: "error", a 15s timeout and a 32 MiB body cap.providers.<name>.priceMultiplier(opt-in, > 0 and <= 1000) scales automatic prices for relays that charge a fixed multiple of list price. It never scales manual prices.apiEquivalent, so their cost reads as an API-equivalent estimate./api/modelsrows gainprice(manual | official | models-dev | openrouterplus effective rates,fetchedAt,multiplier,apiEquivalent).POST /api/pricing/refresh(forced, clears cooldowns) andPUT /api/providers/:name/price-multiplier.GET .../model-costsreturnspriceMultiplierwhen set.online_price_catalog,relay_price_multiplier,subscription_api_equivalent.POST /api/providers/probe-capabilitiesinroute-registry.ts. feat(models): infer unknown model capabilities by name, label their source, and probe what a relay accepts #8674 added the route without an entry, which failed the registry reconciliation test.Models with no public price anywhere (
composer-2.5,grok-composer-2.5-fast,codex-auto-review) stay unknown.Verification
bun run typecheck: pass.bun run privacy:scan: pass.bun run structure:check: pass.tests/usage/price-catalog.test.tscovers parsing, key matching, ambiguity, TTL, coalescing, offline fallback, corrupt cache, backoff and forced refresh.tests/usage/price-catalog-miss.test.tscovers miss trigger, dedupe, cooldown, merge-only-missing, and that resolution returns synchronously while a fetch is still pending.tests/usage/price-catalog-resolution.test.tscovers priority, user zero still wins, multiplier, subscription and cost reasons.tests/server/price-catalog-management-api.test.tscovers the multiplier GET/PUT sparse round trip with rollback, the refresh route, load-degrade and the/api/modelspricefield.test-layout,test-layout-tooling,structure-ssot, file-size ratchet,cli-capability-dataand the model-costs suites pass.tests/server/model-costs-pricing-cache.test.tsstage 2 is flaky on Windows on the base commit as well: one clean-HEAD run failed identically and the next passed.bun x tsc -b,bun run lint:gui,bun run lint:i18nandbun run build:guipass.gui/tests/models-price-editor.test.tsx(29) and the locale parity tests pass.bun test tests/usage/price-catalog*.test.ts tests/server/price-catalog-management-api.test.tsgives 69 pass and 0 fail.bun run test:changeddid not complete. The lane hit its timeout (exit 124) on a machine running several concurrent test suites, so I am not claiming a pass and am leaving the full run to CI.route-intent-asidefiles pass, so those were timeouts under load.tests/routing/routing-compatibility-boundaries.test.ts(2 fail) andtests/clients/route-intent-connected.test.ts(childbun.exeETIMEDOUT) fail the same way on a clean checkout of the base commit787f8c111.cn:deepseek-v4.1-flashcame from OpenRouter (0.3/1.2).cn:kimi-k3.1-pro,cn:glm-5.3-flash,zd/gpt-6-solandzd/gemini-4-argoncame from models.dev.priceMultiplier: 0.5showed official Claude rates halved, while its manual row stayed unscaled.Model list with source chips:
Price dialog with the automatic price, adopt, refresh and multiplier controls:
Refreshed on
dev86be880 (2026-11-13)Merged the current
devhead86be8805cinto the branch (merge commit, no history rewrite). Stacked on #8674: this branch merges the refreshedfeat/model-capability-inference.bun run typecheck,bun run structure:check,bun run skill:surface:checkandbun run privacy:scanall exit 0. Focused run: the layout, file-size ratchet, repo-hygiene, management route registry, skill-ocx, structure-ssot and import-cycle guards plus every test file this PR adds or changes.npx tsc -bandbun run lintexit 0;models-price-editor: 29 pass, 0 fail.devcheckout on this Windows host (test-layout-toolingrewriteSpecifier / move end to end ×5 and twostructure-ssotgit-index cases): their temp repositories run git withGIT_CONFIG_GLOBAL=NUL, which git 2.56 for Windows rejects withfatal: unable to access 'NUL'. None of them touches a file in this PR.bun run testandbun run test:changedwere not rerun for this refresh. This host runs several worktrees at once and an earliertest:changedhit 900 s with timeouts in unrelated modules, so full coverage is left to CI.maintainer-sponsoredlabel: the hygiene gate flagssrc/server/auth-cors.ts(field-policy lines and the price-multiplier validator). Everything else is ready for review.Checklist
structure/pricing.md, the management API reference, provider configuration and the web dashboard guide (English) are updated.mutates: truebehind the existing management auth.privacy:scanpasses.Review readiness checklist
This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:
Required local validation passed; commands, results, and any full-suite exception are documented.
I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).
I resolved all correct Codex and CodeRabbit findings.
My PR is ready for review.