Skip to content

feat(pricing): fetch model prices from models.dev and OpenRouter, label every price's source - #8708

Closed
andrew05060414 wants to merge 6 commits into
lidge-jun:devfrom
andrew05060414:feat/auto-model-pricing
Closed

andrew05060414 wants to merge 6 commits into
lidge-jun:devfrom
andrew05060414:feat/auto-model-pricing

Conversation

@andrew05060414

@andrew05060414 andrew05060414 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Many models have no price, so the dashboard shows an unknown cost for a large share of traffic. In one local install, the last 30 days included about 15k unpriced deepseek-v4.1-flash attempts across four relays. There were also gemini-4-argon on Vertex/Cursor/Zed, cn:-prefixed Kimi/GLM/MiniMax/Step ids, and zd/-prefixed relay ids. This PR fetches public prices automatically, labels where every price came from, and keeps a manually set price authoritative.

Depends on #8674 (stacked on feat/model-capability-inference; it reuses capabilityInferenceKey). Review only the top commit until #8674 lands.

  • Two new lowest price tiers. The resolution order is manual modelCosts > official (verified overrides, bundled table, expected overlay, model-level vendor fallback) > models.dev > OpenRouter > unknown.
    • models.dev prices come only from original-vendor entries, not reseller listings. OpenRouter per-token prices are converted to per-1M.
    • Catalog prices are verified-derived, so every estimate using one is marked estimated. Each one carries the feed URL and fetch time.
  • Matching. Ids are matched on the feat(models): infer unknown model capabilities by name, label their source, and probe what a relay accepts #8674 key (case, vendor/ prefix, dots vs hyphens, [...] suffix), plus a leading channel: prefix (cn:, global:). If two rows in one feed claim the same key with different prices, the key is dropped.
  • Cache and refresh. Prices are cached in <configDir>/price-catalog-cache.json.
    • A full refresh of each source runs once a day, triggered from the dashboard reads (GET /api/models and GET /api/usage).
    • A pricing miss queues a background fetch that merges only the missing keys. A key that is still missing cools down for 6h, and each source gets at most one miss-driven fetch per window.
    • Nothing runs on the request path or awaits the network there. Failures keep the previous snapshot; with no cache, behavior is unchanged.
    • Fetches are anonymous GETs with redirect: "error", a 15s timeout and a 32 MiB body cap.
  • Relay multiplier. providers.<name>.priceMultiplier (opt-in, > 0 and <= 1000) scales automatic prices for relays that charge a fixed multiple of list price. It never scales manual prices.
  • Subscription label. Providers using OAuth or forward auth, and Codex account identities, carry apiEquivalent, so their cost reads as an API-equivalent estimate.
  • API.
    • /api/models rows gain price (manual | official | models-dev | openrouter plus effective rates, fetchedAt, multiplier, apiEquivalent).
    • New POST /api/pricing/refresh (forced, clears cooldowns) and PUT /api/providers/:name/price-multiplier.
    • GET .../model-costs returns priceMultiplier when set.
    • New cost reasons: online_price_catalog, relay_price_multiplier, subscription_api_equivalent.
  • Dashboard.
    • Every model row shows a price-source chip whose tooltip gives the rates, fetch time and multiplier.
    • The price dialog shows the automatic price, with Adopt as manual and Refresh prices buttons and the provider multiplier field.
    • Logs show the new sources and reasons.
    • All 11 locales are covered.
  • Base-commit fix. Registers POST /api/providers/probe-capabilities in route-registry.ts. feat(models): infer unknown model capabilities by name, label their source, and probe what a relay accepts #8674 added the route without an entry, which failed the registry reconciliation test.

Models with no public price anywhere (composer-2.5, grok-composer-2.5-fast, codex-auto-review) stay unknown.

Verification

  • bun run typecheck: pass.
  • bun run privacy:scan: pass.
  • bun run structure:check: pass.
  • New tests, all against hand-written fixtures with no network access:
    • tests/usage/price-catalog.test.ts covers parsing, key matching, ambiguity, TTL, coalescing, offline fallback, corrupt cache, backoff and forced refresh.
    • tests/usage/price-catalog-miss.test.ts covers miss trigger, dedupe, cooldown, merge-only-missing, and that resolution returns synchronously while a fetch is still pending.
    • tests/usage/price-catalog-resolution.test.ts covers priority, user zero still wins, multiplier, subscription and cost reasons.
    • tests/server/price-catalog-management-api.test.ts covers the multiplier GET/PUT sparse round trip with rollback, the refresh route, load-degrade and the /api/models price field.
    • All pass.
  • Registry, layout and ratchet tests: route-registry reconciliation, test-layout, test-layout-tooling, structure-ssot, file-size ratchet, cli-capability-data and the model-costs suites pass.
  • tests/server/model-costs-pricing-cache.test.ts stage 2 is flaky on Windows on the base commit as well: one clean-HEAD run failed identically and the next passed.
  • GUI:
    • bun x tsc -b, bun run lint:gui, bun run lint:i18n and bun run build:gui pass.
    • gui/tests/models-price-editor.test.tsx (29) and the locale parity tests pass.
    • 597 related GUI tests pass.
  • Pricing tests: bun test tests/usage/price-catalog*.test.ts tests/server/price-catalog-management-api.test.ts gives 69 pass and 0 fail.
  • bun run test:changed did not complete. The lane hit its timeout (exit 124) on a machine running several concurrent test suites, so I am not claiming a pass and am leaving the full run to CI.
    • Before the kill, about 19 files reported failures, all in areas this PR does not touch, and none in a file it adds or edits.
    • Re-run at a 90s timeout, the routing, jev-quota, provider and route-intent-aside files pass, so those were timeouts under load.
    • tests/routing/routing-compatibility-boundaries.test.ts (2 fail) and tests/clients/route-intent-connected.test.ts (child bun.exe ETIMEDOUT) fail the same way on a clean checkout of the base commit 787f8c111.
    • The remote-workspace, anthropic-pool and replay-fixture failures (EBUSY, early fixture exit) were not individually re-checked.
  • End to end on a scratch install (separate config dir, placeholder keys): the first dashboard load fetched both feeds.
    • The previously unpriced rows showed prices. cn:deepseek-v4.1-flash came from OpenRouter (0.3/1.2). cn:kimi-k3.1-pro, cn:glm-5.3-flash, zd/gpt-6-sol and zd/gemini-4-argon came from models.dev.
    • A relay with priceMultiplier: 0.5 showed official Claude rates halved, while its manual row stayed unscaled.
    • Adopt as manual turned a row's chip to Manual and persisted the effective rates.

Model list with source chips:

model rows with price source chips

relay rows

Price dialog with the automatic price, adopt, refresh and multiplier controls:

price dialog

Refreshed on dev 86be880 (2026-11-13)

Merged the current dev head 86be8805c into the branch (merge commit, no history rewrite). Stacked on #8674: this branch merges the refreshed feat/model-capability-inference.

  • bun run typecheck, bun run structure:check, bun run skill:surface:check and bun run privacy:scan all exit 0. Focused run: the layout, file-size ratchet, repo-hygiene, management route registry, skill-ocx, structure-ssot and import-cycle guards plus every test file this PR adds or changes.
  • 296 tests: 288 pass, 1 skip, 7 fail (includes the feat(models): infer unknown model capabilities by name, label their source, and probe what a relay accepts #8674 route-registry fix). GUI: npx tsc -b and bun run lint exit 0; models-price-editor: 29 pass, 0 fail.
  • The 7 failures are the same 7 cases that fail on an untouched dev checkout on this Windows host (test-layout-tooling rewriteSpecifier / move end to end ×5 and two structure-ssot git-index cases): their temp repositories run git with GIT_CONFIG_GLOBAL=NUL, which git 2.56 for Windows rejects with fatal: unable to access 'NUL'. None of them touches a file in this PR.
  • Full-suite exception: the full bun run test and bun run test:changed were not rerun for this refresh. This host runs several worktrees at once and an earlier test:changed hit 900 s with timeouts in unrelated modules, so full coverage is left to CI.
  • Still needs the maintainer-sponsored label: the hygiene gate flags src/server/auth-cors.ts (field-policy lines and the price-multiplier validator). Everything else is ready for review.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed. structure/pricing.md, the management API reference, provider configuration and the web dashboard guide (English) are updated.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.
    • The new outbound fetches are credential-free GETs to two fixed public URLs.
    • Error statuses carry short codes, never bodies.
    • Both new routes are registered mutates: true behind the existing management auth.
    • privacy:scan passes.

Review readiness checklist

This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:

  • Required local validation passed; commands, results, and any full-suite exception are documented.

  • I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).

  • I resolved all correct Codex and CodeRabbit findings.

  • My PR is ready for review.

…xis's source, and probe what a relay accepts

GET /api/models routed rows gain capabilitySources (manual > probed > reported >
inferred > unknown) and display-only inferredCapabilities from a same-name lookup
over the vendor registry, bundled metadata and other providers' declarations.
POST /api/providers/probe-capabilities sends a baseline, a 1x1 image and one
request per effort through the completion diagnostic, asks before spending on a
metered provider, and with apply writes the conclusive axes plus a sparse
modelCapabilityProbes label.
…el every price's source

Automatic prices fill the two lowest tiers of price resolution (manual > official > bundled >
models.dev > OpenRouter). Catalogs are cached in price-catalog-cache.json, refreshed daily and on
a pricing miss (merging only the missing keys, 6h per-key cooldown, never on the request path).
Catalog prices are verified-derived estimates carrying their feed URL and fetch time.

Adds an opt-in providers.<name>.priceMultiplier for relays (never applied to manual prices),
apiEquivalent labelling for subscription providers, a price field on /api/models rows,
POST /api/pricing/refresh and PUT /api/providers/:name/price-multiplier. The dashboard shows a
source chip per model and adds adopt-as-manual, refresh and multiplier controls.

Also registers POST /api/providers/probe-capabilities, which the base commit left out of the
route registry.
@coderabbitai

coderabbitai Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions github-actions Bot added the intake: hygiene-blocked Deterministic PR hygiene checks failed label Nov 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Deterministic hygiene checks failed.

  • unsponsored_surface — This changes an authentication, workflow, release-automation, or dependency surface. MAINTAINERS.md requires security review for these; ask a maintainer to apply maintainer-sponsored once they have reviewed it. Paths: src/server/auth-cors.ts.

@github-actions github-actions Bot added the enhancement New feature or request label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

⏳ DRAFT

  • hygiene: unsponsored_surface.

What to do

  • Fix unsponsored_surface — This changes an authentication, workflow, release-automation, or dependency surface. MAINTAINERS.md requires security review for these; ask a maintainer to apply maintainer-sponsored once they have reviewed it. Paths: src/server/auth-cors.ts.

Review readiness checklist

  • ✅ Required local validation passed; commands, results, and any full-suite exception are documented.
  • ✅ I pushed my PR to a recent dev commit (at most 10 behind; a maintainer may still ask for the exact tip before merge).
  • ✅ I resolved all correct Codex and CodeRabbit findings.
  • ✅ My PR is ready for review.

✅ 4/4 boxes ticked.

This pull request was already a draft. Its draft status will be preserved after every issue above is resolved.

Hygiene

⚠️ Deterministic hygiene checks failed.

  • unsponsored_surface — This changes an authentication, workflow, release-automation, or dependency surface. MAINTAINERS.md requires security review for these; ask a maintainer to apply maintainer-sponsored once they have reviewed it. Paths: src/server/auth-cors.ts.

@andrew05060414

Copy link
Copy Markdown
Contributor Author

Superseded by #8738, which carries this change together with the related model metadata and usage PRs as one commit each on current dev, with no stacking. The design notes in this description still apply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request intake: hygiene-blocked Deterministic PR hygiene checks failed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant