Repository navigation
fix(pricing): add GPT-6.1 Terra, and GPT-6.1 Sol and Terra under Delegate's ids - #263
Conversation
|
Claude finished @ibabencu's task in 1m 14s —— View job PR Review in Progress
|
fc83d77 to
2d88005
Compare
uipreliga
left a comment
There was a problem hiding this comment.
Update (head moved during review): this review ran at
fc83d77. The new head2d88005adds the two missingEXPECTED_USD_PER_MTOKentries intests/test_delegate_agent_registration.py. That fixes the only 🟡 finding (Axis 3): the test now passes locally (4 passed). The 🔵 nits below still apply to2d88005, becausesrc/coder_eval/pricing.pydid not change.
Review: coder_eval — pr:263 (2 files) axis:1,2,3,4,5,6,7,8
Scope: pr:263 (2 files) axis:1,2,3,4,5,6,7,8 · branch pr-263 (ionutb-fix-pricing-gpt-6-1-terra) · fc83d77 · 2026-11-18T08:55Z · workflow variant
Change class: trivial — adds three pricing rate rows (and their generated evalboard mirror) without altering any code path
PR #263 is a small, low-risk pricing change: it adds rates and touches no grading code, and no finding can change a task's score or final_status for the same agent output, but it cannot merge yet because the two new Delegate rows at src/coder_eval/pricing.py:54-55 have no pinned test entries, so test_expected_models_match_delegate_model_ids_exactly fails and so does the Quality Gate CI check; after that fix, what remains is low-severity work on drift, provenance and row order.
Summary
| Axis | Score | 🔴 | 🟠 | 🟡 | 🔵 | Top Issue |
|---|---|---|---|---|---|---|
| 1. Code Quality & Style | 9.9 / 10 | 0 | 0 | 0 | 1 | GPT-6.1 rows are placed between gpt-6-astra and gpt-6-sol, so the GPT-6 group mixes generations |
| 2. Type Safety | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 3. Test Health | 9.5 / 10 | 0 | 0 | 1 | 0 | New _DELEGATE_PRICING rows ship without their pinned-dollar test entries, so make test / make verify fails |
| 4. Security | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 5. Architecture & Design | 9.9 / 10 | 0 | 0 | 0 | 1 | The hyphen/dot twin rate rows for GPT-6.1 Delegate ids are hand-copied, so a reprice must be made in two places (drift hazard) |
| 6. Error Handling & Resilience | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 7. API Surface & Maintainability | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 8. Evaluation Harness Quality | 9.9 / 10 | 0 | 0 | 0 | 1 | The module docstring says every row was re-verified on 2026-10-07, but the new GPT-6.1 rows have no source or verification date |
Overall Score: 9.9 / 10 · Weakest Axis: Test Health at 9.5 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 1 · 🔵 3 across 8 axes.
Blockers
None.
Non-blocking, but please consider before merge
- [Axis 3] New _DELEGATE_PRICING rows ship without their pinned-dollar test entries, so
make test/make verifyfails (src/coder_eval/pricing.py:54) — The PR adds two rows to_DELEGATE_PRICINGat lines 54-55:"gpt-6-1-sol": ModelPricing(2.0, 10.0, 2.50, 0.10),and"gpt-6-1-terra": ModelPricing(1.0, 5.0, 1.25, 0.05),. These rows feedDELEGATE_MODEL_IDS = frozenset(_DELEGATE_PRICING). The PR does not add the matching pinned entries toEXPECTED_USD_PER_MTOKin tests/test_delegate_agent_registration.py (lines 22-35). That table exists so that a new Delegate id cannot ship unverified. I ran the test at head fc83d77 and it fails:tests/test_delegate_agent_registration.py::test_expected_models_match_delegate_model_ids_exactlyreportsassert set(EXPECTED_USD_PER_MTOK) == DELEGATE_MODEL_IDSwith "Extra items in the right set: 'gpt-6-1-terra', 'gpt-6-1-sol'" (1 failed, 3 passed). The PR's "Quality Gate (Format, Lint, Type, Test, Security)" check also reports FAILURE. The routed-k 'pric or cost'run did not select this test, because neither its name nor its module name contains 'pric' or 'cost'. That is why the routed result shows green. Because of this, the new rates have no independent check of the input+output or cache-write dollar figures, so a cache_write/cache_read swap would go unnoticed. Fix: add"gpt-6-1-sol": (2.0 + 10.0, 2.50),and"gpt-6-1-terra": (1.0 + 5.0, 1.25),toEXPECTED_USD_PER_MTOK, then runmake verify, not a keyword-filtered pytest run.
Nits
- [Axis 1] GPT-6.1 rows are placed between gpt-6-astra and gpt-6-sol, so the GPT-6 group mixes generations (
src/coder_eval/pricing.py:127-132) — The block reads# CAVEAT: flat rate; GPT-6's >272K-input tier ..., then"gpt-6-astra","gpt-6.1-sol","gpt-6.1-terra","gpt-6-sol","gpt-6-luna". GPT-6.1 rows sit between GPT-6 rows, so a reader cannot tell whether the GPT-6 CAVEAT also applies to 6.1. Other generations are grouped in order (for example thegpt-5.6-*block). Move thegpt-6.1-*rows into their own group after thegpt-6-*rows. If the >272K caveat also applies to 6.1, say so. - [Axis 5] The hyphen/dot twin rate rows for GPT-6.1 Delegate ids are hand-copied, so a reprice must be made in two places (drift hazard) (
src/coder_eval/pricing.py:129) — The PR adds"gpt-6.1-terra": ModelPricing(1.0, 5.0, 1.25, 0.05),(line 130) to_PRICING, next to the existing"gpt-6.1-sol": ModelPricing(2.0, 10.0, 2.50, 0.10),(line 129). It also adds byte-identical twins under the Delegate ids at lines 54-55 ("gpt-6-1-sol","gpt-6-1-terra"). This follows the existing pattern that .claude/notes/agents.md section 'Delegate agent pricing' documents, and that note records that this pattern already drifted once:gpt-5-6-solis 5.0/30.0 but its dotted twingpt-5.6-solis 4.0/20.0 because a price cut reached only one table. Each new twin pair adds one more place where a future reprice (for example, a promotional-rate expiry) must be applied twice, and nothing enforces that. Because the change follows the documented pattern, it adds to existing debt only a little, so this is Low. If these physical models really share one rate, derive the Delegate key from the dotted row and do not copy the literal (for example,"gpt-6-1-sol": _PRICING_DOTTED["gpt-6.1-sol"], or a small{k.replace('.', '-'): v for k in (...)}alias block). Another option is a lint or test that asserts the two id families stay equal for every pair not listed as deliberately divergent. That way a reprice cannot update only one half. - [Axis 8] The module docstring says every row was re-verified on 2026-10-07, but the new GPT-6.1 rows have no source or verification date (
src/coder_eval/pricing.py:8) — Line 8 says the rate sources were checked in OpenRouter's live/api/v1/modelswith(every row re-verified 2026-10-07, except the Bedrock. The PR adds"gpt-6.1-terra": ModelPricing(1.0, 5.0, 1.25, 0.05),(line 130) and two Delegate rows, with no source and no verification date. The table has no effective dates (see the HAZARD comment at lines 121-123), so this provenance line is the only record that says a rate is still current. Fix: update the date in the docstring after you re-check the new rows against the OpenAI card, or name the new rows as an exception (as the docstring already does for the Bedrock block). This has no effect on score or final_status. It is a reproducibility / provenance paper cut only.
What's Missing
Tests:
- 🟡 Tests: the two new
_DELEGATE_PRICINGrows (gpt-6-1-sol,gpt-6-1-terra) ship without matching pinned entries inEXPECTED_USD_PER_MTOK(tests/test_delegate_agent_registration.py:22-36).test_expected_models_match_delegate_model_ids_exactlyfails and the Quality Gate is red. The PR's-k "pric or cost"run did not select this test. Add(2.0 + 10.0, 2.50)and(1.0 + 5.0, 1.25), then runmake verify. (trigger: src/coder_eval/pricing.py) _(restates: Axis 3: New DELEGATE_PRICING rows ship without their pinned-dollar test entries, somake test/make verifyfails) - 🔵 Tests: the new dotted
_PRICINGrowgpt-6.1-terra(the codex-path id named in the PR's Why section) has no independent dollar check. The pinned guard covers only Delegate's hyphenated ids, and test_pricing_registry.py / test_pricing_mirror.py do not pin rows. A swap of cache_write and cache_read on this row would pass CI, because the CE065 mirror only copies the same wrong number. Add a pinned assertion for the dotted row that this PR introduces, or a twin-equality test againstgpt-6-1-terra. (trigger: src/coder_eval/pricing.py) (restates: Axis 5: The hyphen/dot twin rate rows for GPT-6.1 Delegate ids are hand-copied, so a reprice must be made in two places (drift hazard))
Parallel paths:
- 🔵 Parallel paths: the PR prices only the hyphenated Delegate ids. If the host echoes the underscored spelling (
gpt_6_1_terra,gpt_6_1_sol), the turn still bookscost: Noneuntil open PR #223 lands the_-to--key normalization. The PR body states this dependency, but nothing in this change tracks it. Merge #223 first, or add a follow-up note, so the Autopilot runs this PR targets are fully priced on every id spelling. (trigger: src/coder_eval/pricing.py)
Downstream consumers:
- 🔵 Downstream consumers: the evalboard prices token buckets on read from the regenerated mirror (
resolvePricing/tokenBucketUsdin evalboard/lib/pricing.ts). After deploy, runs made before the merge ongpt-6.1-terra/gpt-6-1-*will show per-bucket USD on the board, but their recorded run.jsoncoststays None andcost_completestays false. The PR does not say whether those historical Autopilot runs are re-run, backfilled, or left unpriced with this split. (trigger: evalboard/lib/pricing.generated.ts)
Nightly pipeline:
- 🔵 Daily/nightly: the PR names the consuming Autopilot RPA eval pipeline (UiPath/Autopilot#9483), but it does not state the blast radius on that pipeline's cost series. Runs before the merge record
cost: None/cost_complete: false, and runs after it record dollars, so a per-model cost trend starts at the merge point. The flat rate under-reports any turn above 272K input tokens (the CAVEAT that the PR says covers these rows). Add one line to the PR description so that nightly-report readers do not read the change as a cost regression or an accurate figure. (trigger: src/coder_eval/pricing.py)
Harness & Lint Improvements
Static checks (lint / type):
- [ce-lint] Delete the hazard before you guard it. In src/coder_eval/pricing.py, replace each byte-identical Delegate twin literal with a reference to its dotted row through a small alias map, for example
_DELEGATE_ALIASES = {"gpt-6-1-sol": "gpt-6.1-sol", "gpt-6-1-terra": "gpt-6.1-terra", "gpt-5-6-terra": "gpt-5.6-terra", "gpt-5-6-luna": "gpt-5.6-luna"}, with_DELEGATE_PRICINGbuilt from it. Then add CE072 as a whole-tree@pytest.mark.lintclass in tests/test_custom_lint.py (CE069/CE070 are claimed by in-flight branches, so CE072 is the next safe id). CE072 importscoder_eval.pricing. For every_DELEGATE_PRICINGkey whose dotted form (regex^(gpt|gemini)-(\d+)-(\d+)to\1-\2.\3) is a_PRICINGkey, it requires (input, output, cache_read) to be equal. A pair that differs must be listed inDELIBERATELY_DIVERGENT_TWINS = {id: reason}, with the same 'stale entry fails' liveness check thatDELIBERATELY_UNMIRROREDuses in tests/lint/pricing_mirror.py. Compare without cache_write: on today's tree the Delegate gemini rows use 0.0 against input, and gpt-5-4/gpt-5-5 use the output rate (15.0/30.0) against the dotted 2.5/5.0. The only (input, output, cache_read) divergence today is gpt-5-6-sol (5.0/30.0/0.50 against 4.0/20.0/0.40), so the exemption set starts with exactly that one entry. NOTE: .claude/notes/agents.md § Delegate agent pricing says gpt-5.4/gpt-5.5 'use the identical convention' as their Delegate twins, but the dotted rows have cache_write 2.5/5.0, not 15.0/30.0. Correct that note when you add the rule. Prevents: Finding 3 (A5/A1/A7 low, pricing.py:54-55 and :129-130): hand-copied twin rows for gpt-6.1-sol/terra under Delegate ids, where a future reprice can update only one half. This is the same failure that already produced the documented gpt-5-6-sol vs gpt-5.6-sol split. With the alias map, the PR's two new Delegate rows cannot drift from their dotted rows. The rule catches any later literal twin that diverges without a recorded reason. - [ci-gate] Add a diff-aware CI step, for example
make pricing-provenancerun in the Quality Gate job againstorigin/$GITHUB_BASE_REF. If the PR adds or changes anyModelPricing(line in src/coder_eval/pricing.py and does not change there-verified YYYY-MM-DDdate in the module docstring, the step fails. The step also passes if each new key is named in the docstring's exception clause, as the Bedrock block already is. Long-term alternative ('remove the sharp edge'): giveModelPricinga requiredverified: str(ISO date) field, so that pyright rejects any new row that has no date. The global docstring claim and this gate can then be deleted. Prevents: Finding 4 (A8/A1 low, pricing.py:8): the docstring says every row was re-verified on 2026-10-07, but the PR adds gpt-6.1-terra, gpt-6-1-sol and gpt-6-1-terra with no source and no verification date. This check is not a pure AST rule because it needs the base-branch diff to know which rows are new. - [ce-lint] Optional, low priority: CE073, an AST rule on the
_PRICINGdict literal in src/coder_eval/pricing.py. It groups keys by their model-generation prefix (regex^(gpt|gemini|claude-[a-z]+)-(\d+(?:\.\d+)?)) and requires each group to be contiguous, so that nogpt-6.1-*key sits betweengpt-6-*keys. Only the contiguity half is static. Whether the GPT-6 >272K CAVEAT comment also applies to 6.1 is a semantic billing fact, and only a reviewer or the provider card can decide it. Prevents: Finding 1 (A1 low, pricing.py:127-132): GPT-6.1 rows placed between gpt-6-astra and gpt-6-sol, so a reader cannot tell which generation the GPT-6 CAVEAT covers.
Harness improvements (not statically reachable):
- In the review workflow's test-routing step, select tests by import graph and not by
-kkeyword. For each changed src module, run every test module that imports it (grep -rlE 'from coder_eval\.pricing|coder_eval import pricing' tests/gives 12 modules for pricing.py, including tests/test_delegate_agent_registration.py), plus all@pytest.mark.lintclasses. For a PR with 5 files or fewer, run plainmake test. Also record in the routed-results line which selector was used, so that a green result states its coverage. Why not static: This is a runtime choice that the review harness makes about which tests to execute. No source-level lint can see that-k 'pric or cost'skipped a module whose file and test names contain neither word. Prevents: Finding 2 (A3 medium, cross-axis 1/3/5/7/8): the routed-k 'pric or cost'run reported green whiletest_expected_models_match_delegate_model_ids_exactlyfailed with 'Extra items in the right set: gpt-6-1-terra, gpt-6-1-sol'. - Use the PR's own CI check status as an automated input to Axis 3. Run
gh pr checks <n>before routing tests, and treat a FAILED 'Quality Gate (Format, Lint, Type, Test, Security)' or 'Windows Smoke Test' check as a confirmed finding to diagnose. The review must not print a green routed result next to a red CI gate. Why not static: This needs live GitHub check state for a specific head SHA. It is a property of a CI run, not of the source tree. Prevents: Finding 2: the Quality Gate check already failed on this exact test (1 failed, 7047 passed), but the review's own routed run hid it. The Windows Smoke Test failure was also left undiagnosed. - Move the Delegate pinned-dollar tests (EXPECTED_USD_PER_MTOK and its three tests) from tests/test_delegate_agent_registration.py to tests/test_pricing_delegate_pins.py. Change the comment above
_DELEGATE_PRICINGin pricing.py to name that file as the place where a new row must also add its pinned (input+output, cache_write) figures. Keep the pins hand-written, because an independent second copy is what catches a cache_write/cache_read transposition, so they must not be derived. Why not static: The guard itself is already a correct test that fails when it should. The gap is discoverability: an author who adds a rate row does not know the pins file exists, and keyword or name-based selection misses it. This is a test-organization and author-workflow fix, not a detectable code pattern. Prevents: Finding 2: the two new _DELEGATE_PRICING rows shipped without their pinned entries. - Add a scheduled (nightly or weekly) job, separate from
make verify, that compares every_PRICING/_DELEGATE_PRICINGrow with OpenRouter's live/api/v1/models(and the provider cards where OpenRouter has no listing). It reports rows whose rates differ, rows that no source lists, and promotional HAZARD dates that have passed (for example, the gpt-5.6-sol promo 're-check 2026-11-21'). The job opens an issue and does not fail any PR build. Why not static: It needs network access to live third-party price lists, and the answer changes over time without any code change, so it cannot run as a deterministic lint in CI. Prevents: Finding 4 (an unverified provenance claim for new rows) and the reprice drift behind Finding 3: a reprice that reaches only one twin table shows up as a mismatch against the live source.
Top 5 Priority Actions
- Make the PR merge-ready: add
"gpt-6-1-sol": (2.0 + 10.0, 2.50)and"gpt-6-1-terra": (1.0 + 5.0, 1.25)toEXPECTED_USD_PER_MTOK(tests/test_delegate_agent_registration.py:22-36), to match the new_DELEGATE_PRICINGrows at src/coder_eval/pricing.py:54-55; this fixes the failing set-equality test, which is the only failure in the Quality Gate CI check, and gives the new rates an independent check on their dollar figures. - Run
make verifyon the PR, not a pytest run filtered with-k 'pric or cost', because that filter skipped the failing Delegate test and showed a false green; also find out why the Windows Smoke Test check fails, because nobody has checked it yet. - Stop hand-copying the twin rate rows: build the Delegate keys at src/coder_eval/pricing.py:54-55 from the dotted rows at :129-130, or add a test that the two id families stay equal, so a future reprice cannot update only one table (this already happened once with gpt-5-6-sol vs gpt-5.6-sol).
- Fix the provenance claim at src/coder_eval/pricing.py:8 ('every row re-verified 2026-10-07'): check the new GPT-6.1 Terra and Delegate rows against the OpenAI card and update the date, or list them as an exception, as the docstring already does for the Bedrock block.
- Move the
gpt-6.1-*rows at src/coder_eval/pricing.py:127-132 into their own group after thegpt-6-*rows, and say whether the GPT-6 >272K-input flat-rate caveat also applies to GPT-6.1.
Stats: 0 🔴 · 0 🟠 · 1 🟡 · 3 🔵 across 8 axes reviewed.
…gate's ids GPT-6.1 Terra has no row, and the Delegate block has no GPT-6.1 row, so coder-eval books cost None for: - codex runs on gpt-6.1-terra; - delegate / delegate-sdk runs on gpt-6-1-terra and gpt-6-1-sol (the Delegate host echoes the caller's hyphenated id back as the model). Rates are OpenAI's standard tier (2026-11-18): Terra $1.00 in / $5.00 out / $1.25 cache write / $0.05 cache read per MTok. The Delegate gpt-6-1-sol row copies the existing gpt-6.1-sol row. pricing.generated.ts regenerated with make pricing-mirror. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2d88005 to
2c49a9e
Compare
|
Thanks for the review. Addressed at
|
|
Two points I missed in my last reply:
|
bai-uipath
left a comment
There was a problem hiding this comment.
LGTM — approving.
- Rates: GPT-6.1 Terra matches OpenAI's standard tier ($1 in / $5 out, $1.25 for the 30-minute cache write, $0.05 cache read). The two Delegate rows match their dotted twins, and the pinned dollar figures cover both new ids.
- Rollout: Autopilot keeps booking
cost: Noneuntil a coder_eval release ships this and itscoderEvalVersionpin moves, so cut the patch release right after merge.

Why
GPT-6.1 Terra has no rate, and the Delegate block has no GPT-6.1 rows. So coder-eval books
cost: None, logs "No pricing rate", and setscost_completeto false for:codexruns ongpt-6.1-terra;delegate/delegate-sdkruns ongpt-6-1-terraandgpt-6-1-sol. The Delegate host echoes the caller's hyphenated id back as the turn's model, andDelegateAgentprices the turn under that id.Autopilot's RPA eval pipeline added GPT-6.1 Terra in UiPath/Autopilot#9483, and the reviewer found these runs would report no cost.
What changes
src/coder_eval/pricing.py:_PRICING: newgpt-6.1-terra, $1.00 in / $5.00 out / $1.25 cache write / $0.05 cache read per MTok._DELEGATE_PRICING: newgpt-6-1-terra(same rates) andgpt-6-1-sol(a copy of the existinggpt-6.1-solrow: $2.00 / $10.00 / $2.50 / $0.10).evalboard/lib/pricing.generated.ts: regenerated withmake pricing-mirror.tests/test_delegate_agent_registration.py: the two new Delegate ids added toEXPECTED_USD_PER_MTOK, the independently pinned dollar figures.Review notes
gpt-6.1-solrow still matches the card.>272K-input tier is not modelled. OpenAI bills GPT-6.1 there at 2x input / 1.5x output, like GPT-6, so the GPT-6 CAVEAT comment now names both, and thegpt-6.1-*rows sit after thegpt-6-*rows.cost: Noneuntil Autopilot moves itscoderEvalVersionpin to a release with this change. Runs after that record dollars, so their cost trend starts there. Runs with more than 272K input tokens read low.run.jsonkeepscost: Noneandcost_complete: false. The evalboard prices token buckets when it reads them, so after deploy it shows per-bucket USD for those runs too._to-rate key for Delegate's underscored ids) touches a different part ofpricing.pyand needs no change for these rows. With fix(pricing): price Delegate's underscored model ids and flag unpriced judge/simulator cost #223,gpt_6_1_terraalso prices.gpt-5-6-sol, so it is a separate change.Testing
calculate_costwith 1M tokens in each bucket:gpt-6-1-terraandgpt-6.1-terra= $7.30;gpt-6-1-solandgpt-6.1-sol= $14.60.tests/test_delegate_agent_registration.py: 4 passed.tests/test_pricing_mirror.py,tests/test_pricing_registry.py: 33 passed.pytest -k "pric or cost": 205 passed, 2 skipped (docker, POSIX-only).tests/test_custom_lint.py(incl. CE065 mirror drift): 753 passed.ruff check,ruff format --check,prose_budget: clean.Jira: PILOT-9277
🤖 Generated with Claude Code