Skip to content

fix(pricing): add GPT-6.1 Terra, and GPT-6.1 Sol and Terra under Delegate's ids - #263

Merged
ibabencu merged 1 commit into
mainfrom
ionutb-fix-pricing-gpt-6-1-terra
Nov 19, 2026
Merged

ibabencu merged 1 commit into
mainfrom
ionutb-fix-pricing-gpt-6-1-terra

Conversation

@ibabencu

@ibabencu ibabencu commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

GPT-6.1 Terra has no rate, and the Delegate block has no GPT-6.1 rows. So coder-eval books cost: None, logs "No pricing rate", and sets cost_complete to false for:

  • codex runs on gpt-6.1-terra;
  • delegate / delegate-sdk runs on gpt-6-1-terra and gpt-6-1-sol. The Delegate host echoes the caller's hyphenated id back as the turn's model, and DelegateAgent prices the turn under that id.

Autopilot's RPA eval pipeline added GPT-6.1 Terra in UiPath/Autopilot#9483, and the reviewer found these runs would report no cost.

What changes

  • src/coder_eval/pricing.py:
    • _PRICING: new gpt-6.1-terra, $1.00 in / $5.00 out / $1.25 cache write / $0.05 cache read per MTok.
    • _DELEGATE_PRICING: new gpt-6-1-terra (same rates) and gpt-6-1-sol (a copy of the existing gpt-6.1-sol row: $2.00 / $10.00 / $2.50 / $0.10).
  • evalboard/lib/pricing.generated.ts: regenerated with make pricing-mirror.
  • tests/test_delegate_agent_registration.py: the two new Delegate ids added to EXPECTED_USD_PER_MTOK, the independently pinned dollar figures.

Review notes

  • Rates come from OpenAI's standard tier, checked 2026-11-18. The cache-write figure is the 30-minute write (1.25x input), the same convention as the other GPT-5.6 and GPT-6 rows. The page's gpt-6.1-sol row still matches the card.
  • The >272K-input tier is not modelled. OpenAI bills GPT-6.1 there at 2x input / 1.5x output, like GPT-6, so the GPT-6 CAVEAT comment now names both, and the gpt-6.1-* rows sit after the gpt-6-* rows.
  • Cost series: Autopilot RPA eval runs on these models report cost: None until Autopilot moves its coderEvalVersion pin to a release with this change. Runs after that record dollars, so their cost trend starts there. Runs with more than 272K input tokens read low.
  • Runs from before the merge are not re-run or backfilled. Their run.json keeps cost: None and cost_complete: false. The evalboard prices token buckets when it reads them, so after deploy it shows per-bucket USD for those runs too.
  • Open fix(pricing): price Delegate's underscored model ids and flag unpriced judge/simulator cost #223 (_ to - rate key for Delegate's underscored ids) touches a different part of pricing.py and needs no change for these rows. With fix(pricing): price Delegate's underscored model ids and flag unpriced judge/simulator cost #223, gpt_6_1_terra also prices.
  • The docstring keeps "every row re-verified 2026-10-07" and now says the new GPT-6.1 rows were added 2026-11-18. I did not re-check the other rows.
  • Not in this PR: deriving the Delegate twin rows from the dotted rows (review nit). It would change every existing twin pair, including the deliberately different gpt-5-6-sol, so it is a separate change.

Testing

  • calculate_cost with 1M tokens in each bucket: gpt-6-1-terra and gpt-6.1-terra = $7.30; gpt-6-1-sol and gpt-6.1-sol = $14.60.
  • tests/test_delegate_agent_registration.py: 4 passed.
  • tests/test_pricing_mirror.py, tests/test_pricing_registry.py: 33 passed.
  • pytest -k "pric or cost": 205 passed, 2 skipped (docker, POSIX-only).
  • tests/test_custom_lint.py (incl. CE065 mirror drift): 753 passed.
  • ruff check, ruff format --check, prose_budget: clean.

Jira: PILOT-9277

🤖 Generated with Claude Code

@ibabencu
ibabencu marked this pull request as ready for review November 18, 2026 08:20
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Claude finished @ibabencu's task in 1m 14s —— View job


PR Review in Progress

Reviewing: fix(pricing): add GPT-6.1 Terra, and GPT-6.1 Sol and Terra under Delegate's ids

Todo List:

  • Read .github/code_review.md for review standards
  • Read CLAUDE.md for project conventions
  • Review git diff for changes
  • Read full pricing.py file for context
  • Read full pricing.generated.ts file
  • Verify pricing rates are correct
  • Check cross-file consistency
  • Look for missing test coverage or documentation
  • Provide structured review feedback

@ibabencu
ibabencu force-pushed the ionutb-fix-pricing-gpt-6-1-terra branch from fc83d77 to 2d88005 Compare November 18, 2026 09:00

@uipreliga uipreliga left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update (head moved during review): this review ran at fc83d77. The new head 2d88005 adds the two missing EXPECTED_USD_PER_MTOK entries in tests/test_delegate_agent_registration.py. That fixes the only 🟡 finding (Axis 3): the test now passes locally (4 passed). The 🔵 nits below still apply to 2d88005, because src/coder_eval/pricing.py did not change.

Review: coder_eval — pr:263 (2 files) axis:1,2,3,4,5,6,7,8

Scope: pr:263 (2 files) axis:1,2,3,4,5,6,7,8 · branch pr-263 (ionutb-fix-pricing-gpt-6-1-terra) · fc83d77 · 2026-11-18T08:55Z · workflow variant

Change class: trivial — adds three pricing rate rows (and their generated evalboard mirror) without altering any code path

PR #263 is a small, low-risk pricing change: it adds rates and touches no grading code, and no finding can change a task's score or final_status for the same agent output, but it cannot merge yet because the two new Delegate rows at src/coder_eval/pricing.py:54-55 have no pinned test entries, so test_expected_models_match_delegate_model_ids_exactly fails and so does the Quality Gate CI check; after that fix, what remains is low-severity work on drift, provenance and row order.

Summary

Axis Score 🔴 🟠 🟡 🔵 Top Issue
1. Code Quality & Style 9.9 / 10 0 0 0 1 GPT-6.1 rows are placed between gpt-6-astra and gpt-6-sol, so the GPT-6 group mixes generations
2. Type Safety 10 / 10 0 0 0 0 —
3. Test Health 9.5 / 10 0 0 1 0 New _DELEGATE_PRICING rows ship without their pinned-dollar test entries, so make test / make verify fails
4. Security 10 / 10 0 0 0 0 —
5. Architecture & Design 9.9 / 10 0 0 0 1 The hyphen/dot twin rate rows for GPT-6.1 Delegate ids are hand-copied, so a reprice must be made in two places (drift hazard)
6. Error Handling & Resilience 10 / 10 0 0 0 0 —
7. API Surface & Maintainability 10 / 10 0 0 0 0 —
8. Evaluation Harness Quality 9.9 / 10 0 0 0 1 The module docstring says every row was re-verified on 2026-10-07, but the new GPT-6.1 rows have no source or verification date

Overall Score: 9.9 / 10 · Weakest Axis: Test Health at 9.5 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 1 · 🔵 3 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

  1. [Axis 3] New _DELEGATE_PRICING rows ship without their pinned-dollar test entries, so make test / make verify fails (src/coder_eval/pricing.py:54) — The PR adds two rows to _DELEGATE_PRICING at lines 54-55: "gpt-6-1-sol": ModelPricing(2.0, 10.0, 2.50, 0.10), and "gpt-6-1-terra": ModelPricing(1.0, 5.0, 1.25, 0.05),. These rows feed DELEGATE_MODEL_IDS = frozenset(_DELEGATE_PRICING). The PR does not add the matching pinned entries to EXPECTED_USD_PER_MTOK in tests/test_delegate_agent_registration.py (lines 22-35). That table exists so that a new Delegate id cannot ship unverified. I ran the test at head fc83d77 and it fails: tests/test_delegate_agent_registration.py::test_expected_models_match_delegate_model_ids_exactly reports assert set(EXPECTED_USD_PER_MTOK) == DELEGATE_MODEL_IDS with "Extra items in the right set: 'gpt-6-1-terra', 'gpt-6-1-sol'" (1 failed, 3 passed). The PR's "Quality Gate (Format, Lint, Type, Test, Security)" check also reports FAILURE. The routed -k 'pric or cost' run did not select this test, because neither its name nor its module name contains 'pric' or 'cost'. That is why the routed result shows green. Because of this, the new rates have no independent check of the input+output or cache-write dollar figures, so a cache_write/cache_read swap would go unnoticed. Fix: add "gpt-6-1-sol": (2.0 + 10.0, 2.50), and "gpt-6-1-terra": (1.0 + 5.0, 1.25), to EXPECTED_USD_PER_MTOK, then run make verify, not a keyword-filtered pytest run.

Nits

  1. [Axis 1] GPT-6.1 rows are placed between gpt-6-astra and gpt-6-sol, so the GPT-6 group mixes generations (src/coder_eval/pricing.py:127-132) — The block reads # CAVEAT: flat rate; GPT-6's >272K-input tier ..., then "gpt-6-astra", "gpt-6.1-sol", "gpt-6.1-terra", "gpt-6-sol", "gpt-6-luna". GPT-6.1 rows sit between GPT-6 rows, so a reader cannot tell whether the GPT-6 CAVEAT also applies to 6.1. Other generations are grouped in order (for example the gpt-5.6-* block). Move the gpt-6.1-* rows into their own group after the gpt-6-* rows. If the >272K caveat also applies to 6.1, say so.
  2. [Axis 5] The hyphen/dot twin rate rows for GPT-6.1 Delegate ids are hand-copied, so a reprice must be made in two places (drift hazard) (src/coder_eval/pricing.py:129) — The PR adds "gpt-6.1-terra": ModelPricing(1.0, 5.0, 1.25, 0.05), (line 130) to _PRICING, next to the existing "gpt-6.1-sol": ModelPricing(2.0, 10.0, 2.50, 0.10), (line 129). It also adds byte-identical twins under the Delegate ids at lines 54-55 ("gpt-6-1-sol", "gpt-6-1-terra"). This follows the existing pattern that .claude/notes/agents.md section 'Delegate agent pricing' documents, and that note records that this pattern already drifted once: gpt-5-6-sol is 5.0/30.0 but its dotted twin gpt-5.6-sol is 4.0/20.0 because a price cut reached only one table. Each new twin pair adds one more place where a future reprice (for example, a promotional-rate expiry) must be applied twice, and nothing enforces that. Because the change follows the documented pattern, it adds to existing debt only a little, so this is Low. If these physical models really share one rate, derive the Delegate key from the dotted row and do not copy the literal (for example, "gpt-6-1-sol": _PRICING_DOTTED["gpt-6.1-sol"], or a small {k.replace('.', '-'): v for k in (...)} alias block). Another option is a lint or test that asserts the two id families stay equal for every pair not listed as deliberately divergent. That way a reprice cannot update only one half.
  3. [Axis 8] The module docstring says every row was re-verified on 2026-10-07, but the new GPT-6.1 rows have no source or verification date (src/coder_eval/pricing.py:8) — Line 8 says the rate sources were checked in OpenRouter's live /api/v1/models with (every row re-verified 2026-10-07, except the Bedrock. The PR adds "gpt-6.1-terra": ModelPricing(1.0, 5.0, 1.25, 0.05), (line 130) and two Delegate rows, with no source and no verification date. The table has no effective dates (see the HAZARD comment at lines 121-123), so this provenance line is the only record that says a rate is still current. Fix: update the date in the docstring after you re-check the new rows against the OpenAI card, or name the new rows as an exception (as the docstring already does for the Bedrock block). This has no effect on score or final_status. It is a reproducibility / provenance paper cut only.

What's Missing

Tests:

  • 🟡 Tests: the two new _DELEGATE_PRICING rows (gpt-6-1-sol, gpt-6-1-terra) ship without matching pinned entries in EXPECTED_USD_PER_MTOK (tests/test_delegate_agent_registration.py:22-36). test_expected_models_match_delegate_model_ids_exactly fails and the Quality Gate is red. The PR's -k "pric or cost" run did not select this test. Add (2.0 + 10.0, 2.50) and (1.0 + 5.0, 1.25), then run make verify. (trigger: src/coder_eval/pricing.py) _(restates: Axis 3: New DELEGATE_PRICING rows ship without their pinned-dollar test entries, so make test / make verify fails)
  • 🔵 Tests: the new dotted _PRICING row gpt-6.1-terra (the codex-path id named in the PR's Why section) has no independent dollar check. The pinned guard covers only Delegate's hyphenated ids, and test_pricing_registry.py / test_pricing_mirror.py do not pin rows. A swap of cache_write and cache_read on this row would pass CI, because the CE065 mirror only copies the same wrong number. Add a pinned assertion for the dotted row that this PR introduces, or a twin-equality test against gpt-6-1-terra. (trigger: src/coder_eval/pricing.py) (restates: Axis 5: The hyphen/dot twin rate rows for GPT-6.1 Delegate ids are hand-copied, so a reprice must be made in two places (drift hazard))

Parallel paths:

  • 🔵 Parallel paths: the PR prices only the hyphenated Delegate ids. If the host echoes the underscored spelling (gpt_6_1_terra, gpt_6_1_sol), the turn still books cost: None until open PR #223 lands the _-to-- key normalization. The PR body states this dependency, but nothing in this change tracks it. Merge #223 first, or add a follow-up note, so the Autopilot runs this PR targets are fully priced on every id spelling. (trigger: src/coder_eval/pricing.py)

Downstream consumers:

  • 🔵 Downstream consumers: the evalboard prices token buckets on read from the regenerated mirror (resolvePricing / tokenBucketUsd in evalboard/lib/pricing.ts). After deploy, runs made before the merge on gpt-6.1-terra / gpt-6-1-* will show per-bucket USD on the board, but their recorded run.json cost stays None and cost_complete stays false. The PR does not say whether those historical Autopilot runs are re-run, backfilled, or left unpriced with this split. (trigger: evalboard/lib/pricing.generated.ts)

Nightly pipeline:

  • 🔵 Daily/nightly: the PR names the consuming Autopilot RPA eval pipeline (UiPath/Autopilot#9483), but it does not state the blast radius on that pipeline's cost series. Runs before the merge record cost: None / cost_complete: false, and runs after it record dollars, so a per-model cost trend starts at the merge point. The flat rate under-reports any turn above 272K input tokens (the CAVEAT that the PR says covers these rows). Add one line to the PR description so that nightly-report readers do not read the change as a cost regression or an accurate figure. (trigger: src/coder_eval/pricing.py)

Harness & Lint Improvements

Static checks (lint / type):

  • [ce-lint] Delete the hazard before you guard it. In src/coder_eval/pricing.py, replace each byte-identical Delegate twin literal with a reference to its dotted row through a small alias map, for example _DELEGATE_ALIASES = {"gpt-6-1-sol": "gpt-6.1-sol", "gpt-6-1-terra": "gpt-6.1-terra", "gpt-5-6-terra": "gpt-5.6-terra", "gpt-5-6-luna": "gpt-5.6-luna"}, with _DELEGATE_PRICING built from it. Then add CE072 as a whole-tree @pytest.mark.lint class in tests/test_custom_lint.py (CE069/CE070 are claimed by in-flight branches, so CE072 is the next safe id). CE072 imports coder_eval.pricing. For every _DELEGATE_PRICING key whose dotted form (regex ^(gpt|gemini)-(\d+)-(\d+) to \1-\2.\3) is a _PRICING key, it requires (input, output, cache_read) to be equal. A pair that differs must be listed in DELIBERATELY_DIVERGENT_TWINS = {id: reason}, with the same 'stale entry fails' liveness check that DELIBERATELY_UNMIRRORED uses in tests/lint/pricing_mirror.py. Compare without cache_write: on today's tree the Delegate gemini rows use 0.0 against input, and gpt-5-4/gpt-5-5 use the output rate (15.0/30.0) against the dotted 2.5/5.0. The only (input, output, cache_read) divergence today is gpt-5-6-sol (5.0/30.0/0.50 against 4.0/20.0/0.40), so the exemption set starts with exactly that one entry. NOTE: .claude/notes/agents.md § Delegate agent pricing says gpt-5.4/gpt-5.5 'use the identical convention' as their Delegate twins, but the dotted rows have cache_write 2.5/5.0, not 15.0/30.0. Correct that note when you add the rule. Prevents: Finding 3 (A5/A1/A7 low, pricing.py:54-55 and :129-130): hand-copied twin rows for gpt-6.1-sol/terra under Delegate ids, where a future reprice can update only one half. This is the same failure that already produced the documented gpt-5-6-sol vs gpt-5.6-sol split. With the alias map, the PR's two new Delegate rows cannot drift from their dotted rows. The rule catches any later literal twin that diverges without a recorded reason.
  • [ci-gate] Add a diff-aware CI step, for example make pricing-provenance run in the Quality Gate job against origin/$GITHUB_BASE_REF. If the PR adds or changes any ModelPricing( line in src/coder_eval/pricing.py and does not change the re-verified YYYY-MM-DD date in the module docstring, the step fails. The step also passes if each new key is named in the docstring's exception clause, as the Bedrock block already is. Long-term alternative ('remove the sharp edge'): give ModelPricing a required verified: str (ISO date) field, so that pyright rejects any new row that has no date. The global docstring claim and this gate can then be deleted. Prevents: Finding 4 (A8/A1 low, pricing.py:8): the docstring says every row was re-verified on 2026-10-07, but the PR adds gpt-6.1-terra, gpt-6-1-sol and gpt-6-1-terra with no source and no verification date. This check is not a pure AST rule because it needs the base-branch diff to know which rows are new.
  • [ce-lint] Optional, low priority: CE073, an AST rule on the _PRICING dict literal in src/coder_eval/pricing.py. It groups keys by their model-generation prefix (regex ^(gpt|gemini|claude-[a-z]+)-(\d+(?:\.\d+)?)) and requires each group to be contiguous, so that no gpt-6.1-* key sits between gpt-6-* keys. Only the contiguity half is static. Whether the GPT-6 >272K CAVEAT comment also applies to 6.1 is a semantic billing fact, and only a reviewer or the provider card can decide it. Prevents: Finding 1 (A1 low, pricing.py:127-132): GPT-6.1 rows placed between gpt-6-astra and gpt-6-sol, so a reader cannot tell which generation the GPT-6 CAVEAT covers.

Harness improvements (not statically reachable):

  • In the review workflow's test-routing step, select tests by import graph and not by -k keyword. For each changed src module, run every test module that imports it (grep -rlE 'from coder_eval\.pricing|coder_eval import pricing' tests/ gives 12 modules for pricing.py, including tests/test_delegate_agent_registration.py), plus all @pytest.mark.lint classes. For a PR with 5 files or fewer, run plain make test. Also record in the routed-results line which selector was used, so that a green result states its coverage. Why not static: This is a runtime choice that the review harness makes about which tests to execute. No source-level lint can see that -k 'pric or cost' skipped a module whose file and test names contain neither word. Prevents: Finding 2 (A3 medium, cross-axis 1/3/5/7/8): the routed -k 'pric or cost' run reported green while test_expected_models_match_delegate_model_ids_exactly failed with 'Extra items in the right set: gpt-6-1-terra, gpt-6-1-sol'.
  • Use the PR's own CI check status as an automated input to Axis 3. Run gh pr checks <n> before routing tests, and treat a FAILED 'Quality Gate (Format, Lint, Type, Test, Security)' or 'Windows Smoke Test' check as a confirmed finding to diagnose. The review must not print a green routed result next to a red CI gate. Why not static: This needs live GitHub check state for a specific head SHA. It is a property of a CI run, not of the source tree. Prevents: Finding 2: the Quality Gate check already failed on this exact test (1 failed, 7047 passed), but the review's own routed run hid it. The Windows Smoke Test failure was also left undiagnosed.
  • Move the Delegate pinned-dollar tests (EXPECTED_USD_PER_MTOK and its three tests) from tests/test_delegate_agent_registration.py to tests/test_pricing_delegate_pins.py. Change the comment above _DELEGATE_PRICING in pricing.py to name that file as the place where a new row must also add its pinned (input+output, cache_write) figures. Keep the pins hand-written, because an independent second copy is what catches a cache_write/cache_read transposition, so they must not be derived. Why not static: The guard itself is already a correct test that fails when it should. The gap is discoverability: an author who adds a rate row does not know the pins file exists, and keyword or name-based selection misses it. This is a test-organization and author-workflow fix, not a detectable code pattern. Prevents: Finding 2: the two new _DELEGATE_PRICING rows shipped without their pinned entries.
  • Add a scheduled (nightly or weekly) job, separate from make verify, that compares every _PRICING / _DELEGATE_PRICING row with OpenRouter's live /api/v1/models (and the provider cards where OpenRouter has no listing). It reports rows whose rates differ, rows that no source lists, and promotional HAZARD dates that have passed (for example, the gpt-5.6-sol promo 're-check 2026-11-21'). The job opens an issue and does not fail any PR build. Why not static: It needs network access to live third-party price lists, and the answer changes over time without any code change, so it cannot run as a deterministic lint in CI. Prevents: Finding 4 (an unverified provenance claim for new rows) and the reprice drift behind Finding 3: a reprice that reaches only one twin table shows up as a mismatch against the live source.

Top 5 Priority Actions

  1. Make the PR merge-ready: add "gpt-6-1-sol": (2.0 + 10.0, 2.50) and "gpt-6-1-terra": (1.0 + 5.0, 1.25) to EXPECTED_USD_PER_MTOK (tests/test_delegate_agent_registration.py:22-36), to match the new _DELEGATE_PRICING rows at src/coder_eval/pricing.py:54-55; this fixes the failing set-equality test, which is the only failure in the Quality Gate CI check, and gives the new rates an independent check on their dollar figures.
  2. Run make verify on the PR, not a pytest run filtered with -k 'pric or cost', because that filter skipped the failing Delegate test and showed a false green; also find out why the Windows Smoke Test check fails, because nobody has checked it yet.
  3. Stop hand-copying the twin rate rows: build the Delegate keys at src/coder_eval/pricing.py:54-55 from the dotted rows at :129-130, or add a test that the two id families stay equal, so a future reprice cannot update only one table (this already happened once with gpt-5-6-sol vs gpt-5.6-sol).
  4. Fix the provenance claim at src/coder_eval/pricing.py:8 ('every row re-verified 2026-10-07'): check the new GPT-6.1 Terra and Delegate rows against the OpenAI card and update the date, or list them as an exception, as the docstring already does for the Bedrock block.
  5. Move the gpt-6.1-* rows at src/coder_eval/pricing.py:127-132 into their own group after the gpt-6-* rows, and say whether the GPT-6 >272K-input flat-rate caveat also applies to GPT-6.1.

Stats: 0 🔴 · 0 🟠 · 1 🟡 · 3 🔵 across 8 axes reviewed.

…gate's ids

GPT-6.1 Terra has no row, and the Delegate block has no GPT-6.1 row, so
coder-eval books cost None for:
- codex runs on gpt-6.1-terra;
- delegate / delegate-sdk runs on gpt-6-1-terra and gpt-6-1-sol (the
  Delegate host echoes the caller's hyphenated id back as the model).

Rates are OpenAI's standard tier (2026-11-18): Terra $1.00 in / $5.00 out /
$1.25 cache write / $0.05 cache read per MTok. The Delegate gpt-6-1-sol row
copies the existing gpt-6.1-sol row. pricing.generated.ts regenerated with
make pricing-mirror.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ibabencu
ibabencu force-pushed the ionutb-fix-pricing-gpt-6-1-terra branch from 2d88005 to 2c49a9e Compare November 18, 2026 09:11
@ibabencu

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Addressed at 2c49a9e8:

  • Pinned Delegate entries (🟡): fixed in 2d880055. gpt-6-1-sol and gpt-6-1-terra are in EXPECTED_USD_PER_MTOK; Quality Gate and Windows Smoke Test both failed only on that test.
  • Row order (🔵): the gpt-6.1-* rows now come after the gpt-6-* rows. OpenAI bills GPT-6.1 over 272K input at 2x input / 1.5x output, like GPT-6, so the CAVEAT now names both.
  • Provenance (🔵): the docstring now says the GPT-6.1 Terra rows and the Delegate GPT-6.1 rows were added 2026-11-18. I checked them against the OpenAI pricing page that day; I did not re-check the other rows, so the 2026-10-07 claim stays for them.
  • Cost series note: added to the PR description.
  • Underscored ids (fix(pricing): price Delegate's underscored model ids and flag unpriced judge/simulator cost #223): not needed for the Autopilot RPA eval: it passes gpt-6-1-terra, and delegate-stdio echoes that id unchanged. Underscored ids still need fix(pricing): price Delegate's underscored model ids and flag unpriced judge/simulator cost #223.
  • Deriving Delegate twins from the dotted rows (🔵), not in this PR: it changes every existing twin pair, including the deliberately different gpt-5-6-sol, so it is better as its own change.
  • Pinned check for the dotted gpt-6.1-terra row: not added. No dotted row has a pinned dollar test today, and adding that guard for one row is a separate test change.

@ibabencu

Copy link
Copy Markdown
Contributor Author

Two points I missed in my last reply:

  • Runs from before the merge (Downstream consumers): they are not re-run or backfilled. Their run.json keeps cost: None, and the evalboard shows per-bucket USD for them after deploy. This is now in the PR description.
  • Harness & Lint Improvements: not in this PR, because they change the lint rules, CI and the review workflow, not these rows. The CE072 twin-equality rule fits best with the twin-derivation change I deferred above.

@ibabencu
ibabencu requested a review from uipreliga November 18, 2026 12:35

@bai-uipath bai-uipath left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — approving.

  • Rates: GPT-6.1 Terra matches OpenAI's standard tier ($1 in / $5 out, $1.25 for the 30-minute cache write, $0.05 cache read). The two Delegate rows match their dotted twins, and the pinned dollar figures cover both new ids.
  • Rollout: Autopilot keeps booking cost: None until a coder_eval release ships this and its coderEvalVersion pin moves, so cut the patch release right after merge.

@ibabencu
ibabencu merged commit 05bab86 into main Nov 19, 2026
19 of 21 checks passed
@ibabencu
ibabencu deleted the ionutb-fix-pricing-gpt-6-1-terra branch November 19, 2026 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants