Repository navigation
fix(codex): give the app-server the custom provider so openai-codex 0.167+ accepts turns - #259
Conversation
…odex 0.167+ accepts turns openai-codex 0.167.x rejects turn/start with JSON-RPC -32600 when only the thread names the CODEX_BASE_URL provider. Build the provider once (_custom_provider) and send it to the app-server as --config overrides (serialised with tomli_w) as well as to the thread. The default endpoint still starts the client with config=None. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…00 symptom Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Type the provider as a recursive _TomlValue alias (str leaves only), so pyright rejects a non-str leaf before _toml_inline crashes start(). - Warn (do not raise) when CODEX_BASE_URL carries userinfo, because the URL reaches the app-server argv; the warning names neither URL nor credential. - Debug log reads the resolved base URL and api-version, not the dict shape. - Docstrings state the contract only; the base-URL env fact lives in one place. - Unit tests: a named Codex spy fixture replaces the duplicated setup; add a URL-credential warning test. - Binary test: parametrize app-server-only and app-server-and-thread, assert the Bearer key arrives, surface communicate() errors, and reap a client that hangs in Codex() construction on timeout. - Notes: drop the plan decision id and change-history wording. - CLAUDE.md: list tomli-w as a runtime dependency. - harness-candidates: latest-openai-codex CI job; no c/ plan refs in docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Claude finished @CarlesUIPath's task in 1m 10s —— View job Code Review
|
…orker CodeQL flags the BaseException handler. Exception covers every error the scenario can raise; a failure still surfaces in the assertion message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uipreliga
left a comment
There was a problem hiding this comment.
Review: coder_eval — pr:259 (7 files) axis:1,2,3,4,5,6,7,8
Scope: pr:259 (7 files) axis:1,2,3,4,5,6,7,8 · branch fix/codex-provider-app-server-config (pr-259) · ad435e1 · 2026-11-13T16:03Z · workflow variant
Change class: complex — changes how the Codex app-server process is configured at start (new --config overrides, TOML serialisation, credential-in-URL warning) on the custom-endpoint path
The codebase is healthy (9.9/10): no finding is above low severity, and no finding can change a task's score or final_status for identical agent output. The real risk is in the new CODEX_BASE_URL provider path in src/coder_eval/agents/codex_agent.py, which puts a URL-embedded credential on the app-server argv, only warns about it, and fails open on a URL it cannot parse. Fix that path in a single small change and the review is clean.
Summary
| Axis | Score | 🔴 | 🟠 | 🟡 | 🔵 | Top Issue |
|---|---|---|---|---|---|---|
| 1. Code Quality & Style | 9.7 / 10 | 0 | 0 | 0 | 3 | CODEX_BASE_URL credential warning sits in the pure _custom_provider() builder: it fires at every provider build (start and each thread), uses the module logger without the task prefix, and the test does not pin the count |
| 2. Type Safety | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 3. Test Health | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 4. Security | 9.9 / 10 | 0 | 0 | 0 | 1 | Credential embedded in CODEX_BASE_URL is now placed on the app-server argv; the guard only warns, and fails open on an unparseable URL |
| 5. Architecture & Design | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 6. Error Handling & Resilience | 9.9 / 10 | 0 | 0 | 0 | 1 | _url_has_userinfo swallows ValueError on an unparseable CODEX_BASE_URL while the other provider seam raises. The test pins tolerance on one seam only |
| 7. API Surface & Maintainability | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 8. Evaluation Harness Quality | 10 / 10 | 0 | 0 | 0 | 0 | — |
Overall Score: 9.9 / 10 · Weakest Axis: Code Quality & Style at 9.7 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 0 · 🔵 5 across 8 axes.
Blockers
None.
Non-blocking, but please consider before merge
None.
Nits
- [Axis 1] CODEX_BASE_URL credential warning sits in the pure _custom_provider() builder: it fires at every provider build (start and each thread), uses the module logger without the task prefix, and the test does not pin the count (
src/coder_eval/agents/codex_agent.py:1529) —_custom_provider()builds a data dict and also logs a warning (if _url_has_userinfo(base_url): logger.warning("CODEX_BASE_URL embeds a credential, which is visible on the codex app-server " ...)). It has two callers._app_server_config_overrides()(line 1592) calls it once in start()._build_thread_options()(line 1776) calls it each timeself.thread is Nonein communicate() (line 1224-1225), which includes every retry after a crash resets the thread. So one misconfiguration logs the same warning several times, and the thread-side copies are wrong because that path does not put the URL on the app-server command line. It is also a@staticmethod, so it logs through the modulelogger. The neighbouring warning at line 1780 usesself._log(thePrefixedAdapterfrom line 1061), so in a parallel run this message has no instance prefix. Fix: keep_custom_provider()pure. Move the userinfo check into_app_server_config_overrides(), the only place where the URL reaches argv, and log withself._log.warning(...). - [Axis 1] Routing debug log reads CODEX_BASE_URL/CODEX_API_VERSION and parses the URL again instead of using the provider definition already built (an open dict, not typed fields) (
src/coder_eval/agents/codex_agent.py:1785) —_custom_provider()has already resolved CODEX_BASE_URL and CODEX_API_VERSION intoprovider, but lines 1785-1786 call both resolvers again:host = urlparse(self._resolve_base_url() or "").hostname or "(unknown)"andapi_version = "set" if self._resolve_api_version() else "unset". Theor ""fallback can never run, becauseprovider is not Nonemeans the base URL is set. It is a second source for the same values in one block. Read them fromprovider["base_url"]and"query_params" in provider, or have_custom_providerreturn the parsed host as well. This also removes one of the twourlparse/urlsplitparses of the same URL in this module (see the next finding). - [Axis 1] Unneeded None-vs-CodexConfig branch, and a new test that pins it (
src/coder_eval/agents/codex_agent.py:1094) —config = (CodexConfig(env=env_override, config_overrides=config_overrides) if env_override or config_overrides else None)branches for no reason. In the installed SDK,Codex.__init__passesconfigtoCodexClient, which doesself.config = config or CodexConfig().CodexConfig's defaults areenv=Noneandconfig_overrides=(), so the two branches behave the same. The PR widens this condition from one input to two, and addstest_start_default_endpoint_passes_no_config(tests/test_codex_agent.py:304,assert codex_spy.config is None), which pins the implementation detail and not the behaviour. KISS: always buildCodexConfig(env=env_override, config_overrides=config_overrides)and assert on its fields in the test. - [Axis 4] Credential embedded in CODEX_BASE_URL is now placed on the app-server argv; the guard only warns, and fails open on an unparseable URL (
src/coder_eval/agents/codex_agent.py:1529) — Before this PR, the base URL went to the app-server only in the thread-start JSON-RPC payload over stdio. Now_app_server_config_overrides()(line 1597:f"model_providers.{_CUSTOM_PROVIDER_ID}={_toml_inline(provider)}") puts the full URL on thecodex --config ... app-servercommand line. The SDK runs it withsubprocess.Popen(args, ...)(an argument list, no shell, so there is no injection). On a multi-user Linux host, other local users can read/proc/<pid>/cmdlineandpsoutput for the life of the app-server. The SDK also keeps the argv inCheckoutCapabilities.command, and aCalledProcessErrorraised from that argv carries it in__cause__(_runtime_requirements.py). This is the same leak shape as the existing harness-candidates entry. The PR detects userinfo (if _url_has_userinfo(base_url):->logger.warning("CODEX_BASE_URL embeds a credential, ...")) but lets the run continue. reqwest honours URL userinfo as basic auth, so an operator who relies on it gets a credential that is newly visible on argv. The detector also fails open: lines 406-407except ValueError:\n return Falsereturn False for a URL such ashttp://u:p@[::1/v1, which urlsplit rejects with 'Invalid IPv6 URL'. That URL gets no warning but still goes onto argv. Fix: when userinfo is present, either raise a clear configuration error (fail loud), or remove the userinfo from the value passed in--configand keep it only in the thread payload. Make the detector treat an unparseable URL as unsafe (return True), or use a raw'@' in netloccheck. The impact is low: the URL is set by a trusted operator, the docs (docs/agents/CODEX.md:65) call this configuration unsupported, and it requires a local co-tenant. CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:L/I:N/A:N - [Axis 6] _url_has_userinfo swallows ValueError on an unparseable CODEX_BASE_URL while the other provider seam raises. The test pins tolerance on one seam only (
src/coder_eval/agents/codex_agent.py:406) — Lines 403-408:try:\n parts = urlsplit(url)\n except ValueError:\n return False. This is the only place the PR parses the URL before the app-server starts, and it turns a malformed required value into 'no credential'. Two results follow. (a) A URL such ashttps://u:p@[bad/puts its credential on the app-server argv and logs no warning. I checked this: urlsplit raises 'Invalid IPv6 URL', so the guard is skipped in exactly the malformed case. (b) start() succeeds with a URL that is not valid, and the first communicate() then raises ValueError at line 1785 (host = urlparse(self._resolve_base_url() or "").hostname or "(unknown)"). The orchestrator records that as a per-turn crash and retries it, so a deterministic configuration error is reported as an agent crash. The test case("http://[::1/v1", False)makes this behaviour part of the contract. Recommendation: in _custom_provider(), fail loud on an unparseable CODEX_BASE_URL. Raise a ValueError that names the variable but not its value, so start() reports 'Failed to initialize Codex client'. Do not return False. Then update the test case to expect the raise.
What's Missing
Parallel paths:
- 🔵 get_environment_info() (codex_agent.py:1378-1386) is a third consumer of the custom-provider routing, and the PR did not move it to _custom_provider(). It still reads CODEX_BASE_URL and CODEX_API_VERSION itself, parses the URL again with urlparse, and hard-codes _CODEX_WIRE_API. The new note (.claude/notes/agents.md § Codex custom provider on the app-server) says the seams 'both read the one helper, so they cannot disagree', but that is true for only two of the three paths. Two effects: the run artifact's routing record can drift from the provider the app-server actually got, and a URL that does not parse raises ValueError here as well. Fix: build the run-artifact keys from the _custom_provider() dict (host from provider['base_url'], api_version from 'query_params' in provider). (trigger: src/coder_eval/agents/codex_agent.py) (restates: Axis 1: Routing debug log reads CODEX_BASE_URL/CODEX_API_VERSION again instead of using the provider definition)
Tests:
- 🔵 No unit test covers start() in the production Azure shape, with CODEX_API_KEY and CODEX_BASE_URL both set. In that case CodexConfig must carry both env (the key) and config_overrides (the provider). test_start_passes_provider_overrides_to_codex_config sets only the base URL and asserts config.env is None. A regression that loses one of the two fields when both are present would not fail any unit test. Add a case that asserts config.env contains CODEX_API_KEY and that config.config_overrides == agent._app_server_config_overrides(). (trigger: tests/test_codex_agent.py)
- 🔵 The regression test for the -32600 defect never runs against a binary in the failing range (cli-bin 0.167.0-0.167.2). In CI, tests/test_codex_provider_binary.py and the codex-live job run only the cli-bin 0.156.1 that the lock pins. That binary accepts thread-only routing, so CI shows that the app-server seam routes a turn but cannot show that a 0.167.x binary accepts it. The PR also states that the binary test has not run on Windows CI, which is the platform of the affected nightly. The PR defers a floating-version job in .claude/harness-candidates.md. Until that job exists, a manual live check is the only gate when the binary pin changes. That manual step is documented in the notes, but the PR checklist does not enforce it. (trigger: tests/test_codex_provider_binary.py)
Daily/nightly:
- 🔵 The PR states the nightly impact: Windows pip installs of the [codex] extra resolve to 0.167.x, and the coder_eval_uipath#275 pin can be lifted after a release. But the run record does not include the openai-codex or openai-codex-cli-bin version. get_environment_info records host, wire_api and api_version, but no binary version. This defect and its fix both depend on the binary version, and the nightly's version floats once the pin is lifted. So a later -32600 or provider failure in a nightly run.json cannot be traced to the binary that produced it. Fix: record the resolved cli-bin version (for example importlib.metadata.version('openai-codex-cli-bin')) in environment_info before the pin is lifted. (trigger: src/coder_eval/agents/codex_agent.py)
Harness & Lint Improvements
Static checks (lint / type):
- [ce-lint] Delete before guard, then add CE069 'URL parsing goes through one helper'. First add one stdlib-only module, for example
src/coder_eval/url_utils.py. It owns three functions.url_host(url) -> str | Nonereturns None when the URL cannot be parsed.url_has_userinfo(url) -> boolfails closed: an unparseable URL returns True.require_http_url(var_name, value) -> SplitResultraises a ValueError that names the variable, does not echo the value, and does not chainfrom exc. Move the five current parse sites to these helpers: config.py:174, orchestrator.py:1771, codex_agent.py:1329 and :1705, delegate_agent.py:209 (_url_host), plus the new_url_has_userinfofrom the PR. Then addtests/lint/rules/ce069_url_parse_via_helper.py. It is an AST rule that flags anyurlparse/urlsplitcall, and anyfrom urllib.parse import urlparse|urlsplit, insrc/coder_eval/outsideurl_utils.py. Theargv_match.pysidecar stays exempt because it is stdlib-only (CE057). Wire the rule intests/lint/runner.pyand give it a.claude/notes/lint-rules.mdentry. CE067 is retired and CE068 is the highest id, so CE069 is the next free number. Prevents: Finding 5 and the fail-open part of finding 4:_url_has_userinforeturns False on a URL that will not parse (http://u:p@[::1/v1), while the other seam raises later in communicate(). The shared helper gives one fail-closed / fail-loud contract for every caller. Finding 2: the routing debug log at codex_agent.py:1785 parses CODEX_BASE_URL a second time even though the provider is already built. With the rule in place, that second parse must go throughurl_host()and is easy to see. It also closes the harness-candidates.md:1028 leak, whereraise ... from excafter urlsplit put a URL password in the traceback, for every URL parse site. - [ce-lint] Add CE070 'an agent with a prefixed logger does not log through the module logger'. In
src/coder_eval/agents/*.py, find each class body that assignsself._log = PrefixedAdapter(...). Flag anylogger.<debug|info|warning|error|exception|critical>(call in a method of that class, @staticmethod and @classmethod included. Module-level helper functions are not checked, and# noqa: CE070suppresses a call that is on purpose. Current hits are claude_code_agent.py (6) and codex_agent.py (2). Per CLAUDE.md, set the rule's cap below the current count, or fix the hits in the same change. Slot:tests/lint/rules/ce070_agent_logs_via_prefixed_adapter.pyplus runner.py. Prevents: The missing-prefix part of finding 1: the CODEX_BASE_URL credential warning comes from@staticmethod _custom_provider()through the modulelogger. In a parallel run its lines have no task prefix, unlike the warning next to it (line 1780), which usesself._log. The rule also pushes such side-effect logging out of static builders and into instance methods. That is the same move the finding asks for: put the warning in_app_server_config_overrides(). - [pyright] Make the Codex custom provider definition a
TypedDict, for exampleclass CodexProviderDef(TypedDict): name: str; base_url: str; wire_api: str; env_key: NotRequired[str]; query_params: NotRequired[dict[str, str]]. Make_custom_provider()returnCodexProviderDef | None, not an opendict[str, Any]. Pyright standard mode then type-checks everyprovider[...]read, which makes readingprovider["base_url"]/"query_params" in providerthe natural, typed source. Calling_resolve_base_url()/_resolve_api_version()again becomes plainly redundant. Prevents: Finding 2: the debug log at codex_agent.py:1785-1786 calls both env resolvers again, and itsor ""fallback can never run. This happens because the built provider is an untyped dict, so reading its fields has no type support.
Harness improvements (not statically reachable):
- Add an argv secret-canary test, parametrized over every built-in AgentKind. Set each credential/endpoint env var the agent reads (
CODEX_BASE_URL=https://canaryuser:canarypass@host/v1, API keys =sk-canary-...) to a sentinel value. Then build the agent's subprocess argv (for Codex:_app_server_config_overrides()plus the SDK--configexpansion) and assert that no sentinel substring appears in argv. If the agent must reject the configuration, assert a clear config error that does not echo the value. Add it next to the existing agent conformance tests so a new agent gets it automatically. Why not static: A secret moves from an env read through_resolve_base_url(),_custom_provider()and_toml_inline()into aconfig_overridestuple, which the SDK passes toPopen. Catching this needs interprocedural data-flow from env sources to argv sinks. harness-candidates.md:1028 records that this data-flow is not cheap for the CE framework, and bandit/CodeQL do not model the codex SDK argv as a sink. Prevents: Finding 4: a credential embedded in CODEX_BASE_URL is now put on thecodex --config ... app-servercommand line, wherepsand/proc/<pid>/cmdlineshow it. Today the guard only logs a warning, and it fails open on an unparseable URL. - Add a lifecycle-repetition test for one-time configuration warnings. Drive one agent through start(), one communicate(), a forced crash that resets
self.thread, and a second communicate(). Assert with caplog an EXACT count (== 1) of each configuration warning, and assert that the record carries the instance prefix. Pin a count, notany(...), for every configuration-warning test in tests/test_codex_agent.py, so that a warning moved into a per-thread builder fails loudly. Why not static: Whether a warning repeats depends on how often a builder runs at runtime, across start, thread creation and crash-retry resets. No AST pattern tells a once-per-run call site from a per-thread call site. Prevents: Finding 1:_custom_provider()is called from both_app_server_config_overrides()and_build_thread_options(), so the credential warning fires at start and again for every thread and retry. The test at tests/test_codex_agent.py:444 did not pin the count, so the repetition was not caught. - Add a review-checklist note, and a matching
.claude/notes/agents.mdentry, for SDK-default mirroring: when an agent passesX if cond else Noneinto an SDK constructor, check whether the SDK already doesx or X()with equal defaults. If it does, always build X, and assert on its fields rather than onis None. Back this with one Codex test that asserts the effectiveCodexConfigfields (env, config_overrides) on the default endpoint. Why not static: To see thatNoneandCodexConfig()behave the same, you must read the installed third-party SDK (CodexClient.__init__:config or CodexConfig()) and compare its field defaults. That is semantic, version-dependent knowledge, not a pattern in this repo's AST. Prevents: Finding 3: the needlessCodexConfig(...) if env_override or config_overrides else Nonebranch at codex_agent.py:1094, and the new testtest_start_default_endpoint_passes_no_config(tests/test_codex_agent.py:304), which pins the implementation detail and not the behaviour.
Top 5 Priority Actions
- No finding can change a task's score or final_status for identical agent output; the most important fix is security: when CODEX_BASE_URL contains userinfo, src/coder_eval/agents/codex_agent.py:1597 (_app_server_config_overrides) must raise a clear configuration error or remove the userinfo from the --config argv value, because the credential is now visible in /proc//cmdline and in SDK CalledProcessError chains.
- Change _url_has_userinfo at src/coder_eval/agents/codex_agent.py:403-408 so that it does not return False on ValueError, and make _custom_provider() raise a ValueError that names CODEX_BASE_URL but not its value; then a malformed URL fails start() once and is not retried as an agent crash at line 1785 or put on argv without a warning, and the test case ("http://[::1/v1", False) at tests/test_codex_agent.py:432 must expect the raise.
- Move the userinfo warning out of the pure builder _custom_provider() at src/coder_eval/agents/codex_agent.py:1529 into _app_server_config_overrides() (line 1592), log it with self._log.warning so that it has the task prefix, and pin a count of one in tests/test_codex_agent.py:444, so that thread builds and retries (line 1776) do not repeat it.
- In the routing debug log at src/coder_eval/agents/codex_agent.py:1785-1786, read the host and api-version state from the provider dict that is already built, and do not resolve CODEX_BASE_URL/CODEX_API_VERSION and parse the URL a second time.
- Simplify src/coder_eval/agents/codex_agent.py:1094 so that it always builds CodexConfig(env=env_override, config_overrides=config_overrides), because the SDK treats None and default CodexConfig the same; change test_start_default_endpoint_passes_no_config (tests/test_codex_agent.py:304) to assert the config fields, not
config is None.
Stats: 0 🔴 · 0 🟠 · 0 🟡 · 5 🔵 across 8 axes reviewed.
uipreliga
left a comment
There was a problem hiding this comment.
Fix identified issues & 🚢
…ew nits - A credential in CODEX_BASE_URL, or a URL that does not parse, now fails start() with a non-retryable AgentConfigError before the app-server spawns, instead of a warning that left the credential on argv. The check runs once, on the argv seam only; the message never echoes the value. - _custom_provider() is a pure builder again. - start() always builds CodexConfig: the SDK treats None and a default CodexConfig the same. - The routing debug log reads the provider dict, not the env again. - Tests: Azure shape (key env plus overrides), default endpoint config fields, rejection before spawn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pip-audit fails the Quality Gate on opentelemetry-api 1.45.1 (ReDoS in TraceContextTextMapPropagator.extract, fixed in 1.45.2). opentelemetry-sdk pins the api in lockstep, so api, sdk, instrumentation and semantic-conventions move together. Lock-only change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Purpose
On openai-codex 0.167.x, every Codex turn on a custom endpoint (
CODEX_BASE_URL) fails before the agent runs a command, with JSON-RPC-32600 "This chat uses custom, but your current provider is openai". Affected: pip/wheel installs of the[codex]extra, which resolve 0.167.x (the Windows nightly; UiPath/coder_eval_uipath#275 pinned 0.156.1 as a stopgap). Not affected: uv-synced repo and CI installs, which[tool.uv]pins to cli-bin 0.156.1.Cause
coder_eval selected the custom provider only on the thread (
thread_startoptions). The app-server process started with its default config, where the provider isopenai. The 0.167.x binary checks onturn/startthat the thread's provider matches the app-server's provider, so it rejects every turn.turn/starthas no provider field, so the turn cannot fix it. Bisect: 0.156.1 OK, 0.167.0 / 0.167.2 fail, 0.168.0 / 0.169.0 OK. The check is inopenai-codex-cli-bin; the SDK API is the same in all versions.Fix
The app-server now gets the same provider as the thread.
_custom_provider()builds the definition once.start()passes it as two--configoverrides throughCodexConfig(config_overrides=...), serialised withtomli_w, and the thread options use the same dict, so the two cannot drift. The default endpoint does not change: no overrides, nomodel_provider. The API key stays in the env (env_key), never in argv. ACODEX_BASE_URLthat embeds a credential or does not parse failsstart()with a non-retryableAgentConfigError, because the URL is on the app-server command line.Also in this PR:
uv.lockbumps opentelemetry to 1.45.2 (CVE-2026-126360), because thepip-auditgate failed on 1.45.1. This is a lock-only change that also affectsmain.Tests
start()wiring (base URL only, and key plus base URL).Live check
CodexAgentturns pass on 0.156.1, 0.167.2 and 0.169.0. A control run on 0.167.2 with the fix disabled gave-32600. Codex runs on gpt-5.6-luna, gpt-5.6-terra and gpt-6.1-sol, on 0.167.2: 6/6 SUCCESS.Windows: ADO nightly dry run 14251616 installed coder_eval from this PR with
openai-codex==0.167.2(cli-bin 0.167.2) on the Windows slice. All Windows tasks and the codex harness smoke succeeded, with 0-32600.Limits
Questions for reviewers
The source change is +82/−30 (
codex_agent.py). Most of the PR (+417/−31 in total) is tests and docs:tests/test_codex_provider_binary.py(+138): the real-binary test against a local stub.tests/test_codex_agent.py(+153): unit tests..claude/notes/agents.md,docs/agents/CODEX.md,.claude/harness-candidates.md,CLAUDE.md(+45).Should we simplify these to keep the PR minimal? Options:
🤖 Generated with Claude Code