Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .claude/harness-candidates.md
Original file line number Diff line number Diff line change
Expand Up @@ -1028,3 +1028,16 @@ re-derive from scratch.
- [ ] `raise X(...) from exc` where `exc` came from parsing an env/URL value leaks the value through `__cause__` in a traceback (urlsplit put a URL password in its port error). Needs data-flow from env reads; not cheap — caught in the network: llm_only Phase 3 review.
- [ ] `await asyncio.to_thread(subprocess.run, ...)` that CREATES a resource inside a try whose `finally` removes it: a cancel leaves the worker thread running, so teardown races the create and leaks it (fixed in `isolation/egress.py` by `_run_to_completion`). Detecting "creates a resource" is not mechanical — caught in the network: llm_only Phase 4 review.
- [ ] The single-`asyncio.shield` join in `fs_permissions.set_permissions` does not survive a SECOND cancel of the caller; `isolation/egress._run_to_completion` loops on the shield. Promote that helper to a shared module and use it in both places — caught in the network: llm_only Phase 4 review.

## From 2026-11-13 Codex app-server provider review

- [ ] A CI job that runs `tests/test_codex_provider_binary.py` against the latest
openai-codex release, in a venv built with `uv pip install --no-config`. CI runs only the
lock-pinned cli-bin 0.156.1, so the 0.167.x `-32600` rejection reached pip installs of the
`[codex]` extra and no check saw it. Deferred because a floating dependency makes the job
flaky by design; it fits a nightly job, not the PR gate — caught in the code review.
- [ ] A doc-surface check that checked-in Markdown (`.claude/notes/`, `.claude/shared/`,
`docs/`, `plugins/coder-eval/`) never references the git-ignored `c/` plan tree
(`c/YYYY-MM-DD-*.md`) or a plan decision id (`decision D<n>`). A notes section cited
"PI decision D1", which a reader of the repo cannot open. Deferred because there was one
hit and review caught it; promote it on a second hit — caught in the code review.
28 changes: 28 additions & 0 deletions .claude/notes/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -753,6 +753,34 @@ profiles and loses the prepend again. Nested zsh keeps it, because `ZDOTDIR` sta
exported. No-op on Windows, where Codex shells through PowerShell (`-NoProfile`) or
`cmd /c`, neither of which re-sources a profile chain.

## Codex custom provider on the app-server

With `CODEX_BASE_URL` set, openai-codex 0.167.0-0.167.2 rejects every turn with JSON-RPC
`-32600 "This chat uses custom, but your current provider is openai"` when only the thread
names the custom provider. The check is in the bundled `openai-codex-cli-bin` binary; the
Python SDK surface (`CodexConfig.config_overrides`) is identical from 0.155.1 to 0.169.0.
An offline probe against a local stub showed thread-only routing works on 0.156.1, 0.168.0
and 0.169.0 and fails on 0.167.2; app-server plus thread works on all of them.

`_custom_provider()` builds the provider once. The thread options embed it, and `start()`
passes it to the app-server as two `--config` overrides (`model_provider` and the
`model_providers.custom` inline table). Both seams select the provider: the thread seam is
the path that every other binary version accepts, and both read the one helper, so they
cannot disagree. The TOML value is written with `tomli_w`, a runtime dependency, with quoted
keys, so `api-version` needs no bare-key rules and `"`, `\`, control characters and U+007F
are escaped. `json.dumps` is not valid TOML for U+007F or surrogate escapes. The
`_TomlValue` alias admits only string leaves, because `_toml_inline` renders only those.

The base URL is on the app-server command line. It is not a secret and also travels in the
thread-start payload; the key stays in the env (`env_key`). A credential embedded in the URL
or a URL that does not parse fails `start()` with a non-retryable `AgentConfigError`
before the app-server spawns, because the URL would go on its command line (visible in
`ps` and `/proc/<pid>/cmdline`). The message names the variable, never the value. A warning
was rejected: it leaves the credential on argv. CI runs only the lock-pinned binary,
so a binary bump needs a manual live check: a fresh venv per version built with
`uv pip install --no-config` (without it the repo's `[tool.uv]` override pins cli-bin
0.156.1 for every version), then `tests/test_codex_provider_binary.py` and one real turn.

## Reaping the CLI harnesses

`opencode run` leaves a local server child alive after the CLI exits, and it INHERITS the
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@ Tasks are YAML. Full reference: [Task Definition Guide](docs/TASK_DEFINITION_GUI
## Dependencies

**Runtime**: pydantic, pydantic-settings, pyyaml, typer, rich, python-dotenv, anthropic,
claude-agent-sdk, anyio, radon, tqdm, jmespath, jsonschema
claude-agent-sdk, anyio, radon, tqdm, jmespath, jsonschema, tomli-w

**Runtime (optional, `[uipath]` extra)**: uipath — for local sandbox parity with tasks
that invoke `uv run uipath eval ...`. Base installs run end-to-end without it;
Expand Down
2 changes: 2 additions & 0 deletions docs/agents/CODEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ CODEX_API_KEY=<azure-openai-key>
```
This registers a custom Codex model provider (`base_url` + `env_key=CODEX_API_KEY` + `query_params={api-version}` + `wire_api=responses`). The Codex CLI only supports the Responses wire API (it rejects `wire_api=chat` as "no longer supported"), so the protocol is fixed. If your Azure deployment requires the key in an `api-key` header rather than `Authorization: Bearer`, that needs an additional provider `http_headers`/`env_http_headers` entry — open an issue if you hit that.

coder_eval selects this provider on the Codex app-server (`--config`) and on each thread. openai-codex 0.167.x requires the app-server half: without it, every turn fails with JSON-RPC `-32600 "This chat uses custom, but your current provider is openai"`. The URL is on the app-server command line, so coder_eval rejects a `CODEX_BASE_URL` that embeds a credential (`https://user:pass@…`) with a configuration error. Put the key only in `CODEX_API_KEY`.

## Usage

### Command Line
Expand Down
112 changes: 82 additions & 30 deletions src/coder_eval/agents/codex_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,17 @@
from pathlib import Path
from types import SimpleNamespace
from typing import Any, ClassVar, Literal, NamedTuple
from urllib.parse import urlparse
from urllib.parse import urlparse, urlsplit

import tomli_w

from coder_eval.agent import Agent, AgentState
from coder_eval.agents._logging import PrefixedAdapter, log_raw_sdk_event
from coder_eval.agents.registry import AgentRegistry
from coder_eval.agents.watchdog import ThreadedWatchdog
from coder_eval.config import settings
from coder_eval.errors import (
AgentConfigError,
AgentCrashError,
TurnTimeoutError,
truncate_crash_message,
Expand Down Expand Up @@ -81,8 +84,8 @@
# Rationale: .claude/notes/agents.md § Codex runs full-access on every permission mode
_CODEX_APPROVAL_MODE = "deny_all"

# Provider id registered in thread config when CODEX_BASE_URL routes to a
# custom endpoint.
# Provider id selected on the app-server and the thread when CODEX_BASE_URL
# routes to a custom endpoint.
_CUSTOM_PROVIDER_ID = "custom"

# apply_patch statuses meaning the patch did not apply. Used ONLY to classify the
Expand Down Expand Up @@ -387,6 +390,32 @@ def _message_uncached_input(m: AssistantMessage) -> int:
# supported"), so this is a fixed constant, not an operator knob.
_CODEX_WIRE_API = "responses"

type _TomlValue = str | dict[str, _TomlValue]


def _toml_inline(value: _TomlValue) -> str:
"""Render a string or a (nested) string table as one TOML inline value, for ``codex --config``."""
if isinstance(value, str):
return tomli_w.dumps({"v": value}).removeprefix("v = ").rstrip("\n")
return "{" + ", ".join(f"{_toml_inline(k)} = {_toml_inline(v)}" for k, v in value.items()) + "}"


def _require_credential_free_base_url(url: str) -> None:
"""Raise ``AgentConfigError`` when CODEX_BASE_URL does not parse or embeds a credential.

The message names the variable, never its value: the URL goes on the app-server command line.
"""
try:
parts = urlsplit(url)
except ValueError:
raise AgentConfigError("CODEX_BASE_URL is not a valid URL.") from None
if parts.username is not None or parts.password is not None:
raise AgentConfigError(
"CODEX_BASE_URL embeds a credential, which would be visible on the codex app-server "
+ "command line; put the key only in CODEX_API_KEY."
)


# Login-shell profiles generated into the per-task HOME. ``.bash_profile`` is
# what ``bash -l`` reads and ``.profile`` covers sh/dash; the zsh trio covers
# macOS, where ``.zshenv`` runs in EVERY zsh, ``.zprofile`` in login shells
Expand Down Expand Up @@ -1064,13 +1093,12 @@ async def start(
self._env_path_prepend = list(env_path_prepend or [])
self._setup_login_shell_home()
self._state = AgentState.WORKING
config_overrides = self._app_server_config_overrides()

try:
from openai_codex import Codex, CodexConfig

# Build CodexConfig with environment variables for custom API configuration
env_override = self._build_codex_env()
config = CodexConfig(env=env_override) if env_override else None
config = CodexConfig(env=self._build_codex_env(), config_overrides=config_overrides)

# Close any prior client FIRST: start() runs through
# execute_with_retry, so a retried start would otherwise orphan the
Expand Down Expand Up @@ -1488,6 +1516,29 @@ def _resolve_api_version() -> str | None:
"""
return os.getenv("CODEX_API_VERSION") or None

@staticmethod
def _custom_provider() -> dict[str, _TomlValue] | None:
"""The CODEX_BASE_URL model provider definition, or None on the default endpoint.

The binary has no base-URL env var, so the provider is defined in config and
selected by id; ``env_key`` names the key's variable, so the key itself never
appears here. On Azure, CODEX_API_VERSION adds the required ``api-version`` query
param and CODEX_MODEL is the deployment name.
"""
base_url = CodexAgent._resolve_base_url()
if not base_url:
return None
provider: dict[str, _TomlValue] = {
"name": "Custom",
"base_url": base_url,
"env_key": "CODEX_API_KEY",
"wire_api": _CODEX_WIRE_API,
}
api_version = CodexAgent._resolve_api_version()
if api_version:
provider["query_params"] = {"api-version": api_version}
return provider

def _effective_model(self) -> str | None:
"""Resolve the model: task/CLI ``agent.model`` wins, else CODEX_MODEL.

Expand All @@ -1499,8 +1550,7 @@ def _build_codex_env(self) -> dict[str, str] | None:
"""Build the environment passed to the Codex app-server.

Carries ``CODEX_API_KEY`` and, when the sandbox resolved mock dirs, a PATH
with those prepended. The base URL is NOT an env var the binary honors —
it goes through the model provider config instead.
with those prepended.

The SDK merges this PARTIAL dict over ``os.environ`` (normalizing the PATH
key case-insensitively), so a full PATH value here safely replaces the
Expand Down Expand Up @@ -1529,6 +1579,25 @@ def _build_codex_env(self) -> dict[str, str] | None:
env["CODEX_HOME"] = str(codex_home)
return env if env else None

def _app_server_config_overrides(self) -> tuple[str, ...]:
"""``--config`` overrides that select the custom provider on the app-server itself.

The definition is the one the thread options carry; empty on the default endpoint.

Raises:
AgentConfigError: CODEX_BASE_URL does not parse or embeds a credential.

Rationale: .claude/notes/agents.md § Codex custom provider on the app-server
"""
provider = self._custom_provider()
if provider is None:
return ()
_require_credential_free_base_url(str(provider["base_url"]))
return (
f"model_provider={_toml_inline(_CUSTOM_PROVIDER_ID)}",
f"model_providers.{_CUSTOM_PROVIDER_ID}={_toml_inline(provider)}",
)

@staticmethod
def _login_shell_profiles_supported() -> bool:
"""Whether to generate login-shell profile shims (POSIX shells only)."""
Expand Down Expand Up @@ -1705,35 +1774,18 @@ def _build_thread_options(self) -> dict[str, Any]:
+ "do not rely on it as a security boundary."
)

# The codex binary has no base-URL env var: a model provider must be
# defined in config and selected, with env_key naming the key's variable.
# For Azure, CODEX_API_VERSION adds the required query param and
# CODEX_MODEL is the deployment name.
base_url = self._resolve_base_url()
if base_url:
provider = self._custom_provider()
if provider is not None:
options["model_provider"] = _CUSTOM_PROVIDER_ID
if not effective_model:
self._log.warning(
"CODEX_BASE_URL is set but no model resolved (agent.model / CODEX_MODEL) "
+ "— the provider may reject the request."
)
provider: dict[str, Any] = {
"name": "Custom",
"base_url": base_url,
"env_key": "CODEX_API_KEY",
# Fixed, not configurable: the pinned binary rejects
# `wire_api = "chat"` as "no longer supported".
"wire_api": _CODEX_WIRE_API,
}
api_version = self._resolve_api_version()
if api_version:
# Azure requires ?api-version=... on every request.
provider["query_params"] = {"api-version": api_version}
tool_config["model_providers"] = {_CUSTOM_PROVIDER_ID: provider}
self._log.debug(
f"Codex routed via custom provider (host={urlparse(base_url).hostname or '(unknown)'}, "
+ f"api_version={'set' if api_version else 'unset'})"
)
host = urlparse(str(provider["base_url"])).hostname or "(unknown)"
api_version_state = "set" if "query_params" in provider else "unset"
self._log.debug(f"Codex routed via custom provider (host={host}, api_version={api_version_state})")

if self.config.context_window is not None:
# Codex then auto-compacts at 90% of this window.
Expand Down
Loading
Loading