Seed-complexity blocks: shell stub and opencode full-harness seeds for Terminal-Bench and GAIA

[varunursekar]

Summary

Adds the two new seed families the EACL rebuttal grid runs on, plus the compiler and rescore changes they needed. Every cell in the TB-stub, TB-opencode and GAIA-opencode blocks was launched from this branch (VERO_SRC_OVERRIDE to this worktree's vero/src).

Seeds

• terminal-bench/baseline/target-shell + build.shell.yaml: non-functional shell stub mirroring the GAIA stub (measured baseline 0.0).
• terminal-bench/baseline/target-opencode, gaia/baseline/target-opencode + build.opencode.yaml: opencode vendored as a pinned submodule (anomalyco/opencode @ e03db9bc), thin harbor OpenCode subclass, host-side Bun build cached per tree hash, VERO_OPENCODE_BINARY override. GAIA e2e smoke variant.
• TB opencode seed sends the vendor-prefixed wire id (xai/grok-build-0.1) through a custom opencode-target provider; the bare id was rejected upstream with 401 and scored 0.0. VERO_OPENCODE_WIRE_MODEL_ID overrides.
• opencode target calls go to the public upstream from inside the task container (task_services_use_upstream).

Compiler

• Materialise git submodules when snapshotting agent_repo (they came out as empty dirs from git archive).
• Allow in-tree relative symlinks in the snapshot (opencode links web assets across packages).

Build timeouts (opencode builds)

• build_timeout_seconds 5400: the image build/start regularly took 45+ min.
• Sandbox idle timeout 4 h: the verifier's test evaluation of an opencode candidate outlives the 1 h window and the sandbox was gone before harbor downloaded the verifier dir.

rescore_candidate

• --build-file for routed builds, --agent-env (harbor --ae) passthrough, --agent-setup-timeout-multiplier (the 143 MB opencode binary upload from the host exceeded harbor's 360 s setup budget at concurrency 24), resolve the harbor_requirement placeholder, skip node_modules/dist/.turbo when copying the seed tree.
• GAIA budget-blind build variant and GAIA shell routed build for transfer passes.

Notes for review

• No secrets: only secrets.env.example files are tracked; added lines were scanned for key patterns.
• The branch is in live use by running cells. Please merge without rewriting its history; if history needs cleaning, do it on a copy.

Test plan

• vero/tests shell-variant config test parametrized over gaia and terminal-bench
• GAIA opencode e2e smoke (build.opencode.e2e.yaml) on Modal
• TB opencode fixed-seed cells start at dev 0.53 to 0.76 (previously 0.0 with the bare wire id)
• Full-block runs: TB stub 27 cells, TB opencode and GAIA opencode blocks in progress

🤖 Generated with Claude Code

Confidence Score: 0/5

The PR is not safe to merge. Four open issues can expose credentials, run edited code on the host, or build the wrong source.

Findings

1.  task services use upstream puts the real OPENAI API KEY in the task environment. This target reads that key from optimizer-editable code instead of using the scoped gateway token. A candidate can send direct requests or print the key, while request logs only catch this later. Keep editable code on a reachable scoped gateway instead of giving it the provider key. ▶
2.  build binary runs optimizer-edited install hooks and script/build.ts on the evaluation host. Both commands inherit all of os.environ . This happens before the binary enters the task container, so task isolation does not protect the host. Build candidate source inside a sandbox and pass only the values it needs. ▶
3.  ls-tree returns the gitlink SHA, but this code keeps only the path and archives the checkout's current HEAD . If an initialized submodule is stale or was moved by a developer, the build silently snapshots that other commit. Archive the gitlink SHA or reject a checkout whose HEAD does not match it. ▶
4.  tree hash skips each symlink, including its path and target. The compiler now keeps safe symlinks because opencode uses them as build inputs. A candidate can change such a link without changing the cache key, and build binary can return the previous binary. Hash the link path and os.readlink path value. ▶
5.  BUN VERSION affects only the fallback download URL. If bun is already on PATH or under ~/.bun , bun returns it without checking the version. Check bun --version and fetch the pinned release when it differs. ▶

Fix with agent prompt

### Issue 1
harness-opt-bench/terminal-bench/baseline/build.opencode.yaml:118-123
`task_services_use_upstream` puts the real `OPENAI_API_KEY` in the task environment. This target reads that key from optimizer-editable code instead of using the scoped gateway token. A candidate can send direct requests or print the key, while request logs only catch this later. Keep editable code on a reachable scoped gateway instead of giving it the provider key.

The GAIA opencode builds use the same setting.

**How this was verified:** `HarborBackend` copies the host provider key into the task environment, and the editable wrapper reads that exact variable.

### Issue 2
harness-opt-bench/terminal-bench/baseline/target-opencode/src/terminal_bench_agent/_build.py:104-115
`build_binary` runs optimizer-edited install hooks and `script/build.ts` on the evaluation host. Both commands inherit all of `os.environ`. This happens before the binary enters the task container, so task isolation does not protect the host. Build candidate source inside a sandbox and pass only the values it needs.

The GAIA copy has the same code.

**How this was verified:** Agent setup invokes both candidate-controlled Bun commands on the host with the full parent environment before uploading the binary.

### Issue 3
vero/src/vero/harbor/build/compiler.py:187-199
`ls-tree` returns the gitlink SHA, but this code keeps only the path and archives the checkout's current `HEAD`. If an initialized submodule is stale or was moved by a developer, the build silently snapshots that other commit. Archive the gitlink SHA or reject a checkout whose `HEAD` does not match it.

### Issue 4
harness-opt-bench/terminal-bench/baseline/target-opencode/src/terminal_bench_agent/_build.py:34-49
`tree_hash` skips each symlink, including its path and target. The compiler now keeps safe symlinks because opencode uses them as build inputs. A candidate can change such a link without changing the cache key, and `build_binary` can return the previous binary. Hash the link path and `os.readlink(path)` value.

### Issue 5
harness-opt-bench/terminal-bench/baseline/target-opencode/src/terminal_bench_agent/_build.py:52-61
`BUN_VERSION` affects only the fallback download URL. If `bun` is already on `PATH` or under `~/.bun`, `_bun` returns it without checking the version. Check `bun --version` and fetch the pinned release when it differs.

The GAIA copy has the same code.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

VeRO adds shell and full-opencode starting points for Terminal-Bench and GAIA, along with compiler and rescoring support for running them. The new seeds compare building an agent from an empty shell with improving a complete coding harness on the same benchmark setup.

• Adds zero-score shell seeds and build variants for both benchmarks.
• Adds vendored, editable opencode seeds with cached host-side builds and binary uploads.
• Teaches snapshots to include git submodules and safe in-tree symlinks.
• Extends rescoring for alternate builds, agent environment values, setup time, and large seed trees.

Diagram

sequenceDiagram
    participant Compiler
    participant Git as Git checkout
    participant Host as Evaluation host
    participant Task as Task container
    participant API as Public upstream

    Compiler->>Git: Archive the seed at HEAD
    Compiler->>Git: Find submodule gitlinks
    Git-->>Compiler: Archive each submodule at its checked-out HEAD
    Compiler-->>Host: Return baseline with vendored opencode
    Host->>Host: Hash the opencode tree
    alt Cached binary exists
        Host->>Host: Reuse cached binary
    else No cached binary
        Host->>Host: Run bun install and build.ts
        Host->>Host: Cache the binary
    end
    Host->>Task: Upload the opencode binary
    Host->>Task: Pass upstream API settings
    Task->>API: Send model requests directly
    API-->>Task: Return model responses

Loading

_{Reviews (2) · Last reviewed commit: "CI: check out the vendored opencode subm..."}

[greptile-apps]

task_services_use_upstream puts the real OPENAI_API_KEY in the task environment. This target reads that key from optimizer-editable code instead of using the scoped gateway token. A candidate can send direct requests or print the key, while request logs only catch this later. Keep editable code on a reachable scoped gateway instead of giving it the provider key.

The GAIA opencode builds use the same setting.

How this was verified: HarborBackend copies the host provider key into the task environment, and the editable wrapper reads that exact variable.

Prompt To Fix With AI

This is a comment left during a code review.
Path: harness-opt-bench/terminal-bench/baseline/build.opencode.yaml
Line: 118-123

Comment:
`task_services_use_upstream` puts the real `OPENAI_API_KEY` in the task environment. This target reads that key from optimizer-editable code instead of using the scoped gateway token. A candidate can send direct requests or print the key, while request logs only catch this later. Keep editable code on a reachable scoped gateway instead of giving it the provider key.

The GAIA opencode builds use the same setting.

**How this was verified:** `HarborBackend` copies the host provider key into the task environment, and the editable wrapper reads that exact variable.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

build_binary runs optimizer-edited install hooks and script/build.ts on the evaluation host. Both commands inherit all of os.environ. This happens before the binary enters the task container, so task isolation does not protect the host. Build candidate source inside a sandbox and pass only the values it needs.

The GAIA copy has the same code.

How this was verified: Agent setup invokes both candidate-controlled Bun commands on the host with the full parent environment before uploading the binary.

Prompt To Fix With AI

This is a comment left during a code review.
Path: harness-opt-bench/terminal-bench/baseline/target-opencode/src/terminal_bench_agent/_build.py
Line: 104-115

Comment:
`build_binary` runs optimizer-edited install hooks and `script/build.ts` on the evaluation host. Both commands inherit all of `os.environ`. This happens before the binary enters the task container, so task isolation does not protect the host. Build candidate source inside a sandbox and pass only the values it needs.

The GAIA copy has the same code.

**How this was verified:** Agent setup invokes both candidate-controlled Bun commands on the host with the full parent environment before uploading the binary.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

ls-tree returns the gitlink SHA, but this code keeps only the path and archives the checkout's current HEAD. If an initialized submodule is stale or was moved by a developer, the build silently snapshots that other commit. Archive the gitlink SHA or reject a checkout whose HEAD does not match it.

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/src/vero/harbor/build/compiler.py
Line: 187-199

Comment:
`ls-tree` returns the gitlink SHA, but this code keeps only the path and archives the checkout's current `HEAD`. If an initialized submodule is stale or was moved by a developer, the build silently snapshots that other commit. Archive the gitlink SHA or reject a checkout whose `HEAD` does not match it.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

tree_hash skips each symlink, including its path and target. The compiler now keeps safe symlinks because opencode uses them as build inputs. A candidate can change such a link without changing the cache key, and build_binary can return the previous binary. Hash the link path and os.readlink(path) value.

Prompt To Fix With AI

This is a comment left during a code review.
Path: harness-opt-bench/terminal-bench/baseline/target-opencode/src/terminal_bench_agent/_build.py
Line: 34-49

Comment:
`tree_hash` skips each symlink, including its path and target. The compiler now keeps safe symlinks because opencode uses them as build inputs. A candidate can change such a link without changing the cache key, and `build_binary` can return the previous binary. Hash the link path and `os.readlink(path)` value.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

BUN_VERSION affects only the fallback download URL. If bun is already on PATH or under ~/.bun, _bun returns it without checking the version. Check bun --version and fetch the pinned release when it differs.

The GAIA copy has the same code.

Prompt To Fix With AI

This is a comment left during a code review.
Path: harness-opt-bench/terminal-bench/baseline/target-opencode/src/terminal_bench_agent/_build.py
Line: 52-61

Comment:
`BUN_VERSION` affects only the fallback download URL. If `bun` is already on `PATH` or under `~/.bun`, `_bun` returns it without checking the version. Check `bun --version` and fetch the pinned release when it differs.

The GAIA copy has the same code.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.