fix(benchmarking): make the SWE-Perf ActorTemplate deployable - #1889
Open
Haowei Cai (Roy) (roycaihw) wants to merge 1 commit into
Open
Haowei Cai (Roy) (roycaihw) wants to merge 1 commit into
Haowei Cai (Roy) (roycaihw) wants to merge 1 commit into
Conversation
The swebench-astropy-7336 ActorTemplate cannot be deployed as shipped. Two independent problems: 1. `image: ""` — the field is empty with a comment saying to fill it in before deploying. There is no way to supply it without editing a tracked file, so every run dirties the tree (and the substrate version label derived from it). 2. The command points at `/opt/swebench/replay.py` and `/opt/swebench/astropy_trace.json`. Neither path exists. The image generator in https://github.com/gke-labs/sweperf writes the replay server to `/replay.py` and the trajectory to `/trace.json` at the image root — which is also the image's own ENTRYPOINT. Nothing in that repo produces `/opt/swebench/`, so the container exits immediately with "can't open file '/opt/swebench/replay.py'". Supply the image through a SWEPERF_IMAGE environment variable, substituted by workloads/deploy.sh alongside the existing template variables, and correct the two paths. Also documents the minimum sweperf commit (c30c0d6, 2026-09-21). The async HTTP execution API this benchmark drives landed 2026-08-31 and `execution_duration_ms` on 2026-09-21; images built earlier ship a replay.py with no HTTP server and fail on the client's first `GET /status`. Verified against a rebuilt image: a 21-step / 4-cycle run completes with 0 failures.
Author
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #1694
The
swebench-astropy-7336ActorTemplate can't be deployed as shipped:image: ""must be hand-edited into a tracked file before every run./opt/swebench/replay.pyand/opt/swebench/astropy_trace.json, which the sweperf image generator neverproduces. It writes
/replay.pyand/trace.jsonat the image root (alsothe image's ENTRYPOINT), so the container exits immediately.
This PR:
SWEPERF_IMAGEenv var, substituted byworkloads/deploy.shlike the other template variables;c30c0d6), since older images have noHTTP server and fail on the first
GET /status.Usage:
Verified on GKE with both gVisor and microVM worker pools: 30-minute sweperf
runs at 1 user, all 4 cycles, 0 failures.
bash -n; exercised end-to-end as above)