Replace the vitest evals with the simulation suite - #63
Merged
Conversation
Agent behavior is now covered by scenarios.yaml, so the in-process evals and the CI job that ran them on every push are gone. One eval stays commented out in src/agent.test.ts as an example of the in-process framework for turn-level checks.
bcherry
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Now that
scenarios.yamlcovers agent behavior end to end (#62), the per-push vitest evals duplicate that coverage. This removes them and thetestCI job that ran them, and keeps one eval commented out insrc/agent.test.tsas an example of the in-process framework for turn-level checks that don't need a live session.src/agent.test.ts: everything commented out except a header explaining why; theoffers assistanceeval remains as the example.vitestand thetestscripts stay so uncommenting it just works..github/workflows/tests.yml: thetestjob is gone, leaving the typecheck/lint/format job. An empty suite makes vitest exit non-zero, so the job would fail on every PR otherwise. If theTest (Node.js 24)check is required in branch protection it will need to be dropped.simulations.ymlheader comment now lead with simulations and point at the commented example instead ofpnpm test. Also dropped the README step about a "Check files not tracked in git" step that no longer existed intests.yml.Verified locally:
pnpm run typecheck,pnpm run lint,pnpm run format:check.