Skip to content

Refine Scenario configuration and sizing - #2377

Open
Roman Lutz (romanlutz) wants to merge 18 commits into
copilot/romanlutz-scenario-pr-05-queue-upstreamfrom
copilot/romanlutz-scenario-pr-06-sizing-upstream
Open

Refine Scenario configuration and sizing#2377
Roman Lutz (romanlutz) wants to merge 18 commits into
copilot/romanlutz-scenario-pr-05-queue-upstreamfrom
copilot/romanlutz-scenario-pr-06-sizing-upstream

Conversation

@romanlutz

@romanlutz Roman Lutz (romanlutz) commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds named technique sets, Adaptive attempt caps, searchable datasets, per-dataset limits, compatibility-aware identities, and exact/ranged backend-owned run arithmetic. Default estimates use bounded concurrency with execution-only timeouts.

What this layer adds

  • Named technique sets plus explicit Custom selection.
  • Adaptive per-objective attempt limits and baseline comparison controls.
  • Searchable dataset selection with bounded per-dataset caps and provenance.
  • Compatibility-aware identities and unsupported Jailbreak-combination filtering.
  • Backend-owned exact/ranged run-size arithmetic shared by preview and launch.
  • Bounded concurrent default estimation with queue time excluded from execution timeout.

Preview

Authoritative Scenario configuration and run-size preview

Stack

This is 6 of 7 in native GitHub Stack #2390 and depends on #2376.

Layer Scope Pull request
1 Progress foundation #2372
2 Catalog and launch #2373
3 Live progress #2374
4 Run history #2375
5 FIFO scheduling #2376
6 Configuration sizing (this PR) #2377
7 Result details #2378

See the full implementation, screenshots, walkthrough, validation, and operational notes.

Validation

  • Backend: 15,212 passed, 120 skipped.
  • Frontend: 68/68 suites, 1,286/1,286 tests; lint, TypeScript, and production build passed.
  • Mock E2E, retries disabled: 91 passed, 0 failed, 0 skipped.
  • Seeded E2E, retries disabled: 62 passed, 0 failed, 0 skipped.
  • All configured pre-commit hooks passed; distinguished-engineer review approved the final stack.

Compatibility notes

Unsupported Jailbreak v4 technique combinations are rejected, v3 persisted Jailbreak runs intentionally do not resume as v4, and configuration-aware Adaptive identities may regroup historical analytics.

This was referenced Aug 12, 2026
Open
Merged
Merged
Merged
Open
Merged
@romanlutz
Roman Lutz (romanlutz) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from 4b81ccf to f316420 Compare August 21, 2026 07:01
@romanlutz
Roman Lutz (romanlutz) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from f316420 to 1128c9b Compare August 21, 2026 14:31
@romanlutz
Roman Lutz (romanlutz) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from 1128c9b to 62a0d01 Compare August 25, 2026 02:25
@romanlutz
Roman Lutz (romanlutz) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from 62a0d01 to 1ea0f0a Compare August 25, 2026 03:45
@romanlutz
Roman Lutz (romanlutz) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from 1ea0f0a to 4a2fbc7 Compare August 25, 2026 06:24
@richlundeen
Richard Lundeen (richlundeen) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from 4a2fbc7 to 77c7986 Compare August 31, 2026 17:23
@richlundeen
Richard Lundeen (richlundeen) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from 77c7986 to 664c8d8 Compare August 31, 2026 18:14
@richlundeen
Richard Lundeen (richlundeen) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from 664c8d8 to 2c83ce3 Compare September 1, 2026 21:32
@richlundeen
Richard Lundeen (richlundeen) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from 2c83ce3 to 521dfee Compare September 1, 2026 21:34
@behnam-o
Behnam (behnam-o) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from 521dfee to e69bc09 Compare September 2, 2026 17:50
@behnam-o
Behnam (behnam-o) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from e69bc09 to 88f163f Compare September 2, 2026 18:10
Copilot AI and others added 7 commits September 3, 2026 13:30
@richlundeen
FEAT: Add scenario configuration sizing
5338137 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@richlundeen
FIX: Cache conditional scenario estimates
e0797eb 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen
FIX: Preserve rich scenario sizing after restack
0bdd7c2 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen
FIX: Clarify conditional scenario estimates
7a70111 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@romanlutz @richlundeen
TEST: Keep auth bootstrap responsive in slow API E2E
1c482f4 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@romanlutz @richlundeen
STYLE: Remove stale jailbreak test import
25c0f70 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Behnam Ousat @richlundeen
FIX: Restore scenario dataset sizing contract
1cacfc9 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 49641a15-4119-43bd-becc-9c43eef407f2
@richlundeen
Richard Lundeen (richlundeen) force-pushed the copilot/romanlutz-scenario-pr-06-sizing-upstream branch from 88f163f to 1cacfc9 Compare September 3, 2026 20:33
@romanlutz
Merge parent updates into scenario sizing
6452761 
Integrate the updated queue layer and repair scenario sizing, configuration, and frontend validation against the current parent architecture.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@romanlutz
TEST: Configure seeded scorer target
8077fa1 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@romanlutz
Merge corrected #2376 parent into #2377
c513eab 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@romanlutz
TEST: Stabilize catalog timeout on Windows
3ae1ba7 
Exercise the async timeout boundary directly so slow Windows thread-pool startup cannot make the test wait forever before the mocked estimate begins.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@romanlutz
TEST: Ignore concurrent catalog call order
20c624c 
Verify that pagination constructs exactly the requested scenarios without assuming thread-pool scheduling order.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@romanlutz
Roman Lutz (romanlutz) marked this pull request as ready for review September 14, 2026 17:54
@romanlutz
Merge corrected #2376 parent into #2377
641397a 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@romanlutz
Initialize memory in scenario sizing test
b5134fe 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@romanlutz
Merge final #2376 parent into #2377
d2523e5 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@romanlutz
Merge latest #2376 parent into #2377
ae392e6 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@romanlutz
Merge repaired #2376 parent into #2377
68fc64a 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@romanlutz
Merge final PR #2376 parent into scenario sizing layer
0856515 
Merge e07691f while preserving PR #2377 scenario configuration and executable sizing behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@romanlutz