Skip to content

Recover bounded capture deadlines and preserve terminal session loss - #2893

Open
miguelg719 wants to merge 42 commits into
mainfrom
evals/consolidation-03-capture-recovery
Open

miguelg719 wants to merge 42 commits into
mainfrom
evals/consolidation-03-capture-recovery

Conversation

@miguelg719

@miguelg719 miguelg719 commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Bounds capture and RPC/batch waits. Two consecutive capture deadlines are recoverable; a third latches terminal session loss, while a successful tool call resets the counter. Late completion cannot overwrite newer snapshot IDs.

Terminal loss rejects queued calls without dispatch. Actions are never automatically replayed. Regressions cover bounded timeout inputs, late completion, counter reset, screenshot transport and first-loss handling.

Preserves provider/session/age/timeout diagnostics through the host, checks screenshot byte limits before decoding, and isolates failures in diagnostic observers. The typed RPC timeout error lives in a focused public module, keeping internal RPC helpers outside the SDK public-field contract.

Validation: 120 focused tests and 47 SDK AST/release guards, SDK/core/eval typechecks, SDK build/publint, and merged ownership/capture regression checks.

Stack base: main. Review this PR relative to its immediate predecessor.

Reviewer entry points:

The CLI startup fix is now included in the parent (#2892), so this PR focuses on capture/RPC deadlines and recovery. Current main is incorporated, preserving both the timeout exports and the runtime-compatibility exports.

Readiness: CI passed on 7b5f94b2de. Human review status is unchanged.

Current open stack: #2893#2894#2902#2903#2889#2904#2905#2906#2907.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 13 files

Architecture diagram
sequenceDiagram
    participant Client as Client/Tool Caller
    participant Facade as StagehandFacadeTools
    participant Queue as Internal Queue
    participant Batch as SDK experimentalBatch
    participant RPC as RPCClient
    participant Executor as Browser Executor
    participant CDP as CDP/Socket Layer

    Note over Client,CDP: Bounded Capture and Session Loss Handling

    Client->>Facade: snapshot() / screenshot() / run()
    
    alt Session already lost
        Facade->>Facade: Check sessionLoss flag
        Facade-->>Client: Throw StagehandFacadeSessionLostError immediately (no dispatch)
    else Healthy session
        Facade->>Queue: Enqueue operation (serialized)
        Queue->>Facade: Execute guarded operation
        
        alt Capture operation (snapshot/screenshot)
            Facade->>Facade: Invalidate previous snapshot state
            Facade->>RPC: Send capture request
            RPC->>Executor: Forward RPC
            alt Response received within 120s
                Executor-->>RPC: Return result
                RPC-->>Facade: Return capture data
                Facade->>Facade: Reset consecutiveDeadlines counter
                Facade->>Facade: Store new snapshot state (valid IDs)
                Facade-->>Client: Return result
            else No response within 120s
                Facade->>Facade: Increment consecutiveDeadlines
                alt Fewer than 3 consecutive timeouts
                    Facade-->>Client: Throw deadline error (recoverable)
                else 3rd consecutive timeout
                    Facade->>Facade: Latch session loss with cause
                    Facade->>Client: Throw StagehandFacadeSessionLostError
                    opt onSessionLost callback configured
                        Facade-->>Facade: Emit telemetry with sanitized error
                    end
                end
            end
        else Batch operation (experimentalBatch)
            Facade->>Batch: runActions() with hydrated selectors
            Batch->>Batch: Calculate client deadline (timeout + 15s grace)
            Batch->>RPC: Send callback batch request with responseTimeoutMs
            
            alt Response received
                Executor-->>RPC: Return batch result
                RPC-->>Batch: Return result
                Batch-->>Facade: Return completed actions
                Facade->>Facade: Reset consecutiveDeadlines
                Facade-->>Client: Return success
            else Client deadline exceeded
                Batch->>Batch: Create RPCResponseTimeoutError
                Batch->>Batch: Wrap in StagehandBatchTimeoutError
                Batch-->>Facade: Throw timeout error
                Facade->>Facade: Identify as session loss cause
                Facade->>Facade: Latch session loss
                Facade-->>Client: Throw StagehandFacadeSessionLostError
            end
        end
    end

    Note over Client,CDP: Late Response Handling
    
    rect rgb(0, 0, 0)
        Note over Facade: After capture deadline fires
        Executor-->>RPC: Late response arrives
        RPC-->>Facade: Return late capture data
        Facade->>Facade: Discard late response (IDs already invalidated)
        Note over Facade: No overwrite of newer snapshot IDs
    end

    Note over Client,CDP: CDP Connection Error Path
    
    CDP->>CDP: Socket closes with code/reason
    CDP->>CDP: Create CDPConnectionClosedError with sanitized details
    CDP-->>Facade: Propagate via RPC rejection
    Facade->>Facade: Extract loss cause (sanitize secrets)
    alt Ordinary tool error (non-loss)
        Facade-->>Client: Re-throw original error (no session loss)
    else Fatal session loss
        Facade->>Facade: Latch session loss permanently
        Facade-->>Client: Throw StagehandFacadeSessionLostError
    end
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/sdk-ts/src/rpcClient.ts Outdated
Comment thread packages/integrations/core/src/facade/stdio-server.ts Outdated
Comment thread packages/integrations/core/src/facade/stdio-server.ts
Comment thread packages/integrations/core/src/facade/screenshot-transport.ts Outdated
Comment thread packages/integrations/core/src/facade/tools.ts Outdated
@changeset-bot

changeset-bot Bot commented Sep 8, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 7b5f94b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages
Name Type
@browserbasehq/stagehand Patch
browse Patch
@browserbasehq/stagehand-integrations Patch
@browserbasehq/stagehand-integrations-example-eve-facade Patch
@browserbasehq/stagehand-integrations-example-pi-facade Patch
@browserbasehq/stagehand-integrations-claude-agent-sdk Patch
@browserbasehq/stagehand-integrations-example-claude-code-facade Patch
@browserbasehq/stagehand-integrations-codex-sdk Patch
@browserbasehq/stagehand-integrations-example-codex-facade Patch
@browserbasehq/stagehand-integrations-cursor-sdk Patch
@browserbasehq/stagehand-integrations-deepagents-sdk Patch
@browserbasehq/stagehand-integrations-eve-sdk Patch
@browserbasehq/stagehand-integrations-fx-sdk Patch
@browserbasehq/stagehand-integrations-mastra-sdk Patch
@browserbasehq/stagehand-integrations-example-mastra-facade Patch
@browserbasehq/stagehand-integrations-pi-sdk Patch
@browserbasehq/stagehand-integrations-example-vercel-ai-facade Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@changeset-bot

changeset-bot Bot commented Sep 8, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8e887b3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
Name Type
@browserbasehq/stagehand Patch
@browserbasehq/stagehand-integrations Patch
@browserbasehq/stagehand-integrations-example-eve-facade Patch
@browserbasehq/stagehand-integrations-example-pi-facade Patch
@browserbasehq/stagehand-integrations-claude-agent-sdk Patch
@browserbasehq/stagehand-integrations-example-claude-code-facade Patch
@browserbasehq/stagehand-integrations-codex-sdk Patch
@browserbasehq/stagehand-integrations-example-codex-facade Patch
@browserbasehq/stagehand-integrations-cursor-sdk Patch
@browserbasehq/stagehand-integrations-deepagents-sdk Patch
@browserbasehq/stagehand-integrations-eve-sdk Patch
@browserbasehq/stagehand-integrations-fx-sdk Patch
@browserbasehq/stagehand-integrations-mastra-sdk Patch
@browserbasehq/stagehand-integrations-example-mastra-facade Patch
@browserbasehq/stagehand-integrations-pi-sdk Patch
@browserbasehq/stagehand-integrations-example-vercel-ai-facade Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Comment thread packages/evals/tui/commands/help.ts
@miguelg719
miguelg719 added this pull request to stack #2932 September 14, 2026 20:04
@miguelg719
miguelg719 removed this pull request from stack #2932 September 17, 2026 07:05
@miguelg719
miguelg719 changed the base branch from evals/consolidation-02-session-ownership to evals/consolidation-11-verifier-results September 17, 2026 08:14
@miguelg719
miguelg719 force-pushed the evals/consolidation-03-capture-recovery branch from 749da11 to d23d757 Compare September 17, 2026 08:14
miguelg719 added a commit that referenced this pull request Sep 17, 2026
Aligns the shared facade’s label matching, strict locator operations and
URL waits with supported Playwright behavior. All consuming harnesses
get these semantics through the same facade.

Scoped role matches now follow shadow hosts instead of relying on DOM
containment. CSS, label, and snapshot-derived queries can access closed
author shadow roots through a narrow extension batch operation.
Temporary CDP references are released best-effort after each query;
fixed typed errors avoid exposing page exception text, and the evaluator
remains off the public SDK type; locator callbacks continue to execute
in the page main world. The Go-embedded extension is rebuilt to match.

Adds local-browser comparisons against native Playwright alongside
deterministic URL-wait tests. Frame composition is isolated in the next
change.

Checkbox probes share the operation timeout; nested role filters are
resolved before matching. The real DOM contract suite is registered in
Turbo.

Validation: 144 shared integration unit tests, five local Chrome DOM
tests, and 19 callback/shadow-reference tests pass. The new
open/closed-root regressions fail against the previous implementation. A
real Stagehand SDK + local extension smoke passes scoped lookup, fill,
click, and main-world callback checks inside a closed root. Extension,
SDK, integration, and eval typechecks pass; extension/integration
builds, formatting, and Go extension drift check pass.

Stack position: 3/11. Base: `evals/consolidation-02-session-ownership`.
Each PR contains the change relative to its immediate predecessor.

Reviewer entry points:

-
[`packages/integrations/core/src/facade/runtime.ts`](https://github.com/browserbase/stagehand/blob/f484d008abb8e0ef5950ffa39bcbee98b84e0fe2/packages/integrations/core/src/facade/runtime.ts)
-
[`packages/integrations/core/integration/facade-dom.test.ts`](https://github.com/browserbase/stagehand/blob/f484d008abb8e0ef5950ffa39bcbee98b84e0fe2/packages/integrations/core/integration/facade-dom.test.ts)
-
[`packages/integrations/core/tests/facade-wait-for-url.test.ts`](https://github.com/browserbase/stagehand/blob/f484d008abb8e0ef5950ffa39bcbee98b84e0fe2/packages/integrations/core/tests/facade-wait-for-url.test.ts)

Reordered stack: #2891#2892#2895#2896#2897#2898#2899#2900#2901#2893#2894.

Readiness: [CI
passed](https://github.com/browserbase/stagehand/actions/runs/35193871165)
on `f484d008ab`. Review threads are triaged and resolved.
miguelg719 added a commit that referenced this pull request Sep 17, 2026
Composes facade locators through same-origin and out-of-process frames
and preserves global snapshot references without duplicating unresolved
frame maps. Adds operator documentation for ownership. Deadline and
disconnect documentation follows in #2893 and #2894.

Also repairs failures exposed by stack CI:

- Background-tab screenshots activate their target and serialize
activation/capture across the browser. Setup and cleanup are serialized
per page, so a stalled page cannot block other tabs. Capture deadlines
stop waiting for stalled CDP responses; late responses cannot re-enter
cleanup. Calls on a page with unfinished timeout recovery fail promptly
until cleanup settles, after which screenshots resume.
- The Vantech Journal scorer identifies the archive’s Load more link by
destination and text instead of an obsolete absolute XPath. Strict DOM
element-identity scoring and the 80% gate remain unchanged.
- CLI test deadlines account for the bounded child processes they
launch, including multi-command tests.

Validation: 24 local screenshot/lifecycle tests and 66 CLI/help tests
passed, along with relevant typechecks and Go extension archive checks.
Tests cover timeout rejection, isolation across tabs, and recovery after
delayed setup/cleanup settles. Live Chrome verification accepts the
recorded correct Vantech observation and rejects an unrelated element.

Stack position: 4/11. Base:
`evals/consolidation-05-locator-wait-contract`.

Reviewer entry points:

- [Frame facade
implementation](https://github.com/browserbase/stagehand/blob/6d79622f5a518b134c1c725e88dd0594d337bff3/packages/integrations/core/src/facade/runtime.ts)
- [Snapshot frame
maps](https://github.com/browserbase/stagehand/blob/6d79622f5a518b134c1c725e88dd0594d337bff3/packages/extension/understudy/a11y/snapshot/capture.ts)
- [Screenshot
lifecycle](https://github.com/browserbase/stagehand/blob/6d79622f5a518b134c1c725e88dd0594d337bff3/packages/extension/understudy/page.ts)
- [Archive
scorer](https://github.com/browserbase/stagehand/blob/6d79622f5a518b134c1c725e88dd0594d337bff3/packages/evals/tasks/bench/observe/observe_vantechjournal.ts)

Reordered stack: #2891#2892#2895#2896#2897#2898#2899#2900#2901#2893#2894.

Readiness: [CI
passed](https://github.com/browserbase/stagehand/actions/runs/35194404031)
on `6d79622f5a`. Review threads are triaged and resolved.
miguelg719 added a commit that referenced this pull request Sep 17, 2026
Bounds stalled DeepAgents setup, stream waits and cleanup. Timeout
settings reject invalid values, preserve the original stop reason and
finish cleanup without losing a completed tool-result batch.

MCP setup, stream and cleanup timeouts preserve AnyIO task ownership;
real in-memory MCP sessions cover the cancel-scope regression and
cleanup deadlines.

Validation: 35 Python runner tests; 29 Python tests and Python
compilation passed, including stalled MCP acquisition/tool discovery,
competing wall/inactivity deadlines, invalid timeout inputs, and
cleanup.

Stack position: 5/11. Base: `evals/consolidation-06-frame-support`. Each
PR contains the change relative to its immediate predecessor.

Reviewer entry points:

-
[`packages/integrations/deepagents/runner/run_eval.py`](https://github.com/browserbase/stagehand/blob/d1d6e48fd946158a042501aad592eb21b01e8481/packages/integrations/deepagents/runner/run_eval.py)
-
[`packages/integrations/deepagents/runner/tests/test_run_eval.py`](https://github.com/browserbase/stagehand/blob/d1d6e48fd946158a042501aad592eb21b01e8481/packages/integrations/deepagents/runner/tests/test_run_eval.py)


Follow-up review validation: 40 Python tests pass. Timeout attribution
now distinguishes the runner’s expired deadline from inner MCP/provider
`TimeoutError`, preserves subsecond diagnostic values, and records the
selected watchdog before cancellation cleanup. Same-task MCP/AnyIO
ownership is preserved. The alleged early-expiry arithmetic was not
confirmed; a separate slow-cleanup attribution regression is covered.

Reordered stack: #2891#2892#2895#2896#2897#2898#2899#2900#2901#2893#2894.

Readiness: [CI
passed](https://github.com/browserbase/stagehand/actions/runs/35195059814)
on `d1d6e48fd9`. Review threads are triaged and resolved.
miguelg719 added a commit that referenced this pull request Sep 17, 2026
Routes direct Mastra and DeepAgents model requests through the intended
native provider while respecting explicit gateway routing. Missing
provider configuration remains an actionable startup error.

Missing xAI credentials fail before client construction, preventing
fallback to another provider’s ambient key.

Validation: 41 Python runner tests including synthetic cross-provider
credential regressions; 32 Python tests, 18 Mastra routing/session
tests, five build tasks, frozen dependency installation, and Mastra
typecheck passed.

Stack position: 6/11. Base:
`evals/consolidation-07-deepagents-lifecycle`. Each PR contains the
change relative to its immediate predecessor.

Reviewer entry points:

-
[`packages/integrations/mastra-sdk/src/session.ts`](https://github.com/browserbase/stagehand/blob/5caa682dca8269b7772a71ad530f4a9ccea87956/packages/integrations/mastra-sdk/src/session.ts)
-
[`packages/integrations/mastra-sdk/tests/model-routing.test.ts`](https://github.com/browserbase/stagehand/blob/5caa682dca8269b7772a71ad530f4a9ccea87956/packages/integrations/mastra-sdk/tests/model-routing.test.ts)
-
[`packages/integrations/deepagents/runner/run_eval.py`](https://github.com/browserbase/stagehand/blob/5caa682dca8269b7772a71ad530f4a9ccea87956/packages/integrations/deepagents/runner/run_eval.py)

Reordered stack: #2891#2892#2895#2896#2897#2898#2899#2900#2901#2893#2894.

Readiness: [CI
passed](https://github.com/browserbase/stagehand/actions/runs/35195660877)
on `5caa682dca`. Review threads are triaged and resolved.
miguelg719 added a commit that referenced this pull request Sep 17, 2026
Bounds retained SDK event logs while keeping actionable diagnostics and
trajectory evidence. Pi screenshot compaction and its Buffer-aware
trajectory consumer land together, preventing intermediate screenshot
loss.

Validation: 118 SDK/event tests, five Pi adapter/pipeline tests, 12
build tasks, and SDK/core/eval typechecks passed. The pipeline test
imports the actual built Pi compactor and checks retained image bytes
for compacted Buffer and historical base64 events.

Stack position: 7/11. Base: `evals/consolidation-08-native-routing`.
Each PR contains the change relative to its immediate predecessor.

Reviewer entry points:

-
[`packages/integrations/core/src/harness/eventLog.ts`](https://github.com/browserbase/stagehand/blob/bfc240ec68d562c190885bc676c57ac6360f835b/packages/integrations/core/src/harness/eventLog.ts)
-
[`packages/integrations/pi-sdk/src/session.ts`](https://github.com/browserbase/stagehand/blob/bfc240ec68d562c190885bc676c57ac6360f835b/packages/integrations/pi-sdk/src/session.ts)
-
[`packages/evals/framework/harnesses/piAdapter.ts`](https://github.com/browserbase/stagehand/blob/bfc240ec68d562c190885bc676c57ac6360f835b/packages/evals/framework/harnesses/piAdapter.ts)

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Bounds SDK event logs and retained event payloads across Claude, Codex,
DeepAgents, Eve, Fx, Mastra, and Pi. Previously, stream noise and
repeated payloads stayed visible; now stream fragments are dropped,
routine events move to debug, and failures stay at level 1 without
changing model-facing results or trajectory evidence.

- Adds a shared event classifier that drops deltas and bare lifecycle
markers and keeps SDK-specific failures visible.
- Retains Pi screenshots once as `Buffer` values, drops duplicate image
payloads from logs and non-assistant messages, and records omission
markers over 8 MiB per image or 64 MiB per run. The Pi adapter accepts
both `Buffer` and legacy base64 image blocks.
- Compacts Mastra `step-finish` and `finish` events to keep usage and
trajectory data without repeated request bodies.
- Redacts credentials before clipping event details.

<sup>Written for commit bfc240e.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2899?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->

Reordered stack: #2891#2892#2895#2896#2897#2898#2899#2900#2901#2893#2894.

Readiness: [CI
passed](https://github.com/browserbase/stagehand/actions/runs/35196277234)
on `bfc240ec68`. Review threads are triaged and resolved.
miguelg719 added a commit that referenced this pull request Sep 17, 2026
Distinguishes unreported usage from observed zero across SDK boundaries
and recovers interrupted Codex usage without replacing authoritative
completed-turn counts. Eval accounting consumes this metadata in the
later adoption PR.

Validation: 167 SDK tests, 38 Python tests, 12 build tasks, and
SDK/core/eval typechecks passed. Authoritative completed Codex usage
takes precedence over interrupted-rollout recovery.

Stack position: 8/11. Base: `evals/consolidation-09-event-retention`.
Each PR contains the change relative to its immediate predecessor.

Reviewer entry points:

-
[`packages/integrations/codex-sdk/src/session.ts`](https://github.com/browserbase/stagehand/blob/b60164d6a5155900c83880a04c0767cab915c562/packages/integrations/codex-sdk/src/session.ts)
-
[`packages/integrations/deepagents/runner/run_eval.py`](https://github.com/browserbase/stagehand/blob/b60164d6a5155900c83880a04c0767cab915c562/packages/integrations/deepagents/runner/run_eval.py)
-
[`packages/integrations/mastra-sdk/src/session.ts`](https://github.com/browserbase/stagehand/blob/b60164d6a5155900c83880a04c0767cab915c562/packages/integrations/mastra-sdk/src/session.ts)

Reordered stack: #2891#2892#2895#2896#2897#2898#2899#2900#2901#2893#2894.

Readiness: [CI
passed](https://github.com/browserbase/stagehand/actions/runs/35197720052)
on `b60164d6a5`. Review threads are triaged and resolved.
miguelg719 added a commit that referenced this pull request Sep 17, 2026
…#2901)

Separates verified task completion from execution health. Verification
errors fail closed, preserve the agent report separately, and retain
captured evidence and raw uncertainty for audit. A supported completion
can still pass after a late browser disconnect.

Preserves judge verdicts beside deterministic evidence gates; missing
criterion scores remain in the denominator. Blocker wording and
execution error alone do not erase a completed rubric requirement.
Numeric grounding remains advisory by default. It includes captured step
and terminal accessibility text, records terminal matches separately,
and treats unknown-page text as untrusted. Batch summaries expose
explicit zero-browser passes; the configured verifiability gate rejects
them. Producer-dependent usage, cost, facade and lifecycle fields are
documented as conditional.

Offline verification uses the same model factory as live grading: CLI
override, then EVAL_VERIFIER_MODEL, then the shared default. Thrown
verification errors and uncertain judgments are preserved as explicit
ungraded artifacts without top-level outcome/process scores, including
live persistence and tracing; the offline command exits unsuccessfully.
Trace labels are restricted to filename components, and failed writes
emit sanitized warnings.

Judge policy still needs confirmation: this proposed change selects
google/gemini-3.5-flash by default, while current main uses 2.5 Flash.
The completed 108-case compatibility gate explicitly used 2.5 Flash and
does not validate the proposed 3.5 default.

Validation: 104 focused verifier/CLI/span/persistence/summary tests at
this boundary, eval typecheck and scoped formatting; 13 assertions
reproduced the new review gaps before fixes. No frozen 108-case
compatibility inputs changed. No evaluator-v4 implementation.

Stack position: 9/11. Base: `evals/consolidation-10-usage-provenance`.
Each PR contains the change relative to its immediate predecessor.

Reviewer entry points:

-
[`packages/evals/framework/verifierAdapter.ts`](https://github.com/browserbase/stagehand/blob/af1cf703925647db722d166cafc75bcfbe3f36df/packages/evals/framework/verifierAdapter.ts)
-
[`packages/evals/framework/verifierGates.ts`](https://github.com/browserbase/stagehand/blob/af1cf703925647db722d166cafc75bcfbe3f36df/packages/evals/framework/verifierGates.ts)
-
[`packages/evals/docs/verifier-gates.md`](https://github.com/browserbase/stagehand/blob/af1cf703925647db722d166cafc75bcfbe3f36df/packages/evals/docs/verifier-gates.md)

Reordered stack: #2891#2892#2895#2896#2897#2898#2899#2900#2901#2893#2894.

Readiness: reordered head `af1cf70392` is awaiting fresh CI and review
triage.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants