Recover bounded capture deadlines and preserve terminal session loss - #2893
Open
miguelg719 wants to merge 42 commits into
Open
miguelg719 wants to merge 42 commits into
miguelg719 wants to merge 42 commits into
Conversation
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 13 files
Architecture diagram
sequenceDiagram
participant Client as Client/Tool Caller
participant Facade as StagehandFacadeTools
participant Queue as Internal Queue
participant Batch as SDK experimentalBatch
participant RPC as RPCClient
participant Executor as Browser Executor
participant CDP as CDP/Socket Layer
Note over Client,CDP: Bounded Capture and Session Loss Handling
Client->>Facade: snapshot() / screenshot() / run()
alt Session already lost
Facade->>Facade: Check sessionLoss flag
Facade-->>Client: Throw StagehandFacadeSessionLostError immediately (no dispatch)
else Healthy session
Facade->>Queue: Enqueue operation (serialized)
Queue->>Facade: Execute guarded operation
alt Capture operation (snapshot/screenshot)
Facade->>Facade: Invalidate previous snapshot state
Facade->>RPC: Send capture request
RPC->>Executor: Forward RPC
alt Response received within 120s
Executor-->>RPC: Return result
RPC-->>Facade: Return capture data
Facade->>Facade: Reset consecutiveDeadlines counter
Facade->>Facade: Store new snapshot state (valid IDs)
Facade-->>Client: Return result
else No response within 120s
Facade->>Facade: Increment consecutiveDeadlines
alt Fewer than 3 consecutive timeouts
Facade-->>Client: Throw deadline error (recoverable)
else 3rd consecutive timeout
Facade->>Facade: Latch session loss with cause
Facade->>Client: Throw StagehandFacadeSessionLostError
opt onSessionLost callback configured
Facade-->>Facade: Emit telemetry with sanitized error
end
end
end
else Batch operation (experimentalBatch)
Facade->>Batch: runActions() with hydrated selectors
Batch->>Batch: Calculate client deadline (timeout + 15s grace)
Batch->>RPC: Send callback batch request with responseTimeoutMs
alt Response received
Executor-->>RPC: Return batch result
RPC-->>Batch: Return result
Batch-->>Facade: Return completed actions
Facade->>Facade: Reset consecutiveDeadlines
Facade-->>Client: Return success
else Client deadline exceeded
Batch->>Batch: Create RPCResponseTimeoutError
Batch->>Batch: Wrap in StagehandBatchTimeoutError
Batch-->>Facade: Throw timeout error
Facade->>Facade: Identify as session loss cause
Facade->>Facade: Latch session loss
Facade-->>Client: Throw StagehandFacadeSessionLostError
end
end
end
Note over Client,CDP: Late Response Handling
rect rgb(0, 0, 0)
Note over Facade: After capture deadline fires
Executor-->>RPC: Late response arrives
RPC-->>Facade: Return late capture data
Facade->>Facade: Discard late response (IDs already invalidated)
Note over Facade: No overwrite of newer snapshot IDs
end
Note over Client,CDP: CDP Connection Error Path
CDP->>CDP: Socket closes with code/reason
CDP->>CDP: Create CDPConnectionClosedError with sanitized details
CDP-->>Facade: Propagate via RPC rejection
Facade->>Facade: Extract loss cause (sanitize secrets)
alt Ordinary tool error (non-loss)
Facade-->>Client: Re-throw original error (no session loss)
else Fatal session loss
Facade->>Facade: Latch session loss permanently
Facade-->>Client: Throw StagehandFacadeSessionLostError
end
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
🦋 Changeset detectedLatest commit: 7b5f94b The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🦋 Changeset detectedLatest commit: 8e887b3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…ion-02-session-ownership
…ion-02-session-ownership
…ion-02-session-ownership
akeimach
approved these changes
Sep 9, 2026
miguelg719
added this pull request to stack #2932
September 14, 2026 20:04
miguelg719
removed this pull request from stack #2932
September 17, 2026 07:05
This was referenced Sep 17, 2026
miguelg719
changed the base branch from
evals/consolidation-02-session-ownership
to
evals/consolidation-11-verifier-results
September 17, 2026 08:14
miguelg719
force-pushed
the
evals/consolidation-03-capture-recovery
branch
from
September 17, 2026 08:14
749da11 to
d23d757
Compare
miguelg719
added a commit
that referenced
this pull request
Sep 17, 2026
Aligns the shared facade’s label matching, strict locator operations and URL waits with supported Playwright behavior. All consuming harnesses get these semantics through the same facade. Scoped role matches now follow shadow hosts instead of relying on DOM containment. CSS, label, and snapshot-derived queries can access closed author shadow roots through a narrow extension batch operation. Temporary CDP references are released best-effort after each query; fixed typed errors avoid exposing page exception text, and the evaluator remains off the public SDK type; locator callbacks continue to execute in the page main world. The Go-embedded extension is rebuilt to match. Adds local-browser comparisons against native Playwright alongside deterministic URL-wait tests. Frame composition is isolated in the next change. Checkbox probes share the operation timeout; nested role filters are resolved before matching. The real DOM contract suite is registered in Turbo. Validation: 144 shared integration unit tests, five local Chrome DOM tests, and 19 callback/shadow-reference tests pass. The new open/closed-root regressions fail against the previous implementation. A real Stagehand SDK + local extension smoke passes scoped lookup, fill, click, and main-world callback checks inside a closed root. Extension, SDK, integration, and eval typechecks pass; extension/integration builds, formatting, and Go extension drift check pass. Stack position: 3/11. Base: `evals/consolidation-02-session-ownership`. Each PR contains the change relative to its immediate predecessor. Reviewer entry points: - [`packages/integrations/core/src/facade/runtime.ts`](https://github.com/browserbase/stagehand/blob/f484d008abb8e0ef5950ffa39bcbee98b84e0fe2/packages/integrations/core/src/facade/runtime.ts) - [`packages/integrations/core/integration/facade-dom.test.ts`](https://github.com/browserbase/stagehand/blob/f484d008abb8e0ef5950ffa39bcbee98b84e0fe2/packages/integrations/core/integration/facade-dom.test.ts) - [`packages/integrations/core/tests/facade-wait-for-url.test.ts`](https://github.com/browserbase/stagehand/blob/f484d008abb8e0ef5950ffa39bcbee98b84e0fe2/packages/integrations/core/tests/facade-wait-for-url.test.ts) Reordered stack: #2891 → #2892 → #2895 → #2896 → #2897 → #2898 → #2899 → #2900 → #2901 → #2893 → #2894. Readiness: [CI passed](https://github.com/browserbase/stagehand/actions/runs/35193871165) on `f484d008ab`. Review threads are triaged and resolved.
miguelg719
added a commit
that referenced
this pull request
Sep 17, 2026
Composes facade locators through same-origin and out-of-process frames and preserves global snapshot references without duplicating unresolved frame maps. Adds operator documentation for ownership. Deadline and disconnect documentation follows in #2893 and #2894. Also repairs failures exposed by stack CI: - Background-tab screenshots activate their target and serialize activation/capture across the browser. Setup and cleanup are serialized per page, so a stalled page cannot block other tabs. Capture deadlines stop waiting for stalled CDP responses; late responses cannot re-enter cleanup. Calls on a page with unfinished timeout recovery fail promptly until cleanup settles, after which screenshots resume. - The Vantech Journal scorer identifies the archive’s Load more link by destination and text instead of an obsolete absolute XPath. Strict DOM element-identity scoring and the 80% gate remain unchanged. - CLI test deadlines account for the bounded child processes they launch, including multi-command tests. Validation: 24 local screenshot/lifecycle tests and 66 CLI/help tests passed, along with relevant typechecks and Go extension archive checks. Tests cover timeout rejection, isolation across tabs, and recovery after delayed setup/cleanup settles. Live Chrome verification accepts the recorded correct Vantech observation and rejects an unrelated element. Stack position: 4/11. Base: `evals/consolidation-05-locator-wait-contract`. Reviewer entry points: - [Frame facade implementation](https://github.com/browserbase/stagehand/blob/6d79622f5a518b134c1c725e88dd0594d337bff3/packages/integrations/core/src/facade/runtime.ts) - [Snapshot frame maps](https://github.com/browserbase/stagehand/blob/6d79622f5a518b134c1c725e88dd0594d337bff3/packages/extension/understudy/a11y/snapshot/capture.ts) - [Screenshot lifecycle](https://github.com/browserbase/stagehand/blob/6d79622f5a518b134c1c725e88dd0594d337bff3/packages/extension/understudy/page.ts) - [Archive scorer](https://github.com/browserbase/stagehand/blob/6d79622f5a518b134c1c725e88dd0594d337bff3/packages/evals/tasks/bench/observe/observe_vantechjournal.ts) Reordered stack: #2891 → #2892 → #2895 → #2896 → #2897 → #2898 → #2899 → #2900 → #2901 → #2893 → #2894. Readiness: [CI passed](https://github.com/browserbase/stagehand/actions/runs/35194404031) on `6d79622f5a`. Review threads are triaged and resolved.
miguelg719
added a commit
that referenced
this pull request
Sep 17, 2026
Bounds stalled DeepAgents setup, stream waits and cleanup. Timeout settings reject invalid values, preserve the original stop reason and finish cleanup without losing a completed tool-result batch. MCP setup, stream and cleanup timeouts preserve AnyIO task ownership; real in-memory MCP sessions cover the cancel-scope regression and cleanup deadlines. Validation: 35 Python runner tests; 29 Python tests and Python compilation passed, including stalled MCP acquisition/tool discovery, competing wall/inactivity deadlines, invalid timeout inputs, and cleanup. Stack position: 5/11. Base: `evals/consolidation-06-frame-support`. Each PR contains the change relative to its immediate predecessor. Reviewer entry points: - [`packages/integrations/deepagents/runner/run_eval.py`](https://github.com/browserbase/stagehand/blob/d1d6e48fd946158a042501aad592eb21b01e8481/packages/integrations/deepagents/runner/run_eval.py) - [`packages/integrations/deepagents/runner/tests/test_run_eval.py`](https://github.com/browserbase/stagehand/blob/d1d6e48fd946158a042501aad592eb21b01e8481/packages/integrations/deepagents/runner/tests/test_run_eval.py) Follow-up review validation: 40 Python tests pass. Timeout attribution now distinguishes the runner’s expired deadline from inner MCP/provider `TimeoutError`, preserves subsecond diagnostic values, and records the selected watchdog before cancellation cleanup. Same-task MCP/AnyIO ownership is preserved. The alleged early-expiry arithmetic was not confirmed; a separate slow-cleanup attribution regression is covered. Reordered stack: #2891 → #2892 → #2895 → #2896 → #2897 → #2898 → #2899 → #2900 → #2901 → #2893 → #2894. Readiness: [CI passed](https://github.com/browserbase/stagehand/actions/runs/35195059814) on `d1d6e48fd9`. Review threads are triaged and resolved.
miguelg719
added a commit
that referenced
this pull request
Sep 17, 2026
Routes direct Mastra and DeepAgents model requests through the intended native provider while respecting explicit gateway routing. Missing provider configuration remains an actionable startup error. Missing xAI credentials fail before client construction, preventing fallback to another provider’s ambient key. Validation: 41 Python runner tests including synthetic cross-provider credential regressions; 32 Python tests, 18 Mastra routing/session tests, five build tasks, frozen dependency installation, and Mastra typecheck passed. Stack position: 6/11. Base: `evals/consolidation-07-deepagents-lifecycle`. Each PR contains the change relative to its immediate predecessor. Reviewer entry points: - [`packages/integrations/mastra-sdk/src/session.ts`](https://github.com/browserbase/stagehand/blob/5caa682dca8269b7772a71ad530f4a9ccea87956/packages/integrations/mastra-sdk/src/session.ts) - [`packages/integrations/mastra-sdk/tests/model-routing.test.ts`](https://github.com/browserbase/stagehand/blob/5caa682dca8269b7772a71ad530f4a9ccea87956/packages/integrations/mastra-sdk/tests/model-routing.test.ts) - [`packages/integrations/deepagents/runner/run_eval.py`](https://github.com/browserbase/stagehand/blob/5caa682dca8269b7772a71ad530f4a9ccea87956/packages/integrations/deepagents/runner/run_eval.py) Reordered stack: #2891 → #2892 → #2895 → #2896 → #2897 → #2898 → #2899 → #2900 → #2901 → #2893 → #2894. Readiness: [CI passed](https://github.com/browserbase/stagehand/actions/runs/35195660877) on `5caa682dca`. Review threads are triaged and resolved.
miguelg719
added a commit
that referenced
this pull request
Sep 17, 2026
Bounds retained SDK event logs while keeping actionable diagnostics and trajectory evidence. Pi screenshot compaction and its Buffer-aware trajectory consumer land together, preventing intermediate screenshot loss. Validation: 118 SDK/event tests, five Pi adapter/pipeline tests, 12 build tasks, and SDK/core/eval typechecks passed. The pipeline test imports the actual built Pi compactor and checks retained image bytes for compacted Buffer and historical base64 events. Stack position: 7/11. Base: `evals/consolidation-08-native-routing`. Each PR contains the change relative to its immediate predecessor. Reviewer entry points: - [`packages/integrations/core/src/harness/eventLog.ts`](https://github.com/browserbase/stagehand/blob/bfc240ec68d562c190885bc676c57ac6360f835b/packages/integrations/core/src/harness/eventLog.ts) - [`packages/integrations/pi-sdk/src/session.ts`](https://github.com/browserbase/stagehand/blob/bfc240ec68d562c190885bc676c57ac6360f835b/packages/integrations/pi-sdk/src/session.ts) - [`packages/evals/framework/harnesses/piAdapter.ts`](https://github.com/browserbase/stagehand/blob/bfc240ec68d562c190885bc676c57ac6360f835b/packages/evals/framework/harnesses/piAdapter.ts) <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Bounds SDK event logs and retained event payloads across Claude, Codex, DeepAgents, Eve, Fx, Mastra, and Pi. Previously, stream noise and repeated payloads stayed visible; now stream fragments are dropped, routine events move to debug, and failures stay at level 1 without changing model-facing results or trajectory evidence. - Adds a shared event classifier that drops deltas and bare lifecycle markers and keeps SDK-specific failures visible. - Retains Pi screenshots once as `Buffer` values, drops duplicate image payloads from logs and non-assistant messages, and records omission markers over 8 MiB per image or 64 MiB per run. The Pi adapter accepts both `Buffer` and legacy base64 image blocks. - Compacts Mastra `step-finish` and `finish` events to keep usage and trajectory data without repeated request bodies. - Redacts credentials before clipping event details. <sup>Written for commit bfc240e. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/browserbase/stagehand/pull/2899?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> Reordered stack: #2891 → #2892 → #2895 → #2896 → #2897 → #2898 → #2899 → #2900 → #2901 → #2893 → #2894. Readiness: [CI passed](https://github.com/browserbase/stagehand/actions/runs/35196277234) on `bfc240ec68`. Review threads are triaged and resolved.
miguelg719
added a commit
that referenced
this pull request
Sep 17, 2026
Distinguishes unreported usage from observed zero across SDK boundaries and recovers interrupted Codex usage without replacing authoritative completed-turn counts. Eval accounting consumes this metadata in the later adoption PR. Validation: 167 SDK tests, 38 Python tests, 12 build tasks, and SDK/core/eval typechecks passed. Authoritative completed Codex usage takes precedence over interrupted-rollout recovery. Stack position: 8/11. Base: `evals/consolidation-09-event-retention`. Each PR contains the change relative to its immediate predecessor. Reviewer entry points: - [`packages/integrations/codex-sdk/src/session.ts`](https://github.com/browserbase/stagehand/blob/b60164d6a5155900c83880a04c0767cab915c562/packages/integrations/codex-sdk/src/session.ts) - [`packages/integrations/deepagents/runner/run_eval.py`](https://github.com/browserbase/stagehand/blob/b60164d6a5155900c83880a04c0767cab915c562/packages/integrations/deepagents/runner/run_eval.py) - [`packages/integrations/mastra-sdk/src/session.ts`](https://github.com/browserbase/stagehand/blob/b60164d6a5155900c83880a04c0767cab915c562/packages/integrations/mastra-sdk/src/session.ts) Reordered stack: #2891 → #2892 → #2895 → #2896 → #2897 → #2898 → #2899 → #2900 → #2901 → #2893 → #2894. Readiness: [CI passed](https://github.com/browserbase/stagehand/actions/runs/35197720052) on `b60164d6a5`. Review threads are triaged and resolved.
miguelg719
added a commit
that referenced
this pull request
Sep 17, 2026
…#2901) Separates verified task completion from execution health. Verification errors fail closed, preserve the agent report separately, and retain captured evidence and raw uncertainty for audit. A supported completion can still pass after a late browser disconnect. Preserves judge verdicts beside deterministic evidence gates; missing criterion scores remain in the denominator. Blocker wording and execution error alone do not erase a completed rubric requirement. Numeric grounding remains advisory by default. It includes captured step and terminal accessibility text, records terminal matches separately, and treats unknown-page text as untrusted. Batch summaries expose explicit zero-browser passes; the configured verifiability gate rejects them. Producer-dependent usage, cost, facade and lifecycle fields are documented as conditional. Offline verification uses the same model factory as live grading: CLI override, then EVAL_VERIFIER_MODEL, then the shared default. Thrown verification errors and uncertain judgments are preserved as explicit ungraded artifacts without top-level outcome/process scores, including live persistence and tracing; the offline command exits unsuccessfully. Trace labels are restricted to filename components, and failed writes emit sanitized warnings. Judge policy still needs confirmation: this proposed change selects google/gemini-3.5-flash by default, while current main uses 2.5 Flash. The completed 108-case compatibility gate explicitly used 2.5 Flash and does not validate the proposed 3.5 default. Validation: 104 focused verifier/CLI/span/persistence/summary tests at this boundary, eval typecheck and scoped formatting; 13 assertions reproduced the new review gaps before fixes. No frozen 108-case compatibility inputs changed. No evaluator-v4 implementation. Stack position: 9/11. Base: `evals/consolidation-10-usage-provenance`. Each PR contains the change relative to its immediate predecessor. Reviewer entry points: - [`packages/evals/framework/verifierAdapter.ts`](https://github.com/browserbase/stagehand/blob/af1cf703925647db722d166cafc75bcfbe3f36df/packages/evals/framework/verifierAdapter.ts) - [`packages/evals/framework/verifierGates.ts`](https://github.com/browserbase/stagehand/blob/af1cf703925647db722d166cafc75bcfbe3f36df/packages/evals/framework/verifierGates.ts) - [`packages/evals/docs/verifier-gates.md`](https://github.com/browserbase/stagehand/blob/af1cf703925647db722d166cafc75bcfbe3f36df/packages/evals/docs/verifier-gates.md) Reordered stack: #2891 → #2892 → #2895 → #2896 → #2897 → #2898 → #2899 → #2900 → #2901 → #2893 → #2894. Readiness: reordered head `af1cf70392` is awaiting fresh CI and review triage.
miguelg719
changed the base branch from
evals/consolidation-11-verifier-results
to
main
September 21, 2026 21:07
This was referenced Sep 21, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bounds capture and RPC/batch waits. Two consecutive capture deadlines are recoverable; a third latches terminal session loss, while a successful tool call resets the counter. Late completion cannot overwrite newer snapshot IDs.
Terminal loss rejects queued calls without dispatch. Actions are never automatically replayed. Regressions cover bounded timeout inputs, late completion, counter reset, screenshot transport and first-loss handling.
Preserves provider/session/age/timeout diagnostics through the host, checks screenshot byte limits before decoding, and isolates failures in diagnostic observers. The typed RPC timeout error lives in a focused public module, keeping internal RPC helpers outside the SDK public-field contract.
Validation: 120 focused tests and 47 SDK AST/release guards, SDK/core/eval typechecks, SDK build/publint, and merged ownership/capture regression checks.
Stack base:
main. Review this PR relative to its immediate predecessor.Reviewer entry points:
packages/integrations/core/src/facade/tools.tspackages/sdk-ts/src/batch.tspackages/integrations/core/tests/facade-tools.test.tsThe CLI startup fix is now included in the parent (#2892), so this PR focuses on capture/RPC deadlines and recovery. Current main is incorporated, preserving both the timeout exports and the runtime-compatibility exports.
Readiness: CI passed on
7b5f94b2de. Human review status is unchanged.Current open stack: #2893 → #2894 → #2902 → #2903 → #2889 → #2904 → #2905 → #2906 → #2907.