Skip to content

fix(cooperation): stop false human_intervention on scripted edits/clicks, cap readiness waits, and unblock failed-load tab management - #783

Merged
NianJiuZst merged 1 commit into
Tencent:mainfrom
NianJiuZst:fix/false-human-intervention-and-failed-load-tabs
Nov 19, 2026
Merged

NianJiuZst merged 1 commit into
Tencent:mainfrom
NianJiuZst:fix/false-human-intervention-and-failed-load-tabs

Conversation

@NianJiuZst

Copy link
Copy Markdown
Collaborator

Summary

Fixes three real-world cooperation failure modes diagnosed from a production bsk interruption log on WeChat MP (mp.weixin.qq.com) where bsk evaluate repeatedly aborted with "the user has control of this tab" despite zero human activity, bsk click on a hidden checkbox timed out after 30s without reporting target_hidden, and a failed-load agent tab deadlocked tab create / tab select / tab close.

Root Causes & Fixes

  1. False human_intervention ("the user has control of this tab" / "User activity revoked this action") on scripted execCommand and .click() (apps/extension/src/content/cooperation-capture.ts):

    • Cause: In Chromium/Blink, document.execCommand('insertHTML' | 'delete' | 'insertText', ...) dispatches a trusted InputEvent("input") (isTrusted: true) with no preceding beforeinput, keyboard, pointer, wheel, drop, or IME event. Similarly, programmatic checkbox.click() / radio.click() / label.click() dispatches an untrusted click (isTrusted: false, previously ignored) followed synchronously by trusted input/change default-activation events (isTrusted: true). cooperation-capture.ts treated any trusted input/change without a native Input.* agent lease as human input, revoking completed bsk evaluate calls and freezing the intent ledger.
    • Fix:
      • Track untrusted programmatic click events within the current task (when no human pointer/key gesture is active) so synchronous default checkbox/radio input/change activations are not misclassified as human activity.
      • Skip humanInput on bare trusted InputEvent("input") (and its same-task blur change) dispatched on editable fields with no originating beforeinput, key, pointer, wheel, drop, or IME composition, while still updating fields/sampledValues/revision and preserving detection of all genuine user interactions.
  2. Hidden/disabled target readiness wait racing Daemon's 30s RPC timeout (apps/extension/src/cooperation/controller.ts):

    • Cause: CooperationController.run polled transient target_hidden / target_disabled / target_readonly checks up to Date.now() + (params.timeout_ms ?? 30_000). Because CLI commands pass timeout_ms: 30000 (and transfers pass 120000) by default with response_grace: false, permanently hidden elements (such as visually hidden WeUI <input type="checkbox"> controls) hung for 30s and lost the race to Daemon's generic tool RPC timed out after 30s.
    • Fix: Cap the pre-execution transient readiness poll at DEFAULT_TARGET_READINESS_TIMEOUT_MS (5s) and below explicit short timeout_ms with a reply margin anchored to runEnteredAt, keeping the full deadline intact for prepareFrames and tool execution.
  3. Tab management deadlocked by an owned agent tab on a page_load_failed error page (apps/extension/src/cooperation/controller.ts):

    • Cause: PR fix(cooperation,cdp,daemon): stop false coverage-changed freezes across SPA routes and subframe churn #762 unblocked navigation actions (navigate/reload/navigate_back/navigate_forward) off page_load_failed error pages, but manageTabs (tab_create/tab_select/tab_borrow/tab_return) and tool.tab_close still refused when any owned non-borrowed agent tab (including human-preserved observedTabs) sat on a settled page_load_failed error page.
    • Fix: Exempt owned non-borrowed tabs (isAgentControlledTab && !borrowedTabs.has) on a settled page_load_failed interstitial (!tab.pendingUrl) in manageTabs and tool.tab_close, while keeping borrowed-tab restrictions and in-flight pendingUrl checks intact.
  4. Changelogs: Updated both CHANGELOG.md and CHANGELOG.zh-CN.md.

Verification

  • Full-stack real-Chrome E2E (node scripts/run-browser-tests.mjs src/cooperation/settlement.browser.test.ts with real bsk CLI, daemon, built Chrome MV3 extension, and Google Chrome): 13/13 passed (0 skipped), including a new end-to-end test covering bsk evaluate with execCommand (delete/insertHTML/insertText + reformat + blur) and checkbox/label/radio .click(), negative controls confirming genuine human checkbox click and typing still trigger human_intervention, hidden-checkbox bsk click returning structured target_hidden (effect_state: "none") under both default and explicit --timeout, and tab create / tab select / tab close escaping a page_load_failed agent tab.
  • Additional real-Chrome browser suites: cooperation-capture.browser.test.ts + live-text.browser.test.ts: 36/36 passed.
  • Extension unit suites: pnpm --filter @browser-skill/extension test: 4,103 passed (190 test files, including new failed-load tab assertions in borrowed-blank-tab.test.ts).
  • Typecheck & lint: tsc --noEmit and pnpm lint (biome check . && stylelint "**/*.css"): 0 errors.

…cks, cap readiness waits, and unblock failed-load tab management

Why:
1. In Chromium, programmatic `document.execCommand('insertHTML' | 'delete' | 'insertText', ...)` on contenteditable/form editors dispatches a trusted `InputEvent("input")` with no preceding `beforeinput`/pointer/key/wheel/drop/IME event, and programmatic `checkbox.click()` / `radio.click()` / `label.click()` dispatches an untrusted `click` whose synchronous default activation emits trusted `input`/`change` events. `cooperation-capture.ts` previously treated any trusted `input`/`change` with no native agent lease as human input, causing `bsk evaluate` (and page scripts) to falsely abort with `human_intervention` ("the user has control of this tab" / "User activity revoked this action") and freeze the intent ledger even when no user touched the browser.
2. `CooperationController.run` polled transient `target_hidden` / `target_disabled` / `target_readonly` checks for the full `params.timeout_ms ?? 30_000` (which CLI interaction commands default to 30s and file transfers default to 2m) with no reply headroom below the daemon's RPC timeout, causing clicks on permanently hidden controls (such as visually hidden WeUI checkbox `<input>` elements) to hang for 30s and lose the race to a generic `tool RPC timed out after 30s`.
3. While PR Tencent#762 allowed navigation actions to recover off `page_load_failed` error pages, `CooperationController.manageTabs` and `tool.tab_close` still refused when any owned non-borrowed agent tab (including human-preserved `observedTabs`) sat on a settled `page_load_failed` interstitial, deadlocking `tab create` / `tab select` / `tab close` behind `cooperation_sync_required` ("shared page control or observation changed").

What:
- Track untrusted programmatic `click` events within the current task in `cooperation-capture.ts` (when no human pointer/key gesture is active) so synchronous default checkbox/radio `input`/`change` activations are not misclassified as human input, and skip `humanInput` on bare trusted `InputEvent("input")` (plus its same-task blur `change`) fired on editable elements with no originating `beforeinput`, key, pointer, wheel, drop, or IME composition while continuing to update `fields`/`revision` and preserving all genuine human input detection.
- Cap pre-execution `target_hidden`/`target_disabled`/`target_readonly` readiness polling in `CooperationController` at `DEFAULT_TARGET_READINESS_TIMEOUT_MS` (5s) and below explicit short `timeout_ms` with a reply margin anchored to `runEnteredAt`, keeping the full `deadline` intact for `prepareFrames` and tool execution.
- Exempt owned non-borrowed tabs (`isAgentControlledTab && !borrowedTabs.has`) on a settled `page_load_failed` error page (`!tab.pendingUrl`) in `manageTabs` and `tool.tab_close`, preserving borrowed-tab restrictions and in-flight `pendingUrl` gating.
- Add full-stack real-Chrome CLI + daemon + extension regression coverage in `settlement.browser.test.ts` and failed-load tab management assertions in `borrowed-blank-tab.test.ts`, and update bilingual `CHANGELOG.md` / `CHANGELOG.zh-CN.md`.
@NianJiuZst
NianJiuZst merged commit 88ca431 into Tencent:main Nov 19, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant