Skip to content

fix(click): report why the overlay could not be cleared from the click point (#556) - #782

Open
haomint wants to merge 2 commits into
Tencent:mainfrom
haomint:fix/click-overlay-diagnostics
Open

haomint wants to merge 2 commits into
Tencent:mainfrom
haomint:fix/click-overlay-diagnostics

Conversation

@haomint

@haomint haomint commented Nov 19, 2026 •

Copy link
Copy Markdown

Summary

Fixes #556.

withClickOverlay reported every overlay-clear failure as one message with no evidence attached:

{"code":"cdp_failed","data":{"effect_state":"none","pointer_moved":false,"reason":"input_not_ready"},
 "message":"Could not clear the extension overlay from the click point"}

Several unrelated causes produce that error, and none of them need the same follow-up. As the report shows, the cost is real: with no coordinates, no probe result and no bypass state, the investigator could not tell the extension overlay apart from the site's own mask ("网络较慢,请耐心等待…") and had to reason about it from a screenshot.


Change

Keep reason: "input_not_ready" and every existing gate exactly as they are — only attribute the failure and carry the evidence:

Field Value
click_point { x, y } the hit test used
overlay_probe last absent / clear / covered / unknown
overlay_bypass requested / failed / not_requested / unsupported
overlay_passthrough attempted / not_started
recovery per-cause follow-up

overlay_bypass reports the request, not a confirmed state: the production callback swallows a failed or timed-out send, so enabled would have claimed more than the code can know.

Causes and messages:

Cause Message
overlay_still_covered The BrowserSkill overlay still covers the click point (40, 60) after bypass and input passthrough
overlay_state_unknown Could not confirm the click point (40, 60) is clear: the overlay hit test did not answer after bypass
passthrough_headroom Input passthrough has too little lease left for this gesture at the click point (40, 60)
passthrough_expired Input passthrough expired before the click point (40, 60) could be verified

passthrough_headroom and passthrough_expired are split because pressDeadlineReached() also rejects a fresh lease when the gesture's input budget exceeds the lease itself (drag with a hold). Waiting and lowering the action timeout cannot fix that, so it recommends shortening the gesture instead.

The gate order (pressDeadlineReached() → covered → covered && unknown) is unchanged; only the attribution is split, so dispatch behaviour is identical.

Tests

apps/extension/src/tools/__tests__/click-overlay.test.ts is new — this module had no test coverage, which is why the collapsed attribution survived:

  1. still-covered → coordinates, overlay_probe: "covered", bypass requested, passthrough attempted, click never dispatched
  2. unreadable hit test after bypass → overlay_probe: "unknown", distinct message
  3. insufficient passthrough headroom → distinct message and a "shorten the gesture" recovery
  4. bypass request throws → overlay_bypass: "failed", not requested
  5. unobstructed click → unchanged: no bypass, no passthrough, beforePress returns null

Local verification: 5/5 new tests pass, full extension unit suite 4107 passed / 8 skipped (pre-existing), tsc --noEmit clean, biome check clean, bilingual changelog sync passes.

Notes

No protocol change: the reason stays input_not_ready, and RpcErrorData already allows extra keys. If you would rather model these as distinct RpcErrorReason variants so the CLI can render its own copy per cause, that needs a bsk-protocol change — say the word and I will follow up with the schema regeneration.

…k point

withClickOverlay collapsed three unrelated causes into one message with
no coordinates and no state, so a reported failure could not be
diagnosed from the receipt (Tencent#556). The report shows how expensive that
is: the investigator could not tell the extension overlay apart from the
site's own mask and had to reason about it from a screenshot.

Keep reason "input_not_ready" and the existing gates, but attribute the
failure and attach the evidence: click_point, overlay_probe,
overlay_bypass, overlay_passthrough, and a recovery hint per cause.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T07:03:00.447379Z e192691 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e192691391

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

pointer_moved: false,
click_point: { x: point.x, y: point.y },
overlay_probe: observed,
overlay_bypass: bypass ? "enabled" : deps.bypassOverlay ? "unavailable" : "unsupported",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Report bypass enablement only after confirmed delivery

When the content script is unavailable or overlay messaging times out, the production click/drag callback in dispatcher.ts:1520-1528 catches the sendTabMessageBounded failure and resolves normally. Consequently, withClickOverlay sets bypass = true, and this new field reports enabled even though the enable message failed. The unavailable value therefore cannot identify actual production transport failures, undermining the new diagnostic evidence. Propagate a delivery result from the callback, or report the bypass as attempted rather than confirmed enabled.

Useful? React with 👍 / 👎.

if (retired || deps.signal?.aborted) return cancelled();
if (pressDeadlineReached() || hit === "covered" || (covered && hit === "unknown"))
return notReady();
if (pressDeadlineReached()) return notReady("passthrough_deadline", hit);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Distinguish insufficient lease headroom from lease expiry

For a drag that needs input passthrough with duration_ms: 5000 and no hold, drag.ts:431-432 supplies a 6000 ms input budget against the fixed 5000 ms lease. This gate therefore rejects immediately on a fresh lease, but the newly selected diagnostic says passthrough expired and recommends waiting or reducing the action deadline. The budget depends on hold_ms and duration_ms, not timeout_ms, so waiting or lowering the timeout cannot resolve this case. Report insufficient remaining lease and recommend shortening the gesture instead of attributing every reserve-budget rejection to an expired lease.

Useful? React with 👍 / 👎.

Address review findings on the diagnostic fields:

- Resolving `bypassOverlay` does not prove delivery: the production
  callback swallows a failed or timed-out send. Report the request state
  (`requested` / `failed` / `not_requested` / `unsupported`) instead of
  claiming `enabled`.
- `pressDeadlineReached()` also rejects a fresh lease when the gesture's
  budget exceeds the lease (drag with a hold), so "expired" was wrong and
  its recovery could not help. Split `passthrough_headroom` from
  `passthrough_expired` and recommend shortening the gesture.
@haomint

haomint commented Nov 19, 2026

Copy link
Copy Markdown
Author

Both findings were right — addressed in e02a620.

Bypass is now a request, not a claim. Resolving bypassOverlay does not prove the content side received the enable message, so the field reports requested / failed / not_requested / unsupported, and enabled is gone. A test makes the callback throw and asserts failed, so the field cannot silently regress to claiming delivery.

Headroom is no longer reported as expiry. You are right that pressDeadlineReached() also rejects a fresh lease once the gesture budget exceeds the lease — drag's 6000 ms budget against a 5000 ms lease. Split into passthrough_headroom ("too little lease left for this gesture", recovery: shorten the gesture, stateing outright that waiting or lowering the timeout cannot help) and passthrough_expired (the lease actually ran out, recovery: wait for the page to settle and retry). The gate itself is untouched.

Tests: 5 in the new click-overlay.test.ts; full suite 4107 passed / 8 skipped; tsc and biome clean.

@NianJiuZst

Copy link
Copy Markdown
Collaborator

Thanks for picking up #556 and iterating on the earlier review feedback! Returning structured diagnostic evidence instead of one opaque message is definitely the right direction, though there are a few issues we should address before merging and closing #556:

  1. Slow normal clicks are misattributed as passthrough_headroom (apps/extension/src/tools/click-overlay.ts)

    • deadlineFailure() currently checks expired() ? "passthrough_expired" : "passthrough_headroom".
    • For a normal single click (inputBudgetMs = 1_000, INPUT_PASSTHROUGH_TTL_MS = 5_000), pressDeadlineReached() trips whenever 4_000ms <= elapsed < 5_000ms (for example on a slow page/IPC). Because expired() is still false, a plain click gets classified as passthrough_headroom with "Shorten the gesture (its hold or drag duration) ... Waiting and lowering the action timeout cannot change this", which doesn't apply to a normal click (waiting and retrying does fix a 4.2s stall).
    • The real distinction is inputBudgetMs >= INPUT_PASSTHROUGH_TTL_MS (the gesture budget cannot fit inside a fresh lease) vs. inputBudgetMs < INPUT_PASSTHROUGH_TTL_MS (lease time ran out while waiting).
    • Also, please remove "or run the action without passthrough" from the recovery copy — passthrough is an internal extension mechanism and callers have no flag/option to disable it.
  2. Error messages can contradict data fields

    • overlay_still_covered unconditionally says "after bypass and input passthrough" (and overlay_state_unknown says "after bypass"), even on the late-covered path inside beforePress() when the initial probe was "clear"/"absent" so neither bypass nor passthrough ran (overlay_bypass: "not_requested", overlay_passthrough: "not_started"), or when bypassOverlay is "unsupported" / "failed".
    • Please phrase the message based on what was actually attempted.
  3. Transport failures are still swallowed for both bypass and passthrough

    • Production bypassOverlay in apps/extension/src/tools/dispatcher.ts still catches and swallows sendTabMessageBounded errors, so overlay_bypass: "failed" is unreachable in production (only hit by the mock in click-overlay.test.ts). To address item 4 of [Bug] 0.4.1: 「Could not clear the extension overlay from the click point」 无坐标/无遮挡元素/无 bypass 失败原因,无法自诊断 #556, bypassOverlay should report delivery failure instead of swallowing it silently.
    • Similarly, sendInputPassthrough already rejects when the content script is unreachable and resolves to { type: INPUT_PASSTHROUGH, ok: true } when applied, yet withClickOverlay still discards the catch error and reports overlay_passthrough: "attempted". Tracking acked vs. failed vs. not_started would let callers tell a dead/unreachable content script apart from an overlay that remained covering.
  4. Remaining diagnostic fields requested in [Bug] 0.4.1: 「Could not clear the extension overlay from the click point」 无坐标/无遮挡元素/无 bypass 失败原因,无法自诊断 #556

    • Since this PR uses Fixes #556, it would be great to include the remaining high-signal evidence without adding extra round-trips:
      • viewport dimensions alongside click_point (so callers can tell whether the click landed under a corner HUD);
      • the unknown probe error/exception summary (currently only logged via console.debug);
      • where accessible (e.g. via the content-side passthrough ack or host metadata), which overlay slot/control is blocking — note that Help / Record / Borrow overlays are intentionally excluded from input passthrough, so recommending "reload the tab once" is misleading when one of those is active.
  5. CLI vs. MCP hint consistency

    • Because reason remains "input_not_ready", CLI hint_for() (crates/bsk-cli/src/cli/error.rs) prefers data.recovery over render_error.rs (dropping the "no click, key press or wheel input was sent" guidance), whereas MCP mcp_hint_for() prefers RpcErrorReason::InputNotReady's mcp_hint over data.recovery. That leaves MCP callers seeing two different hints, and data.recovery currently includes CLI-only command names (get-html). Please keep data.recovery tool-neutral (or wire distinct reasons through bsk-protocol / render_error.rs if per-cause CLI/MCP hints are needed).
  6. Tests & CHANGELOG hygiene

    • withClickOverlay already has top-level coverage through handleClick / handleDrag in apps/extension/src/tools/__tests__/interaction.test.ts, drag.test.ts, and real-browser click.browser.test.ts. Covering these diagnostics through those top-level entrypoints (including a real-browser assertion in click.browser.test.ts) will catch integration mismatches like the dispatcher.ts swallowing issue rather than testing isolated mocks.
    • In both CHANGELOG.md and CHANGELOG.zh-CN.md, the new bullet was inserted above ### Fixed / ### 修复 (so it currently sits at the end of ### Changed / ### 变更), and the copy still refers to "the three causes" / "三类成因" after commit e02a620 split them into four.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] 0.4.1: 「Could not clear the extension overlay from the click point」 无坐标/无遮挡元素/无 bypass 失败原因,无法自诊断

2 participants