Skip to content

improvement(computer-use): report an advertised action the element refuses as unsupported, with the snapshot still valid - #10106

Merged
waleedlatif1 merged 1 commit into
stagingfrom
improvement/computer-use-unsupported-open-hint
Nov 17, 2026
Merged

waleedlatif1 merged 1 commit into
stagingfrom
improvement/computer-use-unsupported-open-hint

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Correction: Finder and file-panel cells refuse AXOpen with kAXErrorAttributeUnsupported (-25205), not kAXErrorActionUnsupported (-25206), so this change alone does not cover them; #10109 handles -25205.

Summary

  • Some elements advertise an Accessibility action and then refuse it with kAXErrorActionUnsupported (-25206); Finder and file-panel sidebar/list cells do this with AXOpen
  • That surfaced as a generic accessibility_error telling the agent to observe again, even though nothing reached the app and the snapshot was still valid, so every occurrence cost an extra observation
  • dispatchAX now reports it as unsupported_action, which the existing request-error path already marks dispatchState: not_started with the snapshot still valid, and the message points to another advertised action or a click (clickCount: 2 opens most files and folders; an element click with a count goes through the pointer path)

Type of Change

  • Improvement

Testing

  • On a GUI-only run of 26 native macOS tasks (MacArena), advertised-but-refused AXOpen accounted for 12 failed calls across 11 episodes, all on Finder/file-panel cells, each followed by a re-observation
  • Finder probe in a macOS VM (no model calls): pending, AXOpen on the Downloads sidebar cell, then a double-click on the same snapshot
  • CI desktop jobs compile the helper

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant