Skip to content

Commit a10bdfd

Browse files
committed
feat(computer-use): add calendar event and contact app commands
1 parent ab00d40 commit a10bdfd

13 files changed

Lines changed: 360 additions & 64 deletions

File tree

‎apps/desktop/build/entitlements.mac.plist‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,12 @@
1414
Hardened Runtime denies the camera no matter what the user allowed. -->
1515
<key>com.apple.security.device.camera</key>
1616
<true/>
17-
<!-- Computer Use app commands add confirmed reminders through EventKit;
18-
the Hardened Runtime denies EventKit access without this key. -->
17+
<!-- Computer Use app commands add confirmed reminders, events and
18+
contacts through EventKit and Contacts; the Hardened Runtime denies
19+
that access without these keys. -->
1920
<key>com.apple.security.personal-information.calendars</key>
2021
<true/>
22+
<key>com.apple.security.personal-information.addressbook</key>
23+
<true/>
2124
</dict>
2225
</plist>

‎apps/desktop/electron-builder.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,8 @@ mac:
7171
extendInfo:
7272
NSScreenCaptureUsageDescription: Mothership captures approved app windows to understand and verify computer-use actions.
7373
NSRemindersFullAccessUsageDescription: Mothership adds reminders you confirm and lists your reminder lists so it can file them correctly.
74+
NSCalendarsFullAccessUsageDescription: Mothership adds calendar events you confirm and lists your calendars so it can file them correctly.
75+
NSContactsUsageDescription: Mothership adds contacts and contact groups you confirm.
7476
NSMicrophoneUsageDescription: Sim uses your microphone for voice input in Chat and for meetings you join in the built-in browser.
7577
NSCameraUsageDescription: Sim uses your camera for meetings you join in the built-in browser, such as Google Meet.
7678
# macOS 15 blocks connections to private addresses (self-hosted servers on a

‎apps/desktop/native/computer-use/ComputerUse.swift‎

Lines changed: 144 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import AppKit
22
import ApplicationServices
3+
import Contacts
34
import EventKit
45
import ScreenCaptureKit
56

@@ -42,15 +43,40 @@ struct Parameters: Decodable {
4243
var accessibilityAction: String?
4344
var command: AppCommand?
4445
}
45-
/// One allowlisted app command. Commands go through EventKit, never Apple Events or scripts, so they
46-
/// cannot reach a shell; each is a typed create or a list read, with no update, delete or search.
46+
/// One allowlisted app command. Commands go through EventKit and Contacts, never Apple Events or
47+
/// scripts, so they cannot reach a shell; each is a typed create or a list read, with no update,
48+
/// delete or search.
4749
struct AppCommand: Decodable {
4850
var name: String
4951
var title: String?
5052
var notes: String?
5153
var due: String?
5254
var listId: String?
5355
var priority: Int?
56+
var start: String?
57+
var end: String?
58+
var allDay: Bool?
59+
var location: String?
60+
var calendarId: String?
61+
var alertMinutesBefore: Int?
62+
var givenName: String?
63+
var familyName: String?
64+
var organization: String?
65+
var jobTitle: String?
66+
var emails: [String]?
67+
var phones: [String]?
68+
var groupId: String?
69+
}
70+
func invalidCommand(_ message: String) -> ComputerError { ComputerError("invalid_arguments", message, dispatchState: "not_started") }
71+
/// A required, trimmed text field of bounded length.
72+
func commandText(_ value: String?, _ field: String, max: Int = 1024) throws -> String {
73+
let text = (value ?? "").trimmingCharacters(in: .whitespacesAndNewlines)
74+
guard !text.isEmpty, text.count <= max else { throw invalidCommand("\(field) is required and limited to \(max) characters.") }
75+
return text
76+
}
77+
func optionalText(_ value: String?, _ field: String, max: Int = 1024) throws -> String? {
78+
guard let value, !value.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty else { return nil }
79+
return try commandText(value, field, max: max)
5480
}
5581
/// The event store for reminders, available only once the user has granted full Reminders access.
5682
/// Checking never prompts; the grant happens from settings, outside the tool call.
@@ -77,14 +103,45 @@ func dueComponents(_ text: String) throws -> DateComponents {
77103
calendar.timeZone = .current
78104
return calendar.dateComponents([.year, .month, .day, .hour, .minute, .timeZone], from: date)
79105
}
80-
/// Reminders access as the status reports it, read without prompting.
81-
func remindersAuthorization() -> String {
82-
switch EKEventStore.authorizationStatus(for: .reminder) {
106+
/// Reminders or Calendar access as the status reports it, read without prompting.
107+
func eventKitAuthorization(_ type: EKEntityType) -> String {
108+
switch EKEventStore.authorizationStatus(for: type) {
83109
case .fullAccess: return "granted"
84110
case .notDetermined: return "not_determined"
85111
default: return "denied"
86112
}
87113
}
114+
func contactsAuthorization() -> String {
115+
switch CNContactStore.authorizationStatus(for: .contacts) {
116+
case .authorized: return "granted"
117+
case .notDetermined: return "not_determined"
118+
default: return "denied"
119+
}
120+
}
121+
func calendarStore() throws -> EKEventStore {
122+
guard EKEventStore.authorizationStatus(for: .event) == .fullAccess else {
123+
throw ComputerError("calendars_permission_required", "Allow Calendar access for Mothership in System Settings, then retry.", dispatchState: "not_started")
124+
}
125+
return EKEventStore()
126+
}
127+
func contactStore() throws -> CNContactStore {
128+
guard CNContactStore.authorizationStatus(for: .contacts) == .authorized else {
129+
throw ComputerError("contacts_permission_required", "Allow Contacts access for Mothership in System Settings, then retry.", dispatchState: "not_started")
130+
}
131+
return CNContactStore()
132+
}
133+
/// An event boundary: an ISO 8601 date-time with an explicit offset, or `YYYY-MM-DD` for all-day events.
134+
func eventDate(_ text: String?, _ field: String, allDay: Bool) throws -> Date {
135+
let text = try commandText(text, field, max: 64)
136+
if allDay {
137+
let parts = try dueComponents(text)
138+
guard parts.hour == nil, let date = Calendar.current.date(from: parts) else { throw invalidCommand("All-day \(field) must be YYYY-MM-DD.") }
139+
return date
140+
}
141+
let formatter = ISO8601DateFormatter(); formatter.formatOptions = [.withInternetDateTime]
142+
guard let date = formatter.date(from: text) else { throw invalidCommand("\(field) must be an ISO 8601 date-time with an offset.") }
143+
return date
144+
}
88145
func runAppCommand(_ command: AppCommand) throws -> [String: Any] {
89146
switch command.name {
90147
case "list_reminder_lists":
@@ -116,6 +173,84 @@ func runAppCommand(_ command: AppCommand) throws -> [String: Any] {
116173
inputDispatched = true
117174
try store.save(reminder, commit: true)
118175
return ["kind": "command", "command": command.name, "created": ["id": reminder.calendarItemIdentifier, "title": title, "list": list.title]]
176+
case "list_calendars":
177+
let calendars = try calendarStore().calendars(for: .event).prefix(200).map { calendar -> [String: Any] in
178+
["id": calendar.calendarIdentifier, "title": String(calendar.title.prefix(1024)), "allowsModifications": calendar.allowsContentModifications]
179+
}
180+
return ["kind": "command", "command": command.name, "items": calendars]
181+
case "create_event":
182+
let title = try commandText(command.title, "Title")
183+
let allDay = command.allDay ?? false
184+
let start = try eventDate(command.start, "Start", allDay: allDay)
185+
let end = try eventDate(command.end, "End", allDay: allDay)
186+
guard allDay ? end >= start : end > start else { throw invalidCommand("End must be after start.") }
187+
guard end.timeIntervalSince(start) <= 31 * 86_400 else { throw invalidCommand("An event can last at most 31 days.") }
188+
let location = try optionalText(command.location, "Location")
189+
let notes = try optionalText(command.notes, "Notes", max: 8000)
190+
if let minutes = command.alertMinutesBefore, !(0...40_320).contains(minutes) { throw invalidCommand("alertMinutesBefore must be 0 to 40320.") }
191+
let store = try calendarStore()
192+
let calendar: EKCalendar
193+
if let calendarId = command.calendarId {
194+
guard let found = store.calendar(withIdentifier: calendarId), found.allowedEntityTypes.contains(.event) else {
195+
throw ComputerError("list_unavailable", "No calendar has this id; call list_calendars.", dispatchState: "not_started")
196+
}
197+
calendar = found
198+
} else {
199+
guard let fallback = store.defaultCalendarForNewEvents else { throw ComputerError("list_unavailable", "Calendar has no default calendar; pass calendarId.", dispatchState: "not_started") }
200+
calendar = fallback
201+
}
202+
guard calendar.allowsContentModifications else { throw ComputerError("list_read_only", "This calendar is read-only.", dispatchState: "not_started") }
203+
let event = EKEvent(eventStore: store)
204+
event.calendar = calendar; event.title = title; event.isAllDay = allDay
205+
event.startDate = start; event.endDate = end; event.location = location; event.notes = notes
206+
if let minutes = command.alertMinutesBefore { event.addAlarm(EKAlarm(relativeOffset: TimeInterval(-minutes * 60))) }
207+
inputDispatched = true
208+
try store.save(event, span: .thisEvent, commit: true)
209+
return ["kind": "command", "command": command.name, "created": ["id": event.eventIdentifier ?? event.calendarItemIdentifier, "title": title, "list": calendar.title]]
210+
case "list_contact_groups":
211+
let groups = try contactStore().groups(matching: nil).prefix(200).map { group -> [String: Any] in
212+
["id": group.identifier, "title": String(group.name.prefix(1024)), "allowsModifications": true]
213+
}
214+
return ["kind": "command", "command": command.name, "items": groups]
215+
case "create_contact_group":
216+
let name = try commandText(command.title, "Group name")
217+
let store = try contactStore()
218+
// A group with this name already exists after a lost reply; return it rather than add a twin.
219+
if let existing = try store.groups(matching: nil).first(where: { $0.name == name }) {
220+
return ["kind": "command", "command": command.name, "created": ["id": existing.identifier, "title": name, "list": "Contacts"]]
221+
}
222+
let group = CNMutableGroup(); group.name = name
223+
let request = CNSaveRequest(); request.add(group, toContainerWithIdentifier: nil)
224+
inputDispatched = true
225+
try store.execute(request)
226+
return ["kind": "command", "command": command.name, "created": ["id": group.identifier, "title": name, "list": "Contacts"]]
227+
case "create_contact":
228+
let given = try optionalText(command.givenName, "Given name"), family = try optionalText(command.familyName, "Family name")
229+
let organization = try optionalText(command.organization, "Organization"), jobTitle = try optionalText(command.jobTitle, "Job title")
230+
guard given != nil || family != nil || organization != nil else { throw invalidCommand("A contact needs a name or an organization.") }
231+
let emails = try (command.emails ?? []).map { try commandText($0, "Email", max: 320) }
232+
let phones = try (command.phones ?? []).map { try commandText($0, "Phone", max: 64) }
233+
guard emails.count <= 5, phones.count <= 5 else { throw invalidCommand("A contact takes at most 5 emails and 5 phones.") }
234+
guard emails.allSatisfy({ $0.contains("@") }) else { throw invalidCommand("Each email needs an @.") }
235+
let store = try contactStore()
236+
var group: CNGroup?
237+
if let groupId = command.groupId {
238+
guard let found = try store.groups(matching: CNGroup.predicateForGroups(withIdentifiers: [groupId])).first else {
239+
throw ComputerError("list_unavailable", "No contact group has this id; call list_contact_groups.", dispatchState: "not_started")
240+
}
241+
group = found
242+
}
243+
let contact = CNMutableContact()
244+
contact.givenName = given ?? ""; contact.familyName = family ?? ""
245+
contact.organizationName = organization ?? ""; contact.jobTitle = jobTitle ?? ""
246+
contact.emailAddresses = emails.map { CNLabeledValue(label: CNLabelWork, value: $0 as NSString) }
247+
contact.phoneNumbers = phones.map { CNLabeledValue(label: CNLabelPhoneNumberMain, value: CNPhoneNumber(stringValue: $0)) }
248+
let request = CNSaveRequest(); request.add(contact, toContainerWithIdentifier: nil)
249+
if let group { request.addMember(contact, to: group) }
250+
inputDispatched = true
251+
try store.execute(request)
252+
let display = [given, family].compactMap { $0 }.joined(separator: " ")
253+
return ["kind": "command", "command": command.name, "created": ["id": contact.identifier, "title": display.isEmpty ? organization ?? "" : display, "list": group?.name ?? "Contacts"]]
119254
default:
120255
throw ComputerError("unknown_command", "Unknown app command.", dispatchState: "not_started")
121256
}
@@ -879,10 +1014,12 @@ final class Driver {
8791014
_ = AXIsProcessTrustedWithOptions(options)
8801015
case "screenCapture": _ = CGRequestScreenCaptureAccess()
8811016
case "reminders": _ = try? await EKEventStore().requestFullAccessToReminders()
1017+
case "calendars": _ = try? await EKEventStore().requestFullAccessToEvents()
1018+
case "contacts": _ = try? await CNContactStore().requestAccess(for: .contacts)
8821019
default: throw ComputerError("invalid_arguments", "Unknown permission.")
8831020
}
884-
return ["kind": "status", "platform": "darwin", "accessibility": AXIsProcessTrusted(), "screenRecording": CGPreflightScreenCaptureAccess(), "reminders": remindersAuthorization()]
885-
case "status": return ["kind": "status", "platform": "darwin", "accessibility": AXIsProcessTrusted(), "screenRecording": CGPreflightScreenCaptureAccess(), "reminders": remindersAuthorization()]
1021+
return ["kind": "status", "platform": "darwin", "accessibility": AXIsProcessTrusted(), "screenRecording": CGPreflightScreenCaptureAccess(), "reminders": eventKitAuthorization(.reminder), "calendars": eventKitAuthorization(.event), "contacts": contactsAuthorization()]
1022+
case "status": return ["kind": "status", "platform": "darwin", "accessibility": AXIsProcessTrusted(), "screenRecording": CGPreflightScreenCaptureAccess(), "reminders": eventKitAuthorization(.reminder), "calendars": eventKitAuthorization(.event), "contacts": contactsAuthorization()]
8861023
case "list_apps": return ["kind": "apps", "apps": try discoverApps()]
8871024
case "app_command":
8881025
inputDispatched = false

‎apps/desktop/native/computer-use/Info.plist‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@
1111
<key>LSMinimumSystemVersion</key><string>14.0</string>
1212
<key>LSUIElement</key><true/>
1313
<key>NSHighResolutionCapable</key><true/>
14+
<key>NSCalendarsFullAccessUsageDescription</key><string>Mothership adds calendar events you confirm and lists your calendars so it can file them correctly.</string>
15+
<key>NSContactsUsageDescription</key><string>Mothership adds contacts and contact groups you confirm.</string>
1416
<key>NSRemindersFullAccessUsageDescription</key><string>Mothership adds reminders you confirm and lists your reminder lists so it can file them correctly.</string>
1517
<key>NSScreenCaptureUsageDescription</key><string>Mothership captures the app window you select to understand and verify computer-use actions.</string>
1618
</dict></plist>

‎apps/desktop/native/computer-use/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@ Compile each architecture with `swiftc -parse-as-library -O -target arm64-apple-
66

77
## Protocol
88

9-
Persistent newline-delimited JSON over stdin/stdout: `{id,method,params}` returns `{id,result}` or `{id,error:{code,message,dispatchState?}}`. The producer contract is `worker/packages/contracts/src/computer-use.ts` in mothership, copied into the desktop bridge by contract sync. `request_permission` is a local onboarding method, never a model tool; params are `{permission:"accessibility"|"screenCapture"|"reminders"}`. Status/preflight never triggers prompts; it reports `reminders` as `granted`, `denied` or `not_determined`.
9+
Persistent newline-delimited JSON over stdin/stdout: `{id,method,params}` returns `{id,result}` or `{id,error:{code,message,dispatchState?}}`. The producer contract is `worker/packages/contracts/src/computer-use.ts` in mothership, copied into the desktop bridge by contract sync. `request_permission` is a local onboarding method, never a model tool; params are `{permission:"accessibility"|"screenCapture"|"reminders"|"calendars"|"contacts"}`. Status/preflight never triggers prompts; it reports `reminders`, `calendars` and `contacts` as `granted`, `denied` or `not_determined`.
1010

11-
`app_command` takes `{command:{name,...}}` and runs one allowlisted command through EventKit, never Apple Events or scripts: `list_reminder_lists` returns `{kind:"command",items:[{id,title,allowsModifications}]}`, and `create_reminder` (`title`, optional `notes`, `due` as `YYYY-MM-DD` or an ISO 8601 date-time with an offset, `listId`, `priority` 0/1/5/9) returns `{kind:"command",created:{id,title,list}}`. Without full Reminders access it fails with `reminders_permission_required` before touching anything. There is no update, delete or search. The desktop service confirms each create with the user before sending it. Errors go through the protocol, and stdout contains no logs.
11+
`app_command` takes `{command:{name,...}}` and runs one allowlisted command through EventKit or Contacts, never Apple Events or scripts. Reads: `list_reminder_lists`, `list_calendars` and `list_contact_groups` return `{kind:"command",items:[{id,title,allowsModifications}]}`. Creates return `{kind:"command",created:{id,title,list}}`: `create_reminder` (`title`, `notes`, `due` as `YYYY-MM-DD` or ISO 8601 with an offset, `listId`, `priority` 0/1/5/9), `create_event` (`title`, `start`/`end` as ISO 8601 with offsets or `YYYY-MM-DD` with `allDay`, at most 31 days, `location`, `notes`, `calendarId`, `alertMinutesBefore`), `create_contact_group` (`title`; an existing group with that name is returned instead of a duplicate) and `create_contact` (`givenName`, `familyName`, `organization`, `jobTitle`, up to 5 `emails` and `phones`, `groupId`). Without the matching access a command fails with `reminders_permission_required`, `calendars_permission_required` or `contacts_permission_required` before touching anything. There is no update, delete or search. The desktop service confirms each create with the user before sending it.
1212

1313
The helper observes up to 2000 accessibility nodes (depth 64, queue 4000, traversal deadline 8 seconds, each AX call timeout 250ms). Selected-window and focused-element ancestry are prioritized. The menu bar and menus are read in place, so an open menu's items and their submenus come first; items of closed menus are read last, capped at 80 menu nodes. Electron accessibility is enabled using its documented `AXManualAccessibility` attribute. The helper observes 100 windows and 1000 apps. Requests are bounded to 128 KiB during ingestion. Text/value input is capped at 32000 UTF-16 units. State references are per-process, per-app, expire after 60 seconds, and are consumed by every action that reaches the app; a request that fails before sending anything reports `dispatchState: "not_started"` and keeps them valid. An Accessibility failure counts as sent unless Accessibility rejected the request before the app saw it (unsupported attribute or action, illegal argument). Restarts invalidate all references. Screen captures exclude window shadows, preserve the selected window’s coordinate geometry, and are capped to 1600 pixels on the longest side and 8 MiB encoded. Complete JSON replies are capped at 16 MiB: oversized observations omit screenshots first, then truncate accessibility nodes, while preserving action outcomes. Screenshots are optional and failure is reported in `screenshotError` without discarding accessibility state. macOS 26 uses the screenshot-specific capture API; the macOS 14/15 API path includes only the selected window in a display filter and crops its exact frame. That older path requires the window to fit within one display and reports `capture_geometry_unavailable` otherwise.
1414

0 commit comments

Comments
 (0)