Skip to content

Bot panel: a bot's commands and browsing under the Computer tab - #96

Open
maybegrim wants to merge 1 commit into
DuskBotAI:devfrom
maybegrim:panel-steps
Open

maybegrim wants to merge 1 commit into
DuskBotAI:devfrom
maybegrim:panel-steps

Conversation

@maybegrim

@maybegrim maybegrim commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Closes #88. This follows #92, which you closed because the chat stays the conversation. It moves the same idea into the bot's panel, as you suggested there, and leaves the chat exactly as it is on dev: no chat file is touched and there is no switch.

The bot panel's Computer tab gains a short list, "Commands and browsing", under the bot's browser or screen:

  • A running command is a small window: the command, an elapsed timer, and the last lines of its output as they arrive, with the bot's pixel cat peeking over the top edge.
  • A finished command folds to one line with its time and a tick or cross. Opening it shows the output. A command too long for its title line is spelled out in full inside.
  • Browsing is one entry per stretch of browser steps: the site, the step count, and the steps in plain words ("Clicking on the page"). The wording never shows what was typed or which element was clicked. On this computer the live browser sits right above the list, so the entry has no picture of its own; where there is no live browser it shows a small picture of the page, with a larger view.
  • Only the latest turn. The list holds what the latest turn ran, not a history of the conversation, and the next turn replaces it. A turn that ran no command and did no browsing lists nothing.
  • Group chats. Each member's panel lists that member's own latest turn.

Server side, kept small:

  • tool.ms on a finished tool message, for the finished line's time.
  • The live output tail goes from 2 lines to 12 (server/harness/live-steps.ts).
  • Group turns now send the speaking member's browser frames on the group's thread, through the same guarded capture 1:1 turns use (extracted as browserPreviewCapture). Frames flow only once that turn's own browser call has claimed the browser, and a group turn still posts no screenshot message when it ends. server/index.ts stays at 16,896 lines.

No new dependency. New strings are in en.json only.

How it was checked

  • pnpm exec vitest run src/components/bot-panel src/components/room-panel src/components/live-cards src/state/command-tails.test.ts src/state/browser-snapshots.test.ts src/state/turn-status.test.ts src/components/type-scale.test.ts server/harness/live-steps.test.ts server/direct-screen-settlement-api.test.ts server/screen-frame-source.test.ts scripts/testing/group-browser-frames.e2e.test.ts: 35 files, 292 tests pass.
  • src/components/bot-panel/PanelSteps.test.ts is new and covers the list: latest turn only, newest first, the running entry open, no picture beside the live browser, one member's steps in a group.
  • scripts/testing/group-browser-frames.e2e.test.ts is new: a real isolated server, the fake Claude engine and the pinned agent-browser on a local page. It passes here and fails on untouched dev, where no frame reaches a group's thread.
  • By hand in isolated fixtures (scripts/control-dusk.ts ui launch, never live data), in a 1:1 chat with the panel open: a command streaming output and folding with the fake Codex engine; a browser run on a local page with the fake Claude engine; a new turn replacing the previous turn's entries.
  • pnpm lint and pnpm i18n:check pass.
  • Screenshots follow in a comment on this PR.

Not seen or not covered:

  • The group chat panel and the picture on a remote computer are covered by tests only; I have not looked at them on screen.
  • Windows, Linux, the phone app, the real desktop app.
  • The fold animation by eye, and reduced motion.
  • After a page reload a running command shows no earlier output until it prints more, because the tail comes from live frames.
  • In a short window the list sits below the live browser, so the panel scrolls to reach it.
  • tool.ms has no server test of its own.

Checks that already fail on dev

All on macOS, on an untouched checkout of dev at 6c14850 as well as on this branch. None is in a file this PR touches.

  • pnpm typecheck: the same five errors, in apps/dashboard/server/clouds/ai-stash.test.ts, server/cloud-mac.e2e.test.ts, server/mac-control/cloud.test.ts, server/mac-control/seal.test.ts and server/group-local-vm.e2e.test.ts.

  • Source guards (the step CI stops at): src/locales/sentence-case.test.ts fails on chat.event.peerReplied ("replied"), and src/locales/server-lines.test.ts fails on a line in shared/computer-unreachable.ts.

  • server/index.test.ts: five cases fail the same way with and without this change: "buzzes when a turn dies before it can start", "creates team computers only with consent…", "reports a failed routine once, not twice", "shares one team computer across direct and room turns…", and "shows a card continuation that cannot start as the root's failed turn…".

  • I did not run the whole two-hour suite on this base; I ran the suites above.

  • pnpm typecheck, pnpm lint and pnpm test pass (lint passes; typecheck and tests fail only where dev already does, see above)

  • New behaviour has a test (except tool.ms, noted above)

  • No secrets, personal data or local paths in the diff or screenshots

@maybegrim

Copy link
Copy Markdown
Contributor Author

Screenshots, all taken in isolated fixtures with the test engines. The bots, their canned replies, the canned command result and the /Users/ada/acme path are the fixtures' own.

Before

The Computer tab on dev while a command runs.

Before: the Computer tab while a command runs, with no list

After

A running command, with output arriving

The Computer tab with a running command and its live output

The finished command, opened

A finished command opened to show its output

Browsing, under the live browser (panel scrolled down)

A browsing entry under the live browser while it runs

The finished browsing entry, opened (panel scrolled down)

A finished browsing entry opened to show its steps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant