diff --git a/docs/plans/2026-11-13-microvm-sessions.md b/docs/plans/2026-11-13-microvm-sessions.md new file mode 100644 index 000000000..e19817e21 --- /dev/null +++ b/docs/plans/2026-11-13-microvm-sessions.md @@ -0,0 +1,151 @@ +# Sessions in Cloud Hypervisor VMs + +**Goal:** every session runs in its own virtual machine instead of a Docker +container. An idle session is saved to disk with every process still +running and restored on its next use, for every agent and session type. +Docker and Sysbox leave the server once nothing runs on them. + +**Status:** approved by the owner on 2026-11-13. Decisions dated 2026-11-13 +are the owner's. Everything marked *Open* is settled by the first +measurement (work item 1). + +## Decisions (2026-11-13) + +- The VM monitor is [Cloud Hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor). + It can share a host folder into a VM (virtio-fs), save and restore a + running VM, and accept live migration from its two previous releases. +- The member home stays one folder on the server that every VM of the + member reads and writes live. Logins, token refreshes and agent settings + keep a single copy, as today ([environment-home.md](../environment-home.md), + [teams.md](../teams.md#agent-accounts)). The VM reaches it through the + in-VM helper, not virtio-fs: a restored VM's virtio-fs mounts return + `EBADF`, and remounting fails while any process holds a file or working + directory in the share (measured with Cloud Hypervisor v54.1 and + virtiofsd 1.10). +- The checkout moves onto the VM's own disk. Git work on it runs inside + the VM; the server keeps the bare repo, session branches, diff history, + evidence and transcripts. The server never mounts a VM's disk. +- Work lands on `main` behind one server setting, `session-runtime` + (`docker` until the cutover, then `vm`). Docker stays the default until + work item 13. + +## A session VM + +| Part | What it is | Lifetime | +| --- | --- | --- | +| Kernel | An Aether-built guest kernel shipped with each server release | Per release | +| Helper disk | A small read-only disk holding the in-VM helper, shipped with each server release | Per release; a saved VM keeps the one it booted with | +| Base disk | The session's image (standard, a saved member environment) flattened to a read-only raw disk | Until the image is swept | +| Root disk | A qcow2 copy-on-write layer over the base disk, so all sessions share one host cache of the base | Disposable; rebuilt on a cold start | +| Data disk | `/workspace`, `/repos/`, `/tmp`, `/var/lib/docker` | Until the session is deleted | +| Home | The member's home folder on the server, reached through the helper's file service | The member's | +| Saved state | The VM's memory and device state while it sleeps | Until the next wake | + +Processes inside the VM: + +- An in-VM helper, `aether-internal guest`, runs as PID 1. It mounts the + filesystems, configures the network from the kernel command line, starts + `dockerd` for sessions that nest Docker, runs the setup script and then the + session's command as the session user, and reaps orphans. +- The helper serves one API to the server over the VM's host socket + (vsock): start a process with argv, environment, user, directory and a + terminal or pipes; stream its input and output; resize; signal; wait; and + attach to the session's main process. It also forwards the coordination + and signing sockets into the VM and reports memory, CPU, process count and + out-of-memory kills. +- Everything Aether runs in a session today (`dev-exec`, `acp-keeper`, + the agent, the MCP helper) runs unchanged on top of that API. + +The VM monitor of each session runs in its own systemd unit inside +`aether.slice`, outside the server's process tree, so sessions keep running +through server restarts and updates. The server finds them again by run ID. + +## Lifecycle + +| Today (Docker) | VM | +| --- | --- | +| create, start | Build the root and data disks, start the VM; the helper runs the setup script, then the command | +| `docker exec`, attach | The helper API | +| pause on close | Save the VM to disk; the monitor exits | +| idle sleep (`docker stop`) | Save the VM to disk; the monitor exits | +| wake | Restore the saved VM. If it cannot be restored (another monitor or kernel version, a deleted file), cold-start from the disks and resume the conversation, as today | +| stop, destroy | Stop the VM; delete the root disk; the data disk follows the checkout retention | +| `docker commit` (Save environment) | Save the environment VM's root disk as a new base disk | + +After a restore the helper corrects the clock and the server reconnects its +streams. Sessions sleep only between turns, so no model request is cut. + +## Network + +Each VM gets one tap device on an Aether bridge. VMs cannot reach each +other. A session with egress control reaches only the server's egress +proxy, which recognises a VM by its address. The cloud instance metadata +address is blocked. Port forwards dial the VM's address or relay through the +helper. The development browser runs inside the session VM. + +## Checkout inside the VM + +Until work items 6 and 7 land, the host checkout, home and caches are +shared into the VM with virtio-fs, so the git engine works unchanged; a VM +with such shares cannot be saved. Work item 6 +then moves the checkout onto the data disk: + +- A prepared disk per workspace and base commit; each session's checkout is + a copy-on-write clone of it. +- The helper watches files and builds diff snapshots, the live patch, Files + reads, writes and uploads, the final commit, subsession fetches and + evidence. It sends git objects to the server; the server fetches session + branches from the VM. +- A sleeping or ended session is shown from its last diff snapshot. Writing + to one wakes it. Cleanup steps that need the checkout run before the VM's + data disk is discarded, or in a VM started briefly for them. + +## Resources + +- vCPUs and memory come from the session's limits. Guest memory returns to + the host through free page reporting and page-cache reclaim inside the + guest. +- The launch capacity check counts VM memory. +- Hosts need `/dev/kvm`: bare metal, or a cloud instance with nested + virtualization (on AWS, the Intel 7th and 8th generation families listed + in the [EC2 guide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-ec2-nested-virtualization.html)). + Saved VMs need local NVMe storage to wake quickly. + +## Work items + +Each item is one pull request through the merge gate, except item 6, which +may take several. Items in a wave run in parallel. + +| # | Wave | Item | Owns | +| --- | --- | --- | --- | +| 1 | 1 | Measure on a KVM host | Scripts and results only, no product code | +| 2 | 1 | VM engine: the `runtime.Runtime` backend, monitor units, save and restore, limits, stats, capacity | `internal/runtime/vm` | +| 3 | 1 | In-VM helper and its API | `internal/guest`, `aether-internal guest` | +| 4 | 1 | Kernel, base disks from images, copy-on-write root disks | `internal/vmimage`, kernel build | +| 5 | 2 | Network: taps, isolation, egress, metadata block, port forwards | `internal/runtime/vm` network, `internal/egress` | +| 6 | 2 | Checkout and live sync inside the VM | `internal/gitengine`, the helper's git service, `internal/sshd/sync.go`, `internal/overlay` | +| 7 | 2 | Home through the helper's file service, caches on the VM disk, Isolated tier | mounts, `internal/scheduler/scratch.go`, ownership, the helper's file service | +| 8 | 2 | Sleep and close save the VM; wake restores; fallback; agent wakeups held by the server | `internal/scheduler` sleep paths | +| 9 | 2 | Environment terminal and Save environment, assistant, agent updater, dictation, browser, cleanup helper, sudo and tiers | their scheduler and runtime files | +| 10 | 3 | Install, update, storage, backups, AWS host guide | `cmd/aether-server`, `internal/serversetup`, docs | +| 11 | 3 | Integration, dashboard end-to-end and real-agent suites on VMs; CI with KVM | tests, `.github` | +| 12 | 3 | Dashboard, CLI and docs wording | `web/`, `cmd/aether`, `docs/` | +| 13 | 3 | Security review, cutover, removal of Docker and Sysbox | everything Docker-specific | + +The pull requests form one stack, each based on the one before it, and +merge bottom-up: this plan, then item 4, item 3, item 2, then the wave 2 +and wave 3 items in the order their dependencies need. Each pull request +passes the merge gate on its own diff: an adversarial review by a separate +session, every real Greptile finding fixed and the rest answered, and a +green `CI gate`. Dashboard changes reuse the existing surfaces and change +wording before they add anything. + +Items 2, 3 and 4 meet at two Go interfaces. Item 3 owns the helper API +definition and its client; item 4 owns the image store (base disk for an +image, new root disk for a session, kernel path). Item 2 builds on both. + +## Open + +- Host memory per awake session, with free page reporting on. +- Save and wake times on local NVMe, and whether nested Docker containers + and a dev server survive a save.