Repository navigation
feat(guest): run a session VM's helper as PID 1 with an API over vsock - #891
Conversation
Sessions are moving into Cloud Hypervisor VMs (docs/plans/ 2026-11-13-microvm-sessions.md). Docker gave a container its init, exec, attach and stats; a VM has none of them. Docker also cannot reattach an exec's streams after a server restart. aether-internal guest is PID 1 of every session VM. It switches from the read-only helper disk to the session's root disk, mounts the system filesystems, the virtio-fs share and the data disk named on the kernel command line, configures the network, relays the coordination and signing sockets to the host over vsock, reaps orphans and forwards PID 1's signals to the main process. Its gRPC API (internal/guest/guestpb) boots the session (dockerd, the setup script, then the main process), spawns processes on a terminal or pipes, and keeps each stream's output under byte offsets, so a restarted server or a restored VM reattaches where it left. internal/guest is the client the VM engine builds on.
Go's poller registered /proc/self/mountinfo and took the kernel's one-shot change mark first, so the helper never saw virtio-fs drop the bind over /run/aether/coord3.sock after the host re-created it.
TestKVMHelper, behind the kvm build tag, boots an image from internal/vmimage disks with the test binary as PID 1 and checks mounts, a terminal, both forwarded sockets including a re-created coordination socket, reattaching from a new client, an out-of-memory kill, the process limit, signal forwarding and power off.
The coordination client retries a refused connection, but treats one cut off after its request was sent as possibly applied. The helper accepted, failed to reach the restarting server and closed, so a call made during a restart failed. A relay now waits up to 60 s for the host socket, as long as the client would retry, and closes a pipe process's input when it exits.
A failed bind or socket stopped the helper from making the ones after it, so one missing source could leave a session without its checkout. The helper now records the failure in Info's boot_error and goes on. The client also refuses a setup exit code outside 0-255 and caps the message of a failed spawn.
fdc1d6e to
5f10e95
Compare
 [High impact] The changes add process management and host-service access to every session VM that uses the guest helper. Fix process identity, command lookup, input cancellation, and lossless output before merging.
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| t.exited = append(t.exited, p.id) | ||
| for len(t.exited) > maxExited { | ||
| delete(t.byID, t.exited[0]) | ||
| t.exited = t.exited[1:] |
There was a problem hiding this comment.
Once 33 non-main processes have exited, retire removes the first ID from the only map that spawn checks. A delayed repeat of a lost Spawn request can then run the command again, despite the single-use promise. Existing handles also resolve to the replacement process. Keep used-ID markers after discarding old output and exit records.
Knowledge Base Used: Environment construction and execution
Prompt To Fix With AI
This is a comment left during a code review.
Path: internal/guest/helper/process_linux.go
Line: 357-360
Comment:
**Old IDs run commands again**
Once 33 non-main processes have exited, `retire` removes the first ID from the only map that `spawn` checks. A delayed repeat of a lost `Spawn` request can then run the command again, despite the single-use promise. Existing handles also resolve to the replacement process. Keep used-ID markers after discarding old output and exit records.
**Knowledge Base Used:** [Environment construction and execution](https://app.greptile.com/-/switch-tenant?tenant=aethercode&next=%2Fcustom-context%2Fknowledge-base%2F3xdevops%2Faether%2F-%2Fdocs%2Fenvironment-construction-and-execution.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
| func drainAfter(f *os.File, done <-chan struct{}) { | ||
| <-done | ||
| _ = f.SetReadDeadline(time.Now().Add(exitDrain)) |
There was a problem hiding this comment.
The exit deadline also applies to lossless streams. A command can fill outputLimit, leave its final bytes in the pipe, and exit. If the reader resumes after one second, room wakes but the expired deadline makes pump close the stream without reading those bytes. The caller receives truncated output followed by normal EOF. Preserve queued lossless bytes when applying the timeout for descendants that keep a pipe open.
Knowledge Base Used: Environment construction and execution
Prompt To Fix With AI
This is a comment left during a code review.
Path: internal/guest/helper/output_linux.go
Line: 150-152
Comment:
**Lossless output loses its end**
The exit deadline also applies to lossless streams. A command can fill `outputLimit`, leave its final bytes in the pipe, and exit. If the reader resumes after one second, `room` wakes but the expired deadline makes `pump` close the stream without reading those bytes. The caller receives truncated output followed by normal EOF. Preserve queued lossless bytes when applying the timeout for descendants that keep a pipe open.
**Knowledge Base Used:** [Environment construction and execution](https://app.greptile.com/-/switch-tenant?tenant=aethercode&next=%2Fcustom-context%2Fknowledge-base%2F3xdevops%2Faether%2F-%2Fdocs%2Fenvironment-construction-and-execution.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| program, err := lookPath(spec.Argv[0], envValue(env, "PATH")) | ||
| if err == nil { | ||
| err = t.start(p, spec, program, env, cred) |
There was a problem hiding this comment.
lookPath checks relative paths from the helper's /, before ForkExec changes to spec.Dir. For example, Argv: []string{"./script"} with Dir: "/workspace" checks /script and returns 127 even when /workspace/script exists. Relative entries in PATH have the same problem. Resolve and check these paths against the requested working directory.
Prompt To Fix With AI
This is a comment left during a code review.
Path: internal/guest/helper/process_linux.go
Line: 102-104
Comment:
**Relative commands fail to start**
`lookPath` checks relative paths from the helper's `/`, before `ForkExec` changes to `spec.Dir`. For example, `Argv: []string{"./script"}` with `Dir: "/workspace"` checks `/script` and returns 127 even when `/workspace/script` exists. Relative entries in `PATH` have the same problem. Resolve and check these paths against the requested working directory.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| stop := context.AfterFunc(ctx, func() { _ = p.stdin.SetWriteDeadline(time.Now()) }) | ||
| defer stop() |
There was a problem hiding this comment.
defer stop() does not wait for an already-started cancellation callback. If a write finishes as its context ends, the callback can run after stdinMu is released and expire the next write's freshly set deadline. It also reads p.stdin while closeInput may replace it. Wait for any running callback before releasing the lock, as the existing input writer does.
Prompt To Fix With AI
This is a comment left during a code review.
Path: internal/guest/helper/process_linux.go
Line: 410-411
Comment:
**Canceled writes break later input**
`defer stop()` does not wait for an already-started cancellation callback. If a write finishes as its context ends, the callback can run after `stdinMu` is released and expire the next write's freshly set deadline. It also reads `p.stdin` while `closeInput` may replace it. Wait for any running callback before releasing the lock, as the existing input writer does.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| if free > 0 { | ||
| return min(free, 32<<10) | ||
| } | ||
| <-changed |
There was a problem hiding this comment.
A full lossless stream waits only for another acknowledgement. If its process exits without a reader, the file deadline cannot wake room. Even after retire removes the process record, the blocked pump keeps its buffer and pipe open forever. Add a retirement path that wakes and stops pumps once their output is no longer available by ID.
Knowledge Base Used: Keep container faults from degrading server availability
Prompt To Fix With AI
This is a comment left during a code review.
Path: internal/guest/helper/output_linux.go
Line: 58
Comment:
**Retired commands keep resources**
A full lossless stream waits only for another acknowledgement. If its process exits without a reader, the file deadline cannot wake `room`. Even after `retire` removes the process record, the blocked pump keeps its buffer and pipe open forever. Add a retirement path that wakes and stops pumps once their output is no longer available by ID.
**Knowledge Base Used:** [Keep container faults from degrading server availability](https://app.greptile.com/-/switch-tenant?tenant=aethercode&next=%2Fcustom-context%2Fknowledge-base%2F3xdevops%2Faether%2F-%2Freverts%2Fincident-mitigation_136-20260915-container-availability-14cb623.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| @@ -0,0 +1,383 @@ | |||
| package helper | |||
There was a problem hiding this comment.
This test file has no Linux build constraint. It imports unix.Prctl and uses helpers defined only in _linux.go files, so the documented go test ./internal/guest/... command fails to compile on macOS despite the package's !linux stub. Rename it to helper_linux_test.go or add //go:build linux.
Prompt To Fix With AI
This is a comment left during a code review.
Path: internal/guest/helper/helper_test.go
Line: 1
Comment:
**Guest tests fail on macOS**
This test file has no Linux build constraint. It imports `unix.Prctl` and uses helpers defined only in `_linux.go` files, so the documented `go test ./internal/guest/...` command fails to compile on macOS despite the package's `!linux` stub. Rename it to `helper_linux_test.go` or add `//go:build linux`.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| if err := executable(candidate); err == nil { | ||
| return candidate, nil | ||
| } | ||
| } | ||
| return "", &execError{code: 127, err: fmt.Errorf("%s: executable file not found in $PATH", name)} |
There was a problem hiding this comment.
lookPath discards permission errors and always returns 127 when no candidate succeeds. If a command exists on PATH but has no execute permission, the documented result is 126, not “not found.” Keep the permission error while searching and return it if no executable candidate is found.
Prompt To Fix With AI
This is a comment left during a code review.
Path: internal/guest/helper/process_linux.go
Line: 300-304
Comment:
**Permission errors report missing commands**
`lookPath` discards permission errors and always returns 127 when no candidate succeeds. If a command exists on `PATH` but has no execute permission, the documented result is 126, not “not found.” Keep the permission error while searching and return it if no executable candidate is found.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Stacked on #890.
What was wrong
Sessions are moving into Cloud Hypervisor VMs (
docs/plans/2026-11-13-microvm-sessions.md). Docker gives a container its init,docker exec,docker attachand stats; a VM has none of them. Docker also cannot reattach an exec's streams after a server restart (internal/runtime/managed_exec.go:71-85), so a restarted server shows a recovered terminal as unavailable.What happens now
aether-internal guestis PID 1 of every session VM (internal/guest/helper):serial=rootdisk on/mnt, moves/dev,/proc,/sysinto it and switches root, staying PID 1./dev/pts,/dev/shm, cgroup2 and/run, then the virtio-fs share and the data disk binds named on the kernel command line. Sources are opened beneath their volume (RESOLVE_BENEATH), so a symlink planted on the persistent data disk cannot redirect a mount.loand the VM's network device by netlink, plus hostname,/etc/hostsand/etc/resolv.conf./run/aether/coord3.sock,/run/aether-signing/agent.sock) to the host over vsock. While the server restarts, a relay holds the connection for up to 60 s, as long as the coordination client retries a refused one, so the call is delivered once instead of failing as "sent, may have applied". A socket inside a read-only share gets the helper's socket bound over it, and bound again when virtio-fs drops the bind after the server re-creates its socket.HUP INT QUIT TERM USR1 USR2 ALRM WINCHto the main process likedocker-init, and enforces the session's process limit with/sys/fs/cgroup/session/pids.max. dockerd gets--cgroup-parent /session, so nested containers count too.Boot(dockerd, the setup script once per boot, then the main process, with Docker's failure rules),Spawnwith caller-chosen single-use IDs, offset-basedReadandWrite,Resize,Signal,Status,Stats,SetClock,PowerOff. Output stays in the VM under byte offsets, so a restarted server, or a client cut off by a save and restore, attaches by ID and reads on.internal/guestis the client the VM engine (item 2) imports:Dial(Cloud Hypervisor'sCONNECThandshake, redials by itself),Pingwith the protocol check,Boot,Spawn,Process(id)for reattach, anAttachmentthat satisfiesruntime.Attachment,Stats,SyncClock,PowerOff, andConfig.KernelArgs, which builds and size-checks the kernel command line. Everything the guest sends is treated as untrusted: reply sizes, offsets and exit codes are checked and strings capped.The API is
internal/guest/guestpb/guest.proto;make protoregenerates it with pinnedbuf,protoc-gen-goandprotoc-gen-go-grpc, needing only Go.acpkeeper.OpenPTYis exported for reuse instead of copied. New modules:github.com/mdlayher/vsockv1.3.0 (chosen over v1.4.0, which would bump fivegolang.org/xmodules) andgithub.com/vishvananda/netlink.docs/vm-guest.mddescribes the helper.How I verified it
Unit tests (
go test ./internal/guest/..., run as a non-root user) drive the real helper service through the real client over a fake hybrid-vsock socket: pipe and terminal processes, resize, input deadlines, a client restart that reattaches and replays, lossless backpressure past the window, spawn failures (127/126, reused IDs, reserved IDs), signals, orphan reaping,Bootonce-only and setup failure, stats, and the host-only listener.TestKVMHelper(-tags kvm,docs/testing.md) bootsghcr.io/3xdevops/aether-standard:v0.8.7-beta.14frominternal/vmimagedisks with the test binary as PID 1 and drives it through the client: mounts, a terminal with resize, both forwarded sockets, a call made while the coordination socket is down for 2 s and then re-created, reattaching from a new client, an OOM kill, the process limit setting,kill -USR1 1reaching the main process, and power off. It passes on the KVM host in 28 s (helper answers about 1 s after the monitor starts).Before that test existed, I drove the helper by hand on the KVM host (Cloud Hypervisor v54.1, item 4's 6.18 kernel, the same image, 2 GiB, 2 vCPUs):
Ping1.05 s after the monitor starts,boot_errorempty;Bootwith dockerd and a setup script takes 0.75-1.7 s;/proc/1/commisinit.nosuid,nodev,rowhere asked, a file bind at/usr/local/bin/aether-internal, data binds at/tmpand/var/lib/docker.ttyis/dev/pts/0,stty sizefollows a resize from 24x80 to 40x132, Ctrl-C endssleep($?130),exit 5returns 5.1000:1000; nested Docker 29.1.3 loads an image and runs a container in/session/<id>./run/aether/coord3.sockand/run/aether-signing/agent.sockreach their host sockets from uid 1000. After the host socket is unlinked and re-created, as a server restart does: one refused connect, then the helper has bound again and the retry gets the new server's answer.Stats.OOMKills1, the kernel log names it.sleep &againstPidsLimit256:pids.currentstays at 235,pids.eventscounts 39 refusals.kill -USR1 1inside the VM ends the main process with 138, and its exit code is read afterwards.SyncClockput it within 2 ms, and a new client reattached to the process that kept running.PowerOffreturns, and the console ends withreboot: Power down.Item 2 also ran its engine against this branch end to end on the same host.
make fmt-check(pinned toolchain),make vet,make lint,make public-auditandgo test -race ./internal/guest/...pass. The Greptile CLI is not installed in this session:greptile: command not found.Open
no-new-privilegesper process (Docker sets it when a session has no sudo) belongs to item 9's tiers.This PR was made with the help of Claude Code.
Opened from Aether run run-ygq840q53x