[ilovecrayons]

Stacked on #889.

Sessions are moving into Cloud Hypervisor VMs (plan, work item 4). Before this, nothing turned today's images into disks a VM boots, and there was no guest kernel. Now:

• Guest kernel (images/kernel/): a miniconfig for Linux 6.18.63 LTS applied with allnoconfig, so every option is deliberate, and build.sh, which fails if any listed option does not reach .config. make kernel builds dist/aether-vmlinux-x86_64 in debian:trixie-slim pinned by digest with packages from snapshot.debian.org 2026-11-10. The release builds it in a new kernel job and ships it as a fifteenth asset in the signed manifest.
• Image store (internal/vmimage), the interface agreed with item 2:
  • BaseDisk(ctx, ref) pulls an image anonymously, as Docker does today, flattens it, and makes a read-only 64 GiB sparse ext4 base disk cached by config digest. It returns the image's User, Env, WorkingDir and Labels. Concurrent first uses share one pull on its own context, so a cancelled launch does not fail the others.
  • aether/ references are never pulled. They are bases saved with SaveBase, or, during the transition, come from docker save through runtime.Docker.SaveImage (new).
  • NewRootDisk makes a qcow2 root disk backed by a hard link <path>.<base id>.base to the base. The link pins the base until the run dir is deleted, so sweeping needs no callback from the engine. A failed rebuild leaves the old root disk and its link as they were.
  • NewDataDisk makes an empty ext4 data disk labelled aether-data and refuses to overwrite one.
  • HelperDisk makes a read-only ext4 disk holding the in-VM helper as /init, with /dev /proc /sys /mnt, once per binary digest, and links it into the run dir.
  • SaveBase saves a stopped VM's root disk as a new base.
  • Tag, Untag, Tags, LookupBase, Sweep and Kernel cover the runtime image methods.

Decisions and why

• go-containerregistry for pulling and flattening. It is pure Go and needs no daemon. mutate.Extract applies whiteouts and opaque directories, refuses unsafe paths and verifies layer digests. It is what crane, ko and cosign use. Its daemon package is not used: Docker-only images arrive as a docker save tar read with its tarball package.

• ext4 made by mkfs.ext4 -d from a staging directory, extracted through os.Root, so no entry, symlink or hard link reaches outside it. hcsshim's tar2ext4 writes without staging but marks the filesystem read-only (RO_COMPAT_READONLY), and a root disk must be writable. Device files are dropped under /dev (the guest mounts devtmpfs there) and refused elsewhere. Limits: 48 GiB and 2M entries per image. Owners, set-user-ID bits, file capabilities and mtimes are kept. mkfs marks the inode tables of a regular file zeroed, so the guest never writes them out (ITABLE_ZEROED on 256 of 257 groups).

• qcow2 over a raw base, not reflink copies. Measured on the KVM host (m8id.4xlarge, CH v54.1, XFS /srv, 4-vCPU guest, standard image, host cache evicted per file, direct=off):

  	qcow2 + backing	reflink raw
  create a root disk	5-7 ms	3 ms
  cold tar of /usr in the guest	4.7 s	3.9 s
  4k random read QD16	86-104k IOPS	83-94k IOPS
  4k random write QD16	17k IOPS	67k IOPS
  on an ext4 host	works (boot verified)	EOPNOTSUPP
  host page cache of the base	one copy for all sessions	one per session

  qcow2 is the only format that works everywhere. Heavy writes go to the raw data disk (/workspace, /tmp, /var/lib/docker), so the root disk's write gap matters little. With direct=on, CH reads qcow2 backing files synchronously (9k IOPS), so the engine keeps the default direct=off.

  Item 1 measured independently with fio in a 2-vCPU guest and CH direct=on, against a 4 GiB random-filled base:

  	qcow2 + backing	reflink raw	full copy
  create	5 ms, 196 KiB	1 ms	1.0 s + 3.5 s sync
  4k random read QD32	11.5k IOPS	175k IOPS	156k IOPS
  4k random write QD32 (first, again)	3.2k, 3.4k IOPS	19.8k, 63.5k IOPS	140k, 144k IOPS
  1M sequential read / write	411 / 274 MiB/s	8752 / 2632 MiB/s	9022 / 4023 MiB/s

  Those qcow2 figures are the synchronous direct=on backing path above; the engine does not run qcow2 that way. A reflink copy also fragmented to about 16k extents under random writes.

  The coordinator settled it (2026-11-13): qcow2 over a raw base with direct=off, one path, no reflink path. It works on ext4 hosts, and the shared host page cache of the base is the density win the move to VMs optimises for. Root-disk writes are rare (package installs). CH docs: --disk ... image_type=qcow2,backing_files=on (vmm/src/config.rs, block/src/formats/qcow/backing.rs at v54.1, raw backing supported). Two VMs on one base run together without a lock conflict. Saving a root disk as a base with qemu-img convert took 4.0 s for a 4.1 GB result.

• Helper disk, not an initramfs, for the in-VM helper (item 3) as PID 1. With a 55 MB init in a 1 GiB guest, the initramfs left 53.7 MB of Shmem that drop_caches cannot free (MemAvailable 812 MB, CH RSS 801 MB at 12 s). The helper disk left 0 Shmem (MemAvailable 909 MB, CH RSS 589 MB). Base disks stay independent of the server version.

• Host tools the server now needs on VM hosts: mkfs.ext4 (e2fsprogs) and qemu-img (qemu-utils). Item 10 owns installing them.

Verification

• go test ./internal/vmimage: extraction refuses .. paths, writes through absolute and relative symlinks, directories through symlinks, hard links outside or through symlinks, device files outside /dev, duplicates, out-of-range owners, oversize and truncated entries. As root it keeps other owners, set-user-ID bits and security.capability. The store test pushes a two-layer image (whiteout, opaque dir, hard link across layers) to an in-memory registry and runs four concurrent BaseDisk calls, which fetch each layer once. It then checks the disk with debugfs, the qcow2 backing with qemu-img info, a root-disk write reaching the saved base (qemu-io), tags, and the sweep with and without a pin. It also covers the Docker source, a wrong architecture, the data disk and the helper disk. Passed on moss (no qemu-img, so those tests skip) and on the KVM host as ubuntu and as root. CI now installs qemu-utils if missing so they run there too.
• make kernel twice on the KVM host: identical SHA-256 84ccf6c354fc66d032fc9329ba691de0a08a007ca6a34be41c8ab7c6ddfc23e4, about 1.5 min each.
• Docker's check-config.sh against the kernel config: every "Generally Necessary" option is enabled. The only optional options missing are swarm overlay networks, macvlan/ipvlan, btrfs, SELinux and AppArmor.
• Real VM on the KVM host, TestKVMStandardImageRunsDocker (-tags kvm, as root, kernel from make kernel, image ghcr.io/3xdevops/aether-standard:v0.8.7-beta.14). The VM boots from disks the store built: helper disk (the static test binary as /init), qcow2 root, raw data.
  • Base disk built from the registry in 18.7-19.9 s; it uses 2549 MiB on the host.
  • Root disk created in 42-45 ms, including the qemu-img process. qemu-img create alone takes 5-7 ms.
  • Root switch at 0.35 s, dockerd ready at 0.88-0.91 s.
  • docker run of an imported image worked, a -p 8080:80 mapping answered, and the iptables nat DOCKER chain was set up. docker info: cgroup v2, overlayfs storage.
  • /usr/bin/sudo kept its set-user-ID bit.
  • SaveBase while the VM ran was refused: qemu-img: Could not open '.../root.qcow2': Failed to lock byte 100. CH's disk lock protects it.
  • After the VM stopped, SaveBase took 0.6 s (2082 MiB). The root disk used 2 MiB after the run.
• Real VM, hand-made disks, before the store existed: an inner container pulled busybox from Docker Hub through both NATs, using a tap with NAT on the host and the kernel's ip= for eth0.
• Real VM, device check with this kernel and CH flags --fs, --vsock and --balloon free_page_reporting=on. Bound drivers: virtio_balloon (reporting feature bit 5 set), virtio_blk, virtio_console, virtio_mem, virtio_net, virtio_rng, virtio_rtc, virtiofs and vmw_vsock_virtio_transport. /dev/vsock was present, a virtio-fs mount read a host file, and /proc/filesystems lists ext4, xfs, overlay and virtiofs.
• The transition path against real Docker 29 (containerd image store), with a throwaway test that is not committed: an aether/ image made by two docker commits became a base disk. The file deleted in the second commit stayed deleted, and su stayed root-owned 04755.
• The Greptile CLI is not installed in this session's container (greptile: command not found), so only the Greptile bot reviews this PR.

Open

• page_reporting.page_reporting_order=0 belongs on the guest command line (item 1's measurement); item 2 owns the command line.
• The kernel is x86_64 only.

This PR was made with the help of Claude Code.

Opened from Aether run run-w212tvpy6z

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The reviewed update appears safe to merge; all four earlier findings are fixed.

Summary

Adds the guest kernel build and the image store that creates VM boot disks.

• A reproducible x86_64 Linux kernel build now ships with each release.
• Container images now become reusable ext4 base disks.
• Session VMs boot from shared image bases and separate run disks.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Registry[Registry image] --> Shared[Shared pull and build]
  Docker[Local Docker image] --> Shared
  Shared --> Base[Cached read-only base]
  Base --> Pin[Base-specific hard link]
  Pin --> Root[Writable qcow2 root]
  Root --> Save[Save stopped root]
  Save --> Base
  Binary[Helper binary] --> Helper[Read-only helper disk]
  Kernel[Pinned kernel build] --> VM[Session VM]
  Root --> VM
  Helper --> VM
  Data[Writable data disk] --> VM

Loading

_{Reviews (2) · Last reviewed commit: "fix(vmimage): keep a root disk paired wi..." · Reviewed by Greptile}