diff --git a/.gitignore b/.gitignore index 41099ea7..81176990 100644 --- a/.gitignore +++ b/.gitignore @@ -48,6 +48,10 @@ Thumbs.db .env.local .env.*.local +# Local Compose customisation (host-specific, not part of the project) +docker-compose.override.yml +docker-compose.override.yaml + # State files (local to each machine) .claude/ralph-loop.local.md diff --git a/docker/.env.example b/docker/.env.example index 70b128f7..12770034 100644 --- a/docker/.env.example +++ b/docker/.env.example @@ -13,12 +13,12 @@ TZ=Australia/Perth # Name of the account that runs Codeman and all local CLI sessions. Changing # this value rebuilds the image with a matching account. -CODEMAN_RUNTIME_USER=opencode +CODEMAN_RUNTIME_USER=codeman # Required. Persistent Codeman application data, CLI credentials, and session # state are stored here on the host and mounted at the runtime account's home # directory in the container. -CODEMAN_APPDATA_PATH=/mnt/user/appdata/Coding/codeman +CODEMAN_APPDATA_PATH=/mnt/user/appdata/codeman # Optional. Absolute host path of this Codeman checkout, mounted at # /opt/codeman so App Settings -> Updates can update Codeman in place. The Bash @@ -29,8 +29,8 @@ CODEMAN_APPDATA_PATH=/mnt/user/appdata/Coding/codeman # Required for Docker cases. This must be an absolute path on the Docker host. # Codeman and each isolated case use this same path, so it cannot be a -# container-only path such as /home/opencode/codeman-cases. -CODEMAN_CASES_PATH=/mnt/user/appdata/Coding/codeman/codeman-cases +# container-only path such as /home/codeman/codeman-cases. +CODEMAN_CASES_PATH=/mnt/user/appdata/codeman/codeman-cases # Required. Network bind address, host port, and local image tag. CODEMAN_HOST=0.0.0.0 diff --git a/docker/README.md b/docker/README.md index e03115e8..7d3a1e84 100644 --- a/docker/README.md +++ b/docker/README.md @@ -22,7 +22,7 @@ Every required value is defined and explained in `.env.example`. `GEMINI_API_KEY On Linux, `Start-Codeman.sh` stops with an error when required paths are missing. It creates the application-data directory when safe, detects its numeric owner as `PUID:PGID`, and detects `DOCKER_SOCKET_GID` from the configured Docker socket. It rejects a root-owned application-data directory because Codeman and its local CLI sessions must remain unprivileged. -Codeman, Claude, OpenCode, and other local sessions run as the unprivileged account named by `CODEMAN_RUNTIME_USER`, which defaults to `opencode`. When Compose is run directly, `PUID` and `PGID` default to `1000:1000`; set them in `.env` when the application-data directory has a different owner. The Bash start script determines them automatically instead. +Codeman, Claude, OpenCode, and other local sessions run as the unprivileged account named by `CODEMAN_RUNTIME_USER`, which defaults to `codeman`. When Compose is run directly, `PUID` and `PGID` default to `1000:1000`; set them in `.env` when the application-data directory has a different owner. The Bash start script determines them automatically instead. To retain Docker-case support without root when running Compose directly, set `DOCKER_SOCKET_GID` to the numeric group ID of the host socket. On a standard Linux Docker host, obtain it with `stat -c '%g' /var/run/docker.sock`. The Bash start script detects it automatically. @@ -38,6 +38,50 @@ Releases that change `server.Dockerfile`, `docker-compose.yaml`, or add a key to changed, and asks you to run `Start-Codeman.sh` here on the host instead. Details: [`../docs/docker-self-update.md`](../docs/docker-self-update.md). +## Local customisation + +Compose merges `docker-compose.override.yml` on top of `docker-compose.yaml`. Keep host-specific changes there rather than editing `docker-compose.yaml`, so this repository can be updated without losing them. Both `docker-compose.override.yml` and `docker-compose.override.yaml` are ignored by Git. + +`Start-Codeman.sh` names the Compose file explicitly, which disables Compose's automatic discovery of the override file, so the script adds it back when one is present and prints the file it used. Running `docker compose` from this folder without any `-f` option finds it automatically. When passing `-f docker/docker-compose.yaml` from the repository root, add `-f docker/docker-compose.override.yml` as well, or the override is silently ignored. + +An override file adds to and replaces individual settings. It cannot delete a key from `docker-compose.yaml`, and Compose concatenates rather than replaces `ports`, so removing a published port still requires editing `docker-compose.yaml`. The example below replaces the restart policy and adds a mount, leaving every other setting in place: + +```yaml +services: + codeman: + restart: always + volumes: + - /srv/projects:/srv/projects +``` + +### Reverse-proxy host allowlist + +Codeman rejects any request whose `Host` header is not on its own allowlist - a +DNS-rebinding guard, not a Compose or Docker concern. Loopback, any IP literal, +the configured `--host`, and a few tunnel-provider suffixes are allowed by +default; a reverse-proxied domain is not, and is rejected with +`403 Forbidden: host not allowed` before the request reaches any handler. + +Add the domain with `CODEMAN_ALLOWED_HOSTS` in `.env`: + +```sh +CODEMAN_ALLOWED_HOSTS='codeman.example.com,.internal.example.com' +``` + +`docker-compose.yaml` does not forward this variable into the container - it +only passes through the environment keys it explicitly lists, and this is not +one of them. Forward it yourself in `docker-compose.override.yml`: + +```yaml +services: + codeman: + environment: + CODEMAN_ALLOWED_HOSTS: ${CODEMAN_ALLOWED_HOSTS} +``` + +See the application's own `docs/wiki/Remote-Access.md` for the full allowlist +format and the tunnel providers it accepts by default. + ## Application data storage The default configuration uses a host-folder bind mount: @@ -49,7 +93,7 @@ volumes: target: /home/${CODEMAN_RUNTIME_USER} ``` -Set `CODEMAN_APPDATA_PATH` in `.env` to a directory that the Docker daemon can access. The example value is `/mnt/user/appdata/Coding/codeman`. +Set `CODEMAN_APPDATA_PATH` in `.env` to a directory that the Docker daemon can access. The example value is `/mnt/user/appdata/codeman`. `CODEMAN_CASES_PATH` is the separate host directory for managed case workspaces. It is mounted into Codeman at the same absolute path, allowing the host Docker daemon to bind it into an isolated case container. Set it to a child directory of `CODEMAN_APPDATA_PATH` unless you deliberately store workspaces elsewhere. @@ -60,7 +104,7 @@ Set `CODEMAN_DOCKER_DISABLE_SWAP_LIMIT=1` when `docker info` reports `SwapLimit= For an existing installation created by a root-running image, change ownership of the application-data directory before upgrading so the configured `PUID` and `PGID` can read the saved credentials and state: ```sh -chown -R 99:100 /mnt/user/appdata/Coding/codeman +chown -R 99:100 /mnt/user/appdata/codeman ``` Replace `99:100` and the path with the values from your `.env` file. @@ -69,7 +113,7 @@ Do not replace this bind mount with a Docker-managed named volume when Docker ca ## Static macvlan networking -The default configuration publishes a host port. It does not use `network_mode: host`. To attach Codeman directly to an existing external macvlan network with a static IP address and MAC address, remove the `ports:` section and add the following to the `codeman` service: +The default configuration publishes a host port. It does not use `network_mode: host`. To attach Codeman directly to an existing external macvlan network with a static IP address and MAC address, remove the `ports:` section from `docker-compose.yaml` and add the following to the `codeman` service. The service and network additions can instead be placed in `docker-compose.override.yml`, but the `ports:` removal cannot, as described under [Local customisation](#local-customisation): ```yaml mac_address: ${CODEMAN_MAC_ADDRESS} diff --git a/docker/Start-Codeman.sh b/docker/Start-Codeman.sh index c0294fe4..92a37424 100644 --- a/docker/Start-Codeman.sh +++ b/docker/Start-Codeman.sh @@ -12,7 +12,21 @@ if [[ ! -f "$env_file" ]]; then exit 1 fi -compose_command=(docker compose --env-file "$env_file" -f "$compose_file") +# Naming a Compose file explicitly disables Compose's automatic discovery of +# the override file, so it has to be added back by hand. Without this, local +# customisation in docker-compose.override.yml is silently ignored. The +# candidates are checked in Compose's own precedence order. +compose_files=(-f "$compose_file") +for override_file in \ + "$script_dir/docker-compose.override.yaml" \ + "$script_dir/docker-compose.override.yml"; do + if [[ -f "$override_file" ]]; then + compose_files+=(-f "$override_file") + printf 'Using Compose override file: %s\n' "$override_file" + break + fi +done +compose_command=(docker compose --env-file "$env_file" "${compose_files[@]}") appdata_path=$( "${compose_command[@]}" config --environment | awk -F= '$1 == "CODEMAN_APPDATA_PATH" { sub(/^[^=]*=/, ""); print; exit }' @@ -92,6 +106,26 @@ if [[ ! -d "$repo_path/.git" ]]; then printf 'Note: %s is not a git checkout, so in-app updates are unavailable.\n' "$repo_path" >&2 fi +# Reads HEAD without requiring a `git` binary on the host — this script +# otherwise checks the checkout only by testing for `.git` as a directory, and +# resolving refs by hand keeps that the same "no host git needed" guarantee. +git_head_commit() { + local git_dir="$1/.git" head_ref ref_path + [[ -d "$git_dir" ]] || return 1 + head_ref=$(cat -- "$git_dir/HEAD" 2>/dev/null) || return 1 + if [[ "$head_ref" == ref:* ]]; then + ref_path="${head_ref#ref: }" + if [[ -f "$git_dir/$ref_path" ]]; then + cat -- "$git_dir/$ref_path" + else + # Packed after a `git gc`; the loose ref file above is gone. + awk -v ref="$ref_path" '$2 == ref { print $1; exit }' "$git_dir/packed-refs" 2>/dev/null + fi + else + printf '%s' "$head_ref" + fi +} + # Record what the container is about to be built and created FROM. The in-app # updater compares these against the release it wants to apply: a release that # changes either file cannot be applied by the container restarting itself (a @@ -126,4 +160,50 @@ else printf 'Warning: no sha256 tool found; in-app updates will not detect environment changes.\n' >&2 fi -exec docker compose --env-file "$env_file" -f "$compose_file" up --build -d +# codeman-node-modules and codeman-dist (docker-compose.yaml) are seeded from +# the image only while EMPTY, so a rebuilt image's fresh output sits unused +# behind old volume content until something clears it. The in-app self-updater +# never hits this — it rebuilds INSIDE the running container, into the very +# volume already in use — but a `docker compose build` triggered from outside +# it (this script, after a `git pull`) does: the container comes back up +# looking unchanged. Detect that here and clear just the affected volume(s) so +# `--build` below actually takes effect. Best-effort: with no sha256 tool this +# quietly does nothing, same as the environment-gate block above. +if [[ -n "$dockerfile_sha" ]]; then + repo_head=$(git_head_commit "$repo_path" || true) + lockfile_sha=$(sha256_of "$repo_path/package-lock.json" 2>/dev/null || true) + source_state_file="$state_dir/docker-build-source.json" + prev_head='' + prev_lockfile_sha='' + if [[ -f "$source_state_file" ]]; then + prev_head=$(sed -n 's/.*"headCommit": *"\([^"]*\)".*/\1/p' "$source_state_file") + prev_lockfile_sha=$(sed -n 's/.*"lockfileSha256": *"\([^"]*\)".*/\1/p' "$source_state_file") + fi + + volumes_to_refresh=() + [[ -n "$repo_head" && "$repo_head" != "$prev_head" ]] && volumes_to_refresh+=('codeman-dist') + [[ -n "$lockfile_sha" && "$lockfile_sha" != "$prev_lockfile_sha" ]] && volumes_to_refresh+=('codeman-node-modules') + + if [[ ${#volumes_to_refresh[@]} -gt 0 ]]; then + # Runs even on this script's very first invocation against an EXISTING + # deployment, deliberately: that deployment's volumes may already be + # stale (there was no earlier version of this check to have caught it), + # and clearing an already-empty or nonexistent volume is a harmless + # no-op, so there is no fresh-install case this needs to avoid. + printf 'Source changed since the last start; refreshing: %s\n' "${volumes_to_refresh[*]}" + "${compose_command[@]}" down + for key in "${volumes_to_refresh[@]}"; do + volume_name=$(docker volume ls -q --filter "label=com.docker.compose.volume=$key" | head -n1) + [[ -n "$volume_name" ]] && docker volume rm -- "$volume_name" + done + fi + + printf '{\n "headCommit": "%s",\n "lockfileSha256": "%s"\n}\n' \ + "$repo_head" "$lockfile_sha" >"$source_state_file.tmp" + mv -- "$source_state_file.tmp" "$source_state_file" + if [[ "$EUID" == '0' ]]; then + chown -- "$PUID:$PGID" "$source_state_file" + fi +fi + +exec "${compose_command[@]}" up --build -d diff --git a/docker/docker-compose.yaml b/docker/docker-compose.yaml index ca61796a..1e888676 100644 --- a/docker/docker-compose.yaml +++ b/docker/docker-compose.yaml @@ -91,6 +91,13 @@ services: - no-new-privileges:true cap_drop: - ALL + cap_add: + # The entrypoint corrects bind-mount ownership as root before dropping to + # PUID:PGID. Everything not listed here remains dropped by cap_drop above. + - CHOWN + - DAC_OVERRIDE + - SETGID + - SETUID healthcheck: test: - CMD-SHELL diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh new file mode 100755 index 00000000..45cf8d5a --- /dev/null +++ b/docker/entrypoint.sh @@ -0,0 +1,48 @@ +#!/bin/sh +# Corrects the ownership of the host bind mounts, then drops to PUID:PGID. +# +# Compose binds CODEMAN_APPDATA_PATH and CODEMAN_CASES_PATH from the host. When +# either path does not exist yet - a first run, a cleared application-data +# directory, a restored backup - the Docker daemon creates it owned by root, +# and an unprivileged server cannot then create its own state directory. The +# result is a container that restarts forever on: +# +# Failed to start web server: EACCES: permission denied, mkdir '/home//.codeman' +# +# Running this as root and dropping afterwards removes that failure mode without +# leaving the server privileged. + +set -eu + +# Honour an explicit `user:` in Compose: when the container was not started as +# root there is nothing to correct and no privilege to drop. +if [ "$(id -u)" -ne 0 ]; then + exec "$@" +fi + +: "${PUID:=1000}" +: "${PGID:=1000}" + +for target in "${HOME:-}" "${CODEMAN_CASES_PATH:-}"; do + [ -n "$target" ] && [ -d "$target" ] || continue + [ "$(stat -c '%u:%g' "$target")" = "${PUID}:${PGID}" ] && continue + + # Deliberately not fatal. A bind mount backed by NFS, CIFS or a rootless + # daemon can refuse chown while still being perfectly writable, and those + # deployments must keep working. A warning is more useful than a container + # that will not start. + if chown -R "${PUID}:${PGID}" "$target" 2>/dev/null; then + printf 'entrypoint: corrected ownership of %s to %s:%s\n' "$target" "$PUID" "$PGID" + else + printf 'entrypoint: warning: cannot change ownership of %s to %s:%s\n' \ + "$target" "$PUID" "$PGID" >&2 + printf 'entrypoint: warning: continuing; set the ownership on the host if startup fails\n' >&2 + fi +done + +# Preserve the supplementary groups Compose granted through group_add - that is +# how the Docker socket stays reachable - while discarding root's own group. +supplementary=$(id -G | tr ' ' '\n' | grep -vx 0 | paste -sd, -) +[ -n "$supplementary" ] || supplementary="$PGID" + +exec setpriv --reuid "$PUID" --regid "$PGID" --groups "$supplementary" "$@" diff --git a/docker/server.Dockerfile b/docker/server.Dockerfile index ecb00d14..c8a37475 100644 --- a/docker/server.Dockerfile +++ b/docker/server.Dockerfile @@ -24,7 +24,7 @@ RUN npm ci \ # docker/docker-compose.yaml. It does not run a Docker daemon in this container. FROM node:22-bookworm-slim -ARG CODEMAN_RUNTIME_USER=opencode +ARG CODEMAN_RUNTIME_USER=codeman ARG PUID=1000 ARG PGID=1000 @@ -93,6 +93,15 @@ RUN npm install --global \ # PGID match the host-owned application-data directory mounted by Compose. The # requested GID may not exist in the base image, and a host UID such as 1000 may # already belong to the baked `node` account, so handle both cases explicitly. +# +# The trailing chown hands the globally-installed CLIs to that same account. +# They were `npm install --global`-ed above while still root, so +# /usr/local/lib/node_modules (and the /usr/local/bin symlinks pointing into it) +# start out root-owned; a session running as the unprivileged runtime user then +# hits EACCES the moment it tries to self-update one in place (observed via +# Codex's own `npm install -g @openai/codex`, which renames the old package dir +# aside before installing the new one — a rename needs write access to the +# PARENT directory, not just the target, so this must chown the whole tree). RUN set -eux; \ case "${PUID}" in ''|*[!0-9]*) echo "PUID must be numeric" >&2; exit 1;; esac; \ case "${PGID}" in ''|*[!0-9]*) echo "PGID must be numeric" >&2; exit 1;; esac; \ @@ -120,7 +129,8 @@ RUN set -eux; \ --home-dir "/home/${CODEMAN_RUNTIME_USER}" \ --shell /bin/bash \ "${CODEMAN_RUNTIME_USER}"; \ - fi + fi; \ + chown -R "${PUID}:${PGID}" /usr/local/lib/node_modules /usr/local/bin WORKDIR /opt/codeman @@ -135,8 +145,19 @@ ENV CODEMAN_IN_CONTAINER=1 \ HOME=/home/${CODEMAN_RUNTIME_USER} \ NODE_ENV=production +# Runtime defaults for the entrypoint, matching the account created above. +ENV PGID=${PGID} PUID=${PUID} + EXPOSE 3000 -USER ${CODEMAN_RUNTIME_USER} +# The container starts as root so the entrypoint can correct the ownership of +# the host bind mounts, which the daemon creates as root whenever they do not +# already exist. The entrypoint then drops to PUID:PGID with setpriv, so the +# server itself never runs privileged. Setting `user:` in Compose bypasses both +# steps, leaving the caller in full control. +COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh +RUN chmod 0755 /usr/local/bin/entrypoint.sh + +ENTRYPOINT ["/usr/local/bin/entrypoint.sh"] CMD ["node", "dist/index.js", "web"] diff --git a/docs/docker-self-update.md b/docs/docker-self-update.md index 6521abd0..6f41505b 100644 --- a/docs/docker-self-update.md +++ b/docs/docker-self-update.md @@ -59,6 +59,7 @@ unchanged. The container path is a new `SupervisorKind`, not a new updater. | `CODEMAN_RESTART_BY_EXIT=1` | The Compose file's declaration of that policy, so the updater may exit even with no Docker socket. | | Toolchain + devDependencies in the image | Lets `npm install` and `npm run build` run inside the container. | | `docker-env-applied.json` | Fingerprint baseline, written by `Start-Codeman.sh` on every start. | +| `docker-build-source.json` | What HEAD/`package-lock.json` the build artefact volumes currently reflect. Written by both `Start-Codeman.sh` and this in-place update, so the two agree on whether those volumes are stale. | ### Why build artefacts are in named volumes @@ -72,6 +73,20 @@ Docker seeds an empty named volume from the image, so the first start inherits t image's already-built `node_modules` and `dist` and pays no bootstrap cost. `docker compose down -v` is the supported reset: the next start re-seeds them. +That seeding-only-while-empty behaviour has a second, less obvious edge: it also +means a plain `docker compose build` triggered from OUTSIDE the container (for +example `Start-Codeman.sh`, after a `git pull` done by hand rather than through +this in-app updater) produces a fresh image whose freshly-built `dist`/ +`node_modules` then sit unused behind the volumes' OLD content — the container +comes back up looking unchanged. `Start-Codeman.sh` detects this by comparing the +checkout's current HEAD and `package-lock.json` hash against `docker-build-source.json`, +and clears just the affected volume(s) before its own `--build` if they moved. +This in-place update writes that same file after a successful build precisely so +that comparison does not fire on stale information: without it, the next plain +`Start-Codeman.sh` run would see the HEAD this update just checked out, not +recognise it as already accounted for, and wipe the volumes this update just +correctly rebuilt right back to the OLDER image. + ### Why the runtime image carries a build toolchain `npm run build` is `tsc` plus `esbuild`, both devDependencies, so the image no diff --git a/scripts/self-update.sh b/scripts/self-update.sh index 6ac3f80d..fb3e7874 100755 --- a/scripts/self-update.sh +++ b/scripts/self-update.sh @@ -193,6 +193,29 @@ run_step "installing" "Installing dependencies" npm install --no-fund --no-audit # 5) Build (gate the restart on success — never restart into a torn dist/). run_step "building" "Building" npm run build || rollback_and_fail "Build failed" +# Docker Compose only: record what HEAD/package-lock.json the freshly-built +# codeman-dist/codeman-node-modules volumes now reflect. `Start-Codeman.sh` +# reads this same file (`$appdata_path/.codeman/…`, i.e. this container's own +# $HOME/.codeman since that path IS the appdata bind mount) to detect source +# changes an EXTERNAL `docker compose build` made and refresh those volumes — +# without this, the next plain `Start-Codeman.sh` run would see the HEAD this +# update just checked out, not recognise it as already accounted for, and wipe +# the volumes this update just correctly rebuilt right back to the OLDER image. +if [[ "$SUPERVISOR" == "docker-compose" ]]; then + build_source_file="$HOME/.codeman/docker-build-source.json" + mkdir -p -- "$HOME/.codeman" + build_head=$(git rev-parse HEAD 2>/dev/null || true) + build_lockfile_sha='' + if command -v sha256sum >/dev/null 2>&1; then + build_lockfile_sha=$(sha256sum -- package-lock.json 2>/dev/null | cut -d' ' -f1) + elif command -v shasum >/dev/null 2>&1; then + build_lockfile_sha=$(shasum -a 256 package-lock.json 2>/dev/null | cut -d' ' -f1) + fi + printf '{\n "headCommit": "%s",\n "lockfileSha256": "%s"\n}\n' \ + "$build_head" "$build_lockfile_sha" >"$build_source_file.tmp" \ + && mv -- "$build_source_file.tmp" "$build_source_file" +fi + # 6) Restart the service so the new code loads. Write the terminal pre-restart # marker FIRST so the freshly-booted server can reconcile it deterministically. write_status "restarting" "Restarting Codeman…"