From bf363214cba50f57b70119b17166add43d8de4fa Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Mon, 24 Aug 2026 01:30:48 +0000 Subject: [PATCH] Add root-ubuntu.sh: provision an Ubuntu/Debian server from one file Moves the server provisioner out of the private dotfiles repo and into cli-tools, where it can be curled onto a box that has nothing on it yet. It is the odd one out here -- bash rather than TypeScript, and not linked onto PATH -- because it has to run before Node exists. Three things had to change for it to live in a public repo and work on a machine that is not one of ours. Dotfiles are now optional and are not in this repository. They cannot be: a dotfiles tree carries ssh config, known_hosts and sometimes keys. The script used to die unless it was sitting inside that checkout, which is exactly what stopped it running anywhere else. It now resolves a source in order -- DOTFILES_DIR, the directory holding the script, DOTFILES_REPO cloned to a cache, or nothing -- and skips only the dotfile stages when there is none. install_ssh had to be split for this: authorising the account's own key is not a dotfiles job, and an earlier cut of this returned early without one, so accounts got created that could not log in. Nothing identifying anybody is left in it. ACME_EMAIL has no default, because a made-up address sends a stranger's certificate warnings into a black hole, and certbot is passed --register-unsafely-without-email rather than an empty -m. There is no default ad slot: a slot id is an account, so shipping one bills every box that ever runs this to whoever owns it. Credentials come from the environment or the config file only. Configuration is read, not sourced. The environment has to win over the file, and `.` assigns unconditionally, so a sourced config would quietly beat the value someone just put on the command line -- and this runs as root, where sourcing hands a config file the whole machine. It is KEY=value, parsed, with nothing executed. Not JSON, because the script runs before apt has installed jq. Also: a POSIX guard that says so in one sentence when the script is run under sh, since /bin/sh on Ubuntu is dash and the old one-liner would have failed on the first [[ with a syntax error naming a line nobody typed; an OS check that warns rather than refuses; --groups, so accounts can be provisioned from a pipe; and the landing page still recognises the marker from when this lived in the dotfiles repo, without which every already-provisioned box would decide its own page was hand-edited and never touch it again. 34 tests cover the pure helpers by cutting them out of the file and running them in a real bash, plus the invariants that make it safe to publish: no personal identifiers, no key material, no default slot. Co-Authored-By: Claude Opus 5 --- README.md | 83 + package.json | 2 +- root-ubuntu.sh | 3941 ++++++++++++++++++++++++++++++++++++++ server.conf.example | 100 + test/root-ubuntu.test.ts | 313 +++ 5 files changed, 4438 insertions(+), 1 deletion(-) create mode 100755 root-ubuntu.sh create mode 100644 server.conf.example create mode 100644 test/root-ubuntu.test.ts diff --git a/README.md b/README.md index 47ca7cf..21990f6 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,12 @@ TypeScript, installed as executables on `PATH`. | [`vid`](#vid) | Inspect, thumbnail, clip and shrink video, through ffmpeg | | [`codeburn`](#codeburn) | See where your AI spend goes, by task, tool, model and project | +One thing here is not a `PATH` command and does not need Node: + +| Script | What it does | +| --- | --- | +| [`root-ubuntu.sh`](#root-ubuntush) | Provision an Ubuntu/Debian server: dev environment, accounts, web, TLS | + ## Requirements - **Node 20+** @@ -683,6 +689,83 @@ before every launch, which is fine for a one-shot and wrong for a dashboard you open twenty times a day. Upstream wants **Node 22.13+**; on an older one it says so and tries anyway, since that floor is theirs to move. +### `root-ubuntu.sh` + +Sets up a server the way we like them, and keeps it that way. It is the odd one +out in this repository: a single bash script rather than a TypeScript command, +because it has to run on a machine where nothing is installed yet — including +Node. Nothing links it onto `PATH`; you curl it onto the box. + +```sh +# on the server, as root +curl -fsSL https://raw.githubusercontent.com/profullstack/cli-tools/master/root-ubuntu.sh \ + | bash -s -- --refresh +``` + +**`bash`, not `sh`.** `/bin/sh` on Ubuntu is dash and this script is bash +throughout. Piping it into `sh` stops with one sentence telling you so rather +than a syntax error on a line you never typed. + +A pipe has no terminal on stdin, so that form is always non-interactive: it +takes defaults instead of reading answers out of its own source. To be asked the +questions, download it first: + +```sh +curl -fsSLO https://raw.githubusercontent.com/profullstack/cli-tools/master/root-ubuntu.sh +chmod +x root-ubuntu.sh +./root-ubuntu.sh # as root +./root-ubuntu.sh alice bob # ...and provision two accounts +./root-ubuntu.sh alice --groups sudo,docker +``` + +What a run does: + +- apt update/upgrade, base packages, unattended security updates +- `ufw`, with ssh opened *before* the firewall is enabled +- accounts and groups — created, or refreshed if an earlier run made them +- zsh + oh-my-zsh, oh-my-tmux, mise, moshcode, chawan +- nginx: `~/public_html` at `/~user` and `user.$WEB_DOMAIN`, plus per-user dev + apps at `..$WEB_DOMAIN`, static or reverse-proxied +- Let's Encrypt, wildcard via DNS-01 where credentials allow it + +**Re-running is the update path.** Every step converges rather than assuming a +blank machine: files it owns are rewritten only when the content really changed, +so nginx is not reloaded for nothing; files a user has since edited are never +clobbered, and the new version is left beside them as `.new`; and a lock file +makes two concurrent runs impossible. On a settled box a re-run reports that +nothing changed, which is the point. + +#### Configuring it + +Read from the environment first, then `$SERVER_CONFIG`, then +`/etc/cli-tools/server.conf`. Copy [`server.conf.example`](server.conf.example), +which documents every value: + +```sh +install -d -m 0755 /etc/cli-tools +install -m 0600 server.conf.example /etc/cli-tools/server.conf +``` + +`KEY=value`, one per line, `#` for comments. The file is **read, not sourced**: +nothing in it executes, so `$(…)` in a config file stays literal text instead of +running as root, and the environment still wins over the file. It is not JSON +either — the script runs before apt has put `jq` on the box, and a bootstrap +that cannot read its own config until it has installed a parser has a hole in +it. + +**Dotfiles are optional and are not in this repository.** They cannot be: a +dotfiles tree carries ssh config, `known_hosts` and sometimes keys, and this +repo is public. Point `DOTFILES_REPO` at your own and the script clones it; +leave it unset and the box still gets everything else, with each account keeping +whatever dotfiles it already had. Running the script from inside a dotfiles +checkout also works — it recognises one by its content, not its name. + +**No credentials, ever, in the file itself.** `ACME_EMAIL` has no default, +because a public script must not ship somebody's address and a made-up one sends +a stranger's certificate warnings into a black hole. There is no default ad slot +for the same shape of reason: a slot id is an account, so shipping one would bill +every box that ever ran this to whoever owned it. + ## As a moshcode plugin This repo is also a plugin marketplace: diff --git a/package.json b/package.json index 094b3ec..7374ec5 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@profullstack/cli-tools", - "version": "0.6.0", + "version": "0.7.0", "private": true, "description": "Local command-line tools, in TypeScript, exposed on PATH.", "type": "module", diff --git a/root-ubuntu.sh b/root-ubuntu.sh new file mode 100755 index 0000000..ff1ec93 --- /dev/null +++ b/root-ubuntu.sh @@ -0,0 +1,3941 @@ +#!/usr/bin/env bash +# +# root-ubuntu.sh -- bootstrap AND maintain an Ubuntu/Debian server as root. +# +# Sets up the standard dev environment and manages the accounts on the box. +# One file, no dependencies beyond what a stock Ubuntu image already has, so it +# can be curled onto a machine that has nothing on it yet: +# +# curl -fsSL https://raw.githubusercontent.com/profullstack/cli-tools/master/root-ubuntu.sh | bash -s -- --refresh +# +# bash, NOT sh. /bin/sh on Ubuntu is dash, this script is bash throughout, and +# piping it into sh fails on the first [[ with a syntax error that names a line +# nobody typed. There is a guard below that says so in one sentence instead. +# +# Piping it gives a NON-INTERACTIVE run: stdin is the script, so there is no +# terminal to prompt at, and every prompt in here is guarded on one (see +# `interactive`). That is the safe direction to fail -- an unattended run takes +# defaults rather than reading answers out of its own source. To be asked the +# questions, download it and run it as a file: +# +# curl -fsSLO https://raw.githubusercontent.com/profullstack/cli-tools/master/root-ubuntu.sh +# chmod +x root-ubuntu.sh && ./root-ubuntu.sh # as root +# +# Safe to re-run, and re-running is the supported way to pick up updates: it +# upgrades packages and tooling, refreshes anything it owns, and leaves anything +# a user has since edited alone (see install_managed below). +# +# Deliberately minimal. Language runtimes/tools come from mise, not apt. +# 1. accounts + groups; users provisioned by an earlier run are picked up +# automatically and refreshed +# 2. apt update/upgrade + unattended security updates +# 3. ufw +# 4. dotfiles (.zsh*, .bash*, .ssh*, ...) from $DOTFILES_REPO, if you have one +# 5. oh-my-zsh + plugins, oh-my-tmux, irssi configs +# 6. mise (curl https://mise.run | sh) +# 7. moshcode (curl https://moshcode.sh/install.sh | sh) +# 8. motd from $MOTD_URL +# 9. nginx per-user pages, per-user dev apps, TLS +# +# Usage, as root: +# ./root-ubuntu.sh # first run, or a refresh +# ./root-ubuntu.sh alice bob # ...and provision two accounts +# ./root-ubuntu.sh alice --groups sudo,docker +# ./root-ubuntu.sh --refresh # update everything, ask nothing +# +# Remote shares (see "remote shares" below for the full options): +# ./root-ubuntu.sh mount user@host:~/data --via peer +# ./root-ubuntu.sh mounts # list what is mounted, and who can reach it +# ./root-ubuntu.sh umount host +# ./root-ubuntu.sh share /mnt/volume -R # open an existing volume +# +# Mounts land at /mnt/./ -- e.g. +# /mnt/tailscale.host/data -- so a remote share is never mistaken for local +# disk, and are persisted to /etc/fstab. They are reachable at the short path +# ~/share/. +# +# Mounts are shared (2775 root:users, 0664 files): every human account can write +# to them. These are team boxes, and a volume only the person who ran the mount +# can write to is the failure that keeps happening -- provider-attached block +# volumes especially, which arrive root:root 0755 and stay that way. Use +# `share` to fix one that is already mounted, and mount --private for a share +# that really does belong to one account. +# +# Flags: +# --refresh non-interactive update pass over the existing box +# --groups LIST groups for the accounts named on this run (no prompt) +# --force-dotfiles overwrite user-edited dotfiles (a .bak is kept) +# --no-reboot never reboot, whatever apt says +# --reboot reboot at the end if the kernel/libc asked for one +# --skip-apt / --skip-web / --skip-tailscale / --skip-tools / --skip-dotfiles +# -h | --help +# +# Configuration, in order of precedence: the environment, then $SERVER_CONFIG +# (default /etc/cli-tools/server.conf). The file is KEY=value, one per line, # +# for comments -- read rather than sourced, so nothing in it executes and the +# environment still wins. Deliberately not JSON, because this runs before apt +# has put jq on the box and a bootstrap script that cannot read its own config +# until it has installed a parser is a bootstrap script with a hole in it. +# Every value below can go in it, and a re-run then needs no environment at all: +# +# WEB_DOMAIN=dev.example.com +# ACME_EMAIL=ops@example.com +# DOTFILES_REPO=git@github.com:example/dotfiles.git +# +# Env overrides: +# SSH_PORT=22 port to open in ufw +# ASSUME_YES=1 don't prompt (defaults: groups sudo,admin; no privkey copy) +# NO_REBOOT=1 skip the reboot at the end +# MOTD_URL=... override the motd endpoint +# TS_AUTHKEY=... tailscale auth key, to join the tailnet unattended +# TS_HOSTNAME=.. name this node takes on the tailnet (default: short hostname) +# WEB_DOMAIN=... domain for the per-user pages +# DEV_APPS=0 turn off ..$WEB_DOMAIN hosting +# DOTFILES_REPO=... git URL of the dotfiles to install (optional) +# SPONSOR_AD_SLOT=... ad slot id; the ad is off until one is set +# PORKBUN_API_KEY=... PORKBUN_SECRET_API_KEY=... +# DNS-01 credentials for the wildcard cert. Without them: +# http only, no wildcard. +# CLOUDFLARE_API_TOKEN=... same, for zones hosted at Cloudflare instead +# +# --- on being re-runnable ----------------------------------------------- +# Every step is written to converge, not to assume a blank machine: +# * files we own are rewritten only when the content actually changes, so +# nginx is not reloaded and services are not restarted for nothing +# * files a USER owns (.zshrc, .gitconfig, .irssi/config, ~/public_html) +# are never clobbered once they have diverged from what we shipped +# * no reboot unless the box says one is required AND you agree to it +# * a lock file makes two concurrent runs impossible +# +# --- on secrets --------------------------------------------------------- +# There are none in this file and there must never be. It is public, it is +# curled onto machines by strangers, and every credential it can use is read +# from the environment or from $SERVER_CONFIG. In particular there is no +# default ad slot: a shared slot bills every install's impressions to one +# account, which is somebody else's bill. + +# Deliberately POSIX so that dash can parse and run it: this is the one thing in +# the file that has to work in the wrong shell, because its whole job is to say +# so. Everything past it is bash. +if [ -z "${BASH_VERSION:-}" ]; then + echo "root-ubuntu.sh: this is a bash script and you are running it under sh." >&2 + echo " curl -fsSL /root-ubuntu.sh | bash -s -- --refresh" >&2 + echo " ...or: bash root-ubuntu.sh" >&2 + exit 1 +fi + +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)" || SCRIPT_DIR="" + +# ------------------------------------------------------------------ config --- + +# Read before anything else looks at a variable, so the file can supply any of +# the defaults below. +# +# READ, not sourced, for two reasons. The environment has to win over the file +# -- that is the rule everywhere else in this repository -- and `.` assigns +# unconditionally, so a sourced config would quietly beat the variable someone +# just put on the command line. And this runs as root: sourcing hands whatever +# is in /etc/cli-tools/server.conf the whole machine, where a config file only +# needs to carry values. +# +# So it is KEY=value, one per line, # for comments, surrounding quotes stripped. +# No expansion, no substitution, nothing executed. +SERVER_CONFIG="${SERVER_CONFIG:-/etc/cli-tools/server.conf}" +read_server_config() { + local file="$1" line key val + [[ -r "$file" ]] || return 0 + while IFS= read -r line || [[ -n "$line" ]]; do + [[ "$line" =~ ^[[:space:]]*# ]] && continue + [[ "$line" =~ ^[[:space:]]*([A-Za-z_][A-Za-z0-9_]*)[[:space:]]*=(.*)$ ]] || continue + key="${BASH_REMATCH[1]}" + val="${BASH_REMATCH[2]}" + # Already in the environment? Then that is the answer, and this line is + # only what the file would have said. + [[ -n "${!key+set}" ]] && continue + # trailing whitespace, then one layer of matching quotes + val="${val%"${val##*[![:space:]]}"}" + if [[ "$val" == \"*\" && ${#val} -ge 2 ]]; then + val="${val:1:${#val}-2}" + elif [[ "$val" == \'*\' && ${#val} -ge 2 ]]; then + val="${val:1:${#val}-2}" + fi + printf -v "$key" '%s' "$val" + done <"$file" + return 0 +} +read_server_config "$SERVER_CONFIG" + +SSH_PORT="${SSH_PORT:-22}" +ASSUME_YES="${ASSUME_YES:-0}" +MOTD_URL="${MOTD_URL:-https://profullstack.com/motd}" +MOTD_CACHE=/var/cache/profullstack-motd + +# Everything this script remembers between runs lives here: which users it +# provisioned, and the checksum of each file it installed into their homes. +# Without that record a re-run cannot tell "we wrote this" from "the user +# wrote this", and the only safe answer would be to never update anything. +STATE_DIR="${STATE_DIR:-/var/lib/profullstack}" +USERS_STATE="$STATE_DIR/users" +LOCK_FILE=/var/lock/root-ubuntu.lock +LOG_FILE="${LOG_FILE:-/var/log/root-ubuntu.log}" + +FORCE_DOTFILES="${FORCE_DOTFILES:-0}" +SKIP_APT="${SKIP_APT:-0}" +SKIP_WEB="${SKIP_WEB:-0}" +SKIP_TAILSCALE="${SKIP_TAILSCALE:-0}" +SKIP_TOOLS="${SKIP_TOOLS:-0}" +SKIP_DOTFILES="${SKIP_DOTFILES:-0}" +# 0 = never, 1 = only if the box says a reboot is required, 2 = always ask +REBOOT_POLICY=1 + +# Dotfiles are OPTIONAL and they are not in this repository. +# +# They cannot be: a dotfiles tree carries ssh config, known_hosts, sometimes +# keys, and this file is public. So the shell/editor/tmux/irssi configuration a +# team wants on its boxes lives in that team's own repo, and this clones it if +# you name one. With no DOTFILES_REPO the box still gets everything else -- +# packages, firewall, accounts, zsh, oh-my-zsh, mise, moshcode, nginx, TLS -- +# and simply keeps whatever dotfiles each account already had. +# +# DOTFILES_DIR points at an existing checkout instead, which is what a run from +# inside such a repo wants: put this script beside the dotfiles and it uses +# them without cloning anything. +DOTFILES_REPO="${DOTFILES_REPO:-}" +DOTFILES_DIR="${DOTFILES_DIR:-}" +DOTFILES_CACHE="${DOTFILES_CACHE:-$STATE_DIR/dotfiles-src}" + +# Where a pasted public key is filed so that re-runs and rebuilds keep working. +# In a dotfiles checkout it belongs with the dotfiles, so the whole team's keys +# travel together; without one it still has to persist somewhere, and that is +# the state directory. +KEYS_DIR="${KEYS_DIR:-}" + +# Tailscale. TS_AUTHKEY joins the tailnet unattended; without it the script +# prints the command to run by hand. +TS_AUTHKEY="${TS_AUTHKEY:-}" +TS_HOSTNAME="${TS_HOSTNAME:-$(hostname -s)}" + +# Per-user web hosting: https://WEB_DOMAIN/~user and https://user.WEB_DOMAIN +WEB_DOMAIN="${WEB_DOMAIN:-dev.profullstack.com}" +# each user's address is @MAIL_DOMAIN +MAIL_DOMAIN="${MAIL_DOMAIN:-profullstack.com}" +# Where the landing page sends people for mail and webmail. Both are only +# links, so a box for a different domain needs nothing here but these two. +MAIL_URL="${MAIL_URL:-https://forwardemail.net/}" +# the comms network, reached over ssh +BBS_DOMAIN="${BBS_DOMAIN:-bbs.profullstack.com}" +WEBMAIL_URL="${WEBMAIL_URL:-https://mail.forwardemail.net/}" +# Wildcard certs require a DNS-01 challenge. Provide a Cloudflare API token +# either in the environment or in CF_CREDENTIALS (ini format certbot expects). +CF_CREDENTIALS="${CF_CREDENTIALS:-/etc/letsencrypt/cloudflare.ini}" +CLOUDFLARE_API_TOKEN="${CLOUDFLARE_API_TOKEN:-}" +PORKBUN_API_KEY="${PORKBUN_API_KEY:-}" +PORKBUN_SECRET_API_KEY="${PORKBUN_SECRET_API_KEY:-}" +ACME_HOME="${ACME_HOME:-/root/.acme.sh}" +# No default, and no personal address baked in. Let's Encrypt uses it only for +# expiry warnings; issuance works without one, and the _issue_cert_* helpers +# say so once rather than failing. +ACME_EMAIL="${ACME_EMAIL:-}" +ACME_WEBROOT="${ACME_WEBROOT:-/var/www/acme}" +CERT_DIR="/etc/letsencrypt/live/$WEB_DOMAIN" +# reissue once the cert has this little life left +CERT_RENEW_DAYS="${CERT_RENEW_DAYS:-30}" +COPY_SSH_PRIVATE_KEYS="${COPY_SSH_PRIVATE_KEYS:-0}" + +# Per-user dev apps: https://..$WEB_DOMAIN +# Static from ~/apps//public, or reverse-proxied to 127.0.0.1: +# when ~/apps//.port holds a port number. +DEV_APPS="${DEV_APPS:-1}" +DEV_APPS_MAP=/etc/nginx/conf.d/profullstack-devapps.conf + +# Block AI/LLM crawlers and aggressive scrapers by User-Agent. +# +# Search engines are deliberately NOT in the list: blocking Googlebot/Bingbot +# would deindex the box rather than protect it. What gets blocked is the +# training/scraping crawlers, which ignore robots.txt often enough that the +# polite file alone is not a control. +# +# The map is written even when this is 0 (with no entries, so $bad_bot is +# always empty). nginx refuses to start when a vhost references a variable no +# map defines -- the same trap DEV_APPS_MAP documents -- so the variable must +# exist unconditionally, and only its contents are conditional. +BLOCK_AI_BOTS="${BLOCK_AI_BOTS:-1}" +BAD_BOTS_MAP=/etc/nginx/conf.d/profullstack-badbots.conf + +# Substrings matched case-insensitively against the User-Agent. Grouped so it +# is obvious what each entry is and nothing gets removed by guesswork. +AI_CRAWLER_AGENTS=( + # OpenAI + GPTBot OAI-SearchBot ChatGPT-User + # Anthropic + ClaudeBot Claude-Web Claude-User Claude-SearchBot anthropic-ai + # Google / Apple opt-out crawlers (NOT Googlebot itself) + Google-Extended Applebot-Extended + # Perplexity + PerplexityBot Perplexity-User + # Meta + meta-externalagent meta-externalfetcher FacebookBot + # Common Crawl -- the corpus most models train on + CCBot + # ByteDance / Amazon / others + Bytespider Amazonbot cohere-ai Diffbot omgili omgilibot + ImagesiftBot YouBot AI2Bot Timpibot iaskspider DuckAssistBot + PanguBot "Kangaroo Bot" Webzio-Extended Scrapy + # generic scraper stacks that ignore robots.txt + python-requests python-httpx libwww-perl HTTrack Nutch +) + +# Sponsor ad shown at the top of the per-user pages: the directory listings +# under ~/public_html, and the default ~/public_html/index.html. +# +# The endpoint returns plain ASCII sized to a column count -- it is the same +# feed the terminal/motd banners use, and it hands back a different creative +# each time you ask. +# +# The ad rotates per page load, but it is NOT fetched per page load: that would +# put an external host in the critical path of every request, and one slow +# response would stall the page. Instead a timer keeps a pool of $SPONSOR_AD_POOL +# pre-rendered creatives on disk and nginx picks one at random per request +# (random_index). Rotation costs one open(); a dead endpoint just stops the pool +# from refreshing and the existing ads keep serving. +# +# Two mechanisms, because the two pages differ in kind: +# listings -- generated by autoindex, so there is no file to edit. nginx +# prepends the fragment with add_before_body. +# index.html -- a real file, so the default page carries an SSI include and +# nginx expands it. That also means a user can move the token, +# and a user who replaces the page entirely drops the ad. +# OFF until a slot id is configured, and there is deliberately no default one. +# An ad slot is an account: baking one in here would bill every box that ever +# runs this script to whoever owns that slot, and the impressions would look +# like traffic they did not have. So SPONSOR_AD_SLOT is the switch -- set it in +# $SERVER_CONFIG to turn the ad on, leave it alone to never see one. +SPONSOR_AD_SLOT="${SPONSOR_AD_SLOT:-}" +SPONSOR_AD="${SPONSOR_AD:-1}" +[[ -z "$SPONSOR_AD_SLOT" ]] && SPONSOR_AD=0 +SPONSOR_AD_ENDPOINT="${SPONSOR_AD_ENDPOINT:-https://crawlproof.com/api/ads/motd}" +# Total width of the ad box, and it has a floor. The endpoint only draws the +# click URL inside the border when it fits -- otherwise it drops it onto a bare +# line underneath, which reads as a stray link rather than part of the ad. The +# URL is 25 chars of prefix + a 36-char id + "?s=$SPONSOR_AD_SRC", and the +# border costs 4 more, so 72 was one short of holding it and 76 is the exact +# floor. 80 leaves headroom for a longer src tag, and matches the ~79-char +# width of the autoindex listing it sits above. +SPONSOR_AD_COLS="${SPONSOR_AD_COLS:-80}" +# rides through to the click URL, so these views are told apart from the motd +SPONSOR_AD_SRC="${SPONSOR_AD_SRC:-userdirs}" +# How many pre-rendered creatives to keep. This is the rotation: nginx picks +# one at random per request, so it also bounds how repetitive a reload feels. +# Duplicates are left in rather than deduped -- the endpoint weights its own +# rotation, and collapsing that here would flatten it. +SPONSOR_AD_POOL="${SPONSOR_AD_POOL:-12}" +SPONSOR_AD_DIR=/var/www/sponsor +SPONSOR_AD_POOL_DIR=/var/www/sponsor/ads +# superseded by the pool; removed on upgrade +SPONSOR_AD_LEGACY_FILE=/var/www/sponsor/ad.html +# nginx URI the pool is served at. Internal, so it is only ever reachable +# through the SSI/add_before_body subrequests -- never fetched directly. The +# trailing slash matters: random_index only fires on a URI that ends in one. +SPONSOR_AD_URI=/.sponsor-ad/ +SPONSOR_AD_BLANK_URI=/.sponsor-ad-blank + +# chawan -- TUI browser and pager. Not in apt: the author ships a .deb, so the +# current version is read off the homepage ("the latest release (vX.Y.Z)") and +# the matching .deb is pulled from SourceHut. Set CHAWAN_VERSION to pin one. +# lynx is in BASE_PACKAGES as the fallback for when chawan cannot be installed +# at all -- non-amd64, or the download is unreachable. +CHAWAN_INDEX="${CHAWAN_INDEX:-https://chawan.net/index.html}" +CHAWAN_VERSION="${CHAWAN_VERSION:-}" +# only used when the homepage cannot be reached and nothing is installed yet +CHAWAN_FALLBACK_VERSION=0.4.4 + +# Logo shown at the top of the landing page. Cached locally so the page does +# not depend on profullstack.com being up. +LOGO_URL="${LOGO_URL:-https://profullstack.com/assets/logo.svg}" +LOGO_FILE=/var/www/userdirs/assets/logo.svg + +# Group menu offered when creating a user. Default selection is 1,2. +GROUP_CHOICES=(sudo admin docker adm www-data users) +DEFAULT_GROUPS="sudo,admin" + +USERS=() # alice@example -- new this run, get the full treatment +USER_GROUPS=() # sudo,admin -- index-matched to USERS +KNOWN_USERS=() # logins provisioned by an earlier run, refreshed not created +FAILED=() +PRESERVED=() # files left alone because the user had edited them +CHANGED=() # things this run actually altered (for the closing summary) + +# ---------------------------------------------------------------- helpers --- + +log() { printf '\n\033[1;32m==>\033[0m %s\n' "$*"; } +info() { printf ' %s\n' "$*"; } +warn() { printf '\033[1;33m[warn]\033[0m %s\n' "$*" >&2; } +die() { printf '\033[1;31m[fail]\033[0m %s\n' "$*" >&2; exit 1; } + +# Run a step; failures are collected and reported at the end instead of +# aborting. The old script died halfway through on one bad package. +try() { + local desc="$1"; shift + info "$desc" + if ! "$@"; then + warn "$desc -- failed (continuing)" + FAILED+=("$desc") + return 1 + fi +} + +note() { CHANGED+=("$*"); info "$*"; } + +interactive() { [[ -t 0 && "$ASSUME_YES" != 1 ]]; } + +confirm() { + local prompt="$1" default="${2:-n}" ans + interactive || { [[ "$default" == y ]]; return; } + read -r -p "$prompt " ans + ans="${ans:-$default}" + [[ "$ans" =~ ^[Yy] ]] +} + +user_login() { printf '%s' "${1%%@*}"; } # alice@example -> alice +user_home() { getent passwd "$1" | cut -d: -f6; } + +valid_login() { [[ "$1" =~ ^[a-z_][a-z0-9_-]{0,31}$ ]]; } + +# Run a command as $1 with a login-ish env (installers write into ~). +# +# runuser -u keeps the caller's environment AND working directory, so both +# have to be replaced: +# HOME -- otherwise installers run for alice still write into /root +# cwd -- otherwise anything touching the cwd dies when the script is run +# from a directory the target user cannot reach, e.g. +# /root/provision ("sh: cd: can't cd to /root/provision") +as_user() { + local login="$1"; shift + local home v + home="$(user_home "$login")" + [[ -n "$home" ]] || { warn "no home dir for $login"; return 1; } + + # env -i, NOT the inherited environment. runuser -u keeps the caller's + # variables, and root's shell exports plenty that are wrong for anybody + # else. This script points root's shell at our .zshrc, which does + # 'export ZSH="$HOME/.oh-my-zsh"' -- so from the second run onwards root + # carries ZSH=/root/.oh-my-zsh, the oh-my-zsh installer honours it over + # $HOME, and the clone dies with + # fatal: cannot mkdir /root/.oh-my-zsh: Permission denied + # while ostensibly installing for someone else. NVM_DIR, ZDOTDIR, + # CARGO_HOME and the MISE_* family all leak the same way. + # + # Starting clean and letting bash -l rebuild from /etc/profile is the only + # version of this that stays correct as people add exports to the dotfiles. + local -a envs=( + HOME="$home" USER="$login" LOGNAME="$login" SHELL=/bin/bash + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + TERM="${TERM:-dumb}" + ) + # a box behind a proxy still has to reach the network + for v in http_proxy https_proxy no_proxy HTTP_PROXY HTTPS_PROXY NO_PROXY; do + [[ -n "${!v:-}" ]] && envs+=("$v=${!v}") + done + + # cd happens here, in the parent, while still root -- root can enter any + # directory, and the child then inherits a cwd its own user can reach. + # stdin from /dev/null: these run inside 'while read ... done < <(...)' + # loops, and anything that decides to prompt (a git credential helper on a + # 401, say) would otherwise eat the rest of the list being iterated. + if [[ "$login" == root ]]; then + ( cd -- "$home" && env -i "${envs[@]}" bash -lc "$*" /dev/null)" ]]; then + warn "$dest exists and is not a git checkout -- left alone (not installing $url)" + return 1 + fi + as_user "$login" "rm -rf '$dest' && git clone --depth 1 --quiet '$url' '$dest'" +} + +# ------------------------------------------------- converge, don't clobber --- + +file_sha() { [[ -f "$1" ]] && sha256sum "$1" 2>/dev/null | cut -d' ' -f1; } + +# Write stdin to $1 only if the content differs. Returns 0 when it changed, 1 +# when it did not -- so callers can reload a service only when there is a +# reason to. Re-running the script should not bounce nginx for nothing. +write_if_changed() { + local dest="$1" mode="${2:-0644}" tmp + tmp="$(mktemp)" || return 1 + cat >"$tmp" + if [[ -f "$dest" ]] && cmp -s "$tmp" "$dest"; then + rm -f "$tmp" + chmod "$mode" "$dest" + return 1 + fi + install -m "$mode" "$tmp" "$dest" + rm -f "$tmp" + return 0 +} + +# Where we remember the checksum of the copy we installed for a user. +_state_path() { + local login="$1" dest="$2" + printf '%s/dotfiles/%s/%s' "$STATE_DIR" "$login" "${dest//\//%}" +} + +# Has this exact content ever been shipped by this repo? +# +# The state file only knows about runs of the NEW script. On a box provisioned +# before it existed there is no record, and every dotfile would look +# user-edited -- which would freeze those boxes forever. So also ask git: if +# the file matches ANY revision of the template in this checkout's history, +# nobody has hand-edited it and updating is safe. +_matches_repo_history() { + local dest="$1" rel="$2" blob want + # No checkout, no history to compare against -- every file then looks + # user-edited, which is the safe answer rather than a wrong one. + [[ -n "$DOTFILES_DIR" ]] || return 1 + command -v git >/dev/null || return 1 + git -C "$DOTFILES_DIR" rev-parse --git-dir >/dev/null 2>&1 || return 1 + blob="$(git -C "$DOTFILES_DIR" hash-object "$dest" 2>/dev/null)" || return 1 + [[ -n "$blob" ]] || return 1 + + want="$(git -C "$DOTFILES_DIR" log --format='%H' --all -- "$rel" 2>/dev/null \ + | sed "s|\$|:$rel|" \ + | git -C "$DOTFILES_DIR" cat-file --batch-check='%(objectname)' 2>/dev/null \ + | grep -qxF "$blob" && echo yes)" + [[ "$want" == yes ]] +} + +# Install $src at $dest for $owner, unless the user has made it theirs. +# +# dest missing .................. install +# dest already identical ........ nothing to do (just record it) +# dest == what we last wrote .... ours, safe to update +# dest is some older template ... ours, safe to update +# anything else ................. THEIRS: leave it, drop a .new beside it +# +# --force-dotfiles overrides the last case, keeping a .bak. +install_managed() { + local src="$1" dest="$2" owner="$3" mode="${4:-0644}" rel="${5:-}" + local src_sha dest_sha recorded state + [[ -f "$src" ]] || return 0 + # repo cloned into the very home we are installing to: same file + [[ "$src" -ef "$dest" ]] && return 0 + + rel="${rel:-$(basename "$dest")}" + state="$(_state_path "$owner" "$dest")" + src_sha="$(file_sha "$src")" + + if [[ -e "$dest" ]]; then + dest_sha="$(file_sha "$dest")" + if [[ -n "$dest_sha" && "$dest_sha" == "$src_sha" ]]; then + _record_managed "$state" "$src_sha" + chown "$owner:$owner" "$dest" 2>/dev/null + return 0 + fi + recorded="$(cat "$state" 2>/dev/null || true)" + if [[ "$FORCE_DOTFILES" == 1 ]]; then + cp -p "$dest" "$dest.bak" 2>/dev/null + warn "overwriting $dest (backup: $dest.bak)" + elif [[ -n "$recorded" && "$dest_sha" == "$recorded" ]]; then + : # we wrote it and it has not been touched since + elif _matches_repo_history "$dest" "$rel"; then + : # an older version of this same template + else + # theirs. Show them the new version without taking anything away. + if ! cmp -s "$src" "$dest.new" 2>/dev/null; then + install -m "$mode" -o "$owner" -g "$owner" "$src" "$dest.new" 2>/dev/null + fi + PRESERVED+=("$dest") + return 0 + fi + fi + + install -D -m "$mode" -o "$owner" -g "$owner" "$src" "$dest" \ + || { warn "could not install $dest"; return 1; } + _record_managed "$state" "$src_sha" + note "updated $dest" + return 0 +} + +_record_managed() { + local state="$1" sha="$2" + [[ -n "$sha" ]] || return 0 + install -d -m 0700 "$(dirname "$state")" 2>/dev/null + printf '%s\n' "$sha" >"$state" 2>/dev/null || true +} + +# --------------------------------------------------------- managed users --- + +# The set of accounts this script looks after. Recorded so that a bare re-run +# refreshes everyone instead of only the users named on the command line. +load_known_users() { + local l + if [[ -s "$USERS_STATE" ]]; then + while read -r l; do + [[ -n "$l" ]] && id -u "$l" >/dev/null 2>&1 && KNOWN_USERS+=("$l") + done <"$USERS_STATE" + fi + + # Nothing recorded: this is either a fresh box or one provisioned by an + # older version of the script. Adopt the real humans already on it -- + # regular uids, a home under /home, an actual login shell. + if [[ ${#KNOWN_USERS[@]} -eq 0 ]]; then + while IFS=: read -r l _ uid _ _ home shell; do + [[ "$uid" -ge 1000 && "$uid" -lt 65534 ]] || continue + [[ "$home" == /home/* && -d "$home" ]] || continue + [[ "$shell" == */nologin || "$shell" == */false ]] && continue + KNOWN_USERS+=("$l") + done < <(getent passwd) + fi +} + +remember_user() { + local login="$1" + install -d -m 0755 "$STATE_DIR" + touch "$USERS_STATE" + grep -qxF "$login" "$USERS_STATE" 2>/dev/null || printf '%s\n' "$login" >>"$USERS_STATE" +} + +# every login this run should touch: previously known + newly created +all_logins() { + local out=() u l + for l in ${KNOWN_USERS[@]+"${KNOWN_USERS[@]}"}; do out+=("$l"); done + for u in ${USERS[@]+"${USERS[@]}"}; do + l="$(user_login "$u")" + printf '%s\n' "${out[@]+"${out[@]}"}" | grep -qxF "$l" || out+=("$l") + done + printf '%s\n' "${out[@]+"${out[@]}"}" +} + +# -------------------------------------------------------- remote shares --- +# +# Mount a share from another box, and keep it mounted across reboots. +# +# The mountpoint is named after where the data actually lives. A remote share +# sitting at a path that reads like local disk is genuinely dangerous: someone +# eventually runs mv or rm -rf against what they believe is a spare local +# volume, and it is in fact the only copy, on another machine, over the wire. +# +# ubuntu@files.example.com:~/Downloads/done +# -> /mnt/tailscale.files.example.com/Downloads/done +# └ how we reach it ┘└ which box ┘└ the remote path, verbatim ┘ +# +# The first label is HOW the box is reached -- "tailscale" for a tailnet peer, +# otherwise the protocol ("nfs" or "sshfs"). Never just the remote username: +# "ubuntu" names an account, not a machine, and there is one on every box. + +MNT_ROOT="${MNT_ROOT:-/mnt}" + +# Who a --shared mount is opened to. Not "everyone": every human account on +# these boxes is in `users` (it is one of GROUP_CHOICES above), and daemons are +# not, so the group is already the line between a person and a service. +SHARE_GROUP="${SHARE_GROUP:-users}" + +# What a shared directory and the files under it end up as. Directories need the +# execute bit to be traversable at all, and the setgid bit to keep new entries in +# the group -- which is why these are not the same number with a digit moved. +SHARE_DIR_MODE="${SHARE_DIR_MODE:-2775}" +SHARE_FILE_MODE="${SHARE_FILE_MODE:-0664}" + +# The account nginx runs as. Only used to let it traverse ~/share (_share_link); +# it is deliberately NOT $SHARE_GROUP, which is who may write to a mount. +WEB_GROUP="${WEB_GROUP:-www-data}" + +# Resolve a tailnet peer name, as it appears in `tailscale status`, to its IP. +_tailnet_ip() { + local peer="$1" ip + command -v tailscale >/dev/null 2>&1 || return 1 + ip="$(tailscale status 2>/dev/null | awk -v p="$peer" '$2 == p { print $1; exit }')" + [[ -n "$ip" ]] || return 1 + printf '%s' "$ip" +} + +_port_open() { timeout 3 bash -c "exec 3<>/dev/tcp/$1/$2" 2>/dev/null; } + +# "alice and root", or just "root" when that is already who we are. +_owner_desc() { + local me="${SUDO_USER:-root}" + [[ "$me" == root ]] && printf 'root' || printf '%s and root' "$me" +} + +# Open a directory to one account, or to everyone in $SHARE_GROUP. +# +# A volume the team is meant to share cannot be 0700 owned by whoever happened +# to run the mount -- that is how you get `touch foo` -> Permission denied on a +# 200G disk sitting empty. Shared mode hands the directory to the group instead: +# +# 2775 directories: rwx for owner and group, r-x for everyone else, and +# setgid so every file and directory created inside inherits +# $SHARE_GROUP rather than the creator's private group. Without the +# setgid bit the first person to write locks the next one out, which +# looks exactly like the bug this is meant to fix. +# 0664 files: rw for owner and group, r for everyone else. +# +# The group is who may WRITE; the world r bit only lets other accounts read. +# That is deliberate -- nginx serving out of a shared volume is the common case +# and does not justify putting www-data in $SHARE_GROUP. Override with +# SHARE_DIR_MODE / SHARE_FILE_MODE if a volume needs to be group-only (2770 and +# 0660), which is the right call for anything actually sensitive. +# +# setgid fixes the group a new file lands in, not its mode -- the group WRITE +# bit comes from the writer's umask. Ubuntu's default 002 grants it (safe here +# because USERGROUPS_ENAB gives each account its own private group). A user who +# has set umask 022 will still create files their colleagues cannot write. +_share_perms() { + local dir="$1" shared="$2" + if [[ "$shared" == 1 ]]; then + getent group "$SHARE_GROUP" >/dev/null \ + || die "share: group '$SHARE_GROUP' does not exist (groupadd $SHARE_GROUP, or set SHARE_GROUP=)" + chgrp "$SHARE_GROUP" "$dir" || warn "could not set group $SHARE_GROUP on $dir" + chmod "$SHARE_DIR_MODE" "$dir" || warn "could not open $dir to $SHARE_GROUP" + info "shared: anyone in '$SHARE_GROUP' can read and write $dir" + else + chown "${SUDO_USER:-root}" "$dir" 2>/dev/null || true + # 00700, not 0700: chmod leaves a directory's setgid bit alone unless a + # numeric mode carries the extra leading zero. Going shared -> private + # with 0700 lands on 2700 -- harmless while the group has no bits, but + # it comes back the moment someone loosens them again. + chmod 00700 "$dir" || warn "could not lock down $dir" + info "private: only $(_owner_desc) can traverse $dir" + fi +} + +# /mnt stays the single source of truth; ~/share/ is the short path a +# human actually types. A symlink rather than a second mountpoint, so findmnt +# and `mounts` keep showing exactly one location for the data. +# +# Who can reach a share is decided by one directory: the one above the +# mountpoint in /mnt. Without execute there, no other account can traverse to +# the data, whatever the server says the files are. Putting the mount in $HOME +# would not have given that control on its own; /home/ is 0751 on these +# boxes, so every account can already walk through it. +# +# That directory is $SHARE_DIR_MODE root:$SHARE_GROUP by default (_share_perms) -- +# these are team boxes and a mount nobody but the person who ran it can write to +# is the common failure, not a safe default. Pass --private for a share that +# genuinely belongs to one account and it goes back to 0700. +# +# The symlink has no say in either case -- a symlink cannot grant what the +# directory withholds. ~/share is a shortcut someone types, not the permission +# boundary, so it must not be what withholds traversal from whoever was +# legitimately pointed at one of these links. +# +# A symlink is resolved by whoever follows it, and the kernel then checks EVERY +# component of the path it expands to. So a 0700 ~/share silently becomes a +# second permission boundary for anything that walks in from outside -- nginx +# following ~/public_html/done -> ~/share/seed gets EACCES on ~/share and 403s, +# while the /mnt directory that is supposed to be making that call sits there +# world-traversable. +# +# What needs to traverse is one account, so name it: the directory goes to +# $WEB_GROUP with 0710. www-data gets the x bit, other accounts get nothing -- +# 0711 would have worked too, but it hands traversal to every process on the box +# to solve a problem only nginx has. Group execute is the same fix, scoped. +# +# Withholding r keeps it per-user either way: nginx can walk through to a link +# it was pointed at, but cannot list which shares exist. +# +# No $WEB_GROUP means no web server, so nothing needs to traverse and it stays +# 0700. That is the safe direction to fail -- a --skip-web box is not silently +# opened up. +_share_link() { + local mp="$1" name="$2" owner="${SUDO_USER:-root}" home grp mode + home="$(getent passwd "$owner" | cut -d: -f6)" + [[ -n "$home" && -d "$home" ]] || { warn "no home for $owner -- skipping the ~/share link"; return 0; } + if getent group "$WEB_GROUP" >/dev/null; then + grp="$WEB_GROUP" mode=0710 + else + grp="$owner" mode=0700 + info "no group '$WEB_GROUP' -- $home/share stays private to $owner" + fi + # -m and -g re-apply to an existing directory too, so a ~/share left 0700 by + # an earlier run is repaired by the next mount rather than staying broken. + install -d -o "$owner" -g "$grp" -m "$mode" "$home/share" || return 0 + ln -sfn "$mp" "$home/share/$name" || return 0 + chown -h "$owner:$owner" "$home/share/$name" 2>/dev/null || true + info "link: $home/share/$name -> $mp" +} + +# ubuntu + ~/Downloads/done -> /home/ubuntu/Downloads/done (what NFS exports) +_expand_remote() { + local user="$1" path="$2" + case "$path" in + '~/'*) printf '/home/%s/%s' "$user" "${path#\~/}" ;; + '~') printf '/home/%s' "$user" ;; + /*) printf '%s' "$path" ;; + *) printf '/home/%s/%s' "$user" "$path" ;; + esac +} + +mount_usage() { + cat <<-'EOF' + Usage: + root-ubuntu.sh mount [user@]host:/remote/path [options] + root-ubuntu.sh umount + root-ubuntu.sh mounts + root-ubuntu.sh share ... [--private] [-R] + + Mounts a remote share at /mnt/./ and adds an fstab + entry so it comes back after a reboot. Re-running for the same share just + rewrites the entry, so it is safe to repeat. + + Shares are shared by default: the directory above the mountpoint becomes + 2775 root:$SHARE_GROUP (default: users), so every human account on the box + can read and write it. The setgid bit keeps new files in the group, so the + first writer does not lock everyone else out. Other accounts get read only. + The short path is linked as ~/share/. + + Modes come from $SHARE_DIR_MODE (2775) and $SHARE_FILE_MODE (0664); set both + in the environment for a volume that should be group-only (2770 and 0660). + + Pass --private for a share that belongs to one account: that directory goes + to 0700 instead and nobody else on the box can traverse to the data. + + `share` applies the same thing to a volume that is already mounted -- a + provider-attached block volume, say, which lands root:root 0755 with its own + fstab line and was never touched by this script. Add -R to sweep contents + that are already there. + + Options: + --link NAME name for the ~/share/ symlink (default: first label of the + host, e.g. files.example.com -> files) + --no-link do not create the ~/share symlink at all + --via PEER reach the host over this tailnet peer (from `tailscale status`). + Use when the DNS name resolves to a public IP but you want the + traffic on the tailnet -- the mount is then labelled tailscale. + --nfs force NFS (default when the host answers on 2049) + --sshfs force sshfs (default otherwise) + --name PATH override the derived mountpoint entirely + --shared open the share to the $SHARE_GROUP group (2775, setgid) -- the default + --private keep it to one account (0700) + --ro mount read-only + --no-fstab mount now, do not persist across reboots + --dry-run print the mountpoint, fstab line and unit, change nothing + + Examples: + root-ubuntu.sh mount ubuntu@files.example.com:~/Downloads/done --via ubuntu + root-ubuntu.sh mount media.example.com:/srv/media --ro + root-ubuntu.sh umount /mnt/tailscale.files.example.com/Downloads/done + EOF +} + +cmd_mount() { + local spec="" via="" proto="" override="" persist=1 ro=0 dry=0 + local link_name="" want_link=1 shared=1 + + while [[ $# -gt 0 ]]; do + case "$1" in + --via) via="${2:-}"; shift ;; + --nfs) proto=nfs ;; + --sshfs) proto=sshfs ;; + --name) override="${2:-}"; shift ;; + --link) link_name="${2:-}"; shift ;; + --no-link) want_link=0 ;; + --shared) shared=1 ;; + --private) shared=0 ;; + --ro) ro=1 ;; + --no-fstab) persist=0 ;; + --dry-run) dry=1 ;; + -h|--help) mount_usage; return 0 ;; + -*) die "mount: unknown option: $1" ;; + *) [[ -z "$spec" ]] && spec="$1" || die "mount: unexpected argument: $1" ;; + esac + shift + done + [[ -n "$spec" ]] || { mount_usage; return 2; } + [[ "$spec" == *:* ]] || die "mount: expected [user@]host:/remote/path, got '$spec'" + + local hostpart="${spec%%:*}" rpath="${spec#*:}" user host + if [[ "$hostpart" == *@* ]]; then + user="${hostpart%%@*}"; host="${hostpart#*@}" + else + user=root; host="$hostpart" + fi + [[ -n "$host" && -n "$rpath" ]] || die "mount: could not parse '$spec'" + + local remote; remote="$(_expand_remote "$user" "$rpath")" + + # Where we actually talk to the box, and what we therefore call the mount. + local target label="" + if [[ -n "$via" ]]; then + target="$(_tailnet_ip "$via")" \ + || die "mount: '$via' is not a peer in 'tailscale status'" + label=tailscale + info "routing over the tailnet: $via = $target" + elif target="$(_tailnet_ip "$host")"; then + label=tailscale + info "$host is a tailnet peer ($target)" + else + target="$host" + fi + + # NFS if the box is exporting, sshfs if it is not. Probing beats guessing. + if [[ -z "$proto" ]]; then + if _port_open "$target" 2049; then proto=nfs; else proto=sshfs; fi + info "detected transport: $proto" + fi + [[ -n "$label" ]] || label="$proto" + + # The mountpoint mirrors the path as TYPED (~/Downloads/done -> Downloads/done), + # not the expanded one -- /mnt/... /home/ubuntu/Downloads/done reads terribly. + local mpath="${rpath#\~/}"; mpath="${mpath#/}" + local mp="${override:-$MNT_ROOT/$label.$host/$mpath}" + + local rw=rw; [[ "$ro" == 1 ]] && rw=ro + local tsreq="" + [[ "$label" == tailscale ]] && tsreq=",x-systemd.requires=tailscaled.service" + + local src opts fstype + case "$proto" in + nfs) + fstype=nfs + src="$target:$remote" + opts="nfsvers=4.1,proto=tcp,$rw,hard,_netdev,nofail,x-systemd.automount$tsreq" + ;; + sshfs) + # A dry run must not install anything -- just say it would. + if ! command -v sshfs >/dev/null 2>&1; then + if [[ "$dry" == 1 ]]; then + info "sshfs is not installed; a real run would apt-get install it" + else + log "installing sshfs" + apt-get install -y sshfs >/dev/null || die "mount: could not install sshfs" + fi + fi + fstype=fuse.sshfs + src="$user@$target:$remote" + # root does the mounting, so it is root's key that has to be authorised + # on the far side -- not the invoking user's. + opts="$rw,_netdev,nofail,x-systemd.automount$tsreq,allow_other,reconnect,ServerAliveInterval=15,IdentityFile=/root/.ssh/id_ed25519" + [[ -r /root/.ssh/id_ed25519 ]] \ + || warn "no /root/.ssh/id_ed25519 -- ssh-keygen and copy it to $user@$host first" + ;; + *) die "mount: unknown protocol '$proto'" ;; + esac + + # files.example.com -> files. Short, and it is the name you already say out loud. + [[ -n "$link_name" ]] || link_name="${host%%.*}" + + # The gate is the first directory under /mnt, not the mountpoint itself -- + # once NFS is mounted, the mountpoint's own mode comes from the server. + local rel="${mp#"$MNT_ROOT"/}" share_root + share_root="$MNT_ROOT/${rel%%/*}" + + if [[ "$dry" == 1 ]]; then + log "dry run -- nothing was changed" + info "mountpoint : $mp" + info "fstab : $src $mp $fstype $opts 0 0" + info "unit : $(systemd-escape -p --suffix=automount "$mp")" + if [[ "$shared" == 1 ]]; then + info "shared : $share_root becomes $SHARE_DIR_MODE root:$SHARE_GROUP" + else + info "private : $share_root becomes 0700, reachable by $(_owner_desc)" + fi + [[ "$want_link" == 1 ]] && info "link : ~/share/$link_name -> $mp" + return 0 + fi + + [[ $EUID -eq 0 ]] || die "mount: must run as root (try: sudo $0 mount ...)" + + log "mounting $src -> $mp" + install -d -m 0755 "$mp" || die "mount: could not create $mp" + + _share_perms "$share_root" "$shared" + + if [[ "$persist" == 1 ]]; then + cp -a /etc/fstab "/etc/fstab.bak.$(date +%Y%m%d-%H%M%S)" + # Drop any previous entry for this mountpoint or this source, so a repeat + # run replaces its own line instead of stacking a second one beside it. + local tmp; tmp="$(mktemp)" + awk -v mp="$mp" -v src="$src" '$1 == src || $2 == mp { next } { print }' \ + /etc/fstab >"$tmp" && cat "$tmp" >/etc/fstab + rm -f "$tmp" + printf '%s %s %s %s 0 0\n' "$src" "$mp" "$fstype" "$opts" >>/etc/fstab + systemctl daemon-reload + local unit; unit="$(systemd-escape -p --suffix=automount "$mp")" + systemctl start "$unit" 2>/dev/null || true + fi + + mountpoint -q "$mp" || mount "$mp" 2>/dev/null || ls "$mp" >/dev/null 2>&1 + if ! mountpoint -q "$mp"; then + warn "not mounted yet -- check: systemctl status $(systemd-escape -p --suffix=mount "$mp")" + return 1 + fi + + info "mounted: $(ls -1 "$mp" 2>/dev/null | wc -l) entries, $(df -h --output=used "$mp" | tail -1 | tr -d ' ') used" + [[ "$persist" == 1 ]] && info "persisted in /etc/fstab (survives reboot)" + [[ "$want_link" == 1 ]] && _share_link "$mp" "$link_name" + return 0 +} + +cmd_umount() { + local what="${1:-}" + [[ -n "$what" ]] || { mount_usage; return 2; } + [[ $EUID -eq 0 ]] || die "umount: must run as root (try: sudo $0 umount ...)" + + # Accept either the mountpoint itself or the host it came from. + local mp="$what" + if [[ ! -d "$mp" ]]; then + mp="$(awk -v h="$what" -v root="$MNT_ROOT" '$2 ~ ("^" root "/[^/]*\\." h "/") { print $2; exit }' /etc/fstab)" + [[ -n "$mp" ]] || die "umount: no mount found for '$what'" + fi + + log "unmounting $mp" + systemctl stop "$(systemd-escape -p --suffix=automount "$mp")" 2>/dev/null || true + systemctl stop "$(systemd-escape -p --suffix=mount "$mp")" 2>/dev/null || true + umount "$mp" 2>/dev/null || true + mountpoint -q "$mp" && die "umount: $mp is busy -- something is sitting in it (lsof +D $mp)" + + cp -a /etc/fstab "/etc/fstab.bak.$(date +%Y%m%d-%H%M%S)" + local tmp; tmp="$(mktemp)" + awk -v mp="$mp" '$2 == mp { next } { print }' /etc/fstab >"$tmp" && cat "$tmp" >/etc/fstab + rm -f "$tmp" + systemctl daemon-reload + rmdir "$mp" 2>/dev/null || true + + # Leave no symlink pointing at a mountpoint that no longer exists. + local owner="${SUDO_USER:-root}" home l + home="$(getent passwd "$owner" | cut -d: -f6)" + if [[ -n "$home" && -d "$home/share" ]]; then + while IFS= read -r l; do + [[ "$(readlink "$l")" == "$mp" ]] || continue + rm -f "$l" && info "removed link $l" + done < <(find "$home/share" -maxdepth 1 -type l 2>/dev/null) + fi + + info "unmounted and removed from /etc/fstab" + return 0 +} + +cmd_mounts() { + local found=0 + while read -r src mp fstype _; do + [[ "$mp" == "$MNT_ROOT"/* ]] || continue + found=1 + # The gate is the first directory under /mnt, the same one `share` and + # `mount` set -- reporting the mountpoint's own mode would describe the + # far side's opinion, not this box's. + local rel="${mp#"$MNT_ROOT"/}" gate access + gate="$MNT_ROOT/${rel%%/*}" + access="$(stat -c '%A %U:%G' "$gate" 2>/dev/null || echo '? ?')" + if mountpoint -q "$mp"; then + printf ' %-12s %-52s %-22s %s\n' "$fstype" "$mp" "$access" "$src" + else + printf ' %-12s %-52s %-22s %s (not mounted)\n' "$fstype" "$mp" "$access" "$src" + fi + done < <(grep -vE '^\s*(#|$)' /etc/fstab) + [[ "$found" == 1 ]] || info "no shares configured under $MNT_ROOT" + return 0 +} + +# Open (or re-close) a volume that is already mounted. +# +# `mount` only covers shares this script created. A cloud block volume attached +# by the provider arrives root:root 0755 with its own fstab line, and nothing +# here ever touched it -- so it needs the same treatment applied after the fact, +# which is all this does. +cmd_share() { + local shared=1 recurse=0 paths=() + while [[ $# -gt 0 ]]; do + case "$1" in + --private) shared=0 ;; + --shared) shared=1 ;; + -R|--recursive) recurse=1 ;; + -h|--help) mount_usage; return 0 ;; + -*) die "share: unknown option: $1" ;; + *) paths+=("$1") ;; + esac + shift + done + [[ ${#paths[@]} -gt 0 ]] || { mount_usage; return 2; } + [[ $EUID -eq 0 ]] || die "share: must run as root (try: $0 share ... as root)" + + local p + for p in "${paths[@]}"; do + [[ -d "$p" ]] || { warn "share: $p is not a directory -- skipped"; continue; } + mountpoint -q "$p" || warn "share: $p is not a mountpoint (setting it anyway)" + _share_perms "$p" "$shared" + + # Only for a volume that already has data in it. Directories and files get + # separate modes -- a single -R chmod cannot express "traversable dirs, + # non-executable files" without the capital-X trick, and $SHARE_FILE_MODE + # is meant to be an exact mode, not a set of bits to add. + # + # Which means a flat $SHARE_FILE_MODE takes the execute bit off scripts + # and binaries living on the volume. That is the right default for a data + # share and wrong for one holding anything runnable, so it is said out + # loud rather than worked around. + if [[ "$recurse" == 1 ]]; then + log "applying to existing contents of $p" + if [[ "$shared" == 1 ]]; then + local execs + execs="$(find "$p" -type f -perm -u+x -printf . 2>/dev/null | wc -c)" + [[ "$execs" -gt 0 ]] \ + && warn "$execs executable file(s) under $p will lose +x (mode $SHARE_FILE_MODE)" + chgrp -R "$SHARE_GROUP" "$p" 2>/dev/null || warn "chgrp -R fell short on $p" + find "$p" -type d -exec chmod "$SHARE_DIR_MODE" {} + 2>/dev/null || warn "chmod on dirs fell short under $p" + find "$p" -type f -exec chmod "$SHARE_FILE_MODE" {} + 2>/dev/null || warn "chmod on files fell short under $p" + else + chown -R "${SUDO_USER:-root}" "$p" 2>/dev/null || warn "chown -R fell short on $p" + chmod -R go-rwx,g-s "$p" 2>/dev/null || warn "chmod -R fell short on $p" + fi + fi + done + return 0 +} + +usage() { + sed -n '2,/^$/p' "${BASH_SOURCE[0]}" | sed 's/^# \?//' + exit "${1:-0}" +} + +# Subcommands are peeled off before the provisioning flags, so that `mount` +# can take its own options without colliding with them. +SUBCMD="" +SUBARGS=() +case "${1:-}" in + mount|umount|mounts|share) SUBCMD="$1"; shift; SUBARGS=("$@"); set -- ;; +esac + +ARGS=() +CLI_GROUPS="" +while [[ $# -gt 0 ]]; do + case "$1" in + --refresh) ASSUME_YES=1 ;; + --force-dotfiles) FORCE_DOTFILES=1 ;; + --no-reboot) REBOOT_POLICY=0 ;; + --reboot) REBOOT_POLICY=2 ;; + --skip-apt) SKIP_APT=1 ;; + --skip-web) SKIP_WEB=1 ;; + --skip-tailscale) SKIP_TAILSCALE=1 ;; + --skip-tools) SKIP_TOOLS=1 ;; + --skip-dotfiles) SKIP_DOTFILES=1 ;; + # Groups for every account named on this run. Without it an interactive + # run asks per account and an unattended one takes $DEFAULT_GROUPS -- + # which left no way at all to say "these two, in these groups" from a + # pipe, and provisioning accounts from a pipe is the whole point of + # being curl-able. + --groups) CLI_GROUPS="${2:-}"; shift ;; + --groups=*) CLI_GROUPS="${1#*=}" ;; + --yes|-y) ASSUME_YES=1 ;; + -h|--help) usage 0 ;; + -*) die "unknown option: $1 (try --help)" ;; + *) ARGS+=("$1") ;; + esac + shift +done +set -- ${ARGS[@]+"${ARGS[@]}"} + +[[ "${NO_REBOOT:-0}" == 1 ]] && REBOOT_POLICY=0 + +# The share helpers are self-contained: no apt, no lock, no dotfiles +# checkout needed. They come before the root check so that --help and +# --dry-run work as a normal user; each asks for root only when it mutates. +case "$SUBCMD" in + mount) cmd_mount ${SUBARGS[@]+"${SUBARGS[@]}"}; exit $? ;; + umount) cmd_umount ${SUBARGS[@]+"${SUBARGS[@]}"}; exit $? ;; + mounts) cmd_mounts; exit $? ;; + share) cmd_share ${SUBARGS[@]+"${SUBARGS[@]}"}; exit $? ;; +esac + +# Root, not sudo-capable: this writes to /etc, creates accounts and drives +# systemd. Saying which is which up front beats failing on the twentieth step. +if [[ $EUID -ne 0 ]]; then + if command -v sudo >/dev/null 2>&1; then + die "must run as root -- try: sudo $0${*:+ $*}" + fi + die "must run as root (and there is no sudo on this box -- log in as root)" +fi + +# What this actually supports, checked rather than assumed. +# +# The floor is "modern": every mechanism in here -- systemd timers, ufw, +# unattended-upgrades, nginx maps, `http2 on` -- is present on Ubuntu 22.04 and +# newer and on Debian 12. Older releases mostly work and are not worth blocking +# over, so an unrecognised version is a warning: the box may be fine, and a +# hard refusal on a version bump nobody predicted would be its own outage. +check_os() { + local id="" ver="" pretty="" + if [[ -r /etc/os-release ]]; then + # shellcheck disable=SC1091 + . /etc/os-release + id="${ID:-}"; ver="${VERSION_ID:-}"; pretty="${PRETTY_NAME:-}" + fi + command -v apt-get >/dev/null \ + || die "this script targets Debian/Ubuntu (no apt-get found${pretty:+ on $pretty})" + + case "$id" in + ubuntu) + # 22.04 and up. Compared as a number so 24.04 > 22.04 and, more to + # the point, 26.04 does not sort below 9.10 as a string would. + if [[ -n "$ver" ]] && awk -v v="$ver" 'BEGIN { exit !(v + 0 < 22) }'; then + warn "${pretty:-Ubuntu $ver} is older than 22.04 -- expect some steps to fail" + fi + ;; + debian) + if [[ -n "$ver" ]] && awk -v v="$ver" 'BEGIN { exit !(v + 0 < 12) }'; then + warn "${pretty:-Debian $ver} is older than 12 -- expect some steps to fail" + fi + ;; + '') + warn "no /etc/os-release -- assuming Debian-like because apt-get is here" + ;; + *) + warn "${pretty:-$id} is not Ubuntu or Debian; apt-get is here, so continuing" + ;; + esac + + # Not fatal either. Every systemd user in here already falls back to cron or + # degrades with a warning, so a container without it still gets the packages, + # the accounts and the dotfiles. + command -v systemctl >/dev/null 2>&1 \ + || warn "no systemd -- timers become cron jobs and services are not managed" + info "${pretty:-unknown OS} on $(uname -m)" + return 0 +} +check_os + +# Two copies racing each other through apt, chsh and nginx is a good way to +# corrupt exactly the box you were trying to maintain. +exec 9>"$LOCK_FILE" +flock -n 9 || die "another run of $0 is already in progress" + +# Unattended runs (cron, --refresh) leave a trail; interactive ones stay on the +# terminal only, so prompts behave normally. +if ! interactive; then + exec > >(tee -a "$LOG_FILE") 2>&1 + printf '\n===== %s %s =====\n' "$(date -Is)" "$0 $*" +fi + +# Where the dotfiles come from, if they come from anywhere. +# +# This used to be a hard requirement -- the script died unless it was sitting +# inside a dotfiles checkout -- which is exactly what stopped it being usable on +# a machine that had only curled the one file. Dotfiles are now one optional +# stage of many, resolved in this order: +# +# 1. $DOTFILES_DIR -- an existing checkout you point at +# 2. the directory holding this script, if it looks like one (so running it +# from inside a dotfiles repo still Just Works, as it always did) +# 3. $DOTFILES_REPO, cloned to $DOTFILES_CACHE and fast-forwarded on re-runs +# 4. nothing, and the dotfile stages are skipped +# +# A checkout is recognised by its content, not its name: any tree with a +# .zshrc or a .bashrc in it is one. +looks_like_dotfiles() { + [[ -n "${1:-}" && -d "$1" ]] || return 1 + [[ -f "$1/.zshrc" || -f "$1/.bashrc" ]] +} + +resolve_dotfiles() { + if [[ "$SKIP_DOTFILES" == 1 ]]; then + DOTFILES_DIR="" + return 0 + fi + + if looks_like_dotfiles "$DOTFILES_DIR"; then + info "dotfiles: $DOTFILES_DIR" + return 0 + fi + if [[ -n "$DOTFILES_DIR" ]]; then + warn "DOTFILES_DIR=$DOTFILES_DIR has no .zshrc or .bashrc -- ignoring it" + DOTFILES_DIR="" + fi + + if looks_like_dotfiles "$SCRIPT_DIR"; then + DOTFILES_DIR="$SCRIPT_DIR" + info "dotfiles: $DOTFILES_DIR (running from inside them)" + return 0 + fi + + if [[ -n "$DOTFILES_REPO" ]]; then + # Same converge-don't-clobber rule as everywhere else: fetch and reset + # onto the remote head, because a --depth 1 clone cannot always + # fast-forward and a half-updated dotfiles tree is worse than a stale one. + if [[ -d "$DOTFILES_CACHE/.git" ]]; then + git -C "$DOTFILES_CACHE" fetch --quiet --depth 1 origin HEAD \ + && git -C "$DOTFILES_CACHE" reset --quiet --hard FETCH_HEAD \ + || warn "could not update $DOTFILES_CACHE -- using the copy on disk" + else + install -d -m 0755 "$(dirname "$DOTFILES_CACHE")" + git clone --depth 1 --quiet "$DOTFILES_REPO" "$DOTFILES_CACHE" \ + || warn "could not clone $DOTFILES_REPO" + fi + # A dotfiles tree is somebody's shell config and may carry ssh config or + # known_hosts; it has no business being world-readable on a shared box. + [[ -d "$DOTFILES_CACHE" ]] && chmod 0700 "$DOTFILES_CACHE" + if looks_like_dotfiles "$DOTFILES_CACHE"; then + DOTFILES_DIR="$DOTFILES_CACHE" + info "dotfiles: $DOTFILES_REPO -> $DOTFILES_DIR" + return 0 + fi + warn "$DOTFILES_REPO has no .zshrc or .bashrc at its root -- skipping dotfiles" + fi + + DOTFILES_DIR="" + info "no dotfiles source (set DOTFILES_REPO to install a team's shell config)" + return 0 +} + +# The pasted-key store follows the dotfiles when there are some, so a team's +# keys travel with a team's config, and falls back to this box's state +# directory when there are not. Either way it has to be somewhere that survives +# a re-run, or every rebuild would ask for every key again. +resolve_keys_dir() { + [[ -n "$KEYS_DIR" ]] && return 0 + if [[ -n "$DOTFILES_DIR" ]]; then + KEYS_DIR="$DOTFILES_DIR/ssh-keys" + else + KEYS_DIR="$STATE_DIR/ssh-keys" + fi + return 0 +} + +# Both of these are CALLED after the apt stage, not here: cloning a dotfiles +# repo needs git, and git is one of the packages apt_stage installs. Resolving +# before that would fail on exactly the bare box this is meant to bootstrap. + +export DEBIAN_FRONTEND=noninteractive +# needrestart on Ubuntu 22.04+ opens a whiptail dialog mid-upgrade and hangs an +# unattended run forever. Suspend it and report /var/run/reboot-required at the +# end instead of restarting services under people's feet. +export NEEDRESTART_SUSPEND=1 NEEDRESTART_MODE=l + +install -d -m 0755 "$STATE_DIR" +if [[ -e "$STATE_DIR/provisioned" ]]; then + log "maintenance run -- converging $(hostname -s) to the current templates" +else + log "first run on $(hostname -s)" +fi + +load_known_users + +# ------------------------------------------------- collect users + groups --- + +# Numbered menu; accepts "1 3", "1,3" or group names. Empty = $DEFAULT_GROUPS. +ask_groups() { + local login="$1" raw tok out=() i + # --groups was given: that is an answer, so do not ask the question. It wins + # over the prompt as well as over the default, because someone who spelled + # the groups out on the command line has already decided. + if [[ -n "$CLI_GROUPS" ]]; then printf '%s' "$CLI_GROUPS"; return; fi + if ! interactive; then printf '%s' "$DEFAULT_GROUPS"; return; fi + + printf '\n groups for %s:\n' "$login" >&2 + for i in "${!GROUP_CHOICES[@]}"; do + printf ' %d) %s\n' "$((i + 1))" "${GROUP_CHOICES[$i]}" >&2 + done + read -r -p " pick (numbers or names, space/comma separated) [${DEFAULT_GROUPS}]: " raw >&2 + raw="${raw//,/ }" + [[ -z "${raw// }" ]] && { printf '%s' "$DEFAULT_GROUPS"; return; } + + for tok in $raw; do + if [[ "$tok" =~ ^[0-9]+$ ]]; then + if (( tok >= 1 && tok <= ${#GROUP_CHOICES[@]} )); then + out+=("${GROUP_CHOICES[$((tok - 1))]}") + else + warn "ignoring out-of-range choice '$tok'" + fi + else + out+=("$tok") + fi + done + + [[ ${#out[@]} -eq 0 ]] && { printf '%s' "$DEFAULT_GROUPS"; return; } + printf '%s' "$(IFS=,; echo "${out[*]}")" +} + +add_user_spec() { + local spec="$1" login + login="$(user_login "$spec")" + valid_login "$login" || { warn "invalid username derived from '$spec' -- skipped"; return 1; } + USERS+=("$spec") + USER_GROUPS+=("$(ask_groups "$login")") + info "queued ${login} -> groups: ${USER_GROUPS[-1]}" +} + +for arg in "$@"; do + add_user_spec "$arg" +done + +if [[ ${#KNOWN_USERS[@]} -gt 0 ]]; then + log "already provisioned (will be refreshed, not recreated)" + info "${KNOWN_USERS[*]}" +fi + +if interactive; then + log "user accounts" + echo " Existing users above are refreshed automatically -- press enter to skip." + echo " To add someone NEW, enter one account at a time, e.g. alice@example" + while true; do + read -r -p " add user: " spec + [[ -z "${spec// }" ]] && break + case "${spec,,}" in + n|no|done|q|quit|exit) break ;; + esac + add_user_spec "$spec" + done +fi + +# No prompt for the private keys any more. It asked on every interactive run, +# and one mistaken 'y' scatters your own private keys through other people's +# home directories with nothing to undo it. COPY_SSH_PRIVATE_KEYS=1 in the +# environment is now the only way, which is deliberate rather than reflexive. + +# ------------------------------------------------------------------- apt --- + +# Only the OS-level bits: shell, terminal apps we keep configs for, firewall. +# Language runtimes and dev tools are mise's job. +# mosh ships mosh, mosh-client and mosh-server in one package. +BASE_PACKAGES=( + ca-certificates curl git zsh tmux irssi mosh ufw unattended-upgrades + nginx certbot python3-certbot-nginx python3-certbot-dns-cloudflare + # the tools whose dotfiles this repo ships -- without them the configs + # are dead weight and the box feels half-provisioned + vim htop ack screen rsync unzip jq openssl + # ripgrep gives you rg, which is what anyone reaching for `find | rg` + # expects to already be there. ack stays -- it is what .ackrc configures. + ripgrep + # lynx is the backup terminal browser: chawan is the one we want, but it + # is amd64-only and comes from a .deb off the author's site, so there has + # to be something in apt that always works + lynx +) + +# "You can download the latest release (v0.4.4)" on the homepage +chawan_latest_version() { + curl -fsSL --max-time 20 "$CHAWAN_INDEX" 2>/dev/null \ + | sed -n 's/.*latest release (v\([0-9][0-9.]*\)).*/\1/p' | head -1 +} + +# chawan lives outside apt, so this is a hand-rolled "is it current?" check: +# compare the installed version against the one the homepage advertises, and +# only fetch when they differ. Every user gets it -- a .deb lands in /usr/bin. +install_chawan() { + local arch want have tmp got + + arch="$(dpkg --print-architecture)" + if [[ "$arch" != amd64 ]]; then + info "chawan publishes an amd64 .deb only -- skipping on $arch (lynx covers it)" + return 0 + fi + + have="$(dpkg-query -W -f '${Version}' chawan 2>/dev/null)" + + want="$CHAWAN_VERSION" + if [[ -z "$want" ]]; then + want="$(chawan_latest_version)" + if [[ -z "$want" ]]; then + # the site is down: keep whatever is installed rather than churn + [[ -n "$have" ]] && { info "chawan $have installed (could not reach $CHAWAN_INDEX)"; return 0; } + want="$CHAWAN_FALLBACK_VERSION" + warn "could not read the current chawan version -- trying $want" + fi + fi + + if [[ "$have" == "$want" ]]; then + info "chawan $have already current" + return 0 + fi + + tmp="$(mktemp -d)" || return 1 + local url="${CHAWAN_DEB_URL:-https://git.sr.ht/~bptato/chawan/refs/download/v${want}/chawan-${want//./-}-amd64.deb}" + if ! curl -fsSL --max-time 180 "$url" -o "$tmp/chawan.deb"; then + warn "could not download chawan $want from $url" + rm -rf "$tmp"; return 1 + fi + + # A 404 page or a truncated transfer is still a file. Installing it would + # be worse than not installing at all, so make dpkg confirm it is a + # package and that it is the version we went looking for. + if ! dpkg-deb -I "$tmp/chawan.deb" >/dev/null 2>&1; then + warn "the chawan download is not a valid .deb -- skipping (lynx still available)" + rm -rf "$tmp"; return 1 + fi + got="$(dpkg-deb -f "$tmp/chawan.deb" Version 2>/dev/null)" + if [[ -n "$got" && "$got" != "$want" ]]; then + warn "chawan .deb says version $got, expected $want -- installing it anyway" + fi + + # apt-get rather than dpkg -i: it resolves dependencies instead of leaving + # a half-configured package behind if the author ever adds one + if apt-get install -y -qq "$tmp/chawan.deb"; then + note "chawan ${got:-$want} (cha, mancha)" + rm -rf "$tmp"; return 0 + fi + warn "chawan install failed -- lynx is still available" + rm -rf "$tmp"; return 1 +} + +apt_stage() { + log "apt update / upgrade" + try "apt-get update" apt-get update -qq + + # --with-new-pkgs is what unattended-upgrades uses: a plain 'upgrade' holds + # back anything needing a new dependency, so kernels and security updates + # quietly never land. confdef+confold keeps every conffile prompt away. + local upgraded_before upgraded_after + upgraded_before="$(dpkg-query -W -f '${Package}=${Version}\n' 2>/dev/null | sha256sum)" + try "apt-get upgrade" apt-get -y --with-new-pkgs \ + -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold upgrade + try "apt-get autoremove" apt-get -y autoremove + upgraded_after="$(dpkg-query -W -f '${Package}=${Version}\n' 2>/dev/null | sha256sum)" + [[ "$upgraded_before" != "$upgraded_after" ]] && note "apt packages changed" + + log "installing base packages" + # One apt-get for everything present, then retry the stragglers one at a + # time so a single unavailable package cannot take the whole batch down. + local missing=() p + for p in "${BASE_PACKAGES[@]}"; do + dpkg-query -W -f '${Status}' "$p" 2>/dev/null | grep -q "^install ok installed$" \ + || missing+=("$p") + done + if [[ ${#missing[@]} -gt 0 ]]; then + note "installing: ${missing[*]}" + apt-get install -y -qq "${missing[@]}" || for p in "${missing[@]}"; do + try "install $p" apt-get install -y -qq "$p" + done + else + info "all base packages already installed" + fi + + # outside apt, so it gets its own step rather than a package name + log "installing chawan (TUI browser)" + try "chawan" install_chawan + + log "enabling unattended security upgrades" + write_if_changed /etc/apt/apt.conf.d/20auto-upgrades <<'EOF' && note "20auto-upgrades" +APT::Periodic::Update-Package-Lists "1"; +APT::Periodic::Unattended-Upgrade "1"; +APT::Periodic::AutocleanInterval "7"; +EOF + return 0 +} + +if [[ "$SKIP_APT" == 1 ]]; then + log "skipping apt (--skip-apt)" +else + apt_stage +fi + +# Now that git exists, work out where the dotfiles are coming from -- see +# resolve_dotfiles for the order it tries. +log "resolving dotfiles source" +resolve_dotfiles +resolve_keys_dir + +# ------------------------------------------------------------------- ufw --- + +log "configuring ufw" +# The -n guard is not redundant with the -f test: with no dotfiles checkout the +# path collapses to /etc/default/ufw, which is the very file being written, and +# the comparison would be a file against itself. +if [[ -n "$DOTFILES_DIR" && -f "$DOTFILES_DIR/etc/default/ufw" ]] \ + && ! cmp -s "$DOTFILES_DIR/etc/default/ufw" /etc/default/ufw; then + try "install /etc/default/ufw" install -m 0644 "$DOTFILES_DIR/etc/default/ufw" /etc/default/ufw \ + && note "/etc/default/ufw" +fi +try "ufw default deny incoming" ufw --force default deny incoming +try "ufw default allow outgoing" ufw --force default allow outgoing + +# ssh first -- opening it before enabling is what stops you locking yourself out +try "ufw allow ${SSH_PORT}/tcp (ssh)" ufw allow "${SSH_PORT}/tcp" +try "ufw allow 80/tcp (http)" ufw allow 80/tcp +try "ufw allow 443/tcp (https)" ufw allow 443/tcp +# mosh picks a UDP port in this range per session; without it mosh cannot connect +try "ufw allow 60000:61000/udp (mosh)" ufw allow 60000:61000/udp + +# 'ufw enable' on an already-active firewall reloads the whole ruleset, which +# briefly drops packets on a box people are logged into. Only enable it when +# it is actually off. +if ufw status 2>/dev/null | head -1 | grep -q 'inactive'; then + try "ufw enable" ufw --force enable && note "ufw enabled" +else + info "ufw already active" +fi +try "enable ufw at boot" systemctl enable ufw +ufw status verbose || true + +# ------------------------------------------------------------------ motd --- + +# Fetched into a cache file; login just cats it and kicks off a background +# refresh, so a slow/dead endpoint can never hang an ssh login. +install_motd() { + local units_changed=0 + write_if_changed /usr/local/bin/profullstack-motd 0755 </dev/null && [[ -d /etc/systemd/system ]]; then + write_if_changed /etc/systemd/system/profullstack-motd.service <<'EOF' && units_changed=1 +[Unit] +Description=Refresh the profullstack motd +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=/usr/local/bin/profullstack-motd +EOF + write_if_changed /etc/systemd/system/profullstack-motd.timer <<'EOF' && units_changed=1 +[Unit] +Description=Refresh the profullstack motd daily + +[Timer] +OnCalendar=daily +RandomizedDelaySec=1h +Persistent=true + +[Install] +WantedBy=timers.target +EOF + # reloading systemd on every run is pointless churn + [[ "$units_changed" == 1 ]] && { systemctl daemon-reload; note "motd timer units"; } + systemctl is-enabled profullstack-motd.timer >/dev/null 2>&1 \ + || systemctl enable --now profullstack-motd.timer + systemctl is-active profullstack-motd.timer >/dev/null 2>&1 \ + || systemctl start profullstack-motd.timer + else + warn "no systemd -- falling back to /etc/cron.daily for the motd refresh" + ln -sf /usr/local/bin/profullstack-motd /etc/cron.daily/profullstack-motd + fi + + # Canonical's motd-news would otherwise print ads above ours. + [[ -f /etc/default/motd-news ]] \ + && sed -i 's/^ENABLED=.*/ENABLED=0/' /etc/default/motd-news + + # populate the cache now so the very first login already shows it + /usr/local/bin/profullstack-motd + [[ -s "$MOTD_CACHE" ]] || { warn "could not fetch ${MOTD_URL} (will retry on next login)"; return 1; } + return 0 +} + +log "configuring motd from ${MOTD_URL}" +try "motd" install_motd + +# --------------------------------------------------------------- dotfiles --- + +# Plain copies. .tmux.conf / .irssi are handled by their own installers below. +DOTFILES=( + .zshrc .zshenv .zsh_aliases + .bashrc .bash_profile .bash_aliases .bash_aliases_linux .bash_env .bash_env_linux + .vimrc .ackrc .htoprc .gitconfig .screenrc +) + +install_ssh() { + local home="$1" owner="$2" f + local src="" + [[ -n "$DOTFILES_DIR" && -d "$DOTFILES_DIR/.ssh" ]] && src="$DOTFILES_DIR/.ssh" + + # NOT gated on there being a dotfiles tree. This function does two separate + # jobs -- authorise the key filed for this account, and install the ssh + # config that came with the dotfiles -- and only the second one needs a + # checkout. Returning early when there is no $src (which is what an earlier + # version did) meant a box with no dotfiles repo silently authorised nobody, + # so accounts were created that could not log in. + [[ -d "$home/.ssh" ]] || install -d -m 0700 -o "$owner" -g "$owner" "$home/.ssh" + + # authorized_keys belongs to the user. FULL STOP. + # + # This used to merge the repo's own .ssh/authorized_keys -- the admin's + # keys -- into every account on every run, and rewrite the file with + # sort -u while it was at it. That is not this script's file to edit: + # who may log in as someone is their decision, re-adding keys they have + # removed defeats the point of removing them, and rewriting the file at + # all risks the access it is supposed to protect. + # + # The ONLY thing written here is an explicit per-user key you have put in + # ssh-keys/.pub, appended if it is not already present. No file in + # ssh-keys means nothing is touched. + local ukey="$KEYS_DIR/$owner.pub" + if [[ -s "$ukey" ]]; then + local ak="$home/.ssh/authorized_keys" keydata + # compare on the key body, so a changed comment is not a new key + keydata="$(awk '{print $1" "$2}' "$ukey" | head -1)" + if [[ -s "$ak" ]] && awk '{print $1" "$2}' "$ak" | grep -qxF "$keydata"; then + : # already authorised + else + # append on a line of its own: if the existing file has no trailing + # newline, a bare >> would graft this key onto the last one and + # destroy both + [[ -s "$ak" && -n "$(tail -c1 "$ak" 2>/dev/null)" ]] && printf '\n' >>"$ak" + cat "$ukey" >>"$ak" + note "authorised ssh-keys/$owner.pub for $owner" + fi + chown "$owner:$owner" "$ak" 2>/dev/null + chmod 0600 "$ak" 2>/dev/null + fi + + # Everything below comes OUT of a dotfiles checkout, so it only runs when + # there is one. Without it the account keeps whatever ssh config it had, + # which is the right answer -- this script has no config of its own to put + # there and inventing one would overwrite theirs with nothing. + if [[ -n "$src" ]]; then + # ~/.ssh/config is personal (host aliases, ProxyJump, per-host keys), so + # it gets the same treatment as any other dotfile: ours until they edit it. + install_managed "$src/config" "$home/.ssh/config" "$owner" 0600 .ssh/config + + # known_hosts is append-only by nature -- overwriting it throws away every + # host the user has accepted since, and then ssh starts asking again. + if [[ -s "$src/known_hosts" ]]; then + touch "$home/.ssh/known_hosts" + if ! sort -u "$src/known_hosts" "$home/.ssh/known_hosts" \ + | cmp -s - "$home/.ssh/known_hosts"; then + sort -u "$src/known_hosts" "$home/.ssh/known_hosts" \ + >"$home/.ssh/known_hosts.new" \ + && mv "$home/.ssh/known_hosts.new" "$home/.ssh/known_hosts" + fi + fi + + # Handing your own private keys to other accounts is never something to + # do by accident, so there is no prompt for it -- it happens only if + # COPY_SSH_PRIVATE_KEYS=1 is set in the environment on purpose, and even + # then never over a key the user already has. + if [[ "$COPY_SSH_PRIVATE_KEYS" == 1 ]]; then + warn "COPY_SSH_PRIVATE_KEYS=1: copying $src's PRIVATE keys into $home/.ssh" + for f in "$src"/id_*; do + [[ -f "$f" && ! -e "$home/.ssh/$(basename "$f")" ]] \ + && cp -f "$f" "$home/.ssh/$(basename "$f")" + done + fi + fi + + # Permissions only, and only on what we may have created. sshd StrictModes + # needs 0700 on ~/.ssh and 0600 on private files, but chown -R over the + # whole directory would also rewrite files this script never wrote. + chmod 0700 "$home/.ssh" + chown "$owner:$owner" "$home/.ssh" + [[ -e "$home/.ssh/authorized_keys" ]] && chmod 0600 "$home/.ssh/authorized_keys" + return 0 +} + +# oh-my-tmux. The .tmux.conf vendored in this repo is an OLD oh-my-tmux with no +# tpm support, while our .tmux.conf.local sets '@plugin tmux-yank' -- so the +# plugin never loaded. Clone upstream and keep only our .local overrides, which +# is the layout oh-my-tmux actually supports. +install_tmux() { + local home="$1" owner="$2" + + # a ~/.tmux that is not our checkout is the user's own tmux setup; leave + # the whole thing alone rather than half-converting it to oh-my-tmux + if [[ -d "$home/.tmux" && ! -d "$home/.tmux/.git" ]]; then + info "$owner has their own ~/.tmux -- left alone" + return 0 + fi + + if clone_or_pull "$owner" https://github.com/gpakosz/.tmux.git "$home/.tmux"; then + # only (re)point the symlink -- if the user replaced ~/.tmux.conf with + # a real file of their own, that is their config now + if [[ -L "$home/.tmux.conf" || ! -e "$home/.tmux.conf" ]]; then + as_user "$owner" "ln -sfn '$home/.tmux/.tmux.conf' '$home/.tmux.conf'" + else + info "$owner has their own ~/.tmux.conf -- left alone" + fi + elif [[ -n "$DOTFILES_DIR" && -f "$DOTFILES_DIR/.tmux.conf" ]]; then + warn "oh-my-tmux clone failed -- falling back to the dotfiles .tmux.conf" + install_managed "$DOTFILES_DIR/.tmux.conf" "$home/.tmux.conf" "$owner" 0644 .tmux.conf + else + warn "no tmux config available for $owner" + return 1 + fi + + # .tmux.conf.local is THE file oh-my-tmux expects you to customise, so it + # is the one most likely to have been edited. install_managed keeps theirs. + [[ -n "$DOTFILES_DIR" ]] \ + && install_managed "$DOTFILES_DIR/.tmux.conf.local" "$home/.tmux.conf.local" \ + "$owner" 0644 .tmux.conf.local + + # our .tmux.conf.local enables tpm plugins; pre-seed tpm so the first + # tmux launch works even with no network. + if [[ -f "$home/.tmux.conf.local" ]] \ + && grep -qE '^[[:space:]]*set -g @plugin' "$home/.tmux.conf.local"; then + clone_or_pull "$owner" https://github.com/tmux-plugins/tpm.git \ + "$home/.tmux/plugins/tpm" || warn "tpm clone failed for $owner" + as_user "$owner" "'$home/.tmux/plugins/tpm/bin/install_plugins' >/dev/null 2>&1" || true + fi + + chown -h "$owner:$owner" "$home/.tmux.conf" 2>/dev/null + chown -R "$owner:$owner" "$home/.tmux" "$home/.tmux.conf.local" 2>/dev/null + return 0 +} + +# irssi: configs + perl scripts. autorun/ holds relative symlinks into +# scripts/, so copy with -a to keep them intact. +install_irssi() { + local home="$1" owner="$2" f + [[ -n "$DOTFILES_DIR" ]] || return 0 + local src="$DOTFILES_DIR/.irssi" + [[ -d "$src" ]] || return 0 + + install -d -m 0700 -o "$owner" -g "$owner" "$home/.irssi" + # irssi rewrites its own config on /save -- servers, channels, nickserv + # passwords. Clobbering it on a re-run would sign people out of their + # networks, so these are managed files like any other. + install_managed "$src/config" "$home/.irssi/config" "$owner" 0600 .irssi/config + install_managed "$src/config-tor" "$home/.irssi/config-tor" "$owner" 0600 .irssi/config-tor + install_managed "$src/default.theme" "$home/.irssi/default.theme" "$owner" 0644 .irssi/default.theme + + # scripts/ is ours: perl we ship, plus autorun/ symlinks into it. -a keeps + # the symlinks; -u leaves anything the user has since made newer. + [[ -d "$src/scripts" ]] && cp -au "$src/scripts" "$home/.irssi/" + + chown -R "$owner:$owner" "$home/.irssi" + return 0 +} + +# oh-my-zsh. KEEP_ZSHRC is essential -- the installer would otherwise replace +# the .zshrc we just installed with its template. +# A ~/.oh-my-zsh with no oh-my-zsh.sh and no .git is wreckage from a run that +# failed partway: the plugin mkdir below builds the tree even when the +# installer died. The installer then refuses to touch an existing $ZSH folder +# ("The $ZSH folder already exists"), so the account can never heal itself -- +# which is how one bad run left ubuntu without oh-my-zsh permanently. +# +# custom/ is the only part worth keeping: it is where a user's own themes and +# plugins live. Salvage it, clear the rest, let the installer run. +_repair_partial_omz() { + local login="$1" home="$2" + [[ -d "$home/.oh-my-zsh" ]] || return 0 + [[ -f "$home/.oh-my-zsh/oh-my-zsh.sh" ]] && return 0 + [[ -d "$home/.oh-my-zsh/.git" ]] && return 0 + + warn "$login has a half-installed ~/.oh-my-zsh -- repairing" + rm -rf "$home/.oh-my-zsh.salvage" + [[ -d "$home/.oh-my-zsh/custom" ]] \ + && mv "$home/.oh-my-zsh/custom" "$home/.oh-my-zsh.salvage" + rm -rf "$home/.oh-my-zsh" + return 0 +} + +_restore_omz_custom() { + local login="$1" home="$2" + [[ -d "$home/.oh-my-zsh.salvage" ]] || return 0 + if [[ -d "$home/.oh-my-zsh" ]]; then + install -d -m 0755 -o "$login" -g "$login" "$home/.oh-my-zsh/custom" + # -n: never overwrite what the fresh install just put there + cp -an "$home/.oh-my-zsh.salvage/." "$home/.oh-my-zsh/custom/" 2>/dev/null + rm -rf "$home/.oh-my-zsh.salvage" + chown -R "$login:$login" "$home/.oh-my-zsh" + fi + return 0 +} + +install_omz() { + local login="$1" home; home="$(user_home "$login")" + + _repair_partial_omz "$login" "$home" + + if [[ ! -f "$home/.oh-my-zsh/oh-my-zsh.sh" ]]; then + # Do not abort on a non-zero exit here: the installer can clone + # ~/.oh-my-zsh successfully and still fail on a later step, and + # bailing out skipped the plugin clones below -- which is exactly how + # 'plugin zsh-autosuggestions not found' survived a "successful" run. + # ZSH= and ZDOTDIR= are pinned, not left to the environment: the + # installer prefers whatever it finds there over $HOME, and getting + # that wrong clones into somebody else's home. as_user already starts + # from a clean environment; this keeps it right even if that changes. + as_user "$login" "ZSH='$home/.oh-my-zsh' ZDOTDIR='$home' RUNZSH=no CHSH=no KEEP_ZSHRC=yes sh -c \"\$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\"" \ + || warn "oh-my-zsh installer exited non-zero for $login -- checking result anyway" + elif [[ -d "$home/.oh-my-zsh/.git" ]]; then + # already installed: this run is an update. Reset rather than pull -- + # omz is a full clone people sometimes edit in place, and a dirty tree + # would make every future run fail on "local changes would be lost". + # + # FETCH_HEAD, not origin/HEAD: the omz installer sets the repo up with + # git init + remote add + a shallow fetch, which leaves no + # refs/remotes/origin/HEAD on the git in 22.04/24.04. + as_user "$login" "git -C '$home/.oh-my-zsh' fetch --quiet --depth 1 origin HEAD \ + && git -C '$home/.oh-my-zsh' reset --quiet --hard FETCH_HEAD" \ + || warn "could not update oh-my-zsh for $login" + fi + + _restore_omz_custom "$login" "$home" + + local rc=0 pdir="$home/.oh-my-zsh/custom/plugins" + if [[ ! -f "$home/.oh-my-zsh/oh-my-zsh.sh" ]]; then + warn "oh-my-zsh missing for $login" + rc=1 + fi + + # external plugins our .zshrc references. Verify afterwards rather than + # trusting the clone: a missing plugin only shows up as an [oh-my-zsh] + # warning at login, long after this script has said it succeeded. + as_user "$login" "mkdir -p '$pdir'" + + clone_or_pull "$login" https://github.com/zsh-users/zsh-autosuggestions.git \ + "$pdir/zsh-autosuggestions" || true + if [[ ! -f "$pdir/zsh-autosuggestions/zsh-autosuggestions.plugin.zsh" ]]; then + warn "zsh-autosuggestions missing for $login (plugins=() in .zshrc references it)" + rc=1 + fi + + clone_or_pull "$login" https://github.com/lukechilds/zsh-nvm.git "$home/.zsh-nvm" || true + if [[ ! -f "$home/.zsh-nvm/zsh-nvm.plugin.zsh" ]]; then + warn "zsh-nvm missing for $login (.zshrc sources it)" + rc=1 + fi + + chown -R "$login:$login" "$home/.oh-my-zsh" "$home/.zsh-nvm" 2>/dev/null + return $rc +} + +install_dotfiles() { + local home="$1" owner="$2" f + [[ -d "$home" ]] || { warn "no home dir $home"; return 1; } + + if [[ -n "$DOTFILES_DIR" ]]; then + for f in "${DOTFILES[@]}"; do + # install_managed handles "not there", "same file" (repo cloned to + # /root) and "the user has since edited it" + install_managed "$DOTFILES_DIR/$f" "$home/$f" "$owner" 0644 "$f" + done + fi + + # These three run either way. install_ssh authorises the account's own key, + # and install_tmux installs oh-my-tmux from upstream -- neither needs a + # dotfiles checkout, and skipping them without one is how an account ends up + # provisioned but unable to log in. + install_ssh "$home" "$owner" + install_tmux "$home" "$owner" + install_irssi "$home" "$owner" +} + +# chsh refuses any shell that is not listed in /etc/shells, and it fails +# quietly enough to leave a user on bash. Register zsh, switch, then confirm +# from the passwd entry rather than trusting the exit status. +ensure_zsh_shell() { + local login="$1" zsh_bin actual + zsh_bin="$(command -v zsh)" || { warn "zsh is not installed"; return 1; } + grep -qxF "$zsh_bin" /etc/shells 2>/dev/null || echo "$zsh_bin" >>/etc/shells + + actual="$(getent passwd "$login" | cut -d: -f7)" + [[ "$actual" == "$zsh_bin" ]] && return 0 + + # Only convert from the distro defaults. If someone has deliberately moved + # to fish or dash, a maintenance run has no business dragging them back -- + # that is precisely the kind of "it broke my account" a re-run must avoid. + case "$actual" in + */zsh) return 0 ;; + */bash|*/sh|"") ;; + *) info "$login chose $actual as their shell -- left alone"; return 0 ;; + esac + + chsh -s "$zsh_bin" "$login" 2>/dev/null || warn "chsh failed for $login" + actual="$(getent passwd "$login" | cut -d: -f7)" + [[ "$actual" == "$zsh_bin" ]] || { warn "$login login shell is $actual, not $zsh_bin"; return 1; } + note "$login login shell -> zsh" + return 0 +} + +# sshd runs with StrictModes yes by default and silently IGNORES +# authorized_keys when the home or .ssh is writable by group or other -- giving +# the same "Permission denied" as having no key at all, which is miserable to +# debug. nginx separately needs o+x on the home to reach public_html. So the +# target is: owned by the user, group/other write stripped, o+x kept. +fix_home_permissions() { + local home="$1" login="$2" oct + [[ -d "$home" ]] || { warn "no home dir $home"; return 1; } + + chown "$login:$login" "$home" + chmod g-w,o-w "$home" + chmod o+x "$home" + + if [[ -d "$home/.ssh" ]]; then + chown -R "$login:$login" "$home/.ssh" + chmod 0700 "$home/.ssh" + find "$home/.ssh" -type f ! -name '*.pub' -exec chmod 0600 {} + + find "$home/.ssh" -type f -name '*.pub' -exec chmod 0644 {} + + fi + + # nginx (www-data) has to be able to read these. a+rX adds what it needs + # without flattening modes the user set deliberately -- a blanket 0644 + # would, for instance, strip the +x off anything they keep in there. + if [[ -d "$home/public_html" ]]; then + chown -R "$login:$login" "$home/public_html" + chmod -R a+rX,go-w "$home/public_html" + fi + # ~/share holds symlinks to /mnt mountpoints (_share_link). Someone who + # publishes ~/public_html/done -> ~/share/seed means for nginx to follow it, + # and no x here breaks that whatever the mountpoint allows. Hand the x to + # $WEB_GROUP rather than to the world: nginx is the only thing that needs to + # walk through. No r, so it still cannot list which shares exist, and what + # may actually be reached is still decided in /mnt. + # + # Only the shortcut directory itself -- never the mounts under it, which + # would push a chmod out over NFS. + if [[ -d "$home/share" ]] && getent group "$WEB_GROUP" >/dev/null; then + chgrp "$WEB_GROUP" "$home/share" && chmod 0710 "$home/share" + fi + + # Dev apps: only open up what is actually published. An app is published + # when it has a public/ or a .port -- everything else under ~/apps is + # private work that nginx never serves and must stay that way. + if [[ -d "$home/apps" ]]; then + chmod o+x "$home/apps" + for _app in "$home"/apps/*/; do + [[ -d "$_app" ]] || continue + [[ -d "$_app/public" || -f "$_app/.port" ]] || continue + chmod o+x "$_app" + [[ -d "$_app/public" ]] && chmod -R a+rX "$_app/public" + done + unset _app + fi + + # report, and complain if sshd would still refuse the keys + oct="$(stat -c '%a' "$home")" + info "$home mode=$oct owner=$(stat -c '%U:%G' "$home")" + if (( (8#$oct & 8#022) != 0 )); then + warn "$home is group/other writable -- sshd StrictModes will ignore authorized_keys" + return 1 + fi + if [[ "$(stat -c '%U' "$home")" != "$login" ]]; then + warn "$home is not owned by $login -- sshd StrictModes will ignore authorized_keys" + return 1 + fi + return 0 +} + +install_mise() { as_user "$1" 'curl -fsSL https://mise.run | sh'; } + +# Installs/repairs moshcode itself. Re-running is the supported update path and +# it replaces ~/.moshcode/pkg wholesale, which is what heals a partial install. +install_moshcode() { as_user "$1" 'curl -fsSL https://moshcode.sh/install.sh | sh'; } + +# ...and this updates the engines and workflow CLIs moshcode manages. +# +# The two are NOT the same command, which is the trap. The wrapper at +# ~/.local/bin/moshcode intercepts `upgrade` and re-runs the installer, so +# `moshcode upgrade` only ever refreshes moshcode/node/bun and reports +# "Update complete" -- while gh, supabase, doctl and friends quietly rot. The +# CLI's real upgrade, the one that walks the installed tools, is reachable +# only by calling bin/moshcode.mjs directly. +# +# node comes from mise and is not on the default PATH, so the shims directory +# has to be prepended exactly as the wrapper does. +update_moshcode_tools() { + local login="$1" home out failed rc=0 + home="$(user_home "$login")" + [[ -f "$home/.moshcode/pkg/bin/moshcode.mjs" ]] || return 0 + + out="$(as_user "$login" \ + "PATH=\"\$HOME/.local/share/mise/shims:\$HOME/.local/bin:\$PATH\" \ + node '$home/.moshcode/pkg/bin/moshcode.mjs' upgrade" 2>&1)" + + # show the per-tool lines, not the whole download log + printf '%s\n' "$out" | grep -aE '^(⬆|✓ upgraded|✗ upgraded)' | sed 's/^/ /' + + # tailscale is the expected casualty: `tailscale update` refuses to run as + # anyone but root, and on this box tailscale comes from apt and is + # upgraded in apt_stage anyway. Failing the whole step over it would mean + # every run ends with a red mark nobody can act on. + failed="$(printf '%s\n' "$out" | grep -a 'failed:' \ + | sed 's/.*failed: //; s/🤘.*//' \ + | tr ',' '\n' | tr -d ' ' | grep -v '^$' | grep -vx tailscale)" + + # Reported, never fatal. moshcode installs from the tip of main, so the box + # tracks whatever is on that branch -- including half-finished refactors, + # where a tool this ran fine against yesterday becomes an "unknown upgrade + # target" today. None of that is something this script can act on, and a + # provisioned box is not broken because a third-party updater had a bad + # day. Same call as tailscale, for the same reason. + if [[ -n "$failed" ]]; then + warn "$login: moshcode could not upgrade: $(printf '%s' "$failed" | tr '\n' ' ')" + info " (moshcode tracks moshcoder/moshcode@main -- retry later, or pin MOSHCODE_REF)" + fi + return 0 +} + +# ------------------------------------------------------------- tailscale --- + +# Joining a tailnet needs a credential. With TS_AUTHKEY it is unattended; +# without one, tailscale up prints a URL to approve in a browser -- so this +# reports what to run rather than hanging on an interactive login. +install_tailscale() { + if ! command -v tailscale >/dev/null; then + curl -fsSL https://tailscale.com/install.sh | sh || return 1 + fi + systemctl enable --now tailscaled || return 1 + + # tailscale ip only succeeds once the node is actually logged in + if tailscale ip -4 >/dev/null 2>&1; then + info "already on the tailnet as $(tailscale ip -4 | head -1)" + elif [[ -n "$TS_AUTHKEY" ]]; then + tailscale up --authkey="$TS_AUTHKEY" --hostname="$TS_HOSTNAME" \ + || { warn "tailscale up failed (bad or expired auth key?)"; return 1; } + else + warn "not joined to a tailnet: no TS_AUTHKEY set" + info " unattended: sudo TS_AUTHKEY=tskey-auth-... $0" + info " or by hand: sudo tailscale up --hostname=$TS_HOSTNAME" + return 1 + fi + + # Tailnet traffic is trusted; without this the default deny-incoming + # policy drops it. 41641/udp lets peers connect directly instead of + # relaying through DERP. + ufw allow in on tailscale0 >/dev/null 2>&1 || warn "could not allow tailscale0 in ufw" + ufw allow 41641/udp >/dev/null 2>&1 || warn "could not allow 41641/udp in ufw" + return 0 +} + +# ----------------------------------------------------------- user web dirs --- + +# Every default page this script has ever shipped, by revision. A file that +# matches one of them byte for byte has never been edited by its owner, so +# rewriting it is safe; anything else is the user's page and is left alone. +# +# Add a revision here rather than editing an old one -- the old text is what +# lets an already-provisioned box recognise its own default and move on. +_public_html_default_page() { + local login="$1" rev="$2" + + # rev 1: before the sponsor ad + [[ "$rev" == 1 ]] && cat < + + +${login} +

${login}

+

Default page for ${login} on ${WEB_DOMAIN}. +

Email: ${login}@${MAIL_DOMAIN} +

Edit ~/public_html/index.html to replace it. +EOF + + # rev 2: sponsor ad at the top, expanded by nginx on the way out. The URI is + # spelled out rather than interpolated: $SPONSOR_AD_URI has moved on since, + # and a revision that changes under us stops recognising the pages it wrote. + [[ "$rev" == 2 ]] && cat < + + +${login} + + +

${login}

+

Default page for ${login} on ${WEB_DOMAIN}. +

Email: ${login}@${MAIL_DOMAIN} +

Edit ~/public_html/index.html to replace it. +EOF + + # rev 3: the ad became a pool that rotates per request, so the token gained + # the trailing slash random_index needs. Keep the token to keep the ad; + # delete the line to drop it. + [[ "$rev" == 3 ]] && cat < + + +${login} + + +

${login}

+

Default page for ${login} on ${WEB_DOMAIN}. +

Email: ${login}@${MAIL_DOMAIN} +

Edit ~/public_html/index.html to replace it. +EOF + return 0 +} + +# newest revision, i.e. what a fresh page gets written as +_public_html_current_rev() { [[ "$SPONSOR_AD" == 1 ]] && echo 3 || echo 1; } + +# ~/public_html served at https://$WEB_DOMAIN/~user and https://user.$WEB_DOMAIN +# +# nginx runs as www-data, so it needs to traverse the home directory. o+x (not +# o+r) on $HOME lets it walk through without making the home itself listable. +install_public_html() { + local home="$1" login="$2" doc="$1/public_html" rev want cur + + # create if absent; an existing one keeps whatever mode its owner chose + [[ -d "$doc" ]] || install -d -m 0755 -o "$login" -g "$login" "$doc" + chmod o+x "$home" + + cur="$(_public_html_current_rev)" + want=0 + if [[ ! -e "$doc/index.html" ]]; then + want=1 + else + # still one of our defaults? then it is ours to update. Anything the + # user has touched -- including a page they wrote from scratch -- fails + # every comparison and is never clobbered. + for rev in 1 2 3; do + [[ "$rev" == "$cur" ]] && continue + if cmp -s <(_public_html_default_page "$login" "$rev") "$doc/index.html"; then + want=1 + break + fi + done + fi + + if [[ "$want" == 1 ]]; then + _public_html_default_page "$login" "$cur" >"$doc/index.html" + chown "$login:$login" "$doc/index.html" + chmod 0644 "$doc/index.html" + fi + return 0 +} + +# ~/apps/ becomes https://..$WEB_DOMAIN +# +# Static: files in ~/apps//public. Dynamic: put the port in +# ~/apps//.port and listen on 127.0.0.1: -- nginx proxies to it, +# so nothing extra has to be opened in ufw. +install_dev_apps_dir() { + local home="$1" login="$2" dir="$1/apps" fresh=0 + [[ "$DEV_APPS" == 1 ]] || return 0 + + # ~/apps is a common enough name that an adopted account may already have + # one full of private work. Create it if it is absent, but never restat an + # existing one -- 'install -d -m 0755' would relax a 0700 directory and + # publish the listing. + if [[ ! -d "$dir" ]]; then + install -d -m 0755 -o "$login" -g "$login" "$dir" + fresh=1 + fi + # o+x is traverse-only: nginx can reach a published app underneath without + # ~/apps itself becoming listable. + chmod o+x "$home" "$dir" + + if [[ "$fresh" == 1 && ! -e "$dir/README" ]]; then + cat >"$dir/README" <.${login}.${WEB_DOMAIN} + +Static site -- files go in public/: + mkdir -p ~/apps/blog/public + echo hello > ~/apps/blog/public/index.html + -> https://blog.${login}.${WEB_DOMAIN} + +Running app (anything that listens on 127.0.0.1): + mkdir -p ~/apps/api + echo 3000 > ~/apps/api/.port + # start your server on 127.0.0.1:3000 + -> https://api.${login}.${WEB_DOMAIN} + +Names must be lowercase letters, digits and dashes. New apps are picked up +within a minute (profullstack-devapps.timer); no root, no restart, no +re-provision. Bind to 127.0.0.1, not 0.0.0.0 -- the firewall blocks the port +directly and nginx is what terminates TLS for you. +EOF + chown "$login:$login" "$dir/README" + fi + return 0 +} + +# nginx cannot read a port out of a file per request, so the ~/apps/*/.port +# files are compiled into one map. A timer re-runs this, which is what makes +# "create a directory and it is live" work without root. +write_devapps_generator() { + [[ "$DEV_APPS" == 1 ]] || return 0 + local units_changed=0 + + write_if_changed /usr/local/bin/profullstack-devapps 0755 <..${WEB_DOMAIN} port map from ~/apps/*/.port. +# Generated by cli-tools/root-ubuntu.sh -- edits here are overwritten. +set -uo pipefail +map_file='${DEV_APPS_MAP}' +tmp="\$(mktemp)" || exit 1 + +{ + echo "# generated by profullstack-devapps -- do not edit" + echo 'map "\$devapp_user/\$devapp_name" \$devapp_port {' + echo ' default "";' + for pf in /home/*/apps/*/.port; do + [ -f "\$pf" ] || continue + appdir="\$(dirname "\$pf")" + app="\$(basename "\$appdir")" + user="\$(basename "\$(dirname "\$(dirname "\$appdir")")")" + port="\$(tr -cd '0-9' <"\$pf" | head -c 5)" + # a user must not be able to point us at someone else's service, and + # the low ports are root's + [ -n "\$port" ] || continue + [ "\$port" -ge 1024 ] 2>/dev/null || continue + [ "\$port" -le 65535 ] 2>/dev/null || continue + case "\$app" in *[!a-z0-9-]*|-*|'') continue ;; esac + case "\$user" in *[!a-z0-9_-]*|'') continue ;; esac + printf '\t"%s/%s" %s;\n' "\$user" "\$app" "\$port" + done + echo '}' +} >"\$tmp" + +if ! cmp -s "\$tmp" "\$map_file"; then + install -m 0644 "\$tmp" "\$map_file" + nginx -t >/dev/null 2>&1 && systemctl reload nginx +fi +rm -f "\$tmp" +EOF + + write_if_changed /etc/systemd/system/profullstack-devapps.service <<'EOF' && units_changed=1 +[Unit] +Description=Compile per-user dev app routes for nginx +After=nginx.service + +[Service] +Type=oneshot +ExecStart=/usr/local/bin/profullstack-devapps +EOF + + write_if_changed /etc/systemd/system/profullstack-devapps.timer <<'EOF' && units_changed=1 +[Unit] +Description=Pick up new per-user dev apps + +[Timer] +OnBootSec=1min +OnUnitActiveSec=1min +AccuracySec=15s + +[Install] +WantedBy=timers.target +EOF + + [[ "$units_changed" == 1 ]] && { systemctl daemon-reload; note "devapps timer units"; } + systemctl is-enabled profullstack-devapps.timer >/dev/null 2>&1 \ + || systemctl enable profullstack-devapps.timer + systemctl is-active profullstack-devapps.timer >/dev/null 2>&1 \ + || systemctl start profullstack-devapps.timer + + # the map file has to exist before nginx -t runs, or the vhost that + # references $devapp_port will not load + [[ -f "$DEV_APPS_MAP" ]] || printf '%s\n' \ + '# generated by profullstack-devapps -- do not edit' \ + 'map "$devapp_user/$devapp_name" $devapp_port {' \ + ' default "";' \ + '}' >"$DEV_APPS_MAP" + return 0 +} + +# The ad endpoint returns plain ASCII. Turning it into the HTML fragment nginx +# serves is three steps -- escape, linkify, wrap in 

 -- and doing them here
+# rather than per request means a page view never touches the ad network.
+#
+# Escaping comes first so a & in the ad text cannot become markup; linkifying
+# second so the URL it produces is a real link on a page in a browser. Both
+# leave the box drawing untouched:  renders zero-width inside 
, so the
+# +--+ borders still line up.
+write_sponsor_ad_generator() {
+	[[ "$SPONSOR_AD" == 1 ]] || return 0
+	local units_changed=0
+
+	install -d -m 0755 "$SPONSOR_AD_DIR"
+
+	write_if_changed /usr/local/bin/profullstack-sponsor-ad 0755 <'
+		printf '%s' '
'
+		printf '%s\n' "\$raw" \\
+			| sed -e 's/&/\&/g' -e 's//\>/g' \\
+			| sed -E 's|(https?://[A-Za-z0-9._~:/?@!\$()*+,;=%&-]+)|\1|g'
+		echo '
'
+		echo '
'
+	} >"\$tmp"
+
+	# Never publish a truncated fragment: it would show up on user pages at
+	# once. install(1) is the atomic step -- readers see old or new, not half.
+	[ -s "\$tmp" ] && install -m 0644 "\$tmp" "\$slot"
+	rm -f "\$tmp"
+done
+EOF
+
+	if command -v systemctl >/dev/null && [[ -d /etc/systemd/system ]]; then
+		write_if_changed /etc/systemd/system/profullstack-sponsor-ad.service <<'EOF' && units_changed=1
+[Unit]
+Description=Refill the sponsor ad pool for the per-user pages
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/profullstack-sponsor-ad
+# one slot can stall for 5s; this bounds a whole pass over the pool
+TimeoutStartSec=180
+EOF
+
+		write_if_changed /etc/systemd/system/profullstack-sponsor-ad.timer <<'EOF' && units_changed=1
+[Unit]
+Description=Refill the sponsor ad pool on the per-user pages
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=10min
+AccuracySec=1min
+
+[Install]
+WantedBy=timers.target
+EOF
+
+		# reloading systemd on every run is pointless churn
+		[[ "$units_changed" == 1 ]] && { systemctl daemon-reload; note "sponsor ad timer units"; }
+		systemctl is-enabled profullstack-sponsor-ad.timer >/dev/null 2>&1 \
+			|| systemctl enable profullstack-sponsor-ad.timer
+		systemctl is-active profullstack-sponsor-ad.timer >/dev/null 2>&1 \
+			|| systemctl start profullstack-sponsor-ad.timer
+	else
+		warn "no systemd -- rotating the sponsor ad from /etc/cron.hourly instead"
+		ln -sf /usr/local/bin/profullstack-sponsor-ad /etc/cron.hourly/profullstack-sponsor-ad
+	fi
+
+	# The pool directory must exist before the first page is served. Empty is
+	# survivable -- the vhost turns the resulting 403/404 into an empty 204 --
+	# but the directory itself missing would mean that guard is load-bearing on
+	# every request rather than kept for genuine accidents.
+	install -d -m 0755 "$SPONSOR_AD_POOL_DIR"
+
+	# the single-file fragment this pool replaced
+	rm -f "$SPONSOR_AD_LEGACY_FILE"
+
+	# fill it now rather than serving no ad until the timer first fires
+	/usr/local/bin/profullstack-sponsor-ad >/dev/null 2>&1 || true
+	return 0
+}
+
+# SPONSOR_AD=0 after a run that had it on: stop the timer and drop the
+# fragment. The vhosts stop referencing it in the same pass, so nothing is
+# left pointing at a file that will not be refreshed.
+remove_sponsor_ad() {
+	systemctl disable --now profullstack-sponsor-ad.timer >/dev/null 2>&1
+	rm -f /etc/cron.hourly/profullstack-sponsor-ad "$SPONSOR_AD_LEGACY_FILE"
+	rm -rf "$SPONSOR_AD_POOL_DIR"
+	return 0
+}
+
+# A wildcard cert can only come from a DNS-01 challenge -- HTTP-01 cannot
+# validate *.domain -- so it needs API access to whoever hosts the zone.
+# profullstack.com is on Porkbun; the Cloudflare path is kept for other zones.
+#
+# Wildcards are single-label on BOTH sides of this problem:
+#   DNS: *.dev.profullstack.com answers alice.dev.profullstack.com but NOT
+#        api.alice.dev.profullstack.com -- that needs its own
+#        *.alice.dev.profullstack.com record.
+#   TLS: the same, so every user needs *..dev.profullstack.com as a SAN.
+# Hence one cert carrying the base name plus a wildcard per user.
+
+have_dns01() {
+	[[ -n "$PORKBUN_API_KEY" && -n "$PORKBUN_SECRET_API_KEY" ]] && return 0
+	[[ -n "$CLOUDFLARE_API_TOKEN" || -s "$CF_CREDENTIALS" ]] && return 0
+	return 1
+}
+
+# users with a home we actually serve something from
+web_users() {
+	local d u
+	for d in /home/*; do
+		[[ -d "$d" ]] || continue
+		u="$(basename "$d")"
+		id -u "$u" >/dev/null 2>&1 || continue
+		[[ -d "$d/public_html" || -d "$d/apps" ]] || continue
+		printf '%s\n' "$u"
+	done
+}
+
+# Every name the cert has to carry, one per line, $WEB_DOMAIN ALWAYS FIRST --
+# acme.sh treats the first -d as the "main domain" and files the certificate
+# under that name, which is the name --install-cert and the renewal check both
+# look for. Sorting this list would silently break both.
+#
+# $1: "dns01" for the wildcard form, "http01" for the spelled-out form.
+# http-01 cannot validate a wildcard at all, so the fallback must never ask
+# for one -- certbot rejects the whole request if it does.
+desired_cert_names() {
+	local mode="${1:-auto}" u app
+	[[ "$mode" == auto ]] && { have_dns01 && mode=dns01 || mode=http01; }
+
+	printf '%s\n' "$WEB_DOMAIN"
+	if [[ "$mode" == dns01 ]]; then
+		printf '*.%s\n' "$WEB_DOMAIN"
+		# one wildcard per user covers all of their current and future apps
+		[[ "$DEV_APPS" == 1 ]] && while read -r u; do
+			printf '*.%s.%s\n' "$u" "$WEB_DOMAIN"
+		done < <(web_users)
+	else
+		# every name spelled out, and each one must already resolve here
+		while read -r u; do
+			printf '%s.%s\n' "$u" "$WEB_DOMAIN"
+			[[ "$DEV_APPS" == 1 ]] || continue
+			for app in "/home/$u"/apps/*; do
+				[[ -d "$app" ]] || continue
+				app="$(basename "$app")"
+				[[ "$app" =~ ^[a-z0-9][a-z0-9-]*$ ]] || continue
+				printf '%s.%s.%s\n' "$app" "$u" "$WEB_DOMAIN"
+			done
+		done < <(web_users)
+	fi
+}
+
+# the -d arguments for an ACME client, order preserved, duplicates dropped
+cert_args() {
+	local n seen=() out=()
+	while read -r n; do
+		[[ -n "$n" ]] || continue
+		printf '%s\n' "${seen[@]+"${seen[@]}"}" | grep -qxF "$n" && continue
+		seen+=("$n"); out+=(-d "$n")
+	done < <(desired_cert_names "${1:-auto}")
+	printf '%s\n' "${out[@]+"${out[@]}"}"
+}
+
+cert_names() {
+	[[ -s "$CERT_DIR/fullchain.pem" ]] || return 1
+	openssl x509 -in "$CERT_DIR/fullchain.pem" -noout -text 2>/dev/null \
+		| grep -A1 'Subject Alternative Name' | tr ',' '\n' \
+		| sed -n 's/.*DNS://p' | tr -d ' '
+}
+
+cert_days_left() {
+	local end now
+	[[ -s "$CERT_DIR/fullchain.pem" ]] || { echo -1; return; }
+	end="$(openssl x509 -in "$CERT_DIR/fullchain.pem" -noout -enddate 2>/dev/null | cut -d= -f2)"
+	end="$(date -d "$end" +%s 2>/dev/null)" || { echo -1; return; }
+	now="$(date +%s)"
+	echo $(( (end - now) / 86400 ))
+}
+
+# missing names, one per line (empty = the cert already covers everything)
+cert_missing_names() {
+	local have
+	have="$(cert_names 2>/dev/null)" || { desired_cert_names; return; }
+	comm -23 <(desired_cert_names | sort -u) <(printf '%s\n' "$have" | sort -u)
+}
+
+# ---- Porkbun DNS ----------------------------------------------------------
+#
+# A per-user wildcard cert is useless without a matching DNS record, and
+# nobody wants to add one by hand each time an account is created.
+PORKBUN_DOMAIN="${PORKBUN_DOMAIN:-$(printf '%s' "$WEB_DOMAIN" | awk -F. '{print $(NF-1)"."$NF}')}"
+PORKBUN_API=https://api.porkbun.com/api/json/v3
+
+_porkbun_call() {
+	local path="$1" body="$2"
+	# no -f: a 4xx body carries the API's own error message, which is worth
+	# more than curl's exit code
+	curl -sS --max-time 30 -H 'Content-Type: application/json' \
+		-d "$body" "$PORKBUN_API/$path" 2>/dev/null
+}
+
+_spf_txt() {
+	dig +short TXT "$1" 2>/dev/null | tr -d '"' | grep -m1 '^v=spf1' || true
+}
+
+# Recursively count the DNS-querying mechanisms in an SPF record.
+#
+# RFC 7208 4.6.4 caps this at 10, and going over is a permerror -- the exact
+# failure mode as having two records. So a merge can happily fix one bug and
+# introduce the other: two includes that each pull in the same four nested
+# includes cost eight lookups between them, not four.
+spf_lookup_count() {
+	local rec="$1" depth="${2:-0}" n=0 t sub add
+	[[ "$depth" -gt 8 ]] && { printf 0; return; }
+	for t in $rec; do
+		case "${t,,}" in
+			include:*|redirect=*)
+				n=$((n+1))
+				sub="$(_spf_txt "${t#*[:=]}")"
+				if [[ -n "$sub" ]]; then
+					add="$(spf_lookup_count "$sub" $((depth+1)))"
+					n=$((n+add))
+				fi
+				;;
+			a|a:*|mx|mx:*|ptr|ptr:*|exists:*) n=$((n+1)) ;;
+		esac
+	done
+	printf '%s' "$n"
+}
+
+# Consolidate the apex SPF record.
+#
+# RFC 7208 allows exactly one "v=spf1" TXT record per name. A second one is a
+# permerror, not a merge -- receivers stop evaluating and the domain fails SPF
+# outright. Combined with "DMARC p=reject" that is a live deliverability bug,
+# and it is easy to end up with because every service that wants SPF tells you
+# to "add a TXT record".
+#
+# Detection always runs and warns. Rewriting only happens with FIX_SPF=1,
+# because silently editing mail DNS during a provisioning run is not something
+# this script should do on its own.
+#
+# The merge is a union of every term except the trailing "all", in first-seen
+# order, so nothing that was authorised before stops being authorised. Two
+# includes that happen to cover the same hosts are left alone -- deciding they
+# are redundant needs a human who knows what still sends mail.
+porkbun_fix_spf() {
+	local auth records spf_json count merged first_id terms all_q
+	[[ -n "$PORKBUN_API_KEY" && -n "$PORKBUN_SECRET_API_KEY" ]] || return 1
+	auth="$(printf '{"apikey":"%s","secretapikey":"%s"}' "$PORKBUN_API_KEY" "$PORKBUN_SECRET_API_KEY")"
+
+	records="$(_porkbun_call "dns/retrieve/$PORKBUN_DOMAIN" "$auth")"
+	[[ "$(printf '%s' "$records" | jq -r '.status? // "ERROR"')" == "SUCCESS" ]] || {
+		warn "could not read DNS for $PORKBUN_DOMAIN -- leaving SPF alone"
+		return 1
+	}
+
+	# apex TXT records only, and only the SPF ones: the same name also carries
+	# site-verification strings that must not be touched.
+	spf_json="$(printf '%s' "$records" | jq -c \
+		--arg d "$PORKBUN_DOMAIN" \
+		'[.records[] | select(.type=="TXT" and .name==$d)
+		  | {id, content: (.content | gsub("^\"|\"$";""))}
+		  | select(.content | test("^v=spf1\\b"))]')"
+	count="$(printf '%s' "$spf_json" | jq 'length')"
+
+	if [[ "$count" -le 1 ]]; then
+		[[ "$count" == 1 ]] && note "SPF on $PORKBUN_DOMAIN: 1 record (correct)"
+		return 0
+	fi
+
+	warn "$PORKBUN_DOMAIN has $count SPF records -- RFC 7208 permits one; this is a permerror"
+	printf '%s' "$spf_json" | jq -r '.[] | "    " + .content' >&2
+
+	# strictest qualifier wins: if any record hard-failed before, keep doing so
+	all_q="$(printf '%s' "$spf_json" | jq -r \
+		'if any(.[].content; test("(^| )-all( |$)")) then "-all" else "~all" end')"
+	terms="$(printf '%s' "$spf_json" | jq -r \
+		'[.[].content | split(" ")[]] | map(select(. != "v=spf1" and (test("all$") | not)))
+		 | unique_by(ascii_downcase) | join(" ")')"
+	merged="v=spf1 $terms $all_q"
+
+	local lookups
+	lookups="$(spf_lookup_count "$terms")"
+	if [[ "${lookups:-0}" -gt 10 ]]; then
+		warn "the union of those records needs $lookups DNS lookups (RFC 7208 caps it at 10)"
+		warn "applying it would swap one permerror for another -- overlapping includes"
+		warn "have to be reduced by hand, by someone who knows what still sends mail:"
+		warn "  $merged"
+		return 1
+	fi
+
+	note "merged SPF ($lookups/10 lookups): $merged"
+	if [[ "${FIX_SPF:-0}" != 1 ]]; then
+		note "re-run with FIX_SPF=1 to apply it (this edits mail DNS)"
+		return 0
+	fi
+
+	first_id="$(printf '%s' "$spf_json" | jq -r '.[0].id')"
+	if ! _porkbun_call "dns/edit/$PORKBUN_DOMAIN/$first_id" \
+		"${auth%\}},\"name\":\"\",\"type\":\"TXT\",\"content\":\"$merged\",\"ttl\":\"600\"}" \
+		| grep -q '"status":"SUCCESS"'; then
+		warn "could not rewrite the SPF record -- nothing deleted, zone unchanged"
+		return 1
+	fi
+	note "SPF rewritten: $merged"
+
+	# only now that one good record exists is it safe to drop the others
+	local id
+	for id in $(printf '%s' "$spf_json" | jq -r '.[1:][].id'); do
+		if _porkbun_call "dns/delete/$PORKBUN_DOMAIN/$id" "$auth" | grep -q '"status":"SUCCESS"'; then
+			note "removed duplicate SPF record $id"
+		else
+			warn "could not remove duplicate SPF record $id -- still a permerror, fix by hand"
+		fi
+	done
+}
+
+# Create . -> $2 if it is not already there.
+#
+# Porkbun happily stores two A records with the same name, so "I could not
+# read the zone" must never be treated as "the record is missing" -- one
+# timeout would otherwise add a duplicate on every run.
+porkbun_ensure_record() {
+	local name="$1" ip="$2" auth existing path
+	[[ -n "$PORKBUN_API_KEY" && -n "$PORKBUN_SECRET_API_KEY" && -n "$ip" ]] || return 1
+	auth="$(printf '{"apikey":"%s","secretapikey":"%s"' "$PORKBUN_API_KEY" "$PORKBUN_SECRET_API_KEY")"
+
+	# the apex record has an empty name; that endpoint wants the name omitted
+	# entirely rather than a trailing empty path segment
+	path="dns/retrieveByNameType/$PORKBUN_DOMAIN/A"
+	[[ -n "$name" ]] && path="$path/$name"
+
+	existing="$(_porkbun_call "$path" "$auth}")"
+	if ! printf '%s' "$existing" | grep -q '"status":"SUCCESS"'; then
+		warn "could not read DNS for ${name:+$name.}$PORKBUN_DOMAIN -- not creating anything"
+		return 1
+	fi
+
+	if printf '%s' "$existing" | grep -q '"content"'; then
+		# -F: an IP is full of dots, which are wildcards to a basic regex
+		if printf '%s' "$existing" | grep -qF "\"content\":\"$ip\""; then
+			return 0            # already correct
+		fi
+		warn "DNS ${name:+$name.}$PORKBUN_DOMAIN points somewhere else -- not touching it"
+		return 1
+	fi
+
+	if _porkbun_call "dns/create/$PORKBUN_DOMAIN" \
+		"$auth,\"type\":\"A\",\"name\":\"$name\",\"content\":\"$ip\",\"ttl\":\"600\"}" \
+		| grep -q '"status":"SUCCESS"'; then
+		note "DNS A ${name:+$name.}$PORKBUN_DOMAIN -> $ip"
+		return 0
+	fi
+	warn "could not create DNS record ${name:+$name.}$PORKBUN_DOMAIN (API access enabled on the domain?)"
+	return 1
+}
+
+# Is this an address the public internet can actually reach? Publishing a
+# NAT address into DNS breaks the zone, and the "points somewhere else" guard
+# above would then refuse to correct it.
+is_public_ip() {
+	local ip="$1"
+	[[ "$ip" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]] || return 1
+	case "$ip" in
+		10.*|127.*|169.254.*|192.168.*|0.*|255.*) return 1 ;;
+		172.1[6-9].*|172.2[0-9].*|172.3[01].*)    return 1 ;;
+		100.6[4-9].*|100.[7-9][0-9].*|100.1[01][0-9].*|100.12[0-7].*) return 1 ;;  # CGNAT/tailscale
+	esac
+	return 0
+}
+
+# the records this box needs: the base name, the user wildcard, and one
+# wildcard per user for their dev apps
+ensure_dns_records() {
+	local ip="$1" sub u rc=0
+	[[ -n "$PORKBUN_API_KEY" && -n "$PORKBUN_SECRET_API_KEY" ]] || return 1
+	[[ -n "$ip" ]] || { warn "no public IP found -- skipping DNS"; return 1; }
+	is_public_ip "$ip" || {
+		warn "$ip is not a public address -- refusing to publish it as DNS"
+		return 1
+	}
+
+	# dev.profullstack.com -> "dev" relative to the zone
+	sub="${WEB_DOMAIN%".$PORKBUN_DOMAIN"}"
+	[[ "$sub" == "$WEB_DOMAIN" ]] && sub=""
+
+	porkbun_ensure_record "$sub" "$ip" || rc=1
+	porkbun_ensure_record "*${sub:+.$sub}" "$ip" || rc=1
+	if [[ "$DEV_APPS" == 1 ]]; then
+		while read -r u; do
+			porkbun_ensure_record "*.$u${sub:+.$sub}" "$ip" || rc=1
+		done < <(web_users)
+	fi
+
+	# Not about this host's records, but this is the one place we already hold
+	# Porkbun credentials -- and a second SPF record breaks mail for the whole
+	# zone silently. Detect-only unless FIX_SPF=1.
+	porkbun_fix_spf || rc=1
+	return $rc
+}
+
+# How each ACME client is told about the contact address, given there may not
+# be one.
+#
+# ACME_EMAIL has no default on purpose -- a public script must not ship
+# somebody's address, and a made-up one is worse than none because Let's
+# Encrypt would send that person's expiry warnings into a black hole. Both
+# clients issue perfectly well without it; the only thing lost is the reminder
+# mail, and the box says so once rather than every run.
+#
+# certbot is the awkward one: it refuses to run non-interactively with neither
+# -m nor --register-unsafely-without-email, so the flag has to be chosen rather
+# than the argument left empty.
+_certbot_email_args() {
+	if [[ -n "$ACME_EMAIL" ]]; then
+		printf '%s\n%s\n' -m "$ACME_EMAIL"
+	else
+		printf '%s\n' --register-unsafely-without-email
+	fi
+}
+
+_warn_no_acme_email() {
+	[[ -n "$ACME_EMAIL" ]] && return 0
+	info "no ACME_EMAIL set -- the cert will issue, but Let's Encrypt cannot"
+	info "      send expiry warnings. Set ACME_EMAIL in $SERVER_CONFIG to get them."
+	return 0
+}
+
+# acme.sh has built-in Porkbun support; certbot has no official plugin for it.
+_issue_cert_porkbun() {
+	local args=() n
+	while read -r n; do args+=("$n"); done < <(cert_args dns01)
+	[[ ${#args[@]} -gt 0 ]] || return 1
+
+	_warn_no_acme_email
+	if [[ ! -x "$ACME_HOME/acme.sh" ]]; then
+		# The installer takes email= as an optional argument; passing an empty
+		# one registers the account to the literal empty string, so it is left
+		# off entirely rather than passed blank.
+		if [[ -n "$ACME_EMAIL" ]]; then
+			curl -fsSL https://get.acme.sh | env HOME=/root sh -s "email=$ACME_EMAIL" \
+				|| { warn "acme.sh install failed"; return 1; }
+		else
+			curl -fsSL https://get.acme.sh | env HOME=/root sh \
+				|| { warn "acme.sh install failed"; return 1; }
+		fi
+	fi
+	install -d -m 0755 "$CERT_DIR"
+
+	# --server letsencrypt: acme.sh defaults to ZeroSSL, which wants a
+	# registered account. --force so a changed name list is honoured rather
+	# than skipped as "cert not yet due for renewal".
+	env HOME=/root \
+		PORKBUN_API_KEY="$PORKBUN_API_KEY" \
+		PORKBUN_SECRET_API_KEY="$PORKBUN_SECRET_API_KEY" \
+		"$ACME_HOME/acme.sh" --issue --server letsencrypt --dns dns_porkbun \
+			--force "${args[@]}" \
+		|| { warn "acme.sh could not issue the cert (API access enabled on the domain?)"; return 1; }
+
+	# land it where nginx already looks, and reload on every renewal
+	env HOME=/root "$ACME_HOME/acme.sh" --install-cert -d "$WEB_DOMAIN" \
+		--fullchain-file "$CERT_DIR/fullchain.pem" \
+		--key-file "$CERT_DIR/privkey.pem" \
+		--reloadcmd "systemctl reload nginx"
+}
+
+_issue_cert_cloudflare() {
+	local args=() n
+	while read -r n; do args+=("$n"); done < <(cert_args dns01)
+	[[ ${#args[@]} -gt 0 ]] || return 1
+
+	if [[ -n "$CLOUDFLARE_API_TOKEN" && ! -s "$CF_CREDENTIALS" ]]; then
+		install -d -m 0700 "$(dirname "$CF_CREDENTIALS")"
+		printf 'dns_cloudflare_api_token = %s\n' "$CLOUDFLARE_API_TOKEN" >"$CF_CREDENTIALS"
+	fi
+	[[ -s "$CF_CREDENTIALS" ]] || return 1
+	chmod 0600 "$CF_CREDENTIALS"
+	_warn_no_acme_email
+	local mail=() m
+	while read -r m; do mail+=("$m"); done < <(_certbot_email_args)
+	certbot certonly --non-interactive --agree-tos --expand \
+		--cert-name "$WEB_DOMAIN" \
+		--dns-cloudflare --dns-cloudflare-credentials "$CF_CREDENTIALS" \
+		--dns-cloudflare-propagation-seconds 30 \
+		"${mail[@]}" "${args[@]}"
+}
+
+# No DNS credentials needed: http-01 validates each name individually by
+# serving a file over port 80. It cannot do *.domain -- hence one -d per
+# existing name -- and it does require this host to be reachable from the
+# internet on port 80, which DNS-01 does not.
+_issue_cert_http01() {
+	local args=() n
+	# always http01: this runs as the fallback with the DNS credentials still
+	# set, and asking certbot --webroot for a wildcard fails the whole request
+	while read -r n; do args+=("$n"); done < <(cert_args http01)
+	[[ ${#args[@]} -gt 0 ]] || return 1
+
+	install -d -m 0755 "$ACME_WEBROOT"
+	info "http-01 for: $(desired_cert_names http01 | tr '\n' ' ')"
+	_warn_no_acme_email
+	local mail=() m
+	while read -r m; do mail+=("$m"); done < <(_certbot_email_args)
+	# --cert-name keeps the path stable so nginx needs no reconfiguration;
+	# --expand lets a later run add newly created users to the same cert.
+	certbot certonly --non-interactive --agree-tos --expand \
+		--cert-name "$WEB_DOMAIN" \
+		--webroot -w "$ACME_WEBROOT" \
+		"${mail[@]}" "${args[@]}"
+}
+
+# Renewal has to be somebody's job or the box quietly goes dark in 90 days.
+# acme.sh installs its own cron on first use; certbot ships a systemd timer.
+ensure_cert_renewal() {
+	if [[ -x "$ACME_HOME/acme.sh" ]] && [[ -d "$ACME_HOME/$WEB_DOMAIN" || -d "$ACME_HOME/${WEB_DOMAIN}_ecc" ]]; then
+		crontab -l 2>/dev/null | grep -q 'acme.sh --cron' \
+			|| env HOME=/root "$ACME_HOME/acme.sh" --install-cronjob >/dev/null 2>&1
+		info "renewal: acme.sh cron"
+		return 0
+	fi
+	if systemctl list-unit-files certbot.timer >/dev/null 2>&1; then
+		systemctl is-enabled certbot.timer >/dev/null 2>&1 \
+			|| systemctl enable --now certbot.timer >/dev/null 2>&1
+		# make sure a renewal actually reaches nginx
+		install -d -m 0755 /etc/letsencrypt/renewal-hooks/deploy
+		write_if_changed /etc/letsencrypt/renewal-hooks/deploy/reload-nginx 0755 <<'EOF' \
+			&& note "certbot deploy hook"
+#!/bin/sh
+# managed by cli-tools/root-ubuntu.sh
+systemctl reload nginx
+EOF
+		info "renewal: certbot.timer"
+		return 0
+	fi
+	warn "no renewal mechanism found -- the certificate will expire"
+	return 1
+}
+
+issue_wildcard_cert() {
+	local missing days
+	days="$(cert_days_left)"
+	missing="$(cert_missing_names)"
+
+	if [[ -s "$CERT_DIR/fullchain.pem" && -z "$missing" && "$days" -gt "$CERT_RENEW_DAYS" ]]; then
+		info "cert for $WEB_DOMAIN covers everything, $days days left"
+		ensure_cert_renewal
+		return 0
+	fi
+	[[ -n "$missing" ]] && info "cert is missing: $(printf '%s' "$missing" | tr '\n' ' ')"
+	[[ -s "$CERT_DIR/fullchain.pem" && "$days" -le "$CERT_RENEW_DAYS" ]] \
+		&& info "cert expires in $days days -- renewing"
+
+	if [[ -n "$PORKBUN_API_KEY" && -n "$PORKBUN_SECRET_API_KEY" ]]; then
+		if _issue_cert_porkbun; then note "certificate"; ensure_cert_renewal; return 0; fi
+		warn "porkbun dns-01 failed -- trying http-01 for the named hosts"
+	elif [[ -n "$CLOUDFLARE_API_TOKEN" || -s "$CF_CREDENTIALS" ]]; then
+		if _issue_cert_cloudflare; then note "certificate"; ensure_cert_renewal; return 0; fi
+		warn "cloudflare dns-01 failed -- trying http-01 for the named hosts"
+	else
+		info "no DNS API credentials -- a wildcard needs dns-01, so falling back"
+		info "to http-01 for $WEB_DOMAIN and each existing name."
+		info "Per-user dev apps (*..$WEB_DOMAIN) NEED dns-01:"
+		info "  sudo PORKBUN_API_KEY=pk1_... PORKBUN_SECRET_API_KEY=sk1_... $0"
+	fi
+
+	_issue_cert_http01 && { note "certificate"; ensure_cert_renewal; return 0; }
+	return 1
+}
+
+LANDING=/var/www/userdirs/index.html
+LANDING_MARKER="generated by cli-tools/root-ubuntu.sh"
+
+# The company blog is one person's userdir rather than its own vhost, so the
+# landing page has to be told whose to link. No default, because whose it is
+# differs per box and guessing a login would produce a link to a 404; with
+# BLOG_USER unset the landing page simply has no blog section.
+BLOG_USER="${BLOG_USER:-}"
+BLOG_TITLE="${BLOG_TITLE:-the team blog}"
+BLOG_DIR="${BLOG_DIR:-${BLOG_USER:+/home/$BLOG_USER/public_html/blog}}"
+
+# Staff landing page for https://$WEB_DOMAIN -- people, company links, stack.
+# Regenerated each run so new users appear, but only if the file is still the
+# generated one; a hand-edited page is left alone.
+# Cached locally so the landing page still has a logo when profullstack.com is
+# unreachable, and so it does not make an off-box request on every view.
+fetch_logo() {
+	local tmp
+	install -d -m 0755 "$(dirname "$LOGO_FILE")"
+	# configure_nginx runs twice per invocation (before and after the cert);
+	# a logo that is already there and less than a week old is good enough.
+	[[ -s "$LOGO_FILE" ]] && [[ -z "$(find "$LOGO_FILE" -mtime +7 2>/dev/null)" ]] && return 0
+	tmp="$(mktemp)" || return 1
+	if curl -fsSL --max-time 10 "$LOGO_URL" -o "$tmp" && [[ -s "$tmp" ]]; then
+		cmp -s "$tmp" "$LOGO_FILE" || { install -m 0644 "$tmp" "$LOGO_FILE"; note "logo"; }
+		rm -f "$tmp"
+		return 0
+	fi
+	rm -f "$tmp"
+	return 1
+}
+
+write_landing_page() {
+	install -d -m 0755 /var/www/userdirs
+
+	# Replaceable if we generated it. The extra patterns are every marker this
+	# script has ever stamped a page with -- the placeholder from before the
+	# marker existed, and the one from when this lived in the dotfiles repo.
+	# Renaming the marker without keeping the old one means every already
+	# provisioned box decides its own landing page was hand-edited and refuses
+	# to touch it again, which is a one-way door: nothing later can tell the
+	# difference between that page and a real one somebody wrote.
+	if [[ -e "$LANDING" ]] \
+		&& ! grep -q "$LANDING_MARKER" "$LANDING" \
+		&& ! grep -q 'generated by dottemplates/root-ubuntu.sh' "$LANDING" \
+		&& ! grep -q 'User pages are at' "$LANDING"; then
+		info "$LANDING was edited by hand -- leaving it alone"
+		return 0
+	fi
+
+	local proto=http u _mail_host _logo _blog
+	[[ -s "$CERT_DIR/fullchain.pem" ]] && proto=https
+	_mail_host="${MAIL_URL#*://}"; _mail_host="${_mail_host%/}"
+
+	# prefer the local copy; fall back to the canonical URL if the fetch failed
+	if [[ -s "$LOGO_FILE" ]]; then
+		_logo="/assets/logo.svg"
+	else
+		_logo="$LOGO_URL"
+	fi
+
+	# The write-ups, linked from the moshcode section. Emitted as a variable
+	# rather than inlined in the heredoc so it can be skipped: on a box with no
+	# such blog it would otherwise be a link to a 404, which is worse than no
+	# link at all. Needs BOTH a configured BLOG_USER and the directory actually
+	# being there -- a name alone is a promise the box may not keep.
+	_blog=""
+	if [[ -n "$BLOG_USER" && -n "$BLOG_DIR" && -d "$BLOG_DIR" ]]; then
+		_blog="  
The long version is written up on
+     $BLOG_TITLE.
"
+	fi
+
+	{
+		cat <
+
+
+
+$WEB_DOMAIN
+
+

+  
+  
$WEB_DOMAIN

+  
Profullstack dev box — staff links and user pages.

+
+  
Company

+  
+
+  
Comms — AgentBBS

+  
Our comms network runs over SSH. First time, connect as
+     join — your SSH key becomes your account and you get a
+     username, a Linux pod and a homepage. After that, sign in as your BBS name.

+  
ssh join@$BBS_DOMAIN          # first time: registers your key
+ssh <your-bbs-name>@$BBS_DOMAIN   # sign in -- arcade, chat, news, mail, pod
+ssh bbs@$BBS_DOMAIN           # look around as a guest

+  
Once signed in the hub reaches everything without separate logins.
+     Handy direct entrances:

+  
ssh mail@$BBS_DOMAIN          # your BBS mailbox
+ssh irc@$BBS_DOMAIN           # members' IRC from your terminal
+ssh -t news@$BBS_DOMAIN       # Usenet-style newsreader
+ssh pod@$BBS_DOMAIN           # your Linux pod

+  
+
+  
People

+  
    
+EOF
+
+		# one card per user that actually has a public_html
+		for u in /home/*/public_html; do
+			[[ -d "$u" ]] || continue
+			u="$(basename "$(dirname "$u")")"
+			printf '    
  • %s%s.%s%s@%sssh %s@%s
    • \n' \
+				"$proto" "$u" "$WEB_DOMAIN" "$u" "$u" "$WEB_DOMAIN" \
+				"$u" "$MAIL_DOMAIN" "$u" "$BBS_DOMAIN"
+		done
+
+		cat <
+
+  
    moshcode
    
+  
    All dev work on this box goes through moshcode. It is
+     already installed for every account — it is the wrapper that installs and
+     drives the coding agents, so you do not set them up yourself. Run
+     moshcode with no arguments for the TUI, which we call the pit. That
+     is where the day starts, and its slash commands are also how you reach people
+     and how you bill — see below.
    
+  
    moshcode                      # the pit, then /agents <engine>
+moshcode engines              # what is installed: claude, codex, gemini, opencode, aider
+moshcode install claude       # add an engine
+moshcode claude               # launch one directly
+moshcode agents claude        # autonomous mode -- auto-approves, use in trusted dirs only
    
+  
    On your own machine — source at
+     github.com/moshcoder/moshcode.
+     Never run it with sudo: it re-runs its own root steps where it
+     needs them, and sudo moshcode installs into /root
+     instead of your account.
    
+  
    curl -fsSL https://moshcoding.com/install.sh | sh
+moshcode upgrade              # once installed, this is how it updates itself
    
+  
    It also fronts the workflow CLIs. Each keeps its own auth, so
+     signing in to one signs you in to nothing else:
    
+  
    moshcode tools                # gh, supabase, railway, doppler, doctl, ugig, coinpay, …
+moshcode gh pr list           # passthrough -- same as running gh yourself
+moshcode upgrade              # update moshcode and everything it installed
    
+
+  
    Run it under tmux, which is installed here and works
+     well with the pit — a dropped connection then costs you nothing, and you
+     can leave an agent running while you go and do something else. mosh
+     above survives the same drop; tmux is what survives you closing the laptop.
    
+  
    tmux new -s work              # start a named session
+tmux attach -t work           # come back to it, from anywhere
+# ctrl-b d detaches and leaves everything running
    
+$_blog
+
+  
    Comms — slash commands
    
+  
    Employees and contractors talk to us through the pit.
+     Not email, not a private DM somewhere we cannot see it — open
+     moshcode and use the slash commands. It keeps the conversation next
+     to the work and reachable by everyone who needs it, and it means you do not need
+     another account or another app to reach your lead.
    
+  
    /chat                         # the room -- ambient, where the team is
+/message <who> <text>         # reach one person
+/msg <who> <text>             # same thing, shorter
    
+  
    Type /help in the pit for the current list — it is
+     the authority, and it grows. The SSH side of the network (rooms, IRC, Usenet,
+     BBS mail) is AgentBBS, described above; the
+     slash commands are the shortest path from where you are already working.
    
+
+  
    Billing
    
+  
    Contractors bill through the pit too, and it lands in
+     CoinPay — so an invoice, its payment and its history are one
+     record rather than a thread and a spreadsheet. Raise it against the work you did;
+     do not send an invoice by email.
    
+  
    /invoice                      # raise an invoice
+/payment                      # pay one, or check a payment
+/billing                      # your account -- invoices, status, history
    
+  
    These front CoinPay, which is
+     also wrapped as a tool if you would rather drive it directly:
    
+  
    moshcode tools coinpay        # or /tools coinpay from inside the pit
    
+
+  
    Two separate logins, and this trips everyone up.
+     Both need --device here: there is no browser on this box, so the
+     loopback flow has nothing to open. You approve the code from your laptop.
    
+  
    moshcode login --device       # app.moshcode.sh -- lets notify()/ask() reach you
+moshcode whoami
+
+moshcode secrets login --device   # LogicSRC -- team vaults. NOT the same account.
+moshcode secrets whoami
    
+  
    Team secrets, once secrets login has been done —
+     <team> and <vault> are your own names, run
+     teams list and teams vaults to see them:
    
+  
    moshcode secrets teams list
+moshcode secrets teams vaults <team>
+moshcode secrets teams pull <team> <vault>   # decrypt into a local .env
+moshcode secrets teams push <team> <vault>   # encrypt a local .env back up
    
+
+  
    Your dev apps
    
+  
    Anything in ~/apps/<name> is published at
+     https://<name>.<you>.$WEB_DOMAIN with TLS. New apps go
+     live within a minute — no root, no restart, no ticket.
    
+  
    # a static site
+mkdir -p ~/apps/blog/public && echo hi > ~/apps/blog/public/index.html
+
+# a running app: declare the port, then listen on it
+mkdir -p ~/apps/api && echo 3000 > ~/apps/api/.port
+node server.js            # bind 127.0.0.1:3000, not 0.0.0.0
    
+  
    Websockets and HMR are proxied through, so vite/next dev servers
+     work as-is. Bind to 127.0.0.1 — the firewall blocks the port
+     directly and nginx is what terminates TLS for you.
    
+
+  
    Stack
    
+  
+
+  
    Access
    
+  
    Straight away, no setup:
    
+  
    ssh <your-username>@$WEB_DOMAIN
+mosh <your-username>@$WEB_DOMAIN   # survives a dropped connection
    
+  
    Or add this to ~/.ssh/config on your laptop and it becomes
+     just ssh dev:
    
+  
    Host dev
+    HostName $WEB_DOMAIN
+    User <your-username>
+    Port ${SSH_PORT}
+    IdentityFile ~/.ssh/id_ed25519
+    ServerAliveInterval 60
    
+  
    Send your public key (~/.ssh/id_ed25519.pub) to get added.
    
+
+  
    Tailscale on your phone or laptop
    
+  
    Optional, and it changes nothing today — the ssh above
+     already works from anywhere. It only starts to matter if public ssh is ever
+     closed. You cannot add your laptop from a shell on this box.
+     Enrolling a device is something that device does, so it happens on the laptop
+     or the phone; nothing you type here reaches it. Do not reach for
+     sudo either — tailscale up run here
+     re-authenticates this box, not your laptop, and takes it out from
+     under everyone else logged in. Ask for a share link instead:
    
+  
    # on your laptop or phone -- not here
+# 1. install tailscale, sign in, make your own (free) tailnet.
+#    that is where your devices get added: to yours, not ours.
+# 2. open the share link you were sent and accept it. this box
+#    then shows up in your tailnet, and nothing else of ours does.
+ssh <your-username>@dev.<tailnet>.ts.net   # exact name is in the share link
    
+
+  
    
+    Your page lives in ~/public_html/index.html.
+    Reachable at $proto://<user>.$WEB_DOMAIN or $proto://$WEB_DOMAIN/~<user>.
+  
    
+

+
+EOF
+	} | write_if_changed "$LANDING" 0644 && note "landing page"
+
+	return 0
+}
+
+# Two separate vhosts, deliberately not one shared body: with a single server
+# block matching both names, a request to user.$WEB_DOMAIN/ finds the bare
+# domain's own index.html and serves the landing page instead of the user's.
+# One http-context map defining $bad_bot, consumed by every vhost below.
+# Matching is case-insensitive (~*) and on substrings, because these crawlers
+# append versions and URLs to their User-Agent.
+write_bad_bots_map() {
+	{
+		echo "# managed by cli-tools/root-ubuntu.sh -- AI crawler / scraper blocklist"
+		echo "map \$http_user_agent \$bad_bot {"
+		echo "	default 0;"
+		if [[ "$BLOCK_AI_BOTS" == 1 ]]; then
+			local ua
+			for ua in "${AI_CRAWLER_AGENTS[@]}"; do
+				echo "	\"~*$ua\" 1;"
+			done
+		fi
+		echo "}"
+	} | write_if_changed "$BAD_BOTS_MAP"
+}
+
+# Emitted into both vhosts that serve ~/public_html, so /~user/x/ and
+# user.$WEB_DOMAIN/x/ cannot drift apart about what they show.
+_nginx_sponsor_ad_locations() {
+	[[ "$SPONSOR_AD" == 1 ]] || return 0
+	cat <, and an exact-match location would not catch
+	# that second URI -- it would fall through to the vhost root and 404.
+	location ^~ ${SPONSOR_AD_URI} {
+		internal;
+		alias ${SPONSOR_AD_POOL_DIR}/;
+		random_index on;
+		default_type text/html;
+		# An empty pool otherwise falls through to the inherited autoindex and
+		# pastes a listing of this directory into every page.
+		autoindex off;
+		# Nothing to serve must render as nothing: 404 for a missing pool, 403
+		# for an empty one. Without this the error body is what gets pasted
+		# into the page that included it.
+		error_page 404 403 = ${SPONSOR_AD_BLANK_URI};
+	}
+	location = ${SPONSOR_AD_BLANK_URI} { internal; return 204; }
+EOF
+}
+
+# The listing half of the ad. autoindex generates the page, so there is no file
+# to hold an SSI token and the fragment is prepended to the response body.
+#
+# Matching on a trailing "/" is what keeps this to listings: a directory that
+# HAS an index.html is internally redirected to .../index.html, which no longer
+# ends in "/" and so misses this block. That page gets the ad from its own SSI
+# token instead -- which is also why replacing index.html drops the ad, rather
+# than having one forced on top of whatever the user wrote.
+#
+# Takes the body of the matching location, so the /~user form can repeat its
+# alias -- a regex location cannot inherit one from a sibling.
+_nginx_sponsor_ad_listing() {
+	[[ "$SPONSOR_AD" == 1 ]] || return 0
+	local pattern="$1" body="${2:-}"
+	printf '\n\n\tlocation ~ "%s" {\n' "$pattern"
+	[[ -n "$body" ]] && printf '%s\n' "$body"
+	printf '\t\tadd_before_body %s;\n\t}\n' "$SPONSOR_AD_URI"
+}
+
+# SSI is on only where ~/public_html is served, and only to expand that token.
+_nginx_sponsor_ad_ssi() {
+	[[ "$SPONSOR_AD" == 1 ]] || return 0
+	printf '\n\t# expands the sponsor-ad token in ~/public_html/index.html\n\tssi on;\n'
+}
+
+# Version-control metadata is not web content. `git init` inside a doc root
+# publishes /.git/config -- remotes, full history, sometimes credentials -- and
+# scanners sweep for exactly that URL. 404 rather than 403, so a probe cannot
+# tell a blocked repo from a directory that never had one.
+#
+# Scoped to VCS directories on purpose. A blanket "deny all dotfiles" would take
+# /.well-known/acme-challenge/ with it and quietly break certificate renewal,
+# and it would swallow the sponsor-ad URIs too.
+#
+# Emitted into every vhost that serves files from a home directory, since the
+# whole point is that it applies to paths nobody remembered to think about.
+_nginx_deny_vcs() {
+	cat <<'EOF'
+
+
+	# see _nginx_deny_vcs in root-ubuntu.sh
+	location ~ "/\.(git|svn|hg|bzr)(/|$)" {
+		access_log off;
+		return 404;
+	}
+EOF
+}
+
+_nginx_userdir_vhosts() {
+	local listen="$1" ssl="$2"
+
+	# ..$WEB_DOMAIN -- one label deeper than the user pages, so the
+	# two regexes cannot collide ([a-z0-9_-] does not match a dot).
+	#
+	# $devapp_port comes from the generated map: set means "proxy to it",
+	# empty means "serve ~/apps//public as a static site".
+	[[ "$DEV_APPS" == 1 ]] && cat <[a-z0-9][a-z0-9-]{0,62})\.(?[a-z_][a-z0-9_-]{0,31})\.${WEB_DOMAIN//./\\.}\$";
+$ssl
+	server_tokens off;
+
+	# AI crawlers / scrapers -- \$bad_bot comes from the map in
+	# profullstack-badbots.conf. 403 rather than 444 so a false positive is a
+	# readable error instead of a hung connection.
+	if (\$bad_bot) { return 403; }
+
+	# The polite half of the same policy, for crawlers that do read it.
+	location = /robots.txt {
+		add_header Content-Type text/plain;
+		return 200 "User-agent: GPTBot\\nUser-agent: OAI-SearchBot\\nUser-agent: ChatGPT-User\\nUser-agent: ClaudeBot\\nUser-agent: anthropic-ai\\nUser-agent: Claude-Web\\nUser-agent: PerplexityBot\\nUser-agent: CCBot\\nUser-agent: Google-Extended\\nUser-agent: Applebot-Extended\\nUser-agent: Bytespider\\nUser-agent: Amazonbot\\nUser-agent: meta-externalagent\\nUser-agent: cohere-ai\\nUser-agent: Diffbot\\nDisallow: /\\n";
+	}$(_nginx_deny_vcs)
+	client_max_body_size 512m;
+
+	location / {
+		root /home/\$devapp_user/apps/\$devapp_name/public;
+		index index.html;
+		autoindex off;
+
+		# A declared port means there is a server running: hand the request
+		# over. 'rewrite ... last' is one of the only two things you may
+		# safely do inside an if -- proxy_pass in here would inherit the
+		# try_files below and 404 every proxied request.
+		if (\$devapp_port) {
+			rewrite ^ /.devapp-proxy last;
+		}
+
+		try_files \$uri \$uri/ =404;
+	}
+
+	# internal: only reachable through the rewrite above. \$request_uri is the
+	# original path and query, so the app sees exactly what the client sent.
+	location = /.devapp-proxy {
+		internal;
+		proxy_pass http://127.0.0.1:\$devapp_port\$request_uri;
+		proxy_http_version 1.1;
+		proxy_set_header Host \$host;
+		proxy_set_header X-Real-IP \$remote_addr;
+		proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
+		proxy_set_header X-Forwarded-Proto \$scheme;
+		# vite/next/HMR all need the websocket upgrade passed through
+		proxy_set_header Upgrade \$http_upgrade;
+		proxy_set_header Connection \$connection_upgrade;
+		# dev servers stream and hot-reload; buffering breaks both
+		proxy_buffering off;
+		proxy_read_timeout 3600s;
+	}
+}
+
+EOF
+
+	# username comes from the hostname here, captured by the server_name regex
+	cat <[a-z_][a-z0-9_-]{0,31})\.${WEB_DOMAIN//./\\.}\$";
+$ssl
+	root /home/\$uname/public_html;
+	index index.html;
+	# A directory with no index.html lists its contents instead of 404ing --
+	# ~/public_html is a place to drop files, and having to write an index by
+	# hand to see them defeats that. index.html still wins where it exists.
+	autoindex on;
+	autoindex_exact_size off;
+	autoindex_localtime on;
+	server_tokens off;$(_nginx_sponsor_ad_ssi)
+
+	# AI crawlers / scrapers -- \$bad_bot comes from the map in
+	# profullstack-badbots.conf. 403 rather than 444 so a false positive is a
+	# readable error instead of a hung connection.
+	if (\$bad_bot) { return 403; }
+
+	# The polite half of the same policy, for crawlers that do read it.
+	location = /robots.txt {
+		add_header Content-Type text/plain;
+		return 200 "User-agent: GPTBot\\nUser-agent: OAI-SearchBot\\nUser-agent: ChatGPT-User\\nUser-agent: ClaudeBot\\nUser-agent: anthropic-ai\\nUser-agent: Claude-Web\\nUser-agent: PerplexityBot\\nUser-agent: CCBot\\nUser-agent: Google-Extended\\nUser-agent: Applebot-Extended\\nUser-agent: Bytespider\\nUser-agent: Amazonbot\\nUser-agent: meta-externalagent\\nUser-agent: cohere-ai\\nUser-agent: Diffbot\\nDisallow: /\\n";
+	}$(_nginx_deny_vcs)$(_nginx_sponsor_ad_locations)$(_nginx_sponsor_ad_listing '/$')
+
+	location / {
+		# \$uri/ has to stay ahead of the fallback: it is what hands a
+		# directory to the index/autoindex handler at all
+		try_files \$uri \$uri/ =404;
+	}
+}
+
+server {
+$listen
+	server_name $WEB_DOMAIN;
+$ssl
+	root /var/www/userdirs;
+	index index.html;
+	autoindex off;
+	server_tokens off;$(_nginx_sponsor_ad_ssi)
+
+	# AI crawlers / scrapers -- \$bad_bot comes from the map in
+	# profullstack-badbots.conf. 403 rather than 444 so a false positive is a
+	# readable error instead of a hung connection.
+	if (\$bad_bot) { return 403; }
+
+	# The polite half of the same policy, for crawlers that do read it.
+	location = /robots.txt {
+		add_header Content-Type text/plain;
+		return 200 "User-agent: GPTBot\\nUser-agent: OAI-SearchBot\\nUser-agent: ChatGPT-User\\nUser-agent: ClaudeBot\\nUser-agent: anthropic-ai\\nUser-agent: Claude-Web\\nUser-agent: PerplexityBot\\nUser-agent: CCBot\\nUser-agent: Google-Extended\\nUser-agent: Applebot-Extended\\nUser-agent: Bytespider\\nUser-agent: Amazonbot\\nUser-agent: meta-externalagent\\nUser-agent: cohere-ai\\nUser-agent: Diffbot\\nDisallow: /\\n";
+	}$(_nginx_deny_vcs)
+
+	# /~user without a trailing slash: redirect so relative links resolve
+	location ~ "^/~([a-z_][a-z0-9_-]{0,31})\$" {
+		return 301 /~\$1/;
+	}$(_nginx_sponsor_ad_locations)$(_nginx_sponsor_ad_listing '^/~([a-z_][a-z0-9_-]{0,31})(/(?:.*/)?)$' \
+"		alias /home/\$1/public_html\$2;
+		index index.html;
+		autoindex on;
+		autoindex_exact_size off;
+		autoindex_localtime on;")
+
+	# same listing behaviour as the subdomain form, so /~user/x/ and
+	# user.$WEB_DOMAIN/x/ do not disagree about what they show
+	location ~ "^/~([a-z_][a-z0-9_-]{0,31})(/.*)?\$" {
+		alias /home/\$1/public_html\$2;
+		index index.html;
+		autoindex on;
+		autoindex_exact_size off;
+		autoindex_localtime on;
+	}
+
+	location / {
+		try_files \$uri \$uri/ =404;
+	}
+}
+EOF
+}
+
+# Port 80 always redirects to 443, so 443 must always answer -- an
+# ssl_certificate pointing at a missing file stops nginx from starting at all.
+# Until the real cert is issued this self-signed one keeps the listener up
+# (browsers will warn, but the redirect works and nothing 5xxs).
+FALLBACK_CERT_DIR=/etc/nginx/ssl
+ensure_fallback_cert() {
+	[[ -s "$FALLBACK_CERT_DIR/fullchain.pem" && -s "$FALLBACK_CERT_DIR/privkey.pem" ]] && return 0
+	install -d -m 0755 "$FALLBACK_CERT_DIR"
+	openssl req -x509 -newkey rsa:2048 -sha256 -days 3650 -nodes \
+		-keyout "$FALLBACK_CERT_DIR/privkey.pem" \
+		-out "$FALLBACK_CERT_DIR/fullchain.pem" \
+		-subj "/CN=$WEB_DOMAIN" \
+		-addext "subjectAltName=DNS:$WEB_DOMAIN,DNS:*.$WEB_DOMAIN" >/dev/null 2>&1 \
+		|| { warn "could not generate the fallback certificate"; return 1; }
+	chmod 0600 "$FALLBACK_CERT_DIR/privkey.pem"
+	warn "using a SELF-SIGNED cert for $WEB_DOMAIN -- browsers will warn until a real one is issued"
+	return 0
+}
+
+configure_nginx() {
+	local crt key changed=0
+	if [[ -s "$CERT_DIR/fullchain.pem" ]]; then
+		crt="$CERT_DIR/fullchain.pem"; key="$CERT_DIR/privkey.pem"
+	else
+		ensure_fallback_cert || return 1
+		crt="$FALLBACK_CERT_DIR/fullchain.pem"; key="$FALLBACK_CERT_DIR/privkey.pem"
+	fi
+
+	# Proxying websockets needs Connection: upgrade on upgrade requests and
+	# Connection: close otherwise -- the one map everybody ends up writing.
+	write_if_changed /etc/nginx/conf.d/profullstack-upgrade.conf <<'EOF' && changed=1
+# managed by cli-tools/root-ubuntu.sh
+map $http_upgrade $connection_upgrade {
+	default upgrade;
+	''      close;
+}
+EOF
+
+	if [[ "$DEV_APPS" == 1 ]]; then
+		write_devapps_generator
+	else
+		# the map references $devapp_user, a variable that only the dev-app
+		# vhost defines. Leaving it behind after DEV_APPS=0 stops nginx from
+		# starting at all ("unknown devapp_user variable").
+		[[ -e "$DEV_APPS_MAP" ]] && { rm -f "$DEV_APPS_MAP"; changed=1; }
+		systemctl disable --now profullstack-devapps.timer >/dev/null 2>&1
+	fi
+
+	# must exist before nginx -t: the vhosts below reference $bad_bot, and
+	# nginx will not start if no map defines it.
+	write_bad_bots_map && changed=1
+
+	# must also come first: the vhosts alias the ad fragment, and it should be
+	# on disk before the reload at the end of this function makes them live
+	if [[ "$SPONSOR_AD" == 1 ]]; then
+		write_sponsor_ad_generator
+	else
+		remove_sponsor_ad
+	fi
+
+	{
+		echo "# managed by cli-tools/root-ubuntu.sh -- per-user public_html"
+		echo "# https://$WEB_DOMAIN/~user  and  https://user.$WEB_DOMAIN"
+		echo
+		echo "# port 80: ACME challenges, everything else redirects"
+		echo "server {"
+		echo "	listen 80 default_server;"
+		echo "	listen [::]:80 default_server;"
+		echo "	server_name $WEB_DOMAIN *.$WEB_DOMAIN;"
+		echo "	# must come before the redirect: an http-01 challenge is fetched"
+		echo "	# over http, and a blanket 301 to https would break issuance"
+		echo "	location ^~ /.well-known/acme-challenge/ {"
+		echo "		root $ACME_WEBROOT;"
+		echo "		default_type \"text/plain\";"
+		echo "	}"
+		echo "	location / { return 301 https://\$host\$request_uri; }"
+		echo "}"
+		echo
+		_nginx_userdir_vhosts \
+			$'\tlisten 443 ssl;\n\tlisten [::]:443 ssl;\n\thttp2 on;' \
+			"	ssl_certificate     $crt;
+	ssl_certificate_key $key;"
+	} | write_if_changed /etc/nginx/sites-available/userdirs.conf && changed=1
+
+	fetch_logo || warn "could not fetch $LOGO_URL (landing page will link to it instead)"
+	write_landing_page
+
+	if [[ ! -L /etc/nginx/sites-enabled/userdirs.conf ]]; then
+		ln -sfn /etc/nginx/sites-available/userdirs.conf \
+			/etc/nginx/sites-enabled/userdirs.conf
+		changed=1
+	fi
+	# default vhost would otherwise claim port 80 for every unmatched host
+	[[ -e /etc/nginx/sites-enabled/default ]] \
+		&& { rm -f /etc/nginx/sites-enabled/default; changed=1; }
+
+	if ! nginx -t; then
+		warn "nginx config test failed -- disabling the userdir vhost"
+		rm -f /etc/nginx/sites-enabled/userdirs.conf
+		nginx -t && systemctl reload nginx
+		return 1
+	fi
+	systemctl is-enabled nginx >/dev/null 2>&1 || systemctl enable nginx
+
+	# Reload only when something actually changed. A maintenance run should be
+	# invisible to anyone using the box, and reloading drops in-flight
+	# websocket connections to everyone's dev servers.
+	if [[ "$changed" == 1 ]] || ! systemctl is-active nginx >/dev/null 2>&1; then
+		note "nginx config"
+		systemctl reload nginx || systemctl restart nginx
+	else
+		info "nginx config unchanged -- not reloading"
+	fi
+	# which cert ended up in use is already reported by ensure_fallback_cert
+	return 0
+}
+
+# ------------------------------------------------------------------ users ---
+
+create_user() {
+	local spec="$1" groups="$2" login home pass g
+	login="$(user_login "$spec")"
+
+	if id -u "$login" >/dev/null 2>&1; then
+		info "user $login already exists"
+	else
+		info "creating user $login"
+		useradd -m -s /bin/zsh -c "$spec" "$login" || { warn "useradd $login failed"; return 1; }
+		if interactive; then
+			read -r -s -p "    password for ${login} (blank = key-only login): " pass; echo
+			if [[ -n "$pass" ]]; then
+				printf '%s:%s\n' "$login" "$pass" | chpasswd
+			else
+				passwd -l "$login" >/dev/null
+			fi
+			unset pass
+		else
+			passwd -l "$login" >/dev/null
+		fi
+	fi
+
+	# sshd here has PasswordAuthentication off, so a user with no key of their
+	# own simply cannot log in -- ask for it now rather than let that surprise
+	# them later. Stored in the repo so re-runs and rebuilds keep working.
+	if interactive && [[ ! -s "$KEYS_DIR/$login.pub" ]]; then
+		echo "    ${login} needs their own public key to log in (password auth is off)."
+		read -r -p "    paste ${login}'s ssh public key (blank to skip): " pubkey
+		if [[ -n "${pubkey// }" ]]; then
+			if [[ "$pubkey" =~ ^(ssh-(rsa|ed25519|dss)|ecdsa-sha2-|sk-ssh-|sk-ecdsa-) ]]; then
+				install -d -m 0755 "$KEYS_DIR"
+				printf '%s\n' "$pubkey" >"$KEYS_DIR/$login.pub"
+				info "saved ssh-keys/$login.pub"
+			else
+				warn "that does not look like an ssh public key -- skipped"
+			fi
+		fi
+		unset pubkey
+	fi
+
+	ensure_zsh_shell "$login" || warn "$login may still be on bash"
+
+	for g in ${groups//,/ }; do
+		getent group "$g" >/dev/null || groupadd "$g" || { warn "cannot create group $g"; continue; }
+		usermod -aG "$g" "$login" || warn "could not add $login to $g"
+	done
+	info "$login groups: $(id -nG "$login")"
+
+	refresh_user "$login"
+	remember_user "$login"
+	return 0
+}
+
+# What a re-run does to an account that already exists: no useradd, no
+# password, no group changes, no questions -- just bring the files we own up
+# to date and re-assert the permissions sshd and nginx depend on.
+refresh_user() {
+	local login="$1" home
+	home="$(user_home "$login")"
+	[[ -n "$home" && -d "$home" ]] || { warn "no home dir for $login"; return 1; }
+
+	# Adopted accounts arrive on whatever shell the cloud image gave them --
+	# /home/ubuntu is the usual one. Without this a refresh left them on bash
+	# forever while installing a zsh setup around them. ensure_zsh_shell only
+	# converts from bash/sh, so a deliberate choice of anything else stands.
+	ensure_zsh_shell "$login" || warn "$login may still be on bash"
+
+	install_dotfiles "$home" "$login"
+	install_public_html "$home" "$login"
+	install_dev_apps_dir "$home" "$login"
+	# last word on permissions, after everything has written into the home
+	fix_home_permissions "$home" "$login" || warn "$login home permissions need attention"
+	return 0
+}
+
+if [[ ${#USERS[@]} -gt 0 ]]; then
+	log "creating users"
+	for i in "${!USERS[@]}"; do
+		try "user ${USERS[$i]}" create_user "${USERS[$i]}" "${USER_GROUPS[$i]}"
+	done
+fi
+
+if [[ ${#KNOWN_USERS[@]} -gt 0 ]]; then
+	log "refreshing existing users"
+	for _l in "${KNOWN_USERS[@]}"; do
+		try "refresh $_l" refresh_user "$_l"
+		remember_user "$_l"
+	done
+	unset _l
+fi
+
+log "installing dotfiles for root"
+try "root dotfiles" install_dotfiles /root root
+# our .zshrc has a dedicated root prompt, so root runs zsh too
+try "root login shell -> zsh" ensure_zsh_shell root
+
+if [[ "$SKIP_TOOLS" == 1 ]]; then
+	log "skipping oh-my-zsh/mise/moshcode (--skip-tools)"
+else
+	log "installing/updating oh-my-zsh, mise, moshcode"
+	# these installers are all "fetch the current version and put it in place",
+	# so running them again is exactly how the box picks up new releases
+	while read -r login; do
+		[[ -n "$login" ]] || continue
+		id -u "$login" >/dev/null 2>&1 || continue
+		try "oh-my-zsh ($login)" install_omz "$login"
+		try "mise ($login)"      install_mise "$login"
+		try "moshcode ($login)"  install_moshcode "$login"
+		# separate step: installing moshcode does not update what it manages
+		try "moshcode tools ($login)" update_moshcode_tools "$login"
+	done < <(printf 'root\n'; all_logins)
+fi
+
+# ---------------------------------------------------------------- wrap up ---
+
+if [[ "$SKIP_TAILSCALE" == 1 ]]; then
+	log "skipping tailscale (--skip-tailscale)"
+else
+	log "installing tailscale"
+	try "tailscale" install_tailscale
+fi
+
+PUBLIC_IP="$(curl -4 -fsS --max-time 5 https://api.ipify.org 2>/dev/null \
+	|| hostname -I | awk '{print $1}')"
+
+if [[ "$SKIP_WEB" == 1 ]]; then
+	log "skipping web/TLS (--skip-web)"
+else
+	log "configuring per-user web hosting on $WEB_DOMAIN"
+
+	# DNS before the cert: a dns-01 challenge for *..$WEB_DOMAIN only
+	# works once the zone knows about the name, and http-01 needs the name to
+	# resolve here at all.
+	if [[ -n "$PORKBUN_API_KEY" && -n "$PORKBUN_SECRET_API_KEY" ]]; then
+		try "dns records at porkbun" ensure_dns_records "$PUBLIC_IP"
+	fi
+
+	# nginx first: http-01 needs something answering on port 80 to serve the
+	# challenge. It comes up on the self-signed fallback if there is no cert yet.
+	try "nginx userdir vhosts" configure_nginx
+	try "certificate for $WEB_DOMAIN" issue_wildcard_cert
+	# again, to pick up a cert that was just issued (no-op otherwise)
+	if [[ -s "$CERT_DIR/fullchain.pem" ]]; then
+		try "nginx with the issued certificate" configure_nginx
+	fi
+	# compile the dev-app routes now rather than waiting for the timer
+	[[ "$DEV_APPS" == 1 && -x /usr/local/bin/profullstack-devapps ]] \
+		&& /usr/local/bin/profullstack-devapps
+fi
+
+# only when it is actually wrong -- otherwise this is noise on every run
+if [[ "$(timedatectl show -p Timezone --value 2>/dev/null)" != UTC ]]; then
+	try "set timezone" timedatectl set-timezone UTC && note "timezone -> UTC"
+fi
+
+touch "$STATE_DIR/provisioned"
+
+echo
+if [[ ${#FAILED[@]} -gt 0 ]]; then
+	warn "${#FAILED[@]} step(s) failed:"
+	printf '       - %s\n' "${FAILED[@]}" >&2
+else
+	log "all steps completed"
+fi
+
+# What a re-run actually did. On a settled box this list should be empty --
+# that is the point.
+echo
+if [[ ${#CHANGED[@]} -gt 0 ]]; then
+	log "changed this run (${#CHANGED[@]})"
+	printf '       - %s\n' "${CHANGED[@]}"
+else
+	log "nothing changed -- the box was already up to date"
+fi
+
+# Files we would have updated but did not, because someone had edited them.
+if [[ ${#PRESERVED[@]} -gt 0 ]]; then
+	echo
+	warn "${#PRESERVED[@]} file(s) kept as the user left them; the new version is beside them as .new:"
+	printf '       - %s\n' "${PRESERVED[@]}" >&2
+	info "      diff them, or re-run with --force-dotfiles to overwrite (a .bak is kept)"
+fi
+
+echo
+info "ufw:  $(ufw status | head -1)"
+info "open: $(ufw status | awk '/ALLOW/{printf "%s ", $1}')"
+info "mosh: $(command -v mosh-server >/dev/null && echo "$(mosh-server --version 2>&1 | head -1)" || echo 'MISSING')"
+info "tail: $(tailscale ip -4 2>/dev/null | head -1 || echo 'not joined to a tailnet')"
+info "motd: $([[ -s $MOTD_CACHE ]] && echo "cached ($(wc -l <"$MOTD_CACHE") lines)" || echo 'empty')"
+echo
+printf '    %-12s %-6s %-6s %-9s %-6s %-5s %s\n' USER OMZ MISE MOSHCODE TMUX APPS SHELL
+while read -r login; do
+	[[ -n "$login" ]] || continue
+	id -u "$login" >/dev/null 2>&1 || continue
+	h="$(user_home "$login")"
+	printf '    %-12s %-6s %-6s %-9s %-6s %-5s %s\n' "$login" \
+		"$([[ -d "$h/.oh-my-zsh" ]]        && echo ok || echo MISS)" \
+		"$([[ -x "$h/.local/bin/mise" ]]   && echo ok || echo MISS)" \
+		"$([[ -e "$h/.local/bin/moshcode" || -e "$h/.moshcode" ]] && echo ok || echo '?')" \
+		"$([[ -e "$h/.tmux.conf" ]]        && echo ok || echo MISS)" \
+		"$(find "$h/apps" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l)" \
+		"$(getent passwd "$login" | cut -d: -f7)"
+done < <(printf 'root\n'; all_logins)
+
+# Flag anyone who cannot actually get in. With sshd refusing passwords, a
+# locked password plus no personal key means no login at all -- which otherwise
+# only shows up when they try and get "Permission denied".
+_pwauth="$(sshd -T 2>/dev/null | awk '/^passwordauthentication/{print $2}')"
+echo
+info "sshd passwordauthentication: ${_pwauth:-unknown}"
+while read -r l; do
+	[[ -n "$l" ]] || continue
+	id -u "$l" >/dev/null 2>&1 || continue
+	h="$(user_home "$l")"
+	_pw="$(passwd -S "$l" 2>/dev/null | awk '{print $2}')"   # P=set L=locked NP=none
+	# Report on the file that actually decides whether they can get in, not on
+	# what this script would have put there. Nothing is added to
+	# authorized_keys unless ssh-keys/.pub exists, so an empty file
+	# really does mean no way in.
+	_nkeys=0
+	[[ -s "$h/.ssh/authorized_keys" ]] \
+		&& _nkeys="$(grep -cvE '^[[:space:]]*(#|$)' "$h/.ssh/authorized_keys" 2>/dev/null || echo 0)"
+	if [[ "$_nkeys" -eq 0 ]]; then
+		if [[ "$_pwauth" == no ]]; then
+			warn "$l has NO keys in ~/.ssh/authorized_keys and sshd refuses passwords -- they cannot log in"
+		elif [[ "$_pw" == L ]]; then
+			warn "$l has no keys and a locked password"
+		fi
+		info "      fix: put their public key in ssh-keys/$l.pub and re-run, or"
+		info "           echo '' >> $h/.ssh/authorized_keys"
+	else
+		info "$l: $_nkeys key(s) authorised"
+	fi
+done < <(all_logins)
+
+echo
+info "DNS records required for $WEB_DOMAIN (point at this host):"
+info "    A    $WEB_DOMAIN      ->  ${PUBLIC_IP:-}"
+info "    A    *.$WEB_DOMAIN    ->  ${PUBLIC_IP:-}"
+if [[ "$DEV_APPS" == 1 ]]; then
+	# DNS wildcards match ONE label, so *.dev.profullstack.com does not answer
+	# for api.alice.dev.profullstack.com. Each user needs their own.
+	while read -r l; do
+		info "    A    *.$l.$WEB_DOMAIN  ->  ${PUBLIC_IP:-}   (${l}'s dev apps)"
+	done < <(web_users)
+fi
+if [[ -s "$CERT_DIR/fullchain.pem" ]]; then
+	info "cert: expires $(openssl x509 -in "$CERT_DIR/fullchain.pem" -noout -enddate 2>/dev/null | cut -d= -f2) ($(cert_days_left) days)"
+	info "      names: $(cert_names | tr '\n' ' ')"
+	_miss="$(cert_missing_names)"
+	[[ -n "$_miss" ]] && warn "not covered by the cert: $(printf '%s' "$_miss" | tr '\n' ' ')"
+else
+	info "cert: none yet (self-signed fallback in use). To issue the real one:"
+	info "      sudo PORKBUN_API_KEY=pk1_... PORKBUN_SECRET_API_KEY=sk1_... $0"
+fi
+while read -r l; do
+	info "    https://$WEB_DOMAIN/~$l  |  https://$l.$WEB_DOMAIN"
+	[[ "$DEV_APPS" == 1 ]] && info "        dev apps: mkdir ~/apps/  ->  https://.$l.$WEB_DOMAIN"
+done < <(all_logins)
+
+# ---------------------------------------------------------------- reboot ---
+#
+# This script is re-run on a live box, so a reboot is never the default. Even
+# when one is genuinely required, someone may be mid-session -- so say who is
+# logged in and let a human decide.
+echo
+_reboot_required=0
+[[ -e /var/run/reboot-required ]] && _reboot_required=1
+if [[ "$_reboot_required" == 1 ]]; then
+	warn "a reboot is required (kernel or libc updated)"
+	[[ -s /var/run/reboot-required.pkgs ]] \
+		&& info "      $(tr '\n' ' ' /dev/null | awk '{print $1}' | sort -u | tr '\n' ' ')"
+
+if [[ "$REBOOT_POLICY" == 0 ]]; then
+	[[ "$_reboot_required" == 1 ]] && info "not rebooting (--no-reboot); do it when it suits you"
+elif [[ "$_reboot_required" == 0 && "$REBOOT_POLICY" != 2 ]]; then
+	: # nothing to do
+elif ! interactive; then
+	# an unattended maintenance run must never take the box down by itself
+	warn "reboot required, but this run is unattended -- not rebooting"
+	info "      run 'sudo reboot' when the box is free"
+elif confirm "Reboot now? ${_sessions:+($_sessions session(s) open) }[y/N]" n; then
+	info "rebooting in 5s..."
+	sleep 5
+	reboot
+fi
diff --git a/server.conf.example b/server.conf.example
new file mode 100644
index 0000000..ee988b6
--- /dev/null
+++ b/server.conf.example
@@ -0,0 +1,100 @@
+# Example configuration for root-ubuntu.sh.
+#
+# Copy to /etc/cli-tools/server.conf on the box, and give it 0600 if you put
+# any credentials in it:
+#
+#   install -d -m 0755 /etc/cli-tools
+#   install -m 0600 server.conf.example /etc/cli-tools/server.conf
+#
+# KEY=value, one per line, # for comments. Surrounding quotes are stripped, so
+# quote anything with a space in it.
+#
+# The file is READ, not sourced: nothing in it is executed, and $(...) or
+# backticks stay literal text rather than running as root. It is deliberately
+# not JSON either, because the script runs before apt has installed jq and a
+# bootstrap that cannot read its own config until it has installed a parser has
+# a hole in it.
+#
+# The environment wins over this file, so a one-off run can override any line
+# here without editing it. Everything is optional; a box with no config file at
+# all still provisions.
+
+# ── the box ──────────────────────────────────────────────────────────────────
+
+# Per-user pages are served at https://$WEB_DOMAIN/~user and
+# https://user.$WEB_DOMAIN, and dev apps at https://..$WEB_DOMAIN.
+#WEB_DOMAIN=dev.example.com
+
+# Each account's address is @$MAIL_DOMAIN on the landing page.
+#MAIL_DOMAIN=example.com
+
+# Port to open in ufw. Change it here if sshd is not on 22, or the firewall
+# will happily lock you out of the box you are provisioning.
+#SSH_PORT=22
+
+# ── accounts ─────────────────────────────────────────────────────────────────
+
+# Groups new accounts land in when nothing is passed on the command line.
+#DEFAULT_GROUPS=sudo,admin
+
+# ── dotfiles (optional) ──────────────────────────────────────────────────────
+
+# The shell/editor/tmux/irssi config to install into every account. It is not
+# in this repository and cannot be: a dotfiles tree carries ssh config,
+# known_hosts and sometimes keys, and cli-tools is public.
+#
+# With none of these set the box still gets everything else -- packages,
+# firewall, accounts, zsh, oh-my-zsh, mise, moshcode, nginx, TLS -- and each
+# account keeps whatever dotfiles it already had.
+#DOTFILES_REPO=git@github.com:example/dotfiles.git
+
+# Or point at a checkout that is already on the box, instead of cloning.
+#DOTFILES_DIR=/root/dotfiles
+
+# Where a pasted public key is filed so re-runs and rebuilds keep working.
+# Defaults to $DOTFILES_DIR/ssh-keys, or /var/lib/profullstack/ssh-keys when
+# there is no dotfiles checkout.
+#KEYS_DIR=/etc/cli-tools/ssh-keys
+
+# ── TLS ──────────────────────────────────────────────────────────────────────
+
+# Where Let's Encrypt sends expiry warnings. There is no default: a public
+# script must not ship somebody's address. Certificates issue without one --
+# you just do not get the reminder mail.
+#ACME_EMAIL=ops@example.com
+
+# A wildcard certificate needs a DNS-01 challenge, which needs API access to
+# whoever hosts the zone. Without either pair below the box still gets a
+# certificate, but only for names that already resolve to it, and per-user dev
+# app wildcards (*..$WEB_DOMAIN) are not possible at all.
+#
+# Keep this file 0600 if you put these here. Better still, pass them in the
+# environment for the one run that issues the certificate and leave them out of
+# the file entirely -- renewal does not need them again.
+#PORKBUN_API_KEY=
+#PORKBUN_SECRET_API_KEY=
+
+# ...or, for a zone at Cloudflare instead:
+#CLOUDFLARE_API_TOKEN=
+
+# ── tailscale ────────────────────────────────────────────────────────────────
+
+# Joins the tailnet unattended. Without it the script prints the command to run
+# by hand, which is the right answer for a box you are sitting in front of.
+#TS_AUTHKEY=
+#TS_HOSTNAME=
+
+# ── the landing page ─────────────────────────────────────────────────────────
+
+# The team blog is one account's userdir rather than its own vhost, so the
+# landing page has to be told whose. Unset means no blog section at all, which
+# beats a link to a 404.
+#BLOG_USER=
+#BLOG_TITLE=the team blog
+
+# ── sponsor ad ───────────────────────────────────────────────────────────────
+
+# Off unless a slot id is set, and there is deliberately no default one: an ad
+# slot is an account, so a shared default would bill every box that ever ran
+# this script to whoever owned that slot.
+#SPONSOR_AD_SLOT=
diff --git a/test/root-ubuntu.test.ts b/test/root-ubuntu.test.ts
new file mode 100644
index 0000000..32ca819
--- /dev/null
+++ b/test/root-ubuntu.test.ts
@@ -0,0 +1,313 @@
+import { execFileSync } from 'node:child_process';
+import { readFileSync } from 'node:fs';
+import { dirname, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { describe, expect, it } from 'vitest';
+
+/**
+ * Tests for root-ubuntu.sh, which is a shell script and therefore cannot be
+ * imported. Two kinds of check live here.
+ *
+ * The first kind is behavioural: the pure helpers are cut out of the script
+ * with sed and run in a real bash, so what is asserted is what the file
+ * actually does rather than what it appears to say. Only self-contained
+ * functions can be treated this way — anything that reaches for apt, systemd
+ * or the network is left to a real box.
+ *
+ * The second kind guards the invariants that make it safe to publish this
+ * file at all. They read like paranoia until you remember where the script
+ * came from: a private dotfiles repo, where an email address and an ad slot id
+ * in the source cost nothing. In a public repository, curled onto other
+ * people's machines, both are somebody else's problem to pay for.
+ */
+
+const here = dirname(fileURLToPath(import.meta.url));
+const SCRIPT = resolve(here, '..', 'root-ubuntu.sh');
+const SOURCE = readFileSync(SCRIPT, 'utf8');
+
+/**
+ * Run one or more of the script's functions without running the script.
+ *
+ * The functions are extracted by name rather than the file being sourced,
+ * because sourcing it would execute the whole provisioner — it is a linear
+ * script, not a library, and there is no import-guard to stop at.
+ */
+function shell(fns: string[], snippet: string): string {
+  const extract = fns.map((fn) => `sed -n '/^${fn}()/,/^}/p' "$S"`).join('; ');
+  const script = `set -uo pipefail\nS=${JSON.stringify(SCRIPT)}\neval "$(${extract})"\n${snippet}`;
+  return execFileSync('bash', ['-c', script], { encoding: 'utf8' }).trim();
+}
+
+/** Same, but for a snippet expected to fail; returns the exit status. */
+function status(fns: string[], snippet: string): number {
+  try {
+    shell(fns, snippet);
+    return 0;
+  } catch (error) {
+    return (error as { status?: number }).status ?? 1;
+  }
+}
+
+describe('the script itself', () => {
+  it('is valid bash', () => {
+    expect(() => execFileSync('bash', ['-n', SCRIPT])).not.toThrow();
+  });
+
+  it('is executable, since the documented use is ./root-ubuntu.sh', () => {
+    execFileSync('test', ['-x', SCRIPT]);
+  });
+
+  it('prints usage without being root, so --help is never a privileged act', () => {
+    const out = execFileSync('bash', [SCRIPT, '--help'], { encoding: 'utf8' });
+    expect(out).toContain('root-ubuntu.sh');
+    expect(out).toContain('--refresh');
+  });
+
+  it('says so in one sentence when run under sh instead of bash', () => {
+    // /bin/sh on Ubuntu is dash. Without this guard, `curl … | sh` dies on the
+    // first [[ with a syntax error naming a line the user never typed.
+    let failed = false;
+    try {
+      execFileSync('dash', [SCRIPT, '--refresh'], { encoding: 'utf8', stdio: 'pipe' });
+    } catch (error) {
+      failed = true;
+      const err = error as { stderr?: Buffer; status?: number };
+      expect(String(err.stderr)).toContain('running it under sh');
+      expect(String(err.stderr)).toContain('bash -s --');
+      expect(err.status).toBe(1);
+    }
+    expect(failed).toBe(true);
+  });
+
+  it('documents the pipe with bash, never sh', () => {
+    expect(SOURCE).toContain('root-ubuntu.sh | bash -s -- --refresh');
+    expect(SOURCE).not.toMatch(/root-ubuntu\.sh \| sh /);
+  });
+
+  it('refuses to do anything as a normal user', () => {
+    let failed = false;
+    try {
+      execFileSync('bash', [SCRIPT, '--refresh'], { encoding: 'utf8', stdio: 'pipe' });
+    } catch (error) {
+      failed = true;
+      expect(String((error as { stderr?: Buffer }).stderr)).toContain('must run as root');
+    }
+    expect(failed).toBe(true);
+  });
+});
+
+describe('what may not be in a public file', () => {
+  // The script is served raw from GitHub and piped into root shells. Anything
+  // in it that identifies a person identifies them to everyone who runs it.
+  it('carries no personal identifiers', () => {
+    const names = /\b(anthony|preshy|phuc|ralyodio|chovy|h4kr|bonita)\b/i;
+    expect(SOURCE).not.toMatch(names);
+  });
+
+  it('has no default ad slot', () => {
+    // A slot id is an account. Shipping one bills every box that ever runs
+    // this script to whoever owns it, and inflates their impressions with
+    // traffic they never had.
+    expect(SOURCE).toMatch(/SPONSOR_AD_SLOT="\$\{SPONSOR_AD_SLOT:-\}"/);
+    expect(SOURCE).not.toMatch(/SPONSOR_AD_SLOT="\$\{SPONSOR_AD_SLOT:-[0-9a-f]{8}-/);
+  });
+
+  it('has no contact address baked into the ACME config', () => {
+    // A real address here would send a stranger's certificate expiry warnings
+    // to a person who has never heard of their box.
+    expect(SOURCE).toMatch(/ACME_EMAIL="\$\{ACME_EMAIL:-\}"/);
+  });
+
+  it('holds no key material or credentials', () => {
+    expect(SOURCE).not.toMatch(/-----BEGIN [A-Z ]*PRIVATE KEY-----/);
+    expect(SOURCE).not.toMatch(/\b(gh[pousr]_[A-Za-z0-9]{20,}|AKIA[0-9A-Z]{16}|sk-ant-)/);
+    // The placeholders in the help text must stay placeholders: three trailing
+    // dots, never a real prefix followed by an actual key.
+    expect(SOURCE).not.toMatch(/\b(pk1|sk1)_[A-Za-z0-9]{12,}/);
+    expect(SOURCE).not.toMatch(/tskey-auth-[A-Za-z0-9]{12,}/);
+  });
+
+  it('reads every credential from the environment or the config file', () => {
+    for (const key of [
+      'PORKBUN_API_KEY',
+      'PORKBUN_SECRET_API_KEY',
+      'CLOUDFLARE_API_TOKEN',
+      'TS_AUTHKEY',
+    ]) {
+      expect(SOURCE).toContain(`${key}="\${${key}:-}"`);
+    }
+  });
+});
+
+describe('read_server_config', () => {
+  const conf = (body: string) =>
+    `f=$(mktemp); cat >"$f" <<'CONF'\n${body}\nCONF\nread_server_config "$f"`;
+
+  it('reads KEY=value', () => {
+    expect(shell(['read_server_config'], `${conf('WEB_DOMAIN=dev.example.com')}; echo "$WEB_DOMAIN"`)).toBe(
+      'dev.example.com',
+    );
+  });
+
+  it('strips one layer of quotes, single or double', () => {
+    expect(shell(['read_server_config'], `${conf('A="one two"')}; echo "$A"`)).toBe('one two');
+    expect(shell(['read_server_config'], `${conf("B='three'")}; echo "$B"`)).toBe('three');
+  });
+
+  it('ignores comments and blank lines', () => {
+    expect(
+      shell(['read_server_config'], `${conf('# a comment\n\n  # indented\nC=kept')}; echo "$C"`),
+    ).toBe('kept');
+  });
+
+  it('lets the environment win over the file', () => {
+    // The rule everywhere else in this repo, and the reason the file is read
+    // rather than sourced: `.` would assign unconditionally and silently beat
+    // the value someone just put on the command line.
+    expect(
+      shell(['read_server_config'], `D=from-env; ${conf('D=from-file')}; echo "$D"`),
+    ).toBe('from-env');
+  });
+
+  it('executes nothing in the file', () => {
+    // It is read as root. A config file only needs to carry values, so a line
+    // that looks like a command must stay a string.
+    const out = shell(
+      ['read_server_config'],
+      `${conf('E=$(touch /tmp/root-ubuntu-pwned)\nF=`id`')}; echo "$E|${'$'}{F}"`,
+    );
+    expect(out).toBe('$(touch /tmp/root-ubuntu-pwned)|`id`');
+  });
+
+  it('is quiet about a config file that is not there', () => {
+    expect(status(['read_server_config'], 'read_server_config /nope/not/here')).toBe(0);
+  });
+});
+
+describe('login handling', () => {
+  it('takes the login from the part before the @', () => {
+    expect(shell(['user_login'], 'user_login alice@example')).toBe('alice');
+    expect(shell(['user_login'], 'user_login bob')).toBe('bob');
+  });
+
+  it('accepts the logins useradd would accept', () => {
+    expect(status(['valid_login'], 'valid_login alice')).toBe(0);
+    expect(status(['valid_login'], 'valid_login a_b-c9')).toBe(0);
+  });
+
+  it('rejects logins that would be trouble', () => {
+    // A leading digit, an uppercase letter or a slash is either refused by
+    // useradd or, worse, accepted and then unusable in the nginx regexes that
+    // map a hostname back to a home directory.
+    expect(status(['valid_login'], 'valid_login 9lives')).toBe(1);
+    expect(status(['valid_login'], 'valid_login Alice')).toBe(1);
+    expect(status(['valid_login'], 'valid_login ../root')).toBe(1);
+    expect(status(['valid_login'], "valid_login ''")).toBe(1);
+  });
+});
+
+describe('is_public_ip', () => {
+  // This is the guard that stops a private address being published into DNS,
+  // where the "points somewhere else" check would then refuse to correct it.
+  it('accepts routable addresses', () => {
+    expect(status(['is_public_ip'], 'is_public_ip 203.0.113.9')).toBe(0);
+    expect(status(['is_public_ip'], 'is_public_ip 152.53.47.37')).toBe(0);
+  });
+
+  it('rejects the private and reserved ranges', () => {
+    for (const ip of ['10.1.2.3', '192.168.0.1', '172.16.0.1', '172.31.255.1', '127.0.0.1']) {
+      expect(status(['is_public_ip'], `is_public_ip ${ip}`)).toBe(1);
+    }
+  });
+
+  it('rejects CGNAT, which is where a tailnet address lives', () => {
+    // 100.64.0.0/10. Publishing one of these makes the name resolve only for
+    // machines on the tailnet, which looks like a DNS outage to everyone else.
+    for (const ip of ['100.64.0.1', '100.99.1.1', '100.127.255.254']) {
+      expect(status(['is_public_ip'], `is_public_ip ${ip}`)).toBe(1);
+    }
+  });
+
+  it('does not mistake a neighbouring range for CGNAT', () => {
+    expect(status(['is_public_ip'], 'is_public_ip 100.63.0.1')).toBe(0);
+    expect(status(['is_public_ip'], 'is_public_ip 100.128.0.1')).toBe(0);
+  });
+
+  it('rejects anything that is not four octets', () => {
+    expect(status(['is_public_ip'], 'is_public_ip not-an-ip')).toBe(1);
+    expect(status(['is_public_ip'], "is_public_ip ''")).toBe(1);
+  });
+});
+
+describe('_expand_remote', () => {
+  // The mountpoint mirrors the path as typed, but what NFS exports is the
+  // expanded one, so the two have to be derived separately.
+  it('expands a tilde against the remote account', () => {
+    expect(shell(['_expand_remote'], '_expand_remote ubuntu "~/Downloads"')).toBe(
+      '/home/ubuntu/Downloads',
+    );
+    expect(shell(['_expand_remote'], '_expand_remote ubuntu "~"')).toBe('/home/ubuntu');
+  });
+
+  it('leaves an absolute path alone', () => {
+    expect(shell(['_expand_remote'], '_expand_remote ubuntu /srv/media')).toBe('/srv/media');
+  });
+
+  it('treats a bare path as relative to the remote home', () => {
+    expect(shell(['_expand_remote'], '_expand_remote ubuntu data')).toBe('/home/ubuntu/data');
+  });
+});
+
+describe('looks_like_dotfiles', () => {
+  // A dotfiles checkout is recognised by content, not by name, so that any
+  // repo can be one.
+  it('accepts a tree with a .zshrc or a .bashrc', () => {
+    expect(
+      status(['looks_like_dotfiles'], 'd=$(mktemp -d); touch "$d/.zshrc"; looks_like_dotfiles "$d"'),
+    ).toBe(0);
+    expect(
+      status(
+        ['looks_like_dotfiles'],
+        'd=$(mktemp -d); touch "$d/.bashrc"; looks_like_dotfiles "$d"',
+      ),
+    ).toBe(0);
+  });
+
+  it('rejects an empty directory, a missing one, and the empty string', () => {
+    expect(status(['looks_like_dotfiles'], 'd=$(mktemp -d); looks_like_dotfiles "$d"')).toBe(1);
+    expect(status(['looks_like_dotfiles'], 'looks_like_dotfiles /nope/not/here')).toBe(1);
+    expect(status(['looks_like_dotfiles'], "looks_like_dotfiles ''")).toBe(1);
+  });
+});
+
+describe('_certbot_email_args', () => {
+  // certbot refuses to run non-interactively with neither -m nor
+  // --register-unsafely-without-email, so the flag has to be chosen rather
+  // than the argument left blank.
+  it('passes -m when there is an address', () => {
+    expect(shell(['_certbot_email_args'], 'ACME_EMAIL=ops@example.com; _certbot_email_args')).toBe(
+      '-m\nops@example.com',
+    );
+  });
+
+  it('registers without one when there is not', () => {
+    expect(shell(['_certbot_email_args'], "ACME_EMAIL=''; _certbot_email_args")).toBe(
+      '--register-unsafely-without-email',
+    );
+  });
+});
+
+describe('upgrading a box that this script already provisioned', () => {
+  // The landing page is only rewritten when it still matches a marker this
+  // script has stamped. Renaming the marker without keeping the old one makes
+  // every already-provisioned box decide its own page was hand-edited, and
+  // nothing afterwards can tell that page from one somebody really wrote.
+  it('still recognises the marker from when this lived in the dotfiles repo', () => {
+    expect(SOURCE).toContain('generated by dottemplates/root-ubuntu.sh');
+    expect(SOURCE).toContain('generated by cli-tools/root-ubuntu.sh');
+  });
+
+  it('still recognises the placeholder from before markers existed', () => {
+    expect(SOURCE).toContain('User pages are at');
+  });
+});