Skip to content

feat(cloud-workspace): cleanup job applies the end-state lifecycle rules (draft) - #9987

Draft
saddlepaddle wants to merge 8 commits into
mainfrom
superset/own-workspace-retention-reaper-49e68bae
Draft

saddlepaddle wants to merge 8 commits into
mainfrom
superset/own-workspace-retention-reaper-49e68bae

Conversation

@saddlepaddle

@saddlepaddle saddlepaddle commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Draft, for review. Spec: the page "Cloud sandbox lifecycle: the end state"; model: #10006 (.agents/skills/verify-sandbox-lifecycle). Plan: plans/done/20261118-1500-cloud-workspace-retention-reaper.md.

What it does

The cleanup job (reap-retention, every 6 hours, 0 */6 * * *) is now the only thing that deletes for age or for being unused. It runs only in production and applies one rules table:

Rule What goes
1 A workspace past its wait: asleep 30 days after last use, archived 7 days, failed 30 days, stuck creating 1 day with no boot. One select over the wait table; each row is retired by one conditional write ("still in that status, still past the wait") that clears its box name (and, for asleep and stuck rows, fails it), then the retried teardown job deletes the box.
2 An archived environment's goldens; a live environment's goldens beyond the active one and the newest others (3 in all), through goldensToDelete, the same as #10000's.
3 A box, drive or golden no production row names, made more than 7 days ago. An agents drive that is not the current release's and that no box in the listing mounts (a stopped box counts).

Every keep and delete is logged as [cleanup] {rule, kind, name, action, reason}. Batches: 200 retires, 20 goldens, 200 leftover boxes, 50 drives. A run stops starting work after 200 s. Counts go to the run log, a retention.run Sentry log line, and the reap-retention cron monitor.

Fails closed. The job lists every box and every drive first, with #9943's checks: every page answered, no cursor seen twice, a non-empty box listing, and a few boxes running now present in it. If any check fails, it deletes nothing and sends a Sentry warning.

Mechanics

  • Only production deletes. One check, sandboxDeletesAllowed in lib/sandbox/deletes.ts, the same file and signature as feat(cloud-workspace): workspace drive per environment, flag-gated (draft) #9847's 038f603. It runs:

    • at the start of teardownSandbox, which returns skipped; that is every box and golden the cleanup job deletes;
    • in deleteDrive;
    • in the promote's delete of a replaced golden.

    Outside production it logs "would delete" and the caller stops. Raw deleteSandbox stays unchecked, so the release probe and real-sandbox still delete the temporary boxes they make. Staging deploys with vercel deploy --prod on the production database, so its VERCEL_ENV is production and it counts. The trpc test preload sets VERCEL_ENV=production, as on feat(cloud-workspace): workspace drive per environment, flag-gated (draft) #9847.

  • Everything else only sets a status:

    • Archive no longer queues the 7-day delete message; the 60 s stop stays. The fallback that deleted at once when queueing failed is gone.
    • A failed create stops its box and marks the row failed.
    • A row archived while its box was being made gets the box stopped, not deleted.
    • A 410 on wake was already status-only.
    • Environment archive no longer deletes its golden.
    • A reap delete message already in QStash from before this change is a no-op.
  • An unarchive after 7 days is refused ({ unarchived: false }), even before the job has run.

  • The release no longer prunes agents drives. Rule 3 does it.

  • snapshotExpiration: 0 for new boxes stays. The backfill script stays, to be run by hand.

Old stages dropped

  • The stranded and archived-backstop stages: rule 1 covers both (failed, stuck creating and archived rows, undated archives included).
  • The orphan stage: it became rule 3, which now covers untagged boxes and drives too.

Where the code is stricter than the model (all extra keeps)

  • An asleep row whose box runs now is kept (billing off has no boot row to guard a wake in flight).
  • A workspace drive that a box mounts is kept, not only an agents drive.
  • An agents drive under a day old is kept: a release publishes its drives before it records its bundle.
  • An incomplete listing deletes nothing at all.

Needs before merge

Before it runs

  • Create the QStash schedule 0 */6 * * * for /api/cloud-workspaces/jobs/reap-retention in Upstash.
  • Run the backfill dry, then decide whether to run --apply.

Checks

  • bun test: packages/trpc 1119 pass, 0 fail; packages/sandbox 82 pass, 0 fail.
  • Biome is clean. tsc passes for trpc, api and sandbox.
  • New or changed tests:
    • cleanup job: one per rule row (asleep, archived, failed, stuck creating), the drop-out property, a wake that wins, a running box kept, the run budget, production only, the fail-closed listing, rule 2 (archived and live, already gone or shared), rule 3 (boxes, drives, agents drives, no current release);
    • listWhole; the production-only delete;
    • archive deletes nothing; the reap delete stage is a no-op; the unarchive refusal;
    • a failed create only stops; environment archive deletes nothing;
    • the retire write carries the status.
  • The model's run.sh (docs(sandbox): TLA+ spec of the end-state sandbox lifecycle rules #10006 at dac36d1, run unchanged): all 20 configs give their expected result, 8 pass and 12 fail the property they are meant to fail; exit 0. No rule's meaning moved. The code is stricter only in keeps the model does not represent (listed above), so the model was not edited.
  • Not run: anything against real Vercel.

New workspace boxes no longer expire their snapshots on Vercel's clock. A
daily QStash job (reap-retention) fails a workspace asleep and unused for 30
days as snapshot_expired, the state the 410 path gives, retires its box name
in the same update, and queues a retried box teardown. In production it also
tears down tagged boxes no row or unarchived environment names, 7 days old,
one delete a second. Counts go to logs, Sentry logs and a cron monitor.

The teardown is box-only with #9847's route and payload, so its drive
version replaces it. A dry-run-by-default backfill sets snapshotExpiration 0
on existing boxes (not run).
@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Preview Deployment

🔗 Preview Links

Service Status Link
Neon Database (Neon) ✅ View Branch
Vercel API (Vercel) ✅ Open Preview
Vercel Web (Vercel) ✅ Open Preview
Vercel Agents Web (Vercel) ➖ Not set up
Vercel Marketing (Vercel) ✅ Open Preview
Vercel Docs (Vercel) ✅ Open Preview

Preview updates automatically with new commits

@expo

expo Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Preview update published for development build 0dd268df (fingerprint 08a5e03ce5fb). Open the build on your phone and scan:

Update group 48cfddd3

…batch

Four runs of 200 clear 800 a day: production made a median 156 workspaces
a day and at most 725. A run stops starting work after 200 s and reports
the rest as deferred, so a backlog takes several runs inside the route's
300 s.
With snapshotExpiration 0, Vercel no longer cleans up after a lost archive
delete. In production each run lists every box once. An archived row past
grace + 1 day whose box is still listed is deleted through the archive's own
delete stage, or through a new guard (archived, no deleted_at, updated_at
older) for rows archived before deleted_at existed. At most 20 a run,
counted as found / alreadyDeleted / deleted / skipped / deferred / failed.
#9847's seam

The idle stage takes the box name off the row with retireAndTearDown(row,
condition), the name, signature and module #9847 uses, then fails the row.
Every box, with a drive or not, goes through the retried teardown job, and
the run paces teardowns in its loop since the seam takes no delay.
saddlepaddle added a commit that referenced this pull request Nov 19, 2026
markSandboxUnavailable no longer retires a drive workspace's names and
tears its box and drive down on an expired snapshot. As on main, it only
moves the row to failed (snapshot_expired); the cleanup job's rule 1
(#9987) deletes a failed workspace's box, snapshots and drive 30 days
later. One test replaces the four that covered the teardown.
saddlepaddle added a commit that referenced this pull request Nov 19, 2026
DISABLE_AUTOUPDATER, OPENCODE_DISABLE_AUTOUPDATE and PI_SKIP_VERSION_CHECK
were only in the login profile. A chat agent is launched from
host-service's login-shell snapshot, which falls back to host-service's
own env when the probe fails, and that env is the env -i list in
superset-boot. The three now ride in that list too.

The plan's retention decision names #9987's reaper, which calls
retireAndTearDown, and its Before-customers item goes.
saddlepaddle added a commit that referenced this pull request Nov 19, 2026
…b's rule 3

Removes the drive-sweep workflow_dispatch job, its input, the package
script, drive-sweep.ts and its test. A drive no row names falls to rule 3
of the cleanup job (#9987), per the sandbox lifecycle end state.
saddlepaddle added a commit that referenced this pull request Nov 19, 2026
markSandboxUnavailable no longer retires a drive workspace's names and
tears its box and drive down on an expired snapshot. As on main, it only
moves the row to failed (snapshot_expired); the cleanup job's rule 1
(#9987) deletes a failed workspace's box, snapshots and drive 30 days
later. One test replaces the four that covered the teardown.
saddlepaddle added a commit that referenced this pull request Nov 19, 2026
A failed create no longer clears the row's drive name or hands the drive
to the teardown. The row keeps naming it, and the box's teardown runs with
keepDrive, so only the box goes, as on main. The cleanup job's rule 1
(#9987) deletes the failed row's drive with it after 30 days; #9987 also
owns turning main's failed-create box delete into a status. forkedDrive
is gone: a forked and a found drive are now treated the same.
…e rules

One rules table, production only, failing closed on an incomplete Vercel
listing. Rule 1 retires workspaces past their wait (asleep 30 d, archived
7 d, failed 30 d, stuck creating 1 d) with one conditional write, then the
retried teardown. Rule 2 deletes archived environments' goldens and, through
goldensToDelete (as #10000), goldens past the newest 3. Rule 3 deletes boxes,
drives and goldens no production row names after 7 days, and agents drives
that are not current and no box mounts (#9943's listing checks).

deleteSandbox and deleteDrive delete only when VERCEL_ENV is production.
Archive, a failed create, the provision race and environment archive only
set a status; archive no longer queues a delayed delete, and an unarchive
after 7 days is refused. The release no longer prunes agents drives.
@saddlepaddle saddlepaddle changed the title feat(cloud-workspace): own workspace retention with a daily reaper (draft) feat(cloud-workspace): cleanup job applies the end-state lifecycle rules (draft) Nov 19, 2026
…eletesAllowed

Same file and signature as #9847 (038f603): teardownSandbox returns
skipped outside production before it stops anything, and deleteDrive and a
replaced golden's delete check it too. deleteSandbox itself stays unchecked
so the release probe and real-sandbox still delete their own temporary
boxes. The test preload sets VERCEL_ENV=production.
reapCloudWorkspaceRetention takes dryRun: it reads everything, listing
checks included, and replaces every writing or deleting dependency
(MUTATING_DEPS) with one that only records, logging "would delete".
scripts/cleanup-dry-run.ts runs it against a real database and the real
Vercel listing: it refuses any role but readonly, sets VERCEL_ENV to
development, and makes the SDK's stop, update, create and delete throw.

This branch was previously deployed

1 inactive deployment
preview — 4012fd86 Deployed Nov 19, 2026 by saddlepaddle via Deploy Database (Neon) #15321
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant