Repository navigation
feat(cloud-workspace): cleanup job applies the end-state lifecycle rules (draft) - #9987
Draft
saddlepaddle wants to merge 8 commits into
Draft
saddlepaddle wants to merge 8 commits into
saddlepaddle wants to merge 8 commits into
Conversation
New workspace boxes no longer expire their snapshots on Vercel's clock. A daily QStash job (reap-retention) fails a workspace asleep and unused for 30 days as snapshot_expired, the state the 410 path gives, retires its box name in the same update, and queues a retried box teardown. In production it also tears down tagged boxes no row or unarchived environment names, 7 days old, one delete a second. Counts go to logs, Sentry logs and a cron monitor. The teardown is box-only with #9847's route and payload, so its drive version replaces it. A dry-run-by-default backfill sets snapshotExpiration 0 on existing boxes (not run).
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Contributor
🚀 Preview Deployment🔗 Preview Links
Preview updates automatically with new commits |
|
Preview update published for development build |
…batch Four runs of 200 clear 800 a day: production made a median 156 workspaces a day and at most 725. A run stops starting work after 200 s and reports the rest as deferred, so a backlog takes several runs inside the route's 300 s.
With snapshotExpiration 0, Vercel no longer cleans up after a lost archive delete. In production each run lists every box once. An archived row past grace + 1 day whose box is still listed is deleted through the archive's own delete stage, or through a new guard (archived, no deleted_at, updated_at older) for rows archived before deleted_at existed. At most 20 a run, counted as found / alreadyDeleted / deleted / skipped / deferred / failed.
#9847's seam The idle stage takes the box name off the row with retireAndTearDown(row, condition), the name, signature and module #9847 uses, then fails the row. Every box, with a drive or not, goes through the retried teardown job, and the run paces teardowns in its loop since the seam takes no delay.
saddlepaddle
added a commit
that referenced
this pull request
Nov 19, 2026
markSandboxUnavailable no longer retires a drive workspace's names and tears its box and drive down on an expired snapshot. As on main, it only moves the row to failed (snapshot_expired); the cleanup job's rule 1 (#9987) deletes a failed workspace's box, snapshots and drive 30 days later. One test replaces the four that covered the teardown.
saddlepaddle
added a commit
that referenced
this pull request
Nov 19, 2026
DISABLE_AUTOUPDATER, OPENCODE_DISABLE_AUTOUPDATE and PI_SKIP_VERSION_CHECK were only in the login profile. A chat agent is launched from host-service's login-shell snapshot, which falls back to host-service's own env when the probe fails, and that env is the env -i list in superset-boot. The three now ride in that list too. The plan's retention decision names #9987's reaper, which calls retireAndTearDown, and its Before-customers item goes.
saddlepaddle
added a commit
that referenced
this pull request
Nov 19, 2026
…b's rule 3 Removes the drive-sweep workflow_dispatch job, its input, the package script, drive-sweep.ts and its test. A drive no row names falls to rule 3 of the cleanup job (#9987), per the sandbox lifecycle end state.
saddlepaddle
added a commit
that referenced
this pull request
Nov 19, 2026
markSandboxUnavailable no longer retires a drive workspace's names and tears its box and drive down on an expired snapshot. As on main, it only moves the row to failed (snapshot_expired); the cleanup job's rule 1 (#9987) deletes a failed workspace's box, snapshots and drive 30 days later. One test replaces the four that covered the teardown.
saddlepaddle
added a commit
that referenced
this pull request
Nov 19, 2026
A failed create no longer clears the row's drive name or hands the drive to the teardown. The row keeps naming it, and the box's teardown runs with keepDrive, so only the box goes, as on main. The cleanup job's rule 1 (#9987) deletes the failed row's drive with it after 30 days; #9987 also owns turning main's failed-create box delete into a status. forkedDrive is gone: a forked and a found drive are now treated the same.
…e rules One rules table, production only, failing closed on an incomplete Vercel listing. Rule 1 retires workspaces past their wait (asleep 30 d, archived 7 d, failed 30 d, stuck creating 1 d) with one conditional write, then the retried teardown. Rule 2 deletes archived environments' goldens and, through goldensToDelete (as #10000), goldens past the newest 3. Rule 3 deletes boxes, drives and goldens no production row names after 7 days, and agents drives that are not current and no box mounts (#9943's listing checks). deleteSandbox and deleteDrive delete only when VERCEL_ENV is production. Archive, a failed create, the provision race and environment archive only set a status; archive no longer queues a delayed delete, and an unarchive after 7 days is refused. The release no longer prunes agents drives.
…eletesAllowed Same file and signature as #9847 (038f603): teardownSandbox returns skipped outside production before it stops anything, and deleteDrive and a replaced golden's delete check it too. deleteSandbox itself stays unchecked so the release probe and real-sandbox still delete their own temporary boxes. The test preload sets VERCEL_ENV=production.
reapCloudWorkspaceRetention takes dryRun: it reads everything, listing checks included, and replaces every writing or deleting dependency (MUTATING_DEPS) with one that only records, logging "would delete". scripts/cleanup-dry-run.ts runs it against a real database and the real Vercel listing: it refuses any role but readonly, sets VERCEL_ENV to development, and makes the SDK's stop, update, create and delete throw.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, for review. Spec: the page "Cloud sandbox lifecycle: the end state"; model: #10006 (
.agents/skills/verify-sandbox-lifecycle). Plan:plans/done/20261118-1500-cloud-workspace-retention-reaper.md.What it does
The cleanup job (
reap-retention, every 6 hours,0 */6 * * *) is now the only thing that deletes for age or for being unused. It runs only in production and applies one rules table:goldensToDelete, the same as #10000's.Every keep and delete is logged as
[cleanup] {rule, kind, name, action, reason}. Batches: 200 retires, 20 goldens, 200 leftover boxes, 50 drives. A run stops starting work after 200 s. Counts go to the run log, aretention.runSentry log line, and thereap-retentioncron monitor.Fails closed. The job lists every box and every drive first, with #9943's checks: every page answered, no cursor seen twice, a non-empty box listing, and a few boxes running now present in it. If any check fails, it deletes nothing and sends a Sentry warning.
Mechanics
Only production deletes. One check,
sandboxDeletesAllowedinlib/sandbox/deletes.ts, the same file and signature as feat(cloud-workspace): workspace drive per environment, flag-gated (draft) #9847's 038f603. It runs:teardownSandbox, which returnsskipped; that is every box and golden the cleanup job deletes;deleteDrive;Outside production it logs "would delete" and the caller stops. Raw
deleteSandboxstays unchecked, so the release probe and real-sandbox still delete the temporary boxes they make. Staging deploys withvercel deploy --prodon the production database, so itsVERCEL_ENVisproductionand it counts. The trpc test preload setsVERCEL_ENV=production, as on feat(cloud-workspace): workspace drive per environment, flag-gated (draft) #9847.Everything else only sets a status:
deletemessage already in QStash from before this change is a no-op.An unarchive after 7 days is refused (
{ unarchived: false }), even before the job has run.The release no longer prunes agents drives. Rule 3 does it.
snapshotExpiration: 0for new boxes stays. The backfill script stays, to be run by hand.Old stages dropped
Where the code is stricter than the model (all extra keeps)
Needs before merge
drive_nameandgolden_drive_nameto rule 3'snamedDrives. Today it is empty, because main has no drive columns;row.driveNamein rule 1, withdrive_name = <read>in the retire condition;drive_nameis not null;retireAndTearDowngains an optional third argumentset(the status written in the same write);liveReferences;nextSandboxNameForfrom./sandbox-name.undeletedGoldensandmarkGoldenDeletedin rule 2, and itsgoldensToDeleteinstead ofgolden-rotation.ts. On main an environment records only its active golden, so rule 2 on live environments is a no-op until then.listWhole.Before it runs
0 */6 * * *for/api/cloud-workspaces/jobs/reap-retentionin Upstash.--apply.Checks
bun test: packages/trpc 1119 pass, 0 fail; packages/sandbox 82 pass, 0 fail.tscpasses for trpc, api and sandbox.listWhole; the production-only delete;run.sh(docs(sandbox): TLA+ spec of the end-state sandbox lifecycle rules #10006 at dac36d1, run unchanged): all 20 configs give their expected result, 8 pass and 12 fail the property they are meant to fail; exit 0. No rule's meaning moved. The code is stricter only in keeps the model does not represent (listed above), so the model was not edited.