Repository navigation
feat(cloud-workspace): workspace drive per environment, flag-gated (draft) - #9847
Draft
saddlepaddle wants to merge 10 commits into
Draft
saddlepaddle wants to merge 10 commits into
saddlepaddle wants to merge 10 commits into
Conversation
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Contributor
🚀 Preview Deployment🔗 Preview Links
Preview updates automatically with new commits |
|
No installed development build can load this pull request as an update: its native fingerprint is |
This was referenced Nov 16, 2026
saddlepaddle
force-pushed
the
superset/thin-workspace-drive-draft-c4ad5f56
branch
from
November 17, 2026 02:40
6cbde6d to
cf0d1a5
Compare
saddlepaddle
force-pushed
the
superset/thin-workspace-drive-draft-c4ad5f56
branch
from
November 17, 2026 03:11
cf0d1a5 to
b68a549
Compare
This was referenced Nov 18, 2026
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
Adds three nullable columns, no defaults, nothing reads them yet: - environments.golden_drive_name: the Vercel drive an environment's workspace drives fork from (0196). - cloud_workspaces.drive_name: the drive mounted on every session of a workspace's box (0196). - cloud_workspaces.drive_laid_out_at: when a boot first answered with the drive laid out (0197). Split from #9847 so the migrations land before the code that uses them.
saddlepaddle
force-pushed
the
superset/workspace-drive-columns
branch
from
November 18, 2026 03:01
99b0fd8 to
8bec0b7
Compare
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
Makes what #9847 now enforces into code guards with file and line (the drive-keyed resume lock, Session-only use of a running box, the create's drive wait, onBox transitions, the lost job's keepDrive teardown, the any-row teardown guard, the sweep's named-drive rule, state/fresh cleared on attempt), models the provision's lock timeout, the drive wait under the lock, duplicate deliveries and a failed first fetch. Seven rules hold; two real bugs and the fresh-fork trade remain, each with a fix. Reviews the model patch on superset/drive-model-9847-local-patch: takes what the code does, rejects the narrowed head_of. SKILL.md: a product change must not edit the model in the same PR.
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
Adds three nullable columns, no defaults, nothing reads them yet: - environments.golden_drive_name: the Vercel drive an environment's workspace drives fork from. - cloud_workspaces.drive_name: the drive mounted on every session of a workspace's box. - cloud_workspaces.drive_laid_out_at: when a boot first answered with the drive laid out. One migration, 0197, generated by drizzle-kit on main after #9933's 0196. Split from #9847 so the migration lands before the code that uses it.
saddlepaddle
force-pushed
the
superset/workspace-drive-columns
branch
from
November 18, 2026 03:14
8bec0b7 to
7b454fc
Compare
saddlepaddle
force-pushed
the
superset/thin-workspace-drive-draft-c4ad5f56
branch
from
November 18, 2026 03:48
3f7b5ec to
1a96ee6
Compare
6 tasks
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
The workspace drive work of #9847 as one commit on main after 78a831b (#9915, migration 0199), resolved by hand against main's M5b failure reasons, M7s logging, M8 metering, M9 non-resuming handles, and the agent-tools drive. The previous 33 commits are kept locally as backup/9847-pre-rebase. Merged with main: - wakeSandbox keeps the resume lock, the drive detach, the composed mounts and withLiveSession, and takes main's onlyIfRunning and its non-resuming handles; a session withLiveSession resumes mid-wake is reported to beforeResume as resumedSession before it boots, so it is metered as M9 meters it. - promoteSandboxToEnvironment takes main's shape: the snapshot ends the source's session and the caller's wakeAfterStop starts it again; the drive fork and the golden drive's strip stay. - provision fails through failCloudWorkspace, which now takes onBox and answers whether it moved the row, so a provision still fails only its own row. - A delete or a failed create still retires the names (retireAndTearDown).
A drive refused for good fails the workspace as host_not_ready ("The
workspace did not become ready."), not sandbox_gone. A drive still
flushing at a wake is a SandboxNotReadyError ("still starting"). The
errors a client can be handed carry the drive in fields, console lines
and Sentry extra, not in the message. A promote of a workspace on a
drive that fails, or is refused with drives off, says only "This
workspace could not be saved as an environment. Try again later.",
translated in every locale.
The daily expiry probe is dropped: retention moves to its own PR, whose
reaper calls retireAndTearDown, as the 410 path here does.
DISABLE_AUTOUPDATER, OPENCODE_DISABLE_AUTOUPDATE and PI_SKIP_VERSION_CHECK were only in the login profile. A chat agent is launched from host-service's login-shell snapshot, which falls back to host-service's own env when the probe fails, and that env is the env -i list in superset-boot. The three now ride in that list too. The plan's retention decision names #9987's reaper, which calls retireAndTearDown, and its Before-customers item goes.
deleteSandboxAndDriveLater no longer deletes a box without a drive inline: every box, with or without a drive, goes to teardownSandbox, which re-reads what still names the box before the delete, and retries. The provision's failed and overtaken creates, the promote's abandoned golden and every retireAndTearDown caller take that path; a test per caller says so. nextSandboxNameFor is imported from ./sandbox-name everywhere; provision no longer re-exports it.
…b's rule 3 Removes the drive-sweep workflow_dispatch job, its input, the package script, drive-sweep.ts and its test. A drive no row names falls to rule 3 of the cleanup job (#9987), per the sandbox lifecycle end state.
sandboxDeletesAllowed (lib/sandbox/deletes.ts) passes only when VERCEL_ENV is production; elsewhere it logs "would delete ..." and the caller stops. teardownSandbox checks it before it stops anything and returns "skipped"; deleteWorkspaceDrive and deleteGoldenDriveIfUnreferenced check it before the Vercel call. Staging deploys with --prod, so it counts as production, as it shares production's database. deleteSandboxAndDrive, used only by the release probe and real-sandbox for the boxes and drives they make themselves, removes its drive directly, so those jobs still clean up after themselves wherever they run. The production release step sets VERCEL_ENV=production so it still deletes the previous internal golden's drive. The trpc test preload sets VERCEL_ENV=production; one test checks a teardown outside production touches nothing. The plan also records the decision, applied in the next commit, that a 410 only marks the workspace failed.
markSandboxUnavailable no longer retires a drive workspace's names and tears its box and drive down on an expired snapshot. As on main, it only moves the row to failed (snapshot_expired); the cleanup job's rule 1 (#9987) deletes a failed workspace's box, snapshots and drive 30 days later. One test replaces the four that covered the teardown.
REAL_SANDBOX_WAKE (the real-sandbox job's real_sandbox_wake input) runs wake-measure.ts instead of the checks: three boxes without a drive (A) and three with one (B), named d-expiry-wake-*, each with a full bun install and bun run typecheck of this repo. "prepare" measures the stop (time, snapshot size), a warm wake right after it (host-service ready, first login shell in /workspace, git status, the typecheck again, an uncached tsc of packages/trpc) and stops again. "cold <suffix>", dispatched at least two hours later, measures one more wake per box and deletes every box and drive; "delete <suffix>" only deletes. One WAKE-MEASURE json line per measure. runOn and stopMeasured move to module scope so both passes use them. The job's timeout goes to 150 minutes for the prepare pass.
A failed create no longer clears the row's drive name or hands the drive to the teardown. The row keeps naming it, and the box's teardown runs with keepDrive, so only the box goes, as on main. The cleanup job's rule 1 (#9987) deletes the failed row's drive with it after 30 days; #9987 also owns turning main's failed-create box delete into a status. forkedDrive is gone: a forked and a found drive are now treated the same.
…s workspace The cold run built its claim with a new random workspace id, so the boot refused every drive as drive.foreign (the drive belongs to the workspace that laid it out). The workspace and organization ids now derive from the run's suffix, the same in both runs. A cold run with a failed wake keeps every box for diagnosis; "delete <suffix>" removes them.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Stacked on #9915 (the database columns), which lands first. Detail:
plans/20261116-workspace-drive.md.What it does
cloud-workspace-drivesflag is on and a release builds that environment's template with a drive. Every other workspace is untouched.How it works
What a customer sees
Evidence
Real Vercel,
REAL_SANDBOX_DRIVE=1: the same work on a drive box and a plain box (clone,bun install, 4 GB written), five persistence sentinels, an unclean stop, and a workspace created from a drive-mode template.The wake times are warm: a wake that the snapshot cache does not serve cannot be forced from the check.
The TLA+ model of the drive lifecycle (#9914) checks 10 rules: nothing lost, never empty shown as work, one writer per drive, no teardown of a live workspace, no change without a drive, never stuck, a fresh fork fetches, a glitch is retried, no drive left behind, only create gives a drive. Its author re-derived it from this code at 3f7b5ec, where 7 held. This head adds the three fixes the model proposed for the other three (FixFailOwnOnly, FixProvLockRetry, FixFetchIfUntouched), each with a unit test or boot-twice case that plays its trace.
./run.sh(the model's design with those fixes on): every scenario passes../run.sh head: still the 3f7b5ec code, so those three rules fail until the model is re-derived from this head.Unit tests replay each trace the model found. boot-twice covers the box side in Docker.
Risks and rollback
cloud-workspace-drivesoff. New workspaces then get no drive, promote refuses a drive workspace, and the next release builds the template without one. Workspaces that already have a drive keep it and keep working. The full procedure is in the plan, under "Rollback"./workspace. Archive them before such a revert.Before internal enable
run.sh headholds on the final head.Before customers
docs/cloud-sandbox-considerations.md.[install](feat(environments): build cloud environments from [install] (SUPER-3017) #9282) must setsource_refandgolden_drive_nametogether.Decisions that are yours
failedwith the reason; its box and drive are kept. Keep 3?Snapshot expiry deletes the drive
Retention will be ours, in #9987: its reaper, every 6 h, deletes an expired workspace's box, snapshots and drive, instead of relying on Vercel's 30-day
snapshotExpiration.This PR keeps the fallback. When a wake, an agent turn or an automation dispatch gets a 410 for an expired snapshot,
markSandboxUnavailablecallsretireAndTearDown. That function retires the row's box and drive names, only while the row isfailedand still names that box and drive. It then queues the retried teardown, which waits for the drive to detach and never deletes a drive that a row or a golden environment names. The new reaper calls the same function:Checks on this head
This head is #9847 rebased on main (e0f54fa) through #9915 (now
0199), squashed into one commit and resolved by hand. The earlier 33 commits are kept locally asbackup/9847-pre-rebase.Ran:
lint:fix.check:i18n: the new message is translated in every locale.image --local, and boot-twice withBOOT_TWICE_CA_FILE: all expectations met. This includes "host-service's own env turns the agents' self-updates off, whatever its shell probe finds":DISABLE_AUTOUPDATER,OPENCODE_DISABLE_AUTOUPDATEandPI_SKIP_VERSION_CHECKare now also in the envsuperset-bootstarts host-service with. A chat agent gets that env when host-service's login-shell probe fails../run.sh head, results in the coordinator thread.Not run: