Repository navigation
fix(sandbox): drop on-demand swap from memguard (draft: trade-off for review) - #9843
Closed
saddlepaddle wants to merge 1 commit into
Closed
saddlepaddle wants to merge 1 commit into
saddlepaddle wants to merge 1 commit into
Conversation
A snapshot is a full block copy of the disk with no discard, so swap memguard writes stays in every later snapshot of the workspace. memguard now reaps on MemAvailable below the floor alone, and still removes any /var/swap/superset-* an earlier bundle left.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Contributor
🚀 Preview Deployment🔗 Preview Links
Preview updates automatically with new commits |
Member
Author
|
Closing: swap stays for now. A full typecheck of this repo on an 8 GB box needs it (measured: swap grew to the 12 GB cap in about a minute, and one run was OOM-killed even with it). The clean fix is swap on the workspace drive (#9847), which keeps it out of snapshots. |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft: a human decides whether this ships. It trades speed on undersized boxes for smaller snapshots.
What changed
superset-memguard: removedgrow_swap/shrink_swap, the swap constants (SWAP_DIR,SWAP_STEP_MB,SWAP_IDLE_S),box_mband the swap cap, and theSwapTotal/SwapFreereads.MemAvailable < floor && SwapFree < SwapTotal/10 + 1. It is nowMemAvailable < floor(8% of the box). The old swap check was there to wait for swap to fill. With no swap,MemAvailablealone is the right signal. Ranking, victim choice (500 → 0 → −300), the cooldown and the log lines did not change./var/swap/superset-*an older bundle left and removes it.superset-bootstill removes the old/swapfile.boot.test.ts: the swap-cap test is replaced with two tests. Nothing in boot or memguard turns swap on, and the stale-file cleanup and the boot/swapfileremoval are still there. memguard reaps on available memory alone.docs/cloud-sandbox-considerations.md: the "Swap is on demand" entry is replaced by "No swap", which keeps the history and records the open telemetry question.The trade
reap pid=… rss=… available=… floor=…line inmemguard.log, and the app's pressure banner shows it. The remedy is a larger machine size.bun run typecheckof the monorepo made memguard grow swap 0 → 12 GB (its cap) in about 60 s.freeshowed 4.3 GB of swap in use afterwards, so at least that much is now written to this workspace's disk and every later snapshot will copy it. Under this change, the same typecheck on the same box would probably have had its largesttscreaped. Earlier measurements (2026-10-22, 2/4 box): theapps/desktoptypecheck needed about 8 GB and finished in 86 s only because of swap. Expect it to be reaped on a 2/4 box now.Alternatives not taken
What would settle it
How many boxes ever grow swap, and how often swap is what saved a job.
memguard.logis box-local today, so we do not know. Shipping theswap.growcount (or the peak swap use) through host-service's existing pressure report (runtime/sandbox-box-pressure) to telemetry for a week, before or instead of merging, would answer it.Checks
Ran:
bun run lint:fix: passed (it reformatted the new test).bun run typecheck(whole monorepo, 42 tasks): passed.bun testinpackages/sandbox: 25 pass, 0 fail.bash -n superset-memguard: passed.bun run --cwd packages/sandbox image --local(Docker started on this box): builtsuperset-sandbox:local.src/runner-check.ts: all expectations met.src/boot-twice.ts: memguard starts (memguard.start,memguard.pidpresent), and the boot, host-service and second-boot skip expectations pass. 3 expectations failed:first boot: checkout done,start hook: the runner asked host-service once the push landed,second boot: checkout marker respected. They come fromcheckout.clone.failed: the container cannot clone from GitHub here (git ls-remotein the image fails with "server certificate verification failed"; this workspace's egress needs a CA the container does not have). memguard is not involved, but I did not run boot-twice onmainin this environment to confirm the same failures happen there./var/swap/superset-0and ran memguard for 3 s. It loggedstartand removed the file.Not run:
Summary by cubic
Drops on-demand swap from memguard so snapshots stop permanently containing swap data. memguard now reaps when available memory drops below the 8% floor instead of growing swapfiles first, because snapshots are full block copies with no discard: swap written once (up to 16 GB per box) was being copied into every later snapshot of the workspace.
The trade-off
reapline inmemguard.log; the remedy is a larger machine size./var/swap/superset-*from older bundles, and boot still removes the old/swapfile.Alternatives not taken
How often swap actually saved a job is unknown because
memguard.logis box-local; reportingswap.growcounts through host-service's pressure report for a week would settle it.Written for commit 1718597. Summary will update on new commits.