Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions .github/CI.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,7 @@ Database integration uses 4-vCPU Blacksmith runners with sticky dependency disks

GitHub runners install dependencies directly rather than transferring Bun and `node_modules` archives. In a same-source paired comparison, those restores took 26.2 seconds before an 82-millisecond install, while a clean install took 12.42 seconds. This is one paired observation; retain the policy only while repeated CI timings support it. Blacksmith keeps its sticky dependency disks. GitHub Turbo output caches remain enabled for jobs that request them, with a commit-specific key and a task/event/fork-scoped restore prefix, so a rerun can reuse an exact entry without creating another archive. Push jobs never restore pull-request entries.

Trusted Blacksmith E2E builders can reuse Next's incremental compiler state when a complete Turbo build is not already cached. The compiler cache has its own mode, event, OS/architecture, resolved toolchain, lockfile and build-recipe namespace; fork PRs and GitHub fallback keep the compiler cache off. The build command always runs, and an uncertain cache probe restores state rather than skipping compilation.

Compiler maintenance enforces a 20 GiB regular-file working-state limit before and after compilation, preserves the mounted root, and fails closed on access or ownership errors. Dev maintenance remains separate and nonblocking. Sticky disks are optional last-good compiler snapshots, not cross-job artifact handoff or validation receipts; no freshest-snapshot or physical-storage eviction guarantee is assumed. Hosted/deployment builds remain cache-off until separately qualified. A cold E2E qualification job took 259 seconds versus 225 uncached; fresh-runner warm and changed-source jobs took 148 and 159 seconds versus 227 and 229 uncached. These paired observations do not establish a universal CI speedup or justify smaller runners. Compilation and all existing acceptance coverage remain required.
E2E builds compile without Next's persistent compiler cache (`turbopackFileSystemCacheForBuild: false`; Next enables it by default). Its earlier qualification showed warm, identical-source builds at 148 seconds, but real PR compiles restoring a shared cache took 2.9-4.6 minutes plus 15-37 seconds writing the cache back, while the same builds without it compiled in 2.5 minutes with no write. A complete Turbo build output is still reused when nothing it depends on changed. `scripts/prune-turbopack-cache.mts` caps only the dev cache.

Closing a same-repository pull request deletes only its GitHub merge-ref caches after verifying that it remains closed or merged. The regular `pull_request` workflow skips forks and Dependabot because their tokens cannot delete caches; those entries retain GitHub's normal cache eviction. Cleanup checks out no code and never selects a branch-wide or default-branch cache. Blacksmith sticky disks retain their separate lifecycle.

Expand Down
41 changes: 3 additions & 38 deletions .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1014,7 +1014,7 @@ jobs:
TOTAL_GB=$(awk '/MemTotal/ {printf "%d", $2/1048576}' /proc/meminfo)
echo "Runner memory: ${TOTAL_GB} GB"
if [ "$TOTAL_GB" -lt 40 ]; then
echo "::warning::Runner has ${TOTAL_GB} GB. A cold-cache build peaks ~51 GB, so this run may be OOM-killed (reported only as 'the runner has received a shutdown signal'). Warm/partial builds should still fit."
echo "::warning::Runner has ${TOTAL_GB} GB. A build peaks ~51 GB, so this run may be OOM-killed (reported only as 'the runner has received a shutdown signal')."
fi

- name: Build application
Expand Down Expand Up @@ -1043,9 +1043,6 @@ jobs:
if: needs.changes.outputs.e2e_build == 'true'
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-16vcpu-ubuntu-2404' || 'linux-x64-8-core' }}
timeout-minutes: 25
env:
NEXT_BUILD_PERSISTENT_CACHE: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && (github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork) && 'true' || 'false' }}

steps:
- *checkout-mirror
- *checkout-plain
Expand All @@ -1064,10 +1061,8 @@ jobs:
echo "::warning::Runner has ${TOTAL_GB} GB. A cold-cache build peaks ~51 GB, so this run may be OOM-killed (reported only as 'the runner has received a shutdown signal'). Warm/partial builds should still fit."
fi

- name: Plan incremental compiler state
id: compiler-cache-plan
if: env.NEXT_BUILD_PERSISTENT_CACHE == 'true'
env: &e2e-build-env
- name: Build application
Comment thread
waleedlatif1 marked this conversation as resolved.
Resolved
env:
NODE_OPTIONS: '--no-warnings --max-old-space-size=8192'
NEXT_PUBLIC_APP_URL: 'http://localhost:3000'
DATABASE_URL: 'postgresql://postgres:postgres@localhost:5432/simstudio'
Expand All @@ -1082,38 +1077,8 @@ jobs:
# already enforce.
NEXT_BUILD_SKIP_TYPE_CHECK: 'true'
NEXT_BUILD_SKIP_SOURCE_MAPS: 'true'
run: |
echo "toolchain=bun$(bun --version)-node$(node --version)" >> "$GITHUB_OUTPUT"
if bunx turbo run build --filter=@sim/app --dry=json > "$RUNNER_TEMP/e2e-build-plan.json" &&
jq -e 'type == "object" and (.tasks | type) == "array" and ([.tasks[] | select(.taskId == "@sim/app#build")] | length == 1 and .[0].cache.status == "HIT")' "$RUNNER_TEMP/e2e-build-plan.json" > /dev/null; then
echo 'restore=false' >> "$GITHUB_OUTPUT"
else
echo 'restore=true' >> "$GITHUB_OUTPUT"
fi

- name: Mount incremental compiler state
id: compiler-cache
if: steps.compiler-cache-plan.outputs.restore == 'true'
uses: ./.github/actions/cache
with:
provider: ${{ vars.CI_PROVIDER }}
key: ${{ github.repository }}-next-build-e2e-v1-${{ runner.os }}-${{ runner.arch }}-${{ github.event_name }}-${{ steps.compiler-cache-plan.outputs.toolchain }}-${{ hashFiles('bun.lock', 'package.json', 'patches/**', 'apps/sim/package.json', 'apps/sim/next.config.ts', 'apps/sim/lib/core/config/env.ts', '.github/workflows/checks.yml', '.github/actions/setup/action.yml') }}
path: ./apps/sim/.next/cache

- name: Bound restored compiler state
if: steps.compiler-cache.outcome == 'success'
working-directory: apps/sim
run: node ../../scripts/prune-turbopack-cache.mts --build

- name: Build application
env: *e2e-build-env
run: bunx turbo run build --filter=@sim/app

- name: Bound compiler state before persistence
if: always() && !cancelled() && steps.compiler-cache.outcome == 'success'
working-directory: apps/sim
run: node ../../scripts/prune-turbopack-cache.mts --build

# Everything `next start` reads, including the spreadsheet parse worker it spawns (named, so a
# missing bundle fails packaging as it fails docker/app.Dockerfile); node_modules and the source
# tree come from e2e-production's own checkout and setup, at the same workspace path and
Expand Down
1 change: 0 additions & 1 deletion apps/sim/lib/core/config/env.ts
Original file line number Diff line number Diff line change
Expand Up @@ -349,7 +349,6 @@ export const env = createEnv({
// Infrastructure & Deployment
NEXT_RUNTIME: z.string().optional(), // Next.js runtime environment
DOCKER_BUILD: z.boolean().optional(), // Flag indicating Docker build environment
NEXT_BUILD_PERSISTENT_CACHE: z.boolean().optional(), // Enables incremental compiler state for qualified CI builds
NEXT_BUILD_SKIP_SOURCE_MAPS: z.boolean().optional(), // Omits unused browser source maps in acceptance builds
NEXT_BUILD_SKIP_TYPE_CHECK: z.boolean().optional(), // Skips `next build`'s TypeScript pass where another CI job type-checks

Expand Down
7 changes: 3 additions & 4 deletions apps/sim/next.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -166,11 +166,10 @@ const nextConfig: NextConfig = {
*/
turbopackFileSystemCacheForDev: true,
/**
* Only qualified CI builds opt into persistent compiler state. Other builds
* remain cache-off until their own memory, invalidation and restore costs are
* qualified. CI keeps this state separate from complete build-output caches.
* Off, explicitly: Next enables it by default, and a CI build compiles code its cache has
* never seen, so it only pays for writing the cache back (15-37s) on a disk that is discarded.
*/
turbopackFileSystemCacheForBuild: isTruthy(env.NEXT_BUILD_PERSISTENT_CACHE),
turbopackFileSystemCacheForBuild: false,
/**
* TypeScript 7 ships no JavaScript compiler API until 7.1, so Next's default
* checker cannot load it — this shells out to the project-local `tsc` instead.
Expand Down
6 changes: 1 addition & 5 deletions apps/sim/turbo.json
Original file line number Diff line number Diff line change
Expand Up @@ -220,11 +220,7 @@
]
},
"build": {
"env": [
"NEXT_BUILD_PERSISTENT_CACHE",
"NEXT_BUILD_SKIP_SOURCE_MAPS",
"NEXT_BUILD_SKIP_TYPE_CHECK"
],
"env": ["NEXT_BUILD_SKIP_SOURCE_MAPS", "NEXT_BUILD_SKIP_TYPE_CHECK"],
"inputs": [
"$TURBO_DEFAULT$",
".env*",
Expand Down
71 changes: 16 additions & 55 deletions scripts/prune-turbopack-cache.mts
Original file line number Diff line number Diff line change
@@ -1,16 +1,12 @@
#!/usr/bin/env node
/**
* Caps each app's Turbopack cache from its working directory.
* Caps an app's Turbopack dev cache (`.next/dev/cache/turbopack`) from its working directory.
*
* Dev maintenance is best effort and never blocks startup. `--build` selects
* `.next/cache`, preserves its mounted root, and fails on maintenance errors;
* run it only between compiler processes. Oversized state is discarded so the
* next compiler starts cold. Child symlinks are neither counted nor followed.
* Best effort: it never blocks `next dev` startup. Oversized state is discarded so the next start
* compiles cold. Child symlinks are neither counted nor followed; a symlinked cache root is unlinked
* without touching its target.
*
* Build maintenance uses Node's bounded directory iterator and stops counting
* above the cap. Dev commands keep Bun and its existing best-effort behavior.
*
* CI build maintenance runs with Node 24: `node scripts/prune-turbopack-cache.mts [--build] [--force] [--dry-run]`.
* Usage: `bun run scripts/prune-turbopack-cache.mts [--force] [--dry-run]`.
* `SIM_TURBOPACK_CACHE_MAX_GB` overrides the default 20 GiB cap.
*/
import { existsSync, lstatSync, statSync } from 'node:fs'
Expand All @@ -19,43 +15,26 @@ import path from 'node:path'

const DEFAULT_MAX_GB = 20
const BYTES_PER_GB = 1024 ** 3
const BUILD_CACHE = process.argv.includes('--build')

/**
* Each app owns its cache under its working directory. Build maintenance selects
* a separate root, so it cannot prune a live dev server's state.
*/
const CACHE_DIR = path.resolve(
process.cwd(),
BUILD_CACHE ? '.next/cache' : '.next/dev/cache/turbopack'
)
/** Each app owns its dev cache under its working directory. */
const CACHE_DIR = path.resolve(process.cwd(), '.next/dev/cache/turbopack')

function isMissingPath(error: unknown): boolean {
return error instanceof Error && 'code' in error && error.code === 'ENOENT'
}

/** Counts regular files without following symlinks or retaining the tree in memory. */
async function directorySize(dir: string, limit: number): Promise<number> {
async function directorySize(dir: string): Promise<number> {
let total = 0
let entries: Awaited<ReturnType<typeof opendir>>
try {
entries = await opendir(dir, { bufferSize: 32 })
} catch (error) {
if (BUILD_CACHE && isMissingPath(error)) return 0
throw error
}
for await (const entry of entries) {
for await (const entry of await opendir(dir, { bufferSize: 32 })) {
const entryPath = path.join(dir, entry.name)
if (entry.isDirectory()) {
total += await directorySize(entryPath, limit - total)
total += await directorySize(entryPath)
} else if (entry.isFile()) {
try {
total += (await stat(entryPath)).size
} catch (error) {
if (BUILD_CACHE && !isMissingPath(error)) throw error
}
} catch {}
}
if (total > limit) break
}
return total
}
Expand Down Expand Up @@ -92,28 +71,11 @@ function parseMaxGb(): number {
async function main() {
const force = process.argv.includes('--force')
const dryRun = process.argv.includes('--dry-run')

if (BUILD_CACHE) {
let root: ReturnType<typeof lstatSync>
try {
root = lstatSync(CACHE_DIR)
} catch (error) {
if (isMissingPath(error)) return
throw error
}
if (root.isSymbolicLink() || !root.isDirectory())
throw new Error('Build cache root must be a directory without symlinks')
} else if (!existsSync(CACHE_DIR) || !statSync(CACHE_DIR).isDirectory()) {
return
}
if (!existsSync(CACHE_DIR) || !statSync(CACHE_DIR).isDirectory()) return

const maxGb = parseMaxGb()
const limit = BUILD_CACHE ? maxGb * BYTES_PER_GB : Number.POSITIVE_INFINITY
const bytes = force ? 0 : await directorySize(CACHE_DIR, limit)
const gb = bytes / BYTES_PER_GB
const sizeLabel = force
? 'cache'
: `${BUILD_CACHE && bytes > limit ? 'at least ' : ''}${gb.toFixed(1)} GB`
const gb = force ? 0 : (await directorySize(CACHE_DIR)) / BYTES_PER_GB
const sizeLabel = force ? 'cache' : `${gb.toFixed(1)} GB`

if (!force && gb <= maxGb) {
if (dryRun) console.log(`[turbopack-cache] ${sizeLabel} (cap ${maxGb} GB) — keeping`)
Expand All @@ -127,15 +89,14 @@ async function main() {
}

console.log(`[turbopack-cache] ${sizeLabel} is ${reason} — pruning; next start recompiles cold`)
if (!BUILD_CACHE && lstatSync(CACHE_DIR).isSymbolicLink()) {
if (lstatSync(CACHE_DIR).isSymbolicLink()) {
await rm(CACHE_DIR, { force: true })
return
}
await clearDirectory(CACHE_DIR)
if (!BUILD_CACHE) await rmdir(CACHE_DIR)
await rmdir(CACHE_DIR)
}

main().catch((error) => {
console.warn('[turbopack-cache] prune skipped:', error)
if (BUILD_CACHE) process.exitCode = 1
})
80 changes: 6 additions & 74 deletions scripts/prune-turbopack-cache.test.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,10 @@
import { spawnSync } from 'node:child_process'
import {
existsSync,
lstatSync,
mkdirSync,
mkdtempSync,
readdirSync,
readFileSync,
rmSync,
statSync,
symlinkSync,
writeFileSync,
} from 'node:fs'
Expand Down Expand Up @@ -36,7 +33,7 @@ function fixture() {
}

function run(root: string, args: string[], maximum = '0.000001') {
return spawnSync(args.includes('--build') ? 'node' : 'bun', [script, ...args], {
return spawnSync('bun', [script, ...args], {
cwd: root,
encoding: 'utf8',
timeout: 10_000,
Expand All @@ -48,58 +45,8 @@ afterEach(() => {
for (const root of fixtures.splice(0)) rmSync(root, { recursive: true, force: true })
})

describe('compiler cache maintenance through the real CLI', () => {
it.each(['size', 'force'])(
'clears only build contents by %s while preserving its mount',
(mode) => {
const { root, build, dev, outside } = fixture()
const inode = statSync(build).ino
const result = run(
root,
mode === 'force' ? ['--build', '--force'] : ['--build'],
mode === 'force' ? '1' : undefined
)
expect(result.status, result.stderr).toBe(0)
expect(existsSync(build)).toBe(true)
expect(statSync(build).ino).toBe(inode)
expect(readdirSync(build)).toEqual([])
expect(existsSync(join(dev, 'state'))).toBe(true)
expect(existsSync(join(outside, 'state'))).toBe(true)
}
)

it('retains useful build state without counting external symlink targets', () => {
const { root, build, dev, outside } = fixture()
const inode = statSync(build).ino
const buildBytes = readFileSync(join(build, 'state'))
const devBytes = readFileSync(join(dev, 'state'))
const outsideBytes = readFileSync(join(outside, 'state'))
expect(run(root, ['--build'], '0.000006').status).toBe(0)
expect(statSync(build).ino).toBe(inode)
expect(readFileSync(join(build, 'state'))).toEqual(buildBytes)
expect(readFileSync(join(dev, 'state'))).toEqual(devBytes)
expect(readFileSync(join(outside, 'state'))).toEqual(outsideBytes)
})

it('does not substitute dev state when the build cache is absent', () => {
const { root, build, dev } = fixture()
rmSync(build, { recursive: true })
expect(run(root, ['--build']).status).toBe(0)
expect(existsSync(join(dev, 'state'))).toBe(true)
})

it('rejects a non-directory ancestor instead of treating it as an absent build cache', () => {
const { root, outside } = fixture()
const parent = join(root, '.next')
rmSync(parent, { recursive: true })
writeFileSync(parent, 'unrelated parent file')
const result = run(root, ['--build'])
expect(result.status, result.stderr).not.toBe(0)
expect(readFileSync(parent, 'utf8')).toBe('unrelated parent file')
expect(existsSync(join(outside, 'state'))).toBe(true)
})

it('keeps the existing dev command scoped to dev state', () => {
describe('dev cache maintenance through the real CLI', () => {
it('prunes only dev state, never the build cache or other directories', () => {
const { root, build, dev, outside } = fixture()
expect(run(root, []).status).toBe(0)
expect(existsSync(dev)).toBe(false)
Expand All @@ -126,25 +73,10 @@ describe('compiler cache maintenance through the real CLI', () => {
}
)

it('leaves both caches intact for a build dry run', () => {
it('keeps dev state on a dry run over the cap', () => {
const { root, build, dev } = fixture()
expect(run(root, ['--build', '--dry-run']).status).toBe(0)
expect(existsSync(join(build, 'state'))).toBe(true)
expect(run(root, ['--dry-run']).status).toBe(0)
expect(existsSync(join(dev, 'state'))).toBe(true)
expect(existsSync(join(build, 'state'))).toBe(true)
})

it.each([true, false])(
'refuses a symlink build root when target existence is %s',
(targetExists) => {
const { root, build, dev, outside } = fixture()
rmSync(build, { recursive: true })
symlinkSync(outside, build, process.platform === 'win32' ? 'junction' : 'dir')
if (!targetExists) rmSync(outside, { recursive: true })
const result = run(root, ['--build', '--force'])
expect(result.status, result.stderr).not.toBe(0)
expect(lstatSync(build).isSymbolicLink()).toBe(true)
expect(existsSync(join(outside, 'state'))).toBe(targetExists)
expect(existsSync(join(dev, 'state'))).toBe(true)
}
)
})
Loading