Repository navigation
Slim Lake setup files before CI caches and validates the pool build - #820
Conversation
Lake writes a .setup.json compiler input per module listing absolute artifact paths for every transitive import (~1.6 MB each, 46 GB of the 69 GB pool build). Lake never reads them back, and the validation cache and exposition extractor only use keys naming local modules. Drop the other keys after every CI build step so restores, saves, uploads, and lint/quality fingerprinting stop handling the dead weight. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both readers use only import names, and big projects list hundreds of local imports with four absolute paths each (4.8 GB pool-wide). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Proof profile (new / modified Lean files)No added or modified Lean files under Advisory only — never blocks merge. |
|
Advisory cross-file review against Lean Pool's supervisory rubric. This is not by itself a significance, source-verification, or merge verdict. [Medium impact] The PR appears safe to merge, with the previous test-coverage gap addressed. SummaryThe PR shrinks Lake setup files before CI validates, packages, or caches build outputs.
Reviews (2) · Last reviewed commit: "Test that Lake replays and rebuilds corr..." · Reviewed by Greptile |
Runs under the pinned-Lean CI step so a toolchain bump that starts reading setup files back fails loudly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Names-only slimming, measured pool-wide on the Azure VM (31,455 setup files, pool at 5dd7126):
🤖 Generated with Claude Code |
Vilin97
left a comment
There was a problem hiding this comment.
Source review passes at e4934b4cc1a8f3ab6e1c8d768fa59cad382ae692; protected validation and separate workflow-change authorization remain outstanding.
The slimmer retains local import names, including missing LeanPool names needed for failure detection, and preserves modification times before atomic replacement. The validation and exposition readers consume those retained names rather than the removed artifact-path payloads. The new pinned-Lake regression checks both unchanged replay and rebuilding after an imported source changes; it addresses the previous inline review's concrete coverage request. I found no actionable defect in the five-file change.
Hosted Python and workflow checks passed in the saved intake. Core was still running, so the real pinned-toolchain regression and full build outcome remain to be verified. The measured VM savings are author claims, not independent evidence here. This PR changes four Lean CI steps and cache inputs; the repository's separate workflow-change authorization requirement still applies, and the current operative authorization names PR758, PR817 and PR819 rather than this PR. No queue or auto-merge action is taken.
Why
Before compiling a module, Lake writes
.lake/build/ir/<Module>.setup.jsonforlean --setup. ItsimportArtslists the absolute artifact paths of every transitive import: about 3,000 entries, roughly 1.6 MB, per Mathlib-importing module. On the Azure VM's current pool build these files are 49 GB of 69 GB, while the oleans are 22 GB.Every CI job pays for this:
ValidationCache.fingerprintover all units took 444 s: 280 s inread+jsonon setup files, against 14 s to hash the 611 MB of sources.Lake never reads a setup file back. It rewrites one when it recompiles that module, as
Lake/Build/Actions.leanshows. The only readers arevalidation_cache._module_sourcesandexposition.extract.project_sources. Both use onlyimportArtskeys that name aLeanPoolmodule or a Lean source in the checkout, and never the artifact paths.What
python/lean_pool/slim_setup.py(stdlib only, parallel). It rewrites each setup file to keep only those local import names, with empty artifact lists, and preserves mtimes so the shards'find -newerpackaging still ships only fresh files. Unreadable files are left alone.Slim Lake setup filesstep after every build step inlean_action_ci.yml: single build, shard, finalize, and rebase. It runs before validation, docs packaging, artifact uploads and cache saves.python/tests/test_slim_setup.py. They check that both readers return identical results before and after, plus mtime preservation, idempotence, unreadable files, and the CLI.python/ci-performance.md.No check changes what it accepts. The validation fingerprints hash the same source set, so existing receipts stay valid.
Verification (Azure VM, v4.35.0, pool at 5dd7126)
LeanPool.Champernowneand slimming,lake build --no-buildreports "All targets up-to-date" (exit 0), and a normallake buildrecompiles nothing. After editing a source, Lake rebuilds that module normally.source_inventoryfor all 343 validation units: 0 differences. With the final names-only format the setup files shrink from 49.0 GB to 0.63 GB (78×), and building the inventories drops from 487 s to 49 s.pytestfor slim_setup, validation_cache and exposition_extract passes, andruffis clean.Rollout
The first run after merge still restores the existing full-size cache and pays a one-time slimming pass: about 7 min with 4 workers on the busy VM. Main's next cache save is slim, and later restores benefit from then on.
🤖 Generated with Claude Code