Compile the Rust kernel's crates at build time instead of at every startup - #105
Open
thomasjm wants to merge 1 commit into
Open
Compile the Rust kernel's crates at build time instead of at every startup#105thomasjm wants to merge 1 commit into
thomasjm wants to merge 1 commit into
Conversation
thomasjm
force-pushed
the
evcxr-prebuilt-deps
branch
from
August 28, 2026 23:56
1c5b26f to
80c28ed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
evcxr compiles the environment's crates into a throwaway directory every time a kernel starts. For
the data-science set (polars, ndarray, plotters, smartcore) that is 115s on this machine, on every
single start, and it's what makes
rust -- use polars...exceed the kernel start timeout.Doing that work once, when the environment is built, takes it to 6s with nothing recompiled.
How
EVCXR_TMPDIRmakes evcxr's build directory persistent, so a derivation runs evcxr once with itpointed at the output. The wrapper unpacks that into a fresh directory per process and points
EVCXR_TMPDIRthere. (evcxr ignoresCARGO_TARGET_DIR— itstarget/lives inside this directory,not the other way round.)
It has to be a copy rather than the store path itself: cargo needs to write, and two kernels sharing
one build directory would overwrite each other's source file.
Why a tarball and not a directory
This is the part that took the longest to find. Cargo decides what to rebuild by comparing mtimes,
and the store normalises every mtime to the epoch. That flattening destroys the ordering between
an artifact and what it was built from, so cargo rebuilt all 61 crates that have build scripts.
Setting them all to "now" doesn't help — equal timestamps aren't ordered ones.
A tarball is a single file as far as the store is concerned, so the timestamps recorded inside it
survive, and unpacking restores the real ordering. This is the same reason
crane packages cargo artifacts rather than
handing cargo a store path.
Three other things had to be right:
executable;
--no-preserve=modebroke them withPermission denied (os error 13).they're stored as a single hardlink, and cargo deadlocks against itself locking one inode twice —
visible as
Blocking waiting for file lock on artifact directorywith both paths reporting thesame inode.
EVCXR_TMPDIRis unset. evcxr re-executes itself through the wrapper,and seeding again gave the child a second build directory to deadlock against.
The prebuild also runs under a cleared environment, since cargo reruns any build script whose
rerun-if-env-changedvariables differ, and stdenv exports a great many of those.Testing
rust -- use polars... (no errors)now passes with the existing 120s timeout, which it did notbefore — that test is the functional regression test, since a missing prebuild puts startup back
over the limit. Added
seeds a prebuilt build directoryalongside it, because the timeout onlycatches this indirectly.
Full
--rustsuite green: 522 tests, no failures.Measurements on this machine,
1+1through the data-science kernel:Note on #103
This should make the
testKernelSucceeds' 600commit there unnecessary — the timeout was coveringexactly this startup cost. #103's other commit, the rust-analyzer diagnostic batching fix, is
unrelated and still needed. Happy to drop the timeout commit from #103, or leave it and follow up.
Tradeoff
The prebuilt tarball is ~640 MB of store per Rust environment, and unpacking costs about a second at
kernel start. That's worth stating plainly given the sandboxed-store-size work, though it's a
trade of teardown bytes for two minutes of startup.