Skip to content

Compile the Rust kernel's crates at build time instead of at every startup - #105

Open
thomasjm wants to merge 1 commit into
mainfrom
evcxr-prebuilt-deps
Open

Compile the Rust kernel's crates at build time instead of at every startup#105
thomasjm wants to merge 1 commit into
mainfrom
evcxr-prebuilt-deps

Conversation

@thomasjm

Copy link
Copy Markdown
Contributor

evcxr compiles the environment's crates into a throwaway directory every time a kernel starts. For
the data-science set (polars, ndarray, plotters, smartcore) that is 115s on this machine, on every
single start
, and it's what makes rust -- use polars... exceed the kernel start timeout.

Doing that work once, when the environment is built, takes it to 6s with nothing recompiled.

How

EVCXR_TMPDIR makes evcxr's build directory persistent, so a derivation runs evcxr once with it
pointed at the output. The wrapper unpacks that into a fresh directory per process and points
EVCXR_TMPDIR there. (evcxr ignores CARGO_TARGET_DIR — its target/ lives inside this directory,
not the other way round.)

It has to be a copy rather than the store path itself: cargo needs to write, and two kernels sharing
one build directory would overwrite each other's source file.

Why a tarball and not a directory

This is the part that took the longest to find. Cargo decides what to rebuild by comparing mtimes,
and the store normalises every mtime to the epoch. That flattening destroys the ordering between
an artifact and what it was built from, so cargo rebuilt all 61 crates that have build scripts.
Setting them all to "now" doesn't help — equal timestamps aren't ordered ones.

A tarball is a single file as far as the store is concerned, so the timestamps recorded inside it
survive, and unpacking restores the real ordering. This is the same reason
crane packages cargo artifacts rather than
handing cargo a store path.

Three other things had to be right:

  • Modes are kept, timestamps are not. Some of these artifacts are build scripts and have to stay
    executable; --no-preserve=mode broke them with Permission denied (os error 13).
  • The lock files are deleted. Cargo takes one lock per artifact directory. Both are empty, so
    they're stored as a single hardlink, and cargo deadlocks against itself locking one inode twice —
    visible as Blocking waiting for file lock on artifact directory with both paths reporting the
    same inode.
  • The seed only runs when EVCXR_TMPDIR is unset. evcxr re-executes itself through the wrapper,
    and seeding again gave the child a second build directory to deadlock against.

The prebuild also runs under a cleared environment, since cargo reruns any build script whose
rerun-if-env-changed variables differ, and stdenv exports a great many of those.

Testing

rust -- use polars... (no errors) now passes with the existing 120s timeout, which it did not
before — that test is the functional regression test, since a missing prebuild puts startup back
over the limit. Added seeds a prebuilt build directory alongside it, because the timeout only
catches this indirectly.

Full --rust suite green: 522 tests, no failures.

Measurements on this machine, 1+1 through the data-science kernel:

before 115s
after 6s, 0 crates recompiled

Note on #103

This should make the testKernelSucceeds' 600 commit there unnecessary — the timeout was covering
exactly this startup cost. #103's other commit, the rust-analyzer diagnostic batching fix, is
unrelated and still needed. Happy to drop the timeout commit from #103, or leave it and follow up.

Tradeoff

The prebuilt tarball is ~640 MB of store per Rust environment, and unpacking costs about a second at
kernel start. That's worth stating plainly given the sandboxed-store-size work, though it's a
trade of teardown bytes for two minutes of startup.

@thomasjm
thomasjm force-pushed the evcxr-prebuilt-deps branch from 1c5b26f to 80c28ed Compare August 28, 2026 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant