Skip to content

Render and deploy the book in CI from a local execution cache - #93

Merged
tsalo merged 2 commits into
mainfrom
ci-build-from-execution-cache
Nov 14, 2026
Merged

tsalo merged 2 commits into
mainfrom
ci-build-from-execution-cache

Conversation

@tsalo

@tsalo tsalo commented Nov 13, 2026

Copy link
Copy Markdown
Member

Summary

The tutorials need several GB of data and more memory than a hosted runner offers, so the book has been built locally and the HTML committed to docs/. This PR separates execution from rendering, so that neither the built HTML nor the execution results live in git:

  • Execution stays local. The book is built with execute_notebooks: cache. Outputs are stored in _build/.jupyter_cache, keyed by a hash of each notebook's code cells, so make book only executes notebooks whose code changed. Editing prose, the table of contents or the configuration never triggers execution.
  • Rendering moves to GitHub Actions. The cache plus content/figures/ is uploaded as the execution-bundle.tar.gz asset of the execution-cache release (make publish-cache). The new book workflow downloads it, renders the HTML and deploys it with the GitHub Pages actions.
  • Outputs cannot silently disappear. scripts/check_cache.py lists notebooks whose code is newer than the cache. On main that fails the workflow before deploying, so the live site keeps the previous build. On pull requests it is a warning.

Changes

  • .github/workflows/book.yml: build on pushes and pull requests, deploy on main.
  • scripts/check_cache.py, scripts/execution_bundle.py: cache check and bundle pack/unpack.
  • scripts/clean.py: keeps the execution cache unless run with --all.
  • Makefile: new check-cache, fetch-cache, publish-cache, deploy, clean-all targets. The unused Jekyll targets, runall, sync-docs and site-publish are removed.
  • docs/ and scripts/sync_docs.py are removed.
  • README.md: new build and publishing instructions.

Two notebooks are excluded from execution

Both already show errors on the live site, and a notebook that fails is never cached, so they would block the cache check:

  • 06_Cerebrovascular_Reactivity_Mapping: illustrative code that reads files that are not part of the tutorial data. It now renders as code without a traceback.
  • plot_approach_figures: fails on an undefined func_dir and a T1w file name that is not in ds006185, so its 16 figures are missing (as they are today). The exclusion should be removed once the notebook is fixed.

Testing

  • Full local build in cache mode: 18 notebooks executed and cached, about 100 minutes.
  • Rebuild afterwards: all 18 read from the cache, none executed, about 2.5 minutes.
  • make check-cache STRICT=1 passes, and make publish-cache uploaded the bundle (23 MB) to the execution-cache release.
  • The workflow in this PR is its first run on GitHub.

Before merging

Settings > Pages > Source must be switched to "GitHub Actions", close to the merge. This PR removes docs/, which the current "Deploy from a branch" setting serves.

🤖 Generated with Claude Code

The tutorials need several GB of data and more memory than a hosted
runner offers, so the book was built locally and the HTML committed to
docs/. Separate execution from rendering instead:

- Build with execute_notebooks: cache. Notebook outputs live in
  _build/.jupyter_cache, keyed by a hash of each notebook's code cells,
  so only notebooks whose code changed are executed.
- Ship the cache and content/figures/ to CI as the
  execution-bundle.tar.gz asset of the execution-cache release
  (scripts/execution_bundle.py, make publish-cache / fetch-cache).
- Add a book workflow that restores the bundle, renders the HTML and
  deploys it with GitHub Pages actions. A notebook missing from the
  cache fails the run on main before deploying and is a warning on
  pull requests (scripts/check_cache.py).
- Keep the execution cache in `make clean`; add `make clean-all`.
- Remove the committed docs/ tree, scripts/sync_docs.py and the unused
  Jekyll, runall and docs-sync Makefile targets.

Pages must be switched to the "GitHub Actions" source, and the cache
published, before this is merged.
06_Cerebrovascular_Reactivity_Mapping is illustrative code that reads
files that are not part of the tutorial data, and plot_approach_figures
fails on an undefined variable and a missing T1w file. A notebook that
fails is never cached, so both would block the strict cache check.
Exclude them from execution and make check_cache.py honor
execute.exclude_patterns.
@tsalo
tsalo merged commit 2eb9a15 into main Nov 14, 2026
2 checks passed
@tsalo
tsalo deleted the ci-build-from-execution-cache branch November 14, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant