Repository navigation
Render and deploy the book in CI from a local execution cache - #93
Merged
Merged
Conversation
The tutorials need several GB of data and more memory than a hosted runner offers, so the book was built locally and the HTML committed to docs/. Separate execution from rendering instead: - Build with execute_notebooks: cache. Notebook outputs live in _build/.jupyter_cache, keyed by a hash of each notebook's code cells, so only notebooks whose code changed are executed. - Ship the cache and content/figures/ to CI as the execution-bundle.tar.gz asset of the execution-cache release (scripts/execution_bundle.py, make publish-cache / fetch-cache). - Add a book workflow that restores the bundle, renders the HTML and deploys it with GitHub Pages actions. A notebook missing from the cache fails the run on main before deploying and is a warning on pull requests (scripts/check_cache.py). - Keep the execution cache in `make clean`; add `make clean-all`. - Remove the committed docs/ tree, scripts/sync_docs.py and the unused Jekyll, runall and docs-sync Makefile targets. Pages must be switched to the "GitHub Actions" source, and the cache published, before this is merged.
06_Cerebrovascular_Reactivity_Mapping is illustrative code that reads files that are not part of the tutorial data, and plot_approach_figures fails on an undefined variable and a missing T1w file. A notebook that fails is never cached, so both would block the strict cache check. Exclude them from execution and make check_cache.py honor execute.exclude_patterns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The tutorials need several GB of data and more memory than a hosted runner offers, so the book has been built locally and the HTML committed to
docs/. This PR separates execution from rendering, so that neither the built HTML nor the execution results live in git:execute_notebooks: cache. Outputs are stored in_build/.jupyter_cache, keyed by a hash of each notebook's code cells, somake bookonly executes notebooks whose code changed. Editing prose, the table of contents or the configuration never triggers execution.content/figures/is uploaded as theexecution-bundle.tar.gzasset of theexecution-cacherelease (make publish-cache). The newbookworkflow downloads it, renders the HTML and deploys it with the GitHub Pages actions.scripts/check_cache.pylists notebooks whose code is newer than the cache. Onmainthat fails the workflow before deploying, so the live site keeps the previous build. On pull requests it is a warning.Changes
.github/workflows/book.yml: build on pushes and pull requests, deploy onmain.scripts/check_cache.py,scripts/execution_bundle.py: cache check and bundle pack/unpack.scripts/clean.py: keeps the execution cache unless run with--all.Makefile: newcheck-cache,fetch-cache,publish-cache,deploy,clean-alltargets. The unused Jekyll targets,runall,sync-docsandsite-publishare removed.docs/andscripts/sync_docs.pyare removed.README.md: new build and publishing instructions.Two notebooks are excluded from execution
Both already show errors on the live site, and a notebook that fails is never cached, so they would block the cache check:
06_Cerebrovascular_Reactivity_Mapping: illustrative code that reads files that are not part of the tutorial data. It now renders as code without a traceback.plot_approach_figures: fails on an undefinedfunc_dirand a T1w file name that is not in ds006185, so its 16 figures are missing (as they are today). The exclusion should be removed once the notebook is fixed.Testing
make check-cache STRICT=1passes, andmake publish-cacheuploaded the bundle (23 MB) to theexecution-cacherelease.Before merging
Settings > Pages > Source must be switched to "GitHub Actions", close to the merge. This PR removes
docs/, which the current "Deploy from a branch" setting serves.🤖 Generated with Claude Code