Update demo notebook dependencies and initialization cells - #531
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #531 +/- ##
========================================
Coverage 87.06% 87.06%
========================================
Files 88 88
Lines 7106 7106
========================================
Hits 6187 6187
Misses 919 919
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Thanks for working on Colab/demo notebook consistency. Please address the following issues before merging.
-
Please fix the CI test failures. When implementing these fixes, it is crucial that the changes injected into the notebooks remain minimal and consistent with the latest
developbranch content. For example, the content indemos/elliptic-pde.ipynbcurrently differs from thedevelopbranch, beyond the cell you added to importgoogle.colab. Leave the other cells (code, markdown, or outputs) unchanged. -
Please correct the typo "dependancies" found in the comments.
-
Please test updated notebooks in Google Colab.
a) I have not been able to open for example
gbm_demo.ipynbin Colab by clicking the "Open in Colab" button in my local notebooks. Please check and make sure the buttons work.b) The notebook
elliptic-pde.ipynbopens successfully via the "Open in Colab" button, but it runs with errors.
There was a problem hiding this comment.
I will restructiure this branch to reflect the new version of develop I have been working on with codex.
- CI tests are passing
- Fix Typo
- Correct updated notebooks in google colab
There was a problem hiding this comment.
Thank you for your continued work on this task.
I have three follow-up questions regarding the implementation and scope:
- Is the redirection
>> /dev/nulltruly necessary in the installation cells and is it a good practice to hide installation error message? - Is "import os" necessary?
- I noticed that
gbm_demo.ipynb,elliptic-pde.ipynb, andiris.ipynbstill have run-time errors in Colab. Could you ensure these notebooks and others are Colab-ready? - Given that the
demosdirectory contains over 40 notebooks, and this PR has covered 15, is there a way we could create a task (perhaps inmakefile) to ensure Colab-notebook consistency across the entire suite, which is growing in number over time?
I appreciate your help and thought in making this process robust!
There was a problem hiding this comment.
- Piping to dev null suppresses all output of installation. If there is an error message we should catch that before we release that
- import os will be removed
- Pending
- We should make a template instead of a makefile. The notebooks would have to be tested on google colab.
|
Following up on our review meeting yesterday, I have added scripts to improve the notebooks' compatibility with Colab. Please review and test these scripts in both the local development environment and the Colab platform. |
|
The information previously in this comment has been moved to the Jupyter notebook QA document see the second sheet of the following https://docs.google.com/spreadsheets/d/17KJoNhVyJBGuWoBqz1oujnAvuUm_8CrJ/edit?pli=1&gid=830782523#gid=830782523 |
There was a problem hiding this comment.
Pull request overview
This PR standardizes “Open in Colab” dependency/install and initialization cells across demo notebooks, and adds tooling + CI coverage to keep notebooks Colab-ready over time.
Changes:
- Introduces a Colab notebook manifest plus scripts to check, harden, smoke-test, and report notebook bootstrap patterns.
- Adds CI and Makefile targets to run strict static checks and lightweight execution smoke tests for Colab-enabled notebooks.
- Updates many demo notebooks to include a consistent Colab badge + bootstrap/install cell near the top.
Reviewed changes
Copilot reviewed 56 out of 57 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/test_colab_notebooks.py | Adds unit tests covering badge stripping, bootstrap detection/generation, and smoke-test helpers. |
| test/README.md | Documents the new Colab-related Make targets and their intent. |
| test/booktests/init.py | Adjusts booktest environment setup (notably Windows Matplotlib config isolation). |
| scripts/smoke_test_colab_notebooks.py | Adds smoke execution that simulates Colab and runs bootstrap + early safe cells. |
| scripts/report_colab_notebook_patterns.py | Adds reporting to group enabled notebooks by bootstrap “family” and placement. |
| scripts/harden_colab_notebook.py | Adds a hardening tool to insert badge/bootstrap and (re)classify notebooks in the manifest. |
| scripts/colab_notebooks_manifest.json | Adds the enabled/disabled notebook classification source of truth for Colab readiness. |
| scripts/check_colab_notebooks.py | Adds strict static validation of manifest coverage, badge/bootstrap placement, and early deps/imports. |
| scripts/init.py | Makes scripts runnable via python -m scripts.<tool>. |
| makefile | Adds Make targets for Colab checks/hardening/smoke tests and a SMOKE_CODE_CELLS knob. |
| docs/tests.md | Mirrors the new Colab Make targets documentation in docs. |
| .github/workflows/alltests.yml | Runs Colab strict checks + smoke tests in Linux CI. |
| .gitignore | Ignores temporary Colab smoke notebook artifacts. |
| demos/vectorized_qmc.ipynb | Inserts standardized Colab badge + bootstrap (qmcpy + LaTeX setup). |
| demos/vectorized_qmc_bayes.ipynb | Inserts standardized Colab badge + bootstrap (qmcpy + scikit-learn + LaTeX setup). |
| demos/talk_paper_demos/why_add_q_to_mc_blog/why_add_q_to_mc_blog.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/talk_paper_demos/SorokinThesis2025/sorokin_thesis_2025.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/talk_paper_demos/Sorokin_random_LD_seq_QMC_fast_kernel_methods_2026/Sorokin_random_LD_seq_QMC_fast_kernel_methods_2026.ipynb | Inserts standardized Colab badge + bootstrap (extra pip installs). |
| demos/talk_paper_demos/pydata_chi_2023.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/talk_paper_demos/Parslfest_2025/output/01_sequential_output.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/talk_paper_demos/Parslfest_2025/03_visualize_speedup.ipynb | Inserts standardized Colab badge + bootstrap (repo-clone + path setup). |
| demos/talk_paper_demos/Parslfest_2025/02_parallel.ipynb | Inserts standardized Colab badge + bootstrap (repo-clone + parsl install + path setup). |
| demos/talk_paper_demos/Parslfest_2025/01_sequential.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/talk_paper_demos/MCQMC_Tutorial_2020/MCQMC_2020_QMC_Software_Tutorial.ipynb | Inserts standardized Colab badge + bootstrap (extra pip install). |
| demos/talk_paper_demos/JOSS2026/joss2026.ipynb | Updates setup section ordering; inserts standardized Colab badge + bootstrap (extra pip installs). |
| demos/statistics_for_TrueMeasure.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/some_true_measures.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/scipywrapper_dependence_custom/scipywrapper_demo.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/sample_scatter_plots.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/ray_tracing.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/quickstart.ipynb | Repositions/standardizes Colab badge + adds bootstrap cell. |
| demos/qmcpy-logo.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/qmcpy_intro.ipynb | Reorders intro vs badge; inserts standardized Colab badge + bootstrap. |
| demos/qei-demo-for-blog.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/product_measure.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/pricing_options.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/plot_proj_function.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/nei_demo.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/linear-scrambled-halton.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/lebesgue_integration.ipynb | Adds bootstrap cell after existing badge. |
| demos/lattice_random_generator.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/korobov_hammersley_latinhypercube_demos.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/iris.ipynb | Separates badge into its own cell; adds bootstrap with extra pip installs. |
| demos/GBM/gbm_examples.ipynb | Inserts standardized Colab badge + bootstrap (repo-clone + path setup). |
| demos/GBM/gbm_demo.ipynb | Repositions/standardizes badge; adds bootstrap (repo-clone + extra pip installs + path setup). |
| demos/gaussian_diagnostics/gaussian_diagnostics_demo.ipynb | Fixes badge URL path and adds bootstrap. |
| demos/elliptic-pde.ipynb | Adds bootstrap cell (qmcpy + LaTeX setup). |
| demos/digital_net_b2.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/demo_resume_data/resume_examples.ipynb | Inserts standardized Colab badge + bootstrap (repo-clone + path setup). |
| demos/demo_resume_data/Iteration_Log_Tolerance_Demo.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/demo_resume_data/accuracy_and_resume.ipynb | Inserts standardized Colab badge + bootstrap (repo-clone + path setup). |
| demos/DAKOTA_Genz/dakota_genz.ipynb | Fixes badge URL path and inserts bootstrap (qmcpy + LaTeX setup). |
| demos/copula_examples.ipynb | Inserts standardized Colab badge + bootstrap. |
| demos/control_variates.ipynb | Inserts a setup section and bootstrap cell. |
| demos/brownian_bridge.ipynb | Reorders intro vs badge; inserts standardized Colab badge + bootstrap. |
| demos/asian-option-mlqmc.ipynb | Repositions/standardizes badge; adds bootstrap cell. |
| demos/acceptance_rejection.ipynb | Reorders setup/reference sections; inserts standardized Colab badge + bootstrap. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
@zitterbewegung and reviewers: I have made a number of commits in the past two days to
NOTE:
|
This PR updates demo notebook dependency/install and initialization cells to improve notebook execution consistency. Two notebooks were removed due to different locations.
Updated notebooks:
Supersedes PR #384.