Skip to content

Update demo notebook dependencies and initialization cells - #531

Merged
sou-cheng-choi merged 41 commits into
developfrom
colab_easy_fix_redo
Sep 3, 2026
Merged

Update demo notebook dependencies and initialization cells#531
sou-cheng-choi merged 41 commits into
developfrom
colab_easy_fix_redo

Conversation

@zitterbewegung

Copy link
Copy Markdown
Collaborator

This PR updates demo notebook dependency/install and initialization cells to improve notebook execution consistency. Two notebooks were removed due to different locations.

Updated notebooks:

  • demos/elliptic-pde.ipynb
  • demos/iris.ipynb
  • demos/qei-demo-for-blog.ipynb

Supersedes PR #384.

@codecov

codecov Bot commented May 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.06%. Comparing base (ab0969b) to head (f359455).
⚠️ Report is 50 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff            @@
##           develop     #531   +/-   ##
========================================
  Coverage    87.06%   87.06%           
========================================
  Files           88       88           
  Lines         7106     7106           
========================================
  Hits          6187     6187           
  Misses         919      919           
Flag Coverage Δ
alltests 87.06% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread demos/elliptic-pde.ipynb

@sou-cheng-choi sou-cheng-choi May 22, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zitterbewegung

Thanks for working on Colab/demo notebook consistency. Please address the following issues before merging.

  1. Please fix the CI test failures. When implementing these fixes, it is crucial that the changes injected into the notebooks remain minimal and consistent with the latest develop branch content. For example, the content in demos/elliptic-pde.ipynb currently differs from the develop branch, beyond the cell you added to import google.colab. Leave the other cells (code, markdown, or outputs) unchanged.

  2. Please correct the typo "dependancies" found in the comments.

  3. Please test updated notebooks in Google Colab.

    a) I have not been able to open for example gbm_demo.ipynb in Colab by clicking the "Open in Colab" button in my local notebooks. Please check and make sure the buttons work.

    b) The notebook elliptic-pde.ipynb opens successfully via the "Open in Colab" button, but it runs with errors.

@zitterbewegung zitterbewegung May 22, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will restructiure this branch to reflect the new version of develop I have been working on with codex.

  • CI tests are passing
  • Fix Typo
  • Correct updated notebooks in google colab

Comment thread demos/iris.ipynb

@sou-cheng-choi sou-cheng-choi May 23, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your continued work on this task.

I have three follow-up questions regarding the implementation and scope:

  1. Is the redirection >> /dev/null truly necessary in the installation cells and is it a good practice to hide installation error message?
  2. Is "import os" necessary?
  3. I noticed that gbm_demo.ipynb, elliptic-pde.ipynb, and iris.ipynb still have run-time errors in Colab. Could you ensure these notebooks and others are Colab-ready?
  4. Given that the demos directory contains over 40 notebooks, and this PR has covered 15, is there a way we could create a task (perhaps in makefile) to ensure Colab-notebook consistency across the entire suite, which is growing in number over time?

I appreciate your help and thought in making this process robust!

@zitterbewegung zitterbewegung May 25, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Piping to dev null suppresses all output of installation. If there is an error message we should catch that before we release that
  2. import os will be removed
  3. Pending
  4. We should make a template instead of a makefile. The notebooks would have to be tested on google colab.

@sou-cheng-choi

Copy link
Copy Markdown
Member

@zitterbewegung

Following up on our review meeting yesterday, I have added scripts to improve the notebooks' compatibility with Colab. Please review and test these scripts in both the local development environment and the Colab platform.

@zitterbewegung

zitterbewegung commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator Author

The information previously in this comment has been moved to the Jupyter notebook QA document see the second sheet of the following https://docs.google.com/spreadsheets/d/17KJoNhVyJBGuWoBqz1oujnAvuUm_8CrJ/edit?pli=1&gid=830782523#gid=830782523

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR standardizes “Open in Colab” dependency/install and initialization cells across demo notebooks, and adds tooling + CI coverage to keep notebooks Colab-ready over time.

Changes:

  • Introduces a Colab notebook manifest plus scripts to check, harden, smoke-test, and report notebook bootstrap patterns.
  • Adds CI and Makefile targets to run strict static checks and lightweight execution smoke tests for Colab-enabled notebooks.
  • Updates many demo notebooks to include a consistent Colab badge + bootstrap/install cell near the top.

Reviewed changes

Copilot reviewed 56 out of 57 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
test/test_colab_notebooks.py Adds unit tests covering badge stripping, bootstrap detection/generation, and smoke-test helpers.
test/README.md Documents the new Colab-related Make targets and their intent.
test/booktests/init.py Adjusts booktest environment setup (notably Windows Matplotlib config isolation).
scripts/smoke_test_colab_notebooks.py Adds smoke execution that simulates Colab and runs bootstrap + early safe cells.
scripts/report_colab_notebook_patterns.py Adds reporting to group enabled notebooks by bootstrap “family” and placement.
scripts/harden_colab_notebook.py Adds a hardening tool to insert badge/bootstrap and (re)classify notebooks in the manifest.
scripts/colab_notebooks_manifest.json Adds the enabled/disabled notebook classification source of truth for Colab readiness.
scripts/check_colab_notebooks.py Adds strict static validation of manifest coverage, badge/bootstrap placement, and early deps/imports.
scripts/init.py Makes scripts runnable via python -m scripts.<tool>.
makefile Adds Make targets for Colab checks/hardening/smoke tests and a SMOKE_CODE_CELLS knob.
docs/tests.md Mirrors the new Colab Make targets documentation in docs.
.github/workflows/alltests.yml Runs Colab strict checks + smoke tests in Linux CI.
.gitignore Ignores temporary Colab smoke notebook artifacts.
demos/vectorized_qmc.ipynb Inserts standardized Colab badge + bootstrap (qmcpy + LaTeX setup).
demos/vectorized_qmc_bayes.ipynb Inserts standardized Colab badge + bootstrap (qmcpy + scikit-learn + LaTeX setup).
demos/talk_paper_demos/why_add_q_to_mc_blog/why_add_q_to_mc_blog.ipynb Inserts standardized Colab badge + bootstrap.
demos/talk_paper_demos/SorokinThesis2025/sorokin_thesis_2025.ipynb Inserts standardized Colab badge + bootstrap.
demos/talk_paper_demos/Sorokin_random_LD_seq_QMC_fast_kernel_methods_2026/Sorokin_random_LD_seq_QMC_fast_kernel_methods_2026.ipynb Inserts standardized Colab badge + bootstrap (extra pip installs).
demos/talk_paper_demos/pydata_chi_2023.ipynb Inserts standardized Colab badge + bootstrap.
demos/talk_paper_demos/Parslfest_2025/output/01_sequential_output.ipynb Inserts standardized Colab badge + bootstrap.
demos/talk_paper_demos/Parslfest_2025/03_visualize_speedup.ipynb Inserts standardized Colab badge + bootstrap (repo-clone + path setup).
demos/talk_paper_demos/Parslfest_2025/02_parallel.ipynb Inserts standardized Colab badge + bootstrap (repo-clone + parsl install + path setup).
demos/talk_paper_demos/Parslfest_2025/01_sequential.ipynb Inserts standardized Colab badge + bootstrap.
demos/talk_paper_demos/MCQMC_Tutorial_2020/MCQMC_2020_QMC_Software_Tutorial.ipynb Inserts standardized Colab badge + bootstrap (extra pip install).
demos/talk_paper_demos/JOSS2026/joss2026.ipynb Updates setup section ordering; inserts standardized Colab badge + bootstrap (extra pip installs).
demos/statistics_for_TrueMeasure.ipynb Inserts standardized Colab badge + bootstrap.
demos/some_true_measures.ipynb Inserts standardized Colab badge + bootstrap.
demos/scipywrapper_dependence_custom/scipywrapper_demo.ipynb Inserts standardized Colab badge + bootstrap.
demos/sample_scatter_plots.ipynb Inserts standardized Colab badge + bootstrap.
demos/ray_tracing.ipynb Inserts standardized Colab badge + bootstrap.
demos/quickstart.ipynb Repositions/standardizes Colab badge + adds bootstrap cell.
demos/qmcpy-logo.ipynb Inserts standardized Colab badge + bootstrap.
demos/qmcpy_intro.ipynb Reorders intro vs badge; inserts standardized Colab badge + bootstrap.
demos/qei-demo-for-blog.ipynb Inserts standardized Colab badge + bootstrap.
demos/product_measure.ipynb Inserts standardized Colab badge + bootstrap.
demos/pricing_options.ipynb Inserts standardized Colab badge + bootstrap.
demos/plot_proj_function.ipynb Inserts standardized Colab badge + bootstrap.
demos/nei_demo.ipynb Inserts standardized Colab badge + bootstrap.
demos/linear-scrambled-halton.ipynb Inserts standardized Colab badge + bootstrap.
demos/lebesgue_integration.ipynb Adds bootstrap cell after existing badge.
demos/lattice_random_generator.ipynb Inserts standardized Colab badge + bootstrap.
demos/korobov_hammersley_latinhypercube_demos.ipynb Inserts standardized Colab badge + bootstrap.
demos/iris.ipynb Separates badge into its own cell; adds bootstrap with extra pip installs.
demos/GBM/gbm_examples.ipynb Inserts standardized Colab badge + bootstrap (repo-clone + path setup).
demos/GBM/gbm_demo.ipynb Repositions/standardizes badge; adds bootstrap (repo-clone + extra pip installs + path setup).
demos/gaussian_diagnostics/gaussian_diagnostics_demo.ipynb Fixes badge URL path and adds bootstrap.
demos/elliptic-pde.ipynb Adds bootstrap cell (qmcpy + LaTeX setup).
demos/digital_net_b2.ipynb Inserts standardized Colab badge + bootstrap.
demos/demo_resume_data/resume_examples.ipynb Inserts standardized Colab badge + bootstrap (repo-clone + path setup).
demos/demo_resume_data/Iteration_Log_Tolerance_Demo.ipynb Inserts standardized Colab badge + bootstrap.
demos/demo_resume_data/accuracy_and_resume.ipynb Inserts standardized Colab badge + bootstrap (repo-clone + path setup).
demos/DAKOTA_Genz/dakota_genz.ipynb Fixes badge URL path and inserts bootstrap (qmcpy + LaTeX setup).
demos/copula_examples.ipynb Inserts standardized Colab badge + bootstrap.
demos/control_variates.ipynb Inserts a setup section and bootstrap cell.
demos/brownian_bridge.ipynb Reorders intro vs badge; inserts standardized Colab badge + bootstrap.
demos/asian-option-mlqmc.ipynb Repositions/standardizes badge; adds bootstrap cell.
demos/acceptance_rejection.ipynb Reorders setup/reference sections; inserts standardized Colab badge + bootstrap.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread demos/elliptic-pde.ipynb
Comment thread scripts/check_colab_notebooks.py
@sou-cheng-choi

sou-cheng-choi commented Sep 1, 2026

Copy link
Copy Markdown
Member

@zitterbewegung and reviewers:

I have made a number of commits in the past two days to

  • bring in the latest changes in develop branch including version 2.4
  • Introduce boolean variableIN_COLAB for clarity
  • fix run time errors in free-tier Colab environment due to limited resources subh as memory
  • enhance related scripts and documentation

NOTE:

  1. I have manually tested in Colab at least one (but not all) notebook in each COLAB pattern as documented in README.md.

  2. Differences between this branch and develop is here

@sou-cheng-choi
sou-cheng-choi self-requested a review September 1, 2026 13:52
@sou-cheng-choi sou-cheng-choi self-assigned this Sep 1, 2026
@sou-cheng-choi
sou-cheng-choi removed their request for review September 1, 2026 13:52
@sou-cheng-choi
sou-cheng-choi self-requested a review September 1, 2026 14:09
Comment thread test/test_colab_notebooks.py Fixed

@fjhickernell fjhickernell left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Joshua.

@sou-cheng-choi
sou-cheng-choi merged commit d8fec00 into develop Sep 3, 2026
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants