MNT, CI: Update pyproject.toml to include [dependency-groups] by aganguly-lab · Pull Request #127 · lanl/GFDL

aganguly-lab · 2026-09-01T23:15:31Z

Fixes #123.

The [project.optional-dependencies] list is removed from pyproject.toml
The [dependency-groups] list is added to pyproject.toml
Two optional groups are added: testing and linting.
Changes are made to all .yml files (except wheel.yml and dependabot.yml) to account for the updated workflow.

With this update, there are three commands that can be used to build the project from source:

python -m pip install .
python -m pip install --group testing
python -m pip install --group linting

The first command installs the project as normal. The second installs the dependencies needed to run the test suite. The last installs all dependencies needed to lint (ruff check and numpydoc lint).

When running, make sure pip is up to date. Installation of the testing or linting group will fail if the version of pip is 25.0 or lower. In particular, I found that the default version of pip on darwin and chicoma was too low for this to work, but everything does work when I update pip using pip install --upgrade pip first. I am not sure if this will cause the CI pipeline to fail. If it does, I will add another commit adding the above incantation to the relevant .yml files.

I did not modify wheel.yml. To my understanding, wheel.yml is intended to test the build as seen by users of the code. Users should not be able to directly install the testing/linting versions of the package.

AI Statement: I did not use AI to write any of the code on this branch. I did consult AI extensively with questions about pip installations and builds. I also used AI for debugging purposes.

Testing the update

I have little experience working with installation, so I figured I would include an outline of how I tested my changes to help reviewers. I ran the code below on my laptop and on darwin (I could not secure a chicoma node). For the unit tests, I first ran the code as described below, but then once I confirmed all tests were collected and a few tests had run, I stopped the testing. I then used python -m pytest -n 8 to verify the tests separately (remember to pip install pytest-xdist first).

%%% start from the root of the directory and deactivate any venvs
%%% make sure unit testing works
git clean -xdf
python -m venv test ; source test/bin/activate
python -m pip install --upgrade pip
python -m pip install -v --group testing .
cd
python -m pytest --pyargs gfdl --cov=gfdl --cov-report=term-missing
deactivate
git clean -xdf

Return to the root directory.

%%% test linting
python -m venv test ; source test/bin/activate
python -m pip install --upgrade pip
python -m pip install -v --group linting .
ruff check
numpydoc lint src/gfdl/model.py src/gfdl/activations.py src/gfdl/weights.py
cd docs && make html SPHINXOPTS="-W --keep-going"
deactivate
cd ..
git clean -xdf

%%% make sure building the project is unaffected (no need to upgrade pip)
python -m venv test ; source test/bin/activate
python -m pip install build pytest pytest-cov
python -m build
python -m pip install dist/gfdl*.whl
python -m pip install pandas ucimlrepo
cd 
python -m pytest --pyargs gfdl --cov=gfdl --cov-report=term-missing

… simplify the process of restoring this version of the code. * Introduces two new dependency-groups in the pyproject.toml file: testing and linting * Removed [project.optional-dependencies] from pyproject.toml: none are needed * Removed the .github/constraints/deps.txt file * Updated .yml files with new install instructions

I forgot to install the project and its dependencies.

tylerjereddy

Thanks, I added some quick review comments, but I'll let Seth do the main review here.

tylerjereddy · 2026-09-02T02:43:23Z

-        python -m pip install ruff==0.15.0
-        python -m pip install -r .github/constraints/deps.txt
+        python -m pip install -v --group testing .
+        python -m pip install -v --group linting .


Here and elsewhere, what's the motivation behind install the project (the dot .) twice?

Good catch. I was running these separately while testing, so it was useful to have both but there is no reason for it. I made the relevant adjustments.

tylerjereddy · 2026-09-02T02:45:33Z

+    "numpydoc",
+    "sphinx_copybutton",
+    "sphinx_design",
+    "pydata_sphinx_theme",


Some of these are docs deps rather than linting, right?

Perhaps it would make sense to have a doc group or similar, to form a clear separation from linting, and then be able to install all development deps with a dev group or something like that (I believe the upstream example does something similar).

I did this. The scipy package also had a build dependency-group, but I couldn't see the point of it in our case. The build project wouldn't install the gfdl module which would be necessary no matter what. And by installing it, we would automatically get any packages in the build dependency-group.

Our dev dependency group simply combines testing, linting and doc.

…nse to #127 (comment) and updated the .yml files accordingly. Changes are untested, but I included this commit for easy retreval during testing.

…ant. Not sure why SciPy has it. I also removed some packages in the linting group that belonged purely in the doc group.

sdtemple

I verified that your commands run Mac OS and on selene (different but same OS from your darwin run).

It would be nice to confirm that the CI failures are from #117. The CI is set by default to fail-fast, whereas with in the .yml file I think we can keep it running for a bit:

    strategy:
      fail-fast: false  # <--- This is the key line

This might not be the desired long term '.yml' file strategy, though. I forked the branch and added this fail-fast: false to the ci.yml file. I also changed on pushes to be "*". I confirmed via this that the base CI failure is from the partial fit test.

Note: this happens for both Windows OS and Ubuntu OS. We might only have been noticing it for Linux because that's what the cluster uses and it is the first crash (temporally) in the CI run.

sdtemple · 2026-09-08T19:23:54Z

-        python -m pip install -v ".[test]"
-        python -m pip install -r .github/constraints/deps.txt
+        python -m pip install -v .
+        python -m pip install -v --group doc


Do you need the -v verbose options in the CI?

I think it is better to leave the -v options here. It should make it easier to debug the issue if the pipeline fails as due to a broken build. I imagine that is why the -v option was used in the previous commit as well.

sdtemple · 2026-09-08T19:29:25Z

 gfdl = ["testdata/**/*"]

-
-[project.optional-dependencies]


The scipy PR you're following as an example mentions that you can keep project.optional-dependencies if there were packages you wanted the user to maybe install.
https://github.com/scipy/scipy/blob/8f7565df3e164e35bbde58e0db735bcf1598715b/pyproject.toml#L78

I'm not saying there is a particular package we want to install. I'm just noting this functionality should we circle back.

sdtemple · 2026-09-08T19:30:07Z

-        python -m pip install -v -e ".[test]"
-        python -m pip install -r .github/constraints/deps.txt
+        python -m pip install -v .
+        python -m pip install -v --group doc


Do you need the -v option in the CI?

sdtemple · 2026-09-08T19:30:12Z

-        python -m pip install -r .github/constraints/deps.txt
+        python -m pip install -v .
+        python -m pip install -v --group testing
+        python -m pip install -v --group linting


Do you need the -v option in the CI?

sdtemple · 2026-09-08T19:46:29Z

+        python -m pip install -v --group doc
    - name: lint and build docs
      run: |
        numpydoc lint src/gfdl/model.py src/gfdl/activations.py src/gfdl/weights.py


Is this line necessary?

AI suggestion:

Create a conf.py with the following:

# docs/conf.py extensions = [ "sphinx.ext.autodoc", "numpydoc", # Crucial: This must be present ] # This is what replaces your "numpydoc lint" command! numpydoc_validation_checks = { "all", # Instructs numpydoc to lint everything during the build }

Then, you don't have to manually add files like src/gfdl/model.py whenever you add a new file to the package.

Such a file belongs under docs/

sdtemple · 2026-09-08T19:53:28Z

-[project.optional-dependencies]
-test = [
+[dependency-groups]
+testing = [


test not testing

sdtemple · 2026-09-08T19:56:29Z

    "pytest",
    "pytest-cov",
 ]
+linting = [


lint not linting. I think the convention is that test, lint, and doc are active verbs.

sdtemple · 2026-09-08T19:59:42Z

-        python -m pip install -r .github/constraints/deps.txt
+        python -m pip install -v .
+        python -m pip install -v --group doc
    - name: lint and build docs


This is really supposed to be build docs and not linting, if you remove the numpydoc line

sdtemple · 2026-09-08T20:37:08Z

+testing = [
    "ucimlrepo",
    "pytest",
    "pytest-cov",


Given that you like to use pytest-xdist, it might be worthwhile to add that to the test dependency.

sdtemple · 2026-09-08T20:37:28Z

+[dependency-groups]
+testing = [
    "ucimlrepo",
    "pytest",


It might be worth specifying a minimum version for pytest.

tylerjereddy · 2026-09-08T22:19:47Z

Just a suggestion--usually best to only review things that changed in the diff vs. stuff that was already there before re: #127 (comment) for example.

Ankan Ganguly added 2 commits September 1, 2026 13:42

BUG, WIP: After some testing, I fixed one minor bug. In the yml files,

311b6f2

I forgot to install the project and its dependencies.

aganguly-lab requested a review from sdtemple September 1, 2026 23:15

tylerjereddy added maintenance continuous integration labels Sep 1, 2026

tylerjereddy reviewed Sep 2, 2026

View reviewed changes

Ankan Ganguly added 2 commits September 2, 2026 18:35

MNT, CI, WIP: Added the doc, build and dev dependency-groups in respo…

9769ec1

…nse to #127 (comment) and updated the .yml files accordingly. Changes are untested, but I included this commit for easy retreval during testing.

MNT, CI: After testing, I've removed the build group. It seems redund…

350165d

…ant. Not sure why SciPy has it. I also removed some packages in the linting group that belonged purely in the doc group.

sdtemple requested changes Sep 8, 2026

View reviewed changes

Uh oh!

Conversation

aganguly-lab commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing the update

Uh oh!

tylerjereddy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdtemple left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tylerjereddy commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aganguly-lab commented Sep 1, 2026 •

edited

Loading