Skip to content

ci: fix red CI on main (zarr pin, heatmap baseline, scanpy 1.13) - #1371

Merged
Marius1311 merged 6 commits into
mainfrom
ci/fix-zarr-pin-accept-macos-gpcca
Aug 24, 2026
Merged

ci: fix red CI on main (zarr pin, heatmap baseline, scanpy 1.13)#1371
Marius1311 merged 6 commits into
mainfrom
ci/fix-zarr-pin-accept-macos-gpcca

Conversation

@Marius1311

@Marius1311 Marius1311 commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Problem

CI has been red on main since the start of August, blocking the All tests pass gate on every PR. Three independent causes:

  1. PETSc / SLEPc + R (conda job) — fails at conftest import:

    ModuleNotFoundError: No module named 'zarr.core.dtype'; 'zarr.core' is not a package
    

    The job installs cellrank (pip install -e '.[plot]', pulling anndata 0.13.2, which requires zarr>=3.1), then the next line force-downgrades to 'zarr<3'. anndata 0.13 imports zarr.core.dtype, a zarr-v3-only module, so the downgrade breaks the whole job. The pin dates to CI workflow modernization #1291 (Feb 2026), when anndata still supported zarr v2 — it's now stale.

  2. All hatch-test stable legs (Linux 3.12/3.14 and macOS) — TestHeatmap::test_heatmap_genes fails with RMS 67.4 against a 50 tolerance, identically on all three. The baseline was rendered with matplotlib 3.11.0; 3.11.1 (in the env since the 2026-08-01 scheduled run) shifts the constrained_layout placement of the genes-mode heatmap. The plotted content is unchanged — only the layout moved.

  3. hatch-test.py3.14-pre — collection of the entire suite fails:

    ModuleNotFoundError: No module named 'scanpy.plotting.palettes'
    

    scanpy 1.13.0a1 moves the legacy plotting API under scanpy.plotting.legacy. cellrank reached into two private scanpy symbols: palettes.vega_20_scanpy and _utils.add_colors_for_categorical_sample_annotation.

Changes

  • Drop the zarr<3 pin in the conda job and let anndata resolve zarr (>=3.1).
  • Refresh the heatmap_genes baseline, promoted from the Linux py3.12-stable rendered-figures artifact as docs/contributing.md prescribes. Reviewed against the old baseline: same curves, same ordering, same colour mapping — layout offset only.
  • Stop importing scanpy's private plotting internals rather than chasing their new location. The palette is a fixed reordering of matplotlib's tab20, so it now lives in _utils/_colors.py, pinned by a test (the committed figure baselines depend on those exact values, and it is byte-identical to scanpy's vega_20_scanpy in both 1.12 and 1.13). The colour assignment is what cellrank's own _get_categorical_colors already did for the plotting functions, so that helper moves from pl/_utils to _utils/_colorsmodels/ can't import from pl/ without a cycle — and gains the one behaviour scanpy's helper contributed: regenerating palettes that are too short or not colour-like.
  • Keep the macOS leg blocking. The pyGPCCA brandts failures that originally motivated continue-on-error for macOS no longer reproduce — on the latest main run macOS is green apart from the shared baseline drift, so it stays a gating job.

Verification

Locally on Linux, both matrix envs pass the full suite: py3.12-stable (matplotlib 3.11.1, scanpy 1.12.3) and py3.14-pre (scanpy 1.13.0a1), the latter having previously failed at collection. The refreshed baseline is byte-identical to what this machine renders.

🤖 Generated with Claude Code

Marius1311 and others added 5 commits July 16, 2026 16:21
The PETSc/SLEPc+R (conda) job force-downgraded to zarr<3 after installing
cellrank, but current anndata (>=0.13) requires zarr>=3.1 and imports
zarr.core.dtype, a zarr-v3-only module. The downgrade broke conftest import
with ModuleNotFoundError, failing the whole job. The pin dates to when anndata
still supported zarr v2; drop it and let anndata resolve zarr.

Also mark the macOS matrix leg continue-on-error: pyGPCCA's brandts Schur
decomposition is numerically unstable on Apple Silicon and fails the GPCCA
tests there. The fix belongs upstream; keep the job for visibility but stop it
gating merges.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `genes`-mode heatmap baseline dates to matplotlib 3.11.0; 3.11.1 shifts the
constrained_layout placement enough to exceed the 50 RMS tolerance while the
plotted content is unchanged. Promoted from the Linux py3.12-stable
`rendered-figures` artifact, as `docs/contributing.md` prescribes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scanpy 1.13 moves the legacy plotting API under `scanpy.plotting.legacy`,
breaking the two private imports cellrank relies on (`palettes.vega_20_scanpy`
and `_utils.add_colors_for_categorical_sample_annotation`) — collection of the
whole test suite failed on the pre-release job. Route both through a small
`_utils/_compat` shim that tries the new path first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pyGPCCA `brandts` failures that motivated making macOS non-blocking no
longer reproduce: on the latest `main` run the macOS leg is green apart from
the shared heatmap baseline drift, so keep it gating.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Marius1311 Marius1311 changed the title ci: fix PETSc/conda zarr pin; accept macOS GPCCA instability ci: fix red CI on main (zarr pin, heatmap baseline, scanpy 1.13) Aug 24, 2026
cellrank pulled two private symbols out of scanpy — `palettes.vega_20_scanpy` and
`_utils.add_colors_for_categorical_sample_annotation` — both of which scanpy 1.13
relocated to `scanpy.plotting.legacy`, breaking collection of the whole suite on the
pre-release job.

Neither needs to come from scanpy: the palette is a fixed reordering of matplotlib's
`tab20`, now defined (and pinned by a test) in `_utils/_colors.py`, and the colour
assignment is what cellrank's own `_get_categorical_colors` already does for the
plotting functions. That helper moves from `pl/_utils` to `_utils/_colors` so
`models/` can use it without importing from `pl/` and now also regenerates palettes
that are too short or not colour-like, which is the part scanpy's helper contributed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Marius1311
Marius1311 merged commit ab23d63 into main Aug 24, 2026
10 checks passed
@Marius1311
Marius1311 deleted the ci/fix-zarr-pin-accept-macos-gpcca branch August 24, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant