ci: fix red CI on main (zarr pin, heatmap baseline, scanpy 1.13) - #1371
Merged
Conversation
The PETSc/SLEPc+R (conda) job force-downgraded to zarr<3 after installing cellrank, but current anndata (>=0.13) requires zarr>=3.1 and imports zarr.core.dtype, a zarr-v3-only module. The downgrade broke conftest import with ModuleNotFoundError, failing the whole job. The pin dates to when anndata still supported zarr v2; drop it and let anndata resolve zarr. Also mark the macOS matrix leg continue-on-error: pyGPCCA's brandts Schur decomposition is numerically unstable on Apple Silicon and fails the GPCCA tests there. The fix belongs upstream; keep the job for visibility but stop it gating merges. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `genes`-mode heatmap baseline dates to matplotlib 3.11.0; 3.11.1 shifts the constrained_layout placement enough to exceed the 50 RMS tolerance while the plotted content is unchanged. Promoted from the Linux py3.12-stable `rendered-figures` artifact, as `docs/contributing.md` prescribes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scanpy 1.13 moves the legacy plotting API under `scanpy.plotting.legacy`, breaking the two private imports cellrank relies on (`palettes.vega_20_scanpy` and `_utils.add_colors_for_categorical_sample_annotation`) — collection of the whole test suite failed on the pre-release job. Route both through a small `_utils/_compat` shim that tries the new path first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pyGPCCA `brandts` failures that motivated making macOS non-blocking no longer reproduce: on the latest `main` run the macOS leg is green apart from the shared heatmap baseline drift, so keep it gating. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cellrank pulled two private symbols out of scanpy — `palettes.vega_20_scanpy` and `_utils.add_colors_for_categorical_sample_annotation` — both of which scanpy 1.13 relocated to `scanpy.plotting.legacy`, breaking collection of the whole suite on the pre-release job. Neither needs to come from scanpy: the palette is a fixed reordering of matplotlib's `tab20`, now defined (and pinned by a test) in `_utils/_colors.py`, and the colour assignment is what cellrank's own `_get_categorical_colors` already does for the plotting functions. That helper moves from `pl/_utils` to `_utils/_colors` so `models/` can use it without importing from `pl/` and now also regenerates palettes that are too short or not colour-like, which is the part scanpy's helper contributed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CI has been red on
mainsince the start of August, blocking the All tests pass gate on every PR. Three independent causes:PETSc / SLEPc + R(conda job) — fails at conftest import:The job installs cellrank (
pip install -e '.[plot]', pulling anndata 0.13.2, which requires zarr>=3.1), then the next line force-downgrades to'zarr<3'. anndata 0.13 importszarr.core.dtype, a zarr-v3-only module, so the downgrade breaks the whole job. The pin dates to CI workflow modernization #1291 (Feb 2026), when anndata still supported zarr v2 — it's now stale.All
hatch-teststable legs (Linux 3.12/3.14 and macOS) —TestHeatmap::test_heatmap_genesfails with RMS 67.4 against a 50 tolerance, identically on all three. The baseline was rendered with matplotlib 3.11.0; 3.11.1 (in the env since the 2026-08-01 scheduled run) shifts theconstrained_layoutplacement of thegenes-mode heatmap. The plotted content is unchanged — only the layout moved.hatch-test.py3.14-pre— collection of the entire suite fails:scanpy 1.13.0a1 moves the legacy plotting API under
scanpy.plotting.legacy. cellrank reached into two private scanpy symbols:palettes.vega_20_scanpyand_utils.add_colors_for_categorical_sample_annotation.Changes
zarr<3pin in the conda job and let anndata resolve zarr (>=3.1).heatmap_genesbaseline, promoted from the Linuxpy3.12-stablerendered-figuresartifact asdocs/contributing.mdprescribes. Reviewed against the old baseline: same curves, same ordering, same colour mapping — layout offset only.tab20, so it now lives in_utils/_colors.py, pinned by a test (the committed figure baselines depend on those exact values, and it is byte-identical to scanpy'svega_20_scanpyin both 1.12 and 1.13). The colour assignment is what cellrank's own_get_categorical_colorsalready did for the plotting functions, so that helper moves frompl/_utilsto_utils/_colors—models/can't import frompl/without a cycle — and gains the one behaviour scanpy's helper contributed: regenerating palettes that are too short or not colour-like.brandtsfailures that originally motivatedcontinue-on-errorfor macOS no longer reproduce — on the latestmainrun macOS is green apart from the shared baseline drift, so it stays a gating job.Verification
Locally on Linux, both matrix envs pass the full suite:
py3.12-stable(matplotlib 3.11.1, scanpy 1.12.3) andpy3.14-pre(scanpy 1.13.0a1), the latter having previously failed at collection. The refreshed baseline is byte-identical to what this machine renders.🤖 Generated with Claude Code