Skip to content

ci: run the test matrix on the actual matrix interpreter, split lint from tests - #768

Open
mdwcrft wants to merge 1 commit into
masterfrom
cicd/python-version-matrix-fix
Open

ci: run the test matrix on the actual matrix interpreter, split lint from tests#768
mdwcrft wants to merge 1 commit into
masterfrom
cicd/python-version-matrix-fix

Conversation

@mdwcrft

@mdwcrft mdwcrft commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

The problem

The CI matrix has not been testing the Python versions it claims to. The setup-python-build-env composite action installs Poetry via pipx, and Poetry then builds the project virtualenv from pipx's own interpreter, not the one selected by actions/setup-python. Every matrix job — including the "3.8" one — has actually been running against the runner's default Python (3.12). This was spotted by @hikinggrass in the #740 discussion, with a job run demonstrating it: https://github.com/mobilityhouse/ocpp/actions/runs/16022303628/job/45201948389

Practical consequence: the matrix has never genuinely verified support below 3.11, which is how the classifier/requires_python drift shipped in 2.1.0 without any red CI.

The fix

.github/workflows/pull-request.yml

  • After setup-python, run poetry env use "$(which python)" so the project venv is built from the matrix interpreter
  • Add a verification step that asserts sys.version_info inside the venv matches matrix.version — the matrix can no longer silently lie; if the binding ever regresses, the job fails loudly
  • Trim the matrix to 3.11 / 3.12 / 3.13 — the range pyproject.toml actually declares (^3.11). The 3.8–3.10 jobs are removed rather than fixed because, once honest, they cannot pass today (poetry install refuses interpreters outside the project's range). They return in the follow-up to Update minimum python version (3.9) in pyproject.toml and docs #740 when the floor is genuinely lowered.
  • Split lint out of the matrix: a single lint job on 3.13 runs black/isort/flake8; matrix jobs run only the test suite. Lint results don't vary by interpreter, and this also unblocks the Update minimum python version (3.9) in pyproject.toml and docs #740 follow-up where lint tools (black 26 ≥3.10, sphinx ≥3.11) won't be installed on older-Python jobs.

Makefile

  • tests split into lint and unit-tests targets; make tests runs both, so local usage is unchanged

.github/workflows/publish-to-pypi.yml

  • Pinned interpreter bumped 3.10 → 3.13 — it was below the package's own ^3.11 floor and only worked via the same pipx accident — and Poetry is bound with poetry env use the same way

Verification

  • make tests (lint + 181 tests) passes locally through the new targets
  • The interpreter assertion one-liner validated standalone
  • This PR's own CI run exercises the new workflow

Sequencing

This is deliberately infrastructure-only — no pyproject.toml changes. It merges first so the matrix is trustworthy before the Python-floor change: the follow-up to #740 (lower floor to ^3.9, lockfile regen, dev-dependency markers) will extend the matrix to 3.9/3.10, and its green checks will then actually mean the suite runs on those interpreters.

🤖 Generated with Claude Code

… tests

The pull-request workflow installed Poetry via pipx, which built the
project virtualenv from pipx's own interpreter rather than the one
selected by actions/setup-python - every matrix job was actually
testing against the runner's default Python (spotted in #740's
discussion: the "3.8" job ran 3.12).

- Bind Poetry to the matrix interpreter with `poetry env use` and
  assert sys.version_info inside the venv so the matrix cannot
  silently drift again
- Trim the matrix to the versions pyproject.toml currently supports
  (^3.11); the untruthful 3.8-3.10 jobs are removed until the floor
  is genuinely lowered (#740)
- Split `make tests` into `lint` + `unit-tests`; lint runs once on
  3.13, matrix jobs run only the test suite. `make tests` still runs
  both, so local workflow is unchanged
- publish-to-pypi: bump the pinned interpreter 3.10 -> 3.13 (it was
  below the package's own ^3.11 floor and only worked via the same
  pipx accident) and bind Poetry the same way

Constraint: matrix must reflect pyproject's python range at every commit
Rejected: extending matrix to 3.9/3.10 here | fails until #740 lowers the floor
Confidence: high
Scope-risk: narrow
Not-tested: publish workflow path (only runs on tag push)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mdwcrft
mdwcrft requested a review from proelke as a code owner July 19, 2026 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant