ci: verify the built artifact before publishing it - #89
Open
pofallon wants to merge 2 commits into
Open
Conversation
The release workflow's `test` job runs `uv sync --all-extras --group dev`,
which installs the *source tree*. `publish` then ran `uv build` and shipped
the result. Nothing between them ever installed the wheel, so the first
person to install a release was a user, and every packaging-layer bug
reached PyPI unopposed: a wrong `packages =`, a runtime dependency only the
dev group was satisfying, a broken extra, an sdist exclude that dropped
something needed at runtime.
Restructures to test → build → install-matrix → release → publish:
- `build` runs `uv build --no-sources`, then `twine check` and
`check-wheel-contents`, and uploads dist/ as an artifact. `--no-sources`
proves the build works with `tool.uv.sources` disabled, which is how
every other build tool sees the project. No sources are declared today,
so it is insurance rather than a fix.
- `install-matrix` installs that artifact into a clean venv across
{3.12, 3.13} x {no extras, each of the six extras, all} and runs
scripts/smoke_install.py against it.
- `publish` no longer checks out or rebuilds. It downloads and ships the
exact artifact the matrix exercised; a rebuild there could differ from
what was verified.
The bare "no extras" row is the point. It is the only configuration where
the vendor SDKs are genuinely absent, so it is the only place the
lazy-import invariant from CLAUDE.md can actually fail — and it is a
configuration CI has never once executed.
Install by wheel PATH, not by name. This was found the hard way while
testing locally: `--find-links` only *adds* to the index, so with PyPI
already carrying 0.9.2 and the local build at 0.9.1, the resolver picked
0.9.2 and the smoke test silently verified the previously-published
package. Appending the extra to the path still resolves that extra's
dependencies from PyPI, which is the behaviour wanted.
Also adds tests/test_lazy_imports.py so the invariant is enforced at PR
time rather than only at release. It probes in a subprocess because the
pytest process has already imported the vendor SDKs deliberately, which
would make an in-process sys.modules check pass vacuously. Verified it
fails: injecting a top-level `import openai` into openrouter.py turns 10
of the 11 tests red. The last test guards the guard — it asserts every
adapter's REQUIRES_PACKAGE appears in the watched list, so a new adapter
with a new SDK cannot silently create an unwatched eager-import path.
scripts/smoke_install.py deliberately does not assert an exact provider
roster; tests/test_discovery.py already does, on every PR. What only an
installed artifact can prove is that every adapter module the wheel claims
to serve is importable from the wheel, which it checks via runtime_for()
per provider.
Verified end to end locally against a real build: all seven extras rows
pass 15/15 checks, twine check and check-wheel-contents both clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UBTr6Q6kTGUMQiwwBHcdMj
v4 is several majors stale. Checked the release notes across v5-v8: the breaking changes are Node runtime bumps and download path/decompression behaviour, not input removals — name, path, retention-days and if-no-files-found are unchanged. download-artifact v8 also enforces artifact hash checks by default rather than warning on mismatch, which supplies the build-to-publish integrity guarantee this restructure depends on. Noted inline so the absence of a separate checksum step reads as deliberate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UBTr6Q6kTGUMQiwwBHcdMj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the gap that has been letting broken releases reach PyPI.
The hole
release.yml'stestjob runsuv sync --all-extras --group dev— it installs the source tree.publishthen ranuv buildand shipped the result. Nothing in between ever installed the wheel, so the first person to install a release was a user, and every packaging-layer bug sailed through:[tool.hatch.build.targets.wheel] packages =devgroup was satisfying[project.optional-dependencies]entryexcludethat dropped something needed at runtimeThe restructure
test→build→install-matrix→release→publishbuild—uv build --no-sources, thentwine check+check-wheel-contents, uploadsdist/as an artifact.--no-sourcesproves the build works withtool.uv.sourcesdisabled, which is how every other build tool (and therefore PyPI) sees the project. No sources are declared today, so it's insurance, not a fix.install-matrix— installs that artifact into a clean venv across {3.12, 3.13} × {no extras, each of the six,all} = 14 combinations, and runsscripts/smoke_install.py.publish— no checkout, no rebuild. Downloads and ships the exact artifact the matrix exercised.The bare "no extras" row is the point. It's the only configuration where the vendor SDKs are genuinely absent, so it's the only place the lazy-import invariant from CLAUDE.md can actually fail — and it's a configuration CI has never executed.
One bug found while building this
Install is by wheel path, not by name, and that's load-bearing. My first version used
uv pip install --find-links dist/ "airframe-agents[claude]".--find-linksonly adds to the index — PyPI already has 0.9.2, the local build is 0.9.1, so the resolver picked 0.9.2 and the smoke test silently verified the previously-published package. A gate that tests the wrong artifact is worse than no gate. Appending the extra to the path ("${WHEEL}[claude]") still resolves that extra's dependencies from PyPI, which is what we want.tests/test_lazy_imports.pySo the invariant is enforced at PR time, not only at release. Probes in a subprocess — the pytest process has already imported the vendor SDKs deliberately, so an in-process
sys.modulescheck would pass vacuously.Verified it can fail: injecting a top-level
import openaiintoopenrouter.pyturns 10 of the 11 tests red; reverting returns them to green. The 11th guards the guard — it asserts every adapter'sREQUIRES_PACKAGEappears in the watched list, so a new adapter with a new SDK can't silently create an unwatched eager-import path.On what the smoke script checks
It deliberately does not assert an exact provider roster —
tests/test_discovery.pyalready does that on every PR and doesn't need a release to catch a regression. What only an installed artifact can prove is that every adapter module the wheel claims to serve is importable from the wheel, so it callsruntime_for()per provider and requires either a class or the documentedImportErrornaming the extra.Verification
Ran the whole thing locally against a real build before pushing:
twine checkandcheck-wheel-contentsboth clean on the built sdist and wheel.make cigreen (1035 passed, +11 new).Note the workflow itself only executes on a
v*tag push, so this PR's CI will not exercise it — the local run above is the evidence.🤖 Generated with Claude Code
https://claude.ai/code/session_01UBTr6Q6kTGUMQiwwBHcdMj