Skip to content

ci: verify the built artifact before publishing it - #89

Open
pofallon wants to merge 2 commits into
mainfrom
ci/verify-built-artifact-before-publish
Open

ci: verify the built artifact before publishing it#89
pofallon wants to merge 2 commits into
mainfrom
ci/verify-built-artifact-before-publish

Conversation

@pofallon

Copy link
Copy Markdown
Contributor

Fixes the gap that has been letting broken releases reach PyPI.

The hole

release.yml's test job runs uv sync --all-extras --group dev — it installs the source tree. publish then ran uv build and shipped the result. Nothing in between ever installed the wheel, so the first person to install a release was a user, and every packaging-layer bug sailed through:

  • a wrong [tool.hatch.build.targets.wheel] packages =
  • a runtime dependency that only the dev group was satisfying
  • a broken [project.optional-dependencies] entry
  • an sdist exclude that dropped something needed at runtime

The restructure

testbuildinstall-matrixreleasepublish

  • builduv build --no-sources, then twine check + check-wheel-contents, uploads dist/ as an artifact. --no-sources proves the build works with tool.uv.sources disabled, which is how every other build tool (and therefore PyPI) sees the project. No sources are declared today, so it's insurance, not a fix.
  • install-matrix — installs that artifact into a clean venv across {3.12, 3.13} × {no extras, each of the six, all} = 14 combinations, and runs scripts/smoke_install.py.
  • publish — no checkout, no rebuild. Downloads and ships the exact artifact the matrix exercised.

The bare "no extras" row is the point. It's the only configuration where the vendor SDKs are genuinely absent, so it's the only place the lazy-import invariant from CLAUDE.md can actually fail — and it's a configuration CI has never executed.

One bug found while building this

Install is by wheel path, not by name, and that's load-bearing. My first version used uv pip install --find-links dist/ "airframe-agents[claude]". --find-links only adds to the index — PyPI already has 0.9.2, the local build is 0.9.1, so the resolver picked 0.9.2 and the smoke test silently verified the previously-published package. A gate that tests the wrong artifact is worse than no gate. Appending the extra to the path ("${WHEEL}[claude]") still resolves that extra's dependencies from PyPI, which is what we want.

tests/test_lazy_imports.py

So the invariant is enforced at PR time, not only at release. Probes in a subprocess — the pytest process has already imported the vendor SDKs deliberately, so an in-process sys.modules check would pass vacuously.

Verified it can fail: injecting a top-level import openai into openrouter.py turns 10 of the 11 tests red; reverting returns them to green. The 11th guards the guard — it asserts every adapter's REQUIRES_PACKAGE appears in the watched list, so a new adapter with a new SDK can't silently create an unwatched eager-import path.

On what the smoke script checks

It deliberately does not assert an exact provider roster — tests/test_discovery.py already does that on every PR and doesn't need a release to catch a regression. What only an installed artifact can prove is that every adapter module the wheel claims to serve is importable from the wheel, so it calls runtime_for() per provider and requires either a class or the documented ImportError naming the extra.

Verification

Ran the whole thing locally against a real build before pushing:

(no extras)      exit=0  passed=15
[claude]         exit=0  passed=15
[copilot]        exit=0  passed=15
[openai-compat]  exit=0  passed=15
[bedrock]        exit=0  passed=15
[opencode]       exit=0  passed=15
[all]            exit=0  passed=15
OVERALL: PASS

twine check and check-wheel-contents both clean on the built sdist and wheel. make ci green (1035 passed, +11 new).

Note the workflow itself only executes on a v* tag push, so this PR's CI will not exercise it — the local run above is the evidence.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UBTr6Q6kTGUMQiwwBHcdMj

The release workflow's `test` job runs `uv sync --all-extras --group dev`,
which installs the *source tree*. `publish` then ran `uv build` and shipped
the result. Nothing between them ever installed the wheel, so the first
person to install a release was a user, and every packaging-layer bug
reached PyPI unopposed: a wrong `packages =`, a runtime dependency only the
dev group was satisfying, a broken extra, an sdist exclude that dropped
something needed at runtime.

Restructures to test → build → install-matrix → release → publish:

- `build` runs `uv build --no-sources`, then `twine check` and
  `check-wheel-contents`, and uploads dist/ as an artifact. `--no-sources`
  proves the build works with `tool.uv.sources` disabled, which is how
  every other build tool sees the project. No sources are declared today,
  so it is insurance rather than a fix.
- `install-matrix` installs that artifact into a clean venv across
  {3.12, 3.13} x {no extras, each of the six extras, all} and runs
  scripts/smoke_install.py against it.
- `publish` no longer checks out or rebuilds. It downloads and ships the
  exact artifact the matrix exercised; a rebuild there could differ from
  what was verified.

The bare "no extras" row is the point. It is the only configuration where
the vendor SDKs are genuinely absent, so it is the only place the
lazy-import invariant from CLAUDE.md can actually fail — and it is a
configuration CI has never once executed.

Install by wheel PATH, not by name. This was found the hard way while
testing locally: `--find-links` only *adds* to the index, so with PyPI
already carrying 0.9.2 and the local build at 0.9.1, the resolver picked
0.9.2 and the smoke test silently verified the previously-published
package. Appending the extra to the path still resolves that extra's
dependencies from PyPI, which is the behaviour wanted.

Also adds tests/test_lazy_imports.py so the invariant is enforced at PR
time rather than only at release. It probes in a subprocess because the
pytest process has already imported the vendor SDKs deliberately, which
would make an in-process sys.modules check pass vacuously. Verified it
fails: injecting a top-level `import openai` into openrouter.py turns 10
of the 11 tests red. The last test guards the guard — it asserts every
adapter's REQUIRES_PACKAGE appears in the watched list, so a new adapter
with a new SDK cannot silently create an unwatched eager-import path.

scripts/smoke_install.py deliberately does not assert an exact provider
roster; tests/test_discovery.py already does, on every PR. What only an
installed artifact can prove is that every adapter module the wheel claims
to serve is importable from the wheel, which it checks via runtime_for()
per provider.

Verified end to end locally against a real build: all seven extras rows
pass 15/15 checks, twine check and check-wheel-contents both clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UBTr6Q6kTGUMQiwwBHcdMj
v4 is several majors stale. Checked the release notes across v5-v8: the
breaking changes are Node runtime bumps and download path/decompression
behaviour, not input removals — name, path, retention-days and
if-no-files-found are unchanged.

download-artifact v8 also enforces artifact hash checks by default rather
than warning on mismatch, which supplies the build-to-publish integrity
guarantee this restructure depends on. Noted inline so the absence of a
separate checksum step reads as deliberate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UBTr6Q6kTGUMQiwwBHcdMj
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant