Skip to content

fix(ci): close release/wheel smoke harness drift gap (#995) #996

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dekobon merged 3 commits into from
Jun 30, 2026
Merged

fix(ci): close release/wheel smoke harness drift gap (#995) #996

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6e23d46
test(ci): close release/wheel smoke drift gap
dekobon Jun 29, 2026
c5d42d6
docs(lessons): add #80 on release-time smoke drift
dekobon Jun 30, 2026
d6bd8c3
fix(ci): address review findings on smoke harness
dekobon Jun 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 17 additions & 27 deletions .github/workflows/python-cli-wheels.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,16 @@ jobs:
runs-on: ${{ matrix.runs-on }}
timeout-minutes: 15
steps:
# Sparse, submodule-free checkout of just the smoke harness so the
# step below can run `scripts/smoke/cli_wheel_smoke.sh` against the
# downloaded wheel (#995 — extracted from the former inline `run:`
# block so the assertions are shellcheck-lintable and also run
# per-PR via smoke-dryrun.yml).
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
sparse-checkout: scripts/smoke
submodules: false

- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: ${{ env.PYTHON_VERSION_HOST }}
Expand All @@ -334,33 +344,13 @@ jobs:
# rather than silently installing a stale published version.
python -m pip install --no-index --find-links=dist big-code-analysis-cli
# The console script lands in the setup-python interpreter's
# scripts dir, which is on PATH. Capture the version rather than
# piping to `head` (a `| head -n1` would SIGPIPE the producer
# under pipefail — see release.yml's note).
ver_out=$(bca --version)
echo "$ver_out"
# On a tag build, prove the binary reports the lockstep release
# version (maturin reads `version.workspace = true`, so the
# wheel version must equal the tag minus its leading `v`).
if [[ -n "$EXPECTED_TAG" ]]; then
want="${EXPECTED_TAG#v}"
case "$ver_out" in
*"$want"*) : ;;
*) echo "::error::bca --version '$ver_out' does not contain tag version $want"; exit 1 ;;
esac
fi
bca list-metrics names >/dev/null
# Parse two unrelated languages to prove the all-languages
# grammar set is compiled in, not just the host language.
printf 'def add(a, b):\n if a > b:\n return a\n return b\n' > smoke.py
printf 'fn main() { if true { println!("x"); } }\n' > smoke.rs
py_cc=$(bca metrics --paths smoke.py -O json | python -c "import sys,json; print(json.load(sys.stdin)['metrics']['cyclomatic']['sum'])")
rs_cc=$(bca metrics --paths smoke.rs -O json | python -c "import sys,json; print(json.load(sys.stdin)['metrics']['cyclomatic']['sum'])")
# 2.0 serializes integer-valued metrics as integers (#530),
# so cyclomatic.sum prints as `3`, not the pre-2.0 `3.0`.
test "$py_cc" = "3" || { echo "::error::python cyclomatic.sum=$py_cc, expected 3"; exit 1; }
test "$rs_cc" = "3" || { echo "::error::rust cyclomatic.sum=$rs_cc, expected 3"; exit 1; }
echo "smoke OK"
# scripts dir, which is on PATH. The assertions (version
# lockstep against EXPECTED_TAG, the all-languages grammar set,
# and the #530 integer-metric serialization) live in the
# checked-in smoke harness so they also run per-PR via
# smoke-dryrun.yml (#995). Invoke via `bash` so the Windows
# matrix leg (Git Bash, no executable bit) runs it too.
bash scripts/smoke/cli_wheel_smoke.sh

# Aggregate gate so branch protection can require a single check name.
# Every dependency is label-gated at PR time, so on an unlabelled PR all
Expand Down
116 changes: 19 additions & 97 deletions .github/workflows/python-wheels.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,12 +256,18 @@ jobs:
runs-on: ${{ matrix.runs-on }}
timeout-minutes: 15
steps:
# No `actions/checkout` — the smoke step reads only the
# downloaded wheel artefact and runs an inline Python heredoc;
# no repo file (fixture, helper, doc) is touched. Skipping the
# checkout saves ~10-20 s per matrix leg on aarch64 (the
# native-ARM runner's clone over LFS-adjacent submodules takes
# the wall-clock penalty even when we skip submodules).
# Sparse, submodule-free checkout of just the smoke harness. The
# smoke step runs `scripts/smoke/lib_wheel_smoke.py` against the
# downloaded wheel (#995 — extracted from the former inline heredoc
# so the assertions are lintable and also run per-PR via
# smoke-dryrun.yml). `sparse-checkout` + `submodules: false` keep
# the aarch64 native-ARM clone cheap: the wall-clock cost this job
# historically skipped was the full clone's LFS-adjacent submodules,
# not the handful of files under scripts/smoke.
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
sparse-checkout: scripts/smoke
submodules: false
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: ${{ matrix.python }}
Expand All @@ -286,103 +292,19 @@ jobs:
# analyze_source returns a dict, flatten_spaces yields function
# records, to_sarif emits SARIF JSON, language_for_file resolves
# extensions. If any of these regress the smoke fails before
# the wheel can publish.
#
# Assertions are deliberately VALUE-bearing, not just shape-
# bearing — a binding regression that returned `{"spaces": []}`
# would satisfy a bare `isinstance(result, dict)` check but
# fail the `records[*]["name"] == "add"` check below.
# the wheel can publish. The assertions are deliberately
# VALUE-bearing, not just shape-bearing — see the per-API
# `_check_*` helpers in scripts/smoke/lib_wheel_smoke.py.
- name: Import + analyse smoke
env:
# Force unbuffered stdout so failure assertions show up
# immediately in the action log instead of after the python
# process has been torn down.
PYTHONUNBUFFERED: "1"
run: |
python - <<'PY'
import json
import tempfile
from pathlib import Path

import big_code_analysis as bca

# Sanity: package + extension loaded.
assert bca.__version__, repr(bca.__version__)
assert "python" in bca.supported_languages()
# Extensions are bare names (no leading dot) — match the
# `language_extensions` contract in the stub. The CLI used
# to expose dotted forms; the binding does not.
assert "py" in bca.language_extensions("python"), bca.language_extensions("python")

# `language_for_file` reads the file (parity with `analyze`,
# #318), so a stub path that does not exist raises
# FileNotFoundError. Materialise an empty fixture in a
# tempdir so the assertion exercises the extension table
# path rather than crashing on I/O.
with tempfile.TemporaryDirectory() as td:
fixture = Path(td) / "foo.py"
fixture.write_bytes(b"")
resolved = bca.language_for_file(fixture)
assert resolved == "python", resolved

# `language` is a positional-only parameter on
# `analyze_source`; passing it as a kwarg raises TypeError.
src = "def add(a, b):\n if a > b:\n return a\n return b\n"
result = bca.analyze_source(src, "python")
assert isinstance(result, dict), type(result)
assert result.get("kind") == "unit", result.get("kind")
# The unit-level cyclomatic is 3 for this fixture (one
# function spanning the file + one `if` branch). A binding
# regression that produced null / zero metrics would slip
# past `isinstance(result, dict)` but trip this assertion.
cyclomatic_sum = result["metrics"]["cyclomatic"]["sum"]
assert cyclomatic_sum == 3.0, f"expected unit cyclomatic.sum == 3.0, got {cyclomatic_sum}"

# `flatten_spaces` yields one record per space; for this
# fixture that is the unit + the `add` function. The
# function record's cyclomatic.sum is 2 (1 entry + 1 if).
records = list(bca.flatten_spaces(result))
assert records, "flatten_spaces returned no records"
add_records = [r for r in records if r.get("name") == "add"]
assert add_records, f"no 'add' function in records: {[r.get('name') for r in records]}"
assert add_records[0].get("kind") == "function", add_records[0].get("kind")
assert add_records[0].get("cyclomatic.sum") == 2.0, add_records[0].get("cyclomatic.sum")

# SARIF must be a well-formed 2.1.0 document with at least
# one run carrying the bca tool driver. A stub
# `{"version": "2.1.0"}` (the most plausible regression of
# an empty-run SARIF writer) would fail the runs check.
sarif_obj = json.loads(bca.to_sarif(result))
assert sarif_obj.get("version") == "2.1.0", sarif_obj.get("version")
runs = sarif_obj.get("runs") or []
assert len(runs) == 1, f"expected 1 SARIF run, got {len(runs)}"
driver = runs[0].get("tool", {}).get("driver", {})
assert driver.get("name"), f"SARIF run missing tool.driver.name: {driver}"

# analyze_batch's never-raise contract: a missing path in
# the batch must be returned as an AnalysisFailure instance
# interleaved with the successful dicts, never as an
# exception. A binding regression that switched batch back
# to raising would crash here; one that dropped error
# records entirely would shrink `len(batch)` below the
# input count and trip the assertion.
with tempfile.TemporaryDirectory() as td:
good = Path(td) / "good.py"
good.write_text("def f():\n pass\n")
missing = Path(td) / "does-not-exist.py"
batch = bca.analyze_batch([good, missing])
assert len(batch) == 2, f"expected 2 batch results, got {len(batch)}"
# Good path → dict.
assert isinstance(batch[0], dict), type(batch[0])
# Missing path → AnalysisFailure with `error_kind == "IoError"`
# (the type was renamed from AnalysisError at 2.0, #614).
err = batch[1]
assert isinstance(err, bca.AnalysisFailure), type(err)
assert err.error_kind == "IoError", err.error_kind
assert str(missing) in err.path, (err.path, str(missing))

print("smoke OK:", bca.__version__)
PY
# The assertions live in a checked-in, mypy-strict, ruff-linted
# script (#995) so a binding rename/serialization change reds a
# PR via smoke-dryrun.yml instead of only this tag-gated job.
run: python scripts/smoke/lib_wheel_smoke.py

# pip-audit deliberately NOT run here. The pre-publish wheel
# has zero Python runtime deps (the binary surface is the
Expand Down
108 changes: 108 additions & 0 deletions .github/workflows/smoke-dryrun.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
name: Smoke dry-run

# Per-PR drift gate for the release/wheel smoke harnesses (#995).
#
# The wheel + release smoke scripts only ran on a `v*` tag push (or via an
# opt-in `python-*-wheels` PR label), so an assertion that fell out of step
# with a breaking change rotted silently until release day. Three did,
# blocking the `v2.0.0` cut: the #530 integer-metric serialization (CLI
# smoke still expected `"3.0"`) and the #614 `AnalysisError` ->
# `AnalysisFailure` rename (library smoke still referenced the old name).
#
# This job runs the *same* checked-in smoke scripts the wheel workflows now
# invoke (scripts/smoke/*), but against a cheap local dev build instead of a
# packaged wheel — `cargo build` for the CLI and `maturin develop` for the
# bindings. It fires only when the wheel-smoke plumbing or the smoke scripts
# themselves change, so the common case pays nothing.
#
# What this gate catches: drift between the extracted smoke scripts and the
# code they assert against, on a PR that edits the scripts or the wheel
# workflows that invoke them. It is the script/plumbing half of #995's fix.
# The *other* half — a src-side metric rename or serialization change (the
# #530 / #614 class) — is caught on every PR by the per-PR unit/integration
# tests (`cli_metrics_json_serializes_integer_metrics_as_integers` in
# big-code-analysis-cli/tests/format_smoke.rs and the `AnalysisFailure`
# assertions in big-code-analysis-py/tests/test_batch.py), which run under
# normal CI whenever src/** changes. This job does not trigger on src/**,
# by design: those tests already guard that surface, and widening the filter
# would rebuild a wheel on nearly every PR.
#
# Scope note: this validates smoke-script-vs-code drift, not OS-package
# *packaging* (the `.apk` basename collision of bug #1). `release.yml` is
# deliberately NOT a trigger here — its deb/rpm/apk smoke is package-manager-
# specific, not extracted into scripts/smoke/*, so a dev-build run would
# validate nothing in it and produce a hollow green check. Building a signed
# multi-arch `.apk` needs the Alpine/abuild toolchain release.yml carries and
# is too heavy for a per-PR job; that path stays tag-gated, with the arch-
# qualified asset name (c53e504b) guarding the specific regression.

on:
pull_request:
paths:
- '.github/workflows/python-wheels.yml'
- '.github/workflows/python-cli-wheels.yml'
- '.github/workflows/smoke-dryrun.yml'
- 'scripts/smoke/**'
workflow_dispatch:

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
CARGO_TERM_COLOR: always
CARGO_INCREMENTAL: 0
CARGO_NET_RETRY: 10
RUSTUP_MAX_RETRIES: 10
PYTHON_VERSION_HOST: "3.12"

jobs:
smoke:
name: smoke (dev build)
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
# `submodules: false` — the smoke fixtures are synthesised in-process
# (scripts/smoke/*); the test-repository submodules are not needed and
# are the slow part of a full clone.
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
submodules: false

- uses: dtolnay/rust-toolchain@67ef31d5b988238dd797d409d6f9574278e20537 # stable tip
with:
toolchain: stable

- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1

- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: ${{ env.PYTHON_VERSION_HOST }}

# CLI smoke against a debug `bca` (the wheel workflow runs the same
# script against the packaged console binary). Runs before the venv
# below so `python` resolves to the setup-python interpreter.
- name: CLI smoke (cargo build)
env:
PYTHONUNBUFFERED: "1"
run: |
set -euo pipefail
cargo build -p big-code-analysis-cli
BCA="$PWD/target/debug/bca" bash scripts/smoke/cli_wheel_smoke.sh

# Library smoke against a `maturin develop` build of the bindings (the
# wheel workflow runs the same script against the packaged abi3 wheel).
- name: Library smoke (maturin develop)
env:
PYTHONUNBUFFERED: "1"
run: |
set -euo pipefail
python -m venv .venv
# shellcheck disable=SC1091 # venv activate script is created above
. .venv/bin/activate
python -m pip install --upgrade pip maturin
maturin develop --manifest-path big-code-analysis-py/Cargo.toml
python scripts/smoke/lib_wheel_smoke.py
19 changes: 19 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,25 @@ for historical reference.

## [Unreleased]

### Added

- Per-PR coverage for the release/wheel smoke harnesses, closing the
drift gap that let three stale assertions block the `v2.0.0` cut
(#995). The integer-valued-metric JSON serialization (#530) is now
pinned by a CLI integration test
(`cli_metrics_json_serializes_integer_metrics_as_integers`) that
asserts `cyclomatic.sum` serializes as a JSON integer (`is_u64()`),
which the existing `as_f64()`-coercing round-trip tests could not
catch. The previously-inline library and CLI wheel smokes were
extracted into checked-in, lint-gated scripts under
[`scripts/smoke/`](scripts/smoke/) (mypy `--strict` / ruff for the
Python script, shellcheck for the shell script), referenced by both
wheel workflows and runnable locally via `make smoke`. A new
path-filtered [`smoke-dryrun.yml`](.github/workflows/smoke-dryrun.yml)
workflow runs those scripts against a cheap dev build on any PR that
touches the release/wheel plumbing, so a future metric rename or
serialization change reds a PR check instead of a release.

## [2.0.0] - 2026-06-29

The project's first major version bump since `1.0`, and the first
Expand Down
Loading
Loading