Skip to content

Add test, pre-commit, uv-lock-check, and dependency-update workflows - #15

Merged
shekharprateek merged 2 commits into
mainfrom
ci/full-test-suite
Sep 15, 2026
Merged

shekharprateek merged 2 commits into
mainfrom
ci/full-test-suite

Conversation

@shekharprateek

@shekharprateek shekharprateek commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Summary

PR #14 imported 21 test files and nothing was running them. This closes that gap, and picks up two loose ends along the way.

Workflow What it does
test.yml The PR merge gate — matrix over benchmarks/ and self-hosted/vllm/ running uv sync, ruff check, mypy, unittest discover
pre-commit.yml + .pre-commit-config.yaml Ruff lint/format plus trailing-whitespace, EOF, merge-marker and YAML hooks
uv-lock-check.yml Fails a PR whose pyproject.toml changed without a matching uv lock
dependency-update.yml Weekly uv lock refresh, 7-day supply-chain quarantine on new releases

main's lint.yml is left untouched — it carries the Bandit test-dir exclude and per-job permissions blocks added in PR #14.

Also in this PR

bedrock/ deleted. This was approved for PR #13, but my commit landed after that PR was merged so the directory survived. It came from the original sample-claude-code-multi-model repo — Amit's fork never had it — and its LiteLLM proxy setup is superseded by benchmarks/config/litellm-mantle.yaml and benchmarks/scripts/bedrock-mantle-proxy.sh. Two of its files also failed ruff-format, so deleting it resolves those. Verified no markdown links point into it.

AGENTS.md — dropped the bedrock/ entry, dropped the stale Ollama mention (removed in PR #13), added vend/swe-router/.

Fixes needed to get the suite green

Ruff lint (2 auto-fixes). extract-metrics.py had an unused glob import — the rglob calls in it are Path methods, not the module — and replay-benchmark.py had an f"..." with no placeholders.

Ruff format (2 files). Same two scripts. Formatted with ruff 0.14.0, the version .pre-commit-config.yaml pins. Worth noting: ruff 0.16+ also reformats Python inside markdown code fences, which would have rewritten the style examples in .claude/skills/*/SKILL.md and collapsed them to one line — directly contradicting the "one parameter per line" convention in CLAUDE.md. Pin-matching matters here.

Removed tests/test_build_vended_models.py. It asserts the committed vend/swe-router/models.json can be regenerated from source, which needs per-tier scores out of benchmarks/swe-benchmark-data/ run-summary files — the 1790-file raw dataset deliberately excluded from this repo. Importing just docs/metrics/pareto-frontier-omp-swe3.json is not enough; score_by_complexity comes back empty. The generator script build_vended_models.py is kept, since someone benchmarking their own repo would run it.

Verified

All 10 checks green, including both Test: legs. The one failure I saw locally — test_judge_common.test_lowercase_preferred_on_collision — is macOS-only: it creates EXPERT_REVIEW.md and expert_review.md in one directory, which a case-insensitive filesystem collapses into a single file. On ubuntu-latest both exist and it passes, which CI confirms.

The 21 test files imported in PR #14 had no CI running them. test.yml closes
that gap: a matrix over benchmarks/ and self-hosted/vllm/ running uv sync,
ruff check, mypy, and unittest discover.

Workflows added:
- test.yml: the PR merge gate (ruff + mypy + unittest per uv project)
- pre-commit.yml + .pre-commit-config.yaml: ruff lint/format and whitespace hooks
- uv-lock-check.yml: fails a PR whose pyproject.toml changed without uv lock
- dependency-update.yml: weekly uv lock refresh with a 7-day supply-chain quarantine

Left main's lint.yml untouched (it carries the Bandit test-dir exclude and the
per-job permissions blocks added in PR #14).

Two fixes needed to make the suite green:
- ruff: removed an unused glob import in extract-metrics.py and an extraneous
  f-prefix in replay-benchmark.py. Both are local-only scripts that predate the
  fork import, so they never went through its CI.
- removed tests/test_build_vended_models.py. It asserts the committed
  vend/swe-router/models.json can be regenerated, which needs per-tier scores
  from benchmarks/swe-benchmark-data/ run-summary files -- the 1790-file raw
  dataset deliberately excluded from this repo. The generator script itself is
  kept, since a user benchmarking their own repo would run it.

Verified locally: benchmarks 445 tests pass, self-hosted/vllm 45 tests pass,
ruff and mypy clean on both, both uv.lock files in sync.
Follow-ups needed to get pre-commit green on this branch.

bedrock/ deletion: this was approved for PR #13 but my commit landed after that
PR was merged, so the directory survived. It is carried over from the original
sample-claude-code-multi-model repo -- Amit's fork never had it -- and its
LiteLLM proxy setup is superseded by benchmarks/config/litellm-mantle.yaml and
benchmarks/scripts/bedrock-mantle-proxy.sh. Two of its files also failed
ruff-format, so deleting it resolves those.

ruff-format on extract-metrics.py and replay-benchmark.py: both are local-only
scripts that predate the fork import and never went through its CI. Formatted
with ruff 0.14.0 to match the version .pre-commit-config.yaml pins -- note that
ruff 0.16+ also reformats Python inside markdown code fences, which would have
rewritten the style examples in .claude/skills/*/SKILL.md and contradicted the
one-parameter-per-line convention in CLAUDE.md.

AGENTS.md: dropped the bedrock/ entry, dropped the stale Ollama mention (that
path was removed in PR #13), and added vend/swe-router/.
@shekharprateek
shekharprateek merged commit 6a8180a into main Sep 15, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant