Add test, pre-commit, uv-lock-check, and dependency-update workflows - #15
Merged
Merged
Conversation
The 21 test files imported in PR #14 had no CI running them. test.yml closes that gap: a matrix over benchmarks/ and self-hosted/vllm/ running uv sync, ruff check, mypy, and unittest discover. Workflows added: - test.yml: the PR merge gate (ruff + mypy + unittest per uv project) - pre-commit.yml + .pre-commit-config.yaml: ruff lint/format and whitespace hooks - uv-lock-check.yml: fails a PR whose pyproject.toml changed without uv lock - dependency-update.yml: weekly uv lock refresh with a 7-day supply-chain quarantine Left main's lint.yml untouched (it carries the Bandit test-dir exclude and the per-job permissions blocks added in PR #14). Two fixes needed to make the suite green: - ruff: removed an unused glob import in extract-metrics.py and an extraneous f-prefix in replay-benchmark.py. Both are local-only scripts that predate the fork import, so they never went through its CI. - removed tests/test_build_vended_models.py. It asserts the committed vend/swe-router/models.json can be regenerated, which needs per-tier scores from benchmarks/swe-benchmark-data/ run-summary files -- the 1790-file raw dataset deliberately excluded from this repo. The generator script itself is kept, since a user benchmarking their own repo would run it. Verified locally: benchmarks 445 tests pass, self-hosted/vllm 45 tests pass, ruff and mypy clean on both, both uv.lock files in sync.
Follow-ups needed to get pre-commit green on this branch. bedrock/ deletion: this was approved for PR #13 but my commit landed after that PR was merged, so the directory survived. It is carried over from the original sample-claude-code-multi-model repo -- Amit's fork never had it -- and its LiteLLM proxy setup is superseded by benchmarks/config/litellm-mantle.yaml and benchmarks/scripts/bedrock-mantle-proxy.sh. Two of its files also failed ruff-format, so deleting it resolves those. ruff-format on extract-metrics.py and replay-benchmark.py: both are local-only scripts that predate the fork import and never went through its CI. Formatted with ruff 0.14.0 to match the version .pre-commit-config.yaml pins -- note that ruff 0.16+ also reformats Python inside markdown code fences, which would have rewritten the style examples in .claude/skills/*/SKILL.md and contradicted the one-parameter-per-line convention in CLAUDE.md. AGENTS.md: dropped the bedrock/ entry, dropped the stale Ollama mention (that path was removed in PR #13), and added vend/swe-router/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #14 imported 21 test files and nothing was running them. This closes that gap, and picks up two loose ends along the way.
test.ymlbenchmarks/andself-hosted/vllm/runninguv sync,ruff check,mypy,unittest discoverpre-commit.yml+.pre-commit-config.yamluv-lock-check.ymlpyproject.tomlchanged without a matchinguv lockdependency-update.ymluv lockrefresh, 7-day supply-chain quarantine on new releasesmain'slint.ymlis left untouched — it carries the Bandit test-dir exclude and per-jobpermissionsblocks added in PR #14.Also in this PR
bedrock/deleted. This was approved for PR #13, but my commit landed after that PR was merged so the directory survived. It came from the originalsample-claude-code-multi-modelrepo — Amit's fork never had it — and its LiteLLM proxy setup is superseded bybenchmarks/config/litellm-mantle.yamlandbenchmarks/scripts/bedrock-mantle-proxy.sh. Two of its files also failedruff-format, so deleting it resolves those. Verified no markdown links point into it.AGENTS.md— dropped thebedrock/entry, dropped the stale Ollama mention (removed in PR #13), addedvend/swe-router/.Fixes needed to get the suite green
Ruff lint (2 auto-fixes).
extract-metrics.pyhad an unusedglobimport — therglobcalls in it arePathmethods, not the module — andreplay-benchmark.pyhad anf"..."with no placeholders.Ruff format (2 files). Same two scripts. Formatted with ruff 0.14.0, the version
.pre-commit-config.yamlpins. Worth noting: ruff 0.16+ also reformats Python inside markdown code fences, which would have rewritten the style examples in.claude/skills/*/SKILL.mdand collapsed them to one line — directly contradicting the "one parameter per line" convention inCLAUDE.md. Pin-matching matters here.Removed
tests/test_build_vended_models.py. It asserts the committedvend/swe-router/models.jsoncan be regenerated from source, which needs per-tier scores out ofbenchmarks/swe-benchmark-data/run-summary files — the 1790-file raw dataset deliberately excluded from this repo. Importing justdocs/metrics/pareto-frontier-omp-swe3.jsonis not enough;score_by_complexitycomes back empty. The generator scriptbuild_vended_models.pyis kept, since someone benchmarking their own repo would run it.Verified
All 10 checks green, including both
Test:legs. The one failure I saw locally —test_judge_common.test_lowercase_preferred_on_collision— is macOS-only: it createsEXPERT_REVIEW.mdandexpert_review.mdin one directory, which a case-insensitive filesystem collapses into a single file. Onubuntu-latestboth exist and it passes, which CI confirms.