Add an aggregate CI check and run every PR job from test.yml

[zeevmoney]

Linear issues

• PER-16779: one aggregate required CI check for permit-python, following the Terraform provider's pattern.
• PER-16336: the SDK upgrade guide; this PR applies its section 9, "One aggregate required check".

Stacked on #144, the top of the 3.x stack.

Why

The ruleset on main requires six job names spread across three workflows (test.yml, security.yml, pre-commit.yml). A renamed job silently stops being required, a new job is not required until someone adds it to the ruleset, and GitHub counts a job skipped by an if: as success for a required check. needs works only inside one workflow, so the jobs a pull request must pass move into test.yml, where one CI job checks all of them.

What changed

• One workflow for PR checks. test.yml now holds pre-commit (moved from pre-commit.yml, which is deleted) and Dependency Audit, Post Audit Comment, Dependency Review, Audit Script Tests and Workflow Hardening (moved from security.yml), next to its existing jobs. Job ids and names are unchanged, so the six check names the ruleset requires today still report.
• One copy of the audit. The audit's steps live in a local composite action, .github/actions/dependency-audit. The audit jobs in test.yml and security.yml both check out the repository and call it. The jobs keep contents: read, and the artifact is still named dependency-audit. The gate_failed job output, which nothing read, is removed.
• security.yml runs only on the Monday 09:00 UTC schedule and on manual dispatch, with audit and notify. The Slack message is unchanged.
• CI job, copied from the Terraform provider. It has if: always() and permissions: {}, and needs 10 jobs: api-coverage, audit, audit-scripts-test, comment, compatibility, dependency-review, migration-skill, pre-commit, pytest and workflow-hardening (EXPECTED_JOBS: 10). It exits 2 when the job results cannot be read or their count is not EXPECTED_JOBS, and 1 when any result is not success. The one exception: dependency-review skipped passes on push.
• e2e-unpinned-pdp (the latest-PDP-image and cloud-PDP legs) stays in test.yml after pytest and does not block a PR: CI does not need it.
• Post Audit Comment runs on every event, still after a failed audit. Its steps run only on a PR from a branch of this repository, so on pushes and fork PRs the job succeeds with its steps skipped. The comment step also skips posting when the PR head has moved past the commit the run audited.
• Workflow Hardening gains two steps:
  • "Check that CI needs every job" fails in four cases: test.yml's jobs, minus ci and ADVISORY_JOBS, differ from ci.needs; an ADVISORY_JOBS entry is not a job, is listed twice or is also in needs; or EXPECTED_JOBS is not the number of jobs in needs. It exits 2 when it reads no jobs. ADVISORY_JOBS holds e2e-unpinned-pdp, with a comment giving the reason.
  • "Shellcheck the local actions" shellchecks each shell: bash step under .github/actions with actionlint's options, because actionlint does not lint composite actions. It exits 1 on a finding and 2 when it reads no bash step.
• Tests. .github/scripts/test_ci_checks.py (50 tests, run by Audit Script Tests) reads the bash and env: of those three steps from test.yml with yq. It runs them the way a shell: bash step runs, against planted job results, workflows and actions.
• Dependabot. The github-actions entry also scans /.github/actions/*.
• Publish workflow. One comment in python-sdk-publish.yml pointed at the Trivy step in security.yml; it now points at .github/actions/dependency-audit. Nothing else in that workflow changes.
• Docs. CONTRIBUTING.md has a new CI section. It covers CI as the one check to require, how to add a job, what the job-list check enforces, the six check names not to rename until the ruleset switch, and security.yml as the home of the weekly audit. The headers of .github/scripts/pytest.ini and skills/tests/README.md, and the Dependabot comments, no longer name the old workflows or describe only some of the tests. The repository has no CLAUDE.md or AGENTS.md.

Behaviour changes

• The PR dependency audit now runs on every push to main/master, because test.yml has no path filter. Before, a push ran it only when pyproject.toml, security.yml, audit-deps.sh or format_audit.py changed.
• New pushes to a PR no longer cancel the older runs of the audit, comment, dependency review, audit script tests and workflow hardening, because test.yml has no concurrency group. The audit comment is not posted from a run whose commit is no longer the PR head. Dependency Review's own on-failure summary comment can still come from a superseded run.
• security.yml no longer runs on pull_request or push. Its concurrency group stays; the cancel-on-PR setting is dropped.
• Post Audit Comment runs on every event and succeeds with its steps skipped where it cannot post. Same-repo PRs, Dependabot's included, get the comment as before.
• Once the ruleset requires CI, more jobs block a merge: API Coverage, the compatibility legs, the Migration Skill Tests legs, Dependency Review on PRs and Post Audit Comment. e2e-unpinned-pdp still does not block.
• Workflow Hardening now fails in three more cases: a test.yml job is in neither ci.needs nor ADVISORY_JOBS, EXPECTED_JOBS is wrong, or a local action's bash has a shellcheck finding.
• Audit Script Tests also runs test_ci_checks.py, which uses bash, jq, yq and shellcheck from the runner image.
• The audit's run steps now run as composite steps under shell: bash (bash --noprofile --norc -eo pipefail) instead of the default bash -e. Every script already sets pipefail.
• Dependabot also opens updates for the composite action's pins. Major bumps are not grouped, so a major bump to an action pinned in both the workflows and the composite action arrives as one PR per directory, and the two pins can differ until both merge.
• The Notify Slack check run (skipped) no longer appears on PRs.

How it was tested

• .github/scripts suite, run as Audit Script Tests runs it: 273 passed, 50 of them in test_ci_checks.py. Local tools: bash 3.2.57, yq v4.53.3, jq, shellcheck 0.11.0.

• Planted inputs, run locally against the step bash extracted from test.yml. Exit codes:

  • CI: all success on pull_request 0 and on push 0; pytest skipped 1; audit cancelled 1; compatibility failed 1; 9 results 2; 11 results 2; unreadable JSON 2; dependency-review skipped on push 0 and on pull_request 1; comment skipped on push 1.
  • Job-list check: committed test.yml 0; a job missing from needs 1; a needs entry that is not a job 1; a stale advisory entry 1; a duplicated advisory entry 1; an advisory job also in needs 1; EXPECTED_JOBS 9 gives 1 and 11 gives 1; no jobs read 2.
  • Local actions' shellcheck: the committed action (5 bash steps) 0; a planted echo $x 1; a missing directory 2.

• Mutation check: I made 23 mutations. 17 were in the CI and job-list bash:

  • skipped or cancelled counted as success
  • the push exception applied on every event, or to any dependency-review result
  • no count check
  • unreadable JSON exiting 1
  • each job-list check removed in turn
  • ci not dropped from the job list
  • an empty ADVISORY_JOBS
  • EXPECTED_JOBS off by one
  • a job dropped from needs

  The other 6 were in the shellcheck step: no placeholder, no exclusions, a finding not failing, bash steps skipped, no exit 2 when nothing is read, and an unreadable action exiting 0. Each mutation failed at least one test.

• The comment script was run under Node 24 with a mocked GitHub client:

  • head unchanged: it updates the existing comment or creates one
  • head moved: it only logs a notice
  • no report: it warns
  • report over the length limit: it posts the link body

  The script before this change updated the comment on a moved head.

• actionlint 1.7.12: no findings. zizmor 1.30.1, offline and online: no findings (2 ignored, 17 suppressed). uv run pre-commit run --all-files passes.

• Job ids and names, compared with the base using yq, are unchanged. The bodies of pre-commit, dependency-review, notify, pytest, e2e-unpinned-pdp, api-coverage, compatibility and migration-skill are identical. The composite action's steps match the old audit steps except for the python-version input, shell: bash, one comment and the removed gate_failed lines.

• On GitHub:

  • The first run on this PR (bdb639c) passed, CI included, with 29 checks green. The composite action ran inside Dependency Audit, and Post Audit Comment posted the audit comment. CI didn't wait for the non-blocking e2e (latest PDP image) leg.
  • A temporary commit (6ca2789) forced Migration Skill Tests to skip. CI failed with "Jobs that did not succeed: migration-skill skipped" and exit code 1. The next commit (559241a) reverts it, so the tree is the same as at bdb639c.
  • Not yet seen for real: the audit comment's moved-head check, which first runs when a push lands while an older run is still going.

Owner actions before merge

1. After this merges and main produces the CI check, replace the six per-job required checks in the main ruleset with CI alone. Do it only then: requiring CI earlier leaves every PR waiting. The six checks are pytest (Pydantic pydantic<2.0.0), pytest (Pydantic pydantic>=2.0.0), pre-commit, Dependency Audit, Audit Script Tests and Workflow Hardening. Then, in a follow-up, remove the note that lists those six names from CI's leading comment in test.yml and from CONTRIBUTING.md's CI section.
2. Replace pre-commit with prek #149 (prek) edits the hooks job this PR moves. Whichever of the two merges second ports Replace pre-commit with prek #149's hooks-job changes (the prek run command and the PREK_HOME cache) into the pre-commit job in test.yml, and keeps pre-commit.yml deleted.

🤖 Generated with Claude Code

[linear-code]

PER-16779

[github-actions]

Dependency Security Audit

Scanned: pyproject.toml dependencies + dev group, resolved at Python 3.10 (the current resolution, and the lowest versions the published specs permit under each pydantic major)

✅ No known vulnerabilities found.

Both the resolved dependency set and the lowest versions the published specs permit are clean at HIGH and CRITICAL.