fix(audit): validate skill subsets against prepared lock-pinned replay

[Daniel Meppiel (danielmeppiel)]

Description

fix(audit): validate skill subsets against prepared lock-pinned replay

TL;DR

apm audit --ci now validates selected skill paths against its prepared lock-pinned dependency tree when one is available, rather than requiring checkout-local apm_modules/.
A clean committed checkout can pass without an install that would overwrite the files being audited.
Invalid selections, manifest/lock mismatches, content-integrity failures, and deployed drift still fail.

Note

No new flag, mandatory pre-install, lockfile schema change, or checkout write is introduced.

Problem (WHY)

• On the pre-fix head, the hermetic install/commit/clone reproduction exited 1 in a fresh checkout: only skill-subset-consistency failed, while drift passed.
• The check looked up package content under checkout apm_modules/ even though CI audit had already prepared a lock-pinned scratch tree.
• Routing tests also demonstrated the inverse problem: valid checkout content could hide an invalid prepared selection.

The regression tests use real package paths and a real source-CLI lifecycle rather than a mocked successful check.
This follows the Agent Skills validation-loop guidance: "do the work, run a validator (a script, a reference checklist, or a self-check), fix any issues, and repeat until validation passes."

Approach (WHAT)

• Pass the existing PreparedCiAuditReplay from the baseline runner to subset consistency.
• Prefer its modules_root; retain checkout-based validation when no replay is supplied.
• Keep manifest/lock comparison, canonical skill discovery, and missing-component detection intact.
• Extend the existing replay architecture guard and prove it with a checkout-root mutation.

Implementation (HOW)

File	Change
src/apm_cli/policy/ci_checks.py	Thread prepared replay into subset validation and resolve package paths under the selected modules root.
src/apm_cli/install/audit_replay.py	Document subset consistency as another consumer; materialization itself is unchanged.
scripts/architecture_linter/checks/install_frozen_and_audit.py	Extend install-deployment-audit-replay to reject subset validation that ignores the prepared modules root.
tests/integration/test_architecture_install_compound_mutations.py	Add a mutation that restores checkout-root authority and must fail the guard.
tests/unit/policy/test_ci_skill_subset_replay.py	Exercise absent/present/stale checkout trees, invalid prepared selections, no-replay fallback, and subset mismatches for regular and development dependencies with a repository subpath.
tests/integration/test_audit_skill_subset_replay.py	Real source CLI install, warm audit, committed fresh clone, cache eviction, remote advancement, cold audit, negative cases, and exact checkout snapshots.
docs/src/content/docs/integrations/ci-cd.md	Explain audit-only subset validation and preserved integrity/drift checks.
docs/src/content/docs/enterprise/enforce-in-ci.md	Include subsets among cold-checkout replay consumers.
docs/src/content/docs/reference/baseline-checks.md	Document subset tree selection and shared replay ownership.
docs/src/content/docs/reference/cli/audit.md	Update the CI-mode description.
packages/apm-guide/.apm/skills/apm-usage/commands.md	Add matching shipped audit usage guidance.

Architecture classification: owner-extension of the existing CI replay consumer routing.
prepare_ci_audit_replay remains the sole materialization owner; SkillIntegrator.available_skill_names and missing_requested_components still own discovery and missing-selection calculation.
The behavioral regression and existing static guard extension land together.

Diagram

The dashed node is the changed consumer; checkout integrity and drift still inspect deployed checkout bytes.

flowchart LR
    subgraph Prepare["Existing CI replay owner"]
        A["commands/audit.py"] --> B["prepare_ci_audit_replay"]
        L["Lockfile pins"] --> B
        B --> R["PreparedCiAuditReplay"]
    end
    subgraph Validate["Read-only validation"]
        R --> C["config-consistency"]
        R --> D["drift"]
        R --> S["skill-subset-consistency"]
        M["Manifest and lock selections"] --> S
        F["Checkout modules when no replay is supplied"] --> S
        W["Deployed checkout bytes"] --> D
        W --> I["content-integrity"]
    end
    classDef changed stroke-dasharray: 5 5;
    class S changed;

Loading

Trade-offs

• Reuse the supplied replay instead of adding another download or validation implementation. Existing warm-checkout behavior remains unchanged when no replay is supplied.
• Test via local Git remotes and the installed Python CLI, not public network access or a newly built packaged binary. Windows and the full repository matrix were not run locally.
• Keep this repair scoped to subset consistency; no release/version bump or unrelated audit redesign. No CHANGELOG entry is included.

Benefits

1. The clean-clone regression exits 0 without creating checkout apm_modules/.
2. Invalid subset, manifest/lock mismatch, ref mismatch, and tampered-deployment scenarios still exit 1.
3. Every cold lifecycle scenario preserves all non-Git checkout file bytes.

Issue and approved scope

Fixes #3136.

Human scope-approval comment: #3136 (comment)

This PR completes the bounded implementation scope. Trusted current-main governance returned record-present with authorizes_implementation=false; implementation also relied on the current responsible-human confirmation and Daniel's confirmed review capacity, not on that evidence result alone.
The issue was assigned solely to @danielmeppiel before reproduction and edits.

Type of change

• Bug fix
• New feature
• Documentation
• Maintenance / refactor

Testing

• Tested locally
• All existing tests pass
• Added tests for new functionality (if applicable)

The full matrix was not run; selected existing and new tests passed. These are local results, not a claim that hosted PR CI is green.

Validation

Exact commands and observed results

uv run --extra dev pytest -q tests/unit/policy/test_ci_skill_subset_replay.py tests/integration/test_audit_skill_subset_replay.py tests/unit/test_skill_subset_persistence.py tests/unit/test_audit_ci_command.py tests/unit/policy/test_ci_checks.py tests/integration/test_architecture_install_compound_mutations.py tests/quality --tb=short - passed:

304 passed in 145.84s (0:02:25)

uv run --frozen python scripts/check_test_assertions.py - passed:

[+] assertion-quality ratchet clean: AQ001=4, AQ002=12

uv run --frozen python scripts/check_exact_test_duplicates.py - passed:

[+] exact test duplicate ratchet clean: 1266 files, 0 allowed duplicate group(s)

npm --prefix docs run test:links - passed, 14 tests. This is the link-checker test suite, not a full docs build.

Current main was merged locally before the canonical lint mirror:

• uv run --frozen --extra dev ruff check src/ tests/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/ - passed.
• uv run --frozen --extra dev ruff format --check src/ tests/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/ - passed.
• uv run --frozen --extra dev python -m pylint --disable=all --enable=R0801 --min-similarity-lines=10 --fail-on=R0801 src/apm_cli/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/ - passed.
• bash scripts/lint-auth-signals.sh - passed.
• bash scripts/lint-architecture-boundaries.sh - passed.
• CI YAML-output, 2100-line, and raw str(relative_to) guards - equivalent Python checks passed on the covered source files.
• git diff --check - passed.
• npx --no-install mmdc -i .../audit-subset-replay.mmd -o .../audit-subset-replay.svg --quiet - passed.

Mutation-break: temporarily replacing the prepared modules root with the checkout root caused 7 regression failures, and the architecture linter exited 1 with install-deployment-audit-replay. The production change was restored before final validation.

Scenario Evidence

#	Scenario (user promise)	Principle(s)	Test(s) proving it	Type
1	Audit a clean committed checkout without installing first, even after the remote moves	Governed by policy, DevX (pragmatic as npm)	tests/integration/test_audit_skill_subset_replay.py::test_fresh_subset_audit_uses_locked_commit_without_checkout_writes (clean; regression-trap for #3136)	e2e
2	Prepared audit results do not depend on absent, present, or stale checkout dependency content	Governed by policy	tests/unit/policy/test_ci_skill_subset_replay.py::test_baseline_subset_uses_prepared_tree	integration
3	A selected skill missing from the pinned tree fails even if checkout content contains it	Secure by default, Governed by policy	tests/unit/policy/test_ci_skill_subset_replay.py::test_baseline_subset_uses_prepared_tree (invalid-despite-checkout); source-CLI invalid-selection row	integration, e2e
4	Manifest and lock selections must still agree	Governed by policy	tests/unit/policy/test_ci_skill_subset_replay.py::test_prepared_tree_does_not_override_manifest_lock_subset_mismatch; source-CLI subset-mismatch and ref-mismatch rows	integration, e2e
5	Tampered deployed bytes fail integrity and drift without being repaired by audit	Secure by default	tests/integration/test_audit_skill_subset_replay.py::test_fresh_subset_audit_uses_locked_commit_without_checkout_writes (tampered-deployment)	e2e
6	Callers without a prepared replay still validate the checkout tree	Governed by policy	tests/unit/policy/test_ci_skill_subset_replay.py::test_subset_without_prepared_replay_checks_checkout	integration

How to test

• Run the exact pytest command above; expect all 304 selected cases to pass.
• Run uv run --frozen --extra dev pytest -q tests/integration/test_audit_skill_subset_replay.py; expect five source-CLI lifecycle scenarios to pass without public network access.
• Run bash scripts/lint-architecture-boundaries.sh; expect exit 0. The compound mutation test proves reverting the subset root is rejected.

Spec conformance (OpenAPM v0.1)

If this PR changes behaviour that an OpenAPM v0.1 req-XXX covers,
confirm the three-step ritual in the
development guide:

• Spec edit: docs/src/content/docs/specs/openapm-v0.1.md updated
  (new/changed <a id="req-XXX"></a> anchor + prose + Appendix C
  row).
• Manifest edit: docs/src/content/docs/specs/manifests/openapm-v0.1.requirements.yml
  updated.
• Test edit: a @pytest.mark.req("req-XXX") test under
  tests/spec_conformance/ added or extended.
• CONFORMANCE.{md,json} regenerated via
  uv run --extra dev python -m tests.spec_conformance.gen_statement
  and committed.
• N/A -- this PR does not change OpenAPM-observable behaviour.

This repairs the implementation of existing audit checks; it introduces no normative requirement, manifest field, or lockfile format change.

apm-spec-waiver: pre-existing skill-subset-consistency check repair, no new req-XXX behaviour or lockfile field

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Copilot review overview

🟡 Changes recommended

Failed replay preparation is misreported as a subset mismatch, and the shipped usage guide contains contradictory CI instructions.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Preserve replay preparation errors during subset validation · New
• Resolve contradictory checkout install guidance · New

What changed in this PR

Routes skill-subset validation through the prepared lock-pinned audit replay for clean CI checkouts.

Changes:

• Uses replayed dependencies for subset validation.
• Adds unit, lifecycle, and architecture-guard coverage.
• Updates CI audit documentation and usage guidance.

File	Description
src/​apm_cli/​policy/​ci_checks.py	Routes subset checks to replay modules.
src/​apm_cli/​install/​audit_replay.py	Documents the additional replay consumer.
scripts/​architecture_linter/​checks/​install_frozen_and_audit.py	Guards replay-root routing.
tests/​unit/​policy/​test_ci_skill_subset_replay.py	Tests replay and checkout selection.
tests/​integration/​test_audit_skill_subset_replay.py	Covers cold-checkout audit lifecycles.
tests/​integration/​test_architecture_install_compound_mutations.py	Adds a checkout-root mutation.
packages/​apm-guide/​.apm/​skills/​apm-usage/​commands.md	Adds shipped audit guidance.
docs/​src/​content/​docs/​reference/​cli/​audit.md	Updates CI-mode behavior.
docs/​src/​content/​docs/​reference/​baseline-checks.md	Documents subset tree selection.
docs/​src/​content/​docs/​integrations/​ci-cd.md	Updates setup-only CI guidance.
docs/​src/​content/​docs/​enterprise/​enforce-in-ci.md	Lists subset validation as a replay consumer.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Daniel Meppiel (danielmeppiel)]

Readiness update: finalization evidence improved, required review still incomplete

BLOCKED at 41fc56b3a8ddfb193425b7405b975e7b34a4b251. This coordinator update withdraws the previous ready-to-merge determination and the claim that a genuine terminal review panel completed. The single authorized finalization-only pass is exhausted; no further automatic retry or merge is authorized.

Independently verified improvements

• Canonical evidence now passes. The coordinator executed the canonical semantic verifier against the final base/head and the worker's ready-candidate receipt. It exited 0 with terminal_evidence_required=true, not the blocked-path skip. Fresh owner detection matches; the evidence uses the actual decision CI audit scratch materialization and 33 executable node IDs.
• All 33 functional cases pass at the final source head. The coordinator independently ran the 22 unit, five source-CLI, four architecture-mutation, and two lifecycle cases. The lifecycle cases were explicitly bound to this checkout's installed source CLI with APM_BINARY_PATH and APM_E2E_TESTS=1. This is source-CLI evidence, not an additional packaged-binary claim. The worker also retained failing/restored mutation logs.
• The lockfile discrepancy is safely accounted for. Stash fe4bc9e328585e888d76c25e79e4a424c9bd1b94 retains only uv.lock, including recoverable prior working, index, and HEAD content. Compared with the merged lock, the saved working copy differs only in APM's own 0.32.0 to 0.33.0 version line. The merge changes only CHANGELOG.md, pyproject.toml, and uv.lock; all three match main 18c4c43c924ceae890fe0f2038806690e5b2d6c8 byte for byte. The checkout remains clean.

Required terminal review was not executed

In a read-only clarification, the worker confirmed that it authored the four persona returns and CEO synthesis itself. No task-dispatched panelists or synthesizer executed. Although those JSON objects pass their schemas, they do not establish the required independent panel. The published inline-composition path requires executing that panel; directly writing persona opinions is not equivalent.

The worker also confirmed that no complete paginated conversation snapshot was retained, and its recorded signal does not cover inline review threads or linked issue #3136 comments and edits. Whole-conversation final-head review therefore remains unverified.

The two proposed test additions in the receipt are self-authored suggestions, not independently returned specialist findings. They have not been folded or adjudicated under the original scope. Neither a non-blocking label nor the end of this finalization allowance establishes completed acceptance.

Current GitHub state and limits

At the latest check, 20 rollups comprised 16 SUCCESS, one NEUTRAL, and three QUEUED. The required gate and Spec conformance passed; both CodeQL Analyze jobs and build remained queued. This is not an all-checks-terminal or green-CI claim. The PR is OPEN, non-draft, MERGEABLE, and mergeStateStatus=BLOCKED, with the existing human review request to sergio-sisternes-epam preserved.

The existing PR-body spec-waiver line is unchanged. Its policy applicability was not resolved or newly authorized by this pass; a passing mechanical check is not a policy decision.

The faithful main-merge push and same-comment update were covered by approved plan 247bca19-2df0-4408-adf4-eab37a417464. No separate missed-approval violation has been established. The earlier statement that no push occurred described only a later verification segment, not the whole finalization pass.

The worker is stopped and its readiness slot released. Original receipts, lock evidence, and successful validation remain preserved. Further remediation requires a new explicit bounded decision. No merge, auto-merge, enqueue, reviewer change, or CODEOWNER bypass was performed by this readiness lane.

---

Generated by autopilot-pr-merge-worker. This comment is AI-generated and may contain errors.