Test policy validation against historical pull requests #258

Parent Quest: #254

Story

As an effectionx maintainer, I want to replay policy validation against known historical defects and their fixes, so we know the complete XMD mechanism distinguishes real violations from compliant code.

Current gap

Individual rules and prompts can be tested with synthetic examples, but those examples do not show that policy discovery, change selection, validator invocation, and verdict reporting work together on changes that actually occurred in effectionx.

PRs #226 and #252 contain documented before-and-after cases for two existing policies. Their revisions are currently external history rather than stable verification inputs owned by the policy suite.

Contract

Create a repository-owned historical corpus that exercises policy validation through the same XMD entrypoints used by contributors.

The initial corpus pins these revision pairs:

Fixtures preserve only the repository inputs needed to reproduce each verdict and record their source revisions. Automated verification does not fetch mutable pull-request state or depend on GitHub availability.

Each historical scenario runs the applicable policy’s public XMD Validate entrypoint. It records the selected policy, validator, expected verdict, actual verdict, and evidence. The corpus also contains a clean negative control that would fail if a validator reported every inspected change.

Acceptance

• The violating revision from PR 🐛 Move async teardown from finally to ensure #226 fails the Async Teardown policy, and its fixed revision passes.
• The violating revision from PR 🐛 Make @effectionx/node once() scope-bound #252 fails the Scope-Bound Event Registration policy, and its fixed revision passes.
• Each result is produced through the policy’s XMD entrypoint rather than a private rule-test shortcut.
• A clean negative control passes and produces no policy violation.
• Swapping an expected failing fixture for its fixed counterpart causes the corresponding regression assertion to fail, demonstrating that the corpus tests the verdict rather than command completion alone.
• Historical verification runs without network access and produces the same selected validators, findings, and exit statuses for unchanged inputs.
• Fixture provenance identifies the exact pull request and revision without requiring readers to reconstruct the source manually.

Evidence

A single repository command runs the historical corpus and reports every scenario by policy and source revision. CI executes that command before the README policy workflow can be considered delivered.

Dependencies

• Depends on Make each policy independently executable through XMD #257 for public per-policy XMD entrypoints.
• Uses lint rules delivered by Publish Effection policy rules through @effectionx/lint #256.
• Blocks README and CI policy composition in Quest: Run deterministic policy validation from the effectionx README #254.

Out of scope

• Replaying every historical pull request
• Fetching GitHub data during tests
• Benchmarking AI models
• Deno lint compatibility
• Treating synthetic rule fixtures as substitutes for end-to-end historical cases