Skip to content

[PRVC-APPLY] Apply the Corpus to the live ProofRunner API #28

Description

@ilyar

Outcome

All eligible corpus variants must be executed through the real API, runner, and container sandbox. Their product-native status, verdict, and reason code must be compared with the oracle from actual execution. This issue remains incomplete until reproducible live-run evidence exists.

Scope

  • packages/corpus
  • apps/api
  • apps/runner

Acceptance

  • Candidate corpus and checksum baseline are generated and validated without claiming live execution.
  • Imported real.* candidates are rejected before fixture materialization when provenance/recipe is missing.
  • On a Docker/Podman-capable target, a real API + runner execution iterates every eligible corpus variant through /api/verify/api/runs/:id; no oracle-replay executor is allowed.
  • Every deterministic eligible execution is repeated three times, with all repetitions retained and included in stability/certification counts.
  • Core and sandbox/resource cases produce the exact product-native terminal status, verdict, and reason-code contract from actual execution.
  • The black-box flow retrieves each /api/receipts/:id, validates response schemas and the receipt signature, compares the oracle and metamorphic relations, and verifies container/workspace cleanup.
  • At least one provenance-pinned real bug pair produces FAIL for buggy and PASS for fixed.
  • A retained report from that exact run receives independent review before certification counts become positive.
  • manifests/certification-report.json advances from IMPORTED only after the preceding evidence exists.
  • Manifest checksums and the full product suite pass on the evidence-producing commit.

Evidence

  • Corrective commit 2082987b55777813ca7f1ef4abbc69000c461350 on PR #36 removes the mocked corpus-application suite and its generated run-report.json instead of presenting simulated output as live evidence.
  • Current committed baseline: 56 cases / 65 variants, certification IMPORTED, passed_cases: 0.
  • Local verification of the corrective commit: corpus 34/34 tests; runner 46 passed with 5 Docker-gated skipped; API 33/33; manifest checksum validation passed.
  • Controller tracking: proof-runner-agent PR #34.

Blockers

  • The current host has Docker CLI but no reachable daemon. A Docker/Podman-capable worker is required.
  • Imported real.* candidates need provenance-pinned source and a reproducible materialization recipe.

Source task: tasks/prvc-apply.md in ever-guild/proof-runner-agent.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions