Establish an evidence-driven Workshop conformance corpus and seasonal feature census

[Teakowa]

Goal

Establish a durable Workshop conformance corpus that turns real projects, minimized regressions, a canonical full-feature Workshop census, and periodic live-client observations into executable evidence for workshop-rs and downstream language implementations.

Context

workshop-rs owns the canonical Workshop capability identities and Workshop-observable expectations used by this corpus. Source-language repositories remain responsible for their own syntax, semantics, and source-level fixtures.

The corpus must make correctness expectations independently attributable and must distinguish implementation regression evidence from externally grounded or preserved behavioral evidence.

Scope

• Define a provenance-aware fixture model that distinguishes at least:
  • small synthetic/unit or property cases;
  • minimized regressions extracted from real projects;
  • complete real-world project/corpus cases;
  • live-client/seasonal observations.
• Require correctness expectations to identify an independent evidence source such as a Workshop client observation, pinned external oracle, documented semantic contract, or preserved real-project regression. Do not treat implementation output changed in the same patch as independent correctness evidence.
• Preserve minimized reproductions for real-project semantic/compatibility defects where practical while retaining repository/revision/path provenance and the original full-project evidence.
• Define stable Workshop feature identities for conformance results across the declared semantic/catalog surface.
• Maintain a canonical Workshop feature-census fixture/mode that exercises the declared surface and reports failures by feature identity rather than only as a monolithic script diff.
• Use semantic/normalized equivalence as the correctness target where text identity is not part of the contract.
• Provide machine-readable conformance output that distinguishes matched behavior, unsupported surface, known gaps, unexpected regression/divergence, and inconclusive evidence without counting known failures as passing compatibility.
• Define a repeatable seasonal validation workflow in which census artifacts can be round-tripped through the live Overwatch client with game/season/client-locale/catalog provenance.
• Use English client output as the primary semantic baseline where appropriate, with explicit locale sweeps for localization/catalog drift as separately recorded evidence.
• Keep corpus/result contracts reusable by opy-rs and deltin-rs without transferring canonical Workshop ownership out of this repository.

Non-goals

• Claiming exhaustive live-runtime/gameplay correctness solely from import/export evidence.
• Requiring byte-identical or formatting-identical output.
• Building a network service or automatically downloading unreviewed catalog data at runtime.
• Moving OPY or DEL/OSTW source-language fixtures or semantics into workshop-rs.
• Treating one giant census script as a replacement for focused unit/property tests or real-world projects.
• Freezing a separate cross-repository test framework before concrete corpus integration requires one.
• Maintaining child-issue completion state or execution sequencing in this parent issue body.

Acceptance criteria

• The repository documents and enforces an evidence model in which correctness expectations have traceable provenance independent from the implementation being changed.
• Real-project failures can retain project-level provenance and, where valuable, minimized provenance-linked regression fixtures.
• A canonical feature-census fixture covers the declared Workshop semantic/catalog surface with stable machine-readable feature identities.
• Census failures identify affected features rather than reducing the result to a single whole-file pass/fail.
• parse -> canonical semantics -> emit and supported locale conversion are gated against the corpus using semantic/normalized contracts.
• Known gaps are represented explicitly and are not reported as successful parity/conformance.
• A documented seasonal client-validation procedure records client/game/catalog/locale provenance and supports structured comparison with prior captures.
• The resulting corpus/result contracts can be consumed by opy-rs and deltin-rs without transferring canonical Workshop ownership out of this repository.

Relationships

• Extends Establish the canonical multi-locale Workshop core #2 rather than replacing its canonical multi-locale Workshop core scope.
• Builds on Define Workshop catalog, locale, provenance, and version boundaries #1 catalog/locale/provenance/version boundaries.
• Executable work is decomposed in related issues such as Define the Workshop conformance result contract and stable feature identities #18-Establish the seasonal live-client Workshop validation and structured-diff workflow #21; their current state and ordering should be derived from GitHub rather than mirrored here.
• Downstream source-language adoption remains tracked in the owning opy-rs and deltin-rs repositories.

[Teakowa]

Execution update for #18–#21:

• Define the Workshop conformance result contract and stable feature identities #18: Draft PR feat(conformance): define Workshop evidence contract #26 pushed, Architect contract review completed, CI green.
• Build the canonical sharded Workshop feature census #19: Draft PR feat(census): add canonical sharded Workshop feature census #27 pushed on Define the Workshop conformance result contract and stable feature identities #18; initial independent QA found an oracle-boundary defect, which was corrected. Fresh QA now PASS/VERIFIED on 6a23b3e; CI stable/Rust 1.85 green.
• Add provenance-linked real-project and minimized Workshop regression evidence #20: Draft PR feat: add provenance-linked real-project evidence #28 pushed in parallel on Define the Workshop conformance result contract and stable feature identities #18; independent QA VERIFIED on fd21367; full CI green. The complete project remains an explicit KnownGap for ambiguous bare Up, while the minimized regression is separately matched.
• Establish the seasonal live-client Workshop validation and structured-diff workflow #21: Draft PR feat: add seasonal Workshop client evidence workflow #30 pushed on Build the canonical sharded Workshop feature census #19; fresh QA PASS/VERIFIED on 72ff506; CI stable/Rust 1.85 green. Offline capture schema, structured diff, CLI and ADR are complete for repository-owned automation.

No PR has been merged and no release published. No real Overwatch client capture is claimed; #21 retains the manual en-US/zh-CN client-validation boundary for maintainer action.

[Teakowa]

Execution update after final provenance/evidence review:

• Define the Workshop conformance result contract and stable feature identities #18 / PR feat(conformance): define Workshop evidence contract #26: final head 58832ff; Architect review PASS and independent QA VERIFIED. ConformanceResult::validate_against now requires exact evidence.catalog == supplied Catalog::identity(), with a format-valid mismatched-catalog regression test. CI stable and Rust 1.85 green.
• Build the canonical sharded Workshop feature census #19 / PR feat(census): add canonical sharded Workshop feature census #27: final head f9a520b; independent QA PASS and CI green. Public CensusIdentity/SHA-256 report identity is authoritative; every Inconclusive probe executes offline parse/validate/emit/round-trip/locale gates first. Current census truthfully exposes 298 UnexpectedRegression, 18 Unsupported, and 169 Inconclusive cases; the two newly authored localization fixtures remain Inconclusive and are not independent evidence.
• Add provenance-linked real-project and minimized Workshop regression evidence #20 / PR feat: add provenance-linked real-project evidence #28: final head 86d8753; independent QA VERIFIED and CI green. Full and minimized fixtures have separate pinned workshop-rs source tuples; the expected artifact is a separately pinned Wright/OverPy oracle with a reviewed digest. The full project remains KnownGap for ambiguous bare Up; the minimized case remains Matched.
• Establish the seasonal live-client Workshop validation and structured-diff workflow #21 / PR feat: add seasonal Workshop client evidence workflow #30: final head a7215dc; independent QA VERIFIED and CI green. It consumes the canonical feat(census): add canonical sharded Workshop feature census #27 CensusIdentity, provides the offline schema/diff workflow, and contains no fabricated client capture. All commits and the PR body use Refs #21; Issue Establish the seasonal live-client Workshop validation and structured-diff workflow #21 remains OPEN for real en-US/zh-CN client validation and runtime acceptance.

All four PRs remain OPEN Draft PRs, stacked as #26 -> #27/#28 -> #30. No PR was merged, main was not pushed, and no release was published. The remaining evidence boundary is maintainer review/merge plus real-client validation for #21; #27 also records current regression/unsupported gaps for separate follow-up rather than hiding them.

[Teakowa]

Fresh independent QA update for the final heads:

• PR feat(conformance): define Workshop evidence contract #26 58832ff: VERIFIED/PASS; exact catalog evidence binding and negative regression confirmed.
• PR feat(census): add canonical sharded Workshop feature census #27 f9a520b: VERIFIED/PASS; canonical census identity, probes-before-Inconclusive execution, and explicit surfaced failures confirmed.
• PR feat: add provenance-linked real-project evidence #28 86d8753: BLOCKED/FAIL; provenance tuples and digests pass, but the corpus runner records the pinned oracle only as metadata and does not compare oracle content against observed canonical WIR, so a divergent implementation can still report Matched.
• PR feat: add seasonal Workshop client evidence workflow #30 a7215dc: VERIFIED/PASS; canonical CensusIdentity consumption, offline/no-fabrication boundary, and non-closing Refs #21 history confirmed. Issue Establish the seasonal live-client Workshop validation and structured-diff workflow #21 remains OPEN.

No implementation changes were made in this QA pass. Merge order remains #26 first, then #27/#28 only after #28 resolves its oracle-comparison blocker, then #30 after #27. #21 stays open for real en-US and zh-CN client capture review.