Skip to content

test(conformance): enforce authority-defect mutations and helper boundaries - #4

Closed
CompleteDotTech wants to merge 3 commits into
mainfrom
feat/conformance-evidence-38
Closed

test(conformance): enforce authority-defect mutations and helper boundaries#4
CompleteDotTech wants to merge 3 commits into
mainfrom
feat/conformance-evidence-38

Conversation

@CompleteDotTech

@CompleteDotTech CompleteDotTech commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Ready-for-review re-creation of draft PR #3 (same head 9e1162724a60): GitHub's REST PATCH {"draft":false} returns 200 but silently leaves the draft flag set (verified 3×). The working conversion path is the GraphQL mutation markPullRequestReadyForReview (GraphQL mutations work with this token; only queries fail) — confirmed afterwards on this PR (HTTP 200, isDraft:false). The draft was held through the complete gate before this PR was opened: local corepack pnpm@10.34.0 verify exit 0 and fork CI run 33304638130 concluded success on exactly this head.

Summary

Audit of OpenCoven#38 against upstream PR OpenCoven#74 found the aggregate/release-binding machinery fully covered upstream, while two harness-quality requirements of OpenCoven#38 were not yet enforced by tests anywhere in-repo:

  • "mutation-test representative authority defects, not only helper functions"
  • "negative-test every assertion helper" (the SDK-side validator helpers had only indirect coverage)

This PR lands exactly those two suites on top of main (PR OpenCoven#73 state), without duplicating any OpenCoven#74 content (it extends, and is compatible with, OpenCoven#74's lock/schema/release-binding work).

What changes

  • tests/conformance-authority-mutations.spec.ts — a 58-case representative authority-defect matrix (plus controls) proving the aggregator rejects a record whenever an authority misbehaves: wrong pairing secret accepted, replay accepted, pairing deny/expiry hidden, shared failure budget and rate limit dropped, double exchange, missing Content-Length: 0 control case, proxy-rejection envelope collapse, stale/replaced discovery, malformed/non-canonical cursors, revoked-credential reads, missing deadlines, native-keychain/trust-binding fail-open, Unix peer-identity / malicious COVEN_HOME / wrong-peer-UID / Windows constructed- and foreign-pipe defects, evidence-redaction assertions (prompts, message bodies, attachments, command output), authority-takeover downgrades (mode/discovery-version/mechanism), TTL drops, engine/registry/commit/tarball provenance drift, operator-state mutation, retained socket handles/private paths, non-loopback networks, workspace-link and source-checkout dependencies, reordered assertion lists — and, at the engine level, a coverage check blind to missing assertions, a summary that counts skips as passes, and a lying renderer are all caught structurally.
  • tests/conformance-helper-boundaries.spec.ts — direct negative tests for every assertion helper (expectTimestamp, expectCommit, expectSha256, expectTarballs, expectCoverage, expectNotCovered, expectIsolation, expectCrossAssertion, expectAssertionArray, expectString, expectBoolean, expectInteger, expectCaveRecord, expectExactObject, registry validation, duplicate-JSON-key rejection) plus an exhaustive redaction-scan matrix: all 19 forbidden field names (in normal and upper-case forms), all six secret patterns (bearer, header, private-key, JWT, ghp_/npm_ tokens, 43-char credential runs), all five private-path families (19 paths), non-JSON values, the 32-level depth bound, the 50,000-node bound, and the 16 KiB string bound. Also pins that sdk/chat scope IDs can never appear in notCovered ([SDK 0.1.0][P0] Prove the packed SDK with cross-repository real-authority conformance OpenCoven/sdk#38 AC3).
  • .github/workflows/ci.yml — one inert line (workflow_dispatch:) so the fork can run real CI on this branch; droppable in review.

OpenCoven#38 acceptance-criteria audit (this branch + upstream OpenCoven#74)

OpenCoven#38 criterion Status
One command produces a deterministic evidence record ✅ in-repo (conformance:aggregate; order-independence tested here and in OpenCoven#74)
Complete Cave+Coven journeys pass on every supported OS/arch ⛔ cross-repo — pending Chat schema-v2 records (below)
SDK and Chat absent from notCovered for read-only 0.1.0 ✅ in-repo — coverage must be all-true; notCovered allowlist excludes sdk/chat (now directly tested here)
Fixture/vendored/packed/consumer-lock digests match ✅ in-repo — cross-platform digest equality enforced and tested
Every required assertion ID appears exactly once ✅ in-repo — coverage + exact-order checks, missing/duplicate/unexpected/fail/skip all rejected (tested here)
Representative broken-authority mutations are caught landed by this PR (58-case matrix)
Harness cannot touch operator state ✅ in-repo — isolation proofs (before/after digests, owned roots) enforced and mutated-tested here
Retained evidence passes redaction scans ✅ in-repo — bounded scan; exhaustive pattern matrix landed by this PR
Release candidate cannot advance without a named passing aggregate ✅ via upstream OpenCoven#74 (conformanceEvidence.aggregateRecord + publish-mode gate); null on main until records exist

Outstanding cross-repo evidence (cannot be produced in this repo)

OpenCoven/chat (owner: Chat/BunsDev) owes schema-v2 platform records produced by Chat's native harness, one each for darwin-arm64, linux-x64, and win32-x64, embedding the unmodified Cave renderConformanceRecord output, against the committed registry at the candidate SDK commit, with matching Cave/Coven/SDK/Chat commits, release versions, tarball digests, isolation proofs, and redaction. Until all three exist, aggregateRecord stays null and publication stays blocked — by design. No evidence is fabricated here.

Observable behavior / compatibility

Test-only plus one inert CI line: no runtime, schema, CLI, or packaging behavior changes. conformance:aggregate, the evidence schema, and the release gate are untouched. Both new suites run inside the ordinary verify gate and CI; no new dependencies.

Validation

  • corepack pnpm@10.34.0 verify — exit 0 on the pushed head (typecheck, build, 1,366 tests / 60 files incl. 153 new, verify:contracts, verify:package, verify:release, coverage, stress, lint)
  • git diff --check — clean
  • New suites: 69 + 84 tests, all green

Release implications

None by itself; it hardens the evidence gate that OpenCoven#38/OpenCoven#41 depend on. A release candidate still cannot advance without the named passing aggregate — unchanged.

Assignee: @CompleteDotTech

Advances OpenCoven#38

Upstream PR (one click for a maintainer): https://github.com/OpenCoven/sdk/compare/main...CompleteDotTech:sdk:feat/conformance-evidence-38?expand=1

@CompleteDotTech

Copy link
Copy Markdown
Owner Author

Recreated upstream as OpenCoven#77: OpenCoven#77. Closing this duplicate fork PR; review continues upstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant