Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/BUILD_LOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -416,3 +416,14 @@ Evidence: `docs/v0.3/ADR-010-DUAL-SOURCE-OPERATOR-SOURCE.md` and synchronized PR
- Verified the real frozen 18-packet index through CLI init/status/next, then passed `npm run check`, 114/114 runnable extension tests, production build, and VSIX packaging; two Docker-only tests remained skipped in the local Windows run.

This removes clerical friction from the R7 human gate without replacing expert judgment or exposing automatic outcomes. Full R7 remains pending until two experienced TypeScript raters and the blinded adjudicator return real bundles.

## 2026-08-01 — Standalone R7 blind kit

- Added a deterministic offline-kit builder that copies the frozen packet bytes unchanged and prebundles the rating CLI into one Node.js file.
- Added an allowlisted `kit.json` manifest binding every file, byte size, SHA-256, packet-index hash, and the full `kitSha256`; timestamps and repository paths are excluded.
- Added standalone `verify-kit`, strict missing/extra/symlink/special-file rejection, and a protocol allowlist that still rejects `outcome.json` after a malicious manifest rehash.
- Added reproducibility, CLI-tamper, manifest-rewrite, no-overwrite, and no-checkout/no-npm execution tests.
- Built the real 18-packet kit twice on Windows with identical hash `a209f22938a3c4744a1decd80ffb14bd0681b0a111aa50fdd535f88443fbf643`, then ran its bundled verifier and `init/status` outside the repository. This is local build evidence; the hash becomes a study artifact only when the owner freezes and communicates a chosen kit out of band.
- Passed `npm run check`, 119/119 runnable extension tests, production build, and VSIX packaging; two Docker-only tests remained skipped in the local Windows run.

The kit changes no frozen packet byte, question, rating field, compiler result, or threshold. It reduces setup and unblinding risk but cannot act as a human rater.
3 changes: 2 additions & 1 deletion docs/PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ Branch `codex/shadow-cockpit-rnd` resets the product R&D thesis around **Dual-Co
- The preregistered R7 collector froze 30 eligible patches from six repositories after evaluating 457 bounded eligibility records. Manifest `a4ef6cbfa48c66cb9d384bcc2834ecbfae8ff08810abfd1863b395b8aa47d149` contains 12 development and 18 held-out patches; `docs/v0.3/results/R7_CORPUS_COLLECTION.md` reports repository and first-match exclusion counts. No compiler or human outcome influenced selection.
- The frozen R7 automatic audit passed its preregistered automatic threshold: 17/18 held-out identities compiled and 16/18 were valid end-to-end. The frozen blind expert packet set and deterministic rating join exist, but two independent ratings and adjudication remain pending. Full R7 has not passed; R5/R6 stay gated.
- The R7 blind-rating path now includes a local hash-verified workspace CLI. It verifies the exact packet directory, supports resumable human drafts, refuses incomplete export, and prepares a blinded adjudication workspace by copying only exact categorical consensus. It automates no causal rating and exposes no compiler outcome; the two real expert bundles and panel decision are still pending.
- A deterministic offline blind-kit builder now removes the need to share the PureFlow checkout. It copies the frozen packet bytes unchanged, prebundles the CLI, emits an allowlisted integrity manifest, and verifies the full extracted directory without npm or network access. The expected kit hash must be communicated separately; this packaging evidence does not complete either human rating.
- ADR-007 proposes an Executable Operator Model and shadow-control protocol. ADR-008 adds a bidirectional Intent Ledger for artifact accountability. ADR-009 adds Decision Futures and a Takeover Envelope so an on-time pre-reveal human commitment can determine a live integrated branch while agents retain implementation. Together they cover artifact accountability, demonstrated control, and real decision authority; none is implementation evidence.
- ADR-010 proposes Dual Source only as a falsifiable product metaphor: agents maintain the complete Software Source while a pure, sparse Operator Projection renders immutable human commitments and bounded control evidence. It owns no state or readiness authority and must beat the existing representation in a matched pilot before becoming core.
- `docs/v0.3/R8_COMBINED_PILOT_PROTOCOL.md` now defines the draft four-condition human study needed to test the combined architecture against ordinary autonomous use and a fixed active-review comparator. It has no participants or measured outcomes and cannot be frozen until full R7 and the R5/R5.1/R5.2/R6 runtime exist.
Expand Down Expand Up @@ -157,7 +158,7 @@ No external input blocks the repository-owned fixture R0–R4.5 mechanism in `do

## Next ordered actions

1. Use `extension/scripts/r7-rater.mjs` to obtain and adjudicate two independent blind expert ratings for packet index `54d78382b3ddbe15cba1f8153275e8149d32ddaa5192163f99ca5f43d903e8fe`.
1. Build the standalone artifact with `extension/scripts/r7-rater-kit.mjs`, communicate its printed kit hash out of band, then obtain and adjudicate two independent blind expert ratings for packet index `54d78382b3ddbe15cba1f8153275e8149d32ddaa5192163f99ca5f43d903e8fe`.
2. If full R7 passes, review ADR-007–010 and the R8 combined protocol together, then freeze the baseline ledger/model schemas and one offline Operator Source prototype.
3. Implement the local R5 evidence ledger and run Experiment 2.5 with identical evidence in both projections; make Operator Source the R6 default only if it improves causal localization per active minute without increasing false confidence.
4. Build the minimal cockpit with the surviving projection, then freeze and run the four-condition delayed-transfer pilot; run the longitudinal field pilot before making a sustained skill-retention claim.
Expand Down
1 change: 1 addition & 0 deletions docs/v0.3/AGENT_EXECUTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -403,6 +403,7 @@ If the fixture-only probe needs arbitrary participant code or model-generated te
- Freeze both the recovery compiler and semantic-probe compiler before running the held-out set and make no per-patch code changes.
- Use two independent expert raters, blind to compiler outcome, for causal relevance and expected judge result; adjudicate disagreements and report inter-rater agreement.
- Use the hash-verified local rater workspace from `R7_EXPERT_RATING.md` so packet integrity, exact coverage, resume, bundle export, identity separation, and consensus-only adjudication prefill are machine-checked while causal decisions remain human.
- Deliver ratings through the deterministic standalone blind kit, not a repository checkout. Freeze and communicate its `kitSha256` out of band; the kit may contain only byte-identical frozen packets, the prebundled local CLI, instructions, and its allowlisted integrity manifest.
- Compile both a recovery episode and an Explain-to-Break probe for each eligible patch. Dynamic probes implement the Phase-B `SandboxControlProbe` contract: frozen sanitized snapshot/tree, immutable command-registry hash, approved command or controller-generated deterministic oracle, read-only mounts, and selected sandbox. They may never pass model output directly into code, tests, commands, arguments, paths, mounts, or environment.
- Report valid-episode rate, valid-probe rate, capsule rejection/leakage results, false-pass rate, false-fail rate, and confidence intervals separately.
- Run the held-out audit on Windows as well as Linux, including paths with spaces, concurrent twins, cancellation, locked files, and cleanup.
Expand Down
45 changes: 35 additions & 10 deletions docs/v0.3/R7_EXPERT_RATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,20 +29,45 @@ Each rater returns one bundle containing exactly one record for every `packet-*.

Use the local workspace CLI so the engineer spends time on causal judgment rather than copying packet IDs or assembling JSON. The tool reads only the frozen packet directory. It verifies the index and every packet hash, refuses unexpected files or subdirectories, stores a resumable draft outside the packet directory, and will not export until all 18 packets have valid ratings. It has no command that accepts compiler results, audit summaries, commits, transcripts, or protected repairs.

From `extension/`:
### Build the standalone blind kit

The study owner must not give a rater a PureFlow repository checkout. Build a new offline kit containing only the byte-identical frozen packets, a prebundled CLI, a short instruction file, and `kit.json`:

```powershell
cd extension
$packetDir = "..\docs\v0.3\results\held-out-rater-packets"
$kitOutput = Join-Path $env:TEMP "pureflow-r7-blind-kit"
node scripts/r7-rater-kit.mjs $packetDir $kitOutput
```

The output directory must not already exist. The builder prints a deterministic `kitSha256` and packet-index hash. Give the extracted kit directory to each rater and communicate the expected `kitSha256` through a separate channel. Transport may use an archive, but the extracted directory—not the archive—is the verified artifact. Do not add cover notes, outcome files, repository metadata, or rating workspaces inside it.

Before rating, the expert runs from the extracted kit:

```powershell
node .\r7-rater.cjs verify-kit .
```

The printed kit hash must equal the out-of-band value. Verification rejects a changed CLI or packet, missing file, extra file, directory, symlink, and even a recomputed manifest that tries to allow a file outside the frozen kit allowlist. Node.js 22 or newer is required; `npm install`, network access, and a PureFlow checkout are not.

This packaging layer was added after the automatic run. It is allowed to copy and render the already frozen packet bytes and collect the already frozen rating schema only. It may not change packet order, content, questions, rating fields, eligibility, compiler outputs, or thresholds.

### Rate the packets

From the extracted kit:

```powershell
$packetDir = ".\packets"
$ratingRoot = Join-Path $env:TEMP "pureflow-r7-expert-a"
New-Item -ItemType Directory -Path $ratingRoot
$workspace = Join-Path $ratingRoot "workspace.json"
$bundle = Join-Path $ratingRoot "expert-a.json"

node scripts/r7-rater.mjs init $packetDir expert-a $workspace
node scripts/r7-rater.mjs next $packetDir $workspace
node scripts/r7-rater.mjs answer $packetDir $workspace <packet-id> yes pass fail none 4 "Reason grounded in the visible diff"
node scripts/r7-rater.mjs status $packetDir $workspace
node scripts/r7-rater.mjs export $packetDir $workspace $bundle
node .\r7-rater.cjs init $packetDir expert-a $workspace
node .\r7-rater.cjs next $packetDir $workspace
node .\r7-rater.cjs answer $packetDir $workspace <packet-id> yes pass fail none 4 "Reason grounded in the visible diff"
node .\r7-rater.cjs status $packetDir $workspace
node .\r7-rater.cjs export $packetDir $workspace $bundle
```

`next` prints the next unrated packet, visible diff, and frozen questions. `answer` may be repeated for a packet before export to correct a draft. Put neither the workspace nor the exported bundle inside `$packetDir`; the exact-content check deliberately rejects that. Give each rater a separate scratch directory and stable pseudonymous ID.
Expand All @@ -55,10 +80,10 @@ New-Item -ItemType Directory -Path $panelRoot
$panelWorkspace = Join-Path $panelRoot "workspace.json"
$adjudication = Join-Path $panelRoot "adjudication.json"

node scripts/r7-rater.mjs init-adjudication $packetDir <rater-a.json> <rater-b.json> panel-chair $panelWorkspace
node scripts/r7-rater.mjs next $packetDir $panelWorkspace
node scripts/r7-rater.mjs answer $packetDir $panelWorkspace <packet-id> yes pass fail none 4 "Panel reason grounded in the visible diff"
node scripts/r7-rater.mjs export $packetDir $panelWorkspace $adjudication
node .\r7-rater.cjs init-adjudication $packetDir <rater-a.json> <rater-b.json> panel-chair $panelWorkspace
node .\r7-rater.cjs next $packetDir $panelWorkspace
node .\r7-rater.cjs answer $packetDir $panelWorkspace <packet-id> yes pass fail none 4 "Panel reason grounded in the visible diff"
node .\r7-rater.cjs export $packetDir $panelWorkspace $adjudication
```

Resolve all pending disagreements without exposing compiler outcomes. The exported adjudication is a full 18-record third bundle with a distinct panel ID. Then run:
Expand Down
1 change: 1 addition & 0 deletions extension/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,7 @@
"r7:audit": "node scripts/r7-audit.mjs",
"r7:corpus": "node scripts/r7-corpus.mjs",
"r7:rate": "node scripts/r7-rater.mjs",
"r7:kit": "node scripts/r7-rater-kit.mjs",
"test": "npm run fixture:runtime && vitest run",
"package": "npm run build && vsce package --no-dependencies"
},
Expand Down
37 changes: 37 additions & 0 deletions extension/scripts/r7-rater-kit.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { pathToFileURL } from "node:url";
import { build } from "esbuild";

const root = await mkdtemp(join(tmpdir(), "pureflow-r7-kit-cli-"));
const rater = join(root, "r7-rater.cjs");
const builder = join(root, "builder.cjs");

try {
await Promise.all([
build({
entryPoints: ["src/rating/workspace-cli.ts"],
bundle: true,
platform: "node",
format: "cjs",
target: "node22",
legalComments: "none",
outfile: rater,
}),
build({
entryPoints: ["src/rating/kit-build-cli.ts"],
bundle: true,
platform: "node",
format: "cjs",
target: "node22",
legalComments: "none",
outfile: builder,
}),
]);
process.argv.push(rater);
const loaded = await import(pathToFileURL(builder).href);
await (loaded.done ?? loaded.default?.done);
} finally {
await rm(root, { recursive: true, force: true });
}
20 changes: 20 additions & 0 deletions extension/src/rating/kit-build-cli.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
import { readFile } from "node:fs/promises";
import { buildBlindKit } from "./kit";

export const done = main();

async function main(): Promise<void> {
try {
const [packetDir, outputDir, cliBundlePath] = process.argv.slice(2);
if (!packetDir || !outputDir || !cliBundlePath) {
process.stderr.write("Usage: node scripts/r7-rater-kit.mjs <packet-dir> <new-output-dir>\n");
process.exitCode = 2;
return;
}
const manifest = await buildBlindKit(packetDir, outputDir, await readFile(cliBundlePath));
process.stdout.write(`Built offline blind kit ${manifest.kitSha256} for packet index ${manifest.packetIndexSha256}.\n`);
} catch (error) {
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
process.exitCode = 1;
}
}
Loading
Loading