Repository navigation
Add a compile-and-run runtime scoreboard for the passing suite #93
Description
Part of #74 (Phase 1). The M7 gate has no measurement at all: no harness compiles and runs a corpus passing file. Execution evidence today is vertical tests over inline fixtures.
Do
Add a runtime scoreboard that, per passing file, compiles to a component and runs it under Wasmtime, asserting the observable result.
Design constraints, learned from the existing harnesses
crates/psrs-driver/src/tests/mod.rs:25run_with_wasmtimecompiles one module and runswasmtime run.run_with_wasmtime_args(:56) passes argv,run_with_wasmtime_stdin(:62) feeds stdin, andrun_wasmtime_with_dirs(:72) adds--dir host::guestpreopens. The corpus needs all three:get-arguments, stdin reads, andpreopens.- There is a duplicate private
run_program_with_wasmtimeatcrates/psrs-driver/src/tests/integration.rs:237that shadows the one inmod.rsand, unlike it, has noPSRS_REQUIRE_WASMTIMEhandling. Consolidate; a shadowed helper that silently skips is exactly the failure mode issue Add a required Wasmtime execution test gate that cannot silently skip #9 was filed to prevent. - The harness must be
#[ignore]d and skip cleanly withoutwasmtime, and must panic underPSRS_REQUIRE_WASMTIME=1so CI cannot let it silently pass.
Reading the expected result
Most passing files either print through Effect.Console or return a value from main as the exit code. Both are observable. Files that assert with Test.Assert fail by exit code. Decide and record how a file with no observable output is scored — as "compiles, validates, runs" rather than agreement.
Report honestly
Per-file pass/fail with the failure reason. Expect the first run to be near zero, because of Phase 2 and Phase 3. That number is the point: it is the honest L6/M7 baseline, and every later backend improvement is measured against it.
Acceptance
A runtime scoreboard reports per-file compile/validate/run agreement for all 413 non-FFI passing files, and the D-04 L6/M7 row carries the number.
Activity
biuld commentedon Oct 1, 2026
Landed in #107. The gate now has a number, and the number is 0/413 — which is the point of the issue.
Measured baseline: 0 of 413 non-FFI
passingfiles compile, validate, and run (26 excluded as FFI), reported per file byruntime::l6_runtime_scoreboard.mainto select)How agreement is defined, and why. The corpus vendors no execution goldens and upstream's
passingsuite is a compile-time suite, so a file agrees when the pipeline compiles it, the component passes Wasm validation, and the guest runs to completion without trapping.main's return value is recorded as the exit code and stdout is captured rather than compared, somain = 42agrees with exit code 42. A trap or a hang fails, and aTest.Assertfailure has to reach the guest as a trap to be visible at all.The design constraints in the issue, resolved:
run_program_with_wasmtimeintests/integration.rsis deleted. It shadowed the shared helper and had noPSRS_REQUIRE_WASMTIMEhandling. Every execution helper now goes through onewasmtime_availablecheck that also verifieswasmtime --versionsucceeds.rgovertests/upstream/passingfinds noget-arguments, no stdin read, and no filesystem use. That part of the issue's premise does not hold for the vendoredv0.15.16corpus, so no harness path was built for it.wasmtimeand panics underPSRS_REQUIRE_WASMTIME=1.main = 42through the same path and asserts exit code 42: a board that reports zero because its runner is broken is worse than no board. The outcome classifier is a pure function with six unit tests, so all three shapes are covered without a trapping program.Two corrections to the issue's premises.
stdlib/libis already on the scoreboard module path throughcompile_program_sources_with_prelude, so the remaining library gap is #94/#95, not a harness change. And 13 of the 229 missing-module rejections name a sibling corpus module the loader does not read —M1,A,B,M2,M3,Foo,Coercible.Lib2— which is exactly #83's defect 1, so the honest library-blocked count is 216 and the board prints the missing module names to keep the two categories apart until #83 splits them.Evidence:
cargo fmt --all --check;cargo clippy --workspace --all-targets -- -D warnings;PSRS_REQUIRE_WASMTIME=1 cargo test --workspace(975 passed, 0 failed, 0 skipped); CI green on #107.