Skip to content

Phase 1: make every suite gate measurable #74

Closed
4 of 4 issues completed
Closed

Description

@biuld

Why this phase is first

Three of the eight gate rows in docs/design/D-04-suite-roadmap.md cannot be improved blind, because nothing measures them:

Gate Current state
L4 core type checking No scoreboard, and TypeCheckError::error_code maps only EscapedSkolem and InvalidCoercibleInstanceDeclaration. A type mismatch is reported with a message and no code.
L5 classes and instances No scoreboard, and none of the mapped class codes is emitted.
L6/M7 runtime No harness compiles and runs a corpus passing file.
M8-W warnings The corpus's @shouldWarnWith headers are read by nothing.
M8-O optimization optimize is not vendored; its goldens are .out.js, not CoreFn.

Until these exist, every later phase can claim progress that no number supports. This phase buys the ability to measure; it changes no user-visible language behavior.

Scope

  1. Official errorCode mapping for type and class diagnostics. psrs-typecheck's TypeCheckErrorKind already distinguishes TypeMismatch, OccursCheck, IntegerOutOfRange, NoInstance, OverlappingInstances, MissingInstanceMethod, AmbiguousConstraint, and FundepConflict, but error_code() returns None for all of them. Map the ones with an unambiguous purs counterpart. This alone makes 61 M4 cases and 92 M5 cases measurable.
  2. A lenient type-check entry point. psrs_driver exposes check_program_lenient (resolution) and check_program_kinds_lenient; there is no type-check equivalent that tolerates missing support libraries, so L4/L5 cannot be scored against the corpus the way L2/L3 are.
  3. L4 and L5 scoreboards, following the existing l2_resolution_scoreboard_with_annotations / l3_kind_scoreboard_with_annotations pattern: read @shouldFailWith, resolve leniently, report per-code agreement.
  4. Harness module path. The scoreboard loads only a case's same-stem support directory and provides no library modules. Two consequences, both measured: 13 passing files fail because a case's other siblings in the same directory are not loaded (passing/Import/M2.purs, passing/Coercible/Lib.purs), and 230 fail purely on library modules. Putting stdlib/lib on the scoreboard's module path and loading a case's sibling modules from the containing directory is a harness-only change.
  5. A runtime compile/run scoreboard for passing, and a warning-code scoreboard for warning.
  6. A stated M8-O comparison. Decide what "optimize output agrees" means when the goldens are JavaScript, and record it.

Gate

Every row of the D-04 gate table carries a live measured number, and the L1–L3 numbers in that document are reproducible from the vendored corpus with no external dependency.

Sub-issues

Tracked as children of this epic.

References

  • docs/design/D-04-suite-roadmap.md — Progress measurement, gate table
  • crates/psrs-driver/tests/suite.rs — the L1/L2/L3 scoreboards to extend
  • crates/psrs-typecheck/src/typecheck/error.rs — TypeCheckErrorKind and error_code
  • crates/psrs-driver/src/program/lenient.rs — the lenient entry points

Validation

cargo fmt --all --check
cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings
PSRS_ORACLE=annotations cargo test -p psrs-driver --test suite -- --ignored --nocapture

Activity

  1. added
    area:harnessScoreboard, corpus, or measurement infrastructure
    and removed
    area:harnessScoreboard, corpus, or measurement infrastructure
    on Oct 1, 2026
  2. biuld commented on Oct 2, 2026

    @biuld
    OwnerAuthor

    Phase 1 is complete. All four slices are closed, and the four gates that were unmeasurable now report numbers.

    Slice PR What it made measurable
    #93 #107 The L6/M7 runtime scoreboard: compile, validate, and run every non-FFI passing file.
    #83 #108 Each case is resolved the way the compiler loads a program, with the library on the module path.
    #81 + #82 #110 The official type and class errorCodes, a lenient type-check entry point, and the L4/L5 scoreboards.

    Every gate the phase set out to measure now has a number:

    Gate Before Now
    L1 parse 904/908 904/908
    L2 resolution 54/70 failing, 36/413 passing 54/70 failing, 52/413 passing
    L3 kinds 27/48 27/48
    L4 types not measured 12/38
    L5 classes not measured 37/74
    L6/M7 runtime not measured 0/413
    M8-W warnings not measured not measured (#91)

    None of the phase's slices changed a user-visible compiler behavior, which is what the phase promised. The harness work also removed a real defect: integration.rs had a second private run_program_with_wasmtime that shadowed the shared one and lacked PSRS_REQUIRE_WASMTIME handling, so an execution test could silently skip under the required baseline.

    What the numbers say, honestly: the phase made the gates measurable, not passing. The single largest finding is that most of what remains is not a checker gap — 16 of 26 M4 mismatches, 26 of 37 M5 mismatches, and 214 of 361 unresolved passing files are blocked on a library module that does not exist yet. That is Phase 3 (#94 Prelude, #95 Effect/Effect.Console/Test.Assert), and the boards now say so per case rather than reporting one number.

    Evidence for each slice is in its closing comment; CI is green on #107, #108, and #110.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions