Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitattributes
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Check every text file out with LF on all platforms. Generated artifacts embed the bytes of
# descriptors, profiles, and docs verbatim, and the byte-exact tests compare against LF fixtures, so
# a CRLF working tree on Windows silently changes program output.
# line-ending conversion would silently change program output.
* text=auto eol=lf

# Golden fixtures are byte-exact — never EOL-normalize them.
Expand Down
33 changes: 4 additions & 29 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,7 @@ env:
jobs:
test:
name: Test suite
strategy:
# Windows is the platform this matrix exists to cover, so a Linux failure
# must not cancel it — that is exactly when its result is worth having.
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
runs-on: ${{ matrix.os }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Rust toolchain
Expand All @@ -31,33 +25,14 @@ jobs:
components: rustfmt, clippy
- name: Cache cargo build
uses: Swatinem/rust-cache@v2
- name: Install jq
# Git for Windows supplies sh, xargs, tr, and wc, but not jq, and the
# judge-recipe tests execute the shipped pipeline text rather than a
# stand-in for it.
if: runner.os == 'Windows'
run: choco install jq --yes --no-progress
- name: Permit symlink creation
# Windows creates symlinks only under Developer Mode or elevation, and
# the core::fs round-trips need one. Asking for it explicitly beats
# depending on how the runner's token happens to be built.
if: runner.os == 'Windows'
run: >
reg add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock"
/t REG_DWORD /f /v AllowDevelopmentWithoutDevLicense /d 1
- name: Format check
run: cargo fmt --all -- --check
- name: Clippy
run: cargo clippy --all-targets --all-features -- -D warnings
- name: Test
# Capability-gated tests (the POSIX recipe pipelines, symlink
# round-trips, long-path staging) skip with a printed reason on a host
# that lacks the capability. This turns every such skip into a failure,
# so neither runner can quietly stop covering them. Ubuntu ships the
# recipe tools; the steps above provide them on Windows. Long paths need
# no provisioning — the runner passes core.longpaths to git itself.
# EVAL_MAGIC_SH stays unset deliberately: discovering the shell from the
# Git install root is what a Windows user hits, so CI should run it too.
# Capability-gated tests skip with a printed reason on a host that lacks
# the capability. CI turns every such skip into a failure so coverage
# cannot shrink silently.
env:
EVAL_MAGIC_REQUIRE_POSIX_TOOLS: 1
run: cargo test --all-targets
Expand Down
7 changes: 1 addition & 6 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -113,9 +113,6 @@ jobs:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BUILD_MANIFEST_NAME: target/distrib/${{ join(matrix.targets, '-') }}-dist-manifest.json
steps:
- name: enable windows longpaths
run: |
git config --global core.longpaths true
- uses: actions/checkout@v6
with:
persist-credentials: false
Expand Down Expand Up @@ -146,9 +143,7 @@ jobs:
echo "dist ran successfully"
- id: cargo-dist
name: Post-build
# We force bash here just because github makes it really hard to get values up
# to "real" actions without writing to env-vars, and writing to env-vars has
# inconsistent syntax between shell and powershell.
# Force bash so every release runner writes GitHub outputs with one syntax.
shell: bash
run: |
# Parse out what we just built and upload it to scratch storage
Expand Down
43 changes: 19 additions & 24 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,40 +54,35 @@ Extraction is a size decision, not a style preference; don't split a small inlin
**Spawning a child process from a test.** Use the hidden `__fixture` subcommand, never `sh`, `true`,
`printf`, or a `#!/bin/sh` stub. It exits with a chosen code, emits chosen bytes, writes a chosen
file, or checks a file or variable — see `FixtureArgs` in `src/cli/args.rs`. One invocation parses
the same under `sh -c` and `cmd /C`, which is what keeps `command_check` tests off per-OS command
strings. Build the command with the `fixture` helper (`tests/run/helpers.rs` for integration tests,
the one in `src/pipeline/grade/command_check/tests.rs` for unit tests). Because the fixture is the
binary, `cargo test --lib` alone does not build it — run `cargo test`, or `cargo build` first.

**Tests are gated on capabilities, not on the OS.** `#[cfg(unix)]` on a test hides it from
compilation and clippy on the other host and hides the coverage gap. Instead, probe for what the
test actually needs and call `report_skip` (`src/core/runtime.rs`), which prints the reason and
returns `true`. Setting `EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1` turns every skip into a failure; CI sets
it on both runners, so neither can quietly stop covering something. Two capabilities are gated
today: symlink creation, which Windows allows only under Developer Mode, and creating a path past
Windows' 259-character limit (`deep_task_root`, `src/cli/run/orchestrate/git.rs`). The Windows
runner is provisioned for both rather than exempted from them, so a skip there is a red build. The
shell is not one of them; it is a hard requirement, per the section below. Where a genuine per-OS
difference is the behavior under test — signals, path separators — branch on `cfg!(windows)` at
runtime so both arms still compile everywhere.
predictably under `sh -c`, which keeps `command_check` tests focused on runner behavior instead of
the host's utility implementations. Build the command with the `fixture` helper
(`tests/run/helpers.rs` for integration tests, the one in
`src/pipeline/grade/command_check/tests.rs` for unit tests). Because the fixture is the binary,
`cargo test --lib` alone does not build it — run `cargo test`, or `cargo build` first.

**Tests are gated on capabilities, not on broad platform labels.** Probe for what the test actually
needs and call `report_skip` (`src/core/runtime.rs`), which prints the reason and returns `true`.
Setting `EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1` turns every skip into a failure; CI sets it so coverage
cannot quietly shrink. Symlink creation is capability-gated because the backing filesystem may
forbid it. The shell is not capability-gated; it is a hard requirement, per the section below.

**A POSIX shell is required, for use and for development.** Harness `exec_template`s are POSIX
command lines, so the dispatch and probe paths spawn `sh` through `run_in_posix_shell` /
`posix_shell()` (`src/core/runtime.rs`) rather than a hardcoded `/bin/sh`: it searches `PATH`, then
a Git for Windows install. Set `EVAL_MAGIC_SH` to override it. `cargo test` inherits the
requirement — the dispatch tests spawn a `#!/bin/sh` harness stub through the resolved shell and do
not skip — so a host without `sh` fails the suite instead of quietly covering less. The shell is
the whole requirement: `jq` was needed only while operators pasted the generated dispatch and judge
recipes, and `eval-magic dispatch` drives both itself.
checks `/bin/sh`. Set `EVAL_MAGIC_SH` to override it. `cargo test` inherits the requirement — the
dispatch tests spawn a `#!/bin/sh` harness stub through the resolved shell and do not skip — so a
host without `sh` fails the suite instead of quietly covering less. The shell is the whole
requirement: `jq` was needed only while operators pasted the generated dispatch and judge recipes,
and `eval-magic dispatch` drives both itself.
`POSIX_TOOLING_REQUIREMENT` (`src/core/runtime.rs`) is the one wording the
Markdown-carrying surfaces reuse: the shell-discovery errors, the `run` preflight warnings,
`RUNBOOK.md`, and `dispatch-manifest.md`. State the requirement from there rather than rephrasing
it. `--help` is the one deliberate restatement (`AFTER_HELP` in `src/cli/help.rs`), hard-wrapped and
backtick-free because clap renders into a terminal; keep the two in step by hand.

Which platforms that requirement is honored on — and why preparing on Windows but dispatching from
WSL is a correctness boundary rather than a preference — is stated once under "Platform support" in
`docs/developer_overview.md`.
eval-magic supports Linux and macOS. On Windows, use and develop eval-magic entirely inside WSL;
native Windows is unsupported. The complete boundary and the portable-data exception are stated
under "Platform support" in `docs/developer_overview.md`.

**Where user-facing warnings come from.** Library modules (`pipeline`, `workspace`, `sandbox`,
`adapters`) never print. They return warning strings on their result struct — `#[serde(skip)]` when
Expand Down
29 changes: 11 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,31 +37,22 @@ The installed CLI is the primary manual. Start with `eval-magic --help`, and use

## Install

Git is required at runtime, plus a POSIX shell: harness dispatch commands are POSIX command lines,
and `eval-magic dispatch` runs them itself, so the host it runs on needs a shell that resolves the
workspace's own paths. On Windows that is Git Bash (Git for Windows). WSL resolves a different
filesystem namespace, so run eval-magic inside WSL rather than dispatching into it.
Set `EVAL_MAGIC_SH` to select a specific `sh`.
eval-magic supports Linux and macOS. On Windows, install and run eval-magic inside Windows
Subsystem for Linux (WSL); native Windows is unsupported. Keep the repository, workspace, and
harness commands inside the same WSL environment.

Windows support runs through Git Bash and is deprecated: a future release will require WSL.
Git and a POSIX shell are required. Set `EVAL_MAGIC_SH` to select a specific `sh`.

Prebuilt binaries for macOS, Linux, and Windows are attached to each
Prebuilt binaries for macOS and Linux are attached to each
[GitHub release](https://github.com/slowdini/eval-magic/releases).

macOS or Linux:
Install on macOS, Linux, or inside WSL:

```bash
curl --proto '=https' --tlsv1.2 -LsSf \
https://github.com/slowdini/eval-magic/releases/latest/download/eval-magic-installer.sh | sh
```

Windows PowerShell:

```powershell
powershell -ExecutionPolicy Bypass -c \
"irm https://github.com/slowdini/eval-magic/releases/latest/download/eval-magic-installer.ps1 | iex"
```

Or build and install from crates.io:

```bash
Expand Down Expand Up @@ -151,9 +142,11 @@ Issues and planned work are tracked in the

## Development

Development carries the same host requirement as use: a POSIX shell. The dispatch tests spawn
`#!/bin/sh` harness stubs through the resolved shell and do not skip, so the suite cannot pass
without one. Tests that need symlink creation report a skip instead.
Development carries the same host requirement as use: Linux or macOS with a POSIX shell. On
Windows, clone the repository and run the complete toolchain inside WSL; native Windows development
is unsupported. The dispatch tests spawn `#!/bin/sh` harness stubs through the resolved shell and
do not skip, so the suite cannot pass without one. Tests that need symlink creation report a skip
instead.

```bash
cargo fmt --check
Expand Down
7 changes: 5 additions & 2 deletions dist-workspace.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,13 @@ members = ["cargo:."]
cargo-dist-version = "0.32.0"
# CI backends to support
ci = "github"
# cargo-dist adds native Windows setup even when no Windows targets remain.
# Keep the Linux/macOS-only workflow as an intentional local customization.
allow-dirty = ["ci"]
# The installers to generate for each app
installers = ["shell", "powershell"]
installers = ["shell"]
# Target platforms to build apps for (Rust target-triple syntax)
targets = ["aarch64-apple-darwin", "aarch64-unknown-linux-gnu", "x86_64-apple-darwin", "x86_64-unknown-linux-gnu", "x86_64-pc-windows-msvc"]
targets = ["aarch64-apple-darwin", "aarch64-unknown-linux-gnu", "x86_64-apple-darwin", "x86_64-unknown-linux-gnu"]
# Path that installers should place binaries in
install-path = "CARGO_HOME"
# Where to host releases
Expand Down
38 changes: 17 additions & 21 deletions docs/developer_overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,23 +75,19 @@ following authorities:

| Tier | Platform | Verified by |
| --- | --- | --- |
| Supported | Linux, macOS | the `ubuntu-latest` CI job |
| Deprecated | Windows, through Git Bash (Git for Windows) | the `windows-latest` CI job |
| Unsupported | preparing a workspace on Windows and dispatching it from WSL | — |

Windows support is deprecated in favor of WSL. #256 has landed, so the recipe surface that carried
the largest Windows accommodation is gone; the remaining removal — the `cfg(windows)` sites, the CI
leg, and the msvc target — is #275. Until that lands, the Windows runner stays green and
Windows-native behavior is held to the same bar as any other platform: a Windows failure is a real
failure, not an accepted gap. Do not add new Windows-native accommodation in the meantime.

The unsupported row is a correctness boundary rather than a preference. `dispatch` spawns each
harness command line with the workspace's own absolute paths, so the shell it resolves has to
resolve those. Git Bash shares the Windows filesystem, so those paths resolve; WSL resolves its own
namespace, where a `C:\…` path names nothing. Nothing in the tree translates between the two, so
the split fails quietly instead of loudly. `POSIX_TOOLING_REQUIREMENT` (`src/core/runtime.rs`) is
the single wording every user-facing surface reuses to state this; `src/cli/help.rs` restates it
for clap by hand.
| Supported | Linux, macOS, Linux inside WSL | the `ubuntu-latest` CI job |
| Unsupported | native Windows | — |

Windows users run the Linux build inside Windows Subsystem for Linux (WSL). Keep the binary,
repository, eval workspaces, and harness processes inside the same WSL environment. `dispatch`
passes workspace-owned absolute paths to harness command lines, so crossing from a native Windows
process into WSL would change the filesystem namespace and invalidate those paths.

Do not add native Windows accommodations or release targets. Preserve support for Windows-shaped
paths only where they are data read from artifacts or transcripts; those portable-data contracts
do not imply native Windows runtime support. `POSIX_TOOLING_REQUIREMENT` (`src/core/runtime.rs`) is
the single wording every user-facing Markdown surface reuses. `src/cli/help.rs` restates it for
clap by hand.

## Make and verify a change

Expand All @@ -100,11 +96,11 @@ editing. Add a focused failing test at the narrowest useful boundary, implement
run the focused test again. Cross-harness changes belong at shared descriptor, runner, or adapter
boundaries unless the evidence requires a named harness capability.

Development carries the host requirement the tool itself declares: a POSIX shell. The dispatch
Development requires Linux or macOS with a POSIX shell. Windows contributors clone the repository
and run the complete toolchain inside WSL; native Windows development is unsupported. The dispatch
tests spawn `#!/bin/sh` harness stubs through the resolved shell and do not skip, so the suite
cannot pass without one. Tests needing symlink creation or a path past Windows' 259-character limit
report a skip instead; `EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1` turns those skips into failures, as CI
sets it to do on both its Ubuntu and its Windows runner.
cannot pass without one. Tests needing symlink creation report a skip instead;
`EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1` turns those skips into failures in CI.

Before handing work off, run:

Expand Down
2 changes: 1 addition & 1 deletion schema/evals.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@
"command": {
"type": "string",
"minLength": 1,
"description": "Trusted eval-author command executed by the runner in the task environment after agent dispatch."
"description": "Trusted eval-author POSIX shell command executed by the runner with `sh -c` in the task environment after agent dispatch."
},
"env": {
"type": "object",
Expand Down
5 changes: 2 additions & 3 deletions src/adapters/skill_shadow.rs
Original file line number Diff line number Diff line change
Expand Up @@ -288,9 +288,8 @@ impl ShadowSource {
}
}

/// The resolved real path, rendered as wire format. `canonicalize` returns a
/// verbatim (`\\?\`) path on Windows, which `artifact_path` strips — an OS
/// escape hatch has no business in a report an agent and a reviewer both read.
/// The resolved real path, rendered in the artifact wire format shared by
/// agents and reviewers.
fn canonical_path(path: &Path) -> Option<String> {
path.canonicalize().ok().map(|path| artifact_path(&path))
}
Expand Down
7 changes: 3 additions & 4 deletions src/cli/args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -832,8 +832,8 @@ pub(crate) enum Commands {
},
/// Internal test fixture. A predictable child process for the suite to
/// spawn — one that exits with a chosen code, emits chosen bytes, or writes
/// a chosen file — so tests never reach for `sh`, `true`, or `printf`, none
/// of which exist under `cmd.exe`. Not for users; hidden from help.
/// a chosen file — so tests do not depend on the output conventions of
/// utilities such as `true` or `printf`. Not for users; hidden from help.
#[command(hide = true, name = "__fixture")]
Fixture(FixtureArgs),
/// Internal generic PreToolUse hook entry point. Invoked by the installed
Expand Down Expand Up @@ -882,8 +882,7 @@ pub struct FixtureArgs {
#[arg(long)]
pub pad: Option<usize>,
/// Sleep this many milliseconds before doing anything else, so a caller can
/// overrun a deadline. The delay lives here rather than in a `sleep` call
/// because Windows has no such binary.
/// overrun a deadline without depending on an external `sleep` binary.
#[arg(long = "sleep-ms")]
pub sleep_ms: Option<u64>,
/// Joins the fragments. Empty by default.
Expand Down
12 changes: 5 additions & 7 deletions src/cli/commands/fixture.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,9 @@
//!
//! Tests that exercise `command_check` grading need a program that exits with a
//! chosen status, emits chosen bytes, or writes a chosen file. Reaching for
//! `sh`, `true`, or `printf` ties those tests to POSIX, and the `cmd.exe`
//! equivalents are not equivalent — `echo x>>f` appends CRLF, and
//! `echo|set /p=` cannot round-trip a value. One fixture invoked the same way
//! under both shells removes the dialect problem entirely.
//! external utilities such as `true`, `printf`, or `sleep` would make their
//! platform-specific output and availability part of the test. The fixture
//! keeps those effects predictable.

use std::fs::{self, OpenOptions};
use std::io::{self, Write};
Expand Down Expand Up @@ -160,9 +159,8 @@ mod tests {
}

/// `--sleep-ms` delays the fixture before it does anything else, which is
/// what lets a dispatch-timeout test overrun a deadline on any host. `sleep`
/// is a POSIX binary Windows lacks, so the delay has to live in the fixture
/// itself.
/// what lets a dispatch-timeout test overrun a deadline without depending
/// on an external `sleep` binary.
#[test]
fn sleep_ms_delays_the_fixture_before_it_emits() {
let started = std::time::Instant::now();
Expand Down
10 changes: 4 additions & 6 deletions src/cli/help.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,10 @@
/// Worked examples shown at the end of `eval-magic --help`.
pub(super) const AFTER_HELP: &str = "\
REQUIREMENTS:
Git, plus a POSIX shell. Harness dispatch commands are POSIX command
lines, and eval-magic dispatch runs them itself, so the host it runs on
needs a shell that resolves the workspace's own paths. On Windows that is
Git Bash (Git for Windows). WSL resolves a different filesystem namespace,
so run eval-magic inside WSL rather than dispatching into it. Set
EVAL_MAGIC_SH to select a specific sh.
eval-magic supports Linux and macOS. On Windows, install and run eval-magic
inside WSL; native Windows is unsupported. Keep the repository, workspace,
and harness commands inside the same WSL environment. Git and a POSIX shell
are required. Set EVAL_MAGIC_SH to select a specific sh.

EXAMPLES:
# Scaffold a first eval and prepare its isolated comparison environments
Expand Down
Loading
Loading