Skip to content

developer_overview.md

Latest commit

 

History

History
152 lines (126 loc) · 9.04 KB

developer_overview.md

File metadata and controls

152 lines (126 loc) · 9.04 KB
Raw

Developer overview

Audience: contributors changing eval-magic itself. For installed-tool usage, start with eval-magic --help; for harness authoring and dispatch isolation, run eval-magic docs.

eval-magic is a Rust CLI that builds reproducible skill-evaluation campaigns, hands dispatches to an agent harness, ingests what happened, grades the results, and preserves the evidence needed to compare conditions. This page is the repository map and documentation-placement policy for new contributors. It deliberately points to authoritative code, schemas, generated artifacts, and focused internal notes instead of duplicating their details.

How an evaluation moves through the system

  1. eval-magic init scaffolds an eval workspace next to a skill. Eval definitions describe the task, fixtures, assertions, conditions, run count, and — for a multi-turn eval — either scripted follow-up turns or a responder policy that derives them.
  2. eval-magic run validates the configuration, resolves and copies the skill under test into the iteration, creates isolated task roots, stages the requested skill condition from that copy, snapshots the starting state, and writes RUNBOOK.md, dispatch.json, and related campaign artifacts. The iteration lives in the eval home, which defaults outside the skill's own repository (workspace_root_from, src/core/context.rs). The generated runbook—not a checked-in recipe—is the authority for dispatching that particular campaign.
  3. eval-magic dispatch runs every task through the selected harness, --jobs at a time, each in its own private environment. A scripted conversation resumes one native harness session across its turns. Each task ends with a conversation.json, which is also what a rerun skips on.
  4. eval-magic ingest reads the harness outputs, transcript evidence, guard denials, and final task state. Runner-owned deterministic checks and diff-scope evidence are collected here.
  5. eval-magic grade evaluates runner-owned assertions and emits tasks for assertions that require an LLM. eval-magic dispatch --judges runs those judge tasks through the selected harness.
  6. eval-magic finalize checks that required work is complete and writes the final per-run and benchmark artifacts. eval-magic aggregate combines campaigns when a larger comparison is needed.
  7. eval-magic teardown removes staged skills and temporary guard configuration. Campaign artifacts remain available for audit and comparison.

Use each command's --help before changing a phase. It documents the current inputs, outputs, preconditions, handoffs, and recovery commands.

Repository map

  • src/main.rs is the thin binary entry point; src/lib.rs exposes reusable crate logic.
  • src/cli/ owns argument parsing, user-facing help, command handlers, and presentation. Library modules return data and warnings; the CLI decides what to print.
  • src/pipeline/ owns campaign phases and artifact assembly.
  • src/adapters/ loads harness descriptors and implements the shared adapter boundary plus the few named capabilities that require harness-specific code.
  • src/sandbox/, src/workspace/, and src/validation/ own task isolation, filesystem/workspace mechanics, and configuration checks.
  • src/source/ resolves a declared source — a git URL and ref, or a local directory — to a commit, and materializes it as a tree. It knows nothing about what is being sourced, so both the codebase a task environment is built from and the skills under test resolve through it.
  • schema/ contains the JSON schemas for user input and generated artifacts.
  • harnesses/ contains built-in descriptors, descriptor scaffolding, and embedded harness assets.
  • profiles/ contains shared prompt profiles.
  • tests/cli/ covers CLI and packaging contracts; tests/run/ covers campaign behavior across the run boundary. Focused unit tests normally live beside the implementation.
  • docs/guides/ contains Markdown guides embedded in the binary. Other files under docs/ are internal contributor notes.

Sources of truth

When prose and a machine-readable or generated surface disagree, fix the prose against the following authorities:

  • eval-magic <command> --help for the installed CLI contract.
  • The relevant file under schema/ for accepted fields and serialized artifact shapes.
  • Built-in and layered descriptor data for harness behavior. Use eval-magic harness list to see registered harnesses and eval-magic harness show <label> to inspect the resolved descriptor.
  • A campaign's generated RUNBOOK.md, dispatch.json, and dispatch-manifest.md for the exact commands and handoffs of that run.
  • Real harness --help, vendor documentation, and observed output for descriptor values. Never infer one harness's flags or event shapes from another harness.
  • Tests and golden artifacts for behavior that crosses a module or CLI boundary.

Platform support

Tier Platform Verified by
Supported Linux, macOS, Linux inside WSL the ubuntu-latest CI job
Unsupported native Windows —

Windows users run the Linux build inside Windows Subsystem for Linux (WSL). Keep the binary, repository, eval workspaces, and harness processes inside the same WSL environment. dispatch passes workspace-owned absolute paths to harness command lines, so crossing from a native Windows process into WSL would change the filesystem namespace and invalidate those paths.

Do not add native Windows accommodations or release targets. Preserve support for Windows-shaped paths only where they are data read from artifacts or transcripts; those portable-data contracts do not imply native Windows runtime support. POSIX_TOOLING_REQUIREMENT (src/core/runtime.rs) is the single wording every user-facing Markdown surface reuses. src/cli/help.rs restates it for clap by hand.

Make and verify a change

Trace the user-visible behavior from the CLI handler into library-owned logic and artifacts before editing. Add a focused failing test at the narrowest useful boundary, implement the change, then run the focused test again. Cross-harness changes belong at shared descriptor, runner, or adapter boundaries unless the evidence requires a named harness capability.

Development requires Linux or macOS with a POSIX shell. Windows contributors clone the repository and run the complete toolchain inside WSL; native Windows development is unsupported. The dispatch tests spawn #!/bin/sh harness stubs through the resolved shell and do not skip, so the suite cannot pass without one. Tests needing symlink creation report a skip instead; EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1 turns those skips into failures in CI.

Before handing work off, run:

cargo fmt --check
cargo build
cargo test
cargo clippy --all-targets -- -D warnings
git diff --check

Add schema checks, help snapshots, guide-contract tests, or golden-artifact verification when the changed surface calls for them.

Documentation policy

Documentation is a shipped feature. Put each fact on the surface where its audience will look:

  1. CLI help (src/cli/args.rs and src/cli/help.rs) is the primary discovery surface. Every command and flag needs enough context to use it correctly and find the next handoff.
  2. Shipped guides are Markdown files directly under docs/guides/. build.rs discovers every .md file at compile time: the filename stem is the eval-magic docs <topic> name, the first H1 is its listing title, and the complete file is embedded verbatim. Guide names must be ASCII kebab-case. Shipped output refers to guides with eval-magic docs <topic>, not repository paths.
  3. Generated run documentation such as RUNBOOK.md is the authority for commands whose exact form depends on a campaign, harness, or model. Do not duplicate those recipes in static prose.
  4. Internal development docs explain architecture, evidence, and maintenance contracts for contributors. They may link to repository paths, but installed-tool users must not depend on them.

Keep the README as a short landing page: what the tool is, how to install it, one successful first run, and where to continue. Distill operational detail into CLI help or a shipped guide; retain implementation evidence in an internal note.

Internal guide index