Reactivate the agentic eval against the rebuilt CLI

[EiffL]

The eval workflow and its prompt were written for the pre-rebuild CLI — the Snakemake shim, lc run <output_id> --universe, ok/stale/missing states, the Containerfile + requirements.txt container model. With layer 8 landed, this brings the eval back in line with what lc actually is.

evals/prompt.md (rewritten)

• lc materialize is the build verb; lc run <command> is the probe. States are current/behind/stale, lc status always exits 0, and lc materialize --check is the gate.
• Recipes import from the project's own locked environment — packages arrive via uv add in the project root (what the denial messages recommend); uv pip reaches nothing a recipe sees.
• A run refuses a dirty tree, so the agent is told to commit its edits with plain git add/git commit (the annex filter is transparent — nobody runs a git-annex command).
• results/ is the engine's to write; a hand-placed file is now detected as a foreign write and remade — the old "fails verification" warning got teeth.
• The seed spec is stated to be deliberately incomplete: the recipe commands carry no placeholders and the dependency contracts are partial, so completing the spec ({output}, {inputs.<id>}, {decisions.<id>}, and the matching inputs:/decisions: lists) is part of the task. This makes the existing seed's shape an intentional test of spec literacy rather than a leftover.
• The recipe template grammar section is kept — verified it still matches astra's own grammar (astra guide, astra-tools 0.2.16).

.github/workflows/eval.yml

• Check step: the old jq expression (.universes[].outputs[], "ok", "alias") matches nothing any verb emits now. Replaced with the designed gate — lc materialize --check --json | jq -e '.up_to_date and ((.current | length) > 0)' — which exits nonzero while anything is planned or failed, plus the non-emptiness guard the old check had. behind deliberately does not fail the gate.
• Setup step: no scientific deps are pre-seeded — the project starts empty and declaring what the recipes import (uv add, spelled out in the prompt) is part of what the eval measures; the repo-venv uv pip install is dropped (it never reached recipes). Setup commits the seed — lc init never commits, and the first lc materialize would otherwise be refused on a dirty tree before the agent did anything.
• Trigger: the dormant PR plumbing (comment step, concurrency group) gets its trigger back — pull_request: types: [labeled] gated on a run-eval label, alongside workflow_dispatch.
• Artifacts: .snakemake/** exclusion → .lightcone/**; keeping .git via include-hidden-files is now a feature — the run records and annexed results are the provenance.
• Install: lc and astra are installed as uv tools — uv tool install of the checkout (the commit under test, stamped into the version by hatch-vcs) links lc and the four git-annex executables through the mirrored entry points, exercising the exact mechanism a user install relies on; astra-tools is its own tool install, pinned to the branch dependency. No venv is activated, so the runner needs no system git-annex and VIRTUAL_ENV is never set.

Seed + trace prompt

• evals/tasks/snae/astra.yaml drops container: Containerfile — the directive is ignored by recorded decision, and no Containerfile exists to point at.
• trace-analysis-prompt.md names the sandbox instead of the container environment.

Verified locally

• The seed validates against astra-tools 0.2.16 after the edit.
• The full setup + check plumbing runs end-to-end against today's lc: lc init → overlay → astra universe generate -n baseline → uv add → seed commit → lc status (three outputs, direct mode, landlock) → the --check gate correctly failing with all three outputs planned on the unbuilt project.

The agent step itself (Claude Code + astra plugin) is unchanged and needs a real dispatch run to shake out — suggest a workflow_dispatch on this branch once merged, or label this PR run-eval.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx

[github-actions]

✅ Eval

Metric	Value
Outputs check	success
Agent run	success
Turns	40
Tool calls	37
Cost	$1.19
Agent wall time	3m50s
Model	claude-sonnet-5

lc status

  mode:    direct
  sandbox: landlock (fs: declared, network: allowed)

  · current  baseline/best_fit        203627d
  · current  baseline/hubble_diagram  203627d
  · current  baseline/residuals       203627d

3 current

Confusion & pain points (Claude analysis)

Confusion & pain points

• lc run with a relative --output path crashed instead of writing where expected: the agent's ad-hoc test invocation (lc run python scripts/fit.py --output /tmp/lbfgs_test ...) failed with a FileNotFoundError because it referenced /tmp/lbfgs_test before creating the directory — the agent then had to mkdir -p it after the failed run. This is a minor scripting error on the agent's part, but it also reveals the agent reaching for an ad-hoc /tmp scratch path rather than using the spec's own decision/universe mechanism from the start.
• No supported way to "dry-run" alternate decisions without touching the committed spec: to verify the optimizer/error_model/redshift_cut decision axes worked, the agent had to fabricate a throwaway universes/_test_alt.yaml, commit it, materialize it, then delete the universe and its results and commit again — three extra commits (eb9dd3f, three [DATALAD RUNCMD] commits, b6eaf9c) just for exploratory validation. This points to a real product gap: there's no lightweight way to preview a universe's outputs without permanently polluting git history, since every materialize commits.
• lc materialize <universe> doesn't accept a bare universe name: lc materialize _test_alt errored with "no output matches _test_alt", forcing the agent to discover (via the error message itself) that it must enumerate every qualified output id (_test_alt/best_fit _test_alt/hubble_diagram _test_alt/residuals). This is a discoverability gap — the CLI's own error message had to teach the agent the correct invocation, which cost a round trip.
• RO-Crate output required a full round of manual reverse-engineering of publication mechanics: the agent had to read pyproject.toml, infer that adding license = "CC-BY-4.0" would trigger crate generation, then separately notice after a "nothing to do" materialize that ro-crate-metadata.json had appeared and been auto-committed as a second, unprompted commit (d6d191f). Nothing in the CLI output announced that the license change would produce this side effect, so the agent had to infer and verify it after the fact via ls and git log.
• Otherwise the run was largely clean: spec validation, the fit/plot recipes, and lc materialize/--check all worked on the first real attempt once the scripts were written.

Full trace: agent-trace artifact on this run.