From 219ba140e84fd3a5366e74285f21ccbbb42f0392 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Thu, 20 Aug 2026 19:17:37 +0200 Subject: [PATCH 1/4] Reactivate the agentic eval against the rebuilt CLI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The eval workflow and its prompt still spoke the pre-rebuild language — the Snakemake shim, `lc run --universe`, ok/stale/missing states, the Containerfile + requirements.txt environment. Bring both up to what the CLI is now: - prompt.md: `lc materialize` is the build verb and `lc run` the probe; states are current/behind/stale and `lc materialize --check` is the gate; recipes import from the project's own locked environment (`uv add`, never `uv pip`) under the sandbox; a run refuses a dirty tree, so the agent commits its edits with plain git; `results/` is the engine's to write, and a foreign write is detected. The seed spec is now stated to be deliberately incomplete — completing the recipe templates and dependency contracts is part of the task. - eval.yml: the check step gates on `lc materialize --check --json` (plus a non-empty `.current`) instead of a jq shape no verb emits; setup seeds numpy/scipy/matplotlib into the project with `uv add` and commits the seed so the agent starts from a clean tree; the repo-venv `uv pip install` is dropped (recipes never saw it); the dormant PR plumbing gets its trigger back as a `run-eval` label; the artifact exclusions swap `.snakemake/` for `.lightcone/`. - The seed astra.yaml drops its `container:` key — the directive is ignored by decision, and no Containerfile exists to point at. - trace-analysis-prompt.md names the sandbox instead of the container environment. Verified locally: the seed validates against astra-tools 0.2.16, and the setup + check plumbing runs end-to-end against today's `lc` (the gate correctly fails with all three outputs planned on the unbuilt project). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx --- .github/workflows/eval.yml | 41 ++++++++++------- evals/prompt.md | 83 ++++++++++++++++++++-------------- evals/tasks/snae/astra.yaml | 2 - evals/trace-analysis-prompt.md | 2 +- 4 files changed, 75 insertions(+), 53 deletions(-) diff --git a/.github/workflows/eval.yml b/.github/workflows/eval.yml index 9594e56e..8278bc32 100644 --- a/.github/workflows/eval.yml +++ b/.github/workflows/eval.yml @@ -3,10 +3,12 @@ name: Eval # Minimal agentic eval: scaffold a project from the eval seed on the # runner, let Claude Code (with the astra plugin) build it using the # branch's lc/astra, then check the results with astra validate and -# lc status. +# lc materialize --check. on: workflow_dispatch: + pull_request: + types: [labeled] # Only one eval per PR at a time — cancel in-progress runs concurrency: @@ -19,7 +21,7 @@ permissions: jobs: eval: - if: github.event_name == 'workflow_dispatch' || github.event.pull_request.draft == false + if: github.event_name == 'workflow_dispatch' || github.event.label.name == 'run-eval' runs-on: ubuntu-latest # Step-level timeout on the agent step (below) is what actually # bounds the run: a job-level timeout would cancel the always() @@ -36,11 +38,9 @@ jobs: activate-environment: true - name: Install lightcone-cli (branch under test) - run: | - uv sync - # Scientific deps the task's analysis scripts will need — the - # agent can add more with `uv pip install` - uv pip install numpy scipy matplotlib + # The activated venv puts lc, astra, and (via the mirrored entry + # points) git-annex on PATH — the runner needs no system annex. + run: uv sync - name: Install Claude Code + astra plugin run: | @@ -69,6 +69,14 @@ jobs: rm -f universes/baseline.yaml astra universe generate -n baseline \ -d 'Default configuration using standard practices' + # Scientific deps the task's recipes import land in the + # project's own locked environment — recipes see nothing else; + # the agent adds more with `uv add` + uv add numpy scipy matplotlib + # lc materialize refuses a dirty tree — commit the seed so the + # agent starts from a clean one + git add -A + git commit -q -m 'Seed the eval task' - name: Run Claude Code id: agent @@ -91,11 +99,11 @@ jobs: working-directory: ${{ env.PROJECT }} run: | astra validate astra.yaml - lc status --universe baseline - # Fail unless every materializable output is ok - lc status --json | jq -e \ - '[.universes[].outputs[] | select(.status != "alias")] - | length > 0 and all(.status == "ok")' + lc status + # The gate: nothing failed, nothing left to make — and at least + # one output actually got materialized + lc materialize --check --json | jq -e \ + '.up_to_date and ((.current | length) > 0)' - name: Build report if: always() @@ -135,7 +143,7 @@ jobs: echo '
lc status' echo "" echo '```' - (cd "$PROJECT" && lc status --universe baseline) || echo "unavailable" + (cd "$PROJECT" && lc status) || echo "unavailable" echo '```' echo "" echo '
' @@ -198,11 +206,12 @@ jobs: uses: actions/upload-artifact@v4 with: name: eval-project - # include-hidden-files keeps the .lightcone-manifest.json - # provenance files; the venv and snakemake scratch are excluded + # include-hidden-files keeps the manifests, the run records in + # .git, and the annexed results; the venv and lc's per-run + # machine state are excluded path: | ${{ env.PROJECT }} !${{ env.PROJECT }}/.venv/** - !${{ env.PROJECT }}/.snakemake/** + !${{ env.PROJECT }}/.lightcone/** include-hidden-files: true if-no-files-found: warn diff --git a/evals/prompt.md b/evals/prompt.md index 15a7a9db..6c8a82f0 100644 --- a/evals/prompt.md +++ b/evals/prompt.md @@ -8,19 +8,32 @@ This project is driven by two CLIs — use them rather than improvising: `astra validate astra.yaml` checks it against the schema. If an `astra` skill or plugin is available in your environment, load it before reading or editing `astra.yaml` — it documents the full spec format. -- `lc` (lightcone-cli) is the execution layer, a thin shim over Snakemake: - - `lc run --universe baseline` materializes an output (and - anything upstream of it) by running the recipe commands declared in - `astra.yaml`. With no output ids it builds everything. It is - idempotent: re-running only rebuilds what is stale or missing. - - `lc status --universe baseline` reports each output as `ok`, `stale`, - or `missing`; `lc status --json` is the machine-readable form. +- `lc` (lightcone-cli) is the execution layer: + - `lc materialize` makes every output the spec declares, running each + recipe in dependency order and committing each result to git as it + lands, together with a provenance manifest. It refuses to start on + a dirty tree: commit your own edits first, with plain `git add` and + `git commit` — the project's git-annex filter handles large files + transparently, so never run a git-annex command yourself. + - `lc materialize ` (or `/`) narrows + a run to one output and whatever it depends on. Re-running is + idempotent: only what is stale gets remade — an output the spec now + defines differently, or one whose declared inputs changed. + - `lc status` reports each output as `current`, `stale`, or `behind`, + with the commit it was made at; `lc status --json` is the + machine-readable form. It always exits 0. The pass/fail gate is + `lc materialize --check`, which exits 1 while anything still needs + making. + - `lc run ` runs an ad-hoc command in the project + environment under the same isolation a recipe gets — useful for + probing why a recipe would fail. - Outputs land in `results/baseline//`, each with a - `.lightcone-manifest.json` provenance manifest written by the engine. - Files placed in `results/` by hand have no manifest and fail - verification — never write there yourself. - - When `lc run` fails, read the error and the Snakemake log it points - to, fix the script or spec, and re-run. + `.lightcone-manifest.json` manifest written and committed by the + engine. Never write into `results/` yourself: a hand-placed file + has no run record, and the engine detects the foreign write and + remakes the output. + - When a recipe fails, `lc materialize` reports which output failed + and why; fix the script or the spec, commit, and re-run. ## Recipe template grammar @@ -49,37 +62,39 @@ is how the engine orders the build. ## Environment -Recipes and your interactive shell run in two different environments — -keep them straight: +Recipes run in the project's own locked environment (`pyproject.toml` + +`uv.lock` + `.venv`), sandboxed: the project tree is read-only apart from +each recipe's own output directory under `results/`, and only declared +tools are executable. -- **Recipe commands run by `lc run`** may execute inside a container - built from the project's `Containerfile` + `requirements.txt` - (whenever `astra.yaml` declares a `container:` and a runtime is - available). Every package a recipe script imports must therefore be - listed in `requirements.txt` — add it there *before* running, and the - engine rebuilds the content-addressed image automatically. Host-side - installs never reach the container. -- **Your own shell commands** run on the host in an activated uv-managed - virtual environment with numpy, scipy, and matplotlib pre-installed. - For ad-hoc host tools use `uv pip install ` — plain `pip` is - not available in this venv. +- Every package a recipe script imports must be added to the project + with `uv add `, run in the project root, before + materializing. numpy, scipy, and matplotlib are already added. Plain + `pip` or `uv pip` installs reach nothing a recipe sees. +- A sandbox denial names the path or tool that was denied and the + remedy — follow the remedy rather than working around the sandbox. ## Build loop `astra.yaml` is the single source of truth: inputs, outputs, recipes, and -methodological decisions all live there — read it first. For each output -that needs materializing: +methodological decisions all live there — read it first. The seed spec is +deliberately incomplete: recipe commands do not yet pass their inputs, +decisions, or output directory, and outputs may be missing entries in +their `inputs:` / `decisions:` contracts. Completing the spec is part of +the task. For each output: -1. Read the recipe's `command` to see what script and arguments it expects. +1. Complete the recipe `command` so it references `{output}` and the + `{inputs.}` / `{decisions.}` the computation needs, and + declare everything it references in that output's `inputs:` / + `decisions:` lists. 2. Write the script at the path the command names, parameterizing every decision via argparse — never hardcode option values. -3. Run `lc run --universe baseline` to materialize it through - the engine. -4. Commit progress as you go. +3. Commit your edits, then run `lc materialize` (or + `lc materialize `) to build through the engine. -Build iteratively from upstream outputs to downstream. `lc status ---universe baseline` shows you what's `ok`, `stale`, or `missing` — you're -done when every output shows `ok` and `astra validate astra.yaml` passes. +Build iteratively from upstream outputs to downstream. `lc status` shows +where every output stands — you're done when `lc materialize --check` +passes and `astra validate astra.yaml` passes. Skip plan approval and interactive confirmations — this is an automated eval run. diff --git a/evals/tasks/snae/astra.yaml b/evals/tasks/snae/astra.yaml index 3bdf0fb5..41b9d647 100644 --- a/evals/tasks/snae/astra.yaml +++ b/evals/tasks/snae/astra.yaml @@ -9,8 +9,6 @@ description: | using maximum-likelihood (MAP) point estimation. This provides best-fit cosmological parameters as a building block for a larger analysis. -container: Containerfile - inputs: - id: union21 type: data diff --git a/evals/trace-analysis-prompt.md b/evals/trace-analysis-prompt.md index 1058b5f9..120f4eb0 100644 --- a/evals/trace-analysis-prompt.md +++ b/evals/trace-analysis-prompt.md @@ -8,7 +8,7 @@ Write a section titled exactly `### Confusion & pain points` containing - failed commands or errored tool results, and their root cause - moments where the agent misunderstood the spec format, the CLI - surface, or the execution/container environment + surface, the sandbox, or the execution environment - detours: reverse-engineering source code, probing the environment, redoing work, or fixing things the harness should have provided - workarounds the agent invented that hint at a product gap (these are From abf54d554e2e5460ed13b19daa86553066ed77c8 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Thu, 20 Aug 2026 19:20:15 +0200 Subject: [PATCH 2/4] Leave the eval project's dependencies to the agent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The setup step no longer pre-seeds numpy/scipy/matplotlib — declaring what the recipes import is part of what the eval measures. In exchange the prompt spells out the uv workflow: the project starts with no dependencies, `uv add` is the one way packages reach a recipe (it edits pyproject.toml, re-locks, and syncs in one step, and the result is committed like any other edit), and lock-bypassing installs reach nothing a recipe sees. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx --- .github/workflows/eval.yml | 6 ++---- evals/prompt.md | 13 +++++++++---- 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/.github/workflows/eval.yml b/.github/workflows/eval.yml index 8278bc32..a90ab5fe 100644 --- a/.github/workflows/eval.yml +++ b/.github/workflows/eval.yml @@ -69,10 +69,8 @@ jobs: rm -f universes/baseline.yaml astra universe generate -n baseline \ -d 'Default configuration using standard practices' - # Scientific deps the task's recipes import land in the - # project's own locked environment — recipes see nothing else; - # the agent adds more with `uv add` - uv add numpy scipy matplotlib + # No scientific deps are pre-seeded — declaring what the + # recipes import (`uv add`) is part of what the eval measures # lc materialize refuses a dirty tree — commit the seed so the # agent starts from a clean one git add -A diff --git a/evals/prompt.md b/evals/prompt.md index 6c8a82f0..be152292 100644 --- a/evals/prompt.md +++ b/evals/prompt.md @@ -67,10 +67,15 @@ Recipes run in the project's own locked environment (`pyproject.toml` + each recipe's own output directory under `results/`, and only declared tools are executable. -- Every package a recipe script imports must be added to the project - with `uv add `, run in the project root, before - materializing. numpy, scipy, and matplotlib are already added. Plain - `pip` or `uv pip` installs reach nothing a recipe sees. +- The project is managed by uv and starts with **no dependencies**. + Every package a recipe script imports must be declared before + materializing: run `uv add [ ...]` in the project + root (e.g. `uv add numpy scipy`). That updates `pyproject.toml`, + re-locks `uv.lock`, and syncs `.venv` in one step — commit all of it + along with your scripts, like any other edit. +- To remove a package use `uv remove `; to pin a version, + `uv add 'numpy>=2'`. Do **not** use plain `pip` or `uv pip install` — + an install that bypasses the lock reaches nothing a recipe sees. - A sandbox denial names the path or tool that was denied and the remedy — follow the remedy rather than working around the sandbox. From c95a56a518bc9c9e21f710de2f4683f6116f4fa5 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Thu, 20 Aug 2026 20:47:20 +0200 Subject: [PATCH 3/4] Install lc and astra as uv tools in the eval MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The engine is the host's uv tool, so the eval now installs it that way: `uv tool install` of the checkout — the commit under test, stamped into the version by hatch-vcs — links lc and, through the mirrored entry points, the git-annex executables, exercising the exact mechanism a user's `uv tool install lightcone-cli` relies on. astra's executable belongs to astra-tools and becomes its own tool install, pinned to the branch's dependency version. The repo venv and its activation are gone with `uv sync`; nothing sets VIRTUAL_ENV anymore, which also removes the benign environment-mismatch warning uv printed at every project-scoped `uv add`. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx --- .github/workflows/eval.yml | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/.github/workflows/eval.yml b/.github/workflows/eval.yml index a90ab5fe..57490e7e 100644 --- a/.github/workflows/eval.yml +++ b/.github/workflows/eval.yml @@ -35,12 +35,21 @@ jobs: with: python-version: "3.12" enable-cache: true - activate-environment: true - - name: Install lightcone-cli (branch under test) - # The activated venv puts lc, astra, and (via the mirrored entry - # points) git-annex on PATH — the runner needs no system annex. - run: uv sync + - name: Install lightcone-cli + astra (uv tools) + # The engine is the host's uv tool — install the branch under + # test the way a user installs a release: `uv tool install`, + # which links `lc` and (via the mirrored entry points) the + # git-annex executables into ~/.local/bin. The commit under test + # is the checkout itself, stamped into the version by hatch-vcs. + # astra's executable belongs to astra-tools, so it is its own + # tool install, pinned to the version the branch depends on. No + # venv is activated: the agent's shell sees the tools exactly as + # an end user's would. + run: | + uv tool install "$GITHUB_WORKSPACE" + uv tool install "astra-tools==$(grep -oP 'astra-tools==\K[0-9][0-9.]*' pyproject.toml)" + echo "$HOME/.local/bin" >> "$GITHUB_PATH" - name: Install Claude Code + astra plugin run: | From 774997eac376c641611bc588fe234adc5efaedaa Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Thu, 20 Aug 2026 20:58:24 +0200 Subject: [PATCH 4/4] The eval task ends at publication MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After every output is materialized, the agent is asked to declare an SPDX license in pyproject.toml and materialize once more — publication intent derived exactly as layer 8 designed it, and the crate converges on a run that remakes nothing. The check step gates on the crate file existing, so the RO-Crate path is exercised by every eval run instead of warning its absence into the report. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx --- .github/workflows/eval.yml | 3 +++ evals/prompt.md | 19 +++++++++++++++++-- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/.github/workflows/eval.yml b/.github/workflows/eval.yml index 57490e7e..5b24e23e 100644 --- a/.github/workflows/eval.yml +++ b/.github/workflows/eval.yml @@ -111,6 +111,9 @@ jobs: # one output actually got materialized lc materialize --check --json | jq -e \ '.up_to_date and ((.current | length) > 0)' + # The publication view: declaring a license and materializing + # once more is part of the task + test -f ro-crate-metadata.json - name: Build report if: always() diff --git a/evals/prompt.md b/evals/prompt.md index be152292..c0f948f4 100644 --- a/evals/prompt.md +++ b/evals/prompt.md @@ -98,8 +98,23 @@ the task. For each output: `lc materialize `) to build through the engine. Build iteratively from upstream outputs to downstream. `lc status` shows -where every output stands — you're done when `lc materialize --check` -passes and `astra validate astra.yaml` passes. +where every output stands. + +## Publication + +Once every output is materialized, prepare the repository for +publication: + +1. Declare a license in `pyproject.toml`, as an SPDX expression under + `[project]` — e.g. `license = "CC-BY-4.0"`. Declaring one is what + turns publication on: from then on `lc materialize` also maintains + `ro-crate-metadata.json` at the project root, an RO-Crate view of + the project and its provenance. +2. Commit the edit, then run `lc materialize` once more — nothing is + remade, but the crate document is generated and committed. + +You're done when `astra validate astra.yaml` and +`lc materialize --check` pass and `ro-crate-metadata.json` exists. Skip plan approval and interactive confirmations — this is an automated eval run.