diff --git a/README.md b/README.md index faaf15dc..6483fcf7 100644 --- a/README.md +++ b/README.md @@ -5,22 +5,33 @@ [![PyPI](https://img.shields.io/pypi/v/lightcone-cli?style=flat&color=f8f7f3)](https://pypi.org/project/lightcone-cli/) [![Tests](https://img.shields.io/github/actions/workflow/status/LightconeResearch/lightcone-cli/tests.yml?style=flat&color=darkgreen)](https://github.com/LightconeResearch/lightcone-cli/actions/workflows/tests.yml) - - **lightcone-cli** (`lc`) is the execution layer for [ASTRA](https://astra-spec.org/latest/) (Agentic Schema for Transparent Research Analysis). Describe your analysis in an `astra.yaml` -specification and `lc` takes care of the rest — execution and -provenance. +specification and `lc` takes care of the rest — execution, environments, +and provenance. ## Quick Start +**lightcone-cli** only requires you to have `uv` installed on your environment, and will take care of everything else. See how to install uv here: [https://docs.astral.sh/uv/getting-started/installation](https://docs.astral.sh/uv/getting-started/installation). + +Then to install **lightcone-cli**: +```bash +uv tool install lightcone-cli +``` + +Once the CLI is installed, you can use it to create an ASTRA project and generate outputs like so: + ```bash -uv tool install lightcone-cli # or: pip install lightcone-cli lc init my-analysis cd my-analysis -# describe your analysis in astra.yaml, then: -lc run +# describe your analysis in astra.yaml, write your scripts, +# declare what they import through normal uv interactions: +uv add numpy +# When you are done with your edits, commit: +git add -A && git commit -m "First analysis" +# Use the lightcone CLI to generate your outputs with full provenance tracking +lc materialize ``` ASTRA specs are plain, structured YAML — they work well hand-written or @@ -30,10 +41,11 @@ drafted with any AI coding assistant. ## Capabilities -- **Multiverse analysis** — define methodological decisions with multiple options; `lc` runs your analysis across all defensible paths automatically -- **Provenance integrity** — every output gets a content-addressed manifest; `lc verify` detects tampering or broken chains -- **HPC-ready execution** — Snakemake-backed DAG dispatch with SLURM and container support (Docker, Podman, Apptainer) out of the box -- **Reproducible publishing** — `lc export wrroc` emits a [Workflow Run RO-Crate](https://www.researchobject.org/workflow-run-crate/) bundle ready for Zenodo or WorkflowHub +- **Multiverse analysis** — declare methodological decisions with multiple defensible options; `lc` materializes your analysis across every universe you define +- **Provenance by construction** — every output is committed to git together with a content-addressed manifest and a re-runnable run record; git-annex carries the bytes, so results travel with the repository +- **Locked, isolated execution** — a project's environment is `pyproject.toml` + `uv.lock`; recipes run in it under a sandbox (Landlock on Linux, Seatbelt on macOS) that keeps undeclared files out and stray writes contained +- **Containers and HPC** — declare `[tool.lightcone.image]` and recipes run in a content-addressed image archived in the repository itself; a SLURM allocation is detected and used automatically, every node included +- **Publication view** — declare a license and `lc materialize` maintains an [RO-Crate](https://www.researchobject.org/ro-crate/) of the project and its provenance, ready to archive or deposit → [Full documentation](https://docs.lightconeresearch.org) diff --git a/docs/index.md b/docs/index.md index 725ba7e0..31464072 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,7 +6,7 @@ It serves as the machinery that ties an analysis `astra.yaml` specification to a of materialized outputs. !!! warning "Alpha development" - lightcone-cli is in **early alpha**. The CLI, skills, and execution layer are all + lightcone-cli is in **early alpha**. The CLI and the execution layer are still moving — expect breaking changes between minor versions. Bug reports, design challenges, and use cases the tooling doesn't yet cover are exactly what we want to hear at this stage; please open an issue on the @@ -43,7 +43,7 @@ of materialized outputs. - __lightcone-cli__ - The library that ships the `lc` CLI: the workflow execution layer and the provenance integrity layer. Depends on [**astra-tools**][astra-tools], the SDK for working with ASTRA analysis specifications. + The library that ships the `lc` CLI: project scaffolding, locked environments, sandboxed execution, and the provenance layer. Depends on [**astra-tools**][astra-tools], the SDK for working with ASTRA analysis specifications. [:fontawesome-brands-github: Repository][cli]{ .md-button } diff --git a/docs/user/getting-started.md b/docs/user/getting-started.md index d16c6d79..3a61d1fd 100644 --- a/docs/user/getting-started.md +++ b/docs/user/getting-started.md @@ -4,19 +4,19 @@ Let's go from nothing on your disk to a working, reproducible analysis. You can read this top to bottom without running anything, or follow along — every command is copy-paste ready. -**What you'll build:** a small two-output analysis that fits a linear model on -a public dataset and sweeps one methodological decision (whether to standardize -features). The result is two universes, `baseline` and `raw`, each with its -own `r2` metric and `fit_plot` figure — a clean comparison ready for a paper -figure. +**What you'll build:** a small two-output analysis that fits a line to a +noisy dataset and sweeps one methodological decision — whether points far +from an initial fit are kept or clipped. The result is two universes, +`baseline` and `robust`, each with its own fitted slope and figure, and a +project that ends published as an [RO-Crate](https://www.researchobject.org/ro-crate/). Make sure you've finished the [install](install.md) first. ## 1. Create a project ```bash -lc init r2-decision-demo -cd r2-decision-demo +lc init line-fit-demo +cd line-fit-demo ``` `lc init` converges the directory to a small, opinionated layout and @@ -24,97 +24,151 @@ stops; it doesn't ask any questions, and it's idempotent — re-running it later only fills in whatever is missing. ``` -r2-decision-demo/ +line-fit-demo/ ├── astra.yaml # the spec — this is where everything lives +├── pyproject.toml # the project's environment: its dependencies… +├── .python-version # …and the exact interpreter, locked by uv +├── uv.lock +├── .venv/ # built from the lock (local, never committed) +├── .git/ # a git repository, with git-annex initialized +├── .gitattributes # the storage policy: what the annex carries ├── .gitignore -├── .git # initialized git repository (skip with --no-git) -├── .venv/ # Python virtual env with the analysis dependencies (skip with --no-venv) -├── .lightcone/ # internal scratchpad — don't edit by hand -├── Containerfile # build instructions for the project container -├── requirements.txt # software dependencies -├── myst.yml # MyST report configuration -├── index.md # template report that references the spec +├── .datalad/ # dataset identity — the project is a DataLad dataset +├── data/ # declared input data lives here +├── results/ # outputs materialize here — lc's to write, not yours ├── universes/ │ └── baseline.yaml # one universe, built from decision defaults -└── results/ - └── README.md # outputs materialize here via `lc run` +├── myst.yml # MyST report configuration +└── index.md # template report that references the spec ``` -The file you'll actually work in: +Two things are worth registering now: -**`astra.yaml`** — the single source of truth for your analysis. Inputs, -outputs, methodological decisions, recipes. Everything else lightcone-cli does -is downstream of this file. The boilerplate from `lc init` has one example -output and an example decision — enough to run `lc run` and see something -materialize, but not yet a real analysis. +- **The project is a git repository.** Every + output `lc` makes is committed together with the code that produced + it; large files ride in git-annex behind the scenes, but you only + ever type ordinary `git add` and `git commit`. +- **The environment is the lock.** `pyproject.toml` + `uv.lock` define + exactly what your recipes can import, and `.venv` is built from them. + You'll add packages with `uv add` in a moment — never `pip install`. -ASTRA specs are plain YAML, designed to be easy for both humans and AI -assistants to write. In this guide you'll write one by hand — it's short. +The file you'll actually work in is **`astra.yaml`** — the single source +of truth for your analysis. Inputs, outputs, methodological decisions, +recipes: everything else lightcone-cli does is downstream of this file. -## 2. Write the spec +## 2. Add the data -Open `astra.yaml` and replace the boilerplate with our analysis: a linear -regression on sklearn's bundled diabetes dataset, with one decision — whether -to standardize features before fitting. +A real project starts from a dataset; ours will generate a small one — +200 points on a line, with a few outliers thrown far off it: -```yaml -version: "0.0.13" # ASTRA spec version — keep what the scaffold wrote -name: "R² with and without feature standardization" -description: "Linear regression on the diabetes dataset, sweeping the standardization choice." -container: Containerfile +```bash +python3 - <<'EOF' +import random +random.seed(0) +rows = ["x,y"] +for _ in range(200): + x = random.uniform(0, 10) + y = 2.5 * x + 1.0 + random.gauss(0, 1.5) + if random.random() < 0.04: + y += random.gauss(0, 15) + rows.append(f"{x:.6f},{y:.6f}") +open("data/points.csv", "w").write("\n".join(rows) + "\n") +EOF +``` -inputs: [] # the diabetes dataset ships with scikit-learn +`data/` is where declared inputs live. When you commit, the +`.gitattributes` policy routes the file's bytes into git-annex +automatically — the file stays an ordinary readable, writable file in +your tree, and the repository stays light. -decisions: - standardize: - label: "Feature standardization" - rationale: "Standardizing changes coefficient scales and can shift R² for ridge-like models." - default: standardized - options: - standardized: { label: "StandardScaler before fit" } - raw: { label: "No preprocessing" } +## 3. Write the spec + +Open `astra.yaml` and replace the boilerplate with our analysis: + +```yaml +version: "0.0.13" # ASTRA schema version — keep what the scaffold wrote +name: "line_fit" +description: | + Fit a straight line to a small synthetic dataset and sweep one + methodological decision: whether points far from an initial fit are + kept or clipped before the final fit. + +inputs: + - id: points + type: data + source: data/points.csv + description: "200 synthetic (x, y) points, a few of them far off the line" outputs: - - id: r2 + - id: fit type: metric - description: "Coefficient of determination on the test split." - decisions: [standardize] + description: "Slope and intercept of the least-squares line" + inputs: [points] + decisions: [outliers] recipe: - command: python src/fit.py --standardize {decisions.standardize} --output {output} + command: python src/fit.py --points {inputs.points} --outliers {decisions.outliers} --output {output} + - id: fit_plot type: figure - description: "Predicted vs true scatter." - inputs: [r2] + description: "The points and the fitted line" + inputs: [points, fit] recipe: - command: python src/plot.py --r2_dir {inputs.r2} --output {output} + command: python src/plot.py --points {inputs.points} --fit {inputs.fit} --output {output} + +decisions: + outliers: + label: "Outlier handling" + rationale: "A few points sit far off the line; keeping or clipping them shifts the slope." + default: keep + options: + keep: + label: "Keep every point" + clip: + label: "Drop points beyond 3 sigma of an initial fit" ``` A few things to notice: -- Each output declares what it depends on: `r2` depends on the - `standardize` decision, `fit_plot` depends on the sibling output `r2`. +- Each output declares its full dependency contract: `fit` depends on + the `points` input and the `outliers` decision; `fit_plot` depends on + `points` and on the sibling output `fit`. That contract is how `lc` + orders the build — and how it knows what to rebuild when something + changes. - Recipes reference those dependencies through placeholders — - `{decisions.standardize}`, `{inputs.r2}`, `{output}` — which `lc run` - expands at execution time. `{output}` is the output's own results - directory. + `{inputs.points}`, `{decisions.outliers}`, `{output}` — which are + expanded at execution time. `{output}` is the output's own results + directory, `results///`; the engine creates it + before the recipe runs. - The decision's options aren't hardcoded anywhere in code; the scripts will take them as command-line arguments. +`universes/baseline.yaml` was scaffolded against the boilerplate spec, +so point it at our decision instead: + +```yaml +id: baseline +description: "Every point kept — the decision defaults." +decisions: + outliers: keep +``` + +Each universe is one complete selection of decision values; its results +materialize to `results///`. + Check the spec is well-formed: ```bash astra validate astra.yaml ``` -(`astra` is the spec-side CLI; it ships with `astra-tools`, a dependency of -lightcone-cli.) +(`astra` is the spec-side CLI; it ships with `astra-tools`, a dependency +of lightcone-cli.) -## 3. Write the scripts +## 4. Write the scripts Two short scripts, in a `src/` directory (`mkdir src` — the scaffold doesn't create it; where code lives is your choice, the recipes above -just happen to point there). First `src/fit.py` — fits the model, -writes the R² metric and the test-set predictions: +just happen to point there). First `src/fit.py`: ```python import argparse @@ -122,33 +176,29 @@ import json from pathlib import Path import numpy as np -from sklearn.datasets import load_diabetes -from sklearn.linear_model import LinearRegression -from sklearn.model_selection import train_test_split -from sklearn.preprocessing import StandardScaler parser = argparse.ArgumentParser() -parser.add_argument("--standardize", choices=["standardized", "raw"], required=True) +parser.add_argument("--points", required=True) +parser.add_argument("--outliers", choices=["keep", "clip"], required=True) parser.add_argument("--output", required=True) args = parser.parse_args() -X, y = load_diabetes(return_X_y=True) -X_train, X_test, y_train, y_test = train_test_split( - X, y, test_size=0.25, random_state=0 -) -if args.standardize == "standardized": - scaler = StandardScaler().fit(X_train) - X_train, X_test = scaler.transform(X_train), scaler.transform(X_test) - -model = LinearRegression().fit(X_train, y_train) +x, y = np.loadtxt(args.points, delimiter=",", skiprows=1, unpack=True) +if args.outliers == "clip": + slope, intercept = np.polyfit(x, y, 1) + residuals = y - (slope * x + intercept) + mask = np.abs(residuals) < 3 * residuals.std() + x, y = x[mask], y[mask] +slope, intercept = np.polyfit(x, y, 1) out = Path(args.output) -out.mkdir(parents=True, exist_ok=True) -(out / "r2.json").write_text(json.dumps({"r2": model.score(X_test, y_test)})) -np.savez(out / "predictions.npz", y_true=y_test, y_pred=model.predict(X_test)) +(out / "fit.json").write_text( + json.dumps({"slope": slope, "intercept": intercept, "n_used": len(x)}, indent=2) +) ``` -Then `src/plot.py` — reads the upstream output directory, makes the figure: +Then `src/plot.py` — reads the upstream output's directory, makes the +figure: ```python import argparse @@ -162,131 +212,174 @@ import matplotlib.pyplot as plt import numpy as np parser = argparse.ArgumentParser() -parser.add_argument("--r2_dir", required=True) +parser.add_argument("--points", required=True) +parser.add_argument("--fit", required=True) parser.add_argument("--output", required=True) args = parser.parse_args() -r2_dir = Path(args.r2_dir) -r2 = json.loads((r2_dir / "r2.json").read_text())["r2"] -data = np.load(r2_dir / "predictions.npz") +x, y = np.loadtxt(args.points, delimiter=",", skiprows=1, unpack=True) +fit = json.loads((Path(args.fit) / "fit.json").read_text()) fig, ax = plt.subplots() -ax.scatter(data["y_true"], data["y_pred"], s=12) -ax.set_xlabel("true") -ax.set_ylabel("predicted") -ax.set_title(f"R² = {r2:.3f}") - -out = Path(args.output) -out.mkdir(parents=True, exist_ok=True) -fig.savefig(out / "fit_plot.png", dpi=150) +ax.scatter(x, y, s=12) +xs = np.linspace(x.min(), x.max(), 2) +ax.plot(xs, fit["slope"] * xs + fit["intercept"], color="C1") +ax.set_xlabel("x") +ax.set_ylabel("y") +ax.set_title(f"slope = {fit['slope']:.3f}") +fig.savefig(Path(args.output) / "fit_plot.png", dpi=150) ``` -Finally, add the dependencies to `requirements.txt`: +Both scripts import from the project's locked environment, so declare +what they need: -```text -scikit-learn -matplotlib +```bash +uv add numpy matplotlib ``` -The Containerfile installs `requirements.txt` into the project image, so -that's all it takes — `lc run` rebuilds the image automatically when the -dependency files change. (If you're running without a container runtime, -install the same packages into `.venv` instead.) +That one command updates `pyproject.toml`, re-locks `uv.lock`, and syncs +`.venv`. It's the only way packages reach a recipe — recipes run +sandboxed in the locked environment, so a stray `pip install` on your +machine changes nothing they can see. That's a feature: the lock *is* +the record of what your results were computed with. -## 4. Add the second universe +## 5. Materialize -`lc init` scaffolded `universes/baseline.yaml`. Point it at our decision's -default: +Commit, then build: -```yaml -id: baseline -description: "Standardized features (the default)." -decisions: - standardize: standardized +```bash +git add -A && git commit -m "Line-fit analysis" +lc materialize ``` -And add the sweep — `universes/raw.yaml`: +The commit isn't ceremony — every output is committed together with the +code that produced it, so a build refuses to start from a tree with +uncommitted edits (it wouldn't be able to say what code ran). Then: -```yaml -id: raw -description: "No preprocessing before the fit." -decisions: - standardize: raw ``` + ✓ made baseline/fit + ✓ made baseline/fit_plot + ! no [project].license in pyproject.toml, so no RO-Crate publication + view is maintained — declare one to enable it -Each universe is one complete selection of decision values; its results -materialize to `results///`. +✓ Made 2 output(s) in /home/you/line-fit-demo +``` -## 5. Run it +(We'll come back to that license line in step 7.) Each output landed in +`results/baseline//` next to a `.lightcone-manifest.json` — +a manifest recording the recipe, the decisions, the input hashes, the +environment, and the commit — and was committed with a run record that +`datalad rerun` can replay. Look at `git log`: the build wrote history, +not just files. + +Check where things stand any time: ```bash -lc run +lc status ``` -`lc run` materializes every universe it finds under `universes/`. To run just -one, or just one output: +``` + mode: direct + sandbox: landlock (fs: declared, network: allowed) -```bash -lc run --universe baseline -lc run r2 + · current baseline/fit a3f1f11 + · current baseline/fit_plot a3f1f11 + +2 current ``` -Then check where things stand: +The commit column is the answer to "which code made this?" — for every +output, current or not. And `lc materialize` is idempotent: run it again +and it reports the project is up to date without executing anything. -```bash -lc status +## 6. Sweep the decision + +Add the second universe — `universes/robust.yaml`: + +```yaml +id: robust +description: "Points beyond 3 sigma of an initial fit are dropped." +decisions: + outliers: clip ``` -Expected output: +Commit and materialize again: + +```bash +git add -A && git commit -m "Add the robust universe" +lc materialize +``` ``` -Universe baseline - ✓ ok r2 - ✓ ok fit_plot + ✓ made robust/fit + ✓ made robust/fit_plot + · up to date baseline/fit + · up to date baseline/fit_plot -Universe raw - ✓ ok r2 - ✓ ok fit_plot +✓ Made 2 output(s) in /home/you/line-fit-demo ``` -Your comparison is on disk: `results/baseline/r2/r2.json` vs -`results/raw/r2/r2.json`, with a figure next to each. +Only the new universe's outputs ran — `baseline` was already exactly +what the spec asks for, so it wasn't touched. Your comparison is on +disk: with this guide's synthetic dataset, clipping drops 4 points and +moves the slope from 2.414 to 2.450 — visibly closer to the true 2.5 +the data was generated with. -If a recipe fails, `lc run` surfaces the error; fix the script or the spec -and rerun — only the affected outputs re-execute. Commit as you go so your -`git log` is a clean record of the build. +If a recipe fails, `lc materialize` reports which output failed and why, +and leaves the tree as clean as it found it; fix the script or the spec, +commit, and rerun — only the affected outputs re-execute. -## 6. Verify integrity +## 7. Publish + +RO-Crate requires a license, so declaring one is how you tell `lc` the +project is meant for the outside world. Add one line under `[project]` +in `pyproject.toml`: + +```toml +license = "CC-BY-4.0" +``` + +then commit and materialize once more: ```bash -lc verify +git add -A && git commit -m "Declare a license" +lc materialize ``` -This recomputes data hashes for every output and walks the input chain back to -declare whether anything has been tampered with since materialization. Useful -pre-publication, when archiving a project, or any time you want a stronger -guarantee than `lc status`. +Nothing is rebuilt — but `ro-crate-metadata.json` appears at the project +root and is committed automatically. From here on, every materialize +keeps it in line with the repository: the project *is* the crate, and +depositing it is just `git archive` (or `datalad export-archive`) on a +repository you already have. ## What just happened -- `astra.yaml` was the only place your analysis was *described* — inputs, - outputs, the decision, and the recipes all live there. +- `astra.yaml` was the only place your analysis was *described* — + inputs, outputs, the decision, and the recipes all live there. - The scripts take decision values as plain command-line arguments, so nothing methodological is hardcoded. -- `lc run` generated `.lightcone/Snakefile` from your spec, dispatched each - rule through Snakemake, and wrote a per-output sidecar manifest recording the - recipe, container image, decisions, input hashes, and output hash. -- `lc status` and `lc verify` rely on those manifests — they don't re-execute - anything; they just check. - -If your laptop dies tomorrow and you `git clone` the repo on a fresh machine -and run `lc run`, you'll get bit-identical results. +- `lc materialize` ran each recipe in the project's locked environment, + sandboxed — free to write its own output directory and nothing else — + and committed every output with a manifest and a re-runnable run + record. +- `lc status` and `lc materialize --check` read those manifests — they + don't re-execute anything; they just classify. An output is remade + when the spec defines it differently than it was made, or when its + declared inputs changed; an output whose *environment* has since + moved is reported as `behind` and deliberately left alone — the + manifest records exactly which environment and commit produced it. + +Clone this repository on a fresh machine, run `lc init` (it rebuilds +the two pieces of local state git doesn't carry — the `.venv` and the +annex), then `lc materialize`: it reports up to date without fetching a +single data byte, because the provenance travels in git. The bytes +themselves follow with `git annex get` whenever you actually need them. ## Where to next - [Running on a Cluster](cluster.md) — take the same project to SLURM. - [Troubleshooting](troubleshooting.md) — when something goes sideways. -- [Glossary](glossary.md) — terms like universe, decision, and manifest in - plain language. +- [Glossary](glossary.md) — terms like universe, decision, and manifest + in plain language. - The [ASTRA docs](https://astra-spec.org/latest/) — the full spec: sub-analyses, prior insights, findings, and evidence. diff --git a/docs/user/index.md b/docs/user/index.md index 7170e4a9..8680753d 100644 --- a/docs/user/index.md +++ b/docs/user/index.md @@ -4,27 +4,28 @@ a reproducible analysis. You describe what you're trying to learn as a precise specification — an `astra.yaml` file following the [**ASTRA**][astra] schema — and the `lc` command line keeps the -resulting code, decisions, and outputs in sync. +resulting code, environments, decisions, and outputs in sync. ASTRA specs are plain YAML, designed to be easy for both humans and AI assistants to write. However the spec gets written, **you stay in charge of the scientific choices** — every methodological decision is declared -in the open, and `lc` records exactly what produced every result. +in the open, and `lc` records exactly what produced every result: the +recipe, the decisions, the input data, the environment, and the commit. ## What this guide covers - [Install](install.md) — get the `lc` command line running on your machine or on a cluster. - [Getting Started](getting-started.md) — create your first project, - run it end-to-end, and understand what each piece does. + build it end-to-end, and understand what each piece does. - [Running on a Cluster](cluster.md) — taking your analysis to a SLURM - HPC system, including Perlmutter-specific notes. + HPC system. - [Troubleshooting](troubleshooting.md) — common issues and how to unstick them. - [Glossary](glossary.md) — the terms that show up everywhere (universe, decision, manifest, …) explained in plain language. -## What you'll do, in three lines +## What you'll do, in a handful of lines !!! tip "Quick start" @@ -32,27 +33,34 @@ in the open, and `lc` records exactly what produced every result. ```bash uv tool install lightcone-cli lc init my-analysis && cd my-analysis - # describe your analysis in astra.yaml, then: - lc run + # describe your analysis in astra.yaml, write your scripts, + # declare what they import (uv add numpy ...), then: + git add -A && git commit -m "First analysis" + lc materialize ``` === "pip" ```bash pip install lightcone-cli lc init my-analysis && cd my-analysis - # describe your analysis in astra.yaml, then: - lc run + # describe your analysis in astra.yaml, write your scripts, + # declare what they import (uv add numpy ...), then: + git add -A && git commit -m "First analysis" + lc materialize ``` -That's the shortest possible path. The rest of the guide is the unhurried version. +That's the shortest possible path. The rest of the guide is the +unhurried version — and the commit is not ceremony: every output is +committed together with the code that produced it, which is why a build +starts from a clean tree. ## What lightcone-cli is *not* - **A statistics package.** It runs your code; it doesn't compute things itself. -- **A workflow language.** Recipes in `astra.yaml` are short shell or - Python commands, not a DSL. There's no learning curve beyond what's - in [Getting Started](getting-started.md). +- **A workflow language.** Recipes in `astra.yaml` are short shell + commands, not a DSL. There's no learning curve beyond what's in + [Getting Started](getting-started.md). - **An IDE.** `lc` is a command-line tool; write `astra.yaml` and your analysis code with whatever editor or tooling you prefer. diff --git a/docs/user/install.md b/docs/user/install.md index 583dda3c..48f13ed0 100644 --- a/docs/user/install.md +++ b/docs/user/install.md @@ -1,78 +1,37 @@ # Install -To get started on a lightcone project, you need two things on your machine: Python 3.11+ and the lightcone command line tool `lc`. -A container runtime is optional but recommended. +To work on a lightcone project you need two things on your machine: +[uv](https://docs.astral.sh/uv/) and git. Everything else — Python +itself included — is installed by uv or ships with `lc`. -## 1. Python +!!! note "Supported platforms" + Linux (glibc 2.34+, x86_64 or aarch64) and macOS (14+ on Apple + silicon, 15+ on Intel). On Windows, use WSL. -If you don't already have a recent Python +## 1. uv and git -=== "macOS" +`lc` uses uv as its only environment substrate — projects are +`pyproject.toml` + `uv.lock`, and uv manages the Python interpreters +too, so there is no separate Python install step. + +=== "macOS / Linux" ```bash - brew install python@3.12 + curl -LsSf https://astral.sh/uv/install.sh | sh ``` -=== "Linux" - Your package manager (`apt install python3.12`, etc.) or - [pyenv](https://github.com/pyenv/pyenv) - -=== "Windows" - [python.org](https://www.python.org/downloads/) or WSL + git is preinstalled on macOS; on Linux use your package manager + (`apt install git`, `dnf install git`, …). === "NERSC Perlmutter" - NERSC doesn't ship `uv`, but it installs into your home dir with a - single curl: + NERSC doesn't ship `uv`, but it installs into your home directory + with a single curl: ```bash curl -LsSf https://astral.sh/uv/install.sh | sh - uv python install 3.12 ``` - Both `uv` and an isolated Python 3.12 land under `~/.local/`. - Make sure `~/.local/bin` is on your `PATH`. - - ??? note "Alternative: NERSC's `python` module" - `module load python` gives you a ready-to-use distribution with - `conda`, `pip`, and many scientific packages already installed: - - ```bash - module load python # NERSC Python (3.11+) - ``` - - Convenient, but the module is shared and read-only. For custom - packages, build a conda env on top: - - ```bash - conda create -n your-env-name python=3.11 -y - conda activate your-env-name - ``` - - This is NERSC's [recommended path for `pip install`](https://docs.nersc.gov/development/languages/python/nersc-python/) - when you need custom packages. - - !!! warning "Storage: 40 GB home quota" - Conda envs land under `~/.conda/envs/` by default. The - Perlmutter home quota is **40 GB**, which gets eaten quickly. - NERSC recommends `/global/common/software//` for - larger envs. If you want them on `$SCRATCH` (note: 12-week - purge), move and symlink: - - ```bash - conda deactivate - mv ~/.conda/envs/your-env-name $SCRATCH/conda-envs/ - ln -s $SCRATCH/conda-envs/your-env-name ~/.conda/envs/your-env-name - ``` - -!!! tip "Recommendation" - We highly recommend the use of [uv](https://docs.astral.sh/uv/) to manage Python installation and virtual environments. - - `uv` can be installed in a single commandline - - curl -LsSf https://astral.sh/uv/install.sh | sh - - and a subsequent version of Python - - uv python install 3.12 + `uv` lands under `~/.local/bin` — make sure it's on your `PATH`. + git is already on the system. ## 2. lightcone-cli @@ -89,74 +48,41 @@ is `lc`. python -m pip install lightcone-cli ``` -=== "NERSC Perlmutter" - With `uv` (recommended — isolates `lc` under `~/.local/share/uv/tools/`): - - ```bash - uv tool install lightcone-cli - ``` - - With pip, the exact command depends on which Python you're using: - - ```bash - # NERSC python module - module load python - python -m pip install --user lightcone-cli # lands in ~/.local/bin/ - - # Conda env - conda activate your-env-name - python -m pip install lightcone-cli - ``` - - `astra-tools` is a transitive dependency — pulled in automatically. - - ??? note "From source (contributors only)" - ```bash - git clone https://github.com/LightconeResearch/lightcone-cli.git - uv pip install -e ./lightcone-cli - ``` - - To also hack on `astra-tools`: - - ```bash - git clone https://github.com/LightconeResearch/ASTRA.git - uv pip install -e ./ASTRA - ``` - Get a confirmation of the proper installation by running - lc --version # → lightcone-cli, version ... + lc --version # → lc, version ... -> **Note** Some people may have already set a personal shell alias `lc='ls --color'`. If that's you, installing lightcone-cli will shadow the alias — make sure to rebind it (e.g. `alias l='ls --color'`). +> **Note** Some people may have already set a personal shell alias +> `lc='ls --color'`. If that's you, installing lightcone-cli will shadow +> the alias — make sure to rebind it (e.g. `alias l='ls --color'`). -## 3. Global configuration +## 3. Tell git who you are -`~/.lightcone/config.yaml` is created automatically the first time you -run any `lc` command. No manual setup step is needed. The file starts -as: +Every output `lc` makes is committed, so git needs an identity before +the first build — `lc materialize` checks up front rather than failing +after your recipes have run: -```yaml -container: - runtime: auto +```bash +git config --global user.name "Ada Lovelace" +git config --global user.email "ada@example.org" ``` -`auto` detects whichever of `podman`, `docker`, or `podman-hpc` is on -your PATH (and skips docker if its daemon isn't running). Feel free to pin the runtime later by editing this file directly. +If you already commit from this machine, you're done. -## 4. (Optional) Docker or Podman +## 4. (Optional) Podman or Docker -If your analysis declares a `container:` (which it usually should — it -makes the result reproducible across machines), you need a container -runtime: +Only *containerized* projects need a container runtime — a project opts +in by declaring `[tool.lightcone.image]` in its `pyproject.toml`, and +until it does, recipes run directly on your machine in the project's +own locked environment. -- Local laptop: install [Podman](https://podman.io/) (rootless, no +- Local machine: install [Podman](https://podman.io/) (rootless, no daemon) or [Docker](https://docs.docker.com/get-docker/). - HPC login node: see [Running on a Cluster](cluster.md). -The `auto` mode picks whichever container runtime you have. If you don't -have either, you can still use `lc` — set `runtime: none` in -`~/.lightcone/config.yaml` and recipes will run on the host without -isolation. +There is nothing to configure: `lc` detects whichever runtime is +available (`podman-hpc`, then `podman`, then `docker` — skipping docker +if its daemon isn't running). ## Sanity check @@ -164,8 +90,7 @@ isolation. lc init --help Both should print help text. If `lc` is shadowed by an `ls` alias, -unset it (`unalias lc`) or use the full path -(`$(which lc) --version`). +unset it (`unalias lc`) or use the full path (`$(which lc) --version`). ## Updating @@ -176,19 +101,12 @@ unset it (`unalias lc`) or use the full path === "pip" ```bash - pip install -U lightcone-cli astra-tools - ``` - -=== "Source" - ```bash - cd path/to/lightcone-cli - git pull - uv pip install -e . # only needed if pyproject.toml changed + pip install -U lightcone-cli ``` - Editable installs auto-follow source edits — switching branches or - pulling new commits is reflected immediately in `lc`. Re-install - only when `pyproject.toml` adds a new dependency. +An upgrade never invalidates your results: the engine's version is +recorded in every output's manifest, but it is not part of any output's +identity, so nothing gets rebuilt just because `lc` moved. ## Uninstalling @@ -202,6 +120,5 @@ unset it (`unalias lc`) or use the full path pip uninstall lightcone-cli ``` -!!! note "Keep your config?" - `~/.lightcone/config.yaml` survives the uninstall. Delete it too - if you want a clean slate. +Your projects are untouched — everything `lc` knows about an analysis +lives in the project's own repository, not in any global state. diff --git a/src/lightcone/cli/commands.py b/src/lightcone/cli/commands.py index 631ab1da..ed58b020 100644 --- a/src/lightcone/cli/commands.py +++ b/src/lightcone/cli/commands.py @@ -124,8 +124,15 @@ def init(directory: Path, check_only: bool, as_json: bool) -> None: def _render_init_output(report: ConvergenceReport, directory: Path, *, dry_run: bool) -> None: """Print a convergence report: the items, then the verdict. + + Engine prose and paths are escaped — they are data, not markup, and a + literal ``[project]`` or a bracketed path must reach the terminal as + written. """ + from rich.markup import escape + mark, style = ("·", "yellow") if dry_run else ("✓", "green") + where = escape(str(directory)) lines: list[str] = [] for items, label, item_mark, item_style in ( @@ -133,21 +140,24 @@ def _render_init_output(report: ConvergenceReport, directory: Path, *, dry_run: (report.repaired, "would repair" if dry_run else "repaired", mark, style), (report.blocked, "blocked", "✗", "red"), ): - lines += [f" [{item_style}]{item_mark}[/{item_style}] {label} {item}" for item in items] - lines += [f" [yellow]![/yellow] {warning}" for warning in report.warnings] + lines += [ + f" [{item_style}]{item_mark}[/{item_style}] {label} {escape(item)}" + for item in items + ] + lines += [f" [yellow]![/yellow] {escape(warning)}" for warning in report.warnings] if report.converged: # A dry run over a converged project finds nothing to do because # there is nothing to do — one line serves both moods. - verdict = f"[green]✓[/green] {directory} is already converged — nothing to do" + verdict = f"[green]✓[/green] {where} is already converged — nothing to do" elif report.blocked: # A write run that left an item blocked did not converge the # project either; only a dry run gets to be neutral about it. - verdict = f"[red]✗[/red] {directory} is not converged" + verdict = f"[red]✗[/red] {where} is not converged" elif dry_run: - verdict = f"[yellow]![/yellow] {directory} is not converged" + verdict = f"[yellow]![/yellow] {where} is not converged" else: - verdict = f"[green]✓[/green] Project converged at {directory}" + verdict = f"[green]✓[/green] Project converged at {where}" if lines: lines.append("") # space the verdict off the list @@ -347,6 +357,8 @@ def status(as_json: bool) -> None: click.echo(json.dumps(report.as_dict(), indent=2)) return + from rich.markup import escape + lines = [f" mode: {report.mode}"] if report.image is not None: tag, state = report.image["tag"], report.image["state"] @@ -368,10 +380,10 @@ def status(as_json: bool) -> None: # for machine consumers of `--json`. lines += [ f" {marks[o.status]} {o.status:<8} {o.output:<{width}} " - f"{o.git_sha[:7] or '—':<7}" + (f" [dim]{o.why}[/dim]" if o.why else "") + f"{o.git_sha[:7] or '—':<7}" + (f" [dim]{escape(o.why)}[/dim]" if o.why else "") for o in report.outputs ] - lines += [f" [yellow]![/yellow] {warning}" for warning in report.warnings] + lines += [f" [yellow]![/yellow] {escape(warning)}" for warning in report.warnings] counts = report.counts if not report.outputs: @@ -391,9 +403,16 @@ def status(as_json: bool) -> None: def _render_materialize_output(report: MaterializeReport, root: Path, *, dry_run: bool) -> None: """Print what ran, or what would. + + Engine prose and paths are escaped — they are data, not markup, and a + literal ``[project]`` or a bracketed path must reach the terminal as + written. Task names need no escape: ASTRA ids cannot hold a bracket. """ + from rich.markup import escape + + where = escape(str(root)) lines = [ - f" [yellow]·[/yellow] would run {name} — {why}" + f" [yellow]·[/yellow] would run {name} — {escape(why)}" for name, why in report.planned.items() ] lines += [f" [green]✓[/green] made {name}" for name in report.made] @@ -401,21 +420,22 @@ def _render_materialize_output(report: MaterializeReport, root: Path, *, dry_run # an output came from, and the only line here that tells you something # you could not have worked out from the exit code. lines += [ - f" [cyan]·[/cyan] behind {name} — {why}" for name, why in report.behind.items() + f" [cyan]·[/cyan] behind {name} — {escape(why)}" + for name, why in report.behind.items() ] lines += [f" [dim]·[/dim] up to date {name}" for name in report.current] lines += [f" [red]✗[/red] failed {name}" for name in report.failed] lines += [f" [red]✗[/red] blocked {name}" for name in report.blocked] - lines += [f" [yellow]![/yellow] {warning}" for warning in report.warnings] + lines += [f" [yellow]![/yellow] {escape(warning)}" for warning in report.warnings] if not report.ok: - verdict = f"[red]✗[/red] {root} did not finish" + verdict = f"[red]✗[/red] {where} did not finish" elif report.up_to_date: - verdict = f"[green]✓[/green] {root} is up to date — nothing to do" + verdict = f"[green]✓[/green] {where} is up to date — nothing to do" elif dry_run: verdict = f"[yellow]![/yellow] {len(report.planned)} output(s) would be made" else: - verdict = f"[green]✓[/green] Made {len(report.made)} output(s) in {root}" + verdict = f"[green]✓[/green] Made {len(report.made)} output(s) in {where}" # On the verdict line as well as in the listing: on a large analysis the # listing scrolls away, and this is the one state that reports something # rather than doing it. diff --git a/tests/test_cli.py b/tests/test_cli.py index eec31b5c..03206bf8 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -431,6 +431,24 @@ def test_a_failure_exits_nonzero( assert "blocked baseline/report" in result.output +def test_engine_prose_is_rendered_as_written( + runner: CliRunner, project: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A warning naming `[project].license` must reach the terminal with + its brackets — rich would otherwise read them as a style tag and + swallow the word, turning the remedy into "no .license".""" + from lightcone.engine.materialize import MaterializeReport + + _stub( + monkeypatch, + materialize=MaterializeReport(warnings=["no [project].license declared"]), + ) + + result = runner.invoke(main, ["materialize"]) + + assert "no [project].license declared" in result.output + + def test_the_json_report_is_machine_readable( runner: CliRunner, project: Path, monkeypatch: pytest.MonkeyPatch ) -> None: