diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 9c80755..62a2930 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,22 +1,6 @@ -# Copilot Instructions for Analysis Template +# Copilot Instructions -## Project context +Canonical repo guidance lives in `AGENTS.md` — analysis tasks, where outputs go, the data +rules, environments and commands. `README.md` is the user-facing overview. -See the project README for details about the project goal, datasets, and structure. - -## Quick reference - -| Task | Command | -|------|---------| -| Run Python | `pixi run python script.py` | -| Run tests | `pixi run test` | -| Add conda package | `pixi add ` | -| Add PyPI package | `pixi add --pypi ` | - -## Project structure -- **Notebooks**: `analysis/[INITIALS]-[YYYY]-[MM]-[DD]_description.ipynb` -- **Data**: `data//{raw,processed,resources,results}/` -- **Paths**: Use `from import FilePaths` (edit `_constants.py` for datasets) -- **Deps**: All in `pixi.toml` (not pyproject.toml) -- pyproject.toml exists mainly for package metadata and testing -- Run `pixi install` after pulling changes that update `pixi.toml` +If this file conflicts with them, they win. diff --git a/.gitignore b/.gitignore index 54dd2f0..402991b 100644 --- a/.gitignore +++ b/.gitignore @@ -150,8 +150,17 @@ dmypy.json *.gmt *.gmx -# Directories to ignore +# Directories to ignore. Unanchored, so they match at any depth: these are the +# untracked half of an analysis task (see AGENTS.md "Analysis tasks"). figures/ +outputs/ +logs/ + +# ...but a task's results/ and reports/ are tracked on purpose: small evidence +# tables and the HTML report are what a reviewer actually reads on GitHub. +!analysis/**/results/*.csv +!analysis/**/results/*.json +!analysis/**/reports/*.html # OS specifics **.DS_Store diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..1c8d036 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,99 @@ +# AGENTS.md — analysis conventions + +This file owns the working conventions. `README.md` is the user-facing overview; anything +documented elsewhere is referenced from here, never restated. + +## Analysis tasks + +A **task** is one agent session, one git worktree, one branch, and however many PRs it takes. +It gets one directory, `analysis//...//`, carrying a version suffix. The name is +chosen once and never changed, because the path is what links artifacts back to the analysis +that produced them. + +Copy `analysis/_task_template/` to start one. + +Humans also work in `analysis/` in notebooks, in the main checkout — the same conventions apply, +and notebooks keep their `[INITIALS]-[YYYY]-[MM]-[DD]_description.ipynb` naming. + +### Where outputs go + +Placement is decided by *sharing*, on either side: anything another task or a human reads or +writes is central (`data//`); everything else is task-local. That call is made at +end-of-session sign-off rather than at write time — you cannot know at write time whether +something will be reused — so until then artifacts sit in `outputs/`. + +Task-local splits by durability, not by kind: + +| dir | tracked | lives in | holds | +| --- | --- | --- | --- | +| `results/` | yes | this checkout | small evidence tables — csv, json | +| `reports/` | yes | this checkout | HTML, figures embedded as base64 so they render on GitHub | +| `figures/` | no | the **main** checkout | pdf, png | +| `outputs/` | no | the **main** checkout | data artifacts, checkpoints | +| `logs/` | no | the **main** checkout | batch job output | + +The tracked two stay in the worktree so they ride the PR; the untracked three are anchored to +the main checkout so they survive the worktree being removed. Both come from +`task_paths(__file__)` — **never write a bare relative output path.** From a worktree that lands +in the worktree, and since worktrees are gitignored, nothing will warn you. + +Batch jobs are the usual way this bites: schedulers resolve a relative log path against the +*submit* directory. Take the path from `task_paths(__file__).logs` and pass it absolutely. How +your scheduler spells that is its own business, not this template's. + +The top-level `figures/` is for **curated** output — figures chosen for a talk or a paper — not +for task output. + +### The README contract + +Every task delivers a report, however small, and a `README.md` naming the date, the inputs it +consumed, the outputs it wrote and where write-back landed. The version suffix appears in the +write-back key names too, so grepping `analysis/**/README.md` gets you from an `obs` column back +to the task that made it. That reverse index is what keeps central storage from being anonymous. + +## Data + +- Datasets live in `data//{raw,processed,resources,results}/`, gitignored. Note that + `data//results/` is *central and untracked* — not the same thing as a task's tracked + `results/`. +- Reach a dataset with `FilePaths.dataset("")`, which gives `.raw`, `.processed`, + `.resources`, `.results` and a `.create()`. Never hardcode a dataset path. `FilePaths.ROOT` is + the main checkout even when called from a worktree, so central data resolves the same either + way. Task outputs come from `task_paths(__file__)`, the same shape with a `.create()`. +- **Accumulate by addition.** Adding new keys to a freshly re-read object is commutative, so + concurrent sessions cannot lose each other's work whatever the write order. Removing something + is not — that means a new dated copy, keeping the old one so old scripts still run. +- **Never write your in-memory object back over a shared one.** By the time an analysis + finishes, its copy is stale shared state. Re-read from disk, apply your named additions, write + a temp file in the same directory and `os.replace` it. +- Which object is current is stated by the human per session and recorded in the task README — + never hardcoded in a helper or a config, which is how a config ends up pointing at an object + retired months ago. +- Writing back to a shared object needs explicit sign-off on that specific diff. That gate is + also what serializes concurrent sessions, which is why no file lock is needed. + +## Environments + +Dependencies live in `pixi.toml`. **Reusability decides where a package goes, not just whether it +resolves:** one likely to be carried forward is worth making work in the root environment even +when it resists; a stack of packages for a benchmark where at most the winner survives belongs in +an isolated task-local `pixi.toml`, promoted later if it earns it. Only the manifest and lock +persist — the environment is derived, dies with the worktree, and rebuilds with +`pixi install --frozen`. + +Run `pixi install` **in the main checkout only**. If the root manifest declares the package as an +editable `path = "."` dependency, installing from a worktree rebases the package — and every +resolved data path with it — into the worktree. + +| Task | Command | +| --- | --- | +| Run Python | `pixi run python script.py` | +| Run tests | `pixi run test` | +| Add conda package | `pixi add ` | +| Add PyPI package | `pixi add --pypi ` | + +## Sessions + +One task, one session, one worktree. Remove the worktree when you end the session, and push +first — a worktree with unpushed commits is the only unrecoverable state. Don't tear one down +while batch jobs are still queued: they reference scripts by path inside it. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..5ddc5bb --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,3 @@ +# Analysis Agent Entry Point + +@AGENTS.md diff --git a/analysis/ML-2026-01-27_demo_scRNA_workflow.ipynb b/analysis/ML-2026-01-27_demo_scRNA_workflow.ipynb index e93671b..cccbafd 100644 --- a/analysis/ML-2026-01-27_demo_scRNA_workflow.ipynb +++ b/analysis/ML-2026-01-27_demo_scRNA_workflow.ipynb @@ -95,7 +95,7 @@ "outputs": [], "source": [ "sc.settings.verbosity = 2\n", - "sc.settings.datasetdir = FilePaths.EXAMPLE_DATASET / \"raw\"\n", + "sc.settings.datasetdir = FilePaths.dataset(\"example_dataset\").raw\n", "sc.settings.set_figure_params(dpi=100, frameon=False)\n", "sc.settings.figdir = FilePaths.FIGURES / \"example_dataset\"\n", "\n", @@ -498,7 +498,7 @@ "metadata": {}, "outputs": [], "source": [ - "output_path = FilePaths.EXAMPLE_DATASET / \"processed\" / \"pbmc3k_processed.h5ad\"\n", + "output_path = FilePaths.dataset(\"example_dataset\").processed / \"pbmc3k_processed.h5ad\"\n", "adata.write(output_path)\n", "print(f\"Saved to: {output_path}\")" ] diff --git a/analysis/_task_template/README.md b/analysis/_task_template/README.md new file mode 100644 index 0000000..569ca7f --- /dev/null +++ b/analysis/_task_template/README.md @@ -0,0 +1,39 @@ +# + +**Date:** + +One or two sentences on what this task set out to answer. + +## Inputs + +Name the actual files, not concepts — this is what makes the task re-runnable. + +- Working object: `data//processed/.h5ad` +- Other artifacts consumed: `analysis//outputs/`, `data//results/` + +## Outputs + +| what | where | +| --- | --- | +| evidence tables | `results/` (tracked) | +| report | `reports/` (tracked) | +| figures | `figures/` (gitignored, in the main checkout) | +| data artifacts | `outputs/` (gitignored, in the main checkout) | + +## Write-back + +What went into the working object, and under which keys. The version identifier in this +directory's name must appear in the key names, so an `obs` column can be traced back here +by grepping `analysis/**/README.md`. + +- `obs["_"]` — one line on what it holds +- `obsm["X__"]` — likewise + +Artifacts too large to embed stay in `outputs/` and are listed above instead. Anything +promoted to central storage (`data//...`) at sign-off is recorded here with its +final path. + +## Notes + +Decisions a reader would otherwise have to reverse-engineer: what was tried and rejected, +which parameters are load-bearing, what is still provisional. diff --git a/analysis/_task_template/_common.py b/analysis/_task_template/_common.py new file mode 100644 index 0000000..7721e78 --- /dev/null +++ b/analysis/_task_template/_common.py @@ -0,0 +1,22 @@ +"""Shared paths for this task. Copy with the template; edit in place. + +Every output path in the task comes from here, so nothing is a bare relative path. +Nothing is created at import: call ``PATHS.create()`` in the writer, so a dry run stays dry. +""" + +from myanalysis import task_paths + +PATHS = task_paths(__file__) + +#: Tracked: small evidence tables and the report. Ride the pull request. +RESULTS = PATHS.results +REPORTS = PATHS.reports + +#: Gitignored, anchored to the main checkout so they survive worktree teardown. +FIGURES = PATHS.figures +OUTPUTS = PATHS.outputs +LOGS = PATHS.logs + +#: The object this task reads. Supplied per session and recorded in README.md — never +#: hardcoded, because which object is current changes over time. +WORKING_OBJECT = None diff --git a/src/myanalysis/__init__.py b/src/myanalysis/__init__.py index a336f31..46d15d7 100644 --- a/src/myanalysis/__init__.py +++ b/src/myanalysis/__init__.py @@ -1,7 +1,7 @@ from importlib.metadata import version -from ._constants import DatasetPaths, FilePaths +from ._constants import DatasetPaths, FilePaths, TaskPaths, main_checkout, task_paths from .plotting import qc_violin -__all__ = ["DatasetPaths", "FilePaths", "qc_violin"] +__all__ = ["DatasetPaths", "FilePaths", "TaskPaths", "main_checkout", "qc_violin", "task_paths"] __version__ = version("myanalysis") diff --git a/src/myanalysis/_constants.py b/src/myanalysis/_constants.py index ac69cd6..b06125c 100644 --- a/src/myanalysis/_constants.py +++ b/src/myanalysis/_constants.py @@ -1,77 +1,135 @@ -"""Project-wide path constants for notebooks and scripts.""" +"""Project-wide paths for notebooks, scripts and analysis tasks.""" from __future__ import annotations -from dataclasses import dataclass +import subprocess +from dataclasses import dataclass, fields +from functools import lru_cache from pathlib import Path +from typing import Self -# Files that mark the repository root, searched for upward from this module. -_ROOT_MARKERS = ("pixi.toml", ".git") +#: Standard subfolders of ``data//``. +DATASET_DIRS = ("raw", "processed", "resources", "results") +#: Task subdirectories git tracks: small, reviewable, they ride the pull request. +TRACKED_TASK_DIRS = ("results", "reports") -def _find_root(start: Path) -> Path: - """Locate the repo root by walking upward until a marker file is found. +#: Task subdirectories git ignores: heavy or noisy, anchored to the main checkout. +UNTRACKED_TASK_DIRS = ("figures", "outputs", "logs") - Falls back to the fixed ``src//`` layout (three levels up) when no - marker is present, e.g. for a non-editable installed copy. +#: Directory names that sit *inside* a task rather than being one. +_RESERVED_TASK_SUBDIRS = frozenset({*TRACKED_TASK_DIRS, *UNTRACKED_TASK_DIRS, "scripts", "slurm", "notebooks", "docs"}) + + +@lru_cache(maxsize=1) +def main_checkout() -> Path: + """The main checkout, resolved from git so a worktree still points at it. + + Falls back to walking up for ``pixi.toml``/``.git`` outside a repository. """ - for parent in (start, *start.parents): - if any((parent / marker).exists() for marker in _ROOT_MARKERS): - return parent - return start.parents[2] + try: + git_dir = subprocess.run( + ["git", "rev-parse", "--path-format=absolute", "--git-common-dir"], + cwd=Path(__file__).resolve().parent, + capture_output=True, + text=True, + check=True, + ).stdout.strip() + return Path(git_dir).parent + except (subprocess.CalledProcessError, OSError): + here = Path(__file__).resolve() + for parent in (here, *here.parents): + if (parent / "pixi.toml").exists() or (parent / ".git").exists(): + return parent + return here.parents[2] @dataclass(frozen=True) -class DatasetPaths: - """Standard subfolders for a single dataset (``data//``).""" +class _Dirs: + """A named set of directories.""" + + def create(self) -> Self: + """Create them all, idempotently. Call from the writer, never at import.""" + for field in fields(self): + getattr(self, field.name).mkdir(parents=True, exist_ok=True) + return self + + +@dataclass(frozen=True) +class DatasetPaths(_Dirs): + """``data//`` and its standard subfolders.""" root: Path + raw: Path + processed: Path + resources: Path + results: Path - @property - def raw(self) -> Path: - """Original, unmodified input data.""" - return self.root / "raw" - - @property - def processed(self) -> Path: - """Preprocessed / intermediate data.""" - return self.root / "processed" - - @property - def resources(self) -> Path: - """Reference data, gene sets, annotations.""" - return self.root / "resources" - - @property - def results(self) -> Path: - """Analysis outputs (tables, exported objects).""" - return self.root / "results" - - def create(self) -> DatasetPaths: - """Create all standard subfolders (idempotent). Returns ``self``.""" - for path in (self.raw, self.processed, self.resources, self.results): - path.mkdir(parents=True, exist_ok=True) - return self +@dataclass(frozen=True) +class TaskPaths(_Dirs): + """One task's directories, split by durability rather than by kind. -class FilePaths: - """Project-wide paths for notebooks and scripts.""" + ``results``/``reports`` stay in the calling checkout so they ride the pull request; + ``figures``/``outputs``/``logs`` are anchored to the main checkout so they survive the + worktree being removed. + """ + + task: Path + results: Path + reports: Path + figures: Path + outputs: Path + logs: Path - ROOT = _find_root(Path(__file__).resolve()) +class FilePaths: + """Project-wide paths. Add datasets here; never hardcode one.""" + + ROOT = main_checkout() DATA = ROOT / "data" FIGURES = ROOT / "figures" - # The bundled example dataset; customize / add your own via `dataset()`. - EXAMPLE_DATASET = DATA / "example_dataset" - @classmethod def dataset(cls, name: str) -> DatasetPaths: - """Return the standard raw/processed/resources/results paths for a dataset. + """Standard paths for ``data//``. Examples -------- - >>> paths = FilePaths.dataset("pbmc3k").create() - >>> paths.processed / "adata.h5ad" # doctest: +SKIP + >>> FilePaths.dataset("pbmc3k").create().processed # doctest: +SKIP """ - return DatasetPaths(cls.DATA / name) + root = cls.DATA / name + return DatasetPaths(root=root, **{d: root / d for d in DATASET_DIRS}) + + +def task_paths(file: str | Path) -> TaskPaths: + """Output directories for the task ``file`` belongs to. Pass ``__file__``. + + The task is the nearest ancestor under ``analysis/`` that is not a known task + subdirectory, so ``/_common.py`` and ``/scripts/step.py`` both resolve + to ````. + + Examples + -------- + >>> paths = task_paths(__file__).create() # doctest: +SKIP + >>> paths.results / "markers.csv" # tracked # doctest: +SKIP + >>> paths.outputs / "embedding.h5ad" # gitignored, in the main checkout # doctest: +SKIP + """ + path = Path(file).resolve() + parts = path.parts + if "analysis" not in parts: + raise ValueError(f"{path} is not under an 'analysis/' directory") + checkout = Path(*parts[: parts.index("analysis")]) + + task = path.parent + while task.name in _RESERVED_TASK_SUBDIRS: + task = task.parent + if task in (checkout / "analysis", checkout): + raise ValueError(f"{path} is not inside a task directory under 'analysis/'") + + main_task = main_checkout() / "analysis" / task.relative_to(checkout / "analysis") + return TaskPaths( + task=task, + **{d: task / d for d in TRACKED_TASK_DIRS}, + **{d: main_task / d for d in UNTRACKED_TASK_DIRS}, + )