Analysis-task conventions: task_paths, task template, AGENTS.md

.github/copilot-instructions.md

@@ -1,22 +1,6 @@
-# Copilot Instructions for Analysis Template
+# Copilot Instructions
 
-## Project context
+Canonical repo guidance lives in `AGENTS.md` — analysis tasks, where outputs go, the data
+rules, environments and commands. `README.md` is the user-facing overview.
 
-See the project README for details about the project goal, datasets, and structure.
-
-## Quick reference
-
-| Task | Command |
-|------|---------|
-| Run Python | `pixi run python script.py` |
-| Run tests | `pixi run test` |
-| Add conda package | `pixi add <package>` |
-| Add PyPI package | `pixi add --pypi <package>` |
-
-## Project structure
-- **Notebooks**: `analysis/[INITIALS]-[YYYY]-[MM]-[DD]_description.ipynb`
-- **Data**: `data/<dataset>/{raw,processed,resources,results}/`
-- **Paths**: Use `from <package> import FilePaths` (edit `_constants.py` for datasets)
-- **Deps**: All in `pixi.toml` (not pyproject.toml)
-- pyproject.toml exists mainly for package metadata and testing
-- Run `pixi install` after pulling changes that update `pixi.toml`
+If this file conflicts with them, they win.

.gitignore

@@ -150,8 +150,17 @@ dmypy.json
 *.gmt
 *.gmx
 
-# Directories to ignore
+# Directories to ignore. Unanchored, so they match at any depth: these are the
+# untracked half of an analysis task (see AGENTS.md "Analysis tasks").
 figures/
+outputs/
+logs/
+
+# ...but a task's results/ and reports/ are tracked on purpose: small evidence
+# tables and the HTML report are what a reviewer actually reads on GitHub.
+!analysis/**/results/*.csv
+!analysis/**/results/*.json
+!analysis/**/reports/*.html
 
 # OS specifics
 **.DS_Store

AGENTS.md

@@ -0,0 +1,99 @@
+# AGENTS.md — analysis conventions
+
+This file owns the working conventions. `README.md` is the user-facing overview; anything
+documented elsewhere is referenced from here, never restated.
+
+## Analysis tasks
+
+A **task** is one agent session, one git worktree, one branch, and however many PRs it takes.
+It gets one directory, `analysis/<topic>/.../<name_vN>/`, carrying a version suffix. The name is
+chosen once and never changed, because the path is what links artifacts back to the analysis
+that produced them.
+
+Copy `analysis/_task_template/` to start one.
+
+Humans also work in `analysis/` in notebooks, in the main checkout — the same conventions apply,
+and notebooks keep their `[INITIALS]-[YYYY]-[MM]-[DD]_description.ipynb` naming.
+
+### Where outputs go
+
+Placement is decided by *sharing*, on either side: anything another task or a human reads or
+writes is central (`data/<dataset>/`); everything else is task-local. That call is made at
+end-of-session sign-off rather than at write time — you cannot know at write time whether
+something will be reused — so until then artifacts sit in `outputs/`.
+
+Task-local splits by durability, not by kind:
+
+| dir | tracked | lives in | holds |
+| --- | --- | --- | --- |
+| `results/` | yes | this checkout | small evidence tables — csv, json |
+| `reports/` | yes | this checkout | HTML, figures embedded as base64 so they render on GitHub |
+| `figures/` | no | the **main** checkout | pdf, png |
+| `outputs/` | no | the **main** checkout | data artifacts, checkpoints |
+| `logs/` | no | the **main** checkout | batch job output |
+
+The tracked two stay in the worktree so they ride the PR; the untracked three are anchored to
+the main checkout so they survive the worktree being removed. Both come from
+`task_paths(__file__)` — **never write a bare relative output path.** From a worktree that lands
+in the worktree, and since worktrees are gitignored, nothing will warn you.
+
+Batch jobs are the usual way this bites: schedulers resolve a relative log path against the
+*submit* directory. Take the path from `task_paths(__file__).logs` and pass it absolutely. How
+your scheduler spells that is its own business, not this template's.
+
+The top-level `figures/` is for **curated** output — figures chosen for a talk or a paper — not
+for task output.
+
+### The README contract
+
+Every task delivers a report, however small, and a `README.md` naming the date, the inputs it
+consumed, the outputs it wrote and where write-back landed. The version suffix appears in the
+write-back key names too, so grepping `analysis/**/README.md` gets you from an `obs` column back
+to the task that made it. That reverse index is what keeps central storage from being anonymous.
+
+## Data
+
+- Datasets live in `data/<dataset>/{raw,processed,resources,results}/`, gitignored. Note that
+  `data/<dataset>/results/` is *central and untracked* — not the same thing as a task's tracked
+  `results/`.
+- Reach a dataset with `FilePaths.dataset("<name>")`, which gives `.raw`, `.processed`,
+  `.resources`, `.results` and a `.create()`. Never hardcode a dataset path. `FilePaths.ROOT` is
+  the main checkout even when called from a worktree, so central data resolves the same either
+  way. Task outputs come from `task_paths(__file__)`, the same shape with a `.create()`.
+- **Accumulate by addition.** Adding new keys to a freshly re-read object is commutative, so
+  concurrent sessions cannot lose each other's work whatever the write order. Removing something
+  is not — that means a new dated copy, keeping the old one so old scripts still run.
+- **Never write your in-memory object back over a shared one.** By the time an analysis
+  finishes, its copy is stale shared state. Re-read from disk, apply your named additions, write
+  a temp file in the same directory and `os.replace` it.
+- Which object is current is stated by the human per session and recorded in the task README —
+  never hardcoded in a helper or a config, which is how a config ends up pointing at an object
+  retired months ago.
+- Writing back to a shared object needs explicit sign-off on that specific diff. That gate is
+  also what serializes concurrent sessions, which is why no file lock is needed.
+
+## Environments
+
+Dependencies live in `pixi.toml`. **Reusability decides where a package goes, not just whether it
+resolves:** one likely to be carried forward is worth making work in the root environment even
+when it resists; a stack of packages for a benchmark where at most the winner survives belongs in
+an isolated task-local `pixi.toml`, promoted later if it earns it. Only the manifest and lock
+persist — the environment is derived, dies with the worktree, and rebuilds with
+`pixi install --frozen`.
+
+Run `pixi install` **in the main checkout only**. If the root manifest declares the package as an
+editable `path = "."` dependency, installing from a worktree rebases the package — and every
+resolved data path with it — into the worktree.
+
+| Task | Command |
+| --- | --- |
+| Run Python | `pixi run python script.py` |
+| Run tests | `pixi run test` |
+| Add conda package | `pixi add <package>` |
+| Add PyPI package | `pixi add --pypi <package>` |
+
+## Sessions
+
+One task, one session, one worktree. Remove the worktree when you end the session, and push
+first — a worktree with unpushed commits is the only unrecoverable state. Don't tear one down
+while batch jobs are still queued: they reference scripts by path inside it.

CLAUDE.md

@@ -0,0 +1,3 @@
+# Analysis Agent Entry Point
+
+@AGENTS.md

analysis/ML-2026-01-27_demo_scRNA_workflow.ipynb

@@ -95,7 +95,7 @@
    "outputs": [],
    "source": [
     "sc.settings.verbosity = 2\n",
-    "sc.settings.datasetdir = FilePaths.EXAMPLE_DATASET / \"raw\"\n",
+    "sc.settings.datasetdir = FilePaths.dataset(\"example_dataset\").raw\n",
     "sc.settings.set_figure_params(dpi=100, frameon=False)\n",
     "sc.settings.figdir = FilePaths.FIGURES / \"example_dataset\"\n",
     "\n",
@@ -498,7 +498,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "output_path = FilePaths.EXAMPLE_DATASET / \"processed\" / \"pbmc3k_processed.h5ad\"\n",
+    "output_path = FilePaths.dataset(\"example_dataset\").processed / \"pbmc3k_processed.h5ad\"\n",
     "adata.write(output_path)\n",
     "print(f\"Saved to: {output_path}\")"
    ]

analysis/_task_template/README.md

@@ -0,0 +1,39 @@
+# <task name, matching this directory>
+
+**Date:** <YYYY-MM-DD, when the task started>
+
+One or two sentences on what this task set out to answer.
+
+## Inputs
+
+Name the actual files, not concepts — this is what makes the task re-runnable.
+
+- Working object: `data/<dataset>/processed/<object>.h5ad`
+- Other artifacts consumed: `analysis/<other_task>/outputs/<file>`, `data/<dataset>/results/<file>`
+
+## Outputs
+
+| what | where |
+| --- | --- |
+| evidence tables | `results/` (tracked) |
+| report | `reports/` (tracked) |
+| figures | `figures/` (gitignored, in the main checkout) |
+| data artifacts | `outputs/` (gitignored, in the main checkout) |
+
+## Write-back
+
+What went into the working object, and under which keys. The version identifier in this
+directory's name must appear in the key names, so an `obs` column can be traced back here
+by grepping `analysis/**/README.md`.
+
+- `obs["<name>_<version>"]` — one line on what it holds
+- `obsm["X_<name>_<version>"]` — likewise
+
+Artifacts too large to embed stay in `outputs/` and are listed above instead. Anything
+promoted to central storage (`data/<dataset>/...`) at sign-off is recorded here with its
+final path.
+
+## Notes
+
+Decisions a reader would otherwise have to reverse-engineer: what was tried and rejected,
+which parameters are load-bearing, what is still provisional.

analysis/_task_template/_common.py

@@ -0,0 +1,22 @@
+"""Shared paths for this task. Copy with the template; edit in place.
+
+Every output path in the task comes from here, so nothing is a bare relative path.
+Nothing is created at import: call ``PATHS.create()`` in the writer, so a dry run stays dry.
+"""
+
+from myanalysis import task_paths
+
+PATHS = task_paths(__file__)
+
+#: Tracked: small evidence tables and the report. Ride the pull request.
+RESULTS = PATHS.results
+REPORTS = PATHS.reports
+
+#: Gitignored, anchored to the main checkout so they survive worktree teardown.
+FIGURES = PATHS.figures
+OUTPUTS = PATHS.outputs
+LOGS = PATHS.logs
+
+#: The object this task reads. Supplied per session and recorded in README.md — never
+#: hardcoded, because which object is current changes over time.
+WORKING_OBJECT = None

src/myanalysis/__init__.py

@@ -1,7 +1,7 @@
 from importlib.metadata import version
 
-from ._constants import DatasetPaths, FilePaths
+from ._constants import DatasetPaths, FilePaths, TaskPaths, main_checkout, task_paths
 from .plotting import qc_violin
 
-__all__ = ["DatasetPaths", "FilePaths", "qc_violin"]
+__all__ = ["DatasetPaths", "FilePaths", "TaskPaths", "main_checkout", "qc_violin", "task_paths"]
 __version__ = version("myanalysis")