Dev

research/vestibular_schwannoma/README.md

@@ -1,22 +1,17 @@
 # Vestibular Schwannoma Segmentation (CE-T1w)
 
-This project uses fastMONAI for patch-based 3D segmentation of vestibular schwannoma in
-contrast-enhanced T1-weighted MRI. It includes five-fold cross-validation, optional all-data
-training, inference on new cases, and PACS deployment.
+Patch-based 3D segmentation of vestibular schwannoma in contrast-enhanced T1-weighted MRI,
+with five-fold cross-validation, optional all-data training, inference, and PACS deployment.
 
 ## Contents
 
 - `train_5fold.py`: command-line five-fold training and evaluation.
-- `merge_inference_manifests.py`: validate and combine parallel fold subsets for inference.
-- `notebooks/01_five_fold_cross_validation.ipynb`: train and compare UNet, DynUNet, and
-  optional SegMamba models.
-- `notebooks/02_inference_new_cases.ipynb`: run one declared model or an explicit ensemble.
-- `workflow/`: project-local configuration, training model definitions, result aggregation,
-  and inference artifact handling shared by the CLI and notebooks.
+- `merge_parallel_folds.py`: validate and combine parallel fold subsets for inference.
+- `notebooks/`: cross-validation and inference workflows.
+- `workflow/`: shared configuration, training, evaluation, and artifact handling.
 - `data/ml_dataset.csv`: public case index and fixed fold assignments.
 - `deployment/pacs/`: Safetensors bundle builder and ROR/PACS container.
-- `tests/workflow/`: CPU-only workflow contract and orchestration tests.
-- `tests/deployment/`: deployment tests.
+- `tests/`: workflow and deployment tests.
 
 ## Setup and data
 
@@ -26,27 +21,24 @@ Use fastMONAI 0.10.1 or a matching development checkout. From the fastMONAI repo
 pip install -e '.[dev]'
 ```
 
-Medical images are not included. A reproducible workflow for preprocessing the downloaded
-public datasets is in progress. Until it lands, the CSV expects prepared data under
-`../nii_data/`; see [data/README.md](data/README.md).
-UNet and DynUNet use MONAI. The training notebook prints the installation command for the
-optional SegMamba fork when needed.
+Images are not included. The CSV expects prepared data under `../nii_data/`; see
+[data/README.md](data/README.md). UNet and DynUNet use MONAI. The training notebook provides
+setup instructions for optional SegMamba support.
 
-## Run
+## Training
 
-Use the CLI for unattended training. Examples for a quick check, one complete model, and the
-full comparison:
+Use the CLI for unattended training:
 
 ```bash
 python train_5fold.py --models unet --folds 1 --epochs 5 --no-compile
 python train_5fold.py --models unet  # One model, all five folds
 python train_5fold.py --skip-unavailable
 ```
 
-The default requests three models across five folds for 500 epochs; run
-`python train_5fold.py --help` before starting. A launcher processes its requested models
-and folds sequentially. With five GPUs, run one process per fold, assign each process one
-visible GPU, and give it a distinct, previously nonexistent `--results-root`:
+The default is three models, five folds, and 500 epochs. Models and folds run sequentially
+within a launcher. Run `python train_5fold.py --help` for all options.
+
+For one process per GPU, assign one visible GPU and a new `--results-root` to each process:
 
 ```bash
 CUDA_VISIBLE_DEVICES=0 python train_5fold.py --models unet --folds 1 --results-root cv_results/unet_fold_1 &
@@ -57,32 +49,27 @@ CUDA_VISIBLE_DEVICES=4 python train_5fold.py --models unet --folds 5 --results-r
 wait
 ```
 
-Inside each process, its assigned physical GPU is exposed to PyTorch as CUDA device 0.
+Populate `preprocessed/` with one process before launching parallel jobs; concurrent initial
+cache creation is unsupported. Each process sees its assigned GPU as CUDA device 0.
 
-Before starting parallel jobs, populate `preprocessed/` once with a single process;
-concurrent first-time cache creation is not supported. When `MLFLOW_TRACKING_URI` is unset,
-processes launched on the same machine from the same fastMONAI checkout automatically share
-fastMONAI's repository-root SQLite tracking store
-(`sqlite:////absolute/path/to/fastMONAI/mlruns.db`). For multiple machines or a central
-tracking service, configure the same remote URI in every shell:
+All jobs must share one MLflow tracking store. Jobs from the same checkout do this
+automatically when `MLFLOW_TRACKING_URI` is unset. For multiple machines, set the same remote
+URI in every shell:
 
 ```bash
 export MLFLOW_TRACKING_URI=http://mlflow.example:5000
 ```
 
-Do not merge run IDs from independent local MLflow databases: inference must be able to
-resolve every run ID through one tracking URI.
+Never merge run IDs from different tracking stores.
+
+## Merge parallel folds
 
-Each launcher atomically updates `completed_run_ids.json` after every successful fold, so
-completed work remains mergeable if a later fold is interrupted. A subset job intentionally
-does not create `inference_run_ids.json`; its completed registry is also rejected by the
-inference loader. Combine disjoint completed-fold registries into a new inference-only
-results root. The merger requires exactly folds 1-5, verifies that dataset/splits,
-preprocessing, model/loss, and training settings match, and rejects missing or overlapping
-folds, duplicate MLflow run IDs, and an existing output root:
+Each launcher updates `completed_run_ids.json` after every successful fold. A subset run is not
+inference-ready. Merge disjoint registries into a new results root; the merger requires folds
+1-5, a matching training contract, distinct run IDs, no overlaps, and a new output root:
 
 ```bash
-python merge_inference_manifests.py \
+python merge_parallel_folds.py \
   cv_results/unet_fold_1/completed_run_ids.json \
   cv_results/unet_fold_2/completed_run_ids.json \
   cv_results/unet_fold_3/completed_run_ids.json \
@@ -92,42 +79,40 @@ python merge_inference_manifests.py \
   --output-root cv_results/unet_5fold_merged
 ```
 
-To replace only fold 1, train it into a new results root and merge that new
-`completed_run_ids.json` with registries containing folds 2-5; omit the old fold-1
-registry. The replacement is accepted only when its training contract matches, and the
-merged `--output-root` must also be new.
-
-Use `cv_results/unet_5fold_merged/inference_run_ids.json` in notebook 02. For interactive
-inspection and visualizations, start Jupyter from this directory or `notebooks/`:
+Training and merging fail if their output directory already exists; they never overwrite
+weights or manifests. For example, replace an existing `cv_results/unet_fold_1` like this:
 
 ```bash
-jupyter lab notebooks/01_five_fold_cross_validation.ipynb
+python train_5fold.py \
+  --models unet \
+  --folds 1 \
+  --results-root cv_results/unet_fold_1_retrained_20260902
+
+python merge_parallel_folds.py \
+  cv_results/unet_fold_1_retrained_20260902/completed_run_ids.json \
+  cv_results/unet_fold_2/completed_run_ids.json \
+  cv_results/unet_fold_3/completed_run_ids.json \
+  cv_results/unet_fold_4/completed_run_ids.json \
+  cv_results/unet_fold_5/completed_run_ids.json \
+  --model unet \
+  --output-root cv_results/unet_5fold_merged_20260902
 ```
 
-Notebook 01 uses the fixed `fold` column for cross-validation and can optionally train one
-final model on all cases. For all-data fitting, one stable case is duplicated only for fastai's
-validation phase; it remains in training, does not select a best checkpoint, and is not held-out
-evaluation. Notebook 02 reads the preprocessing and output contract embedded
-in each declared Safetensors model. Completed training runs are handed to notebook 02 through
-`cv_results/<RESULTS_RUN>/inference_run_ids.json`; models remain stored in MLflow.
-
-Evaluation and inference preserve all predicted regions without size filtering and use TTA by
-default. Predictions require clinical review.
-
-The notebooks keep the scientific choices visible but delegate reusable project orchestration
-to `workflow/`. Training model configs contain the VS-specific architecture and loss settings, while
-model reconstruction, patch inference, metrics, and artifact formats remain fastMONAI
-responsibilities.
-
-Training writes selected fold checkpoints below
-`<results-root>/<model>/fold_<n>/checkpoints/`, so different folds and models cannot overwrite
-each other. All-data learners are independently scoped below
-`<results-root>/<model>/all_data/`, but their final artifacts are stored in MLflow rather than as
-a local checkpoint. The fold `.pth` files support warm-starting or further fitting with a newly
-initialized optimizer and learning-rate schedule; they are not exact training-resume
-checkpoints. Final and best inference artifacts remain isolated in their MLflow runs. Inference
-and deployment use strict-loaded `.safetensors` artifacts. Generated data, results, tracking
-stores, checkpoints, and model bundles are excluded from Git.
+Contracts must match. The original directory and MLflow run remain unchanged.
+
+## Inference and artifacts
+
+- Notebook 01 uses fixed folds and can train an all-data model. Its duplicated validation case
+  remains in training and is only an internal fastai monitor.
+- Notebook 02 loads declared models from
+  `cv_results/<RESULTS_RUN>/inference_run_ids.json` and enforces their preprocessing and output
+  contracts.
+- Evaluation and inference use TTA by default and preserve every predicted region. Predictions
+  require clinical review.
+- Fold checkpoints live under `<results-root>/<model>/fold_<n>/checkpoints/`. They support
+  warm-starting, not exact resume. MLflow stores the final/best artifacts; inference and
+  deployment strict-load `.safetensors` files.
+- Generated data, results, tracking stores, checkpoints, and bundles are excluded from Git.
 
 For container preparation and execution, see
 [deployment/pacs/README.md](deployment/pacs/README.md).

research/vestibular_schwannoma/data/README.md

@@ -1,14 +1,10 @@
 # Data layout
 
-Medical images and masks are not distributed with this repository.
+Images and masks are not included. Prepare the source datasets in the layout referenced
+by `ml_dataset.csv`; a reproducible preprocessing workflow is still in development.
 
-A reproducible workflow for preprocessing the downloaded public datasets is in progress.
-Until that workflow is available, users must prepare the source datasets themselves in the
-layout referenced by the index.
-
-`ml_dataset.csv` is the current research index used by the training notebook. Paths are
-resolved from the project root; the current index expects the private image tree under
-`../nii_data/`.
+Paths resolve from the `vestibular_schwannoma/` directory. The current index expects data
+under `../nii_data/`.
 
 ## Columns
 
@@ -20,4 +16,5 @@ resolved from the project root; the current index expects the private image tree
 | `fold` | Required fixed cross-validation fold. |
 | `volume_mm3` | Optional tumor volume derived from the reference mask. |
 | `quartile_label` | Optional zero-based volume-quartile label. |
-Only `case_id`, the two paths, and `fold` are required by the current training workflow.
+
+Only `case_id`, both paths, and `fold` are required.

research/vestibular_schwannoma/data/ml_dataset.csv

@@ -175,7 +175,6 @@ vs_gk_54,1592.2794342041016,../nii_data/queen_square_data/vs_gk_54/vs_gk_54_t1_r
 crossmoda2022_etz_85,248.3048658370972,../nii_data/tilburg_data/crossmoda2022_etz_85/crossmoda2022_etz_85_ceT1.nii.gz,../nii_data/tilburg_data/crossmoda2022_etz_85/crossmoda2022_etz_85_Label.nii.gz,5,0
 vs_gk_72,3547.931671142578,../nii_data/queen_square_data/vs_gk_72/vs_gk_72_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_72/vs_gk_72_seg_refT1.nii.gz,2,3
 vs_gk_175,672.7443695068359,../nii_data/queen_square_data/vs_gk_175/vs_gk_175_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_175/vs_gk_175_seg_refT1.nii.gz,3,0
-vs_gk_131,1472.6692199707031,../nii_data/queen_square_data/vs_gk_131/vs_gk_131_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_131/vs_gk_131_seg_refT1.nii.gz,3,2
 vs_gk_65,434.5333099365234,../nii_data/queen_square_data/vs_gk_65/vs_gk_65_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_65/vs_gk_65_seg_refT1.nii.gz,5,0
 vs_gk_137,1936.978912353516,../nii_data/queen_square_data/vs_gk_137/vs_gk_137_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_137/vs_gk_137_seg_refT1.nii.gz,3,2
 vs_gk_200,626.0610580444336,../nii_data/queen_square_data/vs_gk_200/vs_gk_200_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_200/vs_gk_200_seg_refT1.nii.gz,1,0
@@ -292,7 +291,6 @@ vs_gk_186,2341.231155395508,../nii_data/queen_square_data/vs_gk_186/vs_gk_186_t1
 vs_gk_206,4790.717124938965,../nii_data/queen_square_data/vs_gk_206/vs_gk_206_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_206/vs_gk_206_seg_refT1.nii.gz,1,3
 vs_gk_105,907.6749801635742,../nii_data/queen_square_data/vs_gk_105/vs_gk_105_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_105/vs_gk_105_seg_refT1.nii.gz,1,1
 crossmoda2022_etz_103,8508.979797363281,../nii_data/tilburg_data/crossmoda2022_etz_103/crossmoda2022_etz_103_ceT1.nii.gz,../nii_data/tilburg_data/crossmoda2022_etz_103/crossmoda2022_etz_103_Label.nii.gz,5,3
-vs_gk_8,1063.8748168945312,../nii_data/queen_square_data/vs_gk_8/vs_gk_8_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_8/vs_gk_8_seg_refT1.nii.gz,2,1
 vs_gk_30,7943.228530883789,../nii_data/queen_square_data/vs_gk_30/vs_gk_30_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_30/vs_gk_30_seg_refT1.nii.gz,5,3
 crossmoda2022_etz_44,2093.4310913085938,../nii_data/tilburg_data/crossmoda2022_etz_44/crossmoda2022_etz_44_ceT1.nii.gz,../nii_data/tilburg_data/crossmoda2022_etz_44/crossmoda2022_etz_44_Label.nii.gz,4,2
 crossmoda2022_etz_12,802.4482727050781,../nii_data/tilburg_data/crossmoda2022_etz_12/crossmoda2022_etz_12_ceT1.nii.gz,../nii_data/tilburg_data/crossmoda2022_etz_12/crossmoda2022_etz_12_Label.nii.gz,5,1

research/vestibular_schwannoma/deployment/pacs/README.md

@@ -1,17 +1,15 @@
 # Vestibular Schwannoma ROR/PACS Container
 
-This directory builds the CE-T1w vestibular schwannoma segmentation container.
-It can ship `unet`, `dynunet`, or both. One Safetensors member is a single model;
-multiple members form an ensemble and are evaluated sequentially. Eight-flip TTA
-is enabled by default.
+Build the CE-T1w vestibular schwannoma container with `unet`, `dynunet`, or both.
+Multiple Safetensors members form a sequential ensemble. Eight-flip TTA is enabled by default.
 
 See [Research PACS deployment with ROR](../../../RESEARCH_PACS_DEPLOYMENT.md)
 for the general build, qualification, export, and handoff workflow.
 
 ## Prepare model bundles
 
-Use fastMONAI 0.10.1 or the pinned `fastmonai` environment from
-`requirements.yml`. Supply complete MLflow run IDs.
+Use fastMONAI 0.10.1 or the environment pinned in `requirements.yml`. Provide complete
+MLflow run IDs.
 
 One model trained on all data:
 
@@ -35,13 +33,12 @@ python prepare_model_bundle.py \
   --artifact-role best
 ```
 
-Repeat `--run MEMBER=RUN_ID` for other ensemble sizes. For an existing local
-artifact, use `--artifact MEMBER=/path/model.safetensors`. The builder validates
-and strict-loads every member, then writes the ignored
-`model_bundles/<model-type>/` directory.
+Repeat `--run MEMBER=RUN_ID` for other ensemble sizes, or use
+`--artifact MEMBER=/path/model.safetensors` for a local artifact. The builder validates and
+strict-loads each member into the ignored `model_bundles/<model-type>/` directory.
 
-Derived DICOM UIDs use deterministic `2.25` UIDs by default. A site that owns a
-registered prefix reserved for this application can bind it at bundle creation:
+Derived DICOM UIDs use deterministic `2.25` values by default. To use a registered
+application-specific prefix:
 
 ```bash
 python prepare_model_bundle.py ... --dicom-uid-prefix "<registered-prefix>"
@@ -87,13 +84,11 @@ ror trigger -cont "vs-seg:$BUILD_VERSION" -each -keep \
 - `model-type`: `unet` (default) or `dynunet`.
 - `tta`: JSON Boolean, default `true`.
 
-Unknown keys and invalid values fail before inference. A header-only preflight
-then rejects inconsistent Study, Series, SOP, modality, or geometry information.
-Nonstandard source UID syntax and missing optional Frame of Reference metadata
-produce aggregated warnings instead of repeated per-slice warnings.
+Unknown keys and invalid values fail before inference. Header preflight rejects
+inconsistent Study, Series, SOP, modality, or geometry data; nonstandard source UIDs and
+missing optional Frame of Reference metadata produce aggregated warnings.
 
-The runtime writes `mask` and intermediate `vote_map` DICOM series. Fiona's
-`pr2mask` tools then create `fused`, `fused_vote_map`, and `reports`; `vote_map`
-is not published. Probability values are stored as `round(probability x 65535)`
-for the vote-map reader. Existing `mask`, `fused`, `fused_vote_map`, and
-`reports` directories are rejected to avoid overwriting previous results.
+The runtime writes `mask` and intermediate `vote_map` DICOM series. Fiona's `pr2mask` creates
+`fused`, `fused_vote_map`, and `reports`; `vote_map` is not published. Vote-map probabilities
+are `round(probability x 65535)`. Existing `mask`, `fused`, `fused_vote_map`, and `reports`
+directories are rejected to prevent overwrites.

research/vestibular_schwannoma/merge_inference_manifests.py → research/vestibular_schwannoma/merge_parallel_folds.py

@@ -21,8 +21,8 @@
 def _parser() -> argparse.ArgumentParser:
     parser = argparse.ArgumentParser(
         description=(
-            "Validate and merge disjoint fold inference manifests produced by "
-            "independently launched training jobs."
+            "Validate and merge disjoint fold manifests produced by parallel "
+            "training jobs."
         )
     )
     parser.add_argument(

research/vestibular_schwannoma/notebooks/01_five_fold_cross_validation.ipynb

@@ -85,7 +85,7 @@
     "\n",
     "`training_seed` initializes training randomness before each independent run while retaining cuDNN performance optimizations. In an all-data run, every case remains in training and the first case by stable `case_id` order is duplicated only for fastai's validation phase. Its metric is an internal monitor, not held-out evaluation.\n",
     "\n",
-    "Every independently launched job must use a distinct, previously nonexistent `RESULTS_ROOT`. Re-run this configuration cell before starting another sweep. After parallel fold subsets finish (or one is interrupted after completing some folds), use `merge_inference_manifests.py` to validate and combine their `completed_run_ids.json` registries. The merger rejects different dataset, split, preprocessing, model, loss, or training contracts. A partial `completed_run_ids.json` registry cannot be used directly for inference."
+    "Every independently launched job must use a distinct, previously nonexistent `RESULTS_ROOT`. Re-run this configuration cell before starting another sweep. After parallel fold subsets finish (or one is interrupted after completing some folds), use `merge_parallel_folds.py` to validate and combine their `completed_run_ids.json` registries. The merger rejects different dataset, split, preprocessing, model, loss, or training contracts. A partial `completed_run_ids.json` registry cannot be used directly for inference."
    ]
   },
   {