diff --git a/research/vestibular_schwannoma/README.md b/research/vestibular_schwannoma/README.md index 16db028..c21320b 100644 --- a/research/vestibular_schwannoma/README.md +++ b/research/vestibular_schwannoma/README.md @@ -1,22 +1,17 @@ # Vestibular Schwannoma Segmentation (CE-T1w) -This project uses fastMONAI for patch-based 3D segmentation of vestibular schwannoma in -contrast-enhanced T1-weighted MRI. It includes five-fold cross-validation, optional all-data -training, inference on new cases, and PACS deployment. +Patch-based 3D segmentation of vestibular schwannoma in contrast-enhanced T1-weighted MRI, +with five-fold cross-validation, optional all-data training, inference, and PACS deployment. ## Contents - `train_5fold.py`: command-line five-fold training and evaluation. -- `merge_inference_manifests.py`: validate and combine parallel fold subsets for inference. -- `notebooks/01_five_fold_cross_validation.ipynb`: train and compare UNet, DynUNet, and - optional SegMamba models. -- `notebooks/02_inference_new_cases.ipynb`: run one declared model or an explicit ensemble. -- `workflow/`: project-local configuration, training model definitions, result aggregation, - and inference artifact handling shared by the CLI and notebooks. +- `merge_parallel_folds.py`: validate and combine parallel fold subsets for inference. +- `notebooks/`: cross-validation and inference workflows. +- `workflow/`: shared configuration, training, evaluation, and artifact handling. - `data/ml_dataset.csv`: public case index and fixed fold assignments. - `deployment/pacs/`: Safetensors bundle builder and ROR/PACS container. -- `tests/workflow/`: CPU-only workflow contract and orchestration tests. -- `tests/deployment/`: deployment tests. +- `tests/`: workflow and deployment tests. ## Setup and data @@ -26,16 +21,13 @@ Use fastMONAI 0.10.1 or a matching development checkout. From the fastMONAI repo pip install -e '.[dev]' ``` -Medical images are not included. A reproducible workflow for preprocessing the downloaded -public datasets is in progress. Until it lands, the CSV expects prepared data under -`../nii_data/`; see [data/README.md](data/README.md). -UNet and DynUNet use MONAI. The training notebook prints the installation command for the -optional SegMamba fork when needed. +Images are not included. The CSV expects prepared data under `../nii_data/`; see +[data/README.md](data/README.md). UNet and DynUNet use MONAI. The training notebook provides +setup instructions for optional SegMamba support. -## Run +## Training -Use the CLI for unattended training. Examples for a quick check, one complete model, and the -full comparison: +Use the CLI for unattended training: ```bash python train_5fold.py --models unet --folds 1 --epochs 5 --no-compile @@ -43,10 +35,10 @@ python train_5fold.py --models unet # One model, all five folds python train_5fold.py --skip-unavailable ``` -The default requests three models across five folds for 500 epochs; run -`python train_5fold.py --help` before starting. A launcher processes its requested models -and folds sequentially. With five GPUs, run one process per fold, assign each process one -visible GPU, and give it a distinct, previously nonexistent `--results-root`: +The default is three models, five folds, and 500 epochs. Models and folds run sequentially +within a launcher. Run `python train_5fold.py --help` for all options. + +For one process per GPU, assign one visible GPU and a new `--results-root` to each process: ```bash CUDA_VISIBLE_DEVICES=0 python train_5fold.py --models unet --folds 1 --results-root cv_results/unet_fold_1 & @@ -57,32 +49,27 @@ CUDA_VISIBLE_DEVICES=4 python train_5fold.py --models unet --folds 5 --results-r wait ``` -Inside each process, its assigned physical GPU is exposed to PyTorch as CUDA device 0. +Populate `preprocessed/` with one process before launching parallel jobs; concurrent initial +cache creation is unsupported. Each process sees its assigned GPU as CUDA device 0. -Before starting parallel jobs, populate `preprocessed/` once with a single process; -concurrent first-time cache creation is not supported. When `MLFLOW_TRACKING_URI` is unset, -processes launched on the same machine from the same fastMONAI checkout automatically share -fastMONAI's repository-root SQLite tracking store -(`sqlite:////absolute/path/to/fastMONAI/mlruns.db`). For multiple machines or a central -tracking service, configure the same remote URI in every shell: +All jobs must share one MLflow tracking store. Jobs from the same checkout do this +automatically when `MLFLOW_TRACKING_URI` is unset. For multiple machines, set the same remote +URI in every shell: ```bash export MLFLOW_TRACKING_URI=http://mlflow.example:5000 ``` -Do not merge run IDs from independent local MLflow databases: inference must be able to -resolve every run ID through one tracking URI. +Never merge run IDs from different tracking stores. + +## Merge parallel folds -Each launcher atomically updates `completed_run_ids.json` after every successful fold, so -completed work remains mergeable if a later fold is interrupted. A subset job intentionally -does not create `inference_run_ids.json`; its completed registry is also rejected by the -inference loader. Combine disjoint completed-fold registries into a new inference-only -results root. The merger requires exactly folds 1-5, verifies that dataset/splits, -preprocessing, model/loss, and training settings match, and rejects missing or overlapping -folds, duplicate MLflow run IDs, and an existing output root: +Each launcher updates `completed_run_ids.json` after every successful fold. A subset run is not +inference-ready. Merge disjoint registries into a new results root; the merger requires folds +1-5, a matching training contract, distinct run IDs, no overlaps, and a new output root: ```bash -python merge_inference_manifests.py \ +python merge_parallel_folds.py \ cv_results/unet_fold_1/completed_run_ids.json \ cv_results/unet_fold_2/completed_run_ids.json \ cv_results/unet_fold_3/completed_run_ids.json \ @@ -92,42 +79,40 @@ python merge_inference_manifests.py \ --output-root cv_results/unet_5fold_merged ``` -To replace only fold 1, train it into a new results root and merge that new -`completed_run_ids.json` with registries containing folds 2-5; omit the old fold-1 -registry. The replacement is accepted only when its training contract matches, and the -merged `--output-root` must also be new. - -Use `cv_results/unet_5fold_merged/inference_run_ids.json` in notebook 02. For interactive -inspection and visualizations, start Jupyter from this directory or `notebooks/`: +Training and merging fail if their output directory already exists; they never overwrite +weights or manifests. For example, replace an existing `cv_results/unet_fold_1` like this: ```bash -jupyter lab notebooks/01_five_fold_cross_validation.ipynb +python train_5fold.py \ + --models unet \ + --folds 1 \ + --results-root cv_results/unet_fold_1_retrained_20260902 + +python merge_parallel_folds.py \ + cv_results/unet_fold_1_retrained_20260902/completed_run_ids.json \ + cv_results/unet_fold_2/completed_run_ids.json \ + cv_results/unet_fold_3/completed_run_ids.json \ + cv_results/unet_fold_4/completed_run_ids.json \ + cv_results/unet_fold_5/completed_run_ids.json \ + --model unet \ + --output-root cv_results/unet_5fold_merged_20260902 ``` -Notebook 01 uses the fixed `fold` column for cross-validation and can optionally train one -final model on all cases. For all-data fitting, one stable case is duplicated only for fastai's -validation phase; it remains in training, does not select a best checkpoint, and is not held-out -evaluation. Notebook 02 reads the preprocessing and output contract embedded -in each declared Safetensors model. Completed training runs are handed to notebook 02 through -`cv_results//inference_run_ids.json`; models remain stored in MLflow. - -Evaluation and inference preserve all predicted regions without size filtering and use TTA by -default. Predictions require clinical review. - -The notebooks keep the scientific choices visible but delegate reusable project orchestration -to `workflow/`. Training model configs contain the VS-specific architecture and loss settings, while -model reconstruction, patch inference, metrics, and artifact formats remain fastMONAI -responsibilities. - -Training writes selected fold checkpoints below -`//fold_/checkpoints/`, so different folds and models cannot overwrite -each other. All-data learners are independently scoped below -`//all_data/`, but their final artifacts are stored in MLflow rather than as -a local checkpoint. The fold `.pth` files support warm-starting or further fitting with a newly -initialized optimizer and learning-rate schedule; they are not exact training-resume -checkpoints. Final and best inference artifacts remain isolated in their MLflow runs. Inference -and deployment use strict-loaded `.safetensors` artifacts. Generated data, results, tracking -stores, checkpoints, and model bundles are excluded from Git. +Contracts must match. The original directory and MLflow run remain unchanged. + +## Inference and artifacts + +- Notebook 01 uses fixed folds and can train an all-data model. Its duplicated validation case + remains in training and is only an internal fastai monitor. +- Notebook 02 loads declared models from + `cv_results//inference_run_ids.json` and enforces their preprocessing and output + contracts. +- Evaluation and inference use TTA by default and preserve every predicted region. Predictions + require clinical review. +- Fold checkpoints live under `//fold_/checkpoints/`. They support + warm-starting, not exact resume. MLflow stores the final/best artifacts; inference and + deployment strict-load `.safetensors` files. +- Generated data, results, tracking stores, checkpoints, and bundles are excluded from Git. For container preparation and execution, see [deployment/pacs/README.md](deployment/pacs/README.md). diff --git a/research/vestibular_schwannoma/data/README.md b/research/vestibular_schwannoma/data/README.md index 35c64e0..54122cc 100644 --- a/research/vestibular_schwannoma/data/README.md +++ b/research/vestibular_schwannoma/data/README.md @@ -1,14 +1,10 @@ # Data layout -Medical images and masks are not distributed with this repository. +Images and masks are not included. Prepare the source datasets in the layout referenced +by `ml_dataset.csv`; a reproducible preprocessing workflow is still in development. -A reproducible workflow for preprocessing the downloaded public datasets is in progress. -Until that workflow is available, users must prepare the source datasets themselves in the -layout referenced by the index. - -`ml_dataset.csv` is the current research index used by the training notebook. Paths are -resolved from the project root; the current index expects the private image tree under -`../nii_data/`. +Paths resolve from the `vestibular_schwannoma/` directory. The current index expects data +under `../nii_data/`. ## Columns @@ -20,4 +16,5 @@ resolved from the project root; the current index expects the private image tree | `fold` | Required fixed cross-validation fold. | | `volume_mm3` | Optional tumor volume derived from the reference mask. | | `quartile_label` | Optional zero-based volume-quartile label. | -Only `case_id`, the two paths, and `fold` are required by the current training workflow. + +Only `case_id`, both paths, and `fold` are required. diff --git a/research/vestibular_schwannoma/data/ml_dataset.csv b/research/vestibular_schwannoma/data/ml_dataset.csv index 4c60361..9e96d06 100644 --- a/research/vestibular_schwannoma/data/ml_dataset.csv +++ b/research/vestibular_schwannoma/data/ml_dataset.csv @@ -175,7 +175,6 @@ vs_gk_54,1592.2794342041016,../nii_data/queen_square_data/vs_gk_54/vs_gk_54_t1_r crossmoda2022_etz_85,248.3048658370972,../nii_data/tilburg_data/crossmoda2022_etz_85/crossmoda2022_etz_85_ceT1.nii.gz,../nii_data/tilburg_data/crossmoda2022_etz_85/crossmoda2022_etz_85_Label.nii.gz,5,0 vs_gk_72,3547.931671142578,../nii_data/queen_square_data/vs_gk_72/vs_gk_72_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_72/vs_gk_72_seg_refT1.nii.gz,2,3 vs_gk_175,672.7443695068359,../nii_data/queen_square_data/vs_gk_175/vs_gk_175_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_175/vs_gk_175_seg_refT1.nii.gz,3,0 -vs_gk_131,1472.6692199707031,../nii_data/queen_square_data/vs_gk_131/vs_gk_131_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_131/vs_gk_131_seg_refT1.nii.gz,3,2 vs_gk_65,434.5333099365234,../nii_data/queen_square_data/vs_gk_65/vs_gk_65_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_65/vs_gk_65_seg_refT1.nii.gz,5,0 vs_gk_137,1936.978912353516,../nii_data/queen_square_data/vs_gk_137/vs_gk_137_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_137/vs_gk_137_seg_refT1.nii.gz,3,2 vs_gk_200,626.0610580444336,../nii_data/queen_square_data/vs_gk_200/vs_gk_200_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_200/vs_gk_200_seg_refT1.nii.gz,1,0 @@ -292,7 +291,6 @@ vs_gk_186,2341.231155395508,../nii_data/queen_square_data/vs_gk_186/vs_gk_186_t1 vs_gk_206,4790.717124938965,../nii_data/queen_square_data/vs_gk_206/vs_gk_206_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_206/vs_gk_206_seg_refT1.nii.gz,1,3 vs_gk_105,907.6749801635742,../nii_data/queen_square_data/vs_gk_105/vs_gk_105_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_105/vs_gk_105_seg_refT1.nii.gz,1,1 crossmoda2022_etz_103,8508.979797363281,../nii_data/tilburg_data/crossmoda2022_etz_103/crossmoda2022_etz_103_ceT1.nii.gz,../nii_data/tilburg_data/crossmoda2022_etz_103/crossmoda2022_etz_103_Label.nii.gz,5,3 -vs_gk_8,1063.8748168945312,../nii_data/queen_square_data/vs_gk_8/vs_gk_8_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_8/vs_gk_8_seg_refT1.nii.gz,2,1 vs_gk_30,7943.228530883789,../nii_data/queen_square_data/vs_gk_30/vs_gk_30_t1_refT1.nii.gz,../nii_data/queen_square_data/vs_gk_30/vs_gk_30_seg_refT1.nii.gz,5,3 crossmoda2022_etz_44,2093.4310913085938,../nii_data/tilburg_data/crossmoda2022_etz_44/crossmoda2022_etz_44_ceT1.nii.gz,../nii_data/tilburg_data/crossmoda2022_etz_44/crossmoda2022_etz_44_Label.nii.gz,4,2 crossmoda2022_etz_12,802.4482727050781,../nii_data/tilburg_data/crossmoda2022_etz_12/crossmoda2022_etz_12_ceT1.nii.gz,../nii_data/tilburg_data/crossmoda2022_etz_12/crossmoda2022_etz_12_Label.nii.gz,5,1 diff --git a/research/vestibular_schwannoma/deployment/pacs/README.md b/research/vestibular_schwannoma/deployment/pacs/README.md index c7c6035..bb859b5 100644 --- a/research/vestibular_schwannoma/deployment/pacs/README.md +++ b/research/vestibular_schwannoma/deployment/pacs/README.md @@ -1,17 +1,15 @@ # Vestibular Schwannoma ROR/PACS Container -This directory builds the CE-T1w vestibular schwannoma segmentation container. -It can ship `unet`, `dynunet`, or both. One Safetensors member is a single model; -multiple members form an ensemble and are evaluated sequentially. Eight-flip TTA -is enabled by default. +Build the CE-T1w vestibular schwannoma container with `unet`, `dynunet`, or both. +Multiple Safetensors members form a sequential ensemble. Eight-flip TTA is enabled by default. See [Research PACS deployment with ROR](../../../RESEARCH_PACS_DEPLOYMENT.md) for the general build, qualification, export, and handoff workflow. ## Prepare model bundles -Use fastMONAI 0.10.1 or the pinned `fastmonai` environment from -`requirements.yml`. Supply complete MLflow run IDs. +Use fastMONAI 0.10.1 or the environment pinned in `requirements.yml`. Provide complete +MLflow run IDs. One model trained on all data: @@ -35,13 +33,12 @@ python prepare_model_bundle.py \ --artifact-role best ``` -Repeat `--run MEMBER=RUN_ID` for other ensemble sizes. For an existing local -artifact, use `--artifact MEMBER=/path/model.safetensors`. The builder validates -and strict-loads every member, then writes the ignored -`model_bundles//` directory. +Repeat `--run MEMBER=RUN_ID` for other ensemble sizes, or use +`--artifact MEMBER=/path/model.safetensors` for a local artifact. The builder validates and +strict-loads each member into the ignored `model_bundles//` directory. -Derived DICOM UIDs use deterministic `2.25` UIDs by default. A site that owns a -registered prefix reserved for this application can bind it at bundle creation: +Derived DICOM UIDs use deterministic `2.25` values by default. To use a registered +application-specific prefix: ```bash python prepare_model_bundle.py ... --dicom-uid-prefix "" @@ -87,13 +84,11 @@ ror trigger -cont "vs-seg:$BUILD_VERSION" -each -keep \ - `model-type`: `unet` (default) or `dynunet`. - `tta`: JSON Boolean, default `true`. -Unknown keys and invalid values fail before inference. A header-only preflight -then rejects inconsistent Study, Series, SOP, modality, or geometry information. -Nonstandard source UID syntax and missing optional Frame of Reference metadata -produce aggregated warnings instead of repeated per-slice warnings. +Unknown keys and invalid values fail before inference. Header preflight rejects +inconsistent Study, Series, SOP, modality, or geometry data; nonstandard source UIDs and +missing optional Frame of Reference metadata produce aggregated warnings. -The runtime writes `mask` and intermediate `vote_map` DICOM series. Fiona's -`pr2mask` tools then create `fused`, `fused_vote_map`, and `reports`; `vote_map` -is not published. Probability values are stored as `round(probability x 65535)` -for the vote-map reader. Existing `mask`, `fused`, `fused_vote_map`, and -`reports` directories are rejected to avoid overwriting previous results. +The runtime writes `mask` and intermediate `vote_map` DICOM series. Fiona's `pr2mask` creates +`fused`, `fused_vote_map`, and `reports`; `vote_map` is not published. Vote-map probabilities +are `round(probability x 65535)`. Existing `mask`, `fused`, `fused_vote_map`, and `reports` +directories are rejected to prevent overwrites. diff --git a/research/vestibular_schwannoma/merge_inference_manifests.py b/research/vestibular_schwannoma/merge_parallel_folds.py similarity index 92% rename from research/vestibular_schwannoma/merge_inference_manifests.py rename to research/vestibular_schwannoma/merge_parallel_folds.py index 2688b2b..11a533b 100644 --- a/research/vestibular_schwannoma/merge_inference_manifests.py +++ b/research/vestibular_schwannoma/merge_parallel_folds.py @@ -21,8 +21,8 @@ def _parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( description=( - "Validate and merge disjoint fold inference manifests produced by " - "independently launched training jobs." + "Validate and merge disjoint fold manifests produced by parallel " + "training jobs." ) ) parser.add_argument( diff --git a/research/vestibular_schwannoma/notebooks/01_five_fold_cross_validation.ipynb b/research/vestibular_schwannoma/notebooks/01_five_fold_cross_validation.ipynb index d5e43ba..f8adb28 100644 --- a/research/vestibular_schwannoma/notebooks/01_five_fold_cross_validation.ipynb +++ b/research/vestibular_schwannoma/notebooks/01_five_fold_cross_validation.ipynb @@ -85,7 +85,7 @@ "\n", "`training_seed` initializes training randomness before each independent run while retaining cuDNN performance optimizations. In an all-data run, every case remains in training and the first case by stable `case_id` order is duplicated only for fastai's validation phase. Its metric is an internal monitor, not held-out evaluation.\n", "\n", - "Every independently launched job must use a distinct, previously nonexistent `RESULTS_ROOT`. Re-run this configuration cell before starting another sweep. After parallel fold subsets finish (or one is interrupted after completing some folds), use `merge_inference_manifests.py` to validate and combine their `completed_run_ids.json` registries. The merger rejects different dataset, split, preprocessing, model, loss, or training contracts. A partial `completed_run_ids.json` registry cannot be used directly for inference." + "Every independently launched job must use a distinct, previously nonexistent `RESULTS_ROOT`. Re-run this configuration cell before starting another sweep. After parallel fold subsets finish (or one is interrupted after completing some folds), use `merge_parallel_folds.py` to validate and combine their `completed_run_ids.json` registries. The merger rejects different dataset, split, preprocessing, model, loss, or training contracts. A partial `completed_run_ids.json` registry cannot be used directly for inference." ] }, { diff --git a/research/vestibular_schwannoma/tests/workflow/test_merge_inference_manifests.py b/research/vestibular_schwannoma/tests/workflow/test_merge_parallel_folds.py similarity index 86% rename from research/vestibular_schwannoma/tests/workflow/test_merge_inference_manifests.py rename to research/vestibular_schwannoma/tests/workflow/test_merge_parallel_folds.py index 8037cf1..9ae6348 100644 --- a/research/vestibular_schwannoma/tests/workflow/test_merge_inference_manifests.py +++ b/research/vestibular_schwannoma/tests/workflow/test_merge_parallel_folds.py @@ -5,10 +5,10 @@ from pathlib import Path from unittest.mock import patch -from vestibular_schwannoma import merge_inference_manifests +from vestibular_schwannoma import merge_parallel_folds -class MergeInferenceManifestsCliTests(unittest.TestCase): +class MergeParallelFoldsCliTests(unittest.TestCase): def test_main_passes_resolved_inputs_to_fixed_fold_merger(self): with tempfile.TemporaryDirectory() as directory: root = Path(directory) @@ -18,13 +18,13 @@ def test_main_passes_resolved_inputs_to_fixed_fold_merger(self): destination = output_root / "inference_run_ids.json" with ( patch.object( - merge_inference_manifests, + merge_parallel_folds, "merge_fold_run_selections", return_value=destination, ) as merge, redirect_stdout(StringIO()), ): - status = merge_inference_manifests.main( + status = merge_parallel_folds.main( [ str(first), str(second), @@ -47,7 +47,7 @@ def test_custom_fold_set_is_rejected(self): redirect_stderr(StringIO()), self.assertRaises(SystemExit), ): - merge_inference_manifests._parser().parse_args( + merge_parallel_folds._parser().parse_args( [ "fold_1/completed_run_ids.json", "--model",