From d86d968740f59737be5e4ac110f1e4c5b080f825 Mon Sep 17 00:00:00 2001 From: ljwharbers Date: Fri, 2 Oct 2026 14:56:05 +0200 Subject: [PATCH 1/4] Move SigProfiler to the corrected CHM13-T2T payload (SPMG 7894689) The CHM13 archive at chm13_release_2026-08 was replaced by the rebuild that fixes SigProfilerMatrixGenerator#251 (sha256 fe68e840...dc1d), while the 1.3.6-chm13-28a9ce8 image still checks the superseded checksums, so --download_sigprofiler_genome failed verification in SIGPROFILER_INSTALL. Use the 1.3.6-chm13-7894689 image, which carries the new checksums, and fetch the payload from the versioned chm13_release_2026-09 path, so the 2026-08 path can keep serving the original archive for older commits. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + conf/igenomes.config | 2 +- modules/local/sigprofiler/assignment/main.nf | 4 ++-- modules/local/sigprofiler/install/main.nf | 4 ++-- modules/local/sigprofiler/matrixgenerator/main.nf | 4 ++-- 5 files changed, 8 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d398e769..057c2e00 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### `Fixed` +- [#XXX](https://github.com/IntGenomicsLab/lrsomatic/pull/XXX) - `--download_sigprofiler_genome` failed on CHM13 with `CHM13-T2T failed the SigProfilerMatrixGenerator checksum verification`. The published payload had been replaced by the archive rebuilt after [SigProfilerMatrixGenerator#251](https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator/issues/251), which counts transcript-end bases correctly, while the image still carried the old checksums. The SigProfiler modules now use `ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689` (`oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689`), and the CHM13 `sigprofiler_genome_url` points at the versioned `chm13_release_2026-09` payload. **A CHM13 volume installed before this change fails verification and must be reinstalled**. Transcribed-strand matrices (SBS288, SBS384 and the strand-split DBS/ID matrices) change slightly; SBS96 and the COSMIC fits are not expected to change (@ljwharbers). - [#206](https://github.com/IntGenomicsLab/lrsomatic/pull/206) - A remote (http, https or ftp) ClinVar is now downloaded once per run by the new `VEPPLUGIN_CLINVAR` step in `PREPARE_VEP_PLUGINS`, instead of being staged by Nextflow as a foreign file; local and cloud-storage paths are staged as before. `GERMLINE_VEP` and `SOMATIC_VEP` re-checked the foreign file on its host for every sample, and NCBI answered the burst from a multi-sample GRCh38 run with HTTP 503, so `SOMATIC_VEP` failed with `Can't stage file ...clinvar_20260829.vcf.gz`; `-resume` could not recover, since the failed check changed the staging cache key. The download is checked against the new `--vep_clinvar_md5` and `--vep_clinvar_tbi_md5`, set by default to the checksums NCBI (GRCh38, VCF only) and Ensembl (CHM13, VCF and index) publish, so the pinned release cannot change silently. Resuming a run that already finished re-runs `GERMLINE_VEP` and `SOMATIC_VEP` once, since ClinVar now comes from a task rather than the stage cache. The ClinVar sizes in `docs/usage.md` are also corrected, and `docs/output.md` now documents `vep_plugins/` (@AmberVerhasselt). - [#203](https://github.com/IntGenomicsLab/lrsomatic/pull/203) - `CLAIRS` no longer runs with `--haplotagged_tumor_bam_provided_so_skip_intermediate_phasing_and_haplotagging`. Since somatic calling was moved ahead of `PHASING_HAPLOTYPING` (v1.1.0), ClairS has received the untagged minimap2 BAM, so the flag made it skip its own phasing and haplotagging and call every paired sample without haplotype information: the full-alignment model saw no `HP` tags and the haplotype filtering step had nothing to filter on, the same as `--disable_phasing`. ClairS now runs Clair3 on the normal and tumour BAMs and phases the tumour itself. **Paired somatic calls change** (fewer false positives expected), and `CLAIRS` takes longer and uses more work-directory space (@ljwharbers). - [#203](https://github.com/IntGenomicsLab/lrsomatic/pull/203) - `docs/output.md` now lists the ClairS SNV output as `snvs.vcf.gz`, the name the pipeline publishes, instead of `snv.vcf.gz` (@ljwharbers). diff --git a/conf/igenomes.config b/conf/igenomes.config index 430e65a2..5a7971c5 100644 --- a/conf/igenomes.config +++ b/conf/igenomes.config @@ -69,7 +69,7 @@ params.genomes = [ // SigProfilerMatrixGenerator CHM13-T2T payload (SigProfilerSuite/SigProfilerMatrixGenerator#250); // not yet hosted on the AlexandrovLab FTP, so it is fetched from the IntGenomicsLab Globus collection sigprofiler_genome : "CHM13-T2T", - sigprofiler_genome_url : "https://g-608c0c.273595.03c0.data.globus.org/chm13_release_2026-08/CHM13-T2T.tar.gz", + sigprofiler_genome_url : "https://g-608c0c.273595.03c0.data.globus.org/chm13_release_2026-09/CHM13-T2T.tar.gz", gnomad : "${params.igenomes_base}/Homo_sapiens/ClairSTO/CHM13/Annotation/ClairSTO-pon/final_gnomad.vcf.gz", dbsnp : "${params.igenomes_base}/Homo_sapiens/ClairSTO/CHM13/Annotation/ClairSTO-pon/final_dbsnp.vcf.gz", onekgenomes : "${params.igenomes_base}/Homo_sapiens/ClairSTO/CHM13/Annotation/ClairSTO-pon/final_1kgenomes.vcf.gz", diff --git a/modules/local/sigprofiler/assignment/main.nf b/modules/local/sigprofiler/assignment/main.nf index d66c1777..2af5430d 100644 --- a/modules/local/sigprofiler/assignment/main.nf +++ b/modules/local/sigprofiler/assignment/main.nf @@ -4,8 +4,8 @@ process SIGPROFILER_ASSIGNMENT { // No conda: the image uses CHM13-T2T forks of SigProfilerMatrixGenerator (#250) and SigProfilerAssignment; see meta.yml container "${(workflow.containerEngine == 'singularity' || workflow.containerEngine == 'apptainer') && !task.ext.singularity_pull_docker_container - ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-28a9ce8' - : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-28a9ce8'}" + ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689' + : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689'}" input: tuple val(meta), path(sbs96), path(dbs78), path(id83) // SigProfilerMatrixGenerator matrices; dbs78/id83 may be [] diff --git a/modules/local/sigprofiler/install/main.nf b/modules/local/sigprofiler/install/main.nf index 710ecf57..4bc17beb 100644 --- a/modules/local/sigprofiler/install/main.nf +++ b/modules/local/sigprofiler/install/main.nf @@ -5,8 +5,8 @@ process SIGPROFILER_INSTALL { // No conda: the image uses CHM13-T2T forks of SigProfilerMatrixGenerator (#250) and SigProfilerAssignment; see meta.yml container "${(workflow.containerEngine == 'singularity' || workflow.containerEngine == 'apptainer') && !task.ext.singularity_pull_docker_container - ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-28a9ce8' - : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-28a9ce8'}" + ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689' + : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689'}" input: val(genome) // SigProfilerMatrixGenerator genome name, e.g. GRCh38 or CHM13-T2T diff --git a/modules/local/sigprofiler/matrixgenerator/main.nf b/modules/local/sigprofiler/matrixgenerator/main.nf index 934d6d9a..b1ea9dd1 100644 --- a/modules/local/sigprofiler/matrixgenerator/main.nf +++ b/modules/local/sigprofiler/matrixgenerator/main.nf @@ -4,8 +4,8 @@ process SIGPROFILER_MATRIXGENERATOR { // No conda: the image uses CHM13-T2T forks of SigProfilerMatrixGenerator (#250) and SigProfilerAssignment; see meta.yml container "${(workflow.containerEngine == 'singularity' || workflow.containerEngine == 'apptainer') && !task.ext.singularity_pull_docker_container - ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-28a9ce8' - : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-28a9ce8'}" + ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689' + : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689'}" input: tuple val(meta), path(vcf) // somatic small-variant VCF (plain or bgzipped) From f7a45c4ff5816cd12a53313cc096f3ef9aa19bad Mon Sep 17 00:00:00 2001 From: ljwharbers Date: Fri, 2 Oct 2026 15:26:11 +0200 Subject: [PATCH 2/4] Fill in PR number in CHANGELOG entry --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 057c2e00..5f1bce5a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,7 +44,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### `Fixed` -- [#XXX](https://github.com/IntGenomicsLab/lrsomatic/pull/XXX) - `--download_sigprofiler_genome` failed on CHM13 with `CHM13-T2T failed the SigProfilerMatrixGenerator checksum verification`. The published payload had been replaced by the archive rebuilt after [SigProfilerMatrixGenerator#251](https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator/issues/251), which counts transcript-end bases correctly, while the image still carried the old checksums. The SigProfiler modules now use `ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689` (`oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689`), and the CHM13 `sigprofiler_genome_url` points at the versioned `chm13_release_2026-09` payload. **A CHM13 volume installed before this change fails verification and must be reinstalled**. Transcribed-strand matrices (SBS288, SBS384 and the strand-split DBS/ID matrices) change slightly; SBS96 and the COSMIC fits are not expected to change (@ljwharbers). +- [#216](https://github.com/IntGenomicsLab/lrsomatic/pull/216) - `--download_sigprofiler_genome` failed on CHM13 with `CHM13-T2T failed the SigProfilerMatrixGenerator checksum verification`. The published payload had been replaced by the archive rebuilt after [SigProfilerMatrixGenerator#251](https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator/issues/251), which counts transcript-end bases correctly, while the image still carried the old checksums. The SigProfiler modules now use `ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689` (`oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689`), and the CHM13 `sigprofiler_genome_url` points at the versioned `chm13_release_2026-09` payload. **A CHM13 volume installed before this change fails verification and must be reinstalled**. Transcribed-strand matrices (SBS288, SBS384 and the strand-split DBS/ID matrices) change slightly; SBS96 and the COSMIC fits are not expected to change (@ljwharbers). - [#206](https://github.com/IntGenomicsLab/lrsomatic/pull/206) - A remote (http, https or ftp) ClinVar is now downloaded once per run by the new `VEPPLUGIN_CLINVAR` step in `PREPARE_VEP_PLUGINS`, instead of being staged by Nextflow as a foreign file; local and cloud-storage paths are staged as before. `GERMLINE_VEP` and `SOMATIC_VEP` re-checked the foreign file on its host for every sample, and NCBI answered the burst from a multi-sample GRCh38 run with HTTP 503, so `SOMATIC_VEP` failed with `Can't stage file ...clinvar_20260829.vcf.gz`; `-resume` could not recover, since the failed check changed the staging cache key. The download is checked against the new `--vep_clinvar_md5` and `--vep_clinvar_tbi_md5`, set by default to the checksums NCBI (GRCh38, VCF only) and Ensembl (CHM13, VCF and index) publish, so the pinned release cannot change silently. Resuming a run that already finished re-runs `GERMLINE_VEP` and `SOMATIC_VEP` once, since ClinVar now comes from a task rather than the stage cache. The ClinVar sizes in `docs/usage.md` are also corrected, and `docs/output.md` now documents `vep_plugins/` (@AmberVerhasselt). - [#203](https://github.com/IntGenomicsLab/lrsomatic/pull/203) - `CLAIRS` no longer runs with `--haplotagged_tumor_bam_provided_so_skip_intermediate_phasing_and_haplotagging`. Since somatic calling was moved ahead of `PHASING_HAPLOTYPING` (v1.1.0), ClairS has received the untagged minimap2 BAM, so the flag made it skip its own phasing and haplotagging and call every paired sample without haplotype information: the full-alignment model saw no `HP` tags and the haplotype filtering step had nothing to filter on, the same as `--disable_phasing`. ClairS now runs Clair3 on the normal and tumour BAMs and phases the tumour itself. **Paired somatic calls change** (fewer false positives expected), and `CLAIRS` takes longer and uses more work-directory space (@ljwharbers). - [#203](https://github.com/IntGenomicsLab/lrsomatic/pull/203) - `docs/output.md` now lists the ClairS SNV output as `snvs.vcf.gz`, the name the pipeline publishes, instead of `snv.vcf.gz` (@ljwharbers). From 1d3b7c7a6ad546c246e5753d26903ed64e5eca86 Mon Sep 17 00:00:00 2001 From: ljwharbers Date: Fri, 2 Oct 2026 15:44:37 +0200 Subject: [PATCH 3/4] Verify user-supplied SigProfiler volumes up front; cover GRCh38 too Addresses review points 1 and 3 on #216. SIGPROFILER_MATRIXGENERATOR already re-checks the payload for every sample and fails each task with a generic checksum error. The new SIGPROFILER_VERIFY step checks a --sigprofiler_genome_dir volume once, before any sample work, and stops with the checksum table and reinstall instructions. The 7894689 image also carries upstream's corrected GRCh38 payload (the AlexandrovLab FTP has served it as GRCh38.tar.gz since 2026-09-25; the old one is GRCh38_Legacy), so GRCh38 volumes installed before this change need reinstalling as well. The docs, schema help_text and CHANGELOG now say so. The new small-tagged module test feeds stale GRCh38 and CHM13-T2T volumes and checks that each is rejected, with the chr1 checksum the image expects, so CI ties the pinned image to the payload revisions it downloads. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 +- conf/modules.config | 5 ++ docs/usage.md | 5 ++ modules/local/sigprofiler/verify/main.nf | 48 +++++++++++ modules/local/sigprofiler/verify/meta.yml | 58 ++++++++++++++ .../sigprofiler/verify/tests/main.nf.test | 79 +++++++++++++++++++ nextflow_schema.json | 2 +- subworkflows/local/prepare_signatures.nf | 14 +++- .../tsb/CHM13-T2T/1.txt | 1 + .../sigprofiler_stale_volume/tsb/GRCh38/1.txt | 1 + 10 files changed, 211 insertions(+), 4 deletions(-) create mode 100644 modules/local/sigprofiler/verify/main.nf create mode 100644 modules/local/sigprofiler/verify/meta.yml create mode 100644 modules/local/sigprofiler/verify/tests/main.nf.test create mode 100644 tests/fixtures/sigprofiler_stale_volume/tsb/CHM13-T2T/1.txt create mode 100644 tests/fixtures/sigprofiler_stale_volume/tsb/GRCh38/1.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f1bce5a..272209dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,7 +44,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### `Fixed` -- [#216](https://github.com/IntGenomicsLab/lrsomatic/pull/216) - `--download_sigprofiler_genome` failed on CHM13 with `CHM13-T2T failed the SigProfilerMatrixGenerator checksum verification`. The published payload had been replaced by the archive rebuilt after [SigProfilerMatrixGenerator#251](https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator/issues/251), which counts transcript-end bases correctly, while the image still carried the old checksums. The SigProfiler modules now use `ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689` (`oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689`), and the CHM13 `sigprofiler_genome_url` points at the versioned `chm13_release_2026-09` payload. **A CHM13 volume installed before this change fails verification and must be reinstalled**. Transcribed-strand matrices (SBS288, SBS384 and the strand-split DBS/ID matrices) change slightly; SBS96 and the COSMIC fits are not expected to change (@ljwharbers). +- [#216](https://github.com/IntGenomicsLab/lrsomatic/pull/216) - `--download_sigprofiler_genome` failed on CHM13 with `CHM13-T2T failed the SigProfilerMatrixGenerator checksum verification`. The published payload had been replaced by the archive rebuilt after [SigProfilerMatrixGenerator#251](https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator/issues/251), which counts transcript-end bases correctly, while the image still carried the old checksums. The SigProfiler modules now use `ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689` (`oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689`), and the CHM13 `sigprofiler_genome_url` points at the versioned `chm13_release_2026-09` payload. The new SigProfilerMatrixGenerator also includes upstream's corrected GRCh38 payload, which the AlexandrovLab FTP has served as `GRCh38.tar.gz` since 2026-09-25; the old one is now `GRCh38_Legacy`. **GRCh38 and CHM13-T2T volumes installed before this change fail verification and must be reinstalled.** A volume passed with `--sigprofiler_genome_dir` is now checked once, up front, by the new `SIGPROFILER_VERIFY` step, which stops with reinstall instructions instead of failing in every `SIGPROFILER_MATRIXGENERATOR` task. Transcribed-strand matrices (SBS288, SBS384) change slightly; SBS96 and the COSMIC fits are not expected to change (@ljwharbers). - [#206](https://github.com/IntGenomicsLab/lrsomatic/pull/206) - A remote (http, https or ftp) ClinVar is now downloaded once per run by the new `VEPPLUGIN_CLINVAR` step in `PREPARE_VEP_PLUGINS`, instead of being staged by Nextflow as a foreign file; local and cloud-storage paths are staged as before. `GERMLINE_VEP` and `SOMATIC_VEP` re-checked the foreign file on its host for every sample, and NCBI answered the burst from a multi-sample GRCh38 run with HTTP 503, so `SOMATIC_VEP` failed with `Can't stage file ...clinvar_20260829.vcf.gz`; `-resume` could not recover, since the failed check changed the staging cache key. The download is checked against the new `--vep_clinvar_md5` and `--vep_clinvar_tbi_md5`, set by default to the checksums NCBI (GRCh38, VCF only) and Ensembl (CHM13, VCF and index) publish, so the pinned release cannot change silently. Resuming a run that already finished re-runs `GERMLINE_VEP` and `SOMATIC_VEP` once, since ClinVar now comes from a task rather than the stage cache. The ClinVar sizes in `docs/usage.md` are also corrected, and `docs/output.md` now documents `vep_plugins/` (@AmberVerhasselt). - [#203](https://github.com/IntGenomicsLab/lrsomatic/pull/203) - `CLAIRS` no longer runs with `--haplotagged_tumor_bam_provided_so_skip_intermediate_phasing_and_haplotagging`. Since somatic calling was moved ahead of `PHASING_HAPLOTYPING` (v1.1.0), ClairS has received the untagged minimap2 BAM, so the flag made it skip its own phasing and haplotagging and call every paired sample without haplotype information: the full-alignment model saw no `HP` tags and the haplotype filtering step had nothing to filter on, the same as `--disable_phasing`. ClairS now runs Clair3 on the normal and tumour BAMs and phases the tumour itself. **Paired somatic calls change** (fewer false positives expected), and `CLAIRS` takes longer and uses more work-directory space (@ljwharbers). - [#203](https://github.com/IntGenomicsLab/lrsomatic/pull/203) - `docs/output.md` now lists the ClairS SNV output as `snvs.vcf.gz`, the name the pipeline publishes, instead of `snv.vcf.gz` (@ljwharbers). diff --git a/conf/modules.config b/conf/modules.config index 59200256..279a016e 100644 --- a/conf/modules.config +++ b/conf/modules.config @@ -795,6 +795,11 @@ process { ] } + withName: 'SIGPROFILER_VERIFY' { + // Passes the user's volume through unchanged; never publish a copy of it + publishDir = [ enabled: false ] + } + withName : '.*:SIGNATURES_BCFTOOLS_VIEW' { // SigProfilerMatrixGenerator ignores FILTER and reads plain-text VCF only ext.args = { "--apply-filters PASS --types snps,mnps,indels --exclude 'ALT=\"*\"' --output-type v" } diff --git a/docs/usage.md b/docs/usage.md index 1119930a..e46ce940 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -380,6 +380,11 @@ Mutational signature analysis runs [SigProfilerMatrixGenerator](https://github.c Running with neither, and without `--skip_signatures`, stops the pipeline at start-up. +A volume passed with `--sigprofiler_genome_dir` is checked once per run against the chromosome checksums of the pipeline's SigProfilerMatrixGenerator (`SIGPROFILER_VERIFY`). If it does not match, the run stops before any sample is processed. + +> [!WARNING] +> GRCh38 and CHM13-T2T payloads installed before this release no longer pass that check. SigProfilerMatrixGenerator corrected how both are encoded, and the payloads it now downloads differ from the earlier ones. Reinstall once with `--download_sigprofiler_genome`, or with `SigProfilerMatrixGenerator install --volume ` from the same image, and pass the new volume on later runs. Mutation counts and COSMIC fits are unaffected. Only the strand-split matrices (for example SBS288 and SBS384) change, slightly. + | Parameter | Description | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--sigprofiler_genome_dir` | Full path to a SigProfilerMatrixGenerator volume containing `tsb//`. Default = `null` | diff --git a/modules/local/sigprofiler/verify/main.nf b/modules/local/sigprofiler/verify/main.nf new file mode 100644 index 00000000..51e36a13 --- /dev/null +++ b/modules/local/sigprofiler/verify/main.nf @@ -0,0 +1,48 @@ +process SIGPROFILER_VERIFY { + tag "$genome" + label 'process_single' + + // No conda: the image uses CHM13-T2T forks of SigProfilerMatrixGenerator (#250) and SigProfilerAssignment; see meta.yml + container "${(workflow.containerEngine == 'singularity' || workflow.containerEngine == 'apptainer') && !task.ext.singularity_pull_docker_container + ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689' + : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689'}" + + input: + path(volume, stageAs: 'genome_volume') // SigProfilerMatrixGenerator volume containing tsb// + val(genome) // SigProfilerMatrixGenerator genome name, e.g. GRCh38 or CHM13-T2T + + output: + val(true) , emit: verified + tuple val("${task.process}"), val('sigprofilermatrixgenerator'), eval("python -c 'import importlib.metadata as m; print(m.version(\"SigProfilerMatrixGenerator\"))'"), topic: versions, emit: versions_sigprofilermatrixgenerator + + when: + task.ext.when == null || task.ext.when + + script: + if (workflow.profile.tokenize(',').intersect(['conda', 'mamba']).size() >= 1) { + error "SIGPROFILER_VERIFY does not support Conda. Please use Docker / Singularity / Apptainer instead." + } + """ + # SIGPROFILER_MATRIXGENERATOR re-checks the payload for every sample; checking once here fails a stale or damaged + # volume before any sample work, with the reinstall instructions instead of a per-sample checksum error + python - <<'PY' + import sys + from SigProfilerMatrixGenerator.scripts import reference_genome_manager as rgm + + manager = rgm.ReferenceGenomeManager("genome_volume") + if not manager.is_genome_installed("${genome}"): + manager.print_genome_checksum_verification_report("${genome}") + sys.exit( + "ERROR: the ${genome} payload in --sigprofiler_genome_dir does not match the checksums of this pipeline's " + "SigProfilerMatrixGenerator. GRCh38 and CHM13-T2T payloads installed before lrsomatic PR #216 are a " + "superseded revision. Reinstall it with --download_sigprofiler_genome (published to " + "/cache/sigprofiler/volume) and pass that directory on later runs." + ) + PY + """ + + stub: + """ + echo "stub: skipping checksum verification of genome_volume/tsb/${genome}" + """ +} diff --git a/modules/local/sigprofiler/verify/meta.yml b/modules/local/sigprofiler/verify/meta.yml new file mode 100644 index 00000000..586b7f39 --- /dev/null +++ b/modules/local/sigprofiler/verify/meta.yml @@ -0,0 +1,58 @@ +--- +# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/modules/meta-schema.json +name: "sigprofiler_verify" +description: Check a user-supplied SigProfilerMatrixGenerator volume against the chromosome checksums of the pipeline's SigProfilerMatrixGenerator, so a stale or damaged payload fails before any sample is processed +keywords: + - mutational signatures + - reference genome + - sigprofiler + - checksum +tools: + - "sigprofilermatrixgenerator": + description: "SigProfilerMatrixGenerator creates mutational matrices for all types of somatic mutations" + homepage: "https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator" + documentation: "https://sigprofilersuite.github.io/SigProfilerMatrixGenerator/" + tool_dev_url: "https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator" + doi: "10.1186/s12864-019-6041-2" + licence: ["BSD-2-Clause"] + +input: + - volume: + type: directory + description: SigProfilerMatrixGenerator volume containing tsb// (e.g. from --sigprofiler_genome_dir) + - genome: + type: string + description: SigProfilerMatrixGenerator genome name, e.g. GRCh38 or CHM13-T2T + +output: + verified: + - "true": + type: boolean + description: Emitted once the volume's chromosome files match the registered checksums; the task fails otherwise + versions_sigprofilermatrixgenerator: + - - ${task.process}: + type: string + description: The process the versions were collected from + - sigprofilermatrixgenerator: + type: string + description: The tool name + - 'python -c ''import importlib.metadata as m; print(m.version("SigProfilerMatrixGenerator"))''': + type: string + description: The command used to generate the version of the tool + +topics: + versions: + - - ${task.process}: + type: string + description: The process the versions were collected from + - sigprofilermatrixgenerator: + type: string + description: The tool name + - 'python -c ''import importlib.metadata as m; print(m.version("SigProfilerMatrixGenerator"))''': + type: string + description: The command used to generate the version of the tool + +authors: + - "@ljwharbers" +maintainers: + - "@ljwharbers" diff --git a/modules/local/sigprofiler/verify/tests/main.nf.test b/modules/local/sigprofiler/verify/tests/main.nf.test new file mode 100644 index 00000000..9813adbb --- /dev/null +++ b/modules/local/sigprofiler/verify/tests/main.nf.test @@ -0,0 +1,79 @@ +nextflow_process { + + name "Test Process SIGPROFILER_VERIFY" + script "../main.nf" + process "SIGPROFILER_VERIFY" + + tag "modules" + tag "modules_local" + tag "sigprofiler" + tag "sigprofiler_verify" + tag "small" + + // No real payload fits in CI (~3 GB per genome), so these tests check the rejection path. The checksum report + // lists the chr1 MD5 the image expects, which ties the pinned image to the payload revision its URL serves: + // GRCh38 from the AlexandrovLab FTP since 2026-09-25, CHM13-T2T from chm13_release_2026-09. + + test("CHM13-T2T - stale volume is rejected with reinstall instructions") { + + when { + process { + """ + input[0] = file("\${projectDir}/tests/fixtures/sigprofiler_stale_volume", checkIfExists: true) + input[1] = 'CHM13-T2T' + """ + } + } + + then { + def report = (process.stdout + process.stderr).join('\n') + assertAll( + { assert process.failed }, + { assert report.contains('Reinstall it with --download_sigprofiler_genome') }, + // T2T-CHM13v2.0 chr1 of the payload rebuilt after SigProfilerMatrixGenerator#251 + { assert report.contains('8683547ca6e5cbb2afd3633a32e6353a') } + ) + } + } + + test("GRCh38 - stale volume is rejected with reinstall instructions") { + + when { + process { + """ + input[0] = file("\${projectDir}/tests/fixtures/sigprofiler_stale_volume", checkIfExists: true) + input[1] = 'GRCh38' + """ + } + } + + then { + def report = (process.stdout + process.stderr).join('\n') + assertAll( + { assert process.failed }, + { assert report.contains('Reinstall it with --download_sigprofiler_genome') }, + // GRCh38 chr1 of upstream's corrected payload (the pre-2026-09-25 one is GRCh38_Legacy) + { assert report.contains('570ba2c0c11b999a906abd4f854a38be') } + ) + } + } + + test("stub") { + + options "-stub" + + when { + process { + """ + input[0] = file("\${projectDir}/tests/fixtures/sigprofiler_stale_volume", checkIfExists: true) + input[1] = 'CHM13-T2T' + """ + } + } + + then { + assert process.success + assert process.out.verified == [true] + } + } +} diff --git a/nextflow_schema.json b/nextflow_schema.json index 2a5cf13b..ca757203 100644 --- a/nextflow_schema.json +++ b/nextflow_schema.json @@ -481,7 +481,7 @@ "type": "string", "format": "directory-path", "description": "SigProfilerMatrixGenerator volume directory containing tsb// (the installed genome payload).", - "help_text": "Create it once with `--download_sigprofiler_genome` (published to `/cache/sigprofiler/volume`) or with `SigProfilerMatrixGenerator install --volume `, then pass it here on later runs.", + "help_text": "Create it once with `--download_sigprofiler_genome` (published to `/cache/sigprofiler/volume`) or with `SigProfilerMatrixGenerator install --volume `, then pass it here on later runs. The volume is checked against the pipeline's SigProfilerMatrixGenerator checksums before any sample is processed. GRCh38 and CHM13-T2T volumes installed before lrsomatic PR #216 are a superseded payload revision and must be reinstalled.", "fa_icon": "fas fa-folder-open" }, "download_sigprofiler_genome": { diff --git a/subworkflows/local/prepare_signatures.nf b/subworkflows/local/prepare_signatures.nf index e27eee87..4255b336 100644 --- a/subworkflows/local/prepare_signatures.nf +++ b/subworkflows/local/prepare_signatures.nf @@ -1,4 +1,5 @@ include { SIGPROFILER_INSTALL } from '../../modules/local/sigprofiler/install/main' +include { SIGPROFILER_VERIFY } from '../../modules/local/sigprofiler/verify/main' workflow PREPARE_SIGNATURES { @@ -39,9 +40,18 @@ workflow PREPARE_SIGNATURES { if (n_chrom < 24) { error("${tsb_dir} holds ${n_chrom} of the 24 chromosome files of a complete ${genome} install.") } - sigprofiler_volume = channel.value(file(genome_dir, type: 'dir', checkIfExists: true)) + // + // MODULE: SIGPROFILER_VERIFY (label: process_single) -- checksums once, so a stale payload fails before any sample + // + def volume_dir = file(genome_dir, type: 'dir', checkIfExists: true) + SIGPROFILER_VERIFY ( + volume_dir, + genome + ) + // Hand on the user's directory itself, released only once it has been verified + sigprofiler_volume = SIGPROFILER_VERIFY.out.verified.map { _verified -> volume_dir } } - // sigprofiler_volume: path -- SigProfilerMatrixGenerator volume root (downloaded or validated local) + // sigprofiler_volume: path -- SigProfilerMatrixGenerator volume root (downloaded or verified local) emit: volume = sigprofiler_volume // path -- volume directory containing tsb// diff --git a/tests/fixtures/sigprofiler_stale_volume/tsb/CHM13-T2T/1.txt b/tests/fixtures/sigprofiler_stale_volume/tsb/CHM13-T2T/1.txt new file mode 100644 index 00000000..47b63767 --- /dev/null +++ b/tests/fixtures/sigprofiler_stale_volume/tsb/CHM13-T2T/1.txt @@ -0,0 +1 @@ +stale payload stand-in for SIGPROFILER_VERIFY tests diff --git a/tests/fixtures/sigprofiler_stale_volume/tsb/GRCh38/1.txt b/tests/fixtures/sigprofiler_stale_volume/tsb/GRCh38/1.txt new file mode 100644 index 00000000..47b63767 --- /dev/null +++ b/tests/fixtures/sigprofiler_stale_volume/tsb/GRCh38/1.txt @@ -0,0 +1 @@ +stale payload stand-in for SIGPROFILER_VERIFY tests From f20a2a27fec6ffbbb264ac318ff4a9df8d9e07dc Mon Sep 17 00:00:00 2001 From: ljwharbers Date: Fri, 2 Oct 2026 16:08:22 +0200 Subject: [PATCH 4/4] Name the cause in SIGPROFILER_VERIFY errors; stub-test the signatures chain SIGPROFILER_VERIFY reported every failed check as a superseded payload from before PR #216. It now tells apart a genome the image has no checksums for, an incomplete install (lists the missing chromosome files), and a checksum mismatch (stale payload from lrsomatic < 1.2.0, or a corrupted copy). The schema help_text and usage docs say the same. PR CI ran nothing of the signatures step (conf/test.config skips it). New stub tests, all tagged small: - PREPARE_SIGNATURES: the --sigprofiler_genome_dir branch verifies and hands on the user's directory; the download branch installs. - SIGPROFILER_MATRIXGENERATOR: the volume from PREPARE_SIGNATURES, wired as in workflows/lrsomatic.nf, reaches two samples. - SIGPROFILER_ASSIGNMENT: runs in the pinned image. The versions evals run inside the image, so the image tag is checked under each CI profile. The verify module test gains the mismatch and unregistered genome cases and asserts on rendered text only, since Nextflow echoes the script source on failure. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 +- docs/usage.md | 2 +- .../sigprofiler/assignment/tests/main.nf.test | 40 ++++++++++ .../matrixgenerator/tests/main.nf.test | 55 ++++++++++++++ modules/local/sigprofiler/verify/main.nf | 32 ++++++-- .../sigprofiler/verify/tests/main.nf.test | 60 +++++++++++++-- nextflow_schema.json | 2 +- .../local/tests/prepare_signatures.nf.test | 73 +++++++++++++++++++ .../tsb/CHM13-T2T/1.txt | 0 .../tsb/CHM13-T2T/10.txt | 0 .../tsb/CHM13-T2T/11.txt | 0 .../tsb/CHM13-T2T/12.txt | 0 .../tsb/CHM13-T2T/13.txt | 0 .../tsb/CHM13-T2T/14.txt | 0 .../tsb/CHM13-T2T/15.txt | 0 .../tsb/CHM13-T2T/16.txt | 0 .../tsb/CHM13-T2T/17.txt | 0 .../tsb/CHM13-T2T/18.txt | 0 .../tsb/CHM13-T2T/19.txt | 0 .../tsb/CHM13-T2T/2.txt | 0 .../tsb/CHM13-T2T/20.txt | 0 .../tsb/CHM13-T2T/21.txt | 0 .../tsb/CHM13-T2T/22.txt | 0 .../tsb/CHM13-T2T/3.txt | 0 .../tsb/CHM13-T2T/4.txt | 0 .../tsb/CHM13-T2T/5.txt | 0 .../tsb/CHM13-T2T/6.txt | 0 .../tsb/CHM13-T2T/7.txt | 0 .../tsb/CHM13-T2T/8.txt | 0 .../tsb/CHM13-T2T/9.txt | 0 .../tsb/CHM13-T2T/X.txt | 0 .../tsb/CHM13-T2T/Y.txt | 0 32 files changed, 252 insertions(+), 14 deletions(-) create mode 100644 modules/local/sigprofiler/assignment/tests/main.nf.test create mode 100644 modules/local/sigprofiler/matrixgenerator/tests/main.nf.test create mode 100644 subworkflows/local/tests/prepare_signatures.nf.test create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/1.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/10.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/11.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/12.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/13.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/14.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/15.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/16.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/17.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/18.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/19.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/2.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/20.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/21.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/22.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/3.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/4.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/5.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/6.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/7.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/8.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/9.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/X.txt create mode 100644 tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/Y.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 272209dd..5b301bf5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,7 +44,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### `Fixed` -- [#216](https://github.com/IntGenomicsLab/lrsomatic/pull/216) - `--download_sigprofiler_genome` failed on CHM13 with `CHM13-T2T failed the SigProfilerMatrixGenerator checksum verification`. The published payload had been replaced by the archive rebuilt after [SigProfilerMatrixGenerator#251](https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator/issues/251), which counts transcript-end bases correctly, while the image still carried the old checksums. The SigProfiler modules now use `ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689` (`oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689`), and the CHM13 `sigprofiler_genome_url` points at the versioned `chm13_release_2026-09` payload. The new SigProfilerMatrixGenerator also includes upstream's corrected GRCh38 payload, which the AlexandrovLab FTP has served as `GRCh38.tar.gz` since 2026-09-25; the old one is now `GRCh38_Legacy`. **GRCh38 and CHM13-T2T volumes installed before this change fail verification and must be reinstalled.** A volume passed with `--sigprofiler_genome_dir` is now checked once, up front, by the new `SIGPROFILER_VERIFY` step, which stops with reinstall instructions instead of failing in every `SIGPROFILER_MATRIXGENERATOR` task. Transcribed-strand matrices (SBS288, SBS384) change slightly; SBS96 and the COSMIC fits are not expected to change (@ljwharbers). +- [#216](https://github.com/IntGenomicsLab/lrsomatic/pull/216) - `--download_sigprofiler_genome` failed on CHM13 with `CHM13-T2T failed the SigProfilerMatrixGenerator checksum verification`. The published payload had been replaced by the archive rebuilt after [SigProfilerMatrixGenerator#251](https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator/issues/251), which counts transcript-end bases correctly, while the image still carried the old checksums. The SigProfiler modules now use `ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689` (`oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689`), and the CHM13 `sigprofiler_genome_url` points at the versioned `chm13_release_2026-09` payload. The new SigProfilerMatrixGenerator also includes upstream's corrected GRCh38 payload, which the AlexandrovLab FTP has served as `GRCh38.tar.gz` since 2026-09-25; the old one is now `GRCh38_Legacy`. **GRCh38 and CHM13-T2T volumes installed before this change fail verification and must be reinstalled.** A volume passed with `--sigprofiler_genome_dir` is now checked once, up front, by the new `SIGPROFILER_VERIFY` step, which stops with reinstall instructions instead of failing in every `SIGPROFILER_MATRIXGENERATOR` task. Its error names the cause: missing chromosome files, a checksum mismatch (stale or corrupted payload), or a genome the image has no checksums for. Stub nf-tests (tag `small`) now cover `PREPARE_SIGNATURES`, the volume reaching every `SIGPROFILER_MATRIXGENERATOR` sample, and the pinned image in `SIGPROFILER_MATRIXGENERATOR` and `SIGPROFILER_ASSIGNMENT`. Transcribed-strand matrices (SBS288, SBS384) change slightly; SBS96 and the COSMIC fits are not expected to change (@ljwharbers). - [#206](https://github.com/IntGenomicsLab/lrsomatic/pull/206) - A remote (http, https or ftp) ClinVar is now downloaded once per run by the new `VEPPLUGIN_CLINVAR` step in `PREPARE_VEP_PLUGINS`, instead of being staged by Nextflow as a foreign file; local and cloud-storage paths are staged as before. `GERMLINE_VEP` and `SOMATIC_VEP` re-checked the foreign file on its host for every sample, and NCBI answered the burst from a multi-sample GRCh38 run with HTTP 503, so `SOMATIC_VEP` failed with `Can't stage file ...clinvar_20260829.vcf.gz`; `-resume` could not recover, since the failed check changed the staging cache key. The download is checked against the new `--vep_clinvar_md5` and `--vep_clinvar_tbi_md5`, set by default to the checksums NCBI (GRCh38, VCF only) and Ensembl (CHM13, VCF and index) publish, so the pinned release cannot change silently. Resuming a run that already finished re-runs `GERMLINE_VEP` and `SOMATIC_VEP` once, since ClinVar now comes from a task rather than the stage cache. The ClinVar sizes in `docs/usage.md` are also corrected, and `docs/output.md` now documents `vep_plugins/` (@AmberVerhasselt). - [#203](https://github.com/IntGenomicsLab/lrsomatic/pull/203) - `CLAIRS` no longer runs with `--haplotagged_tumor_bam_provided_so_skip_intermediate_phasing_and_haplotagging`. Since somatic calling was moved ahead of `PHASING_HAPLOTYPING` (v1.1.0), ClairS has received the untagged minimap2 BAM, so the flag made it skip its own phasing and haplotagging and call every paired sample without haplotype information: the full-alignment model saw no `HP` tags and the haplotype filtering step had nothing to filter on, the same as `--disable_phasing`. ClairS now runs Clair3 on the normal and tumour BAMs and phases the tumour itself. **Paired somatic calls change** (fewer false positives expected), and `CLAIRS` takes longer and uses more work-directory space (@ljwharbers). - [#203](https://github.com/IntGenomicsLab/lrsomatic/pull/203) - `docs/output.md` now lists the ClairS SNV output as `snvs.vcf.gz`, the name the pipeline publishes, instead of `snv.vcf.gz` (@ljwharbers). diff --git a/docs/usage.md b/docs/usage.md index e46ce940..1760510d 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -380,7 +380,7 @@ Mutational signature analysis runs [SigProfilerMatrixGenerator](https://github.c Running with neither, and without `--skip_signatures`, stops the pipeline at start-up. -A volume passed with `--sigprofiler_genome_dir` is checked once per run against the chromosome checksums of the pipeline's SigProfilerMatrixGenerator (`SIGPROFILER_VERIFY`). If it does not match, the run stops before any sample is processed. +A volume passed with `--sigprofiler_genome_dir` is checked once per run against the chromosome checksums of the pipeline's SigProfilerMatrixGenerator (`SIGPROFILER_VERIFY`). If a chromosome file is missing or a checksum differs, the run stops before any sample is processed, and the error says which of the two it found. A genome the image has no checksums for is rejected too. > [!WARNING] > GRCh38 and CHM13-T2T payloads installed before this release no longer pass that check. SigProfilerMatrixGenerator corrected how both are encoded, and the payloads it now downloads differ from the earlier ones. Reinstall once with `--download_sigprofiler_genome`, or with `SigProfilerMatrixGenerator install --volume ` from the same image, and pass the new volume on later runs. Mutation counts and COSMIC fits are unaffected. Only the strand-split matrices (for example SBS288 and SBS384) change, slightly. diff --git a/modules/local/sigprofiler/assignment/tests/main.nf.test b/modules/local/sigprofiler/assignment/tests/main.nf.test new file mode 100644 index 00000000..ee178d3c --- /dev/null +++ b/modules/local/sigprofiler/assignment/tests/main.nf.test @@ -0,0 +1,40 @@ +nextflow_process { + + name "Test Process SIGPROFILER_ASSIGNMENT" + script "../main.nf" + process "SIGPROFILER_ASSIGNMENT" + + tag "modules" + tag "modules_local" + tag "sigprofiler" + tag "sigprofiler_assignment" + tag "small" + + // Under -stub; the versions eval still runs in the pinned image, so its tag must resolve under each profile + + test("stub") { + + options "-stub" + + when { + process { + """ + // any file stands in for the SBS96 matrix; the stub does not read it + def matrix = file("\${projectDir}/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/1.txt", checkIfExists: true) + input[0] = [ [ id:'sample1' ], matrix, [], [] ] + input[1] = 'CHM13-T2T' + input[2] = 3.6 + """ + } + } + + then { + assertAll( + { assert process.success }, + { assert process.out.activities.size() == 1 }, + { assert process.out.versions_sigprofilerassignment.size() == 1 }, + { assert process.out.versions_sigprofilerassignment[0][2] ==~ /\d+\.\d+.*/ } + ) + } + } +} diff --git a/modules/local/sigprofiler/matrixgenerator/tests/main.nf.test b/modules/local/sigprofiler/matrixgenerator/tests/main.nf.test new file mode 100644 index 00000000..564d3230 --- /dev/null +++ b/modules/local/sigprofiler/matrixgenerator/tests/main.nf.test @@ -0,0 +1,55 @@ +nextflow_process { + + name "Test Process SIGPROFILER_MATRIXGENERATOR" + script "../main.nf" + process "SIGPROFILER_MATRIXGENERATOR" + + tag "modules" + tag "modules_local" + tag "sigprofiler" + tag "sigprofiler_matrixgenerator" + tag "small" + + // No real payload fits in CI (~3 GB per genome), so this runs under -stub. The volume comes from + // PREPARE_SIGNATURES and is wired as in workflows/lrsomatic.nf, so every sample must receive it. + // The versions eval still runs in the pinned image, so its tag must resolve under each profile. + + test("stub - the verified volume reaches every sample") { + + options "-stub" + + setup { + run("PREPARE_SIGNATURES") { + script "../../../../../subworkflows/local/prepare_signatures.nf" + workflow { + """ + input[0] = 'CHM13-T2T' + input[1] = null + input[2] = "\${projectDir}/tests/fixtures/sigprofiler_stub_volume" + input[3] = false + """ + } + } + } + + when { + process { + """ + def vcf = file("\${projectDir}/tests/fixtures/vcfsplit_indel.vcf", checkIfExists: true) + input[0] = channel.of([ [ id:'sample1' ], vcf ], [ [ id:'sample2' ], vcf ]) + input[1] = PREPARE_SIGNATURES.out.volume.map { volume -> [ [:], volume ] } + input[2] = 'CHM13-T2T' + """ + } + } + + then { + assertAll( + { assert process.success }, + { assert process.out.sbs96.collect { meta, _matrix -> meta.id }.sort() == ['sample1', 'sample2'] }, + { assert process.out.versions_sigprofilermatrixgenerator.size() == 2 }, + { assert process.out.versions_sigprofilermatrixgenerator.every { _proc, _tool, version -> version ==~ /\d+\.\d+.*/ } } + ) + } + } +} diff --git a/modules/local/sigprofiler/verify/main.nf b/modules/local/sigprofiler/verify/main.nf index 51e36a13..4bf86194 100644 --- a/modules/local/sigprofiler/verify/main.nf +++ b/modules/local/sigprofiler/verify/main.nf @@ -29,14 +29,34 @@ process SIGPROFILER_VERIFY { import sys from SigProfilerMatrixGenerator.scripts import reference_genome_manager as rgm + genome = "${genome}" + if genome not in rgm.CHECKSUMS: + sys.exit( + f"ERROR: this pipeline's SigProfilerMatrixGenerator has no checksums for {genome} (registered: " + f"{', '.join(sorted(rgm.CHECKSUMS))}). Set --sigprofiler_genome to one of them or use --skip_signatures." + ) + manager = rgm.ReferenceGenomeManager("genome_volume") - if not manager.is_genome_installed("${genome}"): - manager.print_genome_checksum_verification_report("${genome}") + if not manager.is_genome_installed(genome): + manager.print_genome_checksum_verification_report(genome) + tsb = manager.reference_dir.get_tsb_dir() / genome + expected = rgm.CHECKSUMS[genome] + missing = [chrom for chrom in expected if not (tsb / f"{chrom}.txt").is_file()] + if missing: + cause = ( + f"is an incomplete install: {len(missing)} of {len(expected)} chromosome files are missing " + f"({', '.join(missing)})." + ) + else: + cause = ( + "does not match the checksums of this pipeline's SigProfilerMatrixGenerator. Either it is a stale " + "payload (GRCh38 and CHM13-T2T installed for lrsomatic < 1.2.0 are a superseded revision) or the " + "copy is corrupted." + ) sys.exit( - "ERROR: the ${genome} payload in --sigprofiler_genome_dir does not match the checksums of this pipeline's " - "SigProfilerMatrixGenerator. GRCh38 and CHM13-T2T payloads installed before lrsomatic PR #216 are a " - "superseded revision. Reinstall it with --download_sigprofiler_genome (published to " - "/cache/sigprofiler/volume) and pass that directory on later runs." + f"ERROR: the {genome} payload in --sigprofiler_genome_dir {cause} Reinstall it with " + "--download_sigprofiler_genome (published to /cache/sigprofiler/volume) and pass that directory " + "on later runs." ) PY """ diff --git a/modules/local/sigprofiler/verify/tests/main.nf.test b/modules/local/sigprofiler/verify/tests/main.nf.test index 9813adbb..cea6e43f 100644 --- a/modules/local/sigprofiler/verify/tests/main.nf.test +++ b/modules/local/sigprofiler/verify/tests/main.nf.test @@ -10,11 +10,11 @@ nextflow_process { tag "sigprofiler_verify" tag "small" - // No real payload fits in CI (~3 GB per genome), so these tests check the rejection path. The checksum report + // No real payload fits in CI (~3 GB per genome), so these tests check the rejection paths. The checksum report // lists the chr1 MD5 the image expects, which ties the pinned image to the payload revision its URL serves: // GRCh38 from the AlexandrovLab FTP since 2026-09-25, CHM13-T2T from chm13_release_2026-09. - test("CHM13-T2T - stale volume is rejected with reinstall instructions") { + test("CHM13-T2T - incomplete volume is rejected with reinstall instructions") { when { process { @@ -29,14 +29,17 @@ nextflow_process { def report = (process.stdout + process.stderr).join('\n') assertAll( { assert process.failed }, - { assert report.contains('Reinstall it with --download_sigprofiler_genome') }, + // rendered message only: the script source Nextflow echoes on failure has the template + { assert report.contains('X, Y). Reinstall it with --download_sigprofiler_genome') }, + // the fixture holds chromosome 1 only + { assert report.contains('incomplete install: 23 of 24 chromosome files are missing') }, // T2T-CHM13v2.0 chr1 of the payload rebuilt after SigProfilerMatrixGenerator#251 { assert report.contains('8683547ca6e5cbb2afd3633a32e6353a') } ) } } - test("GRCh38 - stale volume is rejected with reinstall instructions") { + test("GRCh38 - incomplete volume is rejected with reinstall instructions") { when { process { @@ -51,13 +54,60 @@ nextflow_process { def report = (process.stdout + process.stderr).join('\n') assertAll( { assert process.failed }, - { assert report.contains('Reinstall it with --download_sigprofiler_genome') }, + { assert report.contains('Y, MT). Reinstall it with --download_sigprofiler_genome') }, + // the fixture holds chromosome 1 only; GRCh38 also registers MT + { assert report.contains('incomplete install: 24 of 25 chromosome files are missing') }, // GRCh38 chr1 of upstream's corrected payload (the pre-2026-09-25 one is GRCh38_Legacy) { assert report.contains('570ba2c0c11b999a906abd4f854a38be') } ) } } + test("CHM13-T2T - complete volume with wrong checksums is rejected as stale or corrupted") { + + when { + process { + """ + // all 24 chromosome files, but empty + input[0] = file("\${projectDir}/tests/fixtures/sigprofiler_stub_volume", checkIfExists: true) + input[1] = 'CHM13-T2T' + """ + } + } + + then { + def report = (process.stdout + process.stderr).join('\n') + assertAll( + { assert process.failed }, + // rendered message only: the script source Nextflow echoes on failure has the template + { assert report.contains('CHM13-T2T payload in --sigprofiler_genome_dir does not match the checksums') }, + { assert !(report =~ /incomplete install: \d/) }, + { assert report.contains('copy is corrupted. Reinstall it with --download_sigprofiler_genome') } + ) + } + } + + test("genome without checksums in the image is rejected") { + + when { + process { + """ + input[0] = file("\${projectDir}/tests/fixtures/sigprofiler_stale_volume", checkIfExists: true) + input[1] = 'GRCh37_unregistered' + """ + } + } + + then { + def report = (process.stdout + process.stderr).join('\n') + assertAll( + { assert process.failed }, + { assert report.contains('has no checksums for GRCh37_unregistered') }, + { assert report.contains('(registered: CHM13-T2T, GRCh37,') } + ) + } + } + test("stub") { options "-stub" diff --git a/nextflow_schema.json b/nextflow_schema.json index ca757203..ceb7b12f 100644 --- a/nextflow_schema.json +++ b/nextflow_schema.json @@ -481,7 +481,7 @@ "type": "string", "format": "directory-path", "description": "SigProfilerMatrixGenerator volume directory containing tsb// (the installed genome payload).", - "help_text": "Create it once with `--download_sigprofiler_genome` (published to `/cache/sigprofiler/volume`) or with `SigProfilerMatrixGenerator install --volume `, then pass it here on later runs. The volume is checked against the pipeline's SigProfilerMatrixGenerator checksums before any sample is processed. GRCh38 and CHM13-T2T volumes installed before lrsomatic PR #216 are a superseded payload revision and must be reinstalled.", + "help_text": "Create it once with `--download_sigprofiler_genome` (published to `/cache/sigprofiler/volume`) or with `SigProfilerMatrixGenerator install --volume `, then pass it here on later runs. The volume is checked against the pipeline's SigProfilerMatrixGenerator checksums before any sample is processed. The run stops if files are missing or a checksum differs, which means a corrupted copy or, for GRCh38 and CHM13-T2T volumes installed for lrsomatic < 1.2.0, a superseded payload revision; reinstall it in either case.", "fa_icon": "fas fa-folder-open" }, "download_sigprofiler_genome": { diff --git a/subworkflows/local/tests/prepare_signatures.nf.test b/subworkflows/local/tests/prepare_signatures.nf.test new file mode 100644 index 00000000..3ea7f829 --- /dev/null +++ b/subworkflows/local/tests/prepare_signatures.nf.test @@ -0,0 +1,73 @@ +nextflow_workflow { + + name "Test Workflow PREPARE_SIGNATURES" + script "../prepare_signatures.nf" + workflow "PREPARE_SIGNATURES" + + tag "subworkflows" + tag "subworkflows_local" + tag "prepare_signatures" + tag "sigprofiler" + // "small" is what .github/workflows/nf-test.yml selects on for pull_request + tag "small" + + // The pipeline tests all set skip_signatures, so these are the only PR coverage of this subworkflow. + // Run under -stub: what is tested is which branch runs and what it hands on as the volume. + + test("--sigprofiler_genome_dir - the user's volume is verified and handed on") { + + options "-stub" + + when { + workflow { + """ + input[0] = 'CHM13-T2T' + input[1] = null + input[2] = "\${projectDir}/tests/fixtures/sigprofiler_stub_volume" + input[3] = false + """ + } + } + + then { + def processes = workflow.trace.tasks().collect { task -> task.name } + + assertAll( + { assert workflow.success }, + { assert processes.any { name -> name.contains('SIGPROFILER_VERIFY') } }, + { assert !processes.any { name -> name.contains('SIGPROFILER_INSTALL') } }, + // the directory itself, not a staged copy + { assert workflow.out.volume.size() == 1 }, + { assert workflow.out.volume[0].toString().endsWith('tests/fixtures/sigprofiler_stub_volume') } + ) + } + } + + test("--download_sigprofiler_genome - the payload is installed") { + + options "-stub" + + when { + workflow { + """ + input[0] = 'CHM13-T2T' + input[1] = 'https://example.org/CHM13-T2T.tar.gz' + input[2] = null + input[3] = true + """ + } + } + + then { + def processes = workflow.trace.tasks().collect { task -> task.name } + + assertAll( + { assert workflow.success }, + { assert processes.any { name -> name.contains('SIGPROFILER_INSTALL') } }, + { assert !processes.any { name -> name.contains('SIGPROFILER_VERIFY') } }, + { assert workflow.out.volume.size() == 1 }, + { assert workflow.out.volume[0].toString().endsWith('volume') } + ) + } + } +} diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/1.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/1.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/10.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/10.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/11.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/11.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/12.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/12.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/13.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/13.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/14.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/14.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/15.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/15.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/16.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/16.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/17.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/17.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/18.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/18.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/19.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/19.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/2.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/2.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/20.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/20.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/21.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/21.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/22.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/22.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/3.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/3.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/4.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/4.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/5.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/5.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/6.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/6.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/7.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/7.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/8.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/8.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/9.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/9.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/X.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/X.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/Y.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/Y.txt new file mode 100644 index 00000000..e69de29b