diff --git a/CHANGELOG.md b/CHANGELOG.md index d398e769..5b301bf5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### `Fixed` +- [#216](https://github.com/IntGenomicsLab/lrsomatic/pull/216) - `--download_sigprofiler_genome` failed on CHM13 with `CHM13-T2T failed the SigProfilerMatrixGenerator checksum verification`. The published payload had been replaced by the archive rebuilt after [SigProfilerMatrixGenerator#251](https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator/issues/251), which counts transcript-end bases correctly, while the image still carried the old checksums. The SigProfiler modules now use `ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689` (`oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689`), and the CHM13 `sigprofiler_genome_url` points at the versioned `chm13_release_2026-09` payload. The new SigProfilerMatrixGenerator also includes upstream's corrected GRCh38 payload, which the AlexandrovLab FTP has served as `GRCh38.tar.gz` since 2026-09-25; the old one is now `GRCh38_Legacy`. **GRCh38 and CHM13-T2T volumes installed before this change fail verification and must be reinstalled.** A volume passed with `--sigprofiler_genome_dir` is now checked once, up front, by the new `SIGPROFILER_VERIFY` step, which stops with reinstall instructions instead of failing in every `SIGPROFILER_MATRIXGENERATOR` task. Its error names the cause: missing chromosome files, a checksum mismatch (stale or corrupted payload), or a genome the image has no checksums for. Stub nf-tests (tag `small`) now cover `PREPARE_SIGNATURES`, the volume reaching every `SIGPROFILER_MATRIXGENERATOR` sample, and the pinned image in `SIGPROFILER_MATRIXGENERATOR` and `SIGPROFILER_ASSIGNMENT`. Transcribed-strand matrices (SBS288, SBS384) change slightly; SBS96 and the COSMIC fits are not expected to change (@ljwharbers). - [#206](https://github.com/IntGenomicsLab/lrsomatic/pull/206) - A remote (http, https or ftp) ClinVar is now downloaded once per run by the new `VEPPLUGIN_CLINVAR` step in `PREPARE_VEP_PLUGINS`, instead of being staged by Nextflow as a foreign file; local and cloud-storage paths are staged as before. `GERMLINE_VEP` and `SOMATIC_VEP` re-checked the foreign file on its host for every sample, and NCBI answered the burst from a multi-sample GRCh38 run with HTTP 503, so `SOMATIC_VEP` failed with `Can't stage file ...clinvar_20260829.vcf.gz`; `-resume` could not recover, since the failed check changed the staging cache key. The download is checked against the new `--vep_clinvar_md5` and `--vep_clinvar_tbi_md5`, set by default to the checksums NCBI (GRCh38, VCF only) and Ensembl (CHM13, VCF and index) publish, so the pinned release cannot change silently. Resuming a run that already finished re-runs `GERMLINE_VEP` and `SOMATIC_VEP` once, since ClinVar now comes from a task rather than the stage cache. The ClinVar sizes in `docs/usage.md` are also corrected, and `docs/output.md` now documents `vep_plugins/` (@AmberVerhasselt). - [#203](https://github.com/IntGenomicsLab/lrsomatic/pull/203) - `CLAIRS` no longer runs with `--haplotagged_tumor_bam_provided_so_skip_intermediate_phasing_and_haplotagging`. Since somatic calling was moved ahead of `PHASING_HAPLOTYPING` (v1.1.0), ClairS has received the untagged minimap2 BAM, so the flag made it skip its own phasing and haplotagging and call every paired sample without haplotype information: the full-alignment model saw no `HP` tags and the haplotype filtering step had nothing to filter on, the same as `--disable_phasing`. ClairS now runs Clair3 on the normal and tumour BAMs and phases the tumour itself. **Paired somatic calls change** (fewer false positives expected), and `CLAIRS` takes longer and uses more work-directory space (@ljwharbers). - [#203](https://github.com/IntGenomicsLab/lrsomatic/pull/203) - `docs/output.md` now lists the ClairS SNV output as `snvs.vcf.gz`, the name the pipeline publishes, instead of `snv.vcf.gz` (@ljwharbers). diff --git a/conf/igenomes.config b/conf/igenomes.config index 430e65a2..5a7971c5 100644 --- a/conf/igenomes.config +++ b/conf/igenomes.config @@ -69,7 +69,7 @@ params.genomes = [ // SigProfilerMatrixGenerator CHM13-T2T payload (SigProfilerSuite/SigProfilerMatrixGenerator#250); // not yet hosted on the AlexandrovLab FTP, so it is fetched from the IntGenomicsLab Globus collection sigprofiler_genome : "CHM13-T2T", - sigprofiler_genome_url : "https://g-608c0c.273595.03c0.data.globus.org/chm13_release_2026-08/CHM13-T2T.tar.gz", + sigprofiler_genome_url : "https://g-608c0c.273595.03c0.data.globus.org/chm13_release_2026-09/CHM13-T2T.tar.gz", gnomad : "${params.igenomes_base}/Homo_sapiens/ClairSTO/CHM13/Annotation/ClairSTO-pon/final_gnomad.vcf.gz", dbsnp : "${params.igenomes_base}/Homo_sapiens/ClairSTO/CHM13/Annotation/ClairSTO-pon/final_dbsnp.vcf.gz", onekgenomes : "${params.igenomes_base}/Homo_sapiens/ClairSTO/CHM13/Annotation/ClairSTO-pon/final_1kgenomes.vcf.gz", diff --git a/conf/modules.config b/conf/modules.config index 59200256..279a016e 100644 --- a/conf/modules.config +++ b/conf/modules.config @@ -795,6 +795,11 @@ process { ] } + withName: 'SIGPROFILER_VERIFY' { + // Passes the user's volume through unchanged; never publish a copy of it + publishDir = [ enabled: false ] + } + withName : '.*:SIGNATURES_BCFTOOLS_VIEW' { // SigProfilerMatrixGenerator ignores FILTER and reads plain-text VCF only ext.args = { "--apply-filters PASS --types snps,mnps,indels --exclude 'ALT=\"*\"' --output-type v" } diff --git a/docs/usage.md b/docs/usage.md index 1119930a..1760510d 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -380,6 +380,11 @@ Mutational signature analysis runs [SigProfilerMatrixGenerator](https://github.c Running with neither, and without `--skip_signatures`, stops the pipeline at start-up. +A volume passed with `--sigprofiler_genome_dir` is checked once per run against the chromosome checksums of the pipeline's SigProfilerMatrixGenerator (`SIGPROFILER_VERIFY`). If a chromosome file is missing or a checksum differs, the run stops before any sample is processed, and the error says which of the two it found. A genome the image has no checksums for is rejected too. + +> [!WARNING] +> GRCh38 and CHM13-T2T payloads installed before this release no longer pass that check. SigProfilerMatrixGenerator corrected how both are encoded, and the payloads it now downloads differ from the earlier ones. Reinstall once with `--download_sigprofiler_genome`, or with `SigProfilerMatrixGenerator install --volume ` from the same image, and pass the new volume on later runs. Mutation counts and COSMIC fits are unaffected. Only the strand-split matrices (for example SBS288 and SBS384) change, slightly. + | Parameter | Description | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--sigprofiler_genome_dir` | Full path to a SigProfilerMatrixGenerator volume containing `tsb//`. Default = `null` | diff --git a/modules/local/sigprofiler/assignment/main.nf b/modules/local/sigprofiler/assignment/main.nf index d66c1777..2af5430d 100644 --- a/modules/local/sigprofiler/assignment/main.nf +++ b/modules/local/sigprofiler/assignment/main.nf @@ -4,8 +4,8 @@ process SIGPROFILER_ASSIGNMENT { // No conda: the image uses CHM13-T2T forks of SigProfilerMatrixGenerator (#250) and SigProfilerAssignment; see meta.yml container "${(workflow.containerEngine == 'singularity' || workflow.containerEngine == 'apptainer') && !task.ext.singularity_pull_docker_container - ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-28a9ce8' - : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-28a9ce8'}" + ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689' + : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689'}" input: tuple val(meta), path(sbs96), path(dbs78), path(id83) // SigProfilerMatrixGenerator matrices; dbs78/id83 may be [] diff --git a/modules/local/sigprofiler/assignment/tests/main.nf.test b/modules/local/sigprofiler/assignment/tests/main.nf.test new file mode 100644 index 00000000..ee178d3c --- /dev/null +++ b/modules/local/sigprofiler/assignment/tests/main.nf.test @@ -0,0 +1,40 @@ +nextflow_process { + + name "Test Process SIGPROFILER_ASSIGNMENT" + script "../main.nf" + process "SIGPROFILER_ASSIGNMENT" + + tag "modules" + tag "modules_local" + tag "sigprofiler" + tag "sigprofiler_assignment" + tag "small" + + // Under -stub; the versions eval still runs in the pinned image, so its tag must resolve under each profile + + test("stub") { + + options "-stub" + + when { + process { + """ + // any file stands in for the SBS96 matrix; the stub does not read it + def matrix = file("\${projectDir}/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/1.txt", checkIfExists: true) + input[0] = [ [ id:'sample1' ], matrix, [], [] ] + input[1] = 'CHM13-T2T' + input[2] = 3.6 + """ + } + } + + then { + assertAll( + { assert process.success }, + { assert process.out.activities.size() == 1 }, + { assert process.out.versions_sigprofilerassignment.size() == 1 }, + { assert process.out.versions_sigprofilerassignment[0][2] ==~ /\d+\.\d+.*/ } + ) + } + } +} diff --git a/modules/local/sigprofiler/install/main.nf b/modules/local/sigprofiler/install/main.nf index 710ecf57..4bc17beb 100644 --- a/modules/local/sigprofiler/install/main.nf +++ b/modules/local/sigprofiler/install/main.nf @@ -5,8 +5,8 @@ process SIGPROFILER_INSTALL { // No conda: the image uses CHM13-T2T forks of SigProfilerMatrixGenerator (#250) and SigProfilerAssignment; see meta.yml container "${(workflow.containerEngine == 'singularity' || workflow.containerEngine == 'apptainer') && !task.ext.singularity_pull_docker_container - ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-28a9ce8' - : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-28a9ce8'}" + ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689' + : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689'}" input: val(genome) // SigProfilerMatrixGenerator genome name, e.g. GRCh38 or CHM13-T2T diff --git a/modules/local/sigprofiler/matrixgenerator/main.nf b/modules/local/sigprofiler/matrixgenerator/main.nf index 934d6d9a..b1ea9dd1 100644 --- a/modules/local/sigprofiler/matrixgenerator/main.nf +++ b/modules/local/sigprofiler/matrixgenerator/main.nf @@ -4,8 +4,8 @@ process SIGPROFILER_MATRIXGENERATOR { // No conda: the image uses CHM13-T2T forks of SigProfilerMatrixGenerator (#250) and SigProfilerAssignment; see meta.yml container "${(workflow.containerEngine == 'singularity' || workflow.containerEngine == 'apptainer') && !task.ext.singularity_pull_docker_container - ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-28a9ce8' - : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-28a9ce8'}" + ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689' + : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689'}" input: tuple val(meta), path(vcf) // somatic small-variant VCF (plain or bgzipped) diff --git a/modules/local/sigprofiler/matrixgenerator/tests/main.nf.test b/modules/local/sigprofiler/matrixgenerator/tests/main.nf.test new file mode 100644 index 00000000..564d3230 --- /dev/null +++ b/modules/local/sigprofiler/matrixgenerator/tests/main.nf.test @@ -0,0 +1,55 @@ +nextflow_process { + + name "Test Process SIGPROFILER_MATRIXGENERATOR" + script "../main.nf" + process "SIGPROFILER_MATRIXGENERATOR" + + tag "modules" + tag "modules_local" + tag "sigprofiler" + tag "sigprofiler_matrixgenerator" + tag "small" + + // No real payload fits in CI (~3 GB per genome), so this runs under -stub. The volume comes from + // PREPARE_SIGNATURES and is wired as in workflows/lrsomatic.nf, so every sample must receive it. + // The versions eval still runs in the pinned image, so its tag must resolve under each profile. + + test("stub - the verified volume reaches every sample") { + + options "-stub" + + setup { + run("PREPARE_SIGNATURES") { + script "../../../../../subworkflows/local/prepare_signatures.nf" + workflow { + """ + input[0] = 'CHM13-T2T' + input[1] = null + input[2] = "\${projectDir}/tests/fixtures/sigprofiler_stub_volume" + input[3] = false + """ + } + } + } + + when { + process { + """ + def vcf = file("\${projectDir}/tests/fixtures/vcfsplit_indel.vcf", checkIfExists: true) + input[0] = channel.of([ [ id:'sample1' ], vcf ], [ [ id:'sample2' ], vcf ]) + input[1] = PREPARE_SIGNATURES.out.volume.map { volume -> [ [:], volume ] } + input[2] = 'CHM13-T2T' + """ + } + } + + then { + assertAll( + { assert process.success }, + { assert process.out.sbs96.collect { meta, _matrix -> meta.id }.sort() == ['sample1', 'sample2'] }, + { assert process.out.versions_sigprofilermatrixgenerator.size() == 2 }, + { assert process.out.versions_sigprofilermatrixgenerator.every { _proc, _tool, version -> version ==~ /\d+\.\d+.*/ } } + ) + } + } +} diff --git a/modules/local/sigprofiler/verify/main.nf b/modules/local/sigprofiler/verify/main.nf new file mode 100644 index 00000000..4bf86194 --- /dev/null +++ b/modules/local/sigprofiler/verify/main.nf @@ -0,0 +1,68 @@ +process SIGPROFILER_VERIFY { + tag "$genome" + label 'process_single' + + // No conda: the image uses CHM13-T2T forks of SigProfilerMatrixGenerator (#250) and SigProfilerAssignment; see meta.yml + container "${(workflow.containerEngine == 'singularity' || workflow.containerEngine == 'apptainer') && !task.ext.singularity_pull_docker_container + ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689' + : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689'}" + + input: + path(volume, stageAs: 'genome_volume') // SigProfilerMatrixGenerator volume containing tsb// + val(genome) // SigProfilerMatrixGenerator genome name, e.g. GRCh38 or CHM13-T2T + + output: + val(true) , emit: verified + tuple val("${task.process}"), val('sigprofilermatrixgenerator'), eval("python -c 'import importlib.metadata as m; print(m.version(\"SigProfilerMatrixGenerator\"))'"), topic: versions, emit: versions_sigprofilermatrixgenerator + + when: + task.ext.when == null || task.ext.when + + script: + if (workflow.profile.tokenize(',').intersect(['conda', 'mamba']).size() >= 1) { + error "SIGPROFILER_VERIFY does not support Conda. Please use Docker / Singularity / Apptainer instead." + } + """ + # SIGPROFILER_MATRIXGENERATOR re-checks the payload for every sample; checking once here fails a stale or damaged + # volume before any sample work, with the reinstall instructions instead of a per-sample checksum error + python - <<'PY' + import sys + from SigProfilerMatrixGenerator.scripts import reference_genome_manager as rgm + + genome = "${genome}" + if genome not in rgm.CHECKSUMS: + sys.exit( + f"ERROR: this pipeline's SigProfilerMatrixGenerator has no checksums for {genome} (registered: " + f"{', '.join(sorted(rgm.CHECKSUMS))}). Set --sigprofiler_genome to one of them or use --skip_signatures." + ) + + manager = rgm.ReferenceGenomeManager("genome_volume") + if not manager.is_genome_installed(genome): + manager.print_genome_checksum_verification_report(genome) + tsb = manager.reference_dir.get_tsb_dir() / genome + expected = rgm.CHECKSUMS[genome] + missing = [chrom for chrom in expected if not (tsb / f"{chrom}.txt").is_file()] + if missing: + cause = ( + f"is an incomplete install: {len(missing)} of {len(expected)} chromosome files are missing " + f"({', '.join(missing)})." + ) + else: + cause = ( + "does not match the checksums of this pipeline's SigProfilerMatrixGenerator. Either it is a stale " + "payload (GRCh38 and CHM13-T2T installed for lrsomatic < 1.2.0 are a superseded revision) or the " + "copy is corrupted." + ) + sys.exit( + f"ERROR: the {genome} payload in --sigprofiler_genome_dir {cause} Reinstall it with " + "--download_sigprofiler_genome (published to /cache/sigprofiler/volume) and pass that directory " + "on later runs." + ) + PY + """ + + stub: + """ + echo "stub: skipping checksum verification of genome_volume/tsb/${genome}" + """ +} diff --git a/modules/local/sigprofiler/verify/meta.yml b/modules/local/sigprofiler/verify/meta.yml new file mode 100644 index 00000000..586b7f39 --- /dev/null +++ b/modules/local/sigprofiler/verify/meta.yml @@ -0,0 +1,58 @@ +--- +# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/modules/meta-schema.json +name: "sigprofiler_verify" +description: Check a user-supplied SigProfilerMatrixGenerator volume against the chromosome checksums of the pipeline's SigProfilerMatrixGenerator, so a stale or damaged payload fails before any sample is processed +keywords: + - mutational signatures + - reference genome + - sigprofiler + - checksum +tools: + - "sigprofilermatrixgenerator": + description: "SigProfilerMatrixGenerator creates mutational matrices for all types of somatic mutations" + homepage: "https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator" + documentation: "https://sigprofilersuite.github.io/SigProfilerMatrixGenerator/" + tool_dev_url: "https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator" + doi: "10.1186/s12864-019-6041-2" + licence: ["BSD-2-Clause"] + +input: + - volume: + type: directory + description: SigProfilerMatrixGenerator volume containing tsb// (e.g. from --sigprofiler_genome_dir) + - genome: + type: string + description: SigProfilerMatrixGenerator genome name, e.g. GRCh38 or CHM13-T2T + +output: + verified: + - "true": + type: boolean + description: Emitted once the volume's chromosome files match the registered checksums; the task fails otherwise + versions_sigprofilermatrixgenerator: + - - ${task.process}: + type: string + description: The process the versions were collected from + - sigprofilermatrixgenerator: + type: string + description: The tool name + - 'python -c ''import importlib.metadata as m; print(m.version("SigProfilerMatrixGenerator"))''': + type: string + description: The command used to generate the version of the tool + +topics: + versions: + - - ${task.process}: + type: string + description: The process the versions were collected from + - sigprofilermatrixgenerator: + type: string + description: The tool name + - 'python -c ''import importlib.metadata as m; print(m.version("SigProfilerMatrixGenerator"))''': + type: string + description: The command used to generate the version of the tool + +authors: + - "@ljwharbers" +maintainers: + - "@ljwharbers" diff --git a/modules/local/sigprofiler/verify/tests/main.nf.test b/modules/local/sigprofiler/verify/tests/main.nf.test new file mode 100644 index 00000000..cea6e43f --- /dev/null +++ b/modules/local/sigprofiler/verify/tests/main.nf.test @@ -0,0 +1,129 @@ +nextflow_process { + + name "Test Process SIGPROFILER_VERIFY" + script "../main.nf" + process "SIGPROFILER_VERIFY" + + tag "modules" + tag "modules_local" + tag "sigprofiler" + tag "sigprofiler_verify" + tag "small" + + // No real payload fits in CI (~3 GB per genome), so these tests check the rejection paths. The checksum report + // lists the chr1 MD5 the image expects, which ties the pinned image to the payload revision its URL serves: + // GRCh38 from the AlexandrovLab FTP since 2026-09-25, CHM13-T2T from chm13_release_2026-09. + + test("CHM13-T2T - incomplete volume is rejected with reinstall instructions") { + + when { + process { + """ + input[0] = file("\${projectDir}/tests/fixtures/sigprofiler_stale_volume", checkIfExists: true) + input[1] = 'CHM13-T2T' + """ + } + } + + then { + def report = (process.stdout + process.stderr).join('\n') + assertAll( + { assert process.failed }, + // rendered message only: the script source Nextflow echoes on failure has the template + { assert report.contains('X, Y). Reinstall it with --download_sigprofiler_genome') }, + // the fixture holds chromosome 1 only + { assert report.contains('incomplete install: 23 of 24 chromosome files are missing') }, + // T2T-CHM13v2.0 chr1 of the payload rebuilt after SigProfilerMatrixGenerator#251 + { assert report.contains('8683547ca6e5cbb2afd3633a32e6353a') } + ) + } + } + + test("GRCh38 - incomplete volume is rejected with reinstall instructions") { + + when { + process { + """ + input[0] = file("\${projectDir}/tests/fixtures/sigprofiler_stale_volume", checkIfExists: true) + input[1] = 'GRCh38' + """ + } + } + + then { + def report = (process.stdout + process.stderr).join('\n') + assertAll( + { assert process.failed }, + { assert report.contains('Y, MT). Reinstall it with --download_sigprofiler_genome') }, + // the fixture holds chromosome 1 only; GRCh38 also registers MT + { assert report.contains('incomplete install: 24 of 25 chromosome files are missing') }, + // GRCh38 chr1 of upstream's corrected payload (the pre-2026-09-25 one is GRCh38_Legacy) + { assert report.contains('570ba2c0c11b999a906abd4f854a38be') } + ) + } + } + + test("CHM13-T2T - complete volume with wrong checksums is rejected as stale or corrupted") { + + when { + process { + """ + // all 24 chromosome files, but empty + input[0] = file("\${projectDir}/tests/fixtures/sigprofiler_stub_volume", checkIfExists: true) + input[1] = 'CHM13-T2T' + """ + } + } + + then { + def report = (process.stdout + process.stderr).join('\n') + assertAll( + { assert process.failed }, + // rendered message only: the script source Nextflow echoes on failure has the template + { assert report.contains('CHM13-T2T payload in --sigprofiler_genome_dir does not match the checksums') }, + { assert !(report =~ /incomplete install: \d/) }, + { assert report.contains('copy is corrupted. Reinstall it with --download_sigprofiler_genome') } + ) + } + } + + test("genome without checksums in the image is rejected") { + + when { + process { + """ + input[0] = file("\${projectDir}/tests/fixtures/sigprofiler_stale_volume", checkIfExists: true) + input[1] = 'GRCh37_unregistered' + """ + } + } + + then { + def report = (process.stdout + process.stderr).join('\n') + assertAll( + { assert process.failed }, + { assert report.contains('has no checksums for GRCh37_unregistered') }, + { assert report.contains('(registered: CHM13-T2T, GRCh37,') } + ) + } + } + + test("stub") { + + options "-stub" + + when { + process { + """ + input[0] = file("\${projectDir}/tests/fixtures/sigprofiler_stale_volume", checkIfExists: true) + input[1] = 'CHM13-T2T' + """ + } + } + + then { + assert process.success + assert process.out.verified == [true] + } + } +} diff --git a/nextflow_schema.json b/nextflow_schema.json index 2a5cf13b..ceb7b12f 100644 --- a/nextflow_schema.json +++ b/nextflow_schema.json @@ -481,7 +481,7 @@ "type": "string", "format": "directory-path", "description": "SigProfilerMatrixGenerator volume directory containing tsb// (the installed genome payload).", - "help_text": "Create it once with `--download_sigprofiler_genome` (published to `/cache/sigprofiler/volume`) or with `SigProfilerMatrixGenerator install --volume `, then pass it here on later runs.", + "help_text": "Create it once with `--download_sigprofiler_genome` (published to `/cache/sigprofiler/volume`) or with `SigProfilerMatrixGenerator install --volume `, then pass it here on later runs. The volume is checked against the pipeline's SigProfilerMatrixGenerator checksums before any sample is processed. The run stops if files are missing or a checksum differs, which means a corrupted copy or, for GRCh38 and CHM13-T2T volumes installed for lrsomatic < 1.2.0, a superseded payload revision; reinstall it in either case.", "fa_icon": "fas fa-folder-open" }, "download_sigprofiler_genome": { diff --git a/subworkflows/local/prepare_signatures.nf b/subworkflows/local/prepare_signatures.nf index e27eee87..4255b336 100644 --- a/subworkflows/local/prepare_signatures.nf +++ b/subworkflows/local/prepare_signatures.nf @@ -1,4 +1,5 @@ include { SIGPROFILER_INSTALL } from '../../modules/local/sigprofiler/install/main' +include { SIGPROFILER_VERIFY } from '../../modules/local/sigprofiler/verify/main' workflow PREPARE_SIGNATURES { @@ -39,9 +40,18 @@ workflow PREPARE_SIGNATURES { if (n_chrom < 24) { error("${tsb_dir} holds ${n_chrom} of the 24 chromosome files of a complete ${genome} install.") } - sigprofiler_volume = channel.value(file(genome_dir, type: 'dir', checkIfExists: true)) + // + // MODULE: SIGPROFILER_VERIFY (label: process_single) -- checksums once, so a stale payload fails before any sample + // + def volume_dir = file(genome_dir, type: 'dir', checkIfExists: true) + SIGPROFILER_VERIFY ( + volume_dir, + genome + ) + // Hand on the user's directory itself, released only once it has been verified + sigprofiler_volume = SIGPROFILER_VERIFY.out.verified.map { _verified -> volume_dir } } - // sigprofiler_volume: path -- SigProfilerMatrixGenerator volume root (downloaded or validated local) + // sigprofiler_volume: path -- SigProfilerMatrixGenerator volume root (downloaded or verified local) emit: volume = sigprofiler_volume // path -- volume directory containing tsb// diff --git a/subworkflows/local/tests/prepare_signatures.nf.test b/subworkflows/local/tests/prepare_signatures.nf.test new file mode 100644 index 00000000..3ea7f829 --- /dev/null +++ b/subworkflows/local/tests/prepare_signatures.nf.test @@ -0,0 +1,73 @@ +nextflow_workflow { + + name "Test Workflow PREPARE_SIGNATURES" + script "../prepare_signatures.nf" + workflow "PREPARE_SIGNATURES" + + tag "subworkflows" + tag "subworkflows_local" + tag "prepare_signatures" + tag "sigprofiler" + // "small" is what .github/workflows/nf-test.yml selects on for pull_request + tag "small" + + // The pipeline tests all set skip_signatures, so these are the only PR coverage of this subworkflow. + // Run under -stub: what is tested is which branch runs and what it hands on as the volume. + + test("--sigprofiler_genome_dir - the user's volume is verified and handed on") { + + options "-stub" + + when { + workflow { + """ + input[0] = 'CHM13-T2T' + input[1] = null + input[2] = "\${projectDir}/tests/fixtures/sigprofiler_stub_volume" + input[3] = false + """ + } + } + + then { + def processes = workflow.trace.tasks().collect { task -> task.name } + + assertAll( + { assert workflow.success }, + { assert processes.any { name -> name.contains('SIGPROFILER_VERIFY') } }, + { assert !processes.any { name -> name.contains('SIGPROFILER_INSTALL') } }, + // the directory itself, not a staged copy + { assert workflow.out.volume.size() == 1 }, + { assert workflow.out.volume[0].toString().endsWith('tests/fixtures/sigprofiler_stub_volume') } + ) + } + } + + test("--download_sigprofiler_genome - the payload is installed") { + + options "-stub" + + when { + workflow { + """ + input[0] = 'CHM13-T2T' + input[1] = 'https://example.org/CHM13-T2T.tar.gz' + input[2] = null + input[3] = true + """ + } + } + + then { + def processes = workflow.trace.tasks().collect { task -> task.name } + + assertAll( + { assert workflow.success }, + { assert processes.any { name -> name.contains('SIGPROFILER_INSTALL') } }, + { assert !processes.any { name -> name.contains('SIGPROFILER_VERIFY') } }, + { assert workflow.out.volume.size() == 1 }, + { assert workflow.out.volume[0].toString().endsWith('volume') } + ) + } + } +} diff --git a/tests/fixtures/sigprofiler_stale_volume/tsb/CHM13-T2T/1.txt b/tests/fixtures/sigprofiler_stale_volume/tsb/CHM13-T2T/1.txt new file mode 100644 index 00000000..47b63767 --- /dev/null +++ b/tests/fixtures/sigprofiler_stale_volume/tsb/CHM13-T2T/1.txt @@ -0,0 +1 @@ +stale payload stand-in for SIGPROFILER_VERIFY tests diff --git a/tests/fixtures/sigprofiler_stale_volume/tsb/GRCh38/1.txt b/tests/fixtures/sigprofiler_stale_volume/tsb/GRCh38/1.txt new file mode 100644 index 00000000..47b63767 --- /dev/null +++ b/tests/fixtures/sigprofiler_stale_volume/tsb/GRCh38/1.txt @@ -0,0 +1 @@ +stale payload stand-in for SIGPROFILER_VERIFY tests diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/1.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/1.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/10.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/10.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/11.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/11.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/12.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/12.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/13.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/13.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/14.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/14.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/15.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/15.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/16.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/16.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/17.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/17.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/18.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/18.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/19.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/19.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/2.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/2.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/20.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/20.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/21.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/21.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/22.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/22.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/3.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/3.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/4.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/4.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/5.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/5.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/6.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/6.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/7.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/7.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/8.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/8.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/9.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/9.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/X.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/X.txt new file mode 100644 index 00000000..e69de29b diff --git a/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/Y.txt b/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/Y.txt new file mode 100644 index 00000000..e69de29b