Move SigProfiler to the corrected CHM13-T2T payload (SPMG 7894689) #216
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| nextflow_process { | ||
|
|
||
| name "Test Process SIGPROFILER_ASSIGNMENT" | ||
| script "../main.nf" | ||
| process "SIGPROFILER_ASSIGNMENT" | ||
|
|
||
| tag "modules" | ||
| tag "modules_local" | ||
| tag "sigprofiler" | ||
| tag "sigprofiler_assignment" | ||
| tag "small" | ||
|
|
||
| // Under -stub; the versions eval still runs in the pinned image, so its tag must resolve under each profile | ||
|
|
||
| test("stub") { | ||
|
|
||
| options "-stub" | ||
|
|
||
| when { | ||
| process { | ||
| """ | ||
| // any file stands in for the SBS96 matrix; the stub does not read it | ||
| def matrix = file("\${projectDir}/tests/fixtures/sigprofiler_stub_volume/tsb/CHM13-T2T/1.txt", checkIfExists: true) | ||
| input[0] = [ [ id:'sample1' ], matrix, [], [] ] | ||
| input[1] = 'CHM13-T2T' | ||
| input[2] = 3.6 | ||
| """ | ||
| } | ||
| } | ||
|
|
||
| then { | ||
| assertAll( | ||
| { assert process.success }, | ||
| { assert process.out.activities.size() == 1 }, | ||
| { assert process.out.versions_sigprofilerassignment.size() == 1 }, | ||
| { assert process.out.versions_sigprofilerassignment[0][2] ==~ /\d+\.\d+.*/ } | ||
| ) | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| nextflow_process { | ||
|
|
||
| name "Test Process SIGPROFILER_MATRIXGENERATOR" | ||
| script "../main.nf" | ||
| process "SIGPROFILER_MATRIXGENERATOR" | ||
|
|
||
| tag "modules" | ||
| tag "modules_local" | ||
| tag "sigprofiler" | ||
| tag "sigprofiler_matrixgenerator" | ||
| tag "small" | ||
|
|
||
| // No real payload fits in CI (~3 GB per genome), so this runs under -stub. The volume comes from | ||
| // PREPARE_SIGNATURES and is wired as in workflows/lrsomatic.nf, so every sample must receive it. | ||
| // The versions eval still runs in the pinned image, so its tag must resolve under each profile. | ||
|
|
||
| test("stub - the verified volume reaches every sample") { | ||
|
|
||
| options "-stub" | ||
|
|
||
| setup { | ||
| run("PREPARE_SIGNATURES") { | ||
| script "../../../../../subworkflows/local/prepare_signatures.nf" | ||
| workflow { | ||
| """ | ||
| input[0] = 'CHM13-T2T' | ||
| input[1] = null | ||
| input[2] = "\${projectDir}/tests/fixtures/sigprofiler_stub_volume" | ||
| input[3] = false | ||
| """ | ||
| } | ||
| } | ||
| } | ||
|
|
||
| when { | ||
| process { | ||
| """ | ||
| def vcf = file("\${projectDir}/tests/fixtures/vcfsplit_indel.vcf", checkIfExists: true) | ||
| input[0] = channel.of([ [ id:'sample1' ], vcf ], [ [ id:'sample2' ], vcf ]) | ||
| input[1] = PREPARE_SIGNATURES.out.volume.map { volume -> [ [:], volume ] } | ||
| input[2] = 'CHM13-T2T' | ||
| """ | ||
| } | ||
| } | ||
|
|
||
| then { | ||
| assertAll( | ||
| { assert process.success }, | ||
| { assert process.out.sbs96.collect { meta, _matrix -> meta.id }.sort() == ['sample1', 'sample2'] }, | ||
| { assert process.out.versions_sigprofilermatrixgenerator.size() == 2 }, | ||
| { assert process.out.versions_sigprofilermatrixgenerator.every { _proc, _tool, version -> version ==~ /\d+\.\d+.*/ } } | ||
| ) | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,68 @@ | ||||||||||
| process SIGPROFILER_VERIFY { | ||||||||||
| tag "$genome" | ||||||||||
| label 'process_single' | ||||||||||
|
|
||||||||||
| // No conda: the image uses CHM13-T2T forks of SigProfilerMatrixGenerator (#250) and SigProfilerAssignment; see meta.yml | ||||||||||
| container "${(workflow.containerEngine == 'singularity' || workflow.containerEngine == 'apptainer') && !task.ext.singularity_pull_docker_container | ||||||||||
| ? 'oras://ghcr.io/ljwharbers/sigprofiler-sif:1.3.6-chm13-7894689' | ||||||||||
| : 'ghcr.io/ljwharbers/sigprofiler:1.3.6-chm13-7894689'}" | ||||||||||
|
There was a problem hiding this comment. Registry convention (still open from the first review). The new
Collaborator
Author
There was a problem hiding this comment. We won't address this in this PR. The SigProfiler image stays on ghcr ( |
||||||||||
|
|
||||||||||
| input: | ||||||||||
| path(volume, stageAs: 'genome_volume') // SigProfilerMatrixGenerator volume containing tsb/<genome>/ | ||||||||||
| val(genome) // SigProfilerMatrixGenerator genome name, e.g. GRCh38 or CHM13-T2T | ||||||||||
|
|
||||||||||
| output: | ||||||||||
| val(true) , emit: verified | ||||||||||
| tuple val("${task.process}"), val('sigprofilermatrixgenerator'), eval("python -c 'import importlib.metadata as m; print(m.version(\"SigProfilerMatrixGenerator\"))'"), topic: versions, emit: versions_sigprofilermatrixgenerator | ||||||||||
|
|
||||||||||
| when: | ||||||||||
| task.ext.when == null || task.ext.when | ||||||||||
|
|
||||||||||
| script: | ||||||||||
| if (workflow.profile.tokenize(',').intersect(['conda', 'mamba']).size() >= 1) { | ||||||||||
| error "SIGPROFILER_VERIFY does not support Conda. Please use Docker / Singularity / Apptainer instead." | ||||||||||
| } | ||||||||||
| """ | ||||||||||
| # SIGPROFILER_MATRIXGENERATOR re-checks the payload for every sample; checking once here fails a stale or damaged | ||||||||||
| # volume before any sample work, with the reinstall instructions instead of a per-sample checksum error | ||||||||||
| python - <<'PY' | ||||||||||
| import sys | ||||||||||
| from SigProfilerMatrixGenerator.scripts import reference_genome_manager as rgm | ||||||||||
|
|
||||||||||
| genome = "${genome}" | ||||||||||
| if genome not in rgm.CHECKSUMS: | ||||||||||
| sys.exit( | ||||||||||
| f"ERROR: this pipeline's SigProfilerMatrixGenerator has no checksums for {genome} (registered: " | ||||||||||
| f"{', '.join(sorted(rgm.CHECKSUMS))}). Set --sigprofiler_genome to one of them or use --skip_signatures." | ||||||||||
| ) | ||||||||||
|
|
||||||||||
| manager = rgm.ReferenceGenomeManager("genome_volume") | ||||||||||
| if not manager.is_genome_installed(genome): | ||||||||||
| manager.print_genome_checksum_verification_report(genome) | ||||||||||
| tsb = manager.reference_dir.get_tsb_dir() / genome | ||||||||||
| expected = rgm.CHECKSUMS[genome] | ||||||||||
| missing = [chrom for chrom in expected if not (tsb / f"{chrom}.txt").is_file()] | ||||||||||
| if missing: | ||||||||||
| cause = ( | ||||||||||
| f"is an incomplete install: {len(missing)} of {len(expected)} chromosome files are missing " | ||||||||||
| f"({', '.join(missing)})." | ||||||||||
| ) | ||||||||||
| else: | ||||||||||
| cause = ( | ||||||||||
| "does not match the checksums of this pipeline's SigProfilerMatrixGenerator. Either it is a stale " | ||||||||||
| "payload (GRCh38 and CHM13-T2T installed for lrsomatic < 1.2.0 are a superseded revision) or the " | ||||||||||
|
There was a problem hiding this comment. The error points at the wrong version. No released lrsomatic has a SigProfiler payload. Signatures came in with #190, under Stale volumes come from two places: The same wording is in the
Suggested change
|
||||||||||
| "copy is corrupted." | ||||||||||
| ) | ||||||||||
| sys.exit( | ||||||||||
| f"ERROR: the {genome} payload in --sigprofiler_genome_dir {cause} Reinstall it with " | ||||||||||
| "--download_sigprofiler_genome (published to <outdir>/cache/sigprofiler/volume) and pass that directory " | ||||||||||
| "on later runs." | ||||||||||
| ) | ||||||||||
| PY | ||||||||||
| """ | ||||||||||
|
|
||||||||||
| stub: | ||||||||||
| """ | ||||||||||
| echo "stub: skipping checksum verification of genome_volume/tsb/${genome}" | ||||||||||
| """ | ||||||||||
| } | ||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| --- | ||
| # yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/modules/meta-schema.json | ||
| name: "sigprofiler_verify" | ||
| description: Check a user-supplied SigProfilerMatrixGenerator volume against the chromosome checksums of the pipeline's SigProfilerMatrixGenerator, so a stale or damaged payload fails before any sample is processed | ||
| keywords: | ||
| - mutational signatures | ||
| - reference genome | ||
| - sigprofiler | ||
| - checksum | ||
| tools: | ||
| - "sigprofilermatrixgenerator": | ||
| description: "SigProfilerMatrixGenerator creates mutational matrices for all types of somatic mutations" | ||
| homepage: "https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator" | ||
| documentation: "https://sigprofilersuite.github.io/SigProfilerMatrixGenerator/" | ||
| tool_dev_url: "https://github.com/SigProfilerSuite/SigProfilerMatrixGenerator" | ||
| doi: "10.1186/s12864-019-6041-2" | ||
| licence: ["BSD-2-Clause"] | ||
|
|
||
| input: | ||
| - volume: | ||
| type: directory | ||
| description: SigProfilerMatrixGenerator volume containing tsb/<genome>/ (e.g. from --sigprofiler_genome_dir) | ||
| - genome: | ||
| type: string | ||
| description: SigProfilerMatrixGenerator genome name, e.g. GRCh38 or CHM13-T2T | ||
|
|
||
| output: | ||
| verified: | ||
| - "true": | ||
| type: boolean | ||
| description: Emitted once the volume's chromosome files match the registered checksums; the task fails otherwise | ||
| versions_sigprofilermatrixgenerator: | ||
| - - ${task.process}: | ||
| type: string | ||
| description: The process the versions were collected from | ||
| - sigprofilermatrixgenerator: | ||
| type: string | ||
| description: The tool name | ||
| - 'python -c ''import importlib.metadata as m; print(m.version("SigProfilerMatrixGenerator"))''': | ||
| type: string | ||
| description: The command used to generate the version of the tool | ||
|
|
||
| topics: | ||
| versions: | ||
| - - ${task.process}: | ||
| type: string | ||
| description: The process the versions were collected from | ||
| - sigprofilermatrixgenerator: | ||
| type: string | ||
| description: The tool name | ||
| - 'python -c ''import importlib.metadata as m; print(m.version("SigProfilerMatrixGenerator"))''': | ||
| type: string | ||
| description: The command used to generate the version of the tool | ||
|
|
||
| authors: | ||
| - "@ljwharbers" | ||
| maintainers: | ||
| - "@ljwharbers" |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No CI coverage: No
smallnf-test runs this code. The test profile setsskip_signatures = true, and all tests are GRCh38 anyway. So CI checks neither the new URL, the new image tags, nor the image's embedded CHM13 checksums against this payload. Before merging, please run the Slurm test array or the standalone CHM13--download_sigprofiler_genomerun listed in the PR's to-do, plus the old-vs-new BL1 matrix/fit comparison. Please also run a GRCh38 signatures run on the new image: the image changed for every genome, not just CHM13, and nothing here tests the GRCh38 checksums in the rebuilt SPMG fork.