Fix/matrixgenerator review bugs

.github/workflows/release.yml

@@ -29,6 +29,10 @@ jobs:
       run: |
         python -m pip install --upgrade pip build twine
 
+    - name: Clean previous build outputs
+      run: |
+        rm -rf build dist *.egg-info
+
     - name: Build distributions
       run: |
         python -m build
@@ -37,6 +41,10 @@ jobs:
       run: |
         python -m twine check dist/*
 
+    - name: Verify GRCh38 reference resources
+      run: |
+        python -c "import glob, zipfile; wheel=glob.glob('dist/*.whl')[0]; names=[name.rsplit('/', 1)[-1] for name in zipfile.ZipFile(wheel).namelist()]; assert not any('GRCh38_TSBv2' in name for name in names); assert sum(name.startswith(('context_counts_GRCh38_Legacy_', 'context_distribution_GRCh38_Legacy_')) for name in names) == 48; assert sum(name.startswith(('context_counts_GRCh38_', 'context_distribution_GRCh38_')) and 'GRCh38_Legacy_' not in name for name in names) == 48"
+
     - name: Verify package version
       run: |
         python -m pip install dist/*.whl

CHANGELOG.md

@@ -6,6 +6,49 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ## [Unreleased]
 
+### Added
+- Register `GRCh38_Legacy` with the chromosome checksums and context tables from
+  the previously distributed GRCh38 reference so historical results remain
+  reproducible.
+
+### Changed
+- Promote the corrected transcription-strand reference to the default `GRCh38`
+  identity, including validated strand-aware whole-genome and exome context-count
+  and distribution tables. Existing installations of the former `GRCh38`
+  reference must be reinstalled after upgrading.
+
+### Fixed
+- Rebuild SBS context-distribution tables from the shared five-base opportunity
+  set, preserve valid binary chromosome bytes, include the final valid window,
+  and reverse T/U labels when purine contexts are canonicalized.
+- Preserve full reference IDs during input conversion and in execution logs;
+  explicitly map existing Havana editions to their shared assembly resources.
+- Distinguish missing, incomplete, unregistered, and checksum-mismatched
+  references in matrix-generation errors without modifying installed files.
+- Supply required conversion arguments for `.genome` text input.
+- Corrected TSB reference generation at inclusive transcript ends and across
+  nested or overlapping transcripts on either strand.
+- Read SBS strand labels at the mutation's own position and reject positions
+  without the required left flanking sequence instead of wrapping around.
+- Compare WGS and exome regression matrices against the correct expected files,
+  and fail when required comparisons are missing. The mode-forwarding correction
+  was also contributed by Luuk Harbers in PR #250.
+- Read combined and headerless transcript annotations without dropping the first
+  record. Correctly combine non-adjacent records for the same gene in the internal
+  gene-range reader; the public gene-strand analysis option remains unsupported.
+- Use the registered reference chromosome list for VCF, text, and MAF conversion
+  instead of inferring chromosomes from transcript filenames.
+- Preserve the first BED data row after a header, and fix region comparisons
+  when the lower bound after applying the cushion is zero.
+- Classify FACETS total-copy-number-one segments in existing LOH channels,
+  retaining the CNV48 schema (from `fix/facets-cnv48-tcn1-loh`).
+
+### Reference Data
+- `GRCh38.tar.gz` now contains the corrected transcription-strand reference.
+  The previously distributed data is preserved as `GRCh38_Legacy.tar.gz`.
+  Both archives must be published under these exact filenames before network
+  installation can succeed.
+
 ## [1.3.6] - 2025-10-28
 
 ### Added

README.md

@@ -83,7 +83,7 @@ View the table below for the full list of parameters.
 | ------ | ----------- | ----------- | ----------- |
 | Required |  |  |  |
 |  | project | String | The name of the project. |
-|  | reference_genome | String | The name of the reference genome. Full list of genomes under **Supported Genomes** section. Supported values include the following: {c_elegans, dog, ebv, GRCh37, GRCh38, mm9, mm10, mm39, rn6, yeast} |
+|  | reference_genome | String | The name of the reference genome. Full list of genomes under **Supported Genomes** section. Supported values include the following: {c_elegans, dog, ebv, GRCh37, GRCh38, GRCh38_Legacy, mm9, mm10, mm39, rn6, yeast} |
 |  | path_to_input_files | String | The path to the input files. |
 | Optional |  |  |  |
 |  | exome | Boolean | Downsamples mutational matrices to the exome regions of the genome. Default value False. |
@@ -205,7 +205,32 @@ SigProfilerMatrixGenerator cnv_matrix_generator BATTENBERG ./SigProfilerMatrixGe
 This tool currently supports the following genomes:
 
 GRCh38.p12 [GRCh38] (Genome Reference Consortium Human Reference 38), INSDC
-Assembly GCA_000001405.27, Dec 2013. Released July 2014. Last updated January 2018. This genome was downloaded from ENSEMBL database version 93.38.
+Assembly GCA_000001405.27, Dec 2013. Released July 2014. Last updated January
+2018. The DNA sequence was downloaded from ENSEMBL database version 93.38.
+The current `GRCh38` reference includes corrected transcription-strand
+annotation and should be used for new analyses.
+
+GRCh38 legacy TSB reference [GRCh38_Legacy] contains the GRCh38 reference data
+distributed before the transcription-strand correction. It is retained only
+to reproduce results from earlier SigProfilerMatrixGenerator releases. New
+analyses should use `GRCh38`. Because the corrected reference is now the
+default, users with an older `GRCh38` installation must reinstall `GRCh38`
+after upgrading. To reproduce an old analysis, install and select
+`GRCh38_Legacy` explicitly.
+
+```python
+from SigProfilerMatrixGenerator import install as genInstall
+
+# Corrected reference for new analyses
+genInstall.install("GRCh38")
+
+# Former reference data, only for reproducing historical results
+genInstall.install("GRCh38_Legacy")
+```
+
+The migration can change transcription-strand-aware matrices. Sequence-only
+matrices such as SBS96 and ID83 are expected to remain unchanged because both
+references use the same GRCh38 DNA sequence.
 
 GRCh37.p13 [GRCh37] (Genome Reference Consortium Human Reference 37), INSDC
 Assembly GCA_000001405.14, Feb 2009. Released April 2011. Last updated September 2013. This genome was downloaded from ENSEMBL database version 93.37.

SigProfilerMatrixGenerator/controllers/cli_controller.py

@@ -53,7 +53,7 @@ def parse_arguments_install(args: List[str]) -> argparse.Namespace:
     parser = argparse.ArgumentParser(description="Install reference genome files.")
     parser.add_argument(
         "genome",
-        help="The reference genome to install. Supported genomes include {c_elegans, dog, ebv, GRCh37, GRCh38, mm9, mm10, mm39, rn6, rn7, yeast}.",
+        help="The reference genome to install. Supported genomes include {c_elegans, dog, ebv, GRCh37, GRCh38, GRCh38_Legacy, mm9, mm10, mm39, rn6, rn7, yeast}.",
     )
     parser.add_argument(
         "-l",
@@ -86,7 +86,7 @@ def parse_arguments_matrix_generator(args: List[str]) -> argparse.Namespace:
     parser.add_argument("project", help="The name of the project.")
     parser.add_argument(
         "reference_genome",
-        help="The name of the reference genome. Supported values {c_elegans, dog, ebv, GRCh37, GRCh38, mm9, mm10, mm39, rn6, rn7, yeast}.",
+        help="The name of the reference genome. Supported values {c_elegans, dog, ebv, GRCh37, GRCh38, GRCh38_Legacy, mm9, mm10, mm39, rn6, rn7, yeast}.",
     )
     parser.add_argument("path_to_input_files", help="The path to the input files.")
 

SigProfilerMatrixGenerator/references/chromosomes/context_distributions/context_counts_GRCh38_24.csv

@@ -1,9 +1,9 @@
 ,X,Y,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22
-B:C,255408,0,375437,324146,567775,247956,191661,359946,260800,176163,169261,249014,304234,235099,72087,312098,253930,336877,287644,49111,292373,51053,202140,131280
-B:T,354936,0,485071,434996,808560,377562,236516,486697,373875,224758,214770,334198,369705,299766,111477,384254,360383,375444,303406,66244,294161,55502,268571,117065
-N:C,42149067,8389614,50691663,56326378,40325299,45396614,44159738,39254022,34343656,34703593,29365628,29488929,29621087,27903104,25186668,20846472,17938686,18672539,17136435,20022290,12104549,15703601,10442223,9266873
-N:T,65497013,12950140,72639325,84614752,63049305,74584966,69188527,60682243,50775737,52290341,43302932,41899721,43417609,41510412,40907694,31055592,25016303,24811850,21766001,30942562,14557564,20834720,15861662,11048875
-T:C,9282289,209766,21304543,18855781,18168588,13619869,13577861,14099305,14442872,12051044,10775464,12804487,12799551,12713698,5822488,6966714,8821784,7146882,10520708,5578025,7700698,5006392,3155623,4501307
-T:T,13587113,342446,28973960,28179036,26373541,21475021,20217053,21153535,20491512,17421277,14901363,17768124,17341026,18165736,8980428,9600608,11932037,8187997,12124326,8153935,7752335,5928609,4246082,4876408
-U:C,9534697,663263,23794652,21262712,19516039,13303535,13681950,13646721,15649482,11203141,9960041,12816924,13160160,13423532,6691515,8857471,8564415,10316385,9630560,6206579,7918080,7249400,2611542,4507296
-U:T,14232375,1080926,32215719,30550292,29290916,20747076,20011928,20395993,22632121,16697775,13100911,17901252,17520306,18886353,10210683,12544844,11753715,11957885,11150936,9070638,7820966,9114624,3300572,4710481
+B:C,258726,0,398597,368653,647393,308788,271645,409289,282975,193180,173812,291418,354028,292883,83186,338700,277173,356052,325015,63534,350873,63465,227070,135855
+B:T,358930,0,509541,494847,914151,465863,357585,555114,403160,246970,219345,383596,436949,373972,131122,412938,394500,392127,344070,85976,353299,70864,297083,119910
+N:C,42148788,8389605,50690936,56325981,40324978,45396376,44159482,39253700,34343332,34703361,29365328,29488656,29620662,27902775,25186555,20846276,17938476,18672298,17136049,20022192,12104046,15703406,10442148,9266698
+N:T,65496542,12950105,72638264,84614076,63048759,74584537,69188051,60681696,50775294,52289976,43302557,41899359,43416968,41509905,40907523,31055321,25016027,24811444,21765441,30942424,14556897,20834453,15861549,11048682
+T:C,9551794,446351,23039741,20493588,19205426,13718909,13875390,14156076,15352313,11868813,10600694,13105001,13214469,13316656,6411817,8085129,8888547,8891536,10251629,6011082,7883722,6243673,2929143,4599679
+T:T,13249592,666501,29214450,28012083,26583257,20197923,19172983,19872784,20572314,16309697,13308735,16896889,16674540,17679894,9166947,10496883,11258472,9594480,11090790,8225394,7408379,7175699,3578235,4534999
+U:C,9262153,426687,22037021,19580795,18399904,13143901,13304693,13540929,14718190,11368587,10130560,12474279,12695873,12763119,6091200,7712650,8474619,8552797,9862654,5759197,7677059,5999902,2813167,4404524
+U:T,14566373,756906,31951820,30658070,28976155,21936302,20935405,21608874,22522477,17787508,14689339,18723451,18120189,19298496,10004690,11620156,12393439,10535125,12144368,8979585,8106451,7852439,3940020,5049238

SigProfilerMatrixGenerator/references/chromosomes/context_distributions/context_counts_GRCh38_24_exome.csv

@@ -1,9 +1,9 @@
 ,X,Y,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,MT,M
-B:C,26464,0,45302,35227,45678,17563,20300,39640,25505,18214,17571,21337,47544,27187,6785,35347,15779,47675,46540,9784,41308,10334,15807,13073,0,0
-B:T,27664,0,42937,36274,49429,18783,18455,37684,24008,15704,16434,19927,43451,24471,8420,34236,17685,41981,39795,9434,31819,8047,14172,9218,0,0
-N:C,25538,15521,80220,52038,18044,28344,18199,20529,83277,17291,37887,42449,35668,36862,12438,48114,54590,79918,48888,6044,25307,9169,7795,38463,0,0
-N:T,22050,19928,75915,57065,19397,25438,21019,21015,79816,15943,41435,39635,38043,43003,13527,46282,49961,67606,50211,6650,21925,9348,9775,30206,0,0
-T:C,404782,17206,1149754,791651,643272,369893,428080,552843,521002,401268,467347,412603,662001,597517,186297,347115,419909,505921,787770,151980,886571,291599,113632,279769,0,0
-T:T,413824,24382,1127106,936887,691191,469946,468448,607714,524561,401098,448808,437022,605376,624535,229841,355162,429299,407589,675212,180546,654770,242839,111280,210795,0,0
-U:C,432110,26421,1196325,801641,609448,424331,558771,535447,558925,352131,478900,452966,717046,585213,189522,336568,381914,600472,675090,178967,863711,305232,117007,276794,0,0
-U:T,454213,34995,1185429,892660,659278,516352,642999,570068,563512,391241,450348,488496,654546,628137,239352,360484,383134,465736,558665,211176,628833,263271,109850,208717,0,0
+B:C,26558,0,47581,39500,50112,17767,21834,42876,27887,20600,18532,24165,50499,31750,8249,37364,17885,51048,51863,10897,51821,10756,16407,13925,0,0
+B:T,27713,0,44846,42218,54213,19155,20040,41049,26067,17801,17097,22575,47026,29867,10497,35959,20042,44010,43815,10946,40057,8520,15053,9644,0,0
+N:C,25521,15520,80166,52013,18029,28329,18182,20499,83255,17275,37858,42434,35631,36841,12432,48098,54570,79901,48851,6041,25255,9155,7790,38451,0,0
+N:T,22028,19926,75844,57039,19373,25401,21003,20967,79784,15911,41393,39606,37991,42975,13520,46274,49947,67581,50166,6643,21866,9329,9768,30190,0,0
+T:C,423828,22503,1181317,810876,631022,407284,501322,551208,537969,382550,474617,437070,680819,593972,191766,346578,407821,548401,727369,168792,853259,297953,117055,275653,0,0
+T:T,440306,28782,1157433,918718,673175,497420,558140,593012,541798,396250,451267,466117,620954,621489,236404,358421,410268,433534,616918,196957,649886,254363,109448,209376,0,0
+U:C,412987,21125,1162537,778168,617279,386751,484012,533876,539598,368479,470698,425686,695310,584216,182595,335104,391916,554636,730205,161045,886562,298470,112989,280070,0,0
+U:T,427704,30597,1153264,904911,672534,488543,551738,581453,544248,394024,447268,456782,635445,625815,230719,355510,399822,437787,612984,193260,625538,251293,110808,209726,0,0