Repository navigation
Record each raw data file's format at registration, so mzTab-M ms_run-format survives the raw data - #835
Merged
Conversation
…t tells it without reading the file
mzTab-M names each run's file format by looking at the disk at export time, so
once the raw data are deleted a re-export writes "Unknown .raw format" or
"Unknown .d folder format" for a format MS-DIAL knew when it processed the file.
This adds the place to keep what was known.
- RawFileFormat (MsdialCore/Enum): one value per format the exporter tells
apart, persisted by number. Numbers are explicit and append-only; Unknown is
0.
- AnalysisFileBean.RawFileFormat as MessagePack key 19, without an initializer,
so a project written before it reads as Unknown. Key 19 is unused on master,
on every remote branch and on the 5.5/5.6 tags.
- RawFileFormatClassifier (MsdialCore/Utility, no UI dependency):
- Classify(path) tells the format by extension, in any letter case. A .raw is
a Waters folder or a Thermo file. A .d is checked in RawDataHandler's order,
AcqData first, then analysis.tdf, .tsf and .baf. It never throws: a path
that cannot be checked is Unknown.
- RecordIfUnknown(file) fills the field only when nothing is recorded.
- IsConsistentWith(format, extension) and Resolve(file): the recorded format
when it fits the path, otherwise what the disk tells now.
Tests pin every extension and its letter cases, the .raw and .d shapes
(including a .d holding both AcqData and a Bruker file), unusable paths, the
enum numbers, a round trip of every value, that the 2025-07-31 fixture project
(written with 19 keys) reads as Unknown with keys 0-18 unchanged, and that a
reader declaring only keys 0-18 reads a bean with key 19, alone and inside a
list followed by more data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t for older projects when the raw data are found The format is recorded wherever MS-DIAL registers a file while it is surely on disk: - GUI: DatasetFileSettingModel.SetFiles, when a project is created. - Console: AnalysisFilesParser, in both bean factories, which covers the folder, CSV and single-file inputs. - GUI: AnalysisFileBeanModel.AnalysisFilePath, when a sample is pointed at a different file. The previous file's format no longer describes it, so the path is classified again. MsdialDataStorage.FixDatasetFolder, a folder move, still leaves the recorded format alone. A project saved before formats were recorded learns them lazily. Every disk check in AnalysisFileBeanModel (IsRawDataAvailable, RefreshRawDataAvailability, which DatasetModel.OpenMethodAsync runs over every file when a dataset opens) goes through one helper. When the raw data are present it records the format if none is recorded yet; the value is kept when the project is next saved. MarkRawDataMissing does not classify, and a recorded format is never replaced by a refresh. DataAccess.LoadMeasurement is deliberately not hooked: the 5.6 line deletes it. No Model class gains a UI or ViewModel reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rvives the raw data SetRawFileMetadataDic now describes each run by, in order: 1. the format recorded on the AnalysisFileBean, when it fits the file's extension; 2. otherwise what RawFileFormatClassifier tells from the disk now, which is the check the switch used to make inline and covers projects saved before formats were recorded; 3. otherwise the same Unknown user terms as before: "Unknown .raw format", "Unknown .d folder format", "Unknown file format". The terms move unchanged into MsRunFormatTerms(format, extension), including the nativeID terms that the "MS-DIAL set Datapoint Number" id_format overrides. AnalysisFileExtention, and with it the EDAM CV declaration for .CDF, is untouched. imzML still has no term here. The term corrections for WIFF2, imzML and netCDF are left for a separate change. ms_run-format is byte-identical for every file present on disk, with one exception: a .d holding both AcqData and a Bruker analysis file is now Agilent's, the format RawDataHandler reads it as. It used to be described by the Bruker file. A differential run of the old switch against the new path (74 present and missing paths, every extension, letter case and .raw/.d shape) found no other difference. Tests: - a recorded format is written when the raw data are gone, checked against the PSI-MS file-format children (psi-ms.obo 4.2.2); - a recorded format that does not fit the path is ignored; - every PSI-MS term the exporter can write is a file format; - the .d holding both kinds of file; - end to end: record while the raw data are present, save the project, delete the raw data, reload, export, and assert the terms. Exporting creates no path. The existing .d and .raw tests keep their expectations; their summaries now say they describe the case where nothing is recorded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at only when the disk cannot tell Resolve preferred a recorded format over the disk whenever it fitted the extension. A format is recorded once and never replaced, so if another file of the same extension later takes the recorded one's place (a Waters .raw folder where a Thermo .raw file was, or a Bruker .d where an Agilent one was), ms_run-format named the old format for raw data that are still on disk. Master names what is there, so this changed output for a file that is present, which the change was not to do. Resolve now asks the disk first and falls back to the recorded format only when the disk cannot tell, that is when the raw data are deleted or moved (or a vendor folder no longer holds its marker file). A recorded format that does not fit the extension is still ignored. Output for every file on disk is again what the classifier tells. The one difference from before formats were recorded is still the .d holding both AcqData and a Bruker analysis file, which is now Agilent's. The recorded format is what makes the difference once the raw data are gone, as before. Tests: - Resolve describes a .raw or .d on disk by its contents whatever was recorded, in both directions, in any letter case, and leaves the record untouched; - an emptied .d folder is described by its record; - an export describes a Waters folder recorded as Thermo, and a Thermo file recorded as Waters, by what is on disk. The new cases fail against the previous Resolve. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
htsugawa
pushed a commit
that referenced
this pull request
Oct 15, 2026
The test guards against the old freeze, in which reading the missing raw data waited 5 s before each of its retries (BaseDataProvider). Two seconds also failed a cold CI runner that took 2.8 s to open the sample without any retry (PR #835's PR Check). Below 5 s still fails any retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
mzTab-M export names each run's file format (
ms_run[n]-format) by looking at the disk at export time (MztabFormatExport.SetRawFileMetadataDic)..rawbecomes[,, Unknown .raw format, ]. This branch dates from Open and browse an MS-DIAL 5 project whose raw data files are missing #829 (browsing without raw data); before Open and browse an MS-DIAL 5 project whose raw data files are missing #829 the export threw..dhas become[,, Unknown .d folder format, ]since Console: AIF collision energies from mzML activation, multi-CE AIF alignment, and nearest-apex AIF seed scans #825.So once the raw data are deleted, which the GUI now supports and the public-repository reanalysis does, a re-export loses a format MS-DIAL knew perfectly well when it processed the file.
Two clarifications:
What changes
Recording the format
RawFileFormatenum (append-only, numbered) andAnalysisFileBean.RawFileFormatat MessagePack Key 19. Keys 0–18 are taken; 19 is free on master and on every remote branch, the 5.6 line included.RawFileFormatClassifiernames the format from the path without reading the file. It never throws..raw: a directory is Waters, a file is Thermo..d: checked in RawDataHandler's own order —AcqData(Agilent), thenanalysis.tdf,analysis.tsf,analysis.baf(Bruker).DatasetFileSettingModel);AnalysisFilesParser);AnalysisFileBeanModel).AnalysisFileBeanModel's availability check).MarkRawDataMissingnever touches it.Export.
SetRawFileMetadataDicresolves the format in this order:For every file present on disk the output is byte-identical to master. The one exception is a
.dthat holds bothAcqDataandanalysis.*files: it now follows RawDataHandler's order, AcqData first.id_formatandspectra_refare unchanged.Compatibility
Tests
.rawas a file, a folder and missing;.dwith each marker, empty, missing and AcqData+tdf.AnalysisFileBean: a MessagePack round trip; an old-project fixture reads Unknown; older-reader compatibility; pinned enum numbers.For the reviewers
ChromatogramPeakFeatureKeys 54/55 in Lipid RT-order check: assign Confirmed to lipid annotations whose retention times follow the reversed-phase rules #832).🤖 Generated with Claude Code