Skip to content

Record each raw data file's format at registration, so mzTab-M ms_run-format survives the raw data - #835

Merged
htsugawa merged 5 commits into
masterfrom
feature/record-raw-file-format
Oct 16, 2026
Merged

htsugawa merged 5 commits into
masterfrom
feature/record-raw-file-format

Conversation

@htsugawa

Copy link
Copy Markdown
Contributor

Why

mzTab-M export names each run's file format (ms_run[n]-format) by looking at the disk at export time (MztabFormatExport.SetRawFileMetadataDic).

So once the raw data are deleted, which the GUI now supports and the public-repository reanalysis does, a re-export loses a format MS-DIAL knew perfectly well when it processed the file.

Two clarifications:

What changes

Recording the format

  • New RawFileFormat enum (append-only, numbered) and AnalysisFileBean.RawFileFormat at MessagePack Key 19. Keys 0–18 are taken; 19 is free on master and on every remote branch, the 5.6 line included.
  • RawFileFormatClassifier names the format from the path without reading the file. It never throws.
    • .raw: a directory is Waters, a file is Thermo.
    • .d: checked in RawDataHandler's own order — AcqData (Agilent), then analysis.tdf, analysis.tsf, analysis.baf (Bruker).
    • Other extensions: decided by extension, case-insensitive.
  • The format is recorded where a file is registered:
    • GUI dataset setting (DatasetFileSettingModel);
    • both Console bean factories (AnalysisFilesParser);
    • a GUI path change (AnalysisFileBeanModel).
  • Older projects learn their format lazily, the first time their raw data are found (AnalysisFileBeanModel's availability check). MarkRawDataMissing never touches it.

Export. SetRawFileMetadataDic resolves the format in this order:

  1. what is on disk, as before;
  2. the recorded format, when the disk cannot tell and the record fits the extension;
  3. today's Unknown strings.

For every file present on disk the output is byte-identical to master. The one exception is a .d that holds both AcqData and analysis.* files: it now follows RawDataHandler's order, AcqData first. id_format and spectra_ref are unchanged.

Compatibility

  • Projects written before this change read as Unknown.
  • A reader without Key 19 opens projects written with it (tested with a keys-0–18 mirror class).

Tests

  • Classifier: every extension and its case variants; .raw as a file, a folder and missing; .d with each marker, empty, missing and AcqData+tdf.
  • AnalysisFileBean: a MessagePack round trip; an old-project fixture reads Unknown; older-reader compatibility; pinned enum numbers.
  • Export with the raw data gone: each recorded format gives its PSI-MS term, checked against an allow-list of MS:1000560 children. A record that does not fit the extension is ignored, and a file on disk is described by what is on disk whatever was recorded.
  • End to end: register, save, delete the raw data, reload, export, then assert the terms.
  • Recording: Console parser recording; GUI model backfill, path change and the missing-file case.
  • Results: whole-solution build and test in "Debug vendor unsupported", as PR Check runs them, are green apart from the GUI test that Dispose a model's items even when its collection is cleared first #833 fixes.

For the reviewers

  • Key 19: please reserve it on the 5.6 line, so the next merge does not collide (we hit one such clash on ChromatogramPeakFeature Keys 54/55 in Lipid RT-order check: assign Confirmed to lipid annotations whose retention times follow the reversed-phase rules #832).
  • Older builds: a 5.5 build or a 5.6 beta that re-saves a project written here drops Key 19. The format is then learned again the next time the raw data are present.
  • Deliberately not in this PR: PSI-MS term corrections — WIFF2 should be MS:1003448, imzML MS:1003611, and netCDF MS:1002441 rather than EDAM. Those change output for present files and are left to a separate PR.

🤖 Generated with Claude Code

DESKTOP-382ETUR\Hiroshi Tsugawa and others added 4 commits October 16, 2026 00:48
…t tells it without reading the file

mzTab-M names each run's file format by looking at the disk at export time, so
once the raw data are deleted a re-export writes "Unknown .raw format" or
"Unknown .d folder format" for a format MS-DIAL knew when it processed the file.
This adds the place to keep what was known.

- RawFileFormat (MsdialCore/Enum): one value per format the exporter tells
  apart, persisted by number. Numbers are explicit and append-only; Unknown is
  0.
- AnalysisFileBean.RawFileFormat as MessagePack key 19, without an initializer,
  so a project written before it reads as Unknown. Key 19 is unused on master,
  on every remote branch and on the 5.5/5.6 tags.
- RawFileFormatClassifier (MsdialCore/Utility, no UI dependency):
  - Classify(path) tells the format by extension, in any letter case. A .raw is
    a Waters folder or a Thermo file. A .d is checked in RawDataHandler's order,
    AcqData first, then analysis.tdf, .tsf and .baf. It never throws: a path
    that cannot be checked is Unknown.
  - RecordIfUnknown(file) fills the field only when nothing is recorded.
  - IsConsistentWith(format, extension) and Resolve(file): the recorded format
    when it fits the path, otherwise what the disk tells now.

Tests pin every extension and its letter cases, the .raw and .d shapes
(including a .d holding both AcqData and a Bruker file), unusable paths, the
enum numbers, a round trip of every value, that the 2025-07-31 fixture project
(written with 19 keys) reads as Unknown with keys 0-18 unchanged, and that a
reader declaring only keys 0-18 reads a bean with key 19, alone and inside a
list followed by more data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t for older projects when the raw data are found

The format is recorded wherever MS-DIAL registers a file while it is surely on
disk:
- GUI: DatasetFileSettingModel.SetFiles, when a project is created.
- Console: AnalysisFilesParser, in both bean factories, which covers the folder,
  CSV and single-file inputs.
- GUI: AnalysisFileBeanModel.AnalysisFilePath, when a sample is pointed at a
  different file. The previous file's format no longer describes it, so the path
  is classified again. MsdialDataStorage.FixDatasetFolder, a folder move, still
  leaves the recorded format alone.

A project saved before formats were recorded learns them lazily. Every disk
check in AnalysisFileBeanModel (IsRawDataAvailable, RefreshRawDataAvailability,
which DatasetModel.OpenMethodAsync runs over every file when a dataset opens)
goes through one helper. When the raw data are present it records the format if
none is recorded yet; the value is kept when the project is next saved.
MarkRawDataMissing does not classify, and a recorded format is never replaced
by a refresh.

DataAccess.LoadMeasurement is deliberately not hooked: the 5.6 line deletes it.
No Model class gains a UI or ViewModel reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rvives the raw data

SetRawFileMetadataDic now describes each run by, in order:
1. the format recorded on the AnalysisFileBean, when it fits the file's
   extension;
2. otherwise what RawFileFormatClassifier tells from the disk now, which is the
   check the switch used to make inline and covers projects saved before
   formats were recorded;
3. otherwise the same Unknown user terms as before: "Unknown .raw format",
   "Unknown .d folder format", "Unknown file format".

The terms move unchanged into MsRunFormatTerms(format, extension), including
the nativeID terms that the "MS-DIAL set Datapoint Number" id_format overrides.
AnalysisFileExtention, and with it the EDAM CV declaration for .CDF, is
untouched. imzML still has no term here. The term corrections for WIFF2, imzML
and netCDF are left for a separate change.

ms_run-format is byte-identical for every file present on disk, with one
exception: a .d holding both AcqData and a Bruker analysis file is now
Agilent's, the format RawDataHandler reads it as. It used to be described by
the Bruker file. A differential run of the old switch against the new path
(74 present and missing paths, every extension, letter case and .raw/.d shape)
found no other difference.

Tests:
- a recorded format is written when the raw data are gone, checked against
  the PSI-MS file-format children (psi-ms.obo 4.2.2);
- a recorded format that does not fit the path is ignored;
- every PSI-MS term the exporter can write is a file format;
- the .d holding both kinds of file;
- end to end: record while the raw data are present, save the project, delete
  the raw data, reload, export, and assert the terms. Exporting creates no path.
The existing .d and .raw tests keep their expectations; their summaries now say
they describe the case where nothing is recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at only when the disk cannot tell

Resolve preferred a recorded format over the disk whenever it fitted the
extension. A format is recorded once and never replaced, so if another file
of the same extension later takes the recorded one's place (a Waters .raw
folder where a Thermo .raw file was, or a Bruker .d where an Agilent one
was), ms_run-format named the old format for raw data that are still on
disk. Master names what is there, so this changed output for a file that is
present, which the change was not to do.

Resolve now asks the disk first and falls back to the recorded format only
when the disk cannot tell, that is when the raw data are deleted or moved
(or a vendor folder no longer holds its marker file). A recorded format that
does not fit the extension is still ignored. Output for every file on disk
is again what the classifier tells. The one difference from before formats
were recorded is still the .d holding both AcqData and a Bruker analysis
file, which is now Agilent's. The recorded format is what makes the
difference once the raw data are gone, as before.

Tests:
- Resolve describes a .raw or .d on disk by its contents whatever was
  recorded, in both directions, in any letter case, and leaves the record
  untouched;
- an emptied .d folder is described by its record;
- an export describes a Waters folder recorded as Thermo, and a Thermo file
  recorded as Waters, by what is on disk.
The new cases fail against the previous Resolve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
htsugawa pushed a commit that referenced this pull request Oct 15, 2026
The test guards against the old freeze, in which reading the missing raw
data waited 5 s before each of its retries (BaseDataProvider). Two
seconds also failed a cold CI runner that took 2.8 s to open the sample
without any retry (PR #835's PR Check). Below 5 s still fails any retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@htsugawa
htsugawa merged commit 2910533 into master Oct 16, 2026
1 check passed
@htsugawa
htsugawa deleted the feature/record-raw-file-format branch October 16, 2026 00:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant