Skip to content

Lipid RT-order check: assign Confirmed to lipid annotations whose retention times follow the reversed-phase rules - #832

Merged
htsugawa merged 8 commits into
masterfrom
feature/lipid-rt-confirmation
Oct 16, 2026
Merged

htsugawa merged 8 commits into
masterfrom
feature/lipid-rt-confirmation

Conversation

@htsugawa

Copy link
Copy Markdown
Contributor

Why

Within one lipid class on reversed-phase LC:

  • more carbons elute later;
  • more double bonds elute earlier;
  • more oxygens elute earlier.

Curators use these rules by eye to reject false positives, and that manual pass is the bottleneck. It gets worse on very sensitive instruments: on ZenoTOF 8600 data, false positives are numerous even at a 0.5 min LBM2 RT tolerance. It is also worse for outside users, whose LC differs from the library's predicted RTs.

This PR lets MS-DIAL check lipid annotations against the rules and add the existing PeakSpotTag.CONFIRMED to the consistent ones:

  • on per-sample peaks (ChromatogramPeakFeature);
  • on alignment spots (AlignmentSpotProperty).

It never removes or overrides a person's tag.

The check (rule version 1.0)

  • Species level only. It compares the sum composition (class, total C, total DB, total O; P- counted as O- with one more DB). Fatty-acid composition is not used for RT. A class-level match is eligible, but by default it is Confirmed only when a second adduct or the other polarity corroborates it.

  • Exact order consistency. Within each class, the check finds the maximum-weight consistent subset of the dominance order, exactly, with an integer max-flow (Dilworth antichain). An annotation is Confirmed only if it belongs to every maximum subset and has adjacent directional support (a +C neighbour and a +DB neighbour bracketing it).

  • Rejection rules:

    • margins scale with the run's RT span;
    • a local magnitude check rejects an RT that sits too far from what its neighbours predict.
  • MS1 artefact layer. A brighter co-eluting partner that explains the annotation means it is not Confirmed. Covered: isotopes, Na/H, NH4/K, acetate/formate/Cl, demethylation, in-source chain or head losses, oxidised ions.

  • Propagation. A confirmed feature's other adducts ([M+Na]+ included) are Confirmed with it.

  • Chromatography gate. A positive carbon slope and order concordance are required. HILIC data get no Confirmed.

  • Optional chain-rule plausibility profile "Mammalian" (off by default):

    • no chain has DB ≥ 7;
    • a sphingolipid N-acyl chain has DB ≤ 2;
    • a sphingoid base has C16–20 and DB ≤ 2.

    It has no lipid-class bias, by design.

  • Outcomes. Each annotation gets a verdict: Confirmed, NotEvaluable (with a reason code) or Inconsistent (with evidence). Inconsistent is reported only; the check never writes Misannotation.

  • Cross-polarity corroboration (step 11) is implemented as a pure function with tests. Applying it from the GUI or Console is deferred to v1.1.

Evaluation (Python reference; the C# reproduces it object for object)

The truth sets are human curations. The primary held-out set is the author's own curation of ZT Scan DIA mouse liver, scored only on what was curated. Development used plasma.

Setting Liver (held out): precision / share of curated "confirmed" recovered
All candidates (null) 0.43 / 1.00
HighPrecision (default, class-level corroborated) 0.86 / 0.37
Strict 0.93 / 0.20
Confirmed in both polarities 0.98 / 0.13
  • Other sets: NIST SRM1950 NEG 0.95; a FastLC dataset 0.98 / 1.00.
  • Negative controls: three HILIC datasets get 0 Confirmed. MS1-realistic decoys (isotope, Na/H, in-source fragments) get 0 Confirmed.

Where it runs

  • Hook A: LC-MS FileProcess, after annotation and before save. Console and GUI share it.
  • Hook B: after alignment, after isotope tracking, before save and export.
  • GUI: a ribbon action, "Re-run lipid RT check", for LC-MS alignment results (the MS-CleanR pattern).
  • Parameters: LipidRtCheckParameter at ParameterBase Key 18. Console method-file keys start with Lipid RT check…; they are registered, templated and listed in ParametersAsText.
  • Outputs:
    • the Console writes <alignment>_lipidrt_review.tsv;
    • mzTab-M gets opt_global_lipid_rt_outcome, _reason and _score;
    • tags keep appearing as Tag: Confirmed in the Comment column.

Tag provenance (who set a tag) and compatibility

  • Provenance lives in the existing _tags.xml files, readable by people and external tools:

    • <Tag Source="MsdialLipidRtCheck/1.0">1</Tag> marks a tag set by the check;
    • <Suppressed Source="User">1</Suppressed> records that a person removed a tag the check had set.

    Older readers read only the <Tag> values, so they still see the same tags.

  • How the check treats tags:

    • it removes only its own tags before re-evaluating;
    • a person's tags stay the person's; GUI toggles, "Mark all as confirmed" and undo are routed through the person path;
    • a person's annotation change withdraws the check's tag on that object.
  • Verdict storage: the verdict is a MessagePack record:

    • AlignmentSpotProperty Key 66;
    • ChromatogramPeakFeature Key 56.

    ChromatogramPeakFeature keys 54 and 55 are reserved as integers for the MS-DIAL 5.6 line's RawDataIDType and AccumulatedDataIDType. Replace the placeholders with those members when the 5.6 line is merged.

  • Backward compatibility: old files load, and master's MsdialCore reads files written by this branch.

Verification

  • Builds (Release): MsdialCore, MsdialLcMsApi, the Console and MsdialGuiApp all build with 0 errors.

  • Tests:

    Project Result
    MsdialCoreTests 688 passed, including the reference-gated parity tests (run with MSDIAL_LIPIDRT_REFERENCE_DIR)
    MsdialCoreTestAppTests 194 passed
    MsdialLcMsApiTests 109 passed, 2 skipped (existing skips)
    MsdialGuiAppTests 179 passed, 1 failed

    The GUI failure is LcmsAnalysisModelWithoutRawTests.ASampleWithoutRawDataOpensFromWhatTheProjectStores. Its cleanup cannot delete a locked temporary .dcl, and it fails the same way before this PR's GUI commits.

  • Parity with the Python reference: 100% agreement, object by object (outcome, reason, score and evidence fields). Covered: demo POS/NEG alignments and per-sample peaks, ClassLevelPolicy Consistent and Corroborated, cross-polarity, and seven evaluation runs.

  • Console smoke runs on public verification data:

    • With Lipid RT check: False, or on a non-lipidomics method, outputs match master except for version strings, timestamps and paths.
    • Binary project files still differ by the new nil keys.

Decisions still open

  1. Default. The check is on by default for lipidomics projects (IsEnabled null means on for TargetOmics Lipidomics). An existing lipidomics method file therefore now gets verdicts, Confirmed tags, the review TSV and the mzTab-M columns. An opt-in default is a one-line change.
  2. Re-ticking. A person who removes the check's Confirmed, re-ticks it and removes it again currently lets a later run set it again. The alternative: any removal by a person blocks the check.

Deferred to v1.1: applying cross-polarity from the GUI or Console (MS-DIAL Interactive's lab pos/neg merge will call it), LC-IM-MS / GC-MS / DIMS hooks, and light-mode persistence.

🤖 Generated with Claude Code

DESKTOP-382ETUR\Hiroshi Tsugawa and others added 7 commits October 15, 2026 18:34
…erministic component

The check confirms a lipid annotation when its species-level composition (class,
total C, total DB with P- counted as O- plus one, total O) is consistent with the
reversed-phase rules among the run's other MS/MS-backed annotations of the class,
bracketed on two axes, and not explained as an MS1 artefact of a brighter
co-eluting ion (specification v1.0 with the addendum of 2026-10-15). It is a port
of the frozen Python reference (lipidrt_final, poset, xpol) and reads no files.

- CompMs.MsdialCore.Algorithm.Annotation.LipidRt: candidates from
  ChromatogramPeakFeature (hook A) and AlignmentSpotProperty (hook B) with the
  eligibility rule (IsReferenceMatched read from the representative directly;
  manual annotations, text-library matches and internal standards excluded);
  species composition via LipidomicsConverter VS2 on the species half, with the
  reference grammar for chain roles, sterol cores and the ester scale; the
  Mammalian profile with the chain rules B1-B3 only (no lipid-class bias);
  steps 1-10 (artefact layer, features, RP gate, exact maximum consistent subset
  by integer max-flow with warm-started margin flows, adjacent support and ties,
  class gate, magnitude, outcomes, propagation); step 11 as a pure function over
  two runs; evidence records, the evidence sentence and the review list.
- LipidRtCheckParameter (MessagePack, for ParameterBase Key 18, not wired yet):
  ClassLevelPolicy defaults to Corroborated, PlausibilityProfile to None,
  IsEnabled null meaning on for lipidomics projects.
- LipidRtVerdict, the persisted record for the integration step.

Tests: the 16 synthetic tests of the reference, policies and operating points,
parser and profile cases, the max-flow against brute force, determinism, and a
parity suite gated on MSDIAL_LIPIDRT_REFERENCE_DIR. On the 2026-10-15 demo the C#
reproduces the reference object by object for ClassLevelPolicy Consistent and
Corroborated (single run, profile off and Mammalian, and after cross-polarity),
and every outcome of the reference's DEV and TEST evaluation runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Integrates the lipid retention-time order check (specification v1.0 with the
addendum of 2026-10-15, sections 9-11 and B1-B3) into MS-DIAL: tag provenance in
the tag files, one tag protocol for every caller, the persisted verdict, the
parameter, hooks A and B, and the Console and mzTab-M outputs.

- Tag provenance (B1). PeakSpotTagCollection keeps an optional source per
  selected tag and a set of suppressed tags. Select/Deselect keep their meaning
  (a person's path); Select(tag, source) is a tool's; SelectByPerson and
  DeselectByPerson are what the GUI toggles now call, so turning off a tool's
  tag suppresses it and turning a tag on makes it the person's. The tag files
  (.pai2 peaks, alignment, spectrum features) are written and read by one
  helper, PeakSpotTagXml: a tool's tag gets a Source attribute, a suppression
  is a <Suppressed> element, and a peak with only suppressions is written. A
  tag without a source is written exactly as before, a file from before reads
  as tags with no source, and the previous reader reads a new file as the same
  selected tags (tested against that reader, copied verbatim).
- Tag protocol (SPEC 9 as re-expressed in B1). LipidRtTagProtocol withdraws
  only the check's own Confirmed (source "MsdialLipidRtCheck/", any rule
  version), records each object's verdict (null where there is none), and
  selects Confirmed with source "MsdialLipidRtCheck/1.0" only where the outcome
  is Confirmed, Confirmed is not already selected, Misannotation is not
  selected and Confirmed is not suppressed. It never writes Misannotation.
- Verdict record (B2). LipidRtVerdict on ChromatogramPeakFeature (key 54, the
  next after 53) and AlignmentSpotProperty (key 66, after the post-curation
  keys). Objects written before either key load with no verdict.
- Parameter. LipidRtCheckParameter is ParameterBase key 18 (DIMS uses 20, the
  other modes 150 and up); a project without it gets the defaults. RunsFor
  says the check runs only for lipidomics. The LC-MS parameter text gets a
  "# Lipid RT check" section with the method-file keys, the rule version and
  the parameter hash.
- Hooks. Hook A in the LC-MS FileProcess between annotation and saving, per
  file and stateless; hook B in PeakAligner right after PackingSpots, LC-MS
  only, and in the Console's alignment light mode, where the verdicts reach
  only the exports. Both run only when the check is enabled and the target
  omics is lipidomics; the RP gate decides inside. When the check does not run
  or fails, its own earlier tags and verdicts are withdrawn and the failure is
  reported, so nothing stale or half-written is left.
- Console. The method-file keys "Lipid RT check" (Auto/True/False), "... operating
  point", "... chromatography", "... plausibility profile", "... class-level
  matches", "... minor duplicates" and "... secondary-only features" are read
  by the LC-MS reader through LcmsConsoleLines, so the key record knows them
  and every other mode reports them as unrecognised; values that are not one
  of the choices are recorded as unusable. The LC-MS template writes them with
  their defaults. The Console writes <alignment stem>_lipidrt_review.tsv next
  to the alignment exports whenever the check ran, and a summary line per file
  and for the alignment.
- mzTab-M. opt_global_lipid_rt_outcome, _reason and _score after
  opt_global_Ontology when any spot has a verdict; "null" where a spot has
  none. Tags keep appearing as "Tag: Confirmed" in the Comment column.

Not in this change (v1.1 or a later step): the GUI "Re-run lipid RT check"
action, applying cross-polarity corroboration from the GUI or the Console, and
the LC-IM-MS, GC-MS and DIMS hooks.

Tests: tag collection, tag-file round trips and compatibility both ways, the
protocol and hooks on real peaks and spots (person's tags, another tool's
tags, Misannotation, suppression, re-runs, switched off, failure, save and
load), MessagePack backward compatibility of keys 54, 66 and 18, the mzTab-M
columns, the Comment column, the method-file keys, template and parameter
text, the review list file, and the GUI toggles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…through undo

Completes the GUI part of the lipid retention-time order check (specification
v1.0 with the addendum of 2026-10-15, B1 and B3).

- Re-run lipid RT check. A ribbon button on the Data visualization tab, next to
  MS-CleanR, opens a small window for the loaded LC-MS alignment result with the
  five user-facing settings (operating point, chromatography, plausibility
  profile, class-level matches, minor duplicates). Run writes them into the
  project's LipidRtCheckParameter, as MS-CleanR does with its own, runs the check
  on the result's spots through LipidRtCheckHooks.RunOnSpots and with it the
  shared tag protocol, tells the spots' views, and saves the alignment result
  (spots with their verdicts and the _tags.xml file) through the alignment
  model's SaveAsync. A person's tags, another tool's tags, a Misannotation and a
  suppressed Confirmed are never touched. The window is modal and stays open
  while the check runs, so no tag is edited by hand meanwhile. On a project that
  is not lipidomics the window says why and Run is disabled; a project whose
  check was switched off ("Lipid RT check: False") is told that running switches
  it back to Auto. A failed save is reported in the window, not thrown.
  Cross-polarity corroboration is not applied from the GUI (v1.1).
- Verdict and provenance in view. The peak information panel shows a "Lipid RT
  check:" row (the evidence sentence) wherever a peak or spot has a verdict, with
  the outcome, reason, score, rule version and parameter hash in its tooltip. The
  Confirmed tag checkbox of the peak and spot tables gets a tooltip saying who set
  it (the lipid RT check, another tool by its source, or a person) or that a
  person removed a tool's tag, and the verdict; with nothing to say the shortcut
  legend stays.
- Person path (B1). Every tag toggle of the GUI (checkboxes, Ctrl+1..5, Mark all
  as confirmed/unconfirmed) goes through the three models' setters, which already
  call SelectByPerson/DeselectByPerson. Undo of "Mark all" now puts each tag back
  exactly as it was, source and suppression included (PeakSpotTagState), instead
  of toggling back as a person: undoing "Mark all as unconfirmed" used to hand
  every Confirmed of the check to the person, and undoing "Mark all as confirmed"
  forgot what a person had removed. The three identical nested commands are one
  MarkAllAsConfirmedCommand<T> over IPeakSpotTagEditable.

Tests (MsdialGuiAppTests, net48): the setting model on a synthetic TG grid (same
tags and reasons as hook B on the same spots, settings written to the project,
person's tags and decisions kept, a re-run withdrawing only the check's own
tags, not lipidomics, switched off, failed save), the view model, Mark all and
its undo for spots and peaks, PeakSpotTagState, the tooltip texts, the
information panel, and the XAML loaded on a WPF dispatcher (the window's
bindings, the panel row, the Confirmed tooltip in the shared tag template, and
the ribbon icon).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e tracking, and withdraw the check where a person changes the annotation (review)

Applies the confirmed findings of the branch review.

- Peak verdict key (high). ChromatogramPeakFeature keys 54 and 55 are taken
  in the MS-DIAL 5.6 line (tags MSDIAL-v5.6.250526-alpha, -250820-alpha,
  -260323-beta and seven branches: RawDataIDType and AccumulatedDataIDType,
  a SpectrumIDType int enum). The verdict at 54 made 5.6 peak files
  unreadable here, made every peak file written here unreadable by 5.6, and
  would collide on a merge. The verdict moves to key 56, the next free one
  on every tag and remote branch (AlignmentSpotProperty 66 and ParameterBase
  18 are free everywhere). Keys 54 and 55 are now kept as plain integers,
  ReservedRawDataIDType and ReservedAccumulatedDataIDType: a 5.6 file keeps
  its values when read and saved here, and a file written here holds 0
  (Index, which is what this line's raw spectrum IDs are) where a 5.6 reader
  reads its enum. Without them a 5.6 reader would meet the nil MessagePack
  writes for a key with no member and fail, so moving the key alone would
  not have restored that direction.
- Hook B and isotope tracking (medium). PeakAligner ran the check before
  IsotopeTracking.SetIsotopeTrackingID, which clears the annotation of every
  spot represented by a labelled file, so such spots were saved and exported
  as unknowns carrying the check's Confirmed and a Confirmed verdict about a
  lipid they no longer were. The hook now runs after every step that
  changes annotations and reads the spot collection the tracking replaces.
- Manual annotation changes (medium). Compound search, MS-FINDER and "Set
  unknown" (SetAnnotationUsecase, SetUnknownDoCommand) now withdraw the
  check's own Confirmed and the verdict on the LC-MS peak or spot (addendum
  A6); a person's or another tool's Confirmed stays. Undo of "Set unknown"
  puts both back. ILipidRtCheckedObject carries this for
  AlignmentSpotPropertyModel and ChromatogramPeakFeatureModel.
- Evidence (low). The supporting neighbours kept in the evidence record are
  still listed by |gap| then direction, but the strongest lower and upper
  bound and the strongest of each direction are kept before the rest, so a
  Confirmed's sentence always shows both sides of its bracket (the review
  counted 48 of 1,274 demo Confirmed showing one side only). Only this free
  text differs from the Python reference; outcomes and scores do not.
- "Mark all as confirmed" (low). Rows whose Confirmed a tool had set are now
  taken over by the person (SelectByPerson drops the source in place), so a
  later run of the check no longer withdraws what the person confirmed. Undo
  gives them back to the check.
- Re-run lipid RT check (low). The run clears the alignment's undo history
  (new UndoManager.Clear), so undoing an earlier "Mark all" can no longer
  put back a Confirmed credited to the check against its new verdict. The
  save after the run goes off the UI thread, and the window says it is
  saving meanwhile.
- GUI processing (low). A failure of the check, on an analysis file (hook
  A, now recorded by FileProcess.LipidRtFailures) or on the alignment (hook
  B), is told to the person once after processing; before, it went only to
  a console the GUI does not show.
- Window text (low). The re-run window promised that a Confirmed a person
  removed is never set again. Under addendum B1 a suppression records the
  removal of a tool's tag only, so a Confirmed the person turned on again and
  then removed may be set again; the text now says so. The behaviour is
  unchanged.

Tests: the peak file in both directions with the 5.6 layout (a simulated
5.6 reader, and why nil at key 54 cannot be read there); PeakAligner with
and without isotope tracking; the evidence selection, end to end and as a
unit; takeover in place; the annotation changes and their undo for spots and
peaks; Mark all followed by the check's withdrawal; the undo history after a
re-run; the save off a UI-like synchronization context; the processing
notice; the window text. Each of these fails with its fix reverted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…publishes it and the selection error on the broker

LcmsMethodViewModel no longer constructs LipidRtCheckSettingView, calls
MessageBox.Show or ShowDialog, or reads Application.Current.MainWindow.
The command is a plain DelegateCommand (the ribbon passes no parameter).
With no alignment result selected it publishes an ErrorMessageBoxRequest,
whose defaults (caption "Error", OK, error icon) are the box shown before.
Otherwise it publishes the LipidRtCheckSettingViewModel, as ExportAlignment
does with its view model, and MainWindow shows it with ShowChildDialog:
modal and owned by the main window, so the view model is still disposed
only after the window has closed and the window still refuses to close
while the check runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e view for the error box it showed before

The view model now publishes the box on the broker instead of calling
MessageBox.Show, so pin what the view receives: the same text, the
"Error" caption and the OK button, with no setting window, when the
ribbon button (which passes no parameter) is pressed with no alignment
result selected. The method view model is built over an empty LC-MS
project on an application dispatcher thread, as the view tests do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@htsugawa

Copy link
Copy Markdown
Contributor Author

Follow-up after the lead developer's MVVM review: two commits now remove the ViewModel → View dependency this PR had added to LcmsMethodViewModel.

  • 87691ea: the "Re-run lipid RT check" command no longer creates the window, shows a MessageBox or uses Application.Current.MainWindow. It publishes ErrorMessageBoxRequest (no alignment selected) or the LipidRtCheckSettingViewModel on the broker, and MainWindow opens the window as a modal child dialog with ShowChildDialog. This is the pattern the authors use for AlignmentResultExportViewModel.
  • 4b40331: a test that, without an alignment result, the command asks the View for the same error box it showed before.

The Model side of this PR was already free of ViewModel and WPF references. Merge order: #833 first, which fixes the failing GUI test behind this PR's red check, then #834 (lock files).

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@htsugawa
htsugawa merged commit 40a61a0 into master Oct 16, 2026
1 check passed
@htsugawa
htsugawa deleted the feature/lipid-rt-confirmation branch October 16, 2026 00:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant