Skip to content

Fix #655: size segmentation confusion matrix by largest class index - #656

Open
smitdighe wants to merge 1 commit into
JdeRobot:masterfrom
smitdighe:fix/m2d-noncontiguous-idx
Open

smitdighe wants to merge 1 commit into
JdeRobot:masterfrom
smitdighe:fix/m2d-noncontiguous-idx

Conversation

@smitdighe

Copy link
Copy Markdown

Fixes #655

What changed and why

  • TorchImageSegmentationModel.eval() now sizes the confusion matrix with the largest class index plus one instead of the number of classes, the same expression export() already uses. With translation_direction="model_to_dataset" the evaluation ontology is the dataset ontology, and for non-contiguous indices such as RELLIS-3D (20 classes, indices up to 34) the old size dropped GT pixels with high indices and then raised IndexError when building the metrics table. Unused indices become NaN rows, which the macro average already skips. For contiguous ontologies the value does not change.
  • New tests/models/test_torch_segmentation.py: one 4x4 image with dataset indices 0, 3 and 4 and a model that returns the GT, run in both directions and expecting IoU 1.0 for every class and the macro average. Like tests/utils/test_torch.py, it is skipped when real torch is not installed, so the current CI job skips it.

This PR only touches the image segmentation path. I'll extend it to the LiDAR, TF and eval_preds paths in this PR or in a follow-up, whichever you prefer.

How I tested

pytest -q                                          -> 88 passed (86 on master + 2 new)
pytest -v tests/models/test_torch_segmentation.py  -> 2 passed
  (on master: model_to_dataset case fails with IndexError: index 3 is out of bounds for axis 0 with size 3)
same file with torch import blocked (CI setup)     -> 1 skipped
black --check (26.5.1) on both files               -> 2 files would be left unchanged
  • RELLIS-3D sample data from tutorial_image_segmentation_sample.ipynb, evaluated in both directions: per-class IoU identical, mIoU 0.4617 and micro IoU 0.7737 in both. On master model_to_dataset fails with IndexError: index 23 is out of bounds for axis 0 with size 20.
  • Executed tutorial_image_segmentation_sample.ipynb end to end: no errors.
  • GUI, Image Segmentation evaluator with model_to_dataset (one-image Cityscapes-style folder, tiny TorchScript model): the evaluation now completes.

Screenshots

Before:

Before: error in the evaluator

After:

After: evaluation completes

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Image segmentation eval crashes with model_to_dataset on RELLIS-3D

1 participant