UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run)

astra.yaml

@@ -323,7 +323,8 @@ analyses:
           image, and ShapePipe's tile SExtractor run requests no IMAFLAGS_ISO
           (detection.detection_source_mode). Neighbour pixels are masked
           separately, from the -1e30 markers SExtractor writes into the tile
-          VIGNET (shape_measurement.blend_handling). Sky-fixed masks never touch
+          VIGNET and the coadd segmentation map
+          (shape_measurement.blend_handling). Sky-fixed masks never touch
           pixels: a star halo changes no stamp.
           Values:
           config_exp_psfex.ini#SEXTRACTOR_RUNNER.FLAG_IMAGE = True;
@@ -1567,11 +1568,15 @@ analyses:
           symmetrized mask as in DES, since symmetrizing quadruples m and
           leaves an additive c1 (symmetrized_4fold_noise); the hole this
           leaves near the galaxy is bounded by central_defect_veto. In the
-          tile VIGNET, SExtractor writes -1e30 over other detections and
-          beyond the tile edge. Rows and columns that are entirely -1e30 and
-          start at the stamp border are off-tile, hence defects (the epoch
-          has the object's light there); every other -1e30 pixel is a
-          neighbour, handled by blend_handling.
+          tile VIGNET, SExtractor writes -1e30 beyond the tile edge and over
+          every above-threshold pixel outside the object's pre-CLEAN
+          footprint. Rows and columns that are entirely -1e30 and start at
+          the stamp border are off-tile, hence defects (the epoch has the
+          object's light there); the other -1e30 pixels on another object's
+          segmentation footprint are neighbours, handled by blend_handling.
+          The rest (sub-MINAREA noise peaks, the object's own CLEAN-merged
+          fragments) are not defects either: the epoch stamps hold real
+          light there, so they keep their weight and image.
           Values:
           VIGNETMAKER_RUNNER_RUN_1.MASKING = False;
           VIGNETMAKER_RUNNER_RUN_2.MASKING = False;
@@ -1638,16 +1643,12 @@ analyses:
           Covers only pixels shared with a neighbour; defect_fill does not
           depend on it: defects are filled the same way under every
           BLEND_HANDLING, and the epoch cuts never count neighbour pixels.
-          noisefill (the default; the committed config sets no key) gives
-          weight 0 to the pixels marked -1e30 in the tile VIGNET on other
-          detections' footprints (about 93% of neighbour-footprint pixels on
-          a sims tile), excluding the off-tile
-          rows and columns, which are defects, and replaces them with noise;
-          unmarked
-          neighbour pixels keep their weight and light. uberseg ignores the
-          markers and zeroes the weight of pixels nearer a neighbour's coadd
-          segmentation footprint than the target's (DILATE_NEIGHBOUR, default 1)
-          and leaves the image untouched, as official uberseg does:
+          The committed choice is uberseg (the workflow's default
+          blend_handling), which ignores the -1e30 neighbour markers in the
+          tile VIGNET and zeroes the weight of pixels nearer a neighbour's
+          coadd segmentation footprint than the target's (DILATE_NEIGHBOUR,
+          default 1), leaving the neighbour side's image untouched, as
+          official uberseg does:
           esheldon/meds get_uberseg returns a weight map (a
           nearest-segment-pixel Voronoi split). DES Y1's fiducial metacal and
           the last Y3 config ran on uberseg-weighted stamps with raw neighbour
@@ -1664,9 +1665,23 @@ analyses:
           with simulations (mask_des_y1_uberseg_only,
           mask_blend_bias_detection). Noise-filling the neighbour side would
           instead cut the target's own light along an unsheared boundary, a
-          sharp edge that rings in the FFTs. The recommended comparison arm is
-          uberseg (weight-only); defect_fill is the same in both arms.
-        default: noisefill
+          sharp edge that rings in the FFTs. The alternative, noisefill
+          (ngmix's own default, and the workflow's blend_handling:
+          noisefill), gives weight 0 to the pixels marked -1e30 in the tile
+          VIGNET (about 93% of neighbour-footprint pixels on a simulated
+          tile), excluding the off-tile rows and columns, which are defects,
+          and replaces them with noise; unmarked neighbour pixels keep their
+          weight and light. Under both handlings the neighbours are defined
+          by the coadd segmentation map: a marker counts only on a footprint
+          labelled neither 0 nor the target. SExtractor's MASK_TYPE CORRECT
+          also marks sub-MINAREA sky blobs and fragments CLEAN merged into
+          the target, which the seg labels as sky or as the target; noise-
+          filling those removed the target's own light (15% of
+          NEIGHBOUR_FLAG = 0 objects differed between the arms on 25 tiles).
+          tile_detect therefore writes SEG_VIGNET under both handlings.
+          Uberseg reads no marker, so its output is unchanged. It is
+          the comparison arm, with defect_fill held equal across arms.
+        default: uberseg
         options:
           noisefill:
             label: Noise-fill the marked neighbour pixels (BLEND_HANDLING = noisefill)
@@ -1681,8 +1696,8 @@ analyses:
           uberseg:
             label: UberSeg, weight-only
             description: >-
-              As in DES Y1/Y3. Needs the coadd segmentation stamp
-              (SEG_VIGNET_PATH). DILATE_NEIGHBOUR absorbs the coadd-vs-epoch
+              As in DES Y1/Y3. Needs the coadd segmentation stamp (the tile
+              catalogue's SEG_VIGNET). DILATE_NEIGHBOUR absorbs the coadd-vs-epoch
               overlay offset: ShapePipe reuses one coadd seg stamp for every
               epoch where MEDS reprojects it.
             insights: [mask_uberseg_weight_only, mask_uberseg_neighbour_bias, mask_des_y1_uberseg_only, mask_blend_bias_detection]

src/shapepipe/modules/make_cat_package/make_cat.py

@@ -92,7 +92,8 @@ def read_sextractor_data(sexcat_path, remove_vignet=True):
     sexcat_path : str
         Path to SExtractor catalogue
     remove_vignet : bool
-        If ``True`` will not keep the ``VIGNET`` field
+        If ``True`` will not keep the ``VIGNET`` and ``SEG_VIGNET`` stamp
+        fields
 
     Returns
     -------
@@ -109,7 +110,7 @@ def read_sextractor_data(sexcat_path, remove_vignet=True):
     columns = {
         name: data[name]
         for name in data.dtype.names
-        if not (remove_vignet and name == "VIGNET")
+        if not (remove_vignet and name in ("VIGNET", "SEG_VIGNET"))
     }
 
     tile_name = os.path.basename(sexcat_path)

src/shapepipe/modules/ngmix_package/ngmix.py

@@ -589,13 +589,10 @@ class Tile_cat():
     Parameters
     ----------
     cat_path : str
-        Path to the tile SExtractor catalogue.
-    seg_cat_path : str, optional
-        Path to the coadd-frame segmentation VIGNET catalogue (a CLASSIC-mode
-        vignetmaker output cut from the tile ``SEGMENTATION`` check image),
-        row-aligned to ``cat_path``. When given, ``self.seg`` holds one integer
-        seg stamp per object for the ``"uberseg"`` blend handling; ``None``
-        leaves ``self.seg`` unset, which only ``"uberseg"`` needs.
+        Path to the tile SExtractor catalogue. Its optional ``SEG_VIGNET``
+        column, one integer coadd segmentation stamp per object on the grid
+        of its ``VIGNET``, becomes ``self.seg``, which defines the neighbours
+        under both blend handlings; without it ``self.seg`` is ``None``.
     row_min, row_max : int, optional
         First and last catalogue row of the chunk (1-based, inclusive; see
         :func:`chunk_rows`). The default, ``-1``, is unbounded, so the whole
@@ -605,12 +602,10 @@ class Tile_cat():
     def __init__(
         self,
         cat_path,
-        seg_cat_path=None,
         row_min=-1,
         row_max=-1,
     ):
         self.cat_path = cat_path
-        self.seg_cat_path = seg_cat_path
         self.row_min = row_min
         self.row_max = row_max
         if cat_path:
@@ -639,41 +634,17 @@ def get_data(self, cat_path):
             else None
         )
 
-        tile_cat.close()
+        # Coadd-frame segmentation stamp (integer labels, the catalogue's
+        # NUMBER), one per object on the grid of its VIGNET, overlaid
+        # unchanged on every epoch; it defines the neighbours (shapepipe#776,
+        # split_tile_markers, uberseg_mask).
+        self.seg = (
+            ChunkStamps(data['SEG_VIGNET'], self.rows)
+            if 'SEG_VIGNET' in cols
+            else None
+        )
 
-        # Coadd-frame SExtractor segmentation stamp (integer labels), one per
-        # object and row-aligned to the tile catalogue, overlaid unchanged on
-        # every epoch for uberseg neighbour masking (shapepipe#776). None ->
-        # uberseg unavailable; ``"noisefill"`` does not read it.
-        self.seg = None
-        if self.seg_cat_path:
-            seg_cat = file_io.FITSCatalogue(
-                self.seg_cat_path,
-                SEx_catalogue=True,
-                memmap=True,
-            )
-            seg_cat.open()
-            seg_data = seg_cat.get_data()
-            # The seg VIGNETs are indexed by tile-catalogue row (self.seg[i]),
-            # so the two catalogues MUST be row-aligned. Fail loud at load if
-            # they are not — a silent length/order mismatch would hand every
-            # object the wrong footprint and quietly corrupt every mask.
-            if len(seg_data) != len(self.obj_id):
-                raise ValueError(
-                    f"SEG_VIGNET_PATH '{self.seg_cat_path}' has"
-                    + f" {len(seg_data)} rows but the tile catalogue has"
-                    + f" {len(self.obj_id)}; the segmentation vignets must be"
-                    + " row-aligned to the tile catalogue."
-                )
-            if 'NUMBER' in seg_data.dtype.names:
-                if not np.array_equal(seg_data['NUMBER'], self.obj_id):
-                    raise ValueError(
-                        f"SEG_VIGNET_PATH '{self.seg_cat_path}' NUMBER column"
-                        + " does not match the tile catalogue NUMBER; the"
-                        + " segmentation vignets are misaligned or reordered."
-                    )
-            self.seg = ChunkStamps(seg_data['VIGNET'], self.rows)
-            seg_cat.close()
+        tile_cat.close()
 
 class Postage_stamp():
     """Galaxy Postage Stamp.
@@ -871,13 +842,11 @@ class Ngmix(object):
     blend_handling : {"noisefill", "uberseg"}, optional
         Neighbour treatment. ``"noisefill"`` (default) zero-weights and
         noise-fills the pixels marked -1e30 in the tile VIGNET on other
-        detections' footprints; ``"uberseg"`` ignores those markers,
+        detections' footprints; ``"uberseg"`` ignores those markers and
         zeroes the weight of neighbour-side pixels from the coadd
-        segmentation map and requires ``seg_cat_path``. Defect pixels are
-        filled under both (see :func:`prepare_ngmix_weights`).
-    seg_cat_path : str, optional
-        Path to the coadd-frame segmentation VIGNET catalogue (see
-        :class:`Tile_cat`). Required when ``blend_handling="uberseg"``.
+        segmentation stamps, the tile catalogue's ``SEG_VIGNET`` column (see
+        :class:`Tile_cat`), which it requires. Defect pixels are filled
+        under both (see :func:`prepare_ngmix_weights`).
     dilate_neighbour : int, optional
         Neighbour-mask dilation iterations for ``"uberseg"`` (see
         :func:`uberseg_mask`); the default is ``1``.
@@ -902,7 +871,7 @@ class Ngmix(object):
         If the length of the input file list is incorrect
     ValueError
         If ``blend_handling``, ``defect_weighting`` or ``defect_fill`` is
-        unknown, or ``"uberseg"`` is selected without ``seg_cat_path``.
+        unknown.
 
     """
 
@@ -920,7 +889,6 @@ def __init__(
         bkg_sub=True,
         centroid_source="wcs",
         blend_handling="noisefill",
-        seg_cat_path=None,
         dilate_neighbour=1,
         metacal_psf="fitgauss",
         defect_weighting=DEFECT_WEIGHTING,
@@ -956,14 +924,6 @@ def __init__(
                 + f" {DEFECT_FILLS}"
             )
 
-        # Fail fast at construction (not deep in the per-epoch loop) when
-        # uberseg is requested without its segmentation input (shapepipe#776).
-        if blend_handling == "uberseg" and seg_cat_path is None:
-            raise ValueError(
-                "blend_handling='uberseg' requires SEG_VIGNET_PATH (the coadd"
-                + " SExtractor segmentation vignets); none configured."
-            )
-
         self._tile_cat_path = input_file_list[0]
         if bkg_sub:
             bkg_path, psf_path, weight_path, flag_path = (
@@ -1003,7 +963,6 @@ def __init__(
         self._bkg_sub = bkg_sub
         self._centroid_source = centroid_source
         self._blend_handling = blend_handling
-        self._seg_cat_path = seg_cat_path
         self._dilate_neighbour = dilate_neighbour
         self._metacal_psf = metacal_psf
         self._defect_weighting = defect_weighting
@@ -1400,12 +1359,27 @@ def process(self):
         """
         tile_cat = Tile_cat(
             self._tile_cat_path,
-            self._seg_cat_path,
             self._id_obj_min,
             self._id_obj_max,
         )
         vignet_cat = self._vignet_cat
 
+        # Fail before the per-object loop, whose try/except would otherwise
+        # drop every object one by one (shapepipe#776).
+        if self._blend_handling == "uberseg" and tile_cat.seg is None:
+            raise ValueError(
+                "BLEND_HANDLING = uberseg needs the tile catalogue's"
+                + f" SEG_VIGNET column, which {self._tile_cat_path} lacks;"
+                + " write it at tile detection (SEG_VIGNET = True)."
+            )
+        if tile_cat.seg is None and getattr(tile_cat, "vign", None) is not None:
+            self._w_log.warning(
+                f"{self._tile_cat_path} has no SEG_VIGNET column: every"
+                + " interior -1e30 marker is taken as a neighbour,"
+                + " including the object's own CLEAN-merged fragments"
+                + " (neighbours-are-other-footprints)."
+            )
+
         check_wcs_centroid_offset(
             self._centroid_source, tile_cat, vignet_cat.gal_vign_cat
         )
@@ -1529,7 +1503,7 @@ def process(self):
             res['obj_id'] = obj_id
             # Neighbour flag: does the coadd seg stamp hold any non-central,
             # non-zero label? Systematics-test hook (shapepipe#776); computed
-            # from the raw coadd seg, 0 when uberseg is off / seg absent.
+            # from the raw coadd seg, 0 when the seg is absent.
             res['neighbour_flag'] = int(
                 seg_has_neighbour(tile_cat.seg[i_tile], obj_id)
                 if getattr(tile_cat, "seg", None) is not None
@@ -1630,8 +1604,8 @@ def prepare_postage_stamps(
     that ``DEFECT_WEIGHTING`` zero-weights or interpolates are not counted.
 
     @sc [decision:shape_measurement.central_defect_veto,decision:shape_measurement.epoch_masked_fraction_cut,decision:shape_measurement.blend_handling] neighbour-markers-are-not-defects
-    The tile VIGNET's -1e30 markers on other detections' footprints
-    (:func:`split_tile_markers`) are the epoch's neighbour mask
+    The tile VIGNET's -1e30 markers on other objects' segmentation
+    footprints (:func:`split_tile_markers`) are the epoch's neighbour mask
     (``stamp.neighbours``), not defects: every epoch shares the tile VIGNET,
     so counting them would drop every epoch of a blended object.
 
@@ -1738,7 +1712,9 @@ def prepare_postage_stamps(
         flag_vign = flag_obj[expccd_name]['VIGNET']
         # Off-tile pixels are defects (off-tile-pixels-are-defects); the
         # other -1e30 markers are neighbours (neighbour-markers-are-not-defects).
-        neighbour, off_tile = split_tile_markers(tile_vign, np.shape(gal_vign))
+        neighbour, off_tile = split_tile_markers(
+            tile_vign, np.shape(gal_vign), tile_seg, obj_id
+        )
         weight_vign = np.where(off_tile, 0, weight_obj[expccd_name]['VIGNET'])
         bkg_rms_vign = (
             bkg_rms_obj[expccd_name]['VIGNET']
@@ -1812,25 +1788,39 @@ def prepare_postage_stamps(
 
     return stamp
 
-def split_tile_markers(tile_vign, shape):
+def split_tile_markers(tile_vign, shape, seg=None, object_number=None):
     """Split the tile VIGNET's -1e30 markers into neighbour and off-tile.
 
     @sc [decision:shape_measurement.blend_handling,decision:shape_measurement.defect_fill] off-tile-is-marked-border-rows-and-columns
-    The tile VIGNET holds -1e30 on the footprints of other detections and on
-    stamp pixels beyond the tile's edge (SExtractor writes it). A stamp
-    clipped by the tile's rectangle loses whole rows and whole columns from
-    its border, so the off-tile pixels are the union of the runs of
-    entirely -1e30 rows and columns that start at a stamp border. The remaining markers are
-    neighbour pixels: a footprint touching the stamp border, and a footprint
-    that completes an interior row or column beside an off-tile band, stay
-    neighbours.
+    SExtractor writes -1e30 into the tile VIGNET on stamp pixels beyond the
+    tile's edge and, under MASK_TYPE CORRECT, on every above-threshold pixel
+    that is not the object's own pre-CLEAN footprint. A stamp clipped by the
+    tile's rectangle loses whole rows and whole columns from its border, so
+    the off-tile pixels are the union of the runs of entirely -1e30 rows and
+    columns that start at a stamp border.
+
+    @sc [decision:shape_measurement.blend_handling] neighbours-are-other-footprints
+    The other markers also fall on pixels no other object owns: detections
+    smaller than DETECT_MINAREA (noise peaks, catalogued nowhere) and
+    fragments CLEAN merged into the object itself, which the segmentation
+    map labels as the object. With ``seg``, the neighbour pixels are the
+    markers on another label's footprint (any label but 0 and
+    ``object_number``; the catalogue join's -1 marks a detection that left
+    the catalogue). The markers left over are nothing: -1e30 is a flag in the
+    tile VIGNET only, and the epoch stamps hold real light there, the
+    object's own or sky. Without ``seg``, every interior marker is a
+    neighbour.
 
     Parameters
     ----------
     tile_vign : numpy.ndarray or None
         Tile VIGNET stamp, oriented like the epoch; ``None`` marks nothing.
     shape : tuple of int
         Stamp shape, used when ``tile_vign`` is ``None``.
+    seg : numpy.ndarray, optional
+        The object's SEG_VIGNET stamp, oriented like ``tile_vign``.
+    object_number : int, optional
+        The object's label in ``seg``; required with ``seg``.
 
     Returns
     -------
@@ -1853,7 +1843,10 @@ def border_runs(full):
         border_runs(marker.all(axis=1))[:, None]
         | border_runs(marker.all(axis=0))[None, :]
     )
-    return marker & ~off_tile, off_tile
+    neighbour = marker & ~off_tile
+    if seg is not None:
+        neighbour &= (seg != 0) & (seg != object_number)
+    return neighbour, off_tile
 
 
 def background_subtract(gal,bkg):
@@ -2480,7 +2473,7 @@ def prepare_ngmix_weights(
         raise ValueError(
             "blend_handling='uberseg' requires a segmentation map and the"
             + " central object_number; none reached prepare_ngmix_weights."
-            + " Set SEG_VIGNET_PATH on the ngmix run (see"
+            + " The tile catalogue's SEG_VIGNET column carries the map (see"
             + " CosmoStat/shapepipe#776)."
         )