Repository navigation
UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run) - #925
Merged
Merged
Conversation
…defects Defects (nonzero flag, zero weight, invalid RMS) must be zero-weighted and noise-filled under every BLEND_HANDLING, with neighbour pixels left raw under uberseg. The masked-fraction cut must count that same raw set, and an epoch with a defect closer than EPOCH_CENTRAL_DEFECT_RADIUS to the stamp centre must be dropped, at a strict boundary and at the configured radius. The runner must pass both cut options on, the HSM centroid must ignore raw defect values, each epoch must keep its own OFFSET, and every tile must log its epoch-cut tally. The science test recovers the full 2x2 response for columns, 3-px bleeds, single pixels and edge bands at the veto radius, on 0.3" and 0.5" galaxies through round and elliptical 0.7" PSFs, and requires |c1|, |c2| < 5e-4 and |m11|, |m22| < 1%. Flagged pixels in the simulation hold a hot value, so an unfilled defect shows up. A column two pixels inside the radius is the positive control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA
…ral ones Metacal shears the whole stamp without reading the weights, so a defect left raw under uberseg reached the fit. Every defect (nonzero flag, zero weight, invalid RMS) is now zero-weighted and noise-filled whatever BLEND_HANDLING is; uberseg only zeroes the weight of neighbour-side pixels and keeps their light. The option that meant "no neighbour treatment" is renamed from noisefill to none, and the retired name is rejected. The defect set is not symmetrized: at the veto radius the one-sided fill leaves |c| <= 3e-4, while a four-fold mask would quadruple m (-2.7% against -0.64% for a 3-px bleed at 10 px on a 0.5" galaxy) and, through an elliptical PSF, still leave c1 = 7.3e-4. An epoch is dropped when a defect lies closer than EPOCH_CENTRAL_DEFECT_RADIUS (default 10 px) to the stamp centre, the smallest radius at which columns, 3-px bleeds, single pixels and edge bands give |m11|, |m22| < 1% and |c1|, |c2| < 5e-4 on 0.3" and 0.5" galaxies through a 0.7" PSF. The masked-fraction cut counts the same raw defect set; EPOCH_MASKED_FRACTION_CUT (default 1/3) makes it configurable. Each tile logs "epoch cuts: considered=N masked_fraction=A central_veto=B objects_emptied=C". The HSM centroid is measured on the filled image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA
prepare_postage_stamps already holds the object's galaxy vignet dict, so the per-epoch OFFSET is read from it instead of a second sqlitedict lookup. The pixel scale sets only the centroid-prior width; the comments that also credited it with a noise window are corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The interpolation fills only short defect runs (at most 3 px along a row or column, with clean pixels at both ends); anything wider, and runs reaching the stamp border, stays noise-filled. The interpolant must reproduce linear planes without reading defect values, commute with quarter turns of the stamp, and apply one operator to the science and metacal noise images. The weight is zero on the defects and on the quarter-turn orbit of the interpolated pixels, whose light stays. Each defect is vetoed at the radius of its own fill: EPOCH_INTERPOLATED_DEFECT_RADIUS for interpolated pixels, EPOCH_CENTRAL_DEFECT_RADIUS for noise-filled ones. The runner and Ngmix pass DEFECT_FILL to the epoch cuts and to metacal; noise stays the default. The science test recovers the full response matrix for interpolated columns, full and finite 3-px bleeds and single pixels at the interpolated radius, on 0.3" and 0.5" galaxies through round and elliptical 0.7" PSFs and on a 0.7" galaxy through a 0.9" PSF, and for noise-filled wide defects at the noise radius. A bleed three pixels inside the radius is the positive control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA
… veto DEFECT_FILL = interpolate fills short defect runs (at most 3 px along a row or column, with clean pixels at both ends: columns, 3-px bleeds, single pixels) with a Clough-Tocher interpolant of the clean pixels within 4 px. The interpolant is averaged over the four quarter turns of the stamp, and each turn's triangulation serves the science image and the metacal noise image alike. Wider holes and runs reaching the stamp border keep the noise fill. noise stays the default and is unchanged. Interpolation restores the object's light, so the remaining bias is the hole in the likelihood: a one-sided zero-weight column 8 px from a 0.5" galaxy still gives c1 = -1.3e-3. The weight is therefore also zero on the quarter-turn orbit of the interpolated pixels (their light stays), which brings that column to c1 = +2e-6. Symmetrizing the fill instead would trade true light for interpolated light over four times the area. Each defect pixel is vetoed at the radius of its own fill: EPOCH_INTERPOLATED_DEFECT_RADIUS (default 7 px) for interpolated pixels and EPOCH_CENTRAL_DEFECT_RADIUS (10 px) for noise-filled ones. 7 px is the smallest radius at which interpolated columns, 3-px bleeds and single pixels give |m11|, |m22| < 1% and |c1|, |c2| < 5e-4 on 0.3", 0.5" and 0.7" galaxies, round and elliptical PSFs; it does not scale with galaxy size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA
Brings the decision record to what this branch does: - central_defect_veto: committed option fixed_radii (was disabled); the 10 px EPOCH_CENTRAL_DEFECT_RADIUS is pinned by a Values entry on its module constant, now tagged. - epoch_masked_fraction_cut: the cut counts the defect set against EPOCH_MASKED_FRACTION_CUT (no longer [HARDCODED] on flags). - defect_fill: defects are noise-filled under every BLEND_HANDLING; the docstring [LINT] this branch fixes is dropped. - blend_handling: the no-treatment option is BLEND_HANDLING = none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings develop (#875's decision record) in through the updated base, and amends the record for this branch: - defect_fill: the interpolate option is implemented (DEFECT_FILL = interpolate); noise stays the committed fill. - central_defect_veto: EPOCH_INTERPOLATED_DEFECT_RADIUS = 7 joins the Values, on its now-tagged module constant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts: the ngmix runner imports both the epoch-cut defaults and write_empty_tile_output; Ngmix.process logs the epoch-cut tally and then calls log_run_health. Develop's run-health test stubs prepare_postage_stamps with the epoch-cut kwargs and tally and sets the epoch-cut attributes; the defect-fill process tests give their vignettes an OFFSET and their logger warning/error methods. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The neighbour option keeps develop's name, noisefill, in the code, the runner config key, tests, astra.yaml and the committed universe; the name none is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SExtractor's -1e30 markers in the tile VIGNET form their own per-epoch mask (stamp.neighbours): a footprint 3 px from the centre, or covering 45% of the stamp, drops no epoch; noisefill zero-weights and noise-fills exactly the marked pixels, bit for bit as develop does; uberseg leaves them raw and weighted; a flagged column near the centre is still vetoed; and do_ngmix_metacal hands each epoch's mask to make_ngmix_observation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
prepare_postage_stamps wrote the tile VIGNET's -1e30 markers (other detections' footprints) into each epoch's flag stamp as 2**10, so defect_mask counted them: they fed the masked-fraction cut and the central veto. Every epoch shares the tile VIGNET, so a neighbour within EPOCH_CENTRAL_DEFECT_RADIUS dropped every epoch of the object. The markers are now their own per-epoch mask, stamp.neighbours (MegaCam-flipped with the epoch), threaded through do_ngmix_metacal and make_ngmix_observation to prepare_ngmix_weights. The epoch cuts count genuine defects only. Under noisefill the marked pixels get weight 0 and the defects' noise realisation, the output noisefill gave when they arrived as flags; under uberseg they are ignored and the seg-based weight handles neighbours. Defects are treated identically under both. Docstrings, @sc contracts, runner comments and the astra.yaml decisions say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SExtractor writes -1e30 into the tile VIGNET beyond the tile's edge as well as on other detections' footprints. Beyond the edge the epoch holds the object's own light, cut off, so treating those pixels as neighbours exempted edge objects from both epoch cuts and measured them with a noise-filled band near their centre. split_tile_markers classifies the markers geometrically on the flipped tile VIGNET: the union of stamp rows and columns that are entirely -1e30 (the off-image part of a rectangle clip) is off-tile, the rest is the neighbour mask. Off-tile pixels are flagged OFF_TILE_FLAG (2**10) in the epoch's flag stamp, so under both blend handlings they are zero-weighted, filled, and counted by the masked-fraction cut and the central veto. Docstrings, @sc contracts, runner comments and astra.yaml say so. Tests: an object 3 px from the tile edge loses every epoch; a band 12 px away passes the default cuts and is vetoed at radius 13; a corner's L-shaped off-tile region is flagged exactly and a border-touching neighbour stays a neighbour; off-tile pixels are filled under both blend handlings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under DEFECT_FILL = interpolate and BLEND_HANDLING = noisefill, the Clough-Tocher support included marked neighbour pixels that noisefill replaces with noise, so a defect beside a bright neighbour was filled with light the image no longer contains. The support now excludes ~clean (the defects and the removed neighbour pixels); the NaN fallback to the noise fill covers degenerate supports. Under uberseg the neighbour light stays in the image and the support is unchanged. interpolate_defects names its second argument `excluded`. Test: a flagged pixel next to a 1e4 marked neighbour interpolates to the kept-pixel interpolant (1.4, against 3102 with the neighbour in the support) under noisefill, and from the raw neighbour light under uberseg. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fill_defects used np.where, which promotes a float32 stamp to float64; develop's noisefill kept the galaxy stamp's dtype. The filled image is cast back to the input dtype, so noisefill stays bit-identical to develop on float32 stamps. The develop-equivalence test gains a float32 case and checks dtypes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…amp border Every entirely -1e30 stamp row or column counted as off-tile. In an edge stamp, a neighbour footprint that covers the on-image part of a row (a wide DR6 footprint running from the tile edge) completes that row, so the neighbour's pixels became OFF_TILE_FLAG defects: counted by the epoch cuts, interpolated under DEFECT_FILL = interpolate, filled under uberseg. Only the runs of marked rows and columns that start at a stamp border are off-tile now. Tests build tile VIGNETs with the catalogue-mode converter: on the real 202.301 DR6 patch every stamp splits exactly into off-image pixels and seg-map neighbour pixels, and the edge case above keeps its footprint in the neighbour mask through prepare_postage_stamps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY
… converter The neighbour-marking rationale and the converter's BIG comment describe the markers as ngmix neighbours under flag 2**10; ngmix splits them into off-tile defects and a neighbour mask. The masking text names the tile VIGNET, not SExtractor, as the markers' carrier, since catalogue mode paints them from the DR6 segmentation map. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY
A committed ini can then name a workflow knob with its default, and the workflow exports the variable only when the knob leaves its default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under SEG_VIGNET = True the converter adds one int32 stamp per object of the relabelled segmentation map, cut with the window and centres VIGNET is cut with, so the two register pixel for pixel. The stamp cutter moves to sextractor_script.cut_stamps for the SExtractor path to share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under SEG_VIGNET = True, sextractor_runner cuts the SEGMENTATION check image on each object's VIGNET grid and adds it to the sexcat. VIGNET is centred on the pixel nearest SExtractor's double barycentre; the float32 X_IMAGE/Y_IMAGE round to it except on exact half pixels, which are resolved by matching VIGNET against image minus BACKGROUND. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tile_cat takes the stamps from the catalogue it already reads, and BLEND_HANDLING = uberseg fails before the object loop when the column is missing. SEG_VIGNET_PATH, its separate catalogue and its checks go. The stamp columns stay views into the loaded table, so it is held once, and BLEND_HANDLING is environment-expanded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blend_handling: uberseg (config.yaml, default noisefill) exports
SP_SEG_VIGNET=True to tile_detect and SP_BLEND_HANDLING=uberseg to
tile_ngmix, and adds the SEG_VIGNET column's memory to each ngmix chunk.
The committed inis read them as ${SP_SEG_VIGNET:-False} and
${SP_BLEND_HANDLING:-noisefill}, so noisefill plans the campaign it
planned without the knob and the params pin does not move.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A noisefill campaign launched from a shell that exports SP_SEG_VIGNET or SP_BLEND_HANDLING (bare, or in the APPTAINERENV_/SINGULARITYENV_ forms that survive --cleanenv) would hand them to every job through the slurm executor's --export=ALL. The Snakefile now pops them at parse time, so only blend_env's exports under uberseg set them. Prologues and shells are unchanged; the DAG harness gains a launch_env to test it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under SEG_VIGNET, sextractor_runner runs SExtractor with a copy of the param file that adds X_IMAGE_DBL and Y_IMAGE_DBL; add_seg_vignet centres each stamp at rint(DBL - 1), as SExtractor centres VIGNET, and drops the two columns, so the catalogue is noisefill's plus SEG_VIGNET. The half-pixel tie matching against image minus background goes, and with it the BACKGROUND check image dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blend_handling defaults to uberseg in config.yaml and the Snakefile, so a campaign without the key writes SEG_VIGNET in both detection modes and runs ngmix with BLEND_HANDLING = uberseg. The committed inis keep the modules' own noise-fill defaults and the workflow exports the uberseg values, so blend_handling: noisefill plans exactly the previous noise-fill campaign: params_pin_noisefill.json is the previous pin, and a noise-fill run dir resumes with the key set explicitly. The default pin moves in tile_detect and tile_ngmix only. astra's committed blend_handling becomes uberseg, with noisefill the comparison arm. test_committed_blend_handling_fills_defects checks the committed blend_handling x defect_fill pair against prepare_ngmix_weights: the workflow default is the committed blend handling, and under it every defect is zero-weighted and filled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY
cailmdaley
force-pushed
the
feat/uberseg-seg-vignet
branch
from
September 30, 2026 01:10
7af2d80 to
c704e63
Compare
params_pin.json regenerated with --update-params-pin on the merged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd
The defect_fill rationale in astra.yaml now says what a defect is, why its image value matters under metacal, what the fill does, and why. The ngmix template and config.yaml name the default defect weighting, fourfold_zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd
tests/workflow/README.md and test_dag.py keep both sides (develop's test_tile_store_is_unique_per_campaign and LOCAL_TAG/run-dir-hash probes, this branch's blend-handling and noisefill rows); both params pins regenerated with --update-params-pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd
#922 was rebuilt after this branch took it, so the two carry different histories of the defect fill. Resolved to #922's current design (interpolation is the only fill, DEFECT_WEIGHTING, 10% masked-fraction cut, 13 px noise-fill veto) with this branch's SEG_VIGNET UberSeg on top: - ngmix.py: no seg_cat_path; Raises names blend_handling/defect_weighting. - committed.yaml: defect_fill interpolate, blend_handling uberseg, defect_weighting fourfold_zero, epoch_masked_fraction_cut ten_percent. - tile.smk: tile_ngmix exports SP_DEFECT_WEIGHTING and the blend variables. - astra.yaml: #922's defect_fill rationale; this branch's uberseg-default blend_handling text. - tests: drop the DEFECT_FILL tests #922 removed; the committed-universe test checks the fill is the noisefill one under the committed blend handling; _marked_stamps keeps cut_stamps with #922's 20-px-edge geometry; test_sextractor_seg_vignet uses uberseg_mask; test_dag normalizes the run-dir hash in the node-local store name (develop's LOCAL_TAG), which differs between the campaigns these tests compare. - params pins regenerated with --update-params-pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd
Tile_cat takes no seg_cat_path: it reads row_min/row_max and holds both stamp columns of the tile catalogue, VIGNET and SEG_VIGNET, for the chunk's rows as ChunkStamps. test_tile_cat_reads_seg_vignet compares row by row; test_tile_cat_holds_the_stamp_columns_once gives way to test_ngmix_tile_cat_chunk.py, which now writes SEG_VIGNET into the tile catalogue (its separate-seg-catalogue alignment test goes with the separate catalogue). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd
…aches it A copy wedged in a corner of the clean region (an edge band beside the stamp edge) lies outside the support's hull, so the fill came back NaN and the finiteness guard dropped every epoch of the object. Such copies now keep their light, like copies on already-replaced pixels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An exception while building one epoch's ngmix observation (for example the non-finite interpolant guard in prepare_ngmix_weights) used to reach the per-object except, losing every epoch of the object. do_ngmix_metacal now catches it per epoch, drops that epoch and fits the object on the rest; the existing epoch logic (N_EPOCH, masked-fraction cut) decides the rest. An object whose every epoch fails still raises. The drop is kept: Postage_stamp.epoch_failures holds (epoch name, exception), process() logs each at warning level with tile, object and epoch, adds them to the "epoch cuts" tally as failed=, and writes the count as a new per-object column n_epoch_failed -> NGMIX_N_EPOCH_FAILED (final_cat.param for cfis and cfis_image_sims; never-fit rows 0). astra.yaml epoch_masked_fraction_cut records the rule; contract failed-epoch-dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd
…seg-vignet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd
With MASK_TYPE CORRECT, SExtractor writes -1e30 into an object's tile VIGNET over every above-threshold pixel outside its pre-CLEAN footprint: other detections, but also sub-MINAREA sky blobs and fragments CLEAN merged into the object itself. split_tile_markers took every interior marker as a neighbour, so noisefill noise-filled the object's own light (15% of NEIGHBOUR_FLAG=0 objects differed noisefill vs uberseg on 25 tiles). split_tile_markers(tile_vign, shape, seg, object_number) now keeps as neighbours only markers on a seg label other than 0 and the object's own (the join's -1, a detection that left the catalogue, counts). Off-tile detection is unchanged. The other interior markers become nothing: -1e30 is a flag in the tile VIGNET only, and the epoch stamps hold real light there (the object's own or sky), so they keep weight and image and are neither interpolated nor noise-filled. The narrower neighbour mask feeds the noisefill fill and interpolated_defects consistently; uberseg reads no marker, so its output is bit-identical (tested). Without a seg stamp every interior marker stays a neighbour, and ngmix warns once. tile_detect therefore writes SEG_VIGNET under both blend handlings (SP_SEG_VIGNET exported always; DETECT_SEG_MEM_MB / NGMIX_SEG_MEM_MB unconditional). blend_handling now only moves tile_ngmix's prologue: the uberseg params pin is unchanged and the noisefill pin moves (campaign boundary for noisefill only). astra.yaml blend_handling, defect_fill and pixel-mask text updated; contract neighbours-are-other-footprints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd
cailmdaley
added a commit
that referenced
this pull request
Oct 5, 2026
…node-local # Conflicts: # workflow/rules/tile.smk
defect_fill joins defect_weighting on the same path: run config `defect_fill` -> Snakefile (validated against ngmix.DEFECT_FILLS) -> tile_ngmix exports SP_DEFECT_FILL -> [NGMIX_RUNNER] DEFECT_FILL -> Ngmix(defect_fill=...) -> prepare_postage_stamps / prepare_ngmix_weights. interpolate (default) is unchanged. noise makes interpolated_defects return no pixel, so every defect is noise-filled, and the central veto, which reads the same set, holds every defect at the 13 px noise-fill radius; defect_weighting then has nothing to act on. astra: defect_fill loses [HARDCODED], noise becomes a selectable (non-default) option, Values pins ngmix.py::DEFECT_FILL. The workflow wiring test covers both defect options. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J2y26yR4NvPvSDJtcbwL3M
Conflicts resolved: Ngmix keeps this branch's removal of the uberseg seg_cat_path guard and gains the DEFECT_FILL check; tile_ngmix exports SP_DEFECT_FILL alongside SP_DEFECT_WEIGHTING and blend_env. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J2y26yR4NvPvSDJtcbwL3M
…cal disk (#947) * make_post_process: write the tile catalogue once make_post_process appended each EPOCH HDU through FITSCatalogue.save_as_fits, which flushed and reopened the ~670 MB uberseg catalogue per exposure, then rebuilt LDAC_OBJECTS for N_EPOCH with add_col, another whole-file write. It now reads the catalogue with astropy, as add_seg_vignet and match_catalogue do, builds the EPOCH HDUs and N_EPOCH in memory, and writes the file once. The columns are laid out by FITSCatalogue.fits_column (add_cols' column builder, now a public static method), so the file is byte-identical to the per-step route: a new test builds the reference through save_as_fits and add_col and compares bytes, and on DR6 191.307 (31,736 rows, 8 epochs) the output is byte-identical to the previous code's. Measured on that tile: 16.4 s -> 5.0 s node-local, 43.7 s -> 28.5 s on idle /scratch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW * tile_detect: run SExtractor and its post-processing node-local On nibi tile_detect took 9-44 min per tile in single-allocation campaigns, of which SExtractor was 1-2 min: the SEG_VIGNET cut, the DR6 join and the post-processing each rewrite the whole catalogue (~670 MB under uberseg) on /scratch, which a loaded node slows ~30x (191.307 in smk-g13-us: 18, 15 and 9 min). Every tile's ngmix waits on this step. sextractor_runner takes an optional WORK_DIR: SExtractor writes its catalogue and check images in a fresh directory under it (sp-detect<num>.*, which tile_local's stale-store sweep also matches), the rewrites happen there, and the files are moved to the output directory once complete. The directory is removed on the way out, also on failure, so a failed run publishes nothing. Without WORK_DIR the runner writes in the output directory as before. config_tile_Sx.ini sets WORK_DIR = /local/scratch, which both profiles bind and the node-local tile store already uses. The ini is not part of params.pre, so the params pins do not move and running campaigns are not invalidated; the published files and the completeness counts are unchanged. Measured on DR6 191.307 under uberseg in one 2-core step: 36 s end to end through WORK_DIR against 112 s in place on idle /scratch (44 min in the loaded g13-us allocation). Catalogue, BACKGROUND and SEGMENTATION check images are identical to the in-place run's except SExtractor's own extraction time cards (SEXTIME, SEXELAPS) in LDAC_IMHEAD. tests/module/test_sextractor_work_dir.py checks the staged run publishes the in-place bytes, leaves the work directory empty, and publishes nothing when the join refuses the catalogue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GPLdkKZzVveSKyCw4DGi5X
… regenerate params pins SEG_VIGNET column design from this branch supersedes the separate segmentation-catalogue path in ngmix Tile_cat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GPLdkKZzVveSKyCw4DGi5X
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GPLdkKZzVveSKyCw4DGi5X
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GPLdkKZzVveSKyCw4DGi5X
…gnet; regenerate params pins Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GPLdkKZzVveSKyCw4DGi5X
… pin # Conflicts: # tests/workflow/params_pin.json
…seg-vignet; regenerate params pins # Conflicts: # src/shapepipe/modules/make_cat_package/make_cat.py # tests/workflow/params_pin.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We supply object-aligned segmentation stamps to ngmix and make
blend_handling: ubersegthe workflow default.UberSeg zero-weights pixels nearer a neighbour's segmentation footprint than the target's, while retaining their light for metacalibration.
This is a selectable treatment, not a demonstrated survey-wide shear calibration;
blend_handling: noisefillremains available for simulation comparisons.Segmentation and neighbour masks
sextractor_runnercuts an int32SEG_VIGNETcolumn from the tile segmentation check image, on the same pixel grid asVIGNET, using SExtractor's centring rule.NUMBER, unmatched detections become −1, and sky remains 0. This prevents a dropped detection's label from masquerading as another object's footprint. Duplicate or non-int32-exact catalogue identifiers are refused.Workflow and resources
Tile detection builds its catalogue in memory and runs SExtractor in node-local storage.
The segmentation column adds about 3 GB per tile; detection reserves an additional 3000 MB under both treatments.
The detection environment exports segmentation settings alongside catalogue matching and image-simulation CCD bounds.
Workflow defaults, ASTRA decisions and the committed universe select UberSeg; the standalone ngmix fallback remains noisefill.
Separate workflow pins cover both treatments.
These shell and parameter changes require deployment at a campaign boundary.
Checks and limits
Tests cover stamp centring, complete label remapping, identifier boundaries, ngmix stamp loading, neighbour classification and workflow propagation for both treatments.
The real DR6 tile 202.301 check covered 36,014 matched objects: 36,013 segmentation centres carried their own identifier and one fell on sky; no stamp carried its identifier outside its own footprint.
Peak detection memory was 2.9 GiB with segmentation versus 1.5 GiB without it.
These measurements check segmentation integrity and resources, not shear calibration.
— Codex on behalf of Cail.