Repository navigation
tile_detect: build the catalogue in memory, run SExtractor on node-local disk - #947
Merged
Merged
Conversation
make_post_process appended each EPOCH HDU through FITSCatalogue.save_as_fits, which flushed and reopened the ~670 MB uberseg catalogue per exposure, then rebuilt LDAC_OBJECTS for N_EPOCH with add_col, another whole-file write. It now reads the catalogue with astropy, as add_seg_vignet and match_catalogue do, builds the EPOCH HDUs and N_EPOCH in memory, and writes the file once. The columns are laid out by FITSCatalogue.fits_column (add_cols' column builder, now a public static method), so the file is byte-identical to the per-step route: a new test builds the reference through save_as_fits and add_col and compares bytes, and on DR6 191.307 (31,736 rows, 8 epochs) the output is byte-identical to the previous code's. Measured on that tile: 16.4 s -> 5.0 s node-local, 43.7 s -> 28.5 s on idle /scratch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW
On nibi tile_detect took 9-44 min per tile in single-allocation campaigns, of which SExtractor was 1-2 min: the SEG_VIGNET cut, the DR6 join and the post-processing each rewrite the whole catalogue (~670 MB under uberseg) on /scratch, which a loaded node slows ~30x (191.307 in smk-g13-us: 18, 15 and 9 min). Every tile's ngmix waits on this step. sextractor_runner takes an optional WORK_DIR: SExtractor writes its catalogue and check images in a fresh directory under it (sp-detect<num>.*, which tile_local's stale-store sweep also matches), the rewrites happen there, and the files are moved to the output directory once complete. The directory is removed on the way out, also on failure, so a failed run publishes nothing. Without WORK_DIR the runner writes in the output directory as before. config_tile_Sx.ini sets WORK_DIR = /local/scratch, which both profiles bind and the node-local tile store already uses. The ini is not part of params.pre, so the params pins do not move and running campaigns are not invalidated; the published files and the completeness counts are unchanged. Measured on DR6 191.307 under uberseg in one 2-core step: 36 s end to end through WORK_DIR against 112 s in place on idle /scratch (44 min in the loaded g13-us allocation). Catalogue, BACKGROUND and SEGMENTATION check images are identical to the in-place run's except SExtractor's own extraction time cards (SEXTIME, SEXELAPS) in LDAC_IMHEAD. tests/module/test_sextractor_work_dir.py checks the staged run publishes the in-place bytes, leaves the work directory empty, and publishes nothing when the join refuses the catalogue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW
…node-local # Conflicts: # workflow/rules/tile.smk
This was referenced Oct 10, 2026
Open
cailmdaley
added a commit
that referenced
this pull request
Oct 25, 2026
… SExtractor run) (#925) * test(ngmix): specify defect fill, epoch cuts and shear recovery near defects Defects (nonzero flag, zero weight, invalid RMS) must be zero-weighted and noise-filled under every BLEND_HANDLING, with neighbour pixels left raw under uberseg. The masked-fraction cut must count that same raw set, and an epoch with a defect closer than EPOCH_CENTRAL_DEFECT_RADIUS to the stamp centre must be dropped, at a strict boundary and at the configured radius. The runner must pass both cut options on, the HSM centroid must ignore raw defect values, each epoch must keep its own OFFSET, and every tile must log its epoch-cut tally. The science test recovers the full 2x2 response for columns, 3-px bleeds, single pixels and edge bands at the veto radius, on 0.3" and 0.5" galaxies through round and elliptical 0.7" PSFs, and requires |c1|, |c2| < 5e-4 and |m11|, |m22| < 1%. Flagged pixels in the simulation hold a hot value, so an unfilled defect shows up. A column two pixels inside the radius is the positive control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA * feat(ngmix): noise-fill defects under every BLEND_HANDLING, veto central ones Metacal shears the whole stamp without reading the weights, so a defect left raw under uberseg reached the fit. Every defect (nonzero flag, zero weight, invalid RMS) is now zero-weighted and noise-filled whatever BLEND_HANDLING is; uberseg only zeroes the weight of neighbour-side pixels and keeps their light. The option that meant "no neighbour treatment" is renamed from noisefill to none, and the retired name is rejected. The defect set is not symmetrized: at the veto radius the one-sided fill leaves |c| <= 3e-4, while a four-fold mask would quadruple m (-2.7% against -0.64% for a 3-px bleed at 10 px on a 0.5" galaxy) and, through an elliptical PSF, still leave c1 = 7.3e-4. An epoch is dropped when a defect lies closer than EPOCH_CENTRAL_DEFECT_RADIUS (default 10 px) to the stamp centre, the smallest radius at which columns, 3-px bleeds, single pixels and edge bands give |m11|, |m22| < 1% and |c1|, |c2| < 5e-4 on 0.3" and 0.5" galaxies through a 0.7" PSF. The masked-fraction cut counts the same raw defect set; EPOCH_MASKED_FRACTION_CUT (default 1/3) makes it configurable. Each tile logs "epoch cuts: considered=N masked_fraction=A central_veto=B objects_emptied=C". The HSM centroid is measured on the filled image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA * ngmix: read each epoch's OFFSET from the object dict in hand prepare_postage_stamps already holds the object's galaxy vignet dict, so the per-epoch OFFSET is read from it instead of a second sqlitedict lookup. The pixel scale sets only the centroid-prior width; the comments that also credited it with a noise window are corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA * tests: name the none-mode uberseg test for none, not noisefill Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(ngmix): specify DEFECT_FILL = interpolate and its shear recovery The interpolation fills only short defect runs (at most 3 px along a row or column, with clean pixels at both ends); anything wider, and runs reaching the stamp border, stays noise-filled. The interpolant must reproduce linear planes without reading defect values, commute with quarter turns of the stamp, and apply one operator to the science and metacal noise images. The weight is zero on the defects and on the quarter-turn orbit of the interpolated pixels, whose light stays. Each defect is vetoed at the radius of its own fill: EPOCH_INTERPOLATED_DEFECT_RADIUS for interpolated pixels, EPOCH_CENTRAL_DEFECT_RADIUS for noise-filled ones. The runner and Ngmix pass DEFECT_FILL to the epoch cuts and to metacal; noise stays the default. The science test recovers the full response matrix for interpolated columns, full and finite 3-px bleeds and single pixels at the interpolated radius, on 0.3" and 0.5" galaxies through round and elliptical 0.7" PSFs and on a 0.7" galaxy through a 0.9" PSF, and for noise-filled wide defects at the noise radius. A bleed three pixels inside the radius is the positive control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA * feat(ngmix): DEFECT_FILL = interpolate, with a fill-dependent central veto DEFECT_FILL = interpolate fills short defect runs (at most 3 px along a row or column, with clean pixels at both ends: columns, 3-px bleeds, single pixels) with a Clough-Tocher interpolant of the clean pixels within 4 px. The interpolant is averaged over the four quarter turns of the stamp, and each turn's triangulation serves the science image and the metacal noise image alike. Wider holes and runs reaching the stamp border keep the noise fill. noise stays the default and is unchanged. Interpolation restores the object's light, so the remaining bias is the hole in the likelihood: a one-sided zero-weight column 8 px from a 0.5" galaxy still gives c1 = -1.3e-3. The weight is therefore also zero on the quarter-turn orbit of the interpolated pixels (their light stays), which brings that column to c1 = +2e-6. Symmetrizing the fill instead would trade true light for interpolated light over four times the area. Each defect pixel is vetoed at the radius of its own fill: EPOCH_INTERPOLATED_DEFECT_RADIUS (default 7 px) for interpolated pixels and EPOCH_CENTRAL_DEFECT_RADIUS (10 px) for noise-filled ones. 7 px is the smallest radius at which interpolated columns, 3-px bleeds and single pixels give |m11|, |m22| < 1% and |c1|, |c2| < 5e-4 on 0.3", 0.5" and 0.7" galaxies, round and elliptical PSFs; it does not scale with galaxy size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA * ngmix: restore the BLEND_HANDLING name noisefill The neighbour option keeps develop's name, noisefill, in the code, the runner config key, tests, astra.yaml and the committed universe; the name none is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(ngmix): specify neighbour markers apart from defects SExtractor's -1e30 markers in the tile VIGNET form their own per-epoch mask (stamp.neighbours): a footprint 3 px from the centre, or covering 45% of the stamp, drops no epoch; noisefill zero-weights and noise-fills exactly the marked pixels, bit for bit as develop does; uberseg leaves them raw and weighted; a flagged column near the centre is still vetoed; and do_ngmix_metacal hands each epoch's mask to make_ngmix_observation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ngmix): keep SExtractor's neighbour markers out of the defect set prepare_postage_stamps wrote the tile VIGNET's -1e30 markers (other detections' footprints) into each epoch's flag stamp as 2**10, so defect_mask counted them: they fed the masked-fraction cut and the central veto. Every epoch shares the tile VIGNET, so a neighbour within EPOCH_CENTRAL_DEFECT_RADIUS dropped every epoch of the object. The markers are now their own per-epoch mask, stamp.neighbours (MegaCam-flipped with the epoch), threaded through do_ngmix_metacal and make_ngmix_observation to prepare_ngmix_weights. The epoch cuts count genuine defects only. Under noisefill the marked pixels get weight 0 and the defects' noise realisation, the output noisefill gave when they arrived as flags; under uberseg they are ignored and the seg-based weight handles neighbours. Defects are treated identically under both. Docstrings, @sc contracts, runner comments and the astra.yaml decisions say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ngmix): off-tile pixels are defects, not neighbours SExtractor writes -1e30 into the tile VIGNET beyond the tile's edge as well as on other detections' footprints. Beyond the edge the epoch holds the object's own light, cut off, so treating those pixels as neighbours exempted edge objects from both epoch cuts and measured them with a noise-filled band near their centre. split_tile_markers classifies the markers geometrically on the flipped tile VIGNET: the union of stamp rows and columns that are entirely -1e30 (the off-image part of a rectangle clip) is off-tile, the rest is the neighbour mask. Off-tile pixels are flagged OFF_TILE_FLAG (2**10) in the epoch's flag stamp, so under both blend handlings they are zero-weighted, filled, and counted by the masked-fraction cut and the central veto. Docstrings, @sc contracts, runner comments and astra.yaml say so. Tests: an object 3 px from the tile edge loses every epoch; a band 12 px away passes the default cuts and is vetoed at radius 13; a corner's L-shaped off-tile region is flagged exactly and a border-touching neighbour stays a neighbour; off-tile pixels are filled under both blend handlings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ngmix): interpolate defects from the pixels the image keeps Under DEFECT_FILL = interpolate and BLEND_HANDLING = noisefill, the Clough-Tocher support included marked neighbour pixels that noisefill replaces with noise, so a defect beside a bright neighbour was filled with light the image no longer contains. The support now excludes ~clean (the defects and the removed neighbour pixels); the NaN fallback to the noise fill covers degenerate supports. Under uberseg the neighbour light stays in the image and the support is unchanged. interpolate_defects names its second argument `excluded`. Test: a flagged pixel next to a 1e4 marked neighbour interpolates to the kept-pixel interpolant (1.4, against 3102 with the neighbour in the support) under noisefill, and from the raw neighbour light under uberseg. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ngmix): the defect fill keeps the stamp's dtype fill_defects used np.where, which promotes a float32 stamp to float64; develop's noisefill kept the galaxy stamp's dtype. The filled image is cast back to the input dtype, so noisefill stays bit-identical to develop on float32 stamps. The develop-equivalence test gains a float32 case and checks dtypes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ngmix): off-tile pixels are the marked rows and columns at the stamp border Every entirely -1e30 stamp row or column counted as off-tile. In an edge stamp, a neighbour footprint that covers the on-image part of a row (a wide DR6 footprint running from the tile edge) completes that row, so the neighbour's pixels became OFF_TILE_FLAG defects: counted by the epoch cuts, interpolated under DEFECT_FILL = interpolate, filled under uberseg. Only the runs of marked rows and columns that start at a stamp border are off-tile now. Tests build tile VIGNETs with the catalogue-mode converter: on the real 202.301 DR6 patch every stamp splits exactly into off-image pixels and seg-map neighbour pixels, and the edge case above keeps its footprint in the neighbour mask through prepare_postage_stamps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY * docs: -1e30 tile-VIGNET markers come from SExtractor or the catalogue converter The neighbour-marking rationale and the converter's BIG comment describe the markers as ngmix neighbours under flag 2**10; ngmix splits them into off-tile defects and a neighbour mask. The masking text names the tile VIGNET, not SExtractor, as the markers' carrier, since catalogue mode paints them from the DR6 segmentation map. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY * Expand ${VAR:-default} in config values A committed ini can then name a workflow knob with its default, and the workflow exports the variable only when the knob leaves its default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Write SEG_VIGNET from the catalogue-mode converter Under SEG_VIGNET = True the converter adds one int32 stamp per object of the relabelled segmentation map, cut with the window and centres VIGNET is cut with, so the two register pixel for pixel. The stamp cutter moves to sextractor_script.cut_stamps for the SExtractor path to share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Write SEG_VIGNET from the tile SExtractor run Under SEG_VIGNET = True, sextractor_runner cuts the SEGMENTATION check image on each object's VIGNET grid and adds it to the sexcat. VIGNET is centred on the pixel nearest SExtractor's double barycentre; the float32 X_IMAGE/Y_IMAGE round to it except on exact half pixels, which are resolved by matching VIGNET against image minus BACKGROUND. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ngmix: read UberSeg's seg stamps from the tile catalogue's SEG_VIGNET Tile_cat takes the stamps from the catalogue it already reads, and BLEND_HANDLING = uberseg fails before the object loop when the column is missing. SEG_VIGNET_PATH, its separate catalogue and its checks go. The stamp columns stay views into the loaded table, so it is held once, and BLEND_HANDLING is environment-expanded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Carry blend_handling to tile_detect and ngmix as two prologue exports blend_handling: uberseg (config.yaml, default noisefill) exports SP_SEG_VIGNET=True to tile_detect and SP_BLEND_HANDLING=uberseg to tile_ngmix, and adds the SEG_VIGNET column's memory to each ngmix chunk. The committed inis read them as ${SP_SEG_VIGNET:-False} and ${SP_BLEND_HANDLING:-noisefill}, so noisefill plans the campaign it planned without the knob and the params pin does not move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * make_cat: leave SEG_VIGNET out of final_cat, with VIGNET Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * astra.yaml: UberSeg reads the tile catalogue's SEG_VIGNET Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop inherited blend variables before any rule A noisefill campaign launched from a shell that exports SP_SEG_VIGNET or SP_BLEND_HANDLING (bare, or in the APPTAINERENV_/SINGULARITYENV_ forms that survive --cleanenv) would hand them to every job through the slurm executor's --export=ALL. The Snakefile now pops them at parse time, so only blend_env's exports under uberseg set them. Prologues and shells are unchanged; the DAG harness gains a launch_env to test it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Centre SExtractor-mode SEG_VIGNET on X/Y_IMAGE_DBL Under SEG_VIGNET, sextractor_runner runs SExtractor with a copy of the param file that adds X_IMAGE_DBL and Y_IMAGE_DBL; add_seg_vignet centres each stamp at rint(DBL - 1), as SExtractor centres VIGNET, and drops the two columns, so the catalogue is noisefill's plus SEG_VIGNET. The half-pixel tie matching against image minus background goes, and with it the BACKGROUND check image dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Point the stale SEG_VIGNET_PATH message and astra note at SEG_VIGNET Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Make UberSeg the default blend handling blend_handling defaults to uberseg in config.yaml and the Snakefile, so a campaign without the key writes SEG_VIGNET in both detection modes and runs ngmix with BLEND_HANDLING = uberseg. The committed inis keep the modules' own noise-fill defaults and the workflow exports the uberseg values, so blend_handling: noisefill plans exactly the previous noise-fill campaign: params_pin_noisefill.json is the previous pin, and a noise-fill run dir resumes with the key set explicitly. The default pin moves in tile_detect and tile_ngmix only. astra's committed blend_handling becomes uberseg, with noisefill the comparison arm. test_committed_blend_handling_fills_defects checks the committed blend_handling x defect_fill pair against prepare_ngmix_weights: the workflow default is the committed blend handling, and under it every defect is zero-weighted and filled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY * Measure windowed centroids for UNIONS-catalogue tiles Since #897 the data path takes its tile sample from the UNIONS DR6 catalogue, which has only isophotal barycentres, and read_ext_sexcat copied them into XWIN_*: every tile-side position (stamp centres and OFFSET, the ngmix centroid prior, tile PSF interpolation, epoch membership, the catalogue RA/Dec) was the barycentre, while image simulations get SExtractor's windowed centroid. read_ext_sexcat now measures XWIN_IMAGE/YWIN_IMAGE at every catalogue object: a vectorised numpy port of SExtractor 2.25's compute_winpos (Gaussian window of sigma 2 FLUX_RADIUS / 2.35, 4-sigma aperture with 11x11 edge subsampling, step 2x the weighted mean offset, at most 16 steps, stop below 1e-4 px), on the tile less a SExtractor-like BACK_SIZE 512 / BACK_FILTERSIZE 9 background, with neighbours' segmentation footprints replaced by their mirror image as MASK_TYPE CORRECT does. XWIN_WORLD/YWIN_WORLD come from the tile WCS, 1-based. SExtractor's fallbacks to the barycentre (FLAGS_WIN 1, 2, 4, 8) are reproduced; 16 marks an unmeasurable object and 32 a centroid still moving after 16 steps. FLAGS_WIN joins final_cat.param, and epoch membership reads XWIN_WORLD as in the SExtractor path. On tile 186.307 it reproduces SExtractor's XWIN_IMAGE exactly when SExtractor does not mask neighbours (0 of 41,425 objects off by more than 0.01 px in windows free of zero-weight pixels), and for 97% of objects on the DR6 catalogue path; the barycentre it replaces is off by 0.16 px at the median and 1.4 px at the 99th percentile. astra.yaml's object_position_columns and background_model say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPvmF43rzKrb5eYPNR3Ki * Windowed centroid: mirror no-data pixels, guard NaN and wild FLUX_RADIUS Review of #933 and the 202.301 run on Nibi: - Pixels that are 0 or not finite hold no data in a tile. They were left out of the background but read as signal in the window. They now take their mirror image through the centre, as SExtractor's MASK_TYPE CORRECT treats pixels under the weight threshold, and a mirror that is itself invalid counts as 0. - Non-finite pixels never enter the background or the sums: samples outside the aperture are zeroed before the product, and a non-finite sum gives FLAGS_WIN 16 at the barycentre. - The second moments are normalised as SExtractor's are after its loop, so a negative source is FLAGS_WIN 4, not 6. - FLUX_RADIUS <= 0 or NaN (DR6 carries values down to -1.2e6) is an empty window, FLAGS_WIN 4; windows wider than MAX_HALF = 256 px are FLAGS_WIN 16; and the working box never exceeds the image. The -12402 radius on 202.301 had built an 84k-px box and been killed at 16 GB. On 186.307 the agreement with SExtractor is unchanged (2.8% of objects off by more than 0.01 px on the DR5 tile, 3.4% on the DR6 path). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsPvmF43rzKrb5eYPNR3Ki * Revert the windowed-centroid port The port re-implemented SExtractor's windowed centroid in numpy for UNIONS-catalogue tiles. Running SExtractor itself on the DR6 image and joining its rows to the DR6 catalogue replaces it (next commits). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Config values expand ${VAR:-default} A value with ${VAR:-default} expands to default when VAR is unset or empty, as in the shell; $VAR and ${VAR} stay strict. The same change as feat/uberseg-seg-vignet's, so the two merge cleanly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sextractor_runner: join detections to an external catalogue MATCH_CATALOGUE (an ASCII SExtractor catalogue of the same image) pairs each SExtractor row with its mutual nearest neighbour within MATCH_RADIUS pixels; paired rows take its NUMBER, unpaired rows leave the catalogue, and a SEG_VIGNET column, when present, is relabelled to the new numbering. Fewer than MATCH_MIN_FRACTION of the rows pairing stops the run: that is what a catalogue measured on other pixels (a DR5 image against DR6) looks like. The join runs before the post-processing keys epochs on NUMBER. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Detect tiles with SExtractor on data too; drop the catalogue converter Data and image simulations both run tile_detect as SExtractor (config_tile_Sx.ini), so XWIN, FLAGS_WIN, the VIGNET neighbour markers and every SExtractor column are measured one way. Under tile_detection: unions_catalogue, tile_get_catalogue fetches only the UNIONS .cat and tile_detect joins its detections to it (SP_MATCH_CATALOGUE), keeping the shared NUMBER and TILE_UNIQUE_ID. On DR6 186.307 all 41,201 detections pair (median separation 4e-5 px); 45 catalogue objects, faint deblended children, have no detection. Gone: read_ext_sexcat and its tests, config_tile_Uc.ini, the runner link, the segmentation-map fetch, merge_final_cat's SEXTRACTOR_ONLY_COLUMNS and --tile-detection, and the detection.catalogue_neighbour_marking decision. completeness.py changes, so this lands at a campaign boundary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sextractor_runner: single-image SExtractor without a detection image Without DETECTION_IMAGE the runner called SExtractor in dual-image mode with the same image and weight twice, which changes the background, MAG_AUTO and FLUX_RADIUS of a fifth of a tile's objects against the single-image call and the UNIONS catalogue. Dual-image mode stays for a real detection image. Closes #936 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * match_catalogue: two-sided pairing guard; record the eight-tile validation The run also stops when fewer than MATCH_MIN_FRACTION of the catalogue objects pair, which catches a detection configuration that finds fewer objects than the catalogue. Tests cover the radius bound and both sides of the guard. astra.yaml records the eight-tile numbers, single-image detection, and the MAG_AUTO differences beyond a pixel-stack overflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Config expansion rejects malformed ${ forms ${VAR-x}, ${VAR:=x} and ${ VAR } passed through as literal paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tile_detect: 1 core, 4 GB, 20 min from eight measured tiles Peak RSS 1.75 GiB and wall time 143 s at most; SExtractor is built without threads. One retry doubles memory and runtime. Image simulations export SP_MATCH_CATALOGUE empty. Stale comments on the catalogue fetch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Update the params pin for this campaign boundary Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tile SExtractor: MEMORY_PIXSTACK 3000000, so no object is truncated 300000 overflowed on four of eight DR6 tiles (a cD galaxy, a bright star's halo, 13 times on Alioth's tile); 3000000 clears all eight. Peak RSS rises to 1.84 GiB and wall time to 160 s on the worst tile; tile_detect's 4000 MB stays at 2.1x. The exposure config never overflows on the 40 CCDs checked and is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * match_catalogue: guard at 2% beyond 20 unpaired per side The guard catches other pixels (DR5 pairs ~95%), not deblending near large objects (DR6 tiles pair at least 99.2%) or a few children on a near-empty edge tile. New key MATCH_TOLERATED_UNPAIRED = 20; MATCH_MIN_FRACTION = 0.98. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * SEG_VIGNET keeps own-vs-other through the DR6 join match_catalogue relabels every SEG_VIGNET label through the whole SExtractor -> DR6 map, unpaired footprints to UNMATCHED_LABEL (-1). That separates each row's own footprint from every other one only if the catalogue's NUMBERs are unique and positive, so the join now refuses a catalogue where they are not. Tests, through sextractor_runner with SExtractor's outputs stubbed: on a crowd where each paired row's DR6 NUMBER is another object's SExtractor NUMBER (one of them the dropped detection's), each own footprint carries the final NUMBER and nothing else does, the dropped footprint is -1, and uberseg_weight and seg_has_neighbour see the stamps as they saw the SExtractor-labelled ones. Sims (no join) keep the check image's stamps. A negative control shows a partial relabel collides. make_cat's exclusion of SEG_VIGNET moves here from the deleted converter tests. Swapping the join before add_seg_vignet, relabelling only paired labels, or dropping the NUMBER guard each fails a test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs * tile_detect: 3000 MB more under uberseg for SEG_VIGNET Measured on DR6 tile 202.301 with sextractor_runner as tile_detect runs it (post-processing off): peak RSS 1.53 GiB under noisefill, 2.93 GiB under uberseg. add_seg_vignet reads the 400 MB check image and adds an int32 column the size of VIGNET, which the join holds twice. Scaled to #933's 1.84 GiB worst tile, uberseg needs ~3.5 GiB, above the 4000 MB noisefill request; 7000 MB is 2x that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs * Update the params pins for UberSeg on #933's detection Regenerated with --update-params-pin in shapepipe-develop-240b37e4.sif with host snakemake 9.23.1. The noisefill pin is #933's params_pin.json byte for byte, so blend_handling: noisefill plans exactly #933's chain; the uberseg default moves in tile_detect and tile_ngmix only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs * tile.smk: keep the catalogue path on one line, as the tile-detection test reads it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs * SEG_VIGNET centres as SExtractor centres VIGNET: (int)(mx + 0.49999) SExtractor 2.25.0 (src/analyse.c, ix=(int)(obj->mx+0.49999); the SIF's Debian 2.25.0+ds-3 does not patch it) truncates the 0-based barycentre plus 0.49999. np.rint differs at exact halves with an odd floor and for fractional parts in (0.5, 0.50001); vignet_centre now mirrors SExtractor. The tests' expected centres are written out by hand from that rule and their stamps cut by plain slicing, not by cut_stamps; they cover exact halves and fractions either side of 0.50001, and an object only the double positions resolve. Reverting to np.rint fails five of them. On DR6 tile 202.301 no object falls where the two rules disagree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs * match_catalogue: refuse NUMBERs the int32 columns cannot hold The guard took unique positive NUMBERs but not their type or range: a non-integer, or a value past 2**31 - 1, narrows on the way into the int32 NUMBER and SEG_VIGNET columns (4294967295 becomes -1, UNMATCHED_LABEL). The join now requires integers in [1, MAX_NUMBER]. Boundary tests cover repeats, 0, negatives, 5.5, 5.0, 2**31, 2**32 - 1 and accept 2**31 - 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs * ngmix: each chunk holds only its own rows' tile stamps Tile_cat memory-maps the tile catalogue (and the segmentation vignet catalogue) and copies the stamps of the chunk's rows only, into a ChunkStamps indexed by tile-catalogue row; rows outside the chunk raise IndexError. The per-object columns (obj_id, ra, dec, flux) stay full length. Ngmix.process passes its ID_OBJ_MIN/ID_OBJ_MAX bounds to Tile_cat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs * Masked-pixel figures for #922: pixel classes on a synthetic and a real DR6 blend, and a defect-type gallery Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * Defect gallery from real DR6 epochs; dr6_epochs.py scans an exposure for every defect type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * ngmix: one weight-symmetrization step, SYMMETRIZE_WEIGHTS names the classes it covers prepare_ngmix_weights sorts each stamp pixel into clean, light-replaced (noise-filled defects, noisefill neighbours) and light-kept-at-zero-weight (interpolated defects, the uberseg neighbour side). One step zeroes the weight on the quarter-turn copies, about the stamp centre, of the light-kept holes that SYMMETRIZE_WEIGHTS names: - interpolated (default): the interpolated defects, as before; - interpolated_and_neighbours: also the uberseg neighbour side (#815); - none. Light-replaced pixels are never symmetrized. Under noisefill + noise every value is a no-op, so the defaults are unchanged. uberseg_weight becomes uberseg_mask, returning the neighbour-side pixels, so the weight map is built once from the pixel classes. Measured with defect_response (6 seeds, 0.5" galaxy, 0.7" PSF, neighbour at 10 px): a lightless neighbour footprint gives c1 = -7.8e-3 under uberseg and 2e-6 symmetrized, but with neighbour light (0.1x flux) c1 goes from +1.1e-3 to +1.2e-2, because the one-sided hole partly offsets the neighbour light in the target's cell. Symmetrizing also keeps 117 of 1580 weighted pixels (97% of S/N) at 10 px and 25 (71%) at 7 px. Hence the neighbour side stays opt-in. tests/science/test_weight_symmetrization.py guards both results; defect_response gains a neighbour and the fit S/N. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * ngmix: interpolation is the only defect fill, and the veto reads the fill's interpolated set Short defect runs are always interpolated and the rest noise-filled; DEFECT_FILL, SYMMETRIZE_WEIGHTS and the three epoch-cut keys are gone, and the radii and the masked fraction are module constants. central_defect_vetoes takes the interpolated set prepare_ngmix_weights uses, removed noisefill neighbours included, so a run that ends on a neighbour is vetoed at the noise-fill radius it gets. A non-finite interpolant raises instead of falling back to noise. Off-tile pixels get zero weight instead of a flag sentinel. Only interpolated holes are weight-symmetrized; the #815 neighbour-side option is dropped. The defect science tests merge into tests/science/test_defect_recovery.py, and the module tests lose their option plumbing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * CI: run tests/science once, in its own step Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * ngmix: one-orientation defect interpolant; interpolated_defects names the set interpolate_defects fits one Clough-Tocher interpolant instead of averaging four quarter-turned ones. For the defects the veto keeps the average moves c by at most 5e-5 and m by at most 4e-4 (test_defect_recovery cases), and the quarter-turn weight copies already make the likelihood symmetric. interpolated_defects(defect, neighbour, blend_handling) is the one rule for which defect pixels are interpolated; prepare_postage_stamps (the central veto) and prepare_ngmix_weights (the fill) both call it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * masked-pixel figures: one defect treatment, off-tile as zero weight pixel_classes compares noisefill and uberseg; defect_gallery shows raw, image and weight under the default. Both take the interpolated set from interpolated_defects, and violet marks only pixels zeroed by the quarter-turn symmetrization. dr6_epochs gives off-tile pixels zero weight and stores the off_tile mask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * Bias-vs-distance figure for one defect: production treatment per defect type, noise fill as counterfactual Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * ngmix: epoch masked-fraction cut at 10%, matching DES EPOCH_MASKED_FRACTION_CUT = 0.10 (was 1/3). The central veto is the bias control; far-field defects bias nothing measured up to 58% of the stamp, so the fraction cut is set for DES Y3/Y6 comparability and as a margin against real-data effects the sims do not model. On tile 202.301 it drops 5.08% of epochs (3.38% at 1/3) and empties 0.55% of objects (0.35%); the extra drops are all CCD-edge or off-tile bands 10-20 px from the object. astra.yaml: ten_percent is the default, one_third excluded with the reason; universes/committed.yaml follows. Tests: the defect-set cut test uses 9.8%/11.8% epochs; corner and edge-neighbour classification tests use geometries under 10%; the off-tile central-veto test lifts the fraction cut to isolate the veto; the far-field recovery cases no longer assert against the cut and are documented as the evidence it is not a bias control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * bias_vs_distance: legend names the masked-fraction cut from its constant Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * ngmix: noise-fill central veto at 13 px, covering the 0.7" galaxy through a 0.9" PSF A noise-filled 4-column cluster (8% of the stamp, kept by the 10% cut) on that galaxy gives m11 = -6.6% at 10 px and first passes at 13 px. The science test checks the cluster and the edge band there; the bias-vs-distance script gains the cluster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * ngmix: DEFECT_WEIGHTING option; noise image interpolated in fixnoise's frame One function, defect_weighting_masks, decides how interpolated defects enter the image and the weight map: fourfold_zero (the PR's design, still the default), des_y6 (pizza-cutter: mask OR its k=1 quarter turn, interpolated at full weight), hole and full. The option reaches ngmix from the run config: `defect_weighting` -> SP_DEFECT_WEIGHTING (tile_ngmix) -> DEFECT_WEIGHTING in config_tile_Ng_template.ini, validated at parse time against the module's DEFECT_WEIGHTINGS; tile_vignets carries it too so a change reruns the whole shape group. Params pin updated (tile_ngmix only): a campaign boundary. Under every option the metacal noise image is now interpolated in the frame ngmix's fixnoise shears it in (rot90 k=1, back with k=3), by the science image's operator. Interpolated in the detector frame, its interpolated pixels sat a quarter turn from the science image's and left an S/N-dependent additive bias (c1 = 2.7e-3 for a 3-px bleed at 7 px on the 0.7"/0.9" galaxy at S/N 30). A test pins ngmix's k=1/k=3 behaviourally. astra: weight_symmetrization becomes shape_measurement.defect_weighting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * ngmix: per-object defect diagnostics in the shape catalogue Over the epochs handed to the fit, each object now carries n_epoch_interp (epochs with an interpolated defect pixel), min_dist_interp and min_dist_noisefill (smallest distance in pixels from the stamp centre to an interpolated / noise-filled defect pixel; -1 when there is none). They read only masks, so distance-split null tests are catalogue queries. make_cat writes them as NGMIX_N_EPOCH_INTERP, NGMIX_MIN_DIST_INTERP and NGMIX_MIN_DIST_NOISEFILL (never-fit rows: 0, -1, -1), and both final_cat.param files list them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo * Plain-language defect_fill rationale; name DEFECT_WEIGHTING's default The defect_fill rationale in astra.yaml now says what a defect is, why its image value matters under metacal, what the fill does, and why. The ngmix template and config.yaml name the default defect weighting, fourfold_zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd * des_y6: interpolate a quarter-turn copy only where the interpolant reaches it A copy wedged in a corner of the clean region (an edge band beside the stamp edge) lies outside the support's hull, so the fill came back NaN and the finiteness guard dropped every epoch of the object. Such copies now keep their light, like copies on already-replaced pixels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd * ngmix: a failed epoch drops that epoch, not the object An exception while building one epoch's ngmix observation (for example the non-finite interpolant guard in prepare_ngmix_weights) used to reach the per-object except, losing every epoch of the object. do_ngmix_metacal now catches it per epoch, drops that epoch and fits the object on the rest; the existing epoch logic (N_EPOCH, masked-fraction cut) decides the rest. An object whose every epoch fails still raises. The drop is kept: Postage_stamp.epoch_failures holds (epoch name, exception), process() logs each at warning level with tile, object and epoch, adds them to the "epoch cuts" tally as failed=, and writes the count as a new per-object column n_epoch_failed -> NGMIX_N_EPOCH_FAILED (final_cat.param for cfis and cfis_image_sims; never-fit rows 0). astra.yaml epoch_masked_fraction_cut records the rule; contract failed-epoch-dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd * ngmix: neighbours are markers on another object's segmentation footprint With MASK_TYPE CORRECT, SExtractor writes -1e30 into an object's tile VIGNET over every above-threshold pixel outside its pre-CLEAN footprint: other detections, but also sub-MINAREA sky blobs and fragments CLEAN merged into the object itself. split_tile_markers took every interior marker as a neighbour, so noisefill noise-filled the object's own light (15% of NEIGHBOUR_FLAG=0 objects differed noisefill vs uberseg on 25 tiles). split_tile_markers(tile_vign, shape, seg, object_number) now keeps as neighbours only markers on a seg label other than 0 and the object's own (the join's -1, a detection that left the catalogue, counts). Off-tile detection is unchanged. The other interior markers become nothing: -1e30 is a flag in the tile VIGNET only, and the epoch stamps hold real light there (the object's own or sky), so they keep weight and image and are neither interpolated nor noise-filled. The narrower neighbour mask feeds the noisefill fill and interpolated_defects consistently; uberseg reads no marker, so its output is bit-identical (tested). Without a seg stamp every interior marker stays a neighbour, and ngmix warns once. tile_detect therefore writes SEG_VIGNET under both blend handlings (SP_SEG_VIGNET exported always; DETECT_SEG_MEM_MB / NGMIX_SEG_MEM_MB unconditional). blend_handling now only moves tile_ngmix's prologue: the uberseg params pin is unchanged and the noisefill pin moves (campaign boundary for noisefill only). astra.yaml blend_handling, defect_fill and pixel-mask text updated; contract neighbours-are-other-footprints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd * ngmix: DEFECT_FILL run-config knob (interpolate | noise) defect_fill joins defect_weighting on the same path: run config `defect_fill` -> Snakefile (validated against ngmix.DEFECT_FILLS) -> tile_ngmix exports SP_DEFECT_FILL -> [NGMIX_RUNNER] DEFECT_FILL -> Ngmix(defect_fill=...) -> prepare_postage_stamps / prepare_ngmix_weights. interpolate (default) is unchanged. noise makes interpolated_defects return no pixel, so every defect is noise-filled, and the central veto, which reads the same set, holds every defect at the 13 px noise-fill radius; defect_weighting then has nothing to act on. astra: defect_fill loses [HARDCODED], noise becomes a selectable (non-default) option, Values pins ngmix.py::DEFECT_FILL. The workflow wiring test covers both defect options. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J2y26yR4NvPvSDJtcbwL3M * tile_detect: build the catalogue in memory, run SExtractor on node-local disk (#947) * make_post_process: write the tile catalogue once make_post_process appended each EPOCH HDU through FITSCatalogue.save_as_fits, which flushed and reopened the ~670 MB uberseg catalogue per exposure, then rebuilt LDAC_OBJECTS for N_EPOCH with add_col, another whole-file write. It now reads the catalogue with astropy, as add_seg_vignet and match_catalogue do, builds the EPOCH HDUs and N_EPOCH in memory, and writes the file once. The columns are laid out by FITSCatalogue.fits_column (add_cols' column builder, now a public static method), so the file is byte-identical to the per-step route: a new test builds the reference through save_as_fits and add_col and compares bytes, and on DR6 191.307 (31,736 rows, 8 epochs) the output is byte-identical to the previous code's. Measured on that tile: 16.4 s -> 5.0 s node-local, 43.7 s -> 28.5 s on idle /scratch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW * tile_detect: run SExtractor and its post-processing node-local On nibi tile_detect took 9-44 min per tile in single-allocation campaigns, of which SExtractor was 1-2 min: the SEG_VIGNET cut, the DR6 join and the post-processing each rewrite the whole catalogue (~670 MB under uberseg) on /scratch, which a loaded node slows ~30x (191.307 in smk-g13-us: 18, 15 and 9 min). Every tile's ngmix waits on this step. sextractor_runner takes an optional WORK_DIR: SExtractor writes its catalogue and check images in a fresh directory under it (sp-detect<num>.*, which tile_local's stale-store sweep also matches), the rewrites happen there, and the files are moved to the output directory once complete. The directory is removed on the way out, also on failure, so a failed run publishes nothing. Without WORK_DIR the runner writes in the output directory as before. config_tile_Sx.ini sets WORK_DIR = /local/scratch, which both profiles bind and the node-local tile store already uses. The ini is not part of params.pre, so the params pins do not move and running campaigns are not invalidated; the published files and the completeness counts are unchanged. Measured on DR6 191.307 under uberseg in one 2-core step: 36 s end to end through WORK_DIR against 112 s in place on idle /scratch (44 min in the loaded g13-us allocation). Catalogue, BACKGROUND and SEGMENTATION check images are identical to the in-place run's except SExtractor's own extraction time cards (SEXTIME, SEXELAPS) in LDAC_IMHEAD. tests/module/test_sextractor_work_dir.py checks the staged run publishes the in-place bytes, leaves the work directory empty, and publishes nothing when the join refuses the catalogue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * Regenerate params pins after merging develop Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GPLdkKZzVveSKyCw4DGi5X * test_sextractor_seg_vignet: #925's final-catalogue test on #942's read_sextractor_data (cherry picked from commit 5b0b357) (cherry picked from commit de2b00f) * Regenerate params pins after merging feat/masked-pixels * params pins regenerated for feat/uberseg-seg-vignet on develop 902c32f --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
tile_detectfast on a loaded shared filesystem. Every tile's ngmix waits on this step, and on Nibi it took 9–44 min per tile, about 13% of a campaign's bill. No single step was slow. The problem was that each step rewrote the whole 672 MB catalogue on NFS.make_post_processbuilds the catalogue in memory and writes it once. Before, it saved and reopened the file for every epoch, then rebuilt it again for N_EPOCH.sextractor_runnertakes an optionalWORK_DIR, set to/local/scratchinconfig_tile_Sx.ini. SExtractor, the check images and the rewrites all run on node-local disk. Each file moves to the output directory once, and a failed run publishes nothing.Result: tile 191.307 took 36 s node-local, against 112 s on idle
/scratchand 44 min in the g13 campaign. Post-processing output is byte-identical to before. The other outputs match too, except SExtractor's timestamp header cards.Based on #925, which adds the
SEG_VIGNETfile handling this rewires; retarget to develop once #925 lands.Claude Opus 5.5 on behalf of Cail
🤖 Generated with Claude Code
https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd