Skip to content

Commit 84e40a5

Browse files
martinkilbingermartinkilbingercailmdaleyclaude
authored
Image simulations workflow (#766)
* testing/simulate: added Pujol denoising estimator of shear bias * Running SP v2.0 on image simulations, up to -j 8 * SP v2.0 running on image simulations including ngmix * Running on im sims incl final cat * ÃÃÂadded datetime to Mh run * Âinit v2.0: cd in new subdir * ÂÂimage sims run updates * Âsmall changes * output mean e to ngmix log * doc * ÂÂrun script : added --retry * im_sims docker deployment * added .py to executables * fixe pyproject bug * pyproject moved back to exe scripts without extension * running create_final_cat.py on image_sims * Pipeline running on image sims * config files: change batch size to 1 (default) * snakemake setup for image simulations * fixed snakefile * test for zero size FITS output files * zero-size output fixes * more automatisation for image sims snakemake * ngmix+vignets: handling of empty input PSF cats; scripts: improved snakemake running * retarget(im_sims): revert incidental stale-base edits to shared modules Keep only genuine sims-enablement; revert shared-module changes that were side-effects of the stale ngmix_v2.0 base and collide with in-flight work: - make_cat: restore HSM_G*_PSF grammar (fix/ngmix-psf-columns) + spread_model (714847c, split-761); drops the input_file_List NameError - vignetmaker: drop broad try/except churn (_w_log AttributeError -> test fail) - mask: restore CDSCLIENT_PATH knob (live in 4 example configs) - restore pipeline_tutorial.md (toc.rst ref -> Sphinx build) + run_scratch_local.sh Verification branch only; not for push. Grounds the #766 review. * retarget(im_sims): slim to ShapePipe module enablement; orchestration -> sp_validation The image-sims workflow splits by concern along the repo dependency arrow (shapepipe produces catalogues; sp_validation consumes + validates them): - ShapePipe (this PR): only the pipeline-module changes needed to run on background-free sims -- the fake_psf module, ngmix BKG_SUB path, numeric sim-exposure-ID handling (exp_utils/sextractor/merge_headers), the cfis_image_sims module configs, create_final_cat's image_sims mode, and make_data_pujol. - sp_validation (separate PR): the snakemake orchestration + the merge/calibrate/m-bias half, rebuilt on sp_validation's native container workflow. The orchestration files removed here are preserved in git history (origin/im_sims) and move there. Incidental stale-base edits to shared modules/standard configs/tests reverted to ngmix_v2.0 (which already carries the fixes: 662fad4 HSM g-cols, 714847c spread_model). Retargeted onto ngmix_v2.0; uv.lock regenerated. * fix(im_sims): restore image-sims job dispatch in the ShapePipe runner The ngmix_v2.0 retarget slimmed run_job_sp_canfar_v2.0.bash and job_sp_canfar_v2.0.bash of their image-sims code paths, leaving #766 unable to run the image-sims pipeline at all: run_job had no -t/--type dispatch (so the sp_validation workflow's `-t image_sims` was silently swallowed and it ran standard-DATA mode), and its job-1024 completeness check referenced ngmix_interp_runner, a runner that exists nowhere in the codebase. job_sp_canfar rejected `-r symlink` ("vos only for v2.0"), named the Gie and make_cat configs for the data path only, and still invoked shapepipe_run.py (the console entry point is `shapepipe_run`, no extension, since 797eeff). run_job: restored the canonical image-sims runner (full -t/--type dispatch — cfis_image_sims config dir, symlink retrieve, tile_det=sx, faked Uz weight-uncompress, fake_psf under job 512, job-2048 make_cat, and the correct ngmix_runner check). Made it relocatable per #766's goal: a self-derived $SP_ROOT replaces ~/shapepipe for sourcing functions.sh/job_list_help.bash and resolving config dirs, and job_sp_canfar is invoked by explicit repo path (it is not on the container PATH). run_tile_job now re-anchors cwd to the tile work dir on entry: SP_RUN/SP_CONFIG derive from cwd, and run_exp_job leaves cwd in the last exposure dir, so when run_job is called once for the whole pipeline (-j 4095, as the workflow does) the post-exposure tile jobs would otherwise execute in the stale exposure dir. job_sp_canfar: kept the ngmix_v2.0 base — the #746 NUMBER_LIST mechanism that matches the new-grammar container (the canonical runner still uses the retired shapepipe_run -e flag and would crash) — and grafted the image-sims bits: accept `-r symlink`, config_exp_Gie_$retrieve, config_tile_Mc_$psf (the config present in cfis_image_sims and matching run_job's job-2048 check), the bare `shapepipe_run` entry point, and a self-derived $SP_ROOT. Verified on candide against the post-#761 container: `-t image_sims` selects cfis_image_sims + symlink + sx and runs Git -> faked-Uz -> Fe -> Gie(7/7) -> Sp -> Ma -> Mh_exp, each in its correct work dir with new-grammar shapepipe_run. The two scripts run from the bind-mounted repo, so the container supplies only the Python stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Jd85D8xNZZ43FgesabSnk * fix(im_sims): add SM_DO_CLASSIFICATION to the image-sims make_cat config The image-sims make_cat config lacked SM_DO_CLASSIFICATION, which the newer make_cat_runner reads unconditionally (NoOptionError -> job 2048 crashed, so the per-tile final_cat was never built). The image-sims pipeline has no spread_model_runner stage (make_cat inputs are sextractor + fake_psf + ngmix only), so there is nothing to classify: set it False. With this, a single tile completes end-to-end (Git -> ... -> ngmix -> make_cat), producing a 2345-object final_cat-233-293.fits against the post-#761 container. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Jd85D8xNZZ43FgesabSnk * fix(im_sims): new-grammar final_cat.param for the image-sims pipeline create_final_cat selects this column list from the per-tile make_cat output into final_cat_{sim}.hdf5; the list was still old-grammar (NGMIX_ELL_*, NGMIX_Tpsf_*, NGMIX_T_PSFo_*, NGMIX_MOM_FAIL) while the post-#761 make_cat writes NGMIX_G{1,2}_*, NGMIX_T_PSF_ORIG/RECONV_*, so the merge died on 'NGMIX_Tpsf_2M' is not in list. Rewrite to the v2 grammar, mirroring example/cfis/final_cat.param minus the two columns the simulation pipeline does not emit (IMAFLAGS_ISO: no imaging-flag masking stage; NGMIX_MOM_FAIL: superseded by NGMIX_MCAL_TYPES_FAIL, which is included). Verified: all 65 columns present in the per-tile cat, merge -> extract -> calibrate runs clean on 1z2z_grid_1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D9atA9VPrg7VMC7oBArny9 * fix(mask): don't crash stdout handling when star masking is off With HALO_MAKE and SPIKE_MAKE both False (the image-sims mask config), find_stars never runs, so _CDS_stdout is never set and the unconditional read at the end of make_mask raised AttributeError for every CCD. The flag files were already written, so each mask stage 'failed' with 40 bookkeeping errors and exit 1, relying on the runner's continue-on-error to keep the pipeline alive. Guard the CDS block with hasattr, matching the WeightWatcher/rm blocks below it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TWURLMoixhuMqVrPVdRmCp * ngmix: position-seeded per-object RNG for Pujol noise cancellation (#796) Add an opt-in SEED_FROM_POSITION knob to the ngmix module (default False). When on, the object loop replaces the tile-level RNG with a per-object RandomState seeded from the object's sky position, and rebuilds the prior from that same RNG, so every stochastic step of the fit is a deterministic function of (ra, dec, ccd). Why: image-sim m-bias uses the Pujol estimator — the same scene under different applied shears, with shear response read from the branch difference of the SAME objects. Metacal's fixnoise adds a counter-noise realisation from an RNG seeded per tile, so an object gets different added noise in different branches (detection order differs) and the noise fails to cancel in the difference, inflating sigma_m. Seeding per object from position makes the same object draw the same added noise AND the same fit guesses in every branch, so both cancel and the m-bias error shrinks. Design and box math are Fabian's issue #796. The prior must be rebuilt per object, not just the RNG: the ngmix guesser draws its initial guess via prior.sample(), which consumes the RNG the prior was CONSTRUCTED with (ngmix 2.4.0 guessers.py / joint_prior.py). A per-object RNG alone would leave the guess drawing from the shared tile stream and break cancellation. The fixnoise counter-noise itself rides on the observation's attached noise image (use_noise_image=True), which prepare_ngmix_weights draws from the per-object RNG — so no fresh draw from the bootstrapper RNG is needed. Deviation from #796: Fabian combines the 3-arcsec position boxes as box_x + box_y, which collides along anti-diagonals — boxes (10,20) and (11,19) both give seed 30, so two distinct objects share a noise stream. This replaces the sum with a Cantor pairing (a bijection on the non-negative integers) after a zig-zag fold that maps signed box indices (Dec can be negative) onto non-negative ones, reduced mod 2**32 into the valid RandomState seed range. The box math is kept exactly as his: floor(ra*3600/3)+(ccd+1), floor(dec*3600/3)+(ccd+2), first-epoch coordinates (coord_list[0] convention), so all epochs of one object share the stream. Off by default => production path is byte-identical to develop (verified: metacal g for all five types matches unmodified develop bit-for-bit; the full fast suite passes, 326 tests). New property tests cover the seed's range, sub-box jitter invariance, adjacent-box distinctness, and the resolved anti-diagonal collision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9qEVRCWEekT5ACNB3rjkq * sims config: enable SEED_FROM_POSITION for the A/B noise-cancellation run Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvQvyJTEwYcLCwDEgpWMJ6 * running smoother on im sims * Fix nfev_fit int64 dtype lock crashing batch-saved ngmix output compile_results filled nfev_fit with np.nan on failed fits, the only int-like column with a float fallback. With SAVE_BATCH enabled, save_results locks column dtypes from batch 1; if that batch has no failures the column locks to int64, and the first later batch with a failure feeds NaN into it, raising "cannot convert float NaN to integer" on append. Use ngmix's own -1 failed/absent-nfev sentinel instead of NaN, keeping the column int64 across every batch. Fixes #795 * im sims: coverage scan script for (near-)empty input tiles Companion to the coverage rule in the Snakefile: scans input weight maps (subsampled), writes tile_coverage_grid_{num}.yaml with an exclude list read at DAG-build time. Weight-map location is taken from the get-images ini of the 1p2z variant. Plus two felt fibers documenting the blank-tile and thread-exhaustion findings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * restore apptainer_noslurm.sh dropped in im_sims_candide merge The image_sims_pipeline Snakefile invokes all container commands through this wrapper; it strips SLURM/PMI/PMIX/OMPI env vars that crash OpenMPI inside apptainer on SLURM compute nodes. No replacement exists in run_scratch_local.sh or run_job_sp_canfar_v2.0.bash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Retire in-repo image_sims Snakemake orchestrator The image-simulation pipeline's orchestration now lives in sp_validation (companion PR #225), which drives ShapePipe via `run_job -t image_sims --config_dir`. The in-repo Snakemake driver under scripts/image_sims_pipeline/ (Snakefile, info.py, monitor_mbias.py, config.yaml.template) is superseded and removed to keep a single source of orchestration truth. Its coverage-scan helper, check_tile_coverage.py, is the one piece of reusable logic with no equivalent elsewhere; it moves to scripts/python/ alongside the other run-directly helpers (create_final_cat.py, etc.) and gains the executable bit to match that directory's convention and its own shebang. docs/source/image_sims_calibration.md documented only this retired orchestrator (its Snakemake targets, info.py, monitor_mbias.py) and was an orphan page not wired into any Sphinx toctree; it is removed with the code it described. History of the old pipeline is preserved in the .felt/ notes, which are left untouched as a chronological record. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF * run_job: add -c/--config_dir to override computed config dir The runner self-computes config_dir from --type (example/cfis for data, example/cfis_image_sims for image_sims) and threads it through the cfis symlinks, SP_CONFIG, and job_sp_canfar's own -c. That hardcodes the .ini tree to the in-repo example dirs. Add an optional -c/--config_dir that, when given, overrides the computed config_dir uniformly for any --type. The override is applied after the type branch, so when the flag is absent config_dir is byte-for-byte the historical value and every existing caller (test_tile_det.py, functions.sh, the sp_validation im_pipeline rule) is unaffected. This lets the sp_validation workflow point every stage at an explicit .ini tree, per the image-sims boundary rule (workflow selects and parameterizes the tree). The flag name matches job_sp_canfar_v2.0.bash's own -c/--config_dir. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF * fake_psf: source PSF dictionary path from the environment The cfis_image_sims fake_psf config hardcoded a collaborator-specific absolute path to the SKiLLS PSF dictionary, coupling the in-repo config tree to one machine's layout. Replace it with PSF_DICT_PATH = $PSF_DICT, which the config layer's getexpanded (os.path.expandvars) resolves from the environment. The orchestrating workflow now exports PSF_DICT from its psf_dict config key, so the campaign fact lives upstream in the manifest, not in this repo -- matching the module-config/workflow boundary rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF * docs(cfis_image_sims): document image-sims job-bit dispatch Add a README to example/cfis_image_sims/ giving the accurate, per-bit mapping of the image-sims chain: for each job bit (1..2048), the ShapePipe module, the .ini file job_sp_canfar_v2.0.bash selects under sim settings, and the special-casing run_job_sp_canfar_v2.0.bash applies. Every row is derived by reading the two bash scripts, so the table stays a truthful reference for the sims path rather than a guess. Captures the parts that surprise: the faked weight-uncompress at bit 2, the placeholder bit 64 with fake_psf actually running inside bit 512, the dashed tile-id convention, forced tile_det=sx, and symlink retrieval. Adds the parallelism model (per-tile fan-out, in-job SMP, no MPI) and the $PSF_DICT env requirement for fake_psf. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF * im sims Snakefile small change * improved reading of merged hdf5 cat for info * added comment * fix(packaging): drop stale init_run_v2.0.sh from script-files The [tool.setuptools] script-files list referenced scripts/sh/init_run_v2.0.sh, which was removed in b0804ce when the im_sims retarget moved orchestration to sp_validation. setuptools' build_editable errors on the missing file, breaking every Docker build in CI (`uv pip install --no-deps -e .` → "file '...init_run_v2.0.sh' does not exist"). The other three listed scripts (update_runs_log_file.py, run_job_sp_canfar_v2.0.bash, job_sp_canfar_v2.0.bash) still exist, so this removes only the stale entry. Verified with an editable-install dry-run in the im_sims runtime container: resolves to shapepipe-1.1.0 with no build_editable error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(im_sims): strip MKDEBUG markers; refresh check_tile_coverage docstring Polish over the image-sims files this PR touches: - check_tile_coverage.py: correct the module docstring's stale "the Snakefile reads the exclude list" wording — orchestration moved to sp_validation (f01344a), so the caller is now the sp_validation workflow, not an in-repo Snakefile. - bin/shapepipe_run.py: drop the "MKDEBUG start/end" print statements that wrapped main(). - ngmix.py: drop two "# MKDEBUG TODO this needs to be checked!" tags appended to the zero-sigma / zero-noise weight guards; the preceding comments already explain why the guards exist. Only strips MKDEBUG introduced by the im_sims merges; the pre-existing marker in create_final_cat.py (also on develop) is left untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: untrack .felt fiber notes that leaked in via im_sims Two private fiber notes (blank-input-tiles-grid-4, snakemake-crash- runtimeerror) were committed on origin/im_sims in 9783a2e, before the develop-side .felt untrack (3d56ec6) landed there, so they survived the merge into this branch. The repo gitignores /.felt/ and treats it as a machine-local symlink into the private loom store (see CLAUDE.md); these tracked files bypass that ignore. Untrack them (kept on disk) so no fiber content rides into the shared CosmoStat repo. Their operational lore is preserved in sp_validation's workflow docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: martinkilbinger <martinkilbinger@cea.fr> Co-authored-by: Cail Daley <cail.daley@cea.fr> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent da7a533 commit 84e40a5

45 files changed

Lines changed: 3138 additions & 286 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
233.293
2+
237.292
3+
238.292
4+
239.291
5+
239.292
6+
239.293
7+
240.291
8+
240.292
9+
240.293
10+
241.293
11+
242.290
12+
242.291
13+
242.292
14+
242.293
15+
243.290
16+
243.291
17+
243.292
18+
243.293
19+
244.293
20+
245.290
21+
245.291
22+
245.292
23+
246.290
24+
246.291
25+
247.289
26+
247.291
27+
247.292
28+
247.293
29+
248.288
30+
248.289
31+
248.290
32+
248.291
33+
249.289
34+
249.290
35+
249.291
36+
249.292
37+
250.289
38+
250.290
39+
250.292
40+
251.287
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
251.288
2+
251.289
3+
251.291
4+
252.287
5+
252.288
6+
252.289
7+
252.291
8+
253.287
9+
253.288
10+
253.291
11+
254.286
12+
254.287
13+
254.288
14+
255.286
15+
255.288
16+
255.290
17+
256.286
18+
256.287
19+
257.285
20+
257.286
21+
257.287
22+
257.288
23+
258.287
24+
258.289
25+
259.285
26+
259.286
27+
259.287
28+
259.288
29+
259.289
30+
260.284
31+
260.285
32+
260.286
33+
260.287
34+
260.288
35+
260.289
36+
261.284
37+
261.285
38+
261.286
39+
261.287
40+
261.288
41+
262.285
42+
262.286
43+
262.287
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
262.288
2+
263.283
3+
263.284
4+
263.285
5+
263.286
6+
264.283
7+
264.284
8+
264.286
9+
264.287
10+
265.284
11+
265.285
12+
265.286
13+
265.287
14+
266.282
15+
266.283
16+
266.285
17+
266.286
18+
267.283
19+
267.284
20+
267.285
21+
267.286
22+
268.282
23+
268.283
24+
268.284
25+
268.285
26+
269.281
27+
269.282
28+
269.283
29+
269.284
30+
269.286
31+
270.281
32+
270.282
33+
270.283
34+
271.281
35+
271.282
36+
271.283
37+
271.284
38+
271.285
39+
272.281
40+
272.282
41+
272.283
42+
272.285
43+
273.280
44+
273.282
45+
273.283
46+
273.284
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
274.279
2+
274.280
3+
274.282
4+
274.283
5+
275.279
6+
275.280
7+
275.282
8+
275.283
9+
276.279
10+
276.280
11+
276.281
12+
276.282
13+
276.283
14+
277.279
15+
277.281
16+
277.282
17+
278.279
18+
278.281
19+
278.282
20+
279.281
21+
279.282
22+
280.280
23+
280.281
24+
280.282
25+
281.280
26+
281.281
27+
282.280
28+
282.281
29+
283.280
30+
284.279
31+
285.279
32+
285.280
33+
286.279
34+
286.280
35+
287.279
36+
288.279
37+
289.279

‎auxdir/CFIS/im_sims_202606/tile_numbers_5.txt‎

Whitespace-only changes.

‎example/cfis_image_sims/README.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# cfis_image_sims — module configs for ShapePipe on simulated tiles
2+
3+
This directory is the single `.ini` tree for image-simulation runs. It is
4+
selected by `run_job_sp_canfar_v2.0.bash -t image_sims` (which sets
5+
`config_dir = example/cfis_image_sims`, `retrieve = symlink`, and forces
6+
`tile_det = sx`) and consumed stage-by-stage by `job_sp_canfar_v2.0.bash`.
7+
8+
The pipeline runs as a bit-coded chain of jobs. `run_job_sp_canfar_v2.0.bash`
9+
loops over the bits set in `-j`, initialises the tile/exposure work directories,
10+
runs each completeness check, and delegates the actual `shapepipe_run` calls to
11+
`job_sp_canfar_v2.0.bash -j <bit>`, which is where the `.ini` file for each bit
12+
is selected. A few bits are special-cased for sims *before* that delegation —
13+
those cases are documented in the last column below and, at more length, under
14+
[Sim special-casing](#sim-special-casing).
15+
16+
## Job-bit dispatch
17+
18+
Every row is derived from the two bash scripts. "Module(s)" is the ShapePipe
19+
runner(s) the selected `.ini` names; "`.ini` selected" is what
20+
`job_sp_canfar_v2.0.bash` picks for that bit under sim settings
21+
(`retrieve=symlink`, `psf=psfex`, `tile_det=sx`, `star_cat_for_mask=onthefly`).
22+
23+
| Bit | Stage | Module(s) | `.ini` selected (sim settings) | Sim special-casing |
24+
|----:|-------|-----------|--------------------------------|--------------------|
25+
| 1 | retrieve tile image + weight | `get_images_runner` | `config_tile_Git_symlink.ini` | symlink retrieval (see below); completeness check expects 2 files (image + weight) vs. 4 for data |
26+
| 2 | uncompress tile weight | *(none — faked)* | *(none)* | **Faked weight-uncompress.** run_job skips the `config_tile_Uz.ini` run entirely; sim weights are already uncompressed, so it fabricates a `run_sp_tile_Uz*/uncompress_fits_runner/output` dir and symlinks `input_tiles/CFIS_simu_weight-<ID-dashed>.fits` into it, satisfying downstream `last:uncompress_fits_runner` lookups |
27+
| 4 | find exposures | `find_exposures_runner` | `config_tile_Fe.ini` | — |
28+
| 8 | retrieve exposure images | `get_images_runner` | `config_exp_Gie_symlink.ini` | symlink retrieval; completeness check expects 3 files vs. 6 for data |
29+
| 16 | split exposures, merge WCS headers | `split_exp_runner` | `config_exp_Sp.ini` | — |
30+
| 32 | mask exposures | `mask_runner` | `config_exp_Ma_onthefly.ini` | — |
31+
| 64 | exposure PSF model | *(none — placeholder)* | *(none)* | **Placeholder.** For data this runs full exposure PSF modelling. For sims run_job writes a placeholder log and does nothing here; the sim PSF (`fake_psf_runner`) actually runs inside bit 512 |
32+
| 128 | merge exposure WCS headers → tile sqlite log | `merge_headers_runner` | `config_tile_Mh_exp.ini` | — |
33+
| 256 | object detection on tiles | `sextractor_runner` | `config_tile_Sx_nomask.ini` | `tile_det` is forced to `sx`, so the SExtractor-no-mask branch is always taken (the `uc` external-catalogue branch is never reached for sims) |
34+
| 512 | fake PSF + postage stamps | `fake_psf_runner`, then `vignetmaker_runner` ×2 | `config_exp_psfex.ini` (fake PSF), then `config_tile_PiViVi_canfar_sx.ini` (vignets) | **Two sub-runs.** run_job first calls the job script with `-j 64` → `config_exp_psfex.ini`, which despite its name runs `fake_psf_runner` (needs the sexcat from bit 256; run dir `run_sp_tile_fpsf`), then `-j 512` → `config_tile_PiViVi_canfar_sx.ini` for the two `vignetmaker_runner` runs. Data instead runs `psfex_interp_runner` + vignets here |
35+
| 1024 | multi-epoch shape measurement | `ngmix_runner` | `config_tile_Ng_batch_psfex_sx.ini` | — |
36+
| 2048 | create final catalogue | `make_cat_runner` | `config_tile_Mc_psfex.ini` | — |
37+
38+
Notes on cross-cutting conventions used above:
39+
40+
- **Dashed tile-id convention.** Data tile IDs use dot format (`233.293`); sim
41+
input files use dash format (`233-293`). run_job writes `tile_numbers.txt` in
42+
dash form for `image_sims` (`${ID//./-}`), and the faked bit-2 weight symlink
43+
and sim input patterns all key off the dashed ID.
44+
- **Symlink retrieval.** `-t image_sims` sets `retrieve=symlink`, so bits 1 and 8
45+
select the `*_symlink.ini` get-images configs (`RETRIEVE = symlink`), which
46+
link sim images out of `$SP_DIR/input_tiles` / `$SP_DIR/input_exp` rather than
47+
downloading from VOSpace.
48+
- **Forced `tile_det=sx`.** The `image_sims` type branch forces `tile_det=sx`, so
49+
bit 256 always runs SExtractor detection and bit 512 selects the `_sx`-suffixed
50+
vignet/ngmix/PiViVi configs.
51+
52+
## Sim special-casing
53+
54+
Three bits diverge from the data chain, all handled in
55+
`run_job_sp_canfar_v2.0.bash` before the per-bit delegation:
56+
57+
- **Bit 2 — faked weight-uncompress.** Sim weights ship uncompressed, so instead
58+
of running `uncompress_fits_runner`, run_job fabricates the expected Uz output
59+
directory and symlinks `CFIS_simu_weight-<ID-dashed>.fits` in, so downstream
60+
`last:uncompress_fits_runner` references resolve.
61+
- **Bit 64 — placeholder.** run_job writes a placeholder completeness log and
62+
runs nothing; the sim PSF is produced by `fake_psf_runner` inside bit 512.
63+
- **Bit 512 — fake PSF + vignets.** run_job runs `fake_psf_runner`
64+
(`config_exp_psfex.ini`) followed by the two `vignetmaker_runner` runs
65+
(`config_tile_PiViVi_canfar_sx.ini`). `fake_psf_runner` consumes the sexcat
66+
from bit 256, so bit 256 must precede it.
67+
68+
## Parallelism model and the `PSF_DICT` requirement
69+
70+
The orchestrator (the sp_validation Snakemake workflow) fans out one job per
71+
tile; within a single job ShapePipe uses its own multiprocessing, controlled by
72+
`-N/--N_SMP` (threaded through to each module's `SMP_BATCH_SIZE`). Every sim
73+
config sets `MODE = SMP` — the sims chain launches no MPI.
74+
75+
`fake_psf_runner` reads the sims' pickled PSF dictionary from the `$PSF_DICT`
76+
environment variable (`config_exp_psfex.ini` sets `PSF_DICT_PATH = $PSF_DICT`,
77+
expanded from the environment). The orchestrating workflow exports it (from its
78+
`psf_dict` config key); on candide it points at the SKiLLS PSF dictionary. Bit
79+
512 will fail if `$PSF_DICT` is unset.
Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# ShapePipe configuration file for: get images
2+
3+
4+
## Default ShapePipe options
5+
[DEFAULT]
6+
7+
# verbose mode (optional), default: True, print messages on terminal
8+
VERBOSE = False
9+
10+
# Name of run (optional) default: shapepipe_run
11+
RUN_NAME = run_sp_exp_Gie
12+
13+
# Add date and time to RUN_NAME, optional, default: False
14+
RUN_DATETIME = True
15+
16+
17+
## ShapePipe execution options
18+
[EXECUTION]
19+
20+
# Module name, single string or comma-separated list of valid module runner names
21+
MODULE = get_images_runner
22+
23+
# Parallel processing mode, SMP or MPI
24+
MODE = SMP
25+
26+
27+
## ShapePipe file handling options
28+
[FILE]
29+
30+
# Log file master name, optional, default: shapepipe
31+
LOG_NAME = log_sp
32+
33+
# Runner log file name, optional, default: shapepipe_runs
34+
RUN_LOG_NAME = log_run_sp
35+
36+
# Input directory, containing input files, single string or list of names
37+
INPUT_DIR = $SP_RUN
38+
39+
# Output directory
40+
OUTPUT_DIR = $SP_RUN/output
41+
42+
43+
## ShapePipe job handling options
44+
[JOB]
45+
46+
# Batch size of parallel processing (optional), default is 1, i.e. run all jobs in serial
47+
SMP_BATCH_SIZE = 1
48+
49+
# Timeout value (optional), default is None, i.e. no timeout limit applied
50+
TIMEOUT = 96:00:00
51+
52+
53+
## Module options
54+
55+
# Get exposures
56+
[GET_IMAGES_RUNNER]
57+
58+
INPUT_MODULE = last:find_exposures_runner
59+
60+
FILE_PATTERN = exp_numbers
61+
62+
FILE_EXT = .txt
63+
64+
# NUMBERING_SCHEME (optional) string with numbering pattern for input files
65+
NUMBERING_SCHEME = -000-000
66+
67+
68+
# Paths
69+
70+
# Output path (optional, default is [FILE]:OUTPUT_DIR
71+
# OUTPUT_PATH = input_images
72+
73+
# Input path where original images are stored. Can be local path or vos url.
74+
# Single string or list of strings
75+
INPUT_PATH = $SP_DIR/input_exp/,$SP_DIR/input_exp/,$SP_DIR/input_exp/
76+
77+
# Input file pattern including tile number as dummy template
78+
INPUT_FILE_PATTERN = simu_image-0000000,simu_weight-0000000,simu_flag-0000000
79+
80+
# Input file extensions
81+
INPUT_FILE_EXT = .fits, .fits,.fits
82+
83+
# Input numbering scheme, python regexp
84+
INPUT_NUMBERING = \d{7}
85+
86+
# Output file pattern without number
87+
OUTPUT_FILE_PATTERN = image-,weight-,flag-
88+
89+
# Method to retrieve images, one in 'vos', 'symlink'
90+
RETRIEVE = symlink

0 commit comments

Comments
 (0)