Merge Ralph/LISA work into calmarg#7
Merged
oshaughnessy-junior merged 10 commits intoJun 12, 2026
Merged
Conversation
Third pass of the hyperpipeline-format work. The first two commits standardised the ILE -> CIP shard chain (commit #1) and the CIP -> ILE grid handoff inside create_event_parameter_pipeline_BasicIteration plus the puff / fetch / dag_utils plumbing (commit #2). This commit makes util_RIFT_pseudo_pipe.py -- the standard wrapper that builds args files and then invokes BasicIteration -- respect the same env-var flag, so end-users can flip the entire wrapper-driven workflow over to .dat format with one environment variable. Also includes the test/test_hyperpipeline_io.py file, which the prior two commits referenced ("12 tests"/"17 tests") but did not actually include in the staged file set. Design constraint ----------------- Per the project policy of "operate cohesively in one mode or the other -- no internal conversion": pseudo_pipe is a thin suffix-substituting wrapper. It does NOT convert XML to .dat (or vice versa) for any input. Upstream inputs (manual seed grids, template-bank-derived grids, etc.) must be staged in the format matching the active mode; pseudo_pipe refuses with a clear message when an XML-only auto-generation path would otherwise produce a file the rest of the workflow can't consume. Files ----- * bin/util_RIFT_pseudo_pipe.py Five surgical patches, all gated on _use_hpip_pp derived from the same RIFT_HYPERPIPELINE_FORMAT env var commits #1/#2 use: - Three new variables (_use_hpip_pp, grid_suffix_pp, sim_grid_flag_pp) defined once near the top, immediately after the RIFT_LOWLATENCY block. Mirrors the BasicIteration placement so the two scripts have parallel structure. - target_params writer (~line 639): in hyperpipeline mode, writes target_params.dat via hyperpipeline_io.write_grid_from_P_list with a column set auto-derived from whether P.eccentricity / P.meanPerAno are nonzero. Otherwise legacy ChooseWaveformParams_array_to_xml emits target_params.xml.gz. No behavioural change in legacy mode. - command-single --sim-xml line (~813): swapped to "{sim_grid_flag_pp} target_params.{grid_suffix_pp}", so the sanity-check ILE invocation routes through ILE's --sim-grid path in hyperpipeline mode. This is the path the --sim-grid reader patch from commit #2 was designed for. - --manual-initial-grid copy site (~line 1399): copies to proposed-grid.{grid_suffix_pp} regardless of mode. shutil.copyfile is format-agnostic; the source file's format must match the active mode (per the design constraint above). The --manual-initial-grid-supplements branch (which uses ligolw_add, XML-only) raises SystemExit in hyperpipeline mode with a clear message pointing the user at pre-merging supplements upstream. - --input-grid argument to create_event_parameter_pipeline_BasicIteration (~line 1418): now passes proposed-grid.{grid_suffix_pp}, threading the suffix through to BasicIteration so the two scripts agree on the seed-grid filename. - AMR / template-bank seed-grid auto-generation guard (~line 1506): in hyperpipeline mode without --manual-initial-grid, raises SystemExit with a message asking the user to stage the initial grid as .dat and pass via --manual-initial-grid. The XML-emitting util_AMRGrid.py and util_GridSubsetOfTemplateBank.py are intentionally untouched -- per the design constraint, no internal conversion. - --manual-initial-grid argparse help text updated to advertise both suffixes and note that the source format must match the active mode. * test/test_hyperpipeline_io.py Recovered from the prior two commits, which referenced this file in their commit messages ("12 tests" in commit #1, extended to "17 tests" in commit #2) but did not include it in the staged file set. The file is otherwise byte-identical to the version exercised end-to-end during the prior commits' development. 17 tests: - default_roundtrip - eccentricity_columns - tides_with_eos_index - to_legacy_dat_default - legacy_column_indices_consistency - sniff_distinguishes_legacy - sniff_recognizes_new_format - env_flag - concatenated_shards - read_many_skips_empties_and_mismatches - consolidate_weighted_average - consolidate_drops_high_sigma - grid_write_read_roundtrip_with_units - grid_distance_unit_conversion - grid_auto_suffix_append - column_alias_bridge - grid_no_lal_module_passthrough The file uses an importlib direct-load shim so it runs in stripped- down environments (no lalsuite / scipy required), making it usable for CI on minimal containers. Audit ----- A full pass over util_RIFT_pseudo_pipe.py confirmed every remaining xml.gz / --sim-xml string is one of: * a comment / variable definition / argparse help mentioning the pair of supported suffixes (lines 34, 35, 47, 48, 321, ...); * a defaults string for an external file (PSD, coinc, ini) that is legitimately external and stays XML; * inside a code path I gated to refuse-and-exit in hyperpipeline mode (the AMR seed-grid block at ~line 1556 -- unreachable when _use_hpip_pp is True). No live XML I/O paths reachable in hyperpipeline mode remain. Tests ----- All 17 tests in test/test_hyperpipeline_io.py pass. Every patched file in this commit and its dependents (commits #1, #2) compile via py_compile. Followups --------- This commit covers util_RIFT_pseudo_pipe.py only. Sibling drivers that still need the same treatment: * bin/cepp_basic_htcondor (htcondor-only twin of BasicIteration) * bin/util_RIFT_pseudo_pipe_lowlatency.py * bin/util_RIFT_hyperpipe.py To round out the seed-grid auto-generation paths so hyperpipeline mode no longer requires --manual-initial-grid, the underlying generators need parallel hyperpipeline output support: * bin/util_AMRGrid.py * bin/util_GridSubsetOfTemplateBank.py * bin/helper_LDG_Events.py The EXTR_out -> LI posterior_samples convert path (called by batchConvertExtr_job and friends in BasicIteration) is the last large XML-resident consumer in the intrinsic-pipeline domain; addressing that closes out the workstream.
First slice of a multi-stage LISA-RIFT integration plan: teach the new ASCII hyperpipeline grid format about explicit ecliptic sky columns before pulling in likelihood or pipeline behavior from the LISA branch. This keeps the old positional I/O untouched while giving later stages a named-column path for sky-as-intrinsic parameters.
Import the dormant LISA response and factored-likelihood modules from AasimZJan/LISA-RIFT lisa_rift_paper as the next multi-stage integration slice. This preserves the upstream implementation without wiring it into the active ILE/CIP pipeline yet; follow-up commits add package/CI safety and later pipeline integration. Upstream provenance includes Aasim Jan's response work beginning at f50f6ec and LISA factored likelihood work beginning at 01de363 on lisa_rift_paper.
Add minimal package markers and an explicit CI smoke test for the newly imported dormant LISA response and likelihood modules. Keep the imported modules out of the active pipeline for now, but make CI catch import regressions while later LISA integration stages proceed. Also replace a library sys.exit path in the response helper with ValueError.
…sy-junior/research-projects-RIT into rift_O4d_junior_ralph
Start a LISA-specific integrate_likelihood_extrinsic_batchmode fork as the first wiring surface for the staged LISA-RIFT integration. The file is seeded from Aasim Z. Jan's lisa_rift_paper branch, with only local modernization needed for this tree: keep it separate from the normal ILE to avoid conflicts with ongoing calmarg-in-loop work, and update the XML imports to igwn_ligolw so CI --help checks can exercise the executable.
Add a LISA-local lalsimutils_compat module adapted from Aasim Z. Jan's lisa_rift_paper branch for the helper surface needed by the standalone LISA ILE fork. Route HDF5 frame loading, LISA FD mode generation, and the LISA parameter printer through the compat shim instead of editing shared lalsimutils. Add smoke coverage and include it in import_check so the new fork has a CI tripwire while normal ILE and calmarg-in-loop work remain isolated.
Add an operational LISA smoke test that generates synthetic A/E/T TDI data, writes and reloads it through the HDF5 cache format, runs PrecomputeAlignedSpinLISA, and checks that the injected extrinsics beat an offset point in the likelihood. Wire LISA_response.generate_lisa_TDI through the LISA compat helper so synthetic generation remains isolated from shared lalsimutils. Include the test in import_check as a CI tripwire for the current LISA landing surface.
Add helper_LISA_Events.py as a narrow LISA-specific producer for the files consumed by create_event_parameter_pipeline: a hyperpipeline initial grid, ILE args, CIP args list, test args, transfer file list, and a ready CEPP command. Keep this outside pseudo_pipe and helper_LDG_Events so it can be tested before calmarg-in-loop lands. The contract test runs the helper in a temp directory and asks CEPP to render the DAG from the emitted files, including a zero-likelihood LISA ILE configuration for ILE/CIP handoff shape checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gradually cascades the Ralph/LISA branch into rift_O4d_junior_calmarg_in_loop.\n\nContext: this branch carries the LISA code line related to AasimZJan/LISA-RIFT lisa_rift_paper and upstream PR oshaughn#112.\n\nConflict resolution:\n- Resolved util_RIFT_pseudo_pipe.py by preserving calmarg's explicit --pipeline-builder override while keeping Ralph/LISA's hyperpipeline grid suffix handling: the CEPP command now uses proposed-grid. instead of forcing proposed-grid.xml.gz. This keeps normal XML behavior by default and preserves RIFT_HYPERPIPELINE_FORMAT=.dat behavior for LISA/hyperpipeline workflows.\n\nValidation run in a temp worktree:\n- python -m py_compile MonteCarloMarginalizeCode/Code/bin/util_RIFT_pseudo_pipe.py MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration\n- python -m py_compile LISA modules/helper/integrate_likelihood_extrinsic_batchmode_lisa and LISA tests\n- python MonteCarloMarginalizeCode/Code/test/test_lisa_lalsimutils_compat.py\n- python MonteCarloMarginalizeCode/Code/test/test_lisa_response_import.py\n- python MonteCarloMarginalizeCode/Code/test/test_lisa_helper_contract.py\n\nNote: git diff --check reports trailing whitespace already present in the imported LISA files; this PR intentionally avoids a broad whitespace-only rewrite.