Updating examples - #32
Open
eduardojsbarroso wants to merge 15 commits into
Open
Conversation
…t tags - clpipeline package/module/imports -> clpipe (dir, files, CLI entrypoint, conda env names, CI, docs, examples, notebooks, tutorials) - FirecrownPipeline -> CLPFirecrown, TJPCovPipeline -> CLPCovariance (class names, stage `name`, config yml top-level keys, module_name paths) - FirecrownPipeline outputs renamed at the ceci tag level: cluster_counts_mean_mass_redshift_richness -> sampler_file, cluster_redshift_richness -> likelihood_file, cluster_richness_values -> priors_file. On-disk output filenames are unchanged: added `aliases:` blocks to every example pipeline yml declaring the CLPFirecrown stage so ceci's default tag->filename naming still resolves to the old basenames.
Homepage/Issues still referenced eduardojsbarroso/CLPipe from before the repo lived under the LSSTDESC org; the CLPipeline->CLPipe rename carried that stale owner forward verbatim.
Aliases would have kept output filenames pinned to the old cluster_counts_mean_mass_redshift_richness/cluster_redshift_richness/ cluster_richness_values basenames forever. Instead, let ceci's default tag->filename naming apply, so outputs are written as sampler_file.ini, likelihood_file.py, and priors_file.ini: - Removed the aliases: block from all 36 example pipeline ymls. - clp_firecrown.py's generated sampler ini now cross-references its sibling likelihood/values files by their actual (computed) basenames instead of hardcoding the old ones. - Renamed the 10 checked-in example output artifacts and fixed the values=/likelihood_source= lines inside each generated sampler_file.ini. - Updated every launch_*.sh that invoked `cosmosis cluster_counts_mean_mass_redshift_richness.ini` to `cosmosis sampler_file.ini`. - Updated tests/test_clp_firecrown.py to match.
clean_examples predates fd49847/328363b (tests, CI, fiducial cosmology integration) and all of no_purity's CLPipeline->CLPipe / FirecrownPipeline->CLPFirecrown / TJPCovPipeline->CLPCovariance rename work. Merges all of it in. Conflicts were confined to examples/cosmodc2_halos_wazp_rich/ and examples/cosmodc2_wazp_gaussian/, which this branch's "Updating tests" commit had already deleted while no_purity's rename still touched them. Resolved by keeping them deleted -- that content lives on, renamed, on the wazp_cosmodc2 branch instead. Nothing in the three examples this branch is meant to ship (SDSS_data, cosmoDC2-20deg2, cosmodc2_halos) conflicted.
…nning
Both predate this branch's rename work:
- examples/cosmodc2_halos/Firecrown.yml had a stray trailing `---`
(leftover multi-doc separator), which makes yaml.safe_load() -- what
ceci's own Pipeline.build_config() uses -- refuse to parse it
("expected a single document in the stream").
- None of cosmoDC2-20deg2/{Firecrown,TJPCov}.yml, cosmodc2_halos/
{Firecrown,TJPCov}.yml, or SDSS_data/CL_SDSS_concat.yml declared
fiducial_cosmology as an input, even though #28 added it as a
required input to CLPFirecrown/CLPCovariance. Every other
cosmodc2-based example in the repo (redmapper, capish_simulation)
points it at the same shared IN2P3 path, so these now do too.
Verified by constructing the actual ceci pipeline graph for all 5
files (Pipeline.create()) -- previously all 5 failed at graph
construction; all 5 now succeed and resolve outputs to
sampler_file.ini/likelihood_file.py/priors_file.ini as expected.
…line The assertion compared the SSC-replaced variance against the mock fixture's hardcoded data value (40.0), not against the theory model's own prediction for that bin. But replace_crow_counts only guarantees full_cov[i, i] >= theory_counts[i] (Poisson term + non-negative SSC addition) -- there's no guarantee theory_counts[i] itself equals the arbitrary mock value, and #28's fiducial-cosmology change (using fiducial_cosmology.to_ccl(), which doesn't set transfer_function) shifted that prediction to ~39.67, just under 40, breaking the test. Exposes the actual theory prediction as stage.crow_theory_counts (set right before replace_crow_counts returns) so the test can assert against the real invariant instead of an incidental match to the fixture's round number.
main squash-merged PR #31, which broke the shared-history link this branch's earlier merge relied on, so git needed a fresh three-way merge here. Same resolution as before: the 38 conflicts were all confined to cosmodc2_halos_wazp_rich/ and cosmodc2_wazp_gaussian/ (deleted here, still present on main) plus a few git rename-detection false positives from sibling directories now sharing identical filenames (sampler_file.ini etc.) -- resolved by keeping all of it deleted.
TJPCov (LSSTDESC/TJPCov@clustercount_cov_validation, clusters_helpers.py::extract_indices_rich_z) only recognizes richness tracers prefixed bin_richness_ or bin_rich_; this catalog's tracers were named rich_0/rich_1/rich_2, so CLPCovariance failed immediately with "Could not extract richness or z from tracer combination". Our own test fixtures (tests/conftest.py) already use the bin_rich_ convention and pass, confirming this was a naming gap specific to this catalog, not a CLPCovariance bug. Renamed via sacc.Sacc.rename_tracer(), which only rewrites tracer identity and each data point's tracer-name tuple -- verified data values, data-point order, and the covariance matrix are byte-identical before and after.
Mirrors cosmodc2_halos/Firecrown.yml and cosmoDC2-20deg2/Firecrown.yml so this example can be run the same way (ceci Firecrown.yml) instead of only via CL_SDSS_concat.yml --yamlId Firecrown. Same stage, inputs, and config as the existing concat file -- verified the ceci pipeline graph constructs and resolves outputs correctly.
ceci failed at graph construction: "shear_tomography_classifier needed by TXSourceSelectorMetadetect but not generated by any stage". Every cosmodc2_redmapper TXPipe.yml variant (22 of them) already runs TXSourceTomography first; only this one was missing it. Confirmed against TXPipe's source (txpipe/source_selection/tomography.py) that TXSourceTomography's sole output is exactly shear_tomography_classifier.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.