Running more chains and changing data path - #34
Merged
Conversation
Large chains and sacc files are moving to external hosting (NERSC portal), not git. Redirects every ./chains/ and ./sacc/ reference in the cosmodc2_redmapper and capish_simulation notebooks/scripts to ../../data/<example>/..., and untracks the fits/sacc binaries that were previously committed (kept on disk locally, just out of git now). Also: - cosmosis_mcmc_plots.py: fix FITS header key collisions for params sharing an 8-char prefix (e.g. sigma_8/sigma_0/sigma_m/sigma_z all truncated to RAW_SIGM), and let plot_triangle take pre-loaded samples= as an alternative to paths=. - add the missing cosmo-only baseline snapshot to plot_samples.ipynb regen cell. - capish: consolidate single-realization run_in2p3_both/ into the mean-mocks run_in2p3_both_mean/ config. - .gitignore: broaden slurm-*.out/err to slurm*, add *.debug.
data/README.md documents the NERSC portal hosting setup (structure, refresh procedure, the rsync-permissions gotcha) for maintainers; README.md gains a user-facing download section with a one-shot wget command matching the paths the example notebooks expect. Also adds chart_generation.py + full_pipeline_chart.yml, used to render the baseline pipeline flowchart via ceci make_flow_chart.
Updated section title and clarified source of truth for hosted bundle.
pygobject-stubs was unpinned, pulling in PyGObject unpinned too, which now needs to build pycairo from source -- the CI runner has no cairo/expat pkg-config files, so meson fails with "Package expat was not found in the pkg-config search path". Nothing in the codebase imports gi/PyGObject; pygobject-stubs is a pure type-stub package with no runtime code, so drop it rather than add system build deps to CI.
…notebooks - cosmosis_mcmc_plots.py: chains_to_fits now saves the FULL chain (burn_fraction=0.0 default), tracking the emcee walker count (NWALKERS header, from the raw chain's #walkers=N line) so it can be reshaped back to (n_steps, n_walkers, n_params) later. Burn-in moves to load time instead: fits_to_samples/fits_to_walker_chain/PlotConfig all default to burn_fraction=0.15. fits_to_walker_chain is new -- reloads a saved chain with per-walker structure intact, for autocorrelation-time diagnostics that a flat MCSamples cannot support. - convergence_diagnostics: dropped the hand-rolled Gelman-Rubin check (emcee has no equivalent, and it was never adding anything integrated_time's own tol=50 reliability check does not already cover); autocorrelation-only now, via emcee.autocorr.integrated_time() and its built-in tol=50 convergence rule. Converted from notebook to plain script (save_chains.py-style, runs standalone or via sbatch). - plot_samples.ipynb: best-fit cell now reads both_full.fits directly (best fit = min loglike, loglike = -log posterior) instead of the raw chain file. summary_plot_best_fits.ipynb: fixed a dead burn_fraction parameter that was accepted but never actually passed to fits_to_samples. - Swept every burn_fraction mention (library defaults, hardcoded cells, PlotConfig construction sites) from 0.3 to 0.15 for consistency.
prediction_versus_data.ipynb: the "Best Fit Cosmo" section now reads Omega_c/sigma_8/MOR parameters from plot_samples.ipynb's saved bestfit_full_both.txt instead of hand-copied literals, so it always tracks whatever chain was last saved. summary_plot_best_fits.ipynb: switched from symmetric Gaussian mean +/- std to getdist's own marginalized-density credible intervals (getMargeStats(), same machinery _print_constraints uses via getInlineLatex) for both the summary plot and the tension table. Also fixes the plot's asymmetric-error attempt, which silently fell back to Gaussian every time (mcmc.samples(par) is not a real MCSamples method), and drops a redundant earlier draft cell that plotted the same Gaussian-only figure to the same output file. Also included: capish_simulation review pass (data_generation scripts, regen_capish_mean_sacc.py, save_chains.py/convergence_diagnostics.py doc trims) and a new crow_clcosmo_benchmark.ipynb.
plot_samples.ipynb: dropped the "Chain paths" cell (dead now that the best-fit cell reads straight from both_full.fits), outputs refreshed against the latest, further-extended chains. summary_plot_best_fits.ipynb: outputs refreshed from an actual Jupyter run -- the previous commit's embedded output cells were stale, still showing the old pre-fix Gaussian print format even though the source was already correct (verified only via an extracted script copy at the time, which does not touch the notebook's own cached output). Also a fontsize tweak on the summary plot's value labels. New: fiducial_cosmology_analysis/cosmo_0.22_0.9/run_in2p3_both/ pipeline config (Firecrown/TJPCov/TXPipe/config_in2p3_both.yml).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.