Lo psets now take goodtimes/background information from science dependencies

[vineetbansal]

Closes issue #3079

Change Summary

Lo psets now take goodtimes/background information from science dependencies instead of ancillary files.

Overview

The lo_l1b_goodtimes product has now being producing goodtimes and H/O background rate information (the latter as the lo_l1b_bgrates product, auto-generated when goodtimes is generated). This PR makes the psets product now get this information from these science dependencies instead of ancillary files.

File changes

• imap_processing/cdf/config/imap_lo_global_cdf_attrs.yaml - renamed logical source to match generated filename.
• lo_l1b.py - the functions that were trying to set good/bad times are now gone, in favor of getting this information from the goodtimes dataset. The places in the code that created dataframes out of csvs are gone too, and now use the xr.Dataset directly.

Note that the incoming ancillary file had hardcoded 0 and 59 and a series of 7 1s as certain columns (with the generating code hardcoding these in its logic too). This implied hardcoding was not carried over when generating the bgrates dataset (the background rates are along a single esa_level axis, and there's no concept of bg_start and bg_end (bins)), but to avoid major disruptions in pset use, this hardcoding now shows up in lines:

        # For each ESA step, accumulate the fractional coverage.
        # Add this time fraction to the affected bins.
        # Note that all ESA Levels and all N_SPIN_ANGLE_BINS currently get the
        # same increment, pending algorithmic changes in the future.
        goodtimes_fraction += time_fraction

i.e. it is assumed that the goodtimes_fraction used is always 1.0 for all ESA levsl and all (3600) spin-bins, and is now broadcast to the desired shape.

Testing

Modified tests based on changes.

[Copilot]

Pull request overview

Switches Lo PSET (L1C) processing to consume goodtimes and H/O background rates from the new L1B science dependencies (imap_lo_l1b_goodtimes, imap_lo_l1b_bgrates) instead of ancillary CSV files, removes the now-dead set_bad_times/set_bad_or_goodtimes helpers from L1B, fixes the goodtimes logical source name, and updates the CLI to load the new science dependencies. The per-ESA-step / per-spin-bin filtering that the CSV used to express via bin_start/bin_end/E-Step* columns is no longer applied — fractions/rates are broadcast across all 3600 spin bins and all ESA steps (intentional per the PR description, acknowledged loss of fidelity).

Changes:

• filter_goodtimes, create_goodtimes_fraction, calculate_exposure_times, and set_background_rates are reworked to consume xarray Datasets from sci_dependencies rather than DataFrames built from ancillary CSVs.
• set_bad_times/set_bad_or_goodtimes (and their l1b_de call site) are removed from L1B; CLI now also pulls goodtimes and bgrates L1B CDFs as science inputs.
• Tests are reshaped accordingly (new l1b_bgrates_ds fixture, xarray-based goodtime fixtures, simplified expectations matching the broadened filter logic).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
imap_processing/lo/l1b/lo_l1b.py	Removes set_bad_times/set_bad_or_goodtimes and their invocation from l1b_de, plus the unused pandas import.
imap_processing/lo/l1c/lo_l1c.py	Switches goodtimes/bgrates consumption from ancillary DataFrames to L1B xarray Datasets; simplifies signatures of filter_goodtimes, create_goodtimes_fraction, calculate_exposure_times, and set_background_rates.
imap_processing/cli.py	Adds goodtimes and bgrates L1B descriptors to the Lo science file list.
imap_processing/cdf/config/imap_lo_global_cdf_attrs.yaml	Renames Logical_source from imap_lo_l1b_good-times to imap_lo_l1b_goodtimes.
imap_processing/tests/lo/test_lo_l1b.py	Removes tests for the deleted bad/goodtime helpers.
imap_processing/tests/lo/test_lo_l1c.py	Replaces DataFrame-based fixtures with Dataset-based fixtures and rewrites expectations to match the new broadcast behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tmplummer]

A few things to consider/change.

[tmplummer]

What values are in l1b_de["epoch"]? Does it contain direct event times? or CCSDS packet creation times?

I just want to make sure that this mask is doing what you want it to do. It seems like filtering DE times is probably what you want, but I also don't know what times are recorded in the goodtimes_ds["gt_start/end_met"] DataArrays. Also, it is important to understand that there is some precision error in conversions to/from floating point MET to integer TTJ2000ns so you need to be careful doing these time comparisons and consider what conversions have been applied to each of the times being compared.

[vineetbansal]

I have no idea what's in l1b_de['epoch'], except that its in ttj2000ns. I didn't want to disrupt the workings of wherever that dataset is used down the line (in create_pset_counts I see..). goodtimes_ds` stores the start/end of goodtime windows in met (this was a PR approved last week).

Point noted about loss of precision - one optimization I can think of is to determine pointing_start_met/pointing_end_met directly instead of this round trip to met -> ttjns2000 -> met which is silly. I'll do that now.

Other than that if you still see a possible problem perhaps you can open an issue on this and I'll look at it next week? Thanks!

[tmplummer]

Ok, I just double checked. It does appear that the l1b_de epoch variable contains the TTJ2000ns time of each event. That alleviates my concerns about this masking.

[tmplummer]

Is this in any way already assured in the bg_rates and goodtimes processing? I will point out that this logic will keep any goodtime interval that overhangs the pointing start or end.

Edit: Ok, I see that the overlap is computed below.

[tmplummer]

I noticed that below, you use the Dataset.isel(epoch=mask) method to filter. After investigating a bit, I think that using sel will work here, but it should be isel. From what I found,

If you pass a 1D boolean array to ds.isel(dim=mask), it works like numpy. However, if you pass a boolean array to ds.sel(dim=mask), it may fail or misinterpret the data if the dimension does not have integer labels that match the mask, as described in GitHub issues regarding sel behavior #4892.

[vineetbansal]

Yes I think isel is better. changed.

[jtniehof]

Lo team notes that the goodtimes algorithms do need some updates; Nathan will be in touch regarding that and coordinating those updates with eventually merging this PR.

[jtniehof]

I also want to foreground #3021 to see if we need that before switching over to using this product