Lo L1C - Use Goodtimes CDF for PSET

[sdhoyt]

The Lo L1C Pointing Set algorithm is using the goodtimes ancillary CSV file, but the Lo pipeline is now producing an L1B Goodtimes product that should be used instead. The code should be updated to switch from using the CSV to the CDF.

An example of a goodtimes L1B product imap_lo_l1b_good-times_20260121_v001.cdf.zip

<xarray.Dataset> Size: 440B
Dimensions:             (dim0: 5, dim1: 7)
Dimensions without coordinates: dim0, dim1
Data variables:
    start_met           (dim0) int64 40B 506692022 506704628 ... 506763272
    end_met             (dim0) int64 40B 506700946 506713552 ... 506767993
    bin_start           (dim0) int64 40B 0 0 0 0 0
    bin_end             (dim0) int64 40B 59 59 59 59 59
    esa_goodtime_flags  (dim0, dim1) int64 280B 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1

And an example of the goodtimes CSV: imap_lo_good-times_20250101_20270101_v003.csv

The main functions to look at for this change are lo_l1c.filter_goodtimes and lo_l1c.set_bad_or_goodtimes in imap_processing/lo/lo_l1c.py

The data being used should be the same between the CSV and CDF, the main change is accessing a different structure in the algorithm code and updating tests.

Lo also uses a badtimes ancillary file, but we now have a badtimes CDF. I haven't heard anything about using the badtimes CDF instead of the ancillary CSV, but it could be worth asking the Lo team and keeping in mind.

Note: When this change is done, the Lo dependency file will need to be updated to switch from triggering processing with the ancillary file to triggering with the CDF. See this issue: IMAP-Science-Operations-Center/sds-data-manager#1272

[vineetbansal]

@sdhoyt - This issue was created 3 weeks ago, but since then the entire implementation of the generation for lo/l1b/goodtimes has changed - see the diffs in PR 3117. I was told to "go ahead and change the goodtimes product" (instead of adding a new "bettertimes" product as was my initial proposal).

The lo/l1c/pset product has no dependency on lo/l1b/goodtimes (either before or after that PR), so won't adding a new dependency now change its generation? That may well be what you mean here, but I don't think that's planned (and will actually break things for the upcoming release). What we plan to do is to start generating a (new) quickmap product instead, deriving from the goodtimes product.

If you agree, then this issue reduces to issue #3153 and can be closed. Let me know if I'm misunderstanding anything.

[sdhoyt]

@vineetbansal In L1C, the processing pipeline currently filters the L1B direct events to only keep those that were detected during a goodtimes window and those events are used in the pointing set. The goodtimes window, as currently implemented, is in the goodtimes ancillary CSV file. The plan this ticket was written for was to use the L1B goodtimes data product to do the filtering for the pointing set rather than the ancillary file.

It's not clear to me what the plan is now. Are pointing sets no longer being filtered at all, so they will contain every direct event regardless of whether they were a good or bad time? We don't filter out bad times in the direct events, only flag them, but they are filtered out when we filter for only the goodtimes in the pointing set. The pset has a dependency for the goodtimes ancillary file, so unless we've gone back to using that, then at least that dependency needs to be removed and the filtering logic should be removed as well.

What is the quickmap product? Is goodtimes the only dependency for that? Is that used at all for creating the pointing set?

[vineetbansal]

Thanks @sdhoyt - makes sense. I've opened PR #3192 on this - perhaps you can review?

While in the process of implementing goodtimes filtering by the goodtimes product instead of the ancillary, I also noticed that the hydrogen-background and oxygen-background ancillary files hadn't been updated as well, and this information is also generated after PR 3117 was merged, so I'm now pulling that information from science files as well.

I'll open up a PR on sds-data-manager shortly, to update the dependencies:

 lo, l1b, de, lo, l1c, pset, HARD, DOWNSTREAM
-lo, ancillary, good-times, lo, l1c, pset, HARD, DOWNSTREAM
-lo, ancillary, hydrogen-background, lo, l1c, pset, HARD, DOWNSTREAM
-lo, ancillary, oxygen-background, lo, l1c, pset, HARD, DOWNSTREAM
+lo, l1b, goodtimes, lo, l1c, pset, HARD, DOWNSTREAM
+lo, l1b, bgrates, lo, l1c, pset, HARD, DOWNSTREAM
 spin, spin, historical, lo, l1c, pset, HARD, DOWNSTREAM
 repoint, repoint, historical, lo, l1c, pset, HARD, DOWNSTREAM
 leapseconds, spice, historical, lo, l1c, pset, HARD_NO_TRIGGER, DOWNSTREAM

Let me know if something is amiss.