Skip to content

Update arrakis dataloader - #517

Merged
wbenoit26 merged 11 commits into
ML4GW:mainfrom
wbenoit26:update-arrakis-dataloader
Sep 13, 2026
Merged

wbenoit26 merged 11 commits into
ML4GW:mainfrom
wbenoit26:update-arrakis-dataloader

Conversation

@wbenoit26

Copy link
Copy Markdown
Contributor

@deepchatterjeeligo @bhgupta20 I haven't been able to test yet because our robot credential needs the arrakis.read scope added to it, but would be good to get a couple sets of eyes on this.

Changes:

  1. Renames NGDD to Arrakis
  2. Makes the Arrakis dataloading more robust, with fewer magic numbers
  3. Updates the configs to use the correct AMPLFI architecture and objects
  4. Changes how the kafka producer gets setup because the current approach broke for server="local".
  5. Fixes a small bug with snapshotter.full_psd_present. The update_size being passed was in units of seconds, but was compared against units of samples. Additionally, the check against full_psd_present in main was slightly wrong due to the subtraction of update_size. Removing it entirely doesn't cause any harm; it just means that contiguous_update_size will be incremented once more before stopping.

@wbenoit26

Copy link
Copy Markdown
Contributor Author

Correction: no arrakis-specific token is needed on the cluster. I've got this running right now, but LLPIC/MDC data isn't available through arrakis yet.

@bhgupta20 bhgupta20 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a minor issue, but that version no longer exists, so it looks good to me!

@wbenoit26
wbenoit26 marked this pull request as ready for review September 13, 2026 00:32
@wbenoit26
wbenoit26 merged commit 02e89b4 into ML4GW:main Sep 13, 2026
9 checks passed
@wbenoit26
wbenoit26 deleted the update-arrakis-dataloader branch September 13, 2026 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants