Skip to content

Smartstack: shipper queue - #473

Open
timbeccue wants to merge 7 commits into
mainfrom
feature/smartstack-pr4-ship
Open

Smartstack: shipper queue#473
timbeccue wants to merge 7 commits into
mainfrom
feature/smartstack-pr4-ship

Conversation

@timbeccue

@timbeccue timbeccue commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

This is PR 4 of 8 completing the v1 banzai implementation of Smartstacking at site.

PR1 (#469): combine math
PR2 (#470): JPEG utilities
PR3 (#471): DB updates
PR4 (#473): shipper contract      <-- you are here
PR5 (#474): smartstack data products
PR6 (#475): stack worker polling
PR7 (#476): e2e tests, compose cleanup
PR8 (#477): logging

PR1 combine math ───────┐
PR2 JPEG utilities ─────┤
PR3 DB updates ─────────┼──> PR5 data products --> PR6 stack workers --> PR7 E2E --> PR8 logs/docs
PR4 shipper contract ───┘

PRs 1–4 provide the independent foundations for combining images, rendering JPEGs, storing stack state, and shipper integration (this PR). PR5 builds the products, PR6 adds the polling worker, and PR7–8 add end-to-end proof, logging, and final documentation.

What changes here

Added methods to post data to the shipper queue. In the case of smartstack previews, this includes passing in a subset of the fits header so that the ingester has the metadata it needs without the inclusion of a fits file.

Publishing contract

post_to_shipper_queue() publishes UTF-8, plain-text JSON:

Field Final product Preview
fits e45 FITS path null
small_thumbnail JPEG path JPEG path
large_thumbnail JPEG path JPEG path
instrument_enqueue_timestamp Passed through Passed through
thumbnail_metadata Omitted by the current final caller Included when supplied

The publisher also:

  • Declares the durable fanout exchange, queue, and binding before publishing.
  • Uses publisher confirms so broker rejection or acknowledgement timeout reaches PR6’s retry path.
  • Applies bounded five-second connection and confirmation timeouts.
  • Publishes and declares through the same channel.

New settings provide these defaults:

  • SHIPPER_EXCHANGE=ship_files
  • SHIPPER_QUEUE_NAME=ship
  • SMARTSTACK_PREVIEWS=true

Scope

This PR defines and tests the publishing boundary only. It does not create products or decide when to preview, finalize, or retry; those responsibilities belong to PR5 and PR6.

Focused tests cover the durable binding, plain-text JSON bodies for previews and finals, optional thumbnail metadata, publisher confirms, and timeout options.

@timbeccue
timbeccue marked this pull request as ready for review August 7, 2026 06:40
@timbeccue
timbeccue requested a review from cmccully August 7, 2026 07:01
@timbeccue timbeccue changed the title Smartstack: shipper integration Smartstack: shipper queue Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant