Skip to content

Xiaoya refactor publisher#48

Merged
taxe10 merged 5 commits into
als-computing:mainfrom
xiaoyachong:xiaoya-refactor-publisher
Jun 10, 2026
Merged

Xiaoya refactor publisher#48
taxe10 merged 5 commits into
als-computing:mainfrom
xiaoyachong:xiaoya-refactor-publisher

Conversation

@xiaoyachong

@xiaoyachong xiaoyachong commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

The main purpose of this PR is to move the Tiled publisher from LSE to arroyoXPS, and to have arroyoXPS send results to LSE via ZMQ instead of using a WebSocket connection.

  1. Results are written to Tiled and a ZMQ message is sent to LSE using the same publisher, XPSTiledResultPublisher.
  2. In the XPSTiledResultPublisher, we import:
    from arroyosas.schemas import RawFrameEvent, SASStart, SASStop, SerializableNumpyArrayModel
    Since both arroyosas and arroyoxps send the same message types (RawFrameEvent, SASStart, SASStop) to LSE, it may make sense to move these schemas into arroyopy and rename them (e.g., remove the SAS prefix).
  3. To handle mid-scan issues, I avoid writing to Tiled unless a start message has been received. For example, if splash_timepix starts before arroyoxps, the Tiled publisher logs the following message:
    Received scan 'None' frame 49 but no start message has been received yet — discarding frame to avoid Tiled index desync.
  4. The Tiled structure:
{tiled_base_uri}/{tiled_prefix}/
│
├── {root_segments1:lse_live_results}/
│   └── {YYYY}/
│       └── {MM}/
│           └── {DD}/
│               └── {experiment_name}/
│                   └── {UUID}/
│                       └── feature_vectors  
│                           (table storing feature vector results)
│
└── {root_segments2:xps_processed_images}/
    └── {YYYY}/
        └── {MM}/
            └── {DD}/
                └── {UUID}/
                    └── xps_averaged_heatmaps  
                        (3D array storing shot-mean heatmaps)

Companion PRs:
LSE: mlexchange/mlex_latent_explorer#79
arroyosas: als-computing/arroyosas#22

Comment thread src/tr_ap_xps/xps_tiled_zmq_publisher.py Outdated

@dylanmcreynolds dylanmcreynolds left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can make this simpler. See commend.

@dylanmcreynolds dylanmcreynolds left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approved. Is this mergable?

@taxe10 taxe10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good. I left some very minor comments.

parts.extend(root_segments)
parts.append(date_path)
parts.append(uuid)
parts.append(ARRAY_KEY)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ARRAY_KEY is referenced before it's defined

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a module-level variable defined as "xps_averaged_heatmaps".

self.poll_interval_sec = poll_interval_sec
self.frame_poll_interval_sec = frame_poll_interval_sec
self.unchanged_cycles_threshold = unchanged_cycles_threshold
self.seen_scans_file = seen_scans_file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self.seen_scans_file = seen_scans_file
self.seen_scans_file = Path(seen_scans_file)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The seen scans are stored as UUIDs, so the path is not used.

poll_interval_sec=poll_interval_sec,
frame_poll_interval_sec=frame_poll_interval_sec,
unchanged_cycles_threshold=unchanged_cycles_threshold,
seen_scans_file=seen_scans_file,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
seen_scans_file=seen_scans_file,
self.seen_scans_file = Path(seen_scans_file)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The seen scans are stored as UUIDs, so the path is not used.

Comment thread src/tr_ap_xps/zmq_publisher.py Outdated
return
if isinstance(message, RawFrameEvent):
message = message.model_dump()
# message["image"] = SerializableNumpyArrayModel.serialize_array(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove this comment?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I just removed it.

@xiaoyachong
xiaoyachong requested a review from taxe10 June 9, 2026 22:16
@taxe10
taxe10 merged commit 3cf9810 into als-computing:main Jun 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants