Skip to content

[CHORE]: Handle CDP commands larger than 100MiB #305

Description

@camdecoster

Description

Add handling of oversized-message. choreographer should transparently split CDP payloads that exceed Chrome's 100 MiB pipe buffer.

Why should this change be made?

This feature was added to Kaleido in plotly/kaleido#442, but it had the unintended consequence of breaking serialization. This was due to the fact that the payload had to be measured before sending it to determine the size. The naive serialization (using orjson instead of PlotlyJSONEncoder) broke on a number of data structures. At any rate, chunking belongs where serialization and the wire already live.

Scope

  • Add payload size check to choreographer during serialization
  • If the size is greater than the limit (~100MiB), chunk the payload
  • If the payload can't be chunked, raise an error
  • If the size is under the limit, handle the call normally

Notes

Activity

  1. self-assigned this
    on Jul 31, 2026
  2. added
    P1needs immediate attention
    taskone-off task
    and removed on Jul 31, 2026
  3. camdecoster commented on Jul 31, 2026

    @camdecoster
    ContributorAuthor

    @ayjayt do you have any thoughts on this plan? It was your comment in the Kaleido PR that led me to starting work.

  4. ayjayt commented on Aug 3, 2026

    @ayjayt
    Collaborator
    1. Are we the ones setting that buffer? We make the pipes.
    2. We're otherwise just building a write loop that feeds incremently and detecte pipe blocking ? Is there another way to detect buffer-full? If not, need some buffer to the buffer as writes can be parallel.
  5. camdecoster commented on Aug 4, 2026

    @camdecoster
    ContributorAuthor

    Andrew and I spoke offline. Generally, the issue is present because Chromium hard codes the max buffer size (see OP). We don't detect the issue right now because Chromium returns a JS error that doesn't get handled in Choreographer. I put together a PR to add the chunking in #306. If that gets added, I'll remove the chunking from Kaleido.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1needs immediate attentionsize: 5taskone-off task

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions