Environment
- Python 3.9.6 (macOS arm64)
- zstandard 0.25.0
- protobuf 6.33.4
- azure-messaging-webpubsubclient 1.1.0
Problem
When subscribing to WebSocket hubs (classic, state_gex, state_greeks_zero, state_greeks_one, orderflow), we receive frames that are not valid zstd-compressed data. Passing these to zstandard.ZstdDecompressor().decompress() causes a native segfault (signal 11), crashing the Python process.
Errors before crash:
zstd decompress error: Data corruption detected
zstd decompress error: Unknown frame descriptor
Occurs every ~5-15 minutes during RTH, across all hubs. Not ticker-specific.
Error breakdown by hub (44 total in one session):
| Hub |
zstd errors |
protobuf parse errors |
| classic |
6 |
8 |
| state_gex |
7 |
4 |
| state_greeks_zero |
6 |
4 |
| state_greeks_one |
4 |
4 |
| orderflow |
1 |
0 |
Two distinct error types:
zstd decompress error: Data corruption detected — zstd library rejects the frameProtobuf parse failed — zstd succeeds but protobuf cannot parse the result (possible different message type?)
The decompress call is inside a try/except, but the crash is a native segfault (signal 11) in the zstd C extension — it kills the process before Python's exception handler can catch it.
Likely cause
Per admin, the hub can send both JSON and protobuf messages. We believe non-zstd frames (control messages, JSON acks) are being passed into the zstd decompressor.
Current workaround
We have added a zstd magic byte check (0x28B52FFD) before decompression to skip non-zstd frames. Testing next session.
Questions
- Is there a recommended way to distinguish data frames from control/protocol frames in the
on_group_message callback?
- Should we check the Azure Web PubSub message
dataType field to determine if the payload is binary (zstd protobuf) vs text (JSON)?
- Are there example implementations in the repo that handle mixed frame types?
Timestamps (UTC, Mar 19 2026)
Crash cluster: 06:31, 06:33, 06:37, 06:39, 06:40, 06:41
Later crashes: 12:23, 12:35, 12:39
Environment
Problem
When subscribing to WebSocket hubs (classic, state_gex, state_greeks_zero, state_greeks_one, orderflow), we receive frames that are not valid zstd-compressed data. Passing these to
zstandard.ZstdDecompressor().decompress()causes a native segfault (signal 11), crashing the Python process.Errors before crash:
Occurs every ~5-15 minutes during RTH, across all hubs. Not ticker-specific.
Error breakdown by hub (44 total in one session):
Two distinct error types:
zstd decompress error: Data corruption detected— zstd library rejects the frameProtobuf parse failed— zstd succeeds but protobuf cannot parse the result (possible different message type?)The decompress call is inside a try/except, but the crash is a native segfault (signal 11) in the zstd C extension — it kills the process before Python's exception handler can catch it.
Likely cause
Per admin, the hub can send both JSON and protobuf messages. We believe non-zstd frames (control messages, JSON acks) are being passed into the zstd decompressor.
Current workaround
We have added a zstd magic byte check (
0x28B52FFD) before decompression to skip non-zstd frames. Testing next session.Questions
on_group_messagecallback?dataTypefield to determine if the payload is binary (zstd protobuf) vs text (JSON)?Timestamps (UTC, Mar 19 2026)
Crash cluster: 06:31, 06:33, 06:37, 06:39, 06:40, 06:41
Later crashes: 12:23, 12:35, 12:39