Live capture durability: sidecar, sentinel, multi-frame#14
Conversation
Three related changes to stream_to_file / stream_multi_to_files so
captures survive both clean and unexpected shutdowns and the
SymbolMappingMsg records the Live gateway sends at subscribe time
are persisted alongside each output file.
1. SymbolMappingMsg sidecar (.symbology.jsonl per RotatingDBNFile)
--------------------------------------------------------------
The Live gateway emits SymbolMappingMsg in v1 wire layout
(~80 bytes, hd.length=20). DBN.jl's encoder, writing into a
v3-metadata file, would lay them out as v3 (~180 bytes) while
keeping the on-wire hd.length, corrupting the file. So we route
mappings to a JSONL sidecar (one JSON object per line) next to
each .dbn.zst. _append_symbology! is called from _handle_control!,
flushes per-line for crash safety, and the sidecar opens/closes/
rotates in lockstep with its main DBN file.
Sample sidecar line:
{"ts_event":1779203015494476543,"instrument_id":1191182337,
"stype_in":"INSTRUMENT_ID","stype_in_symbol":"SPX.OPT",
"stype_out":"INSTRUMENT_ID","stype_out_symbol":"SPX 271217C02800000",
"start_ts":-1,"end_ts":-1}
2. Stop-sentinel shutdown
-----------------------
New stop_sentinel kwarg (default joinpath(base_dir,"STOP")).
_watch_sentinel polls once per second; on detection it calls
_request_shutdown! on every ctx so the capture exits through
its normal finally → _close_stack! path (writes final zstd
frame footer cleanly). Any stale sentinel from a previous run
is removed at startup so it doesn't trigger immediate shutdown.
To stop a running capture cleanly:
New-Item -Path '<base_dir>\STOP' -ItemType File
3. In-file zstd frame rotation (crash safety)
-------------------------------------------
New frame_seconds kwarg (default 60s). _rotate_frame_if_needed!
uses TranscodingStreams.TOKEN_END + flush to seal the active
zstd frame and start a new one on the SAME stream — no encoder
or stream re-creation needed. Resulting .dbn.zst is a multi-
frame zstd file (standards-compliant; libzstd concatenates
frame payloads on read). A hard kill loses at most one
open frame instead of corrupting the whole file.
Crucially, _rotate_frame_if_needed! is invoked both from
_write_record! (the high-traffic path) AND from _session_monitor
(the only path that fires for quiet schemas — _write_record!
never runs when no records arrive). The lock-free entry point
wraps the rotation body in lock(f.write_lock); ReentrantLock
allows the writer-path call to re-enter the lock it already
holds. The monitor also calls _flush! directly so a quiet
schema's buffered metadata header reaches disk.
Test coverage: all 236 existing tests still pass. Multi-frame
round-trip and clean-shutdown round-trip were validated by hand
during a live OPRA.PILLAR capture session (T213349Z files, 109-
110 bytes, parse cleanly with strict DBN.read_dbn_with_metadata
after sentinel-triggered shutdown).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ee692c2315
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # caller needs instrument_id → raw_symbol resolution, use the | ||
| # `symbology.resolve` historical endpoint with the same symbols. | ||
| s.mapping_count += 1 | ||
| _append_symbology!(ctx.file, rec) |
There was a problem hiding this comment.
Writing SymbolMappingMsg records from the separate control-drainer task can mis-assign mappings to the wrong sidecar file around rotation boundaries. In _run_session, data records are consumed on one task while control records are consumed on another, and _append_symbology! writes to whichever ctx.file is current at write time; if _rotate_if_needed! has already opened the next file, a mapping that logically belongs with the previous file’s records is persisted in the new file’s .symbology.jsonl. This shows up under load or near rotate_seconds boundaries and breaks the advertised “per-file” mapping alignment.
Useful? React with 👍 / 👎.
|
Superseded by #18 (lifts durability to the subscription layer via Live do-block + open_dbn_writer + write_record!; drops the STOP sentinel and the JSONL sidecar workaround since DatabentoBinaryEncoding 0.1.1 fixes SymbolMappingMsg cross-version encode). |
1 participant
Stacked on #13 (rename adoption). Three related additions to the live-capture path so captures survive both clean and unexpected shutdowns and SymbolMappingMsg records are persisted alongside each output file.
1. SymbolMappingMsg sidecar (
.symbology.jsonlperRotatingDBNFile)The Live gateway emits
SymbolMappingMsgin v1 wire layout (~80 bytes,hd.length=20). DBN.jl's encoder, writing into a v3-metadata file, would lay them out as v3 (~180 bytes) while keeping the on-wirehd.length, corrupting the file. So mappings get routed to a JSONL sidecar (one JSON object per line) sitting next to each.dbn.zst.{"ts_event":1779203015494476543,"instrument_id":1191182337, "stype_in":"INSTRUMENT_ID","stype_in_symbol":"SPX.OPT", "stype_out":"INSTRUMENT_ID","stype_out_symbol":"SPX 271217C02800000", "start_ts":-1,"end_ts":-1}_append_symbology!is called from_handle_control!, flushed per-line for crash safety.The proper fix lives in DBN.jl's encoder (re-layout v1→v3 with correct
hd.length); the sidecar is the right workaround until that lands.2. Stop-sentinel shutdown
New
stop_sentinelkwarg (defaultjoinpath(base_dir, "STOP"))._watch_sentinelpolls once per second; on detection it calls_request_shutdown!on every ctx, so the capture exits through the normalfinally → _close_stack!path (writes the final zstd frame footer cleanly).To stop a running capture cleanly:
3. In-file zstd frame rotation (crash safety)
New
frame_secondskwarg (default60.0)._rotate_frame_if_needed!usesTranscodingStreams.TOKEN_END + flushto seal the active zstd frame and start a new one on the SAME stream — no encoder or stream re-creation needed..dbn.zstis a multi-frame zstd file (standards-compliant; libzstd concatenates frame payloads on read).Crucially,
_rotate_frame_if_needed!is invoked both from_write_record!(the high-traffic path) AND from_session_monitor(the only path that fires for quiet schemas —_write_record!never runs when no records arrive). The function wraps its body inlock(f.write_lock);ReentrantLockallows the writer-path call to re-enter the lock it already holds. The monitor also calls_flush!directly so a quiet schema's buffered metadata header reaches disk.Validation
stream_multi_to_filesreturned normally → 3 zero-dataT213349Z.dbn.zstfiles (109-110 B each) parse cleanly with strictDBN.read_dbn_with_metadata.TaskStop'd previous session (tested before this PR).🤖 Generated with Claude Code