Skip to content

Generic read_record crashes on DBN v3 captures: invalid value for Enum SType: 255 #23

Description

@tbeason

Summary

read_record (the generic decode path the DatabentoAPI.jl untyped live reader
uses) crashes on DBN version-3 captures with:

ArgumentError: invalid value for Enum SType: 255.

255 (0xFF) is the "unset" sentinel for an SType field, but the SType enum
has no member for it, so constructing the enum throws. The crash fires on the
first symbol-mapping / control record of the stream — typically after exactly 1
data record — so it makes v3 captures effectively undecodable through the
generic reader.

DBN version-1 files decode fine; this is v3-specific.

Reproduction

using DatabentoBinaryEncoding, CodecZstd, TranscodingStreams
import DatabentoBinaryEncoding as DBN

function probe(p)
    io  = TranscodingStream(ZstdDecompressor(), open(p, "r"))
    dec = DBN.DBNDecoder(io); DBN.read_header!(dec)
    println("schema=", dec.metadata.schema, "  version=", dec.metadata.version,
            "  stype_in=", dec.metadata.stype_in)
    n = 0
    while true
        r = DBN.read_record(dec); r === nothing && break; n += 1
    end
    println("decoded ", n, " records")
end

# v3 live capture (OPRA.PILLAR cbbo-1s): CRASHES after 1 record
probe("<a v3 .dbn.zst live capture>")
# => schema=MIX  version=3  stype_in=nothing
# => ArgumentError: invalid value for Enum SType: 255.

# v1 batch file (same schema family): OK
probe("replay_SPY.OPT_2026-05-13T13_30_00__2026-05-13T13_31_00.dbn.zst")
# => schema=CBBO_1S  version=1  stype_in=PARENT
# => decoded 7722839 records

Reproduced 2026-06-01 on both Julia-captured and Python-captured
OPRA.PILLAR cbbo-1s files, so this is a decoder-side gap, not a writer/capture
bug (distinct from the known v1-layout SymbolMappingMsg encoder corruption).

Likely root cause

A v3 record (almost certainly the v3 SymbolMappingMsg / a control record)
carries stype_in and/or stype_out = 0xFF meaning "unset", and the decoder
does an unguarded SType(byte) (or SType.T(byte)) on it. The v3 layout
differs from v1, so either the field offset or the unset-sentinel handling is
off for v3.

Suggested fix

  • Map 0xFF to an explicit SType "Undef/Unset" member (matching the
    reference Databento DBN spec), or
  • Guard control-record decode so an unset stype is tolerated rather than passed
    to the enum constructor.

Please add a v3 round-trip / decode regression test using a control record with
an unset stype.

Impact

  • Blocks using real v3 live captures as bench_live_replay fixtures in
    DatabentoAPI.jl (the v1 SPY.OPT fixture is the current workaround).
  • Almost certainly affects real v3 live streaming through DatabentoAPI.jl's
    generic (untyped) live reader, since the gateway interleaves
    SymbolMappingMsg control records with data.

Found while benchmarking DatabentoAPI.jl PR #22 (live reconnect supervisor);
orthogonal to that PR — the crash is present independent of it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions