Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Changelog

All notable changes to DatabentoBinaryEncoding.jl are documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.1.2] - 2026-06-01

### Added

- `SType.UNDEF` (`255` / `0xFF`) enum member representing the Databento "unset"
symbol-type sentinel, so non-nullable `stype` fields can hold and round-trip it.
- v3 `InstrumentDefMsg` byte-offset regression test.
- Documentation refresh: registered-installation instructions, corrected enum
reference lists (`Schema`, `SType`, `InstrumentClass`), and this changelog.

### Fixed

- Generic `read_record` no longer crashes with `ArgumentError: invalid value for
Enum SType: 255` on DBN v3 captures. A v3 live gateway can wire-encode an unset
`stype` as `0xFF` on a `SymbolMappingMsg`; the decoder now tolerates it instead
of throwing on the first control record ([#23], [#24]).
- v3 `InstrumentDefMsg` is decoded/encoded with the fixed binary field layout
(71-byte v2/v3 symbol fields, signed `Int32` leg ratios) and `InstrumentClass`
enum values aligned with the Databento DBN codes (`CALL = 'C'`, `PUT = 'P'`,
`STOCK = 'K'`, …) ([#22]).

## [0.1.1] - 2026-05-25

### Fixed

- `SymbolMappingMsg` `hd.length` is re-derived from the layout being written on a
cross-version encode, so the record header matches the bytes that follow it
([#21]).

## [0.1.0] - 2026-05-18

### Added

- Initial release of DatabentoBinaryEncoding.jl (renamed from DBN.jl): reading,
writing, and streaming of DBN v2/v3 files; Zstd compression; bidirectional
conversion to JSON/Parquet/CSV; byte-for-byte compatibility with the official
Rust implementation ([#18], [#19]).

[Unreleased]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.2...HEAD
[0.1.2]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.1...v0.1.2
[0.1.1]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/releases/tag/v0.1.0
[#18]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/pull/18
[#19]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/pull/19
[#21]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/pull/21
[#22]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/pull/22
[#23]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/issues/23
[#24]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/pull/24
2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name = "DatabentoBinaryEncoding"
uuid = "90689371-c8cb-40d1-831f-18033db90f74"
authors = ["Tyler Beason <tbeas12@gmail.com>"]
version = "0.1.1"
version = "0.1.2"

[deps]
CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
Expand Down
13 changes: 11 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@ For more details, read the [introduction to DBN](https://databento.com/docs/stan
- ✅ Zstd file compression support (read and write)
- ✅ Bidirectional format conversion (DBN ↔ JSON/Parquet/CSV)
- ✅ Byte-for-byte compatibility with official implementations
- ✅ All DBN message types (Trades, MBO, MBP, OHLCV, Status, etc.)
- ✅ All DBN message types (Trades, MBO, MBP, OHLCV, definitions, statistics, imbalance, status)
- ✅ Consolidated & BBO schemas (CBBO, CMBP-1, TCBBO, BBO-1s/1m)
- ✅ Live control records (SymbolMappingMsg, SystemMsg, ErrorMsg), tolerant of v3 unset-`stype` sentinels
- ✅ High-precision timestamp handling
- ✅ Fixed-point price arithmetic

Expand All @@ -31,7 +33,14 @@ dbn version1.dbn --output version2.dbn --upgrade

## Installation

This package is not yet registered. Install directly from GitHub:
The package is registered in the General registry:

```julia
using Pkg
Pkg.add("DatabentoBinaryEncoding")
```

Or, for the latest unreleased changes, install directly from GitHub:

```julia
using Pkg
Expand Down
2 changes: 1 addition & 1 deletion benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@ To compare with the official Rust implementation:
time dbn dump test_data.dbn --output /dev/null

# Benchmark Julia version
julia -e 'using DBN; @time read_dbn("test_data.dbn")'
julia -e 'using DatabentoBinaryEncoding; @time read_dbn("test_data.dbn")'
```

### Across Versions
Expand Down
45 changes: 33 additions & 12 deletions docs/src/api/enums.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,18 @@ The `Schema` enum defines the available data schemas:
- `MBO` - Market-by-order
- `MBP_1` - Market-by-price (top of book)
- `MBP_10` - Market-by-price (10 levels)
- `TBBO` - Top-of-book BBO
- `TBBO` - Top-of-book BBO (trade-sampled)
- `TRADES` - Trade messages
- `OHLCV_1S`, `OHLCV_1M`, `OHLCV_1H`, `OHLCV_1D` - OHLCV bars at different intervals
- `DEFINITION` - Instrument definitions
- `STATISTICS` - Publisher statistics
- `STATUS` - Status messages
- `IMBALANCE` - Imbalance messages
- And more...
- `CBBO`, `CBBO_1S`, `CBBO_1M` - Consolidated BBO (event, 1-second, 1-minute)
- `CMBP_1` - Consolidated market-by-price (top of book)
- `TCBBO` - Consolidated BBO (trade-sampled)
- `BBO_1S`, `BBO_1M` - BBO at 1-second / 1-minute intervals
- `MIX` - Mixed-schema stream (e.g. live captures interleaving data and control records)

For complete schema details, see [Databento Schemas Documentation](https://databento.com/docs/schemas-and-data-formats/whats-a-schema).

Expand All @@ -45,11 +51,21 @@ Record types identify the message type in the binary format. Common values:
SType
```

Symbol types specify how instruments are identified:
- `RAW_SYMBOL` - Raw symbol string
Symbol types specify how instruments are identified. Numeric values match the
official Databento DBN spec (wire-encoded as a `UInt8`):
- `INSTRUMENT_ID` - Numeric instrument ID
- `PARENT` - Parent instrument
- And more...
- `RAW_SYMBOL` - Raw symbol string from the exchange
- `SMART` - Deprecated alias (split into `CONTINUOUS` and `PARENT`)
- `CONTINUOUS` - Continuous contract symbol
- `PARENT` - Parent symbol for derived instruments (e.g. `SPXW.OPT`)
- `NASDAQ_SYMBOL`, `CMS_SYMBOL` - Venue-specific symbols
- `ISIN`, `US_CODE` - Security identifiers
- `BBG_COMP_ID`, `BBG_COMP_TICKER` - Bloomberg composite ID / ticker
- `FIGI`, `FIGI_TICKER` - OpenFIGI identifier / ticker
- `UNDEF` - Unset/undefined sentinel (`0xFF`). A v3 live gateway can wire-encode
an unset `stype` (e.g. on a `SymbolMappingMsg`); this member lets the decoder
represent it instead of erroring. The nullable metadata `stype_in` decodes
`0xFF` to `nothing` instead.

## Action Types

Expand Down Expand Up @@ -103,12 +119,17 @@ File encoding formats:
InstrumentClass
```

Instrument classification:
- `STOCK` - Equity
- `FUTURE` - Futures contract
- `OPTION` - Options contract
- `FX_SPOT` - Foreign exchange spot
- And more...
Instrument classification. Values match the Databento DBN single-character
codes:
- `STOCK` (`'K'`) - Equity
- `CALL` (`'C'`) - Call option
- `PUT` (`'P'`) - Put option
- `FUTURE` (`'F'`) - Futures contract
- `BOND` (`'B'`) - Fixed income
- `FX_SPOT` (`'X'`) - Foreign exchange spot
- `COMMODITY_SPOT` (`'Y'`) - Commodity spot
- `MIXED_SPREAD` (`'M'`), `FUTURE_SPREAD` (`'S'`), `OPTION_SPREAD` (`'T'`) - Spreads
- `OTHER` (`'?'`), `UNKNOWN_0`, `UNKNOWN_45` - Fallbacks for unclassified instruments

## Usage Examples

Expand Down
12 changes: 9 additions & 3 deletions docs/src/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,17 @@

## Installing DatabentoBinaryEncoding.jl

!!! note
DatabentoBinaryEncoding.jl is not yet registered in the Julia General registry. Install directly from GitHub.
### From the General registry

```julia
using Pkg
Pkg.add("DatabentoBinaryEncoding")
```

### From GitHub

For the latest unreleased changes, install directly from the repository:

```julia
using Pkg
Pkg.add(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl")
Expand All @@ -26,7 +32,7 @@ using Pkg
Pkg.develop(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl")
```

This will clone the repository to `~/.julia/dev/DBN`.
This will clone the repository to `~/.julia/dev/DatabentoBinaryEncoding`.

## Verifying Installation

Expand Down
2 changes: 1 addition & 1 deletion docs/src/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Get up and running with DatabentoBinaryEncoding.jl in 5 minutes!

```julia
using Pkg
Pkg.add(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl")
Pkg.add("DatabentoBinaryEncoding")
```

## Basic Usage
Expand Down
11 changes: 6 additions & 5 deletions docs/src/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,13 @@ Common issues and solutions when working with DatabentoBinaryEncoding.jl.

### Package Not Found

**Problem**: `ERROR: Package DBN not found`
**Problem**: `ERROR: Package DatabentoBinaryEncoding not found`

**Solution**: DatabentoBinaryEncoding.jl is not registered. Install from GitHub:
**Solution**: Install it from the General registry (or from GitHub for unreleased changes):
```julia
using Pkg
Pkg.add(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl")
Pkg.add("DatabentoBinaryEncoding")
# or: Pkg.add(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl")
```

### Dependency Conflicts
Expand All @@ -27,9 +28,9 @@ Pkg.resolve() # Resolve conflicts

If problems persist:
```julia
Pkg.rm("DBN")
Pkg.rm("DatabentoBinaryEncoding")
Pkg.gc() # Clean up
Pkg.add(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl")
Pkg.add("DatabentoBinaryEncoding")
```

## Compression Issues
Expand Down
Loading