From b6c1c0e26154f8e17c59aa07d9e9160b12069cd9 Mon Sep 17 00:00:00 2001 From: Tyler Beason Date: Mon, 1 Jun 2026 11:41:48 -0500 Subject: [PATCH] docs: refresh README/docs, add CHANGELOG, bump to v0.1.2 - Update install instructions to the registered form (Pkg.add("DatabentoBinaryEncoding")) across README, installation, quickstart, and troubleshooting; the package is now in the General registry. - Correct the enum reference lists in docs/src/api/enums.md to match src/types.jl: full Schema set (CBBO/CMBP/TCBBO/BBO/MIX), full SType set incl. the new UNDEF sentinel, and the real InstrumentClass members (CALL/PUT/STOCK/... codes). - README features: note consolidated/BBO schemas and v3-tolerant live control records. - Fix leftover old-name references (DBN -> DatabentoBinaryEncoding) in docs/benchmark. - Add CHANGELOG.md (Keep a Changelog) covering v0.1.0 - v0.1.2. - Bump version to 0.1.2. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 57 +++++++++++++++++++++++++++++++++++++ Project.toml | 2 +- README.md | 13 +++++++-- benchmark/README.md | 2 +- docs/src/api/enums.md | 45 +++++++++++++++++++++-------- docs/src/installation.md | 12 ++++++-- docs/src/quickstart.md | 2 +- docs/src/troubleshooting.md | 11 +++---- 8 files changed, 119 insertions(+), 25 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..50db7b8 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,57 @@ +# Changelog + +All notable changes to DatabentoBinaryEncoding.jl are documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +## [0.1.2] - 2026-06-01 + +### Added + +- `SType.UNDEF` (`255` / `0xFF`) enum member representing the Databento "unset" + symbol-type sentinel, so non-nullable `stype` fields can hold and round-trip it. +- v3 `InstrumentDefMsg` byte-offset regression test. +- Documentation refresh: registered-installation instructions, corrected enum + reference lists (`Schema`, `SType`, `InstrumentClass`), and this changelog. + +### Fixed + +- Generic `read_record` no longer crashes with `ArgumentError: invalid value for + Enum SType: 255` on DBN v3 captures. A v3 live gateway can wire-encode an unset + `stype` as `0xFF` on a `SymbolMappingMsg`; the decoder now tolerates it instead + of throwing on the first control record ([#23], [#24]). +- v3 `InstrumentDefMsg` is decoded/encoded with the fixed binary field layout + (71-byte v2/v3 symbol fields, signed `Int32` leg ratios) and `InstrumentClass` + enum values aligned with the Databento DBN codes (`CALL = 'C'`, `PUT = 'P'`, + `STOCK = 'K'`, …) ([#22]). + +## [0.1.1] - 2026-05-25 + +### Fixed + +- `SymbolMappingMsg` `hd.length` is re-derived from the layout being written on a + cross-version encode, so the record header matches the bytes that follow it + ([#21]). + +## [0.1.0] - 2026-05-18 + +### Added + +- Initial release of DatabentoBinaryEncoding.jl (renamed from DBN.jl): reading, + writing, and streaming of DBN v2/v3 files; Zstd compression; bidirectional + conversion to JSON/Parquet/CSV; byte-for-byte compatibility with the official + Rust implementation ([#18], [#19]). + +[Unreleased]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.2...HEAD +[0.1.2]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.1...v0.1.2 +[0.1.1]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.0...v0.1.1 +[0.1.0]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/releases/tag/v0.1.0 +[#18]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/pull/18 +[#19]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/pull/19 +[#21]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/pull/21 +[#22]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/pull/22 +[#23]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/issues/23 +[#24]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/pull/24 diff --git a/Project.toml b/Project.toml index e62118f..5a39ef0 100644 --- a/Project.toml +++ b/Project.toml @@ -1,7 +1,7 @@ name = "DatabentoBinaryEncoding" uuid = "90689371-c8cb-40d1-831f-18033db90f74" authors = ["Tyler Beason "] -version = "0.1.1" +version = "0.1.2" [deps] CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b" diff --git a/README.md b/README.md index fc48414..518fcb4 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,9 @@ For more details, read the [introduction to DBN](https://databento.com/docs/stan - ✅ Zstd file compression support (read and write) - ✅ Bidirectional format conversion (DBN ↔ JSON/Parquet/CSV) - ✅ Byte-for-byte compatibility with official implementations -- ✅ All DBN message types (Trades, MBO, MBP, OHLCV, Status, etc.) +- ✅ All DBN message types (Trades, MBO, MBP, OHLCV, definitions, statistics, imbalance, status) +- ✅ Consolidated & BBO schemas (CBBO, CMBP-1, TCBBO, BBO-1s/1m) +- ✅ Live control records (SymbolMappingMsg, SystemMsg, ErrorMsg), tolerant of v3 unset-`stype` sentinels - ✅ High-precision timestamp handling - ✅ Fixed-point price arithmetic @@ -31,7 +33,14 @@ dbn version1.dbn --output version2.dbn --upgrade ## Installation -This package is not yet registered. Install directly from GitHub: +The package is registered in the General registry: + +```julia +using Pkg +Pkg.add("DatabentoBinaryEncoding") +``` + +Or, for the latest unreleased changes, install directly from GitHub: ```julia using Pkg diff --git a/benchmark/README.md b/benchmark/README.md index d5fa510..781e9e4 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -209,7 +209,7 @@ To compare with the official Rust implementation: time dbn dump test_data.dbn --output /dev/null # Benchmark Julia version -julia -e 'using DBN; @time read_dbn("test_data.dbn")' +julia -e 'using DatabentoBinaryEncoding; @time read_dbn("test_data.dbn")' ``` ### Across Versions diff --git a/docs/src/api/enums.md b/docs/src/api/enums.md index 84ac2c4..50502c9 100644 --- a/docs/src/api/enums.md +++ b/docs/src/api/enums.md @@ -15,12 +15,18 @@ The `Schema` enum defines the available data schemas: - `MBO` - Market-by-order - `MBP_1` - Market-by-price (top of book) - `MBP_10` - Market-by-price (10 levels) -- `TBBO` - Top-of-book BBO +- `TBBO` - Top-of-book BBO (trade-sampled) - `TRADES` - Trade messages - `OHLCV_1S`, `OHLCV_1M`, `OHLCV_1H`, `OHLCV_1D` - OHLCV bars at different intervals +- `DEFINITION` - Instrument definitions +- `STATISTICS` - Publisher statistics - `STATUS` - Status messages - `IMBALANCE` - Imbalance messages -- And more... +- `CBBO`, `CBBO_1S`, `CBBO_1M` - Consolidated BBO (event, 1-second, 1-minute) +- `CMBP_1` - Consolidated market-by-price (top of book) +- `TCBBO` - Consolidated BBO (trade-sampled) +- `BBO_1S`, `BBO_1M` - BBO at 1-second / 1-minute intervals +- `MIX` - Mixed-schema stream (e.g. live captures interleaving data and control records) For complete schema details, see [Databento Schemas Documentation](https://databento.com/docs/schemas-and-data-formats/whats-a-schema). @@ -45,11 +51,21 @@ Record types identify the message type in the binary format. Common values: SType ``` -Symbol types specify how instruments are identified: -- `RAW_SYMBOL` - Raw symbol string +Symbol types specify how instruments are identified. Numeric values match the +official Databento DBN spec (wire-encoded as a `UInt8`): - `INSTRUMENT_ID` - Numeric instrument ID -- `PARENT` - Parent instrument -- And more... +- `RAW_SYMBOL` - Raw symbol string from the exchange +- `SMART` - Deprecated alias (split into `CONTINUOUS` and `PARENT`) +- `CONTINUOUS` - Continuous contract symbol +- `PARENT` - Parent symbol for derived instruments (e.g. `SPXW.OPT`) +- `NASDAQ_SYMBOL`, `CMS_SYMBOL` - Venue-specific symbols +- `ISIN`, `US_CODE` - Security identifiers +- `BBG_COMP_ID`, `BBG_COMP_TICKER` - Bloomberg composite ID / ticker +- `FIGI`, `FIGI_TICKER` - OpenFIGI identifier / ticker +- `UNDEF` - Unset/undefined sentinel (`0xFF`). A v3 live gateway can wire-encode + an unset `stype` (e.g. on a `SymbolMappingMsg`); this member lets the decoder + represent it instead of erroring. The nullable metadata `stype_in` decodes + `0xFF` to `nothing` instead. ## Action Types @@ -103,12 +119,17 @@ File encoding formats: InstrumentClass ``` -Instrument classification: -- `STOCK` - Equity -- `FUTURE` - Futures contract -- `OPTION` - Options contract -- `FX_SPOT` - Foreign exchange spot -- And more... +Instrument classification. Values match the Databento DBN single-character +codes: +- `STOCK` (`'K'`) - Equity +- `CALL` (`'C'`) - Call option +- `PUT` (`'P'`) - Put option +- `FUTURE` (`'F'`) - Futures contract +- `BOND` (`'B'`) - Fixed income +- `FX_SPOT` (`'X'`) - Foreign exchange spot +- `COMMODITY_SPOT` (`'Y'`) - Commodity spot +- `MIXED_SPREAD` (`'M'`), `FUTURE_SPREAD` (`'S'`), `OPTION_SPREAD` (`'T'`) - Spreads +- `OTHER` (`'?'`), `UNKNOWN_0`, `UNKNOWN_45` - Fallbacks for unclassified instruments ## Usage Examples diff --git a/docs/src/installation.md b/docs/src/installation.md index 4cb2efd..0cd5ca2 100644 --- a/docs/src/installation.md +++ b/docs/src/installation.md @@ -7,11 +7,17 @@ ## Installing DatabentoBinaryEncoding.jl -!!! note - DatabentoBinaryEncoding.jl is not yet registered in the Julia General registry. Install directly from GitHub. +### From the General registry + +```julia +using Pkg +Pkg.add("DatabentoBinaryEncoding") +``` ### From GitHub +For the latest unreleased changes, install directly from the repository: + ```julia using Pkg Pkg.add(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl") @@ -26,7 +32,7 @@ using Pkg Pkg.develop(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl") ``` -This will clone the repository to `~/.julia/dev/DBN`. +This will clone the repository to `~/.julia/dev/DatabentoBinaryEncoding`. ## Verifying Installation diff --git a/docs/src/quickstart.md b/docs/src/quickstart.md index a7a8fe5..5ba4d93 100644 --- a/docs/src/quickstart.md +++ b/docs/src/quickstart.md @@ -6,7 +6,7 @@ Get up and running with DatabentoBinaryEncoding.jl in 5 minutes! ```julia using Pkg -Pkg.add(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl") +Pkg.add("DatabentoBinaryEncoding") ``` ## Basic Usage diff --git a/docs/src/troubleshooting.md b/docs/src/troubleshooting.md index 5959293..b066028 100644 --- a/docs/src/troubleshooting.md +++ b/docs/src/troubleshooting.md @@ -6,12 +6,13 @@ Common issues and solutions when working with DatabentoBinaryEncoding.jl. ### Package Not Found -**Problem**: `ERROR: Package DBN not found` +**Problem**: `ERROR: Package DatabentoBinaryEncoding not found` -**Solution**: DatabentoBinaryEncoding.jl is not registered. Install from GitHub: +**Solution**: Install it from the General registry (or from GitHub for unreleased changes): ```julia using Pkg -Pkg.add(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl") +Pkg.add("DatabentoBinaryEncoding") +# or: Pkg.add(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl") ``` ### Dependency Conflicts @@ -27,9 +28,9 @@ Pkg.resolve() # Resolve conflicts If problems persist: ```julia -Pkg.rm("DBN") +Pkg.rm("DatabentoBinaryEncoding") Pkg.gc() # Clean up -Pkg.add(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl") +Pkg.add("DatabentoBinaryEncoding") ``` ## Compression Issues