From e1a338315b57c8c2ffd4327d83f92e08638b7e73 Mon Sep 17 00:00:00 2001 From: Tyler Beason Date: Tue, 2 Jun 2026 16:25:45 -0500 Subject: [PATCH 1/2] Add CLAUDE.md for Claude Code guidance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the DBN delegation boundary, the two clients, Live typed/untyped modes, the TCP read stack, the reconnect supervisor design, and multi-schema capture durability — plus test/build/benchmark/docs commands and the DATABENTO_LIVE_TESTS gating. Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 176 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..f213765 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,176 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +DatabentoAPI.jl is a Julia client for the [Databento](https://databento.com) +market-data APIs: the **Historical** HTTP API (backfills, metadata, batch jobs, +symbology) and the **Live** TCP API (real-time feeds with reconnect, multi-schema +routing, and crash-safe capture to disk). All binary DBN record decoding/encoding +is delegated to a separate package, `DatabentoBinaryEncoding` (imported as `DBN`). + +## Commands + +```bash +# Run the full offline test suite (~30s, 1500+ tests). Uses a temp env with Test. +julia --project=. -e 'using Pkg; Pkg.test()' + +# Run a single test file. Test files are plain includes that assume `using Test`, +# `using DatabentoAPI`, and the tcp_mock are already loaded: +julia --project=. -e 'using Test, DatabentoAPI; include("test/mocks/tcp_mock.jl"); include("test/test_live_subscribe.jl")' + +# Build the package (precompile deps) +julia --project=. -e 'using Pkg; Pkg.instantiate()' + +# Benchmarks (separate benchmark/ project; writes CSVs to benchmark/results/) +julia --project=benchmark benchmark/runbench.jl # default tiers small,medium +julia --project=benchmark benchmark/runbench.jl --profile # + profiling pass +julia --project=benchmark benchmark/runbench.jl --tiers=small # override size tiers + +# Docs (Documenter.jl) +julia --project=docs docs/make.jl +``` + +### Live-network tests + +`test/runtests.jl` runs offline tests by default and **only** runs the +`test/live/` smoke tests when `DATABENTO_LIVE_TESTS=1` is set (they hit the real +gateway and need a valid API key). The offline suite never touches the network — +the Live gateway is faked by an in-process TCP server (`test/mocks/tcp_mock.jl`, +`spawn_mock_gateway` / `spawn_mock_gateway_sequence`); HTTP is mocked by swapping +the `HTTPClient.dispatcher` function. + +The offline `@testset` is wrapped in a `try/catch` so a failure still lets gated +live tests run, then re-throws at the end — i.e. exit status reflects offline +failures even when live tests are enabled. + +## Architecture + +### DBN delegation boundary + +This package owns the **wire protocols** (HTTP requests, the Live TCP/CRAM +handshake, framing) and **session orchestration**. It owns *no* binary record +layout knowledge — every `read_*_msg` / `write_record` / `Metadata` / `Schema` / +`SType` / record struct comes from `DatabentoBinaryEncoding`, aliased throughout +as `DBN` (`import DatabentoBinaryEncoding as DBN`). DBN enums (`Schema`, `SType`, +`Compression`, `Action`, `Side`, `InstrumentClass`) are re-exported so users need +a single `using DatabentoAPI`. If a record decodes wrong or a new schema/rtype is +needed, the fix usually belongs in DBN, not here. + +`src/DatabentoAPI.jl` is the include manifest and the single `export` list — the +public API surface is whatever it exports. + +### Two clients + +- **`Historical`** (`src/historical/`) wraps `HTTPClient` (`src/http.jl`). Leaf + endpoints (`timeseries`, `metadata`, `batch`, `symbology`) are thin functions + that build query params, call `get_json`/`get_bytes`/`open_stream`, and decode. + `get_range` buffers the whole zstd payload then decodes; `foreach_record` + streams (overlaps download with decode, never materializes the payload). +- **`Live`** (`src/live/`) is a `mutable struct` driving the TCP session through + an explicit lifecycle: `connect!` → `subscribe!` (≥1) → `start!` → consume → + `close`. A single `Live` is bound to **one dataset**. + +### Cross-cutting conventions + +- **Enum ↔ wire strings**: `schema_str`/`stype_str`/`compression_str` (in + `src/historical/client.jl`) map DBN enums to Databento's wire spelling + (`Schema.MBP_1 → "mbp-1"`, `SType.RAW_SYMBOL → "raw_symbol"`). Many kwargs also + accept lowercase `Symbol`/`String` spellings, coerced via `getfield(EnumMod, ...)`. +- **API key resolution** (`src/auth.jl`, `load_api_key`): explicit arg → + `~/.databento/config.toml` `[auth] api_key` → `DATABENTO_API_KEY`. Config path + overridable via `DATABENTO_CONFIG_PATH` (used by tests). +- **Errors** (`src/errors.jl`): HTTP 4xx → `BentoClientError`, 5xx → + `BentoServerError`, auth → `BentoAuthError`, all under `BentoError`. +- **Typed decode path**: `record_type_for_schema(schema)` returns the concrete + record struct for type-pure schemas (all but `Schema.MIX`). The typed path + (`DBNStore{T}` with a `Vector{T}`) is ~10× faster / near-zero-alloc vs. the + generic `Vector{DBN.DBNRecord}` Union path. `typed=true` is the default for + `get_range`; `Schema.MIX` falls back to the Union. + +### Live: typed vs untyped mode (frozen at construction) + +`Live(...; typed=false)` (default) uses one `Channel{DBN.DBNRecord}` — all data +*and* control records arrive on `client.channel`; consume via `for rec in client` +or `subscribe_callback`. `Live(...; typed=true)` gives each subscribed schema its +own `Channel{ConcreteT}` (returned by `subscribe!`, or via `channel(client, +schema)`), with `ErrorMsg`/`SystemMsg`/`SymbolMappingMsg` routed to a separate +`control_channel(client)`. Iterating `for rec in client` and `subscribe_callback` +**error** in typed mode — the hot path is type-stable per-channel `put!`s, so the +typed reader (`_reader_loop_typed`) is a verbose flat `if/elseif` tree, one branch +per rtype, rather than dynamic dispatch. + +### Live: TCP read stack + +`_reader_loop[_typed]` (`src/live/reader.jl`) is the `@async` task draining the +socket. The IO stack, bottom-up: `TCPSocket` → optional `ZstdDecompressor` +TranscodingStream (when `compression=zstd`) → `CountingIO` (wraps any IO to +implement `position`, which `TCPSocket` lacks but `DBN.BufferedReader` needs) → +`DBN.BufferedReader` → `DBN.DBNDecoder`. Mid-stream `EOFError` is treated as a +clean end-of-stream (not a crash) — relevant mostly to tests/replay, since the +real gateway streams continuously. + +### Live: reconnect supervisor (the central design point) + +Reconnect lives in the **Live client itself** (`src/live/reconnect.jl`), so every +consumer — iteration, `subscribe_callback`, `stream_to_file` — benefits from one +codepath. `Live`'s default `reconnect_policy = RECONNECT`. Mechanics: + +- `start!` spawns `_supervisor_loop` (an `@async` task) when policy ≠ `NONE`. The + supervisor `wait`s on the reader task; when it dies it decides reconnect vs. + terminate based on a `state` machine (`:fresh → :connecting → :streaming → + :reconnecting → :closed|:failed`) guarded by `state_lock`. +- **Channel ownership differs by policy**: under `NONE`, `start!` `bind`s channels + to the reader so they close when it exits (correct shutdown signal, no + replacement coming). Under `RECONNECT`, channels are owned by the `Live` and + **outlive** each reader incarnation, so the supervisor can spawn a fresh reader + writing into the same channels without terminating consumers. The reader's + `finally` only closes channels under `NONE` or on user `close()`. +- **Retry schedule** (`_reconnect_delay`): the first `immediate_reconnect_attempts` + (default 3) retries fire with no sleep (catch sub-second blips), then full-jitter + exponential backoff (1s base, 60s cap), up to `max_reconnect_attempts` (default + 10; `nothing` = unlimited). The budget **refreshes** once a new reader delivers + ≥1 record (`_record_replay!`), so a long-lived session keeps its full budget + across the connection's lifetime. +- **Replay on reconnect**: the reader records per-instrument last timestamps + (`last_ts_event_by_id` / `last_ts_recv_by_id`, BBO-family schemas key on + `ts_recv`). On reconnect the supervisor resubscribes each schema with + `start=` the min-across-instruments timestamp (bounded to Databento's 24h replay + window), bridging the gap. `add_reconnect_callback(client, cb)` fires + `cb(gap_start_ns, gap_end_ns)` per successful reconnect. +- **Terminal errors**: a gateway `ErrorMsg` sets `c.terminal_error` *before* the + record is published, so the supervisor refuses to reconnect (gateway-side errors + are deterministic). Auth failures during reconnect are likewise terminal. + +### Live: multi-schema capture to file + +`stream_to_file` / `stream_multi_to_files` (`src/live/streaming.jl`) capture one or +more schemas to compressed DBN files using a **single** typed `Live`: one TCP +connection, one typed channel + one drainer task + one `RotatingDBNFile` per +schema, plus a shared control drainer (the gateway dedupes `SymbolMappingMsg` +within a connection). `_run_unified_session` is the coordinator; reconnect is +handled by the Live supervisor (the streaming layer no longer owns its own loop). + +**Durability** (`RotatingDBNFile`): in addition to optional time-based file +rotation (`rotate_seconds`), `frame_seconds` (default 60s) periodically closes the +active zstd frame and starts a new one on the same file — a standards-compliant +multi-frame `.dbn.zst` that reads back as one stream, so a hard kill loses ≤ one +frame instead of corrupting the whole file. A per-schema monitor task also flushes +quiet schemas (whose buffers, including the DBN header, would otherwise never reach +disk). `open_dbn_writer` + `write_record!` expose this writer publicly for custom +subscribe-and-iterate loops. + +**Head-of-line caveat**: the reader `put!`s to each schema's channel in turn; if +one consumer's disk write falls behind and fills its `channel_size` buffer (default +10_000), the reader stalls the TCP socket for *all* schemas. + +## Conventions for changes + +- Match the existing heavy-comment style in `src/live/` — the *why* (race + conditions, ordering constraints, policy rationale) is documented inline and is + load-bearing; preserve it when editing. +- Public API changes mean editing the `export` list in `src/DatabentoAPI.jl` and + adding to the matching `test/test_*.jl` (offline, mock-driven) — add a `live/` + smoke test too if it touches the real gateway path. +- Update `CHANGELOG.md` (Keep-a-Changelog format) under `## [Unreleased]`. +- Requires Julia ≥ 1.12. CI matrix: Julia `1.12` and `1` × {ubuntu, macOS, windows}. +``` From 1dc47dddea6affa81935bedb64c48c095007a616 Mon Sep 17 00:00:00 2001 From: Tyler Beason Date: Tue, 2 Jun 2026 16:32:56 -0500 Subject: [PATCH 2/2] docs: include docs-env setup before build command MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex review flagged that 'julia --project=docs docs/make.jl' fails on a fresh checkout — docs/Project.toml has no manifest and the repo package isn't developed into it. Add the Pkg.develop + Pkg.instantiate step that .github/workflows/documentation.yml runs first. Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index f213765..7edfc23 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,7 +26,8 @@ julia --project=benchmark benchmark/runbench.jl # default tiers julia --project=benchmark benchmark/runbench.jl --profile # + profiling pass julia --project=benchmark benchmark/runbench.jl --tiers=small # override size tiers -# Docs (Documenter.jl) +# Docs (Documenter.jl) — first develop the repo package into the docs env, then build +julia --project=docs -e 'using Pkg; Pkg.develop(PackageSpec(path=pwd())); Pkg.instantiate()' julia --project=docs docs/make.jl ```