Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name = "DBN"
name = "DatabentoBinaryEncoding"
uuid = "90689371-c8cb-40d1-831f-18033db90f74"
authors = ["Tyler Beason <tbeas12@gmail.com>"]
version = "0.1.0"
Expand Down
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# DBN.jl
# DatabentoBinaryEncoding.jl

Julia implementation of the Databento Binary Encoding (DBN) message encoding and storage format for normalized market data.

Expand All @@ -19,7 +19,7 @@ For more details, read the [introduction to DBN](https://databento.com/docs/stan

## Performance

DBN.jl is optimized for high-throughput market data processing:
DatabentoBinaryEncoding.jl is optimized for high-throughput market data processing:
- **Read**: Up to 40M records/sec with near-zero-allocation callback streaming
- **Write**: 11M records/sec with optimized bulk operations
- **Type-specific readers** (`read_trades`, `read_mbo`, etc.) are 5-6x faster than generic `read_dbn()`
Expand All @@ -35,15 +35,16 @@ This package is not yet registered. Install directly from GitHub:

```julia
using Pkg
Pkg.add(url="https://github.com/tbeason/DBN.jl")
Pkg.add(url="https://github.com/tbeason/DatabentoBinaryEncoding.jl")
```

## Usage

### Reading DBN Files

```julia
using DBN
using DatabentoBinaryEncoding
# Tip: `import DatabentoBinaryEncoding as DBN` gives you the terser `DBN.foo` prefix.

# Read entire file into memory (fastest for bulk loading)
records = read_dbn("trades.dbn")
Expand Down Expand Up @@ -72,7 +73,7 @@ println("Total: $(total[])")
### Writing DBN Files

```julia
using DBN, Dates
using DatabentoBinaryEncoding, Dates

# Create metadata for trades
metadata = Metadata(
Expand Down
2 changes: 1 addition & 1 deletion benchmark/BENCHMARK_RESULTS.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
================================================================================
DBN.jl COMPREHENSIVE PERFORMANCE COMPARISON
DatabentoBinaryEncoding.jl COMPREHENSIVE PERFORMANCE COMPARISON
================================================================================


Expand Down
8 changes: 4 additions & 4 deletions benchmark/EXAMPLE_OUTPUT.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Example Benchmark Output

This document shows example output from the DBN.jl benchmark suite.
This document shows example output from the DatabentoBinaryEncoding.jl benchmark suite.

## Throughput Benchmark Example

```
████████████████████████████████████████████████████████████████████
█ █
DBN.jl THROUGHPUT BENCHMARK SUITE █
DatabentoBinaryEncoding.jl THROUGHPUT BENCHMARK SUITE █
█ █
████████████████████████████████████████████████████████████████████

Expand Down Expand Up @@ -134,7 +134,7 @@ Completed at: 2024-01-15 10:45:23

```
======================================================================
DBN.jl BenchmarkTools Suite
DatabentoBinaryEncoding.jl BenchmarkTools Suite
======================================================================

Creating benchmark suite...
Expand Down Expand Up @@ -230,7 +230,7 @@ sys 0m0.012s
```
**Throughput:** ~5.05 million records/second

### Julia (DBN.jl)
### Julia (DatabentoBinaryEncoding.jl)
```julia
julia> @time read_dbn("trades.1m.dbn.zst")
0.245000 seconds (450 allocations: 3.82 MiB)
Expand Down
10 changes: 5 additions & 5 deletions benchmark/PERFORMANCE_REPORT.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# DBN.jl Performance Optimization Report
# DatabentoBinaryEncoding.jl Performance Optimization Report

**Date**: 2025-10-26
**Branch**: `claude/benchmark-package-performance-011CUSRjJF6DMNNwX64UELMY`
Expand All @@ -8,7 +8,7 @@

## Executive Summary

This report documents the performance optimization work on DBN.jl, including critical bug fixes, code quality improvements, and performance benchmarking against the official Rust implementation.
This report documents the performance optimization work on DatabentoBinaryEncoding.jl, including critical bug fixes, code quality improvements, and performance benchmarking against the official Rust implementation.

### Key Achievements

Expand All @@ -19,7 +19,7 @@ This report documents the performance optimization work on DBN.jl, including cri

### Current Performance: Julia vs Rust vs Python

| Metric | Julia DBN.jl | Rust dbn CLI | Python databento | Julia vs Rust | Julia vs Python |
| Metric | Julia DatabentoBinaryEncoding.jl | Rust dbn CLI | Python databento | Julia vs Rust | Julia vs Python |
|--------|--------------|--------------|------------------|---------------|-----------------|
| **Large files (1M+ records)** | 1.18-1.45 M rec/s | 2.22-2.68 M rec/s | 10.04-11.90 M rec/s | **1.77-1.84x slower** | **7.73-10.13x slower** |
| **Medium files (100K records)** | 0.83-0.89 M rec/s | 1.78-1.94 M rec/s | 9.90-10.02 M rec/s | **2.04-2.17x slower** | **11.11-11.48x slower** |
Expand All @@ -28,7 +28,7 @@ This report documents the performance optimization work on DBN.jl, including cri

**Write Performance**: 2.0-2.2 M rec/s (excellent, competitive with Rust)

**Note on Python Performance**: The Python databento client uses Rust bindings (`databento-dbn`) under the hood, essentially providing a thin wrapper over the Rust implementation. This explains its excellent performance - it's doing minimal work in Python, with the heavy lifting done by compiled Rust code. This is a different architecture than Julia DBN.jl which implements the full decoding in Julia.
**Note on Python Performance**: The Python databento client uses Rust bindings (`databento-dbn`) under the hood, essentially providing a thin wrapper over the Rust implementation. This explains its excellent performance - it's doing minimal work in Python, with the heavy lifting done by compiled Rust code. This is a different architecture than Julia DatabentoBinaryEncoding.jl which implements the full decoding in Julia.

---

Expand Down Expand Up @@ -289,7 +289,7 @@ end

## Conclusion

DBN.jl is currently **2.45x slower** than the Rust CLI implementation on average, which is a reasonable starting point for a Julia package. The write performance is already excellent at 2.0-2.2 M rec/s.
DatabentoBinaryEncoding.jl is currently **2.45x slower** than the Rust CLI implementation on average, which is a reasonable starting point for a Julia package. The write performance is already excellent at 2.0-2.2 M rec/s.

**Key findings**:
- ✅ Critical bugs fixed (compression, world age)
Expand Down
12 changes: 6 additions & 6 deletions benchmark/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# DBN.jl Benchmark Suite
# DatabentoBinaryEncoding.jl Benchmark Suite

Comprehensive performance benchmarking tools for DBN.jl, measuring throughput (records/second), latency, memory usage, and bandwidth.
Comprehensive performance benchmarking tools for DatabentoBinaryEncoding.jl, measuring throughput (records/second), latency, memory usage, and bandwidth.

## Quick Start

Expand Down Expand Up @@ -185,7 +185,7 @@ julia benchmark/run_benchmarks.jl --suite-only
### 4. Custom Analysis

```julia
using DBN
using DatabentoBinaryEncoding
include("benchmark/throughput.jl")

# Benchmark specific file
Expand Down Expand Up @@ -262,7 +262,7 @@ For detailed profiling of hot spots:

```julia
using Profile, ProfileView
using DBN
using DatabentoBinaryEncoding

# Profile reading
file = "benchmark/data/trades.1m.dbn"
Expand All @@ -281,7 +281,7 @@ g = flamegraph()
## Memory Profiling

```julia
using DBN
using DatabentoBinaryEncoding

# Track allocations
file = "benchmark/data/trades.1m.dbn"
Expand Down Expand Up @@ -327,7 +327,7 @@ julia benchmark/run_benchmarks.jl --throughput-only

## Contributing

When adding new features to DBN.jl:
When adding new features to DatabentoBinaryEncoding.jl:

1. Add relevant benchmarks to `benchmarks.jl`
2. Run `julia benchmark/run_benchmarks.jl --quick` to verify no regressions
Expand Down
3 changes: 2 additions & 1 deletion benchmark/benchmarks.jl
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ min/median/max times, memory allocations, and garbage collection overhead.
"""

using BenchmarkTools
using DBN
using DatabentoBinaryEncoding
import DatabentoBinaryEncoding as DBN
using DataFrames
using CSV
using Printf
Expand Down
3 changes: 2 additions & 1 deletion benchmark/generate_test_data.jl
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ This script creates DBN files of various sizes with realistic market data
to enable thorough performance testing.
"""

using DBN
using DatabentoBinaryEncoding
import DatabentoBinaryEncoding as DBN
using Dates
using Printf

Expand Down
16 changes: 9 additions & 7 deletions benchmark/run_benchmarks.jl
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""
run_benchmarks.jl

Main entry point for running all DBN.jl benchmarks.
Main entry point for running all DatabentoBinaryEncoding.jl benchmarks.

Usage:
julia benchmark/run_benchmarks.jl [options]
Expand All @@ -18,17 +18,19 @@ Options:
using Pkg

# Ensure we're in the right environment
# Check if DBN is either a dependency or if we're in the DBN.jl project itself
# Check if DatabentoBinaryEncoding is either a dependency or if we're in the
# DatabentoBinaryEncoding.jl project itself.
proj = Pkg.project()
if !haskey(proj.dependencies, "DBN") && proj.name != "DBN"
error("DBN package not found. Make sure you're in the DBN.jl project directory.")
if !haskey(proj.dependencies, "DatabentoBinaryEncoding") && proj.name != "DatabentoBinaryEncoding"
error("DatabentoBinaryEncoding package not found. Make sure you're in the DatabentoBinaryEncoding.jl project directory.")
end

# Activate the project
Pkg.activate(".")

# Load DBN
using DBN
using DatabentoBinaryEncoding
import DatabentoBinaryEncoding as DBN
using Printf
using Dates

Expand Down Expand Up @@ -98,7 +100,7 @@ Print usage information.
"""
function print_help()
println("""
DBN.jl Benchmark Suite
DatabentoBinaryEncoding.jl Benchmark Suite

Usage:
julia benchmark/run_benchmarks.jl [options]
Expand Down Expand Up @@ -136,7 +138,7 @@ function print_banner()
println("\n")
println("█"^80)
println("█" * " "^78 * "█")
println("█" * " DBN.jl - Databento Binary Encoding Performance Benchmark" * " "^12 * "█")
println("█" * " DatabentoBinaryEncoding.jl - Databento Binary Encoding Performance Benchmark" * " "^12 * "█")
println("█" * " "^78 * "█")
println("█"^80)
println()
Expand Down
6 changes: 3 additions & 3 deletions benchmark/verify_setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Verify benchmark suite setup

echo "=================================================="
echo " DBN.jl Benchmark Suite Verification"
echo " DatabentoBinaryEncoding.jl Benchmark Suite Verification"
echo "=================================================="
echo

Expand Down Expand Up @@ -35,7 +35,7 @@ if command -v julia &> /dev/null; then

# Check if DBN can be loaded
echo "Testing DBN package load..."
julia --project=. -e 'using DBN; println("✓ DBN.jl loaded successfully")' 2>&1
julia --project=. -e 'using DatabentoBinaryEncoding; println("✓ DatabentoBinaryEncoding.jl loaded successfully")' 2>&1

if [ $? -eq 0 ]; then
echo
Expand All @@ -48,7 +48,7 @@ if command -v julia &> /dev/null; then
echo
else
echo
echo "⚠ DBN.jl could not be loaded. Run:"
echo "⚠ DatabentoBinaryEncoding.jl could not be loaded. Run:"
echo " julia --project=. -e 'using Pkg; Pkg.instantiate()'"
echo
fi
Expand Down
2 changes: 1 addition & 1 deletion docs/Project.toml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
[deps]
Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
DBN = "90689371-c8cb-40d1-831f-18033db90f74"
DatabentoBinaryEncoding = "90689371-c8cb-40d1-831f-18033db90f74"
10 changes: 5 additions & 5 deletions docs/make.jl
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
using Documenter
using DBN
using DatabentoBinaryEncoding

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the docs environment dependency name

This import was renamed, but docs/Project.toml still declares the package as DBN = "90689371-c8cb-40d1-831f-18033db90f74". When the docs are built with the committed docs environment (the workflow uses julia --project=docs docs/make.jl), the active project does not have a DatabentoBinaryEncoding root dependency unless a prior command mutates docs/Project.toml, so local/CI docs builds can fail with the package not found. Please rename the docs dependency entry to DatabentoBinaryEncoding as part of this package rename.

Useful? React with 👍 / 👎.


makedocs(
sitename = "DBN.jl",
sitename = "DatabentoBinaryEncoding.jl",
format = Documenter.HTML(
prettyurls = get(ENV, "CI", nothing) == "true",
canonical = "https://tbeason.github.io/DBN.jl",
canonical = "https://tbeason.github.io/DatabentoBinaryEncoding.jl",
assets = String[],
),
modules = [DBN],
modules = [DatabentoBinaryEncoding],
pages = [
"Home" => "index.md",
"Installation" => "installation.md",
Expand All @@ -35,7 +35,7 @@ makedocs(
)

deploydocs(
repo = "github.com/tbeason/DBN.jl.git",
repo = "github.com/tbeason/DatabentoBinaryEncoding.jl.git",
devbranch = "main",
push_preview = true,
)
4 changes: 2 additions & 2 deletions docs/src/api/enums.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Enums API Reference

Enumeration types used throughout DBN.jl.
Enumeration types used throughout DatabentoBinaryEncoding.jl.

## Schema Types

Expand Down Expand Up @@ -115,7 +115,7 @@ Instrument classification:
### Working with Schemas

```julia
using DBN
using DatabentoBinaryEncoding

# Check schema type
if metadata.schema == Schema.TRADES
Expand Down
6 changes: 3 additions & 3 deletions docs/src/api/utilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ float_to_price
### Usage Examples

```julia
using DBN
using DatabentoBinaryEncoding

# Convert DBN fixed-point price to float
fixed_price = 1005000 # 100.5000 in fixed-point
Expand Down Expand Up @@ -49,7 +49,7 @@ to_nanoseconds
### Usage Examples

```julia
using DBN, Dates
using DatabentoBinaryEncoding, Dates

# DateTime to nanoseconds
dt = DateTime(2024, 1, 1, 9, 30, 0)
Expand Down Expand Up @@ -111,7 +111,7 @@ DBN_VERSION::UInt8 = 3 # Supported DBN version
DBN uses special sentinel values to indicate undefined/missing data:

```julia
using DBN
using DatabentoBinaryEncoding

# Check for undefined price
if trade.price == UNDEF_PRICE
Expand Down
Loading
Loading