Hybrid C++/LibTorch High-Frequency Backtesting Engine

An institutional-grade, ultra-low-latency backtesting simulation engine combining a high-performance C++ execution loop with a Python quantitative research environment and LibTorch (PyTorch C++ API) model inference.

Key Features

1. Sub-Microsecond Core Latency:

   • Zero-Copy Ingestion: Maps Level 2 binary book data directly into virtual memory using Windows CreateFileMapping / POSIX mmap pointer casting.
   • Hardware Cache-Line Alignment: Order and Tick structures are decorated with alignas(64) to eliminate false sharing. Tick is padded to exactly 256 bytes (a power of 2) for bit-shift indexing stride.
   • Pre-Allocated Memory Pool: Pre-allocates block storage for active execution structs, avoiding OS heap malloc/new page faults on the critical path.
   • Asynchronous, Lock-Free Logger: Main execution threads write to a single-producer single-consumer circular ring buffer to prevent disk IO bottlenecks.
   • CPU Core Thread Pinning: Binds the main execution loop to CPU Core 1 (SetThreadAffinityMask) to minimize OS scheduling overhead.

2. Realistic Market Microstructure Simulation:

   • Latent Signal Execution: Queue-routing delay queue delays signals by simulated inference overhead (35 μs), eliminating look-ahead bias.
   • Stochastic Queue Position Decay: TracksLevel 2 depth changes. If level volume decreases without trade executions, the simulator decays the order's volume_ahead queue priority, modeling competitor cancellations.

---

Performance Statistics

System Configuration: Windows 10, MSVC 2019, LibTorch 2.1.0 CPU, JIT Warmup enabled.

Metric	Total Loop (Core Engine + ML)	ML Inference	C++ Core Engine Overhead
Mean	46.5 μs	46.0 μs	0.5 μs (500 ns)
p50 (Median)	31.2 μs	30.9 μs	0.3 μs (300 ns)
p90	48.8 μs	48.2 μs	0.6 μs (600 ns)
p99	120.1 μs	117.1 μs	3.0 μs (3000 ns)

---

Directory Structure

quant_backtester/
├── CMakeLists.txt              # Root build configuration
├── build.ps1                   # Windows PowerShell CMake builder
├── build.sh                    # Linux Bash CMake builder
├── README.md                   # Project documentation & benchmarks
├── .gitignore                  # Git exclude patterns
│
├── data/
│   ├── raw/                    # Raw L2 CSVs (ignored)
│   ├── processed/              # Memory-mapped (.bin) files (ignored)
│   └── data_pipeline.py        # CSV -> Packed cache-aligned binary conversion
│
├── research/                   # Python Quantitative environment
│   ├── requirements.txt        # Package dependencies
│   ├── train_model.py          # PyTorch L2 features training script
│   ├── export_torchscript.py   # Script compiling weights to JIT traced .pt module
│   ├── dashboard.py            # Generates interactive HTML performance reports
│   └── dashboard.html          # Dynamic UI dashboard with light/dark theme toggle
│
├── models/                     
│   └── sentiment_net.pt        # Traced TorchScript model loaded by C++
│
├── include/                    # Public C++ Header Files
│   ├── core/
│   │   ├── Engine.h            # Main loop, delays, thread-pinning, and benchmarks
│   │   ├── OrderBook.h         # LOB state & queue decay matching matcher
│   │   └── Types.h             # Cache-aligned binary packet layout structs
│   ├── io/
│   │   └── MmapReader.h        # Zero-copy memory-mapped file reader
│   ├── ml/
│   │   └── TorchPredictor.h    # Pimpl-isolated LibTorch inference class
│   └── utils/
│       ├── MemoryPool.h        # Placement-new object recycler
│       └── Logger.h            # Asynchronous SPSC ring-buffer logger
│
├── src/                        # Private C++ Source Files
│   ├── core/
│   │   ├── Engine.cpp
│   │   └── OrderBook.cpp
│   ├── io/
│   │   └── MmapReader.cpp
│   ├── ml/
│   │   └── TorchPredictor.cpp
│   └── main.cpp                # wires up IO, Engine, and ML
│
└── tests/                      # Unit testing framework
    ├── CMakeLists.txt          # GTest download and configurations
    ├── test_orderbook.cpp      # GTest: Queue decay, delays, and priority fills
    ├── test_mmap.cpp           # GTest: Packing offsets & alignments
    └── test_inference.cpp      # GTest: LibTorch forward-pass shapes

---

Getting Started

1. Research Phase (Python Setup)

Install requirements, generate L2 ticks, train, and compile the model:

pip install -r research/requirements.txt
python data/data_pipeline.py
python research/train_model.py
python research/export_torchscript.py

2. Build Phase (C++ Compilation)

Windows (PowerShell)

The build script automatically downloads official LibTorch CPU binary version 2.1.0 if not present:

powershell -ExecutionPolicy Bypass -File build.ps1

Linux

chmod +x build.sh
./build.sh

3. Run Unit Tests

build/tests/Release/run_tests.exe

4. Run Backtester

Run the compiled trading loop with the generated binary ticks and TorchScript JIT model:

build/Release/backtester.exe --data data/processed/history.bin --model models/sentiment_net.pt

5. Generate Visual Performance Dashboard

Compile backtest logs and trade signals into an interactive HTML visualization:

python research/dashboard.py

Open research/dashboard.html in your web browser. Includes a dynamic light/dark mode theme toggle that re-themes all Plotly charts.