This directory contains the tools, scripts, and instructions to reproduce the high-performance backtest throughput benchmarks and strategy parameter optimizations for the PineForge engine.
Below is the boxplot chart showing the distribution of backtest throughput across the 201 bench slots (the 100 public corpus probes and the maintainers' 101 closed scripts). The individual data points (jittered orange circles) correspond to the throughput of each compiled C++ strategy running on the 53,929-bar ETHUSDT 15m feed.
| Quartile Metric | Throughput (Millions of Bars per Second - M/s) | Equiv. Execution Time (per 10k bars) |
|---|---|---|
| Minimum | 0.064 M/s | 156.04 ms |
| Q1 (25th Percentile) | 0.488 M/s | 20.50 ms |
| Median (50th Percentile) | 0.779 M/s | 12.84 ms |
| Q3 (75th Percentile) | 1.137 M/s | 8.79 ms |
| Maximum | 2.095 M/s | 4.77 ms |
Note: the metric is GBench's <slug>/throughput/no_magnifier hot loop. Each strategy's shared library is dlopened once, outside the timed region. Each of the 20 timed iterations runs strategy_create, applies the slot's inputs.json strategy_overrides pins, and runs run_backtest over the whole feed with the bar magnifier off; the table reports the mean.
Last measured 2026-09-29 on a dedicated AWS c7a.8xlarge (AMD EPYC 9R14, 32 cores, SMT off) running Ubuntu 24.04, pinned to one core, with engine main 35db01c8. This is the median of five quiet runs: the five run medians were 0.782, 0.786, 0.778, 0.779 and 0.777 M/s, and the table and chart come from the run with the median result (run 4). Every run started at a 1-minute load below 6 with no build or test process running; the loads are in ../results/speed.md. A public checkout only has the 100 public slots, whose median in the same run is 1.050 M/s. The five runs' Google Benchmark files and load traces are in ../results/raw/2026-09-29-35db01c8/throughput/; run 4 is byte-identical to benchmark_results.json.
Earlier figures were measured on an Apple M4 Max and are not compared with these: 0.638 M/s on 2026-09-22 (engine 063e4460), and 17.05 M/s on 2026-05-29 (100 strategies, a 41,307-bar feed). On this host, in one window, engine 35db01c8 takes 0.35× the time of 063e4460 per strategy at the median with the magnifier on, and the engine of the 2026-06-11 speed table runs the three probes both populations share 4–8× faster than 35db01c8 (see ../results/speed.md, Provenance).
grid_search_repro.py runs a multi-parameter grid search in memory, through Python ctypes FFI, on the compiled C++ shared library for 021-composite-scalping-integration-01: an EMA-cross scalper with take-profit and stop-loss exits in ticks. The sweep covers 27 combinations: Fast EMA 3/5/7 × Slow EMA 11/13/15 × take profit 10/15/20 ticks, with the stop loss held at its default of 7 ticks.
- Best configuration: Fast EMA = 3, Slow EMA = 15, take profit = 20 ticks
- Net profit: −103.16 USDT. The probe is a mechanism test, not a profitable strategy: every combination loses. The sweep demonstrates the FFI loop, not an edge.
- Trade count: 5,153 trades over 53,929 bars of 15m ETHUSDT (recorded 2026-09-29 at engine
35db01c8, as ate9ad37ddon 2026-09-22)
reproduce.sh: The end-to-end automation bash script.plot_quartile.py: Python script using NumPy and Matplotlib to parse benchmark JSON, calculate exact throughput quartiles, and generate the boxplot chart (throughput_quartiles.png).grid_search_repro.py: Multi-parameter grid search optimizer script using ctypes FFI to sweep parameters on the compiled021-composite-scalping-integration-01library.
Initialise the assets submodule (git submodule update --init benchmarks/assets), and have Python 3, CMake ≥ 3.20 (the build fetches Google Benchmark), a C++17 compiler (e.g. clang or gcc), and the Python plotting dependencies installed:
pip install matplotlib numpyExecute the main wrapper script directly from this directory:
chmod +x reproduce.sh
./reproduce.shThis script will:
- Recompile the backtest engine and every bench strategy in Release mode: the 100 public slots, plus the 101 closed slots when the maintainers'
benchmarks/assets-closed/root is present. - Run Google Benchmark suites across all strategies for dynamic throughput measurement, exporting results to
benchmark_results.json. - Compute the exact distribution quartiles (Min, Q1, Median, Q3, Max) and print them.
- Render the distribution boxplot chart to
throughput_quartiles.png. - Run the in-memory parameter sweep grid search on
021-composite-scalping-integration-01and output the optimized parameters.
You can audit these figures (such as the median hot-loop throughput of 0.78 million bars per second) step by step:
Instead of running every slot, you can compile and benchmark a single strategy of your choice (e.g. 001-analyzer-anvil-percent-costs-01) using Google Benchmark directly to eliminate any script wrapper bias:
# From the project root:
cmake -B build -DPINEFORGE_BUILD_BENCH_STRATEGIES=ON -DPINEFORGE_BUILD_SPEED_BENCH=ON -DPINEFORGE_BUILD_TESTS=ON
cmake --build build --target bench_strategies pineforge_bench -j4
# Execute only the chosen strategy benchmark:
./build/bin/pineforge_bench --benchmark_filter="001-analyzer-anvil-percent-costs-01"Every strategy's Pine Script v6 is compiled to native, modern C++17. You can inspect the fully generated C++ files inside each strategy directory (e.g., benchmarks/assets/strategies/001-analyzer-anvil-percent-costs-01/generated.cpp) to verify that they are:
- Performing genuine, complex math and indicators (no mock shortcuts).
- Leveraging in-memory sliding window lookups instead of slow databases.
- Accessing trades and executing orders in
O(1)time complexity.
To verify that the FFI parameters are actually being set correctly and the optimization is genuine, run grid_search_repro.py; it prints every parameter combination it runs:
# Run grid search directly
python3 grid_search_repro.pyThis script accesses the compiled strategy's inputs using Python ctypes by calling strategy_set_input, showing how easily PineForge integrates into modern quantitative analysis environments with zero translation overhead.