A professional benchmarking and auto-tuning framework for llama.cpp.
llama-benchmark-suite is a benchmarking framework designed for llama.cpp and llama-server. It focuses on real-world inference performance rather than synthetic benchmarks.
Unlike llama-bench, which measures kernel-level performance, this project measures the complete inference pipeline through the OpenAI-compatible API exposed by llama-server, providing metrics that closely match actual application performance.
- ✅ Real streaming benchmark based on llama-server
- ✅ Accurate TTFT (Time To First Token)
- ✅ Accurate TPS (Tokens Per Second)
- ✅ Automatic parameter grid search
- ✅ Automatic llama-server lifecycle management
- ✅ JSON / CSV / HTML reports
- ✅ Automatic best configuration selection
- ✅ Historical benchmark comparison
- ✅ Windows first-class support
- ✅ Linux compatible architecture
llama-bench is an excellent microbenchmark for measuring low-level inference throughput.
However, it cannot evaluate the complete inference pipeline used by real applications.
For example:
- API latency
- Streaming performance
- TTFT
- Context initialization
- OpenAI-compatible serving
- End-to-end throughput
This project fills that gap.
config.json
│
▼
batch_test.ps1
│
┌───────────────┴───────────────┐
▼ ▼
start_server.ps1 stop_server.ps1
│
▼
llama-server.exe
│
▼
benchmark/stream.py
│
▼
benchmark/metrics.py
│
▼
analyzer.py
│
▼
report.py
│
▼
HTML / CSV / JSON
llama-benchmark-suite/
├── config/
│ └── config.json
│
├── scripts/
│ ├── batch_test.ps1
│ ├── start_server.ps1
│ ├── stop_server.ps1
│ └── clean.ps1
│
├── python/
│ ├── main.py
│ │
│ ├── core/
│ ├── benchmark/
│ ├── report/
│ └── monitor/
│
├── reports/
├── results/
├── logs/
└── history/
Load Config
│
▼
Start llama-server
│
▼
Wait until Ready
│
▼
Streaming Benchmark
│
▼
Collect Metrics
│
▼
Stop Server
│
▼
Next Parameter
│
▼
Analyze Results
│
▼
Generate Report
The framework measures:
| Metric | Description |
|---|---|
| TTFT | Time To First Token |
| Generation Time | Time spent generating output |
| Total Time | End-to-end latency |
| Tokens | Number of generated tokens |
| TPS | Tokens Per Second |
| Throughput | Overall inference throughput |
| Score | Composite performance score |
Future versions will include:
- GPU Utilization
- GPU Memory
- CPU Utilization
- RAM Usage
- Power Consumption
- Thermal Information
Example:
{
"ngl":[24,28,30,32],
"threads":[8,10,12],
"context":[4096,8192],
"batch":[256,512],
"ubatch":[256,512]
}The framework automatically:
- Starts llama-server
- Runs benchmark
- Saves result
- Stops server
- Continues with the next configuration
The framework generates:
reports/
report.html
report.csv
summary.json
best_config.json
- Streaming Benchmark
- Grid Search
- HTML Report
- Automatic Analysis
- GPU Monitoring
- CPU Monitoring
- VRAM Statistics
- Resource Timeline
- Early Stop
- OOM Detection
- Second-pass Search
- Warm-up Strategy
- Bayesian Optimization
- Evolutionary Search
- Auto Parameter Tuning
- Dashboard
- REST API
- Windows 11 / Linux
- Python 3.11+
- llama.cpp (latest)
- llama-server
- GGUF model
- NVIDIA CUDA (optional)
.\scripts\batch_test.ps1Output:
Best Configuration
NGL : 32
Threads : 12
Context : 8192
Batch : 512
UBatch : 512
TTFT : 0.71 s
TPS : 11.42 tok/s
Score : 98.6
This project aims to become a professional benchmarking and auto-tuning framework for llama.cpp, providing reproducible, real-world inference performance evaluation instead of synthetic kernel benchmarks.
The long-term goal is to evolve into a complete automatic optimization platform capable of finding the best inference configuration with minimal manual intervention.
MIT License