This is an internal tool used by our engineers at Scalattice to benchmark our inference API for price and latency against rivals offering open models. We decided to release it here because we believe it can further the development of inference networks and products, and help build a world of inference abundance.
git clone https://github.com/scalattice/modelmeter.git
cd modelmeter
./run.sh
# → http://127.0.0.1:8765./run.sh creates a venv, installs, and starts the server. Pass-through flags work: ./run.sh --host 0.0.0.0 --port 8765.
Manual equivalent:
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
modelmeter- Search and filter live open-model catalogs
- $/1M input & output, blend, tokens per dollar, and a simple bill estimate
- Compare / Try - same prompt (optional image, headers) against multiple endpoints; latency, tok/s, estimated cost
- Shareable filter URLs (
?q=qwen3-8b&in=10e6&out=2e6) - Per-provider API tokens and custom OpenAI-compatible hosts in Settings
| Source | Notes |
|---|---|
| Scalattice / OpenRouter / DeepInfra | Live APIs (on by default) |
| LiteLLM index | Broad public price table |
| Together / Fireworks / Groq / custom | Enable in Settings; API keys in local SQLite |
Credentials live in ~/.modelmeter/modelmeter.db (override with MODELMETER_DATA_DIR).
modelmeter --host 0.0.0.0 --port 8765Docker:
docker build -t modelmeter .
docker run --rm -p 8765:8765 modelmeterMIT