evalstack

Open-source LLM evaluation for teams that don't want to pay $300/month for Braintrust.

Live demo: evalstack.kartikaneja.com Status: alpha · last shipped 2026-06-22 Built by: Kartik Aneja — AI/ML Platform Engineer

---

Why this exists

You ship an LLM feature. You iterate on the prompt. You ship again. Last week's regression resurfaces. You have no idea.

See PRODUCT.md for the full writeup. TL;DR:

• Who: AI Engineer at a 5–50 person startup running OpenAI or Anthropic in production
• Pain: No CI for prompts. No regression detection. Excel sheets and Slack screenshots.
• Why now: LLM eval is the #1 underserved 2026 topic. Braintrust raised $36M but is closed + paid.

Demo

Runs list — every eval run reported by the SDK or CLI, with a "Compare two runs" picker at the bottom:

Run-level diff — judge means with deltas, 10-bucket score histograms per judge (A in zinc, B in indigo), top regressions/improvements panels, and a per-prompt matched table with drill-down to event-level diff:

Run detail — every event in the run with per-event judge scores + reasoning + a compare-events picker at the bottom:

Event-level diff — word-level LCS diff on prompts + completions plus per-judge A-vs-B deltas:

What works today (alpha MVP)

• Python SDK — one decorator (@evalstack.trace) captures LLM calls + outputs
• FastAPI server — receives traces, runs configured judges, stores results in SQLite by default (Postgres opt-in via EVALSTACK_DB_URL for multi-process deployments)
• LLM-as-judge — accuracy + helpfulness rubrics out of the box, fully extensible
• CLI — evalstack run <eval.yaml> runs a suite of eval cases and prints a diff
• Web dashboard — Next.js + Tailwind UI for browsing runs, events, judge scores, and side-by-side event diffs
• CI regression gate — evalstack gate + a GitHub Action that fails a PR when a prompt change drops judge scores past a threshold

Try it (60 seconds, local)

git clone https://github.com/anejakartik/evalstack.git
cd evalstack
pip install -e ./sdk
export OPENAI_API_KEY=sk-...
docker compose up -d              # starts FastAPI server (:8000) + Next.js dashboard (:3000)
python examples/quickstart.py     # runs a sample eval
open http://localhost:3000        # browse runs in the dashboard

CI: fail PRs on eval regression

Catch a prompt change that quietly drops quality — before it merges:

# Compare a candidate run against a baseline; exit 1 if any judge regresses > 0.05
evalstack gate --baseline main --candidate "pr-$GITHUB_SHA" --threshold 0.05

In GitHub Actions, use the bundled composite action (writes a verdict table to the job summary and fails the check on regression):

- uses: anejakartik/evalstack/.github/actions/eval-gate@main
  with:
    endpoint: ${{ secrets.EVALSTACK_ENDPOINT }}
    baseline: main
    candidate: pr-${{ github.sha }}
    threshold: "0.05"

See .github/workflows/eval-gate-example.yml for a self-contained, no-secrets demo (boots the server, seeds runs, gates them).

Architecture

See docs/architecture.md. Stack: Python + FastAPI + SQLite or Postgres (env-configurable) + Next.js dashboard, deployed on Fly.io + Vercel.

What's next

See ROADMAP.md. Top items: web dashboard, A/B traffic split, Slack alerts on regression.

Contributing

PRs welcome. See AGENTS.md if you're an AI coding agent working on this repo.

License

MIT — see LICENSE.