Repository navigation
batch_evaluation fails on self-hosted v4 events_only deployments (uses unavailable v3 read endpoints) #1861
Description
Environment
- Langfuse server: self-hosted v4 (
docker.io/langfuse/langfuse:4), default write modeevents_only - Python SDK: 4.14.4 (also verified against 4.15.1 source)
What happened
Running batch evaluation against existing traces fails. The runner fetches items via endpoints that are unavailable on v4 events_only deployments:
scope="traces"→client.api.trace.list→GET /api/public/tracesscope="observations"→ legacyobservations_v1→GET /api/public/observations
Both return:
"This endpoint is not available on deployments running in Langfuse v4 events_only mode."
Root cause
BatchEvaluationRunner._fetch_batch_with_retry (langfuse/batch_evaluation.py, line 1139 in v4.14.4) still calls the v3 read APIs. Per the v3→v4 migration guide, reads should go through GET /api/public/v2/observations (client-side grouping by traceId, root observation for trace-level io).
Expected
batch_evaluation works on events_only deployments, e.g. by fetching via the v2 observations API.
Workaround
We currently run our own thin loop: fetch v2 observations grouped by traceId → evaluate → create_score (score-create ingestion works fine on v4).
Happy to contribute a PR if the v2-based approach sounds right — glad to align on the design first.
Activity
Rainbow-prince commentedon Sep 11, 2026
Small correction: the original report referenced
BatchEvaluationRunner._fetch_items, which is a wrong name. The actual method name is_fetch_batch_with_retry(langfuse/batch_evaluation.py, line 1139 in v4.14.4), I've edited the issue bodypassionworkeer commentedon Sep 24, 2026
Repro and workaround sketch confirm the same root cause on a self-hosted v4 events_only stack —
GET /api/public/v3/tracesand the v3 observations path return events_only errors, while v2 observations works. Routing batch_evaluation throughGET /api/public/v2/observationsis the natural fix.One open question that would change the PR shape: would you prefer the default switch (the v3 read path is deprecated with EOL 2026-11-16 anyway) or a flag-gated rollout behind
LANGFUSE_USE_V2_OBSERVATIONS_API? Happy to align to whatever your usual approach is here.I have a fix + regression tests ready. Will send the PR within ~24h once direction is set; if you'd rather I hold until a maintainer pings, just say so.
hassiebp commentedon Oct 14, 2026
Fixed in v5.0.1: batch evaluation now reads through the v2 observations API (#1936), so it works on v4
events_onlydeployments. See the v4 → v5 migration guide.