Skip to content

fix(evaluation): isolate EvalCases per trace in tracing JSON import (#1021) - #1072

Open
klioen wants to merge 1 commit into
volcengine:mainfrom
klioen:fix/1021-multi-trace-evalset-collapse
Open

fix(evaluation): isolate EvalCases per trace in tracing JSON import (#1021)#1072
klioen wants to merge 1 commit into
volcengine:mainfrom
klioen:fix/1021-multi-trace-evalset-collapse

Conversation

@klioen

@klioen klioen commented Sep 7, 2026

Copy link
Copy Markdown

Summary

Fixes #1021: when BaseEvaluator.build_eval_set() imports a tracing JSON file containing multiple traces, the previous code accumulated all spans into a single conversation and a single EvalCase — mixing tool uses, session metadata (app_name/user_id) and user/model turns across independent sessions.

Changes

In veadk/evaluation/base_evaluator.py (_build_eval_set_from_tracing_json):

  1. One EvalCase per traceeval_cases, conversation, app_name/user_id and creation_timestamp are now reset for each trace_id group instead of being shared across the whole file.
  2. Sort spans by start_time within a trace — the first/last call_llm spans now reliably map to that trace's user input and final output, even when spans are not ordered in the JSON file.
  3. EvalSet timestamp — derived from the earliest generated case instead of whatever the last loop iteration left behind.

Test

Added test_tracing_file_multiple_traces_to_evalset covering:

  • two traces → two isolated eval cases
  • out-of-order spans within a trace (regression for the start_time sorting)
  • session metadata, tool uses and conversations never crossing trace boundaries
3 passed in 1.76s

Fixes #1021

When building an EvalSet from a tracing JSON file containing multiple
traces, the previous code accumulated all spans into a single
conversation and a single EvalCase, mixing tool uses, session metadata
(app_name/user_id) and user/model turns across independent sessions.

Fix _build_eval_set_from_tracing_json to:
- group spans by trace_id and sort each trace's spans by start_time so
  the first/last call_llm spans map to that trace's input/output
- emit one isolated EvalCase per trace
- derive the EvalSet creation timestamp from the earliest case

Adds a regression test with two traces (one with out-of-order spans)
verifying cases, metadata, tools and conversations stay isolated.

Fixes volcengine#1021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Multi-trace files collapse into one eval case

1 participant