fix: prevent eval trial upsert collisions (COR-84) - #2476
Open
paultancre-bt wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Related to COR-84. Addresses the JS counterpart of the P1 raised on the Python SDK fix.
The intent is to keep one independently updatable result per evaluation trial. The JS SDK already preserves
upsert_id, but currently reuses that exact record ID for every trial. With multiple trials, their root records overwrite each other, losing trial results and potentially leaving task spans attached to the wrong surviving root. Generating fresh random IDs would avoid collisions but break rerun upserts.This fixes an existing JS multi-trial bug, not Python's original dropped-
upsert_idbug. It matches the per-trial ID scheme in braintrust-sdk-python#763.Repro
upsert_id: "eval-row"andtrialCount: 3, with a task returning a different output for each trial index.id: "eval-row".The regression reproduces this for both experiment-backed and remote-parent execution, and with global or per-row trial counts. Before the fix, four multi-trial cases fail with one logged root instead of three; the other 20 cases pass. Two remote eval columns are not required to trigger this P1.
Fix
upsert_idfor trial zero, preserving single-trial behavior.braintrust:eval:<upsert_id>:trial:<trial_index>. Each trial has a distinct ID that stays stable on rerun, matching Python.braintrust.The behavior change is limited to additional trials with a non-empty upsert ID. This prevents future trial collisions after upgrading; it does not restore overwritten results or clean up historical records.
Test
pnpm test src/framework.test.ts --reporter=dot: all 104 tests pass, including 24 regression cases covering supplied/missing/empty IDs, global/per-row trial counts, and experiment/parent-context execution.pnpm run check:typings: passes for production and test code.pnpm run fix:formattingandgit diff --check: pass.