fix: handle RemoteEvalParameters in eval dev /list endpoint#192
Draft
Alex Jean-Baptiste (justcodebruh) wants to merge 1 commit into
Draft
fix: handle RemoteEvalParameters in eval dev /list endpoint#192Alex Jean-Baptiste (justcodebruh) wants to merge 1 commit into
Alex Jean-Baptiste (justcodebruh) wants to merge 1 commit into
Conversation
`build_eval_definitions` crashed with `AttributeError: 'RemoteEvalParameters'
object has no attribute 'items'` when an `Eval(...)` passed
`parameters=load_parameters(...)`. The runner called
`parameters_to_json_schema(evaluator.parameters)` unconditionally; that
helper expects an `EvalParameters` mapping and calls `.items()` on it,
which `RemoteEvalParameters` does not implement.
The dev server's `/list` handler in Rust only surfaced this as a generic
500 ("Eval runner exited with an error."), so the Playground Remote evals
settings page just said "could not connect to the dev server" with no
indication that the saved-parameters path was the trigger.
Branch on `RemoteEvalParameters` and pass through `.schema` directly,
matching how the SDK itself handles this object in
`framework.py:1515` and `parameters.py:357`. The other dev_mode == "list"
paths (`parameters_to_json_schema` for inline `EvalParameters`, empty
when no parameters are declared) are preserved.
Pylon: #15895
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Latest downloadable build artifacts for this PR commit
Available artifact names
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AttributeError: 'RemoteEvalParameters' object has no attribute 'items'raised bybuild_eval_definitionswhen anEval(...)is declared withparameters=load_parameters(...)./listendpoint was masking this as a generic 500 (Eval runner exited with an error.), so the Playground Remote evals settings page just said "could not connect to the dev server" with no actionable trace.RemoteEvalParametersin the runner and pass through its already-resolved.schema, matching what the SDK itself does atbraintrust/framework.py:1515andbraintrust/parameters.py:357.Bug path
params = load_parameters(project=..., slug=...)→ returns aRemoteEvalParametersobject.parameters=paramsintoEval(...)./liston the dev server → bt spawnseval-runner.pywithBT_EVAL_DEV_MODE=list.build_eval_definitions(waseval-runner.py:570) callsparameters_to_json_schema(evaluator.parameters)unconditionally; that helper callsparameters.items(), whichRemoteEvalParametersdoes not implement → uncaughtAttributeError.dev_server_listreturns a generic 500 with no error detail (the traceback goes to stderr, not throughsse.send(\"error\", ...)).Verification (local)
BT_EVAL_DEV_MODE=list python scripts/eval-runner.py eval_repro_15895.py --localagainst the sameEval(..., parameters=load_parameters(project=\"repro-15895\", slug=\"eval-config\"))repro: previously crashed with theAttributeErrorat line 570 → now emits the correctly-serialized JSON schema.cargo build --releaseand ran the patched binary against the same eval file.curl http://localhost:8301/list(withAuthorization: Bearer+x-bt-org-name) previously returned500 {\"error\":\"Eval runner exited with an error.\"}; now returns200with the full param schema (model + eval_prompt + temperature + system_prompt all resolved).Test plan
bt eval <file>.py --language=python --dev --dev-port 8301against anEval(..., parameters=load_parameters(...))no longer crashes the dev server, and the Playground Remote evals settings page lists the eval with all saved parameters visibleEval(parameters={\"prefix\": PrefixParam, \"main\": {\"type\": \"prompt\", ...}})flow is unchangedPylon: #15895
🤖 Generated with Claude Code