Repository navigation
Feat/metrics eval - #15
Merged
Merged
Conversation
added 24 commits
September 29, 2026 17:30
…ed metrics - Add a tool_step protocol node that calls one MCP tool directly with the upstream payload and wired Script code: optional XGBoost hyperparameter sanitizer, script/payload hash verification, provenance recorded before the call. - Add derived metric producers: asaree.tool_step (scalar result fields by dotted path, e.g. test_metrics.*) and asaree.feature_pipeline (DC/FTE/FS feature counts computed like the notebook's process_metrics). - Frontend: Tool Step node, inspector, connection rules, layout, accent (--node-11), and metrics-editor display for derived producers. - BDM myocardial publications: replace the score agent/parser with the Tool Step, ReAct max_iterations 12, 20 replicates; update README/tests.
- Tool Step card is w-56 (was w-36) with larger icon/text and a stronger glow; layout host width updated so its Tool connector stays aligned. - Add left/right MainEdgeAddStub, like AgentNode. - --node-11 is now a near-white chrome, lighter than every other node. - Re-center the score MCP Tool and Script nodes under the wider Tool Step in the BDM myocardial publications.
Reuse FileDropInput in the create-from-file dialog so the chosen file is visible in the field, and clear the input after each pick so re-selecting the same file still fires.
Arguments are mapped per parameter from the tool's input schema (fixed value, upstream payload, script code, workspace ID), and hash checks are opt-in. Hyperparameter sanitizing moves into run_model_script, driven by a caller-supplied param_spec_json; the BDM publications pass the XGBoost spec as data and bind payload-sha256 to sanitized_payload_sha256.
Replace the standalone floating conversation transcript with a Latest Run results panel, and label every results panel and reopen button with the run kind, short id and start time so two open runs can't be confused.
Stop disabling cell and replicate run buttons when the canvas was never published; the confirm dialog now offers Publish & Run in that case, and names its publish choices after the chosen action instead of Test Run.
Tool Step results and Agent outputs can now be recorded field by field as custom metrics. The BDM definitions use this, rebuild the notebook's MLM brief from field references, give each critic the task_brief, drop FS fields the notebook never had, and restore the notebook's contract descriptions.
The runtime producer now counts critic invocations, rejections split by partial vs. full scope, revision rounds and Reason+Act runs that hit their iteration ceiling. The feature-pipeline producer reports the FTE recipe's op count, chain depth and structural hash. The BDM myocardial definitions bind all of them.
An edge into an Agent can pass the full output (default), only the sender's extracted fields, or selected fields (list fields optionally narrowed to item keys). A gate's outgoing edge offers its worker's fields; edges into a Critic Gate always pass the full output. Narrowed edges fall back to the full output when nothing was extracted, and publish refuses selections the sender's Output Parser doesn't declare. The BDM MLM now receives only the notebook's FS brief fields over its incoming edge, and its goal drops the duplicate FS references.
{{node:X.field[key, ...]}} plucks those keys from each item of a list
field, sharing the narrowing used by edge handoffs. The BDM MLM brief
now passes only the FTE recipe's step names and each encoding's
feature/encoding, matching the notebook's summarize_for_mlm.
Adds prompt_sha256 to the runtime producer (hashed from the published, factor-resolved revision's prompts, critic criteria and output contracts) and a node-scoped runtime producer, asaree.node_runtime, that sums runs launched by the nodes in config.node_ids. The BDM experiments bind one per stage for tokens_<stage> and n_turns_<stage>, and drop v0.8.0 from their filenames.
Manage metrics can now create and edit asaree.node_runtime metrics: pick
Agents and Critic Gates and a measure (tokens, turns, cost, ...), one
binding per metric. Custom metric plan definitions are kept on save
instead of being cut to {id, name}, which would have stripped a node
runtime metric's value type, and the backend fills them in if missing.
Also delete the unused CreateCustomMetricDialog.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds deterministic, code-computed scoring and richer process/runtime metrics to protocol runs, plus a set of canvas and Runs-tab UX improvements. Also brings the BDM myocardial experiments in line with the spinal notebook.
Metrics & scoring
services/tool_steps.py,ToolStepNode/ToolStepNodeInspector); the old Tool Step config format now gets a named validation error.services/derived_metrics.py,nodeRuntimeMetrics.ts,MetricFieldPicker).runtime_metrics.py,reported_metrics.py).promptReferences,metricFields).Protocol canvas
EdgeHandoffPanel); inserting a node on an edge keeps its handoff.Runs & results
Other
Testing
New/updated tests:
test_tool_steps.py,test_runtime_metrics.py,test_reported_metrics.py,test_prompt_references.py,test_bdm_use_cases.py,test_skills_api.py,test_run_model_script_sanitize.py, plus frontend tests for the canvas, run dialogs, metric dialogs, edge handoff and metric fields.