W-22203426: feat: move eval normalizer, formatter, YAML translator, and runner into library#276
Merged
Merged
Conversation
…nd runner into library Extracts evalNormalizer, evalFormatter, yamlSpecTranslator, and the new agentEvalRunner module from plugin-agent into the @salesforce/agents library so any consumer can use them directly without depending on the CLI plugin. All four modules are exported from the package index. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
4 tasks
- Export EvalOutput, EvalResult, TestError, TestResult from evalFormatter so callers get proper types without casting - Rewrite buildResultSummary using the exported types, eliminating all Record<string,unknown> casts - Type AgentEvalRunResult.tests[].evaluations and .outputs as EvalResult[]/EvalOutput[] - Improve resolveAgent error messages to include actionable suggestions - Fix evalCounter to only increment when an id is auto-generated - Replace as-any stub cast with as-unknown StreamPromise in tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
WillieRuemmele
requested changes
May 7, 2026
| const conn = org.getConnection(); | ||
|
|
||
| // Escape single quotes to prevent SOQL injection | ||
| const escapedApiName = apiName.replace(/'/g, "\\'"); |
Member
There was a problem hiding this comment.
I think you'll want to escape it like this
const escapedApiName = apiName.replace(/'/g, "''"); | `SELECT Id FROM BotDefinition WHERE DeveloperName = '${escapedApiName}'` | ||
| ); | ||
| if (!botResult.records.length) { | ||
| throw new Error( |
Member
There was a problem hiding this comment.
these should be SfErrors. There's a few other just plain Errors not SfErrors
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…scratch orgs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…blic API Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Member
QA Notes:done here: salesforcecli/plugin-agent#413 (comment) |
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
WillieRuemmele
approved these changes
May 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@W-22203426@
evalNormalizer,evalFormatter,yamlSpecTranslatorfromplugin-agentinto this shared library so any consumer can use them without depending on the CLI pluginagentEvalRunnermodule that encapsulatesresolveAgent,executeBatches, andbuildResultSummary(API execution logic previously embedded in the run-eval command)index.tsevalNormalizer,evalFormatter,yamlSpecTranslator, andagentEvalRunnerTest plan
yarn compilepasses in agents repoyarn testpasses in agents repo (all new test files: evalNormalizer, evalFormatter, yamlSpecTranslator, agentEvalRunner)plugin-agentresolves correctly🤖 Generated with Claude Code