Repository navigation
fix(eval): judge 预检与解析对同一 env 结论不一致,空白 API key 通过预检、带空白的 provider 路由错端点 - #307
Merged
helsome merged 1 commit intoOct 13, 2026
Merged
Conversation
resolveJudgeConfigStatus validated trimmed provider/model values but handed the untrimmed env to resolveJudgeConfig, which only lowercased. Two consequences, both observable: - a whitespace-only FINAGENT_JUDGE_API_KEY passed the preflight with an empty `missing` list, so the client was built with a blank key and only failed at request time; - a provider value with surrounding whitespace (`" anthropic"`) passed the preflight as anthropic, while resolveJudgeConfig fell through to 'openai-compatible' and picked the OpenAI endpoint for an Anthropic model. Trim provider, model, and apiKey in both functions so "configured" means the same thing on both sides. baseUrl is untouched: neither function normalizes it.
helsome
approved these changes
Oct 13, 2026
helsome
left a comment
Owner
There was a problem hiding this comment.
这次只统一 judge provider/model/apiKey 的输入归一化,修复预检与实际构造配置不一致;没有改变外部接口或扩大预检职责。已核对 #306 认领、完整两文件 diff 和七条回归用例。作者提供 Windows/Bun 及 evaluation/shared/typecheck 命令,focused 由 3/4 变为 7/0;注意全仓报告来自含其他改动的本地工作区,不能当成此 PR 的独立全仓通过,本次批准依据是聚焦回归与小范围配置代码审查。当前 head 无 CI run,不声称 CI 已通过;此纯归一化修复不要求付费模型调用或截图。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题与修复
Closes #306
judge-client.ts里两个相邻函数对同一份配置用了不同的归一化:resolveJudgeConfigStatus(:154-156).trim().trim()resolveJudgeConfig(:178-180).toLowerCase()而
resolveJudgeConfigStatus在:166行把未修剪的值交给后者,于是「预检通过」不等于「配置可用」。修复前直接打印的实际输出:修复后:
用户可见后果:
scripts/eval/run.ts:815打印的会是Judge: openai-compatible/claude-sonnet-4,且 baseUrl 取FINAGENT_JUDGE_BASE_URL ?? OPENAI_BASE_URL(:198)—— 把 Anthropic 的模型名发到 OpenAI 端点。空白 key 则要等到真正判分时才JUDGE_API_ERROR,而resolveJudgeConfigStatus的职责(issue #113:显式失败而不是静默降级为仅确定性评估)恰恰是提前拦住它。修复:新增模块内
judgeSetting(),provider/model/apiKey在两处都经它归一化(baseUrl两边都不归一化,本次不动)。可复现测试报告
3a17eca6;headecf3d64cpackages/shared/src/evaluation/judge-client.test.ts(7 条用例)swap_files.py把judge-client.ts切回 main 原样字节):3 pass / 4 failjudge-client在evaluation/下原本没有对应测试bun test packages/shared/src/evaluation --isolate:195 pass / 1 failbun test packages/shared --isolate(全量):1139 pass / 1 fail(1140 tests / 103 files)bun run typecheck:core / i18n / shared / ui / electron 五工作区 exit 0pr.yml口径,排除node_modules/docs/*.test.ts(x)/bun.lock):命中 0check_collisions.py:81 个 open PR 求交,两个文件均collisions=0关于失败项与本地基线
唯一失败
langfuse backend > does not throw when Langfuse is down — agent path keeps a diagnostic是既存问题:把packages/shared/src/evaluation/langfuse/resolve.ts与packages/shared/src/thesis/repository.ts都还原成 main 原样字节后单跑langfuse.test.ts,结果仍是 12 pass / 1 fail(A/B 后已还原,md5 校验一致)。另需说明本地基线:工作树是「Git Data API 推送」流程留下的累积基线,除本 PR 外还含本轮另外三个 PR(#303 / #305)以及我在 #270 方向上尚未合并的改动(
packages/shared/src/evaluation/aggregate.ts的higherIsBetter方向判定及其测试)。因此全量用例数(1140)比「main + 本 PR」多;失败项与这个数量无关。PR checks工作流本次开工实测仍为disabled_manually,新 PR 不产生 check run,故按仓库等价命令在本地补齐上述检查,不声称 CI 通过。边界与范围
judge-client.ts:新增judgeSetting(),三个字段改走它。resolveJudgeConfigStatus的返回结构与missing文案、不改requested语义、不改resolveJudgeConfig的返回形状、header 解析与baseUrl。scripts/eval/run.ts的调用方式(该文件同时被 open PR test(eval): 补齐 #114 的 CLI→Service→runtime→metadata 链路集成测试 #269 修改,避免无谓重叠)。packages/shared/src/evaluation/judge-client.ts与新增的同名测试文件。UI 变化
无可见 UI 变化。仅影响评估 CLI / 主进程在构造 judge client 时的取值。