Skip to content

fix(eval): judge 预检与解析对同一 env 结论不一致,空白 API key 通过预检、带空白的 provider 路由错端点 - #307

Merged
helsome merged 1 commit into
helsome:mainfrom
wxrbyte:codex/fix-judge-config-normalization
Oct 13, 2026
Merged

helsome merged 1 commit into
helsome:mainfrom
wxrbyte:codex/fix-judge-config-normalization

Conversation

@wxrbyte

@wxrbyte wxrbyte commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

问题与修复

Closes #306

judge-client.ts 里两个相邻函数对同一份配置用了不同的归一化:

provider model apiKey
resolveJudgeConfigStatus(:154-156) .trim() .trim() 原样
resolveJudgeConfig(:178-180) 只 .toLowerCase() 原样 原样

而 resolveJudgeConfigStatus 在 :166 行把未修剪的值交给后者,于是「预检通过」不等于「配置可用」。修复前直接打印的实际输出:

API key = "   "
  preflight : requested=true missing=[] config=true
  resolved  : provider="anthropic" model="claude-sonnet-4" apiKey="   "

provider = " anthropic"
  preflight : requested=true missing=[] config=true
  resolved  : provider="openai-compatible" model="claude-sonnet-4" apiKey="k"

API key = "sk-test\n"
  preflight : requested=true missing=[] config=true
  resolved  : ... apiKey="sk-test\n"

修复后:

API key = "   "
  preflight : requested=true missing=["API key (--judge-api-key / FINAGENT_JUDGE_API_KEY)"] config=false

provider = " anthropic"
  resolved  : provider="anthropic" model="claude-sonnet-4" apiKey="k"

API key = "sk-test\n"
  resolved  : ... apiKey="sk-test"

用户可见后果:scripts/eval/run.ts:815 打印的会是 Judge: openai-compatible/claude-sonnet-4,且 baseUrl 取 FINAGENT_JUDGE_BASE_URL ?? OPENAI_BASE_URL(:198)—— 把 Anthropic 的模型名发到 OpenAI 端点。空白 key 则要等到真正判分时才 JUDGE_API_ERROR,而 resolveJudgeConfigStatus 的职责(issue #113:显式失败而不是静默降级为仅确定性评估)恰恰是提前拦住它。

修复:新增模块内 judgeSetting(),provider / model / apiKey 在两处都经它归一化(baseUrl 两边都不归一化,本次不动)。

可复现测试报告

  • 环境:Windows 11;Bun 1.4.2;上游 main 3a17eca6;head ecf3d64c
  • 新增 packages/shared/src/evaluation/judge-client.test.ts(7 条用例)
    • 修复前(用 swap_files.py 把 judge-client.ts 切回 main 原样字节):3 pass / 4 fail
    • 修复后:7 pass / 0 fail
  • 该测试文件此前不存在:judge-client 在 evaluation/ 下原本没有对应测试
  • bun test packages/shared/src/evaluation --isolate:195 pass / 1 fail
  • bun test packages/shared --isolate(全量):1139 pass / 1 fail(1140 tests / 103 files)
  • bun run typecheck:core / i18n / shared / ui / electron 五工作区 exit 0
  • secret scan(照抄 pr.yml 口径,排除 node_modules / docs / *.test.ts(x) / bun.lock):命中 0
  • check_collisions.py:81 个 open PR 求交,两个文件均 collisions=0

关于失败项与本地基线

唯一失败 langfuse backend > does not throw when Langfuse is down — agent path keeps a diagnostic 是既存问题:把 packages/shared/src/evaluation/langfuse/resolve.ts 与 packages/shared/src/thesis/repository.ts 都还原成 main 原样字节后单跑 langfuse.test.ts,结果仍是 12 pass / 1 fail(A/B 后已还原,md5 校验一致)。

另需说明本地基线:工作树是「Git Data API 推送」流程留下的累积基线,除本 PR 外还含本轮另外三个 PR(#303 / #305)以及我在 #270 方向上尚未合并的改动(packages/shared/src/evaluation/aggregate.ts 的 higherIsBetter 方向判定及其测试)。因此全量用例数(1140)比「main + 本 PR」多;失败项与这个数量无关。

  • 未运行:完整 e2e。PR checks 工作流本次开工实测仍为 disabled_manually,新 PR 不产生 check run,故按仓库等价命令在本地补齐上述检查,不声称 CI 通过。

边界与范围

  • 只改 judge-client.ts:新增 judgeSetting(),三个字段改走它。
  • 不改 resolveJudgeConfigStatus 的返回结构与 missing 文案、不改 requested 语义、不改 resolveJudgeConfig 的返回形状、header 解析与 baseUrl。
  • 不改 scripts/eval/run.ts 的调用方式(该文件同时被 open PR test(eval): 补齐 #114 的 CLI→Service→runtime→metadata 链路集成测试 #269 修改,避免无谓重叠)。
  • 只触碰 packages/shared/src/evaluation/judge-client.ts 与新增的同名测试文件。

UI 变化

无可见 UI 变化。仅影响评估 CLI / 主进程在构造 judge client 时的取值。

resolveJudgeConfigStatus validated trimmed provider/model values but handed the
untrimmed env to resolveJudgeConfig, which only lowercased. Two consequences,
both observable:

- a whitespace-only FINAGENT_JUDGE_API_KEY passed the preflight with an empty
  `missing` list, so the client was built with a blank key and only failed at
  request time;
- a provider value with surrounding whitespace (`" anthropic"`) passed the
  preflight as anthropic, while resolveJudgeConfig fell through to
  'openai-compatible' and picked the OpenAI endpoint for an Anthropic model.

Trim provider, model, and apiKey in both functions so "configured" means the
same thing on both sides. baseUrl is untouched: neither function normalizes it.

@helsome helsome left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这次只统一 judge provider/model/apiKey 的输入归一化,修复预检与实际构造配置不一致;没有改变外部接口或扩大预检职责。已核对 #306 认领、完整两文件 diff 和七条回归用例。作者提供 Windows/Bun 及 evaluation/shared/typecheck 命令,focused 由 3/4 变为 7/0;注意全仓报告来自含其他改动的本地工作区,不能当成此 PR 的独立全仓通过,本次批准依据是聚焦回归与小范围配置代码审查。当前 head 无 CI run,不声称 CI 已通过;此纯归一化修复不要求付费模型调用或截图。

@helsome
helsome merged commit 83fcd40 into helsome:main Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(eval): judge 预检与解析对同一 env 结论不一致,空白 API key 通过预检、带空白的 provider 路由错端点

2 participants