Skip to content

fix(eval): LANGFUSE_TRACING 的开关解析有两套实现,TRUE / ' on ' 等写法在真实后台上静默失效 - #305

Merged
helsome merged 1 commit into
helsome:mainfrom
wxrbyte:codex/fix-langfuse-tracing-flag-parsing
Oct 12, 2026
Merged

helsome merged 1 commit into
helsome:mainfrom
wxrbyte:codex/fix-langfuse-tracing-flag-parsing

Conversation

@wxrbyte

@wxrbyte wxrbyte commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

问题与修复

Closes #304

LANGFUSE_TRACING 有两个解析器,而生产只用了较严格的那个:

  • 生产唯一调用的 resolveLangfuseBackend(resolve.ts:22-26):=== 'true' | '1' | 'yes' | 'on' 精确匹配
  • 被导出、被测试、生产从不调用的 isLangfuseTracingEnabled(backend.ts:224-233):trim().toLowerCase(),并处理 0/false/off/no

全仓对 isLangfuseTracingEnabled 只有一处引用——langfuse.test.ts:356 自己。于是同一环境变量在两个导出函数下结论相反:

LANGFUSE_TRACING isLangfuseTracingEnabled resolveLangfuseBackend().kind(修复前)
true / 1 / yes / on true langfuse
TRUE / True / YES / ON / ' on ' / ' true ' true none
0 / false / off / no / maybe false none

修复:resolveLangfuseBackend 直接调用已导出的 isLangfuseTracingEnabled(input.settings, env),开关只留一份实现。settings.langfuseTracingEnabled === true 的优先级不变(helper 里仍是首个判断),负面词与未知值仍为关闭。

影响面:docs/langfuse-tracing.md:13 把该变量列为开启跟踪的开关。按上表第二行设置时,resolveLangfuseBackend 返回 NoopEvaluationBackend,kernel 的 refreshEvaluationBackend(kernelHost.ts:1294)把 langfuseBackend 置空,scripts/eval/run.ts:826 同样静默跳过——没有任何报错,只是"开了但没数据"。

可复现测试报告

  • 环境:Windows 11;Bun 1.4.2;上游 main 3a17eca6;head 见本 PR 提交
  • 新增 packages/shared/src/evaluation/langfuse/tracing-flag.test.ts(4 条用例)
    • 修复前(用 swap_files.py 把 resolve.ts 切回 main 原样字节):2 pass / 2 fail,失败形式为 Expected: "langfuse" / Received: "none"
    • 修复后:4 pass / 0 fail
  • bun test packages/shared/src/evaluation/langfuse --isolate:16 pass / 1 fail
  • bun test packages/shared --isolate(全量):1131 pass / 2 fail,两条均为既存问题:
    • langfuse backend > does not throw when Langfuse is down — agent path keeps a diagnostic —— 已 A/B 验证在 main 原始字节下同样失败(12 pass / 1 fail),属既存
    • AgentKernel E2E agent loop > rejects deleting a session while a run is active in it —— beforeEach/afterEach 钩子超时(5.9s),时序抖动(同一命令在另一轮里失败项是 ResearchService > lists runs newest-first,同为钩子超时,失败项不稳定本身说明与本次改动无关)
  • bun run typecheck:core / i18n / shared / ui / electron 五工作区 exit 0
  • secret scan(照抄 pr.yml 口径,排除 node_modules / docs / *.test.ts(x) / bun.lock):命中 0
  • 未运行:完整 e2e。PR checks 工作流本次开工实测仍为 disabled_manually,新 PR 不产生 check run,故按仓库等价命令在本地补齐上述检查,不声称 CI 通过

边界与范围

  • 只改 resolve.ts 的开关判定;isLangfuseTracingEnabled 自身语义不变。
  • 不改凭证解析(parseLangfuseCredential / langfuseCredentialsFromEnv)、host 归一化、脱敏级别与任何后端行为。
  • 未改 scripts/eval/run.ts:那里的第三份内联副本(只认 true/1/yes)在最终结果上不起作用,因为 resolve.ts 同时也读 env。该文件同时被 open PR test(eval): 补齐 #114 的 CLI→Service→runtime→metadata 链路集成测试 #269(test(eval): 补齐 #114 的 CLI→Service→runtime→metadata 链路集成测试,非本人)修改,为避免无谓重叠,本次不动它,留作后续清理。
  • 文件级碰撞:check_collisions.py 对 80 个 open PR 求交,resolve.ts / backend.ts / 新测试文件均 collisions=0(仅 scripts/eval/run.ts 命中 test(eval): 补齐 #114 的 CLI→Service→runtime→metadata 链路集成测试 #269,且本 PR 不触碰该文件)。

UI 变化

无可见 UI 变化。仅影响跟踪开关注解后是否真的创建 Langfuse 后端。

本地基线的补充说明

工作树是「Git Data API 推送」流程留下的累积基线,除本 PR 外还含本轮另外几个 PR 以及我在 #270 方向上尚未合并的改动(packages/shared/src/evaluation/aggregate.ts 的 higherIsBetter 方向判定及其测试)。因此上文的全量用例数比「main + 本 PR」多;失败项与这个数量无关,两条失败均可在 main 原始字节下复现。

resolveLangfuseBackend re-implemented the LANGFUSE_TRACING switch with exact
string comparisons, while the exported, tested isLangfuseTracingEnabled
(backend.ts) trims and lowercases the value. Production only ever called the
former, so a switch written as TRUE / True / ' on ' turned tracing on for the
helper and off for every real backend — resolveLangfuseBackend returned a Noop
backend with no error, and neither the kernel nor the eval CLI emitted anything.

Call the exported parser from resolveLangfuseBackend so the switch has one
implementation. The stored setting still wins, and the explicit negative words
(0/false/off/no) still mean off.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(eval): LANGFUSE_TRACING 的开关解析有两套实现,TRUE / ' on ' 等写法在真实后台上静默失效

2 participants