Repository navigation
fix(eval): LANGFUSE_TRACING 的开关解析有两套实现,TRUE / ' on ' 等写法在真实后台上静默失效 - #305
Merged
helsome merged 1 commit intoOct 12, 2026
Merged
Conversation
resolveLangfuseBackend re-implemented the LANGFUSE_TRACING switch with exact string comparisons, while the exported, tested isLangfuseTracingEnabled (backend.ts) trims and lowercases the value. Production only ever called the former, so a switch written as TRUE / True / ' on ' turned tracing on for the helper and off for every real backend — resolveLangfuseBackend returned a Noop backend with no error, and neither the kernel nor the eval CLI emitted anything. Call the exported parser from resolveLangfuseBackend so the switch has one implementation. The stored setting still wins, and the explicit negative words (0/false/off/no) still mean off.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题与修复
Closes #304
LANGFUSE_TRACING有两个解析器,而生产只用了较严格的那个:resolveLangfuseBackend(resolve.ts:22-26):=== 'true' | '1' | 'yes' | 'on'精确匹配isLangfuseTracingEnabled(backend.ts:224-233):trim().toLowerCase(),并处理0/false/off/no全仓对
isLangfuseTracingEnabled只有一处引用——langfuse.test.ts:356自己。于是同一环境变量在两个导出函数下结论相反:LANGFUSE_TRACINGisLangfuseTracingEnabledresolveLangfuseBackend().kind(修复前)true/1/yes/onlangfuseTRUE/True/YES/ON/' on '/' true 'none0/false/off/no/maybenone修复:
resolveLangfuseBackend直接调用已导出的isLangfuseTracingEnabled(input.settings, env),开关只留一份实现。settings.langfuseTracingEnabled === true的优先级不变(helper 里仍是首个判断),负面词与未知值仍为关闭。影响面:
docs/langfuse-tracing.md:13把该变量列为开启跟踪的开关。按上表第二行设置时,resolveLangfuseBackend返回NoopEvaluationBackend,kernel 的refreshEvaluationBackend(kernelHost.ts:1294)把langfuseBackend置空,scripts/eval/run.ts:826同样静默跳过——没有任何报错,只是"开了但没数据"。可复现测试报告
3a17eca6;head 见本 PR 提交packages/shared/src/evaluation/langfuse/tracing-flag.test.ts(4 条用例)swap_files.py把resolve.ts切回 main 原样字节):2 pass / 2 fail,失败形式为Expected: "langfuse" / Received: "none"bun test packages/shared/src/evaluation/langfuse --isolate:16 pass / 1 failbun test packages/shared --isolate(全量):1131 pass / 2 fail,两条均为既存问题:langfuse backend > does not throw when Langfuse is down — agent path keeps a diagnostic—— 已 A/B 验证在 main 原始字节下同样失败(12 pass / 1 fail),属既存AgentKernel E2E agent loop > rejects deleting a session while a run is active in it——beforeEach/afterEach钩子超时(5.9s),时序抖动(同一命令在另一轮里失败项是ResearchService > lists runs newest-first,同为钩子超时,失败项不稳定本身说明与本次改动无关)bun run typecheck:core / i18n / shared / ui / electron 五工作区 exit 0pr.yml口径,排除node_modules/docs/*.test.ts(x)/bun.lock):命中 0PR checks工作流本次开工实测仍为disabled_manually,新 PR 不产生 check run,故按仓库等价命令在本地补齐上述检查,不声称 CI 通过边界与范围
resolve.ts的开关判定;isLangfuseTracingEnabled自身语义不变。parseLangfuseCredential/langfuseCredentialsFromEnv)、host 归一化、脱敏级别与任何后端行为。scripts/eval/run.ts:那里的第三份内联副本(只认true/1/yes)在最终结果上不起作用,因为resolve.ts同时也读 env。该文件同时被 open PR test(eval): 补齐 #114 的 CLI→Service→runtime→metadata 链路集成测试 #269(test(eval): 补齐 #114 的 CLI→Service→runtime→metadata 链路集成测试,非本人)修改,为避免无谓重叠,本次不动它,留作后续清理。check_collisions.py对 80 个 open PR 求交,resolve.ts/backend.ts/ 新测试文件均collisions=0(仅scripts/eval/run.ts命中 test(eval): 补齐 #114 的 CLI→Service→runtime→metadata 链路集成测试 #269,且本 PR 不触碰该文件)。UI 变化
无可见 UI 变化。仅影响跟踪开关注解后是否真的创建 Langfuse 后端。
本地基线的补充说明
工作树是「Git Data API 推送」流程留下的累积基线,除本 PR 外还含本轮另外几个 PR 以及我在 #270 方向上尚未合并的改动(
packages/shared/src/evaluation/aggregate.ts的higherIsBetter方向判定及其测试)。因此上文的全量用例数比「main + 本 PR」多;失败项与这个数量无关,两条失败均可在 main 原始字节下复现。