Ab test/agent action shadow - #69
Merged
Merged
Conversation
feat: 添加 python-multipart 包及其依赖
- Introduced new metrics: action_rejection_count, answer_call_count, decision_call_count, decision_fallback_count, guard_retry_count, and provider_retry_count to conversation-detail.schema.json, workflow-result.schema.json, and workflow-stream-event.schema.json. - Enhanced test coverage for runtime metrics in various test files, including test_ab_runtime.py, test_agent_loop.py, test_eval_runner.py, test_exam_review_plan.py, test_iteration_3_runtime.py, and test_openrouter_models.py. - Implemented logic to ensure metrics are accurately tracked during workflow runs and decision-making processes.
- Removed frozen BYOK providers and replaced with a dynamic connection model. - Updated the API to handle BYOK connection inputs, including display name, base URL, model ID, and API key. - Refactored tests to accommodate changes in BYOK connection structure and validation. - Enhanced the UI for managing BYOK connections, including input validation and connection creation. - Updated model selection logic to reflect the new BYOK connection model. - Adjusted the app configuration to remove deprecated BYOK provider references.
feat: add SCUT Senior documentation and debugging scripts
Enhance database cleanup and support multi-file upload in frontend
同步一下修复，下阶段是RRF混排策略提升和agent loop决策实验
- Updated scut-real-corpus-cases.json to include new evaluation status and notes. - Introduced audit_evaluation_sets.py to inventory legacy annotations against the active corpus. - Added build_reviewed_evaluation.py to materialize authored annotations and scenarios. - Modified test_eval_runner.py to ensure cross-course support is enabled and functioning. - Created test_learning_eval.py to validate various scoring and evaluation scenarios.
- Introduced visual-reviewed.json containing semantic answer keys for image-only source pages. - Implemented build_coverage_harness.py to generate a coverage harness for active courses, focusing on retrieval evidence. - Updated build_reviewed_evaluation.py to merge topics from expanded annotations. - Enhanced test_learning_eval.py with tests for coverage harness validation and visual evaluation integrity.
…narios - Introduced `build_student_scenarios.py` to generate realistic student workflow cases from reviewed semantic anchors. - The script aggregates scenarios by anchor_topic_id, ensuring multi-turn robustness without creating new answer keys. - Added tests to validate scenario grouping by anchor topic and ensure the integrity of student scenario expansion. - Updated existing tests to verify the preservation of anchor clusters and course diversity in generated scenarios.
布置智能基座人员任务至实验分支
- Dense 检索默认改为只读 SQLite 向量快照 + NumPy float32 矩阵余弦搜索，按语料版本、课程和模型身份缓存；可用 SCUT_SENIOR_VECTOR_SEARCH_ENGINE=scalar 回退。 - 同一检索请求会批量编码并复用重复 query variants，跨课程请求使用同一 active corpus snapshot。 - 加入可选 protected_rrf_v1：只硬保护唯一题号或完整非泛化标题，其余词法与 dense 候选按加权 RRF 排序。默认仍是旧的 lexical_first_v1，不会直接改变线上结果。
…ated components - Introduced `PersonaEnhancement` and `PersonaEnhancementOutcome` enums in the workflow-stream-event schema. - Updated workflow request and result interfaces to include persona enhancement properties. - Enhanced tests to cover new persona enhancement features and ensure compatibility with existing functionality. - Modified frontend components to support persona enhancement settings and display relevant information. - Implemented runtime health check to determine if humanizer is configured for persona enhancements.
humanizer终于不是空转 从第一期计划 到第四期计划才接入模型闭合
- Added `decide_action` method to `OpenAICompatibleByokGateway` for bounded agent actions. - Implemented connection validation in BYOK model decision-making. - Introduced `BoundByokActionGateway` to encapsulate BYOK credential handling during agent actions. - Enhanced humanizer response error handling with specific exceptions for invalid responses. - Updated `optional_model_timeout_seconds` to allow longer processing times for optional model work. - Added provider status code tracking to `TraceSafeResult` for improved diagnostics. - Updated schemas to include `provider_status_code` for better traceability. - Enhanced tests to validate new BYOK model behavior and humanizer response parsing. - Documented new memory and context management strategies in plan-5.md.
- Increased the maximum runtime from 120 seconds to 180 seconds across various components, including AgentBudget and main application settings. - Updated soft runtime limits accordingly to 135 seconds. - Introduced optional timeout and repair context parameters in ModelGateway and UserKeyModelGateway protocols. - Enhanced AnswerGenerator to support optional repair instructions without altering user input. - Improved lifecycle management with new methods for calculating primary model timeout. - Refined retrieval logic to ensure fallback mechanisms retain first authorized evidence during query rewrites. - Updated tests to reflect changes in runtime limits and to verify new functionality around citation repair and shadow mode behavior. - Adjusted documentation to align with the new runtime policies and error handling strategies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
第三期智能仓库建设——harness改进计划完成（memory/agent loop/retrieval enhance/context/tool calling）
A/B分支平衡了成本和检索效率，rule为master基线；
export SCUT_SENIOR_AGENT_DECISION_MODE=shadowA/B 后检索决策模式：rule 为默认确定性基线；可改为 shadow、model 或 deterministic 做对照实验。
本次核心改动如下：
仅同一会话内、已完成的最近 6 轮问答进入模型上下文；每个用户问题和助手回答仍截断至 2,000 字符；后续检索只携带最近 2 条用户问题，每条最多 400 字，组合查询最多 1,200 字符；实现滚动摘要来有效避免关键结论丢失。
想体验本产品并提出意见同学联系本人，本项目尚未迁移服务器，考虑到服务器请求openrouter网关问题以及费用问题，可能长期处于当前产品状态。