Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,8 @@ K8S.md

# Machine-local Kernel Lab cluster profiles live under ~/.config/loom and must
# never be copied into this repository.
/loom/kernel_hub/env.*.local
/loom/kernel_hub/env.*.local
# Local runtime state and tool caches
.RUD/
.pytest_cache/
.ruff_cache/
82 changes: 76 additions & 6 deletions docs/AUTO_RESEARCH_SYSTEM_DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ flowchart TB

subgraph LOOP["② Paper 工作台(每篇论文一个回合制状态机)"]
direction TB
AUTHOR["Author Agent(Claude, tmux)<br/>做实验(slurm GPU) + 写 LaTeX"]
AUTHOR["Author Agent(Cursor, tmux)<br/>SSH 空闲 H100 做实验 + 写 LaTeX"]
READY{"Readiness Gate<br/>确定性检查"}
PANEL["Reviewer Panel<br/>三模型读编译后 PDF<br/>最低分定档"]
STOP{"停止条件<br/>达标 / 满轮 / 平台期"}
Expand Down Expand Up @@ -60,7 +60,7 @@ flowchart TB
I1["tmux Agent 池<br/>+ Web 实时面板"]
I2["Web UI / API<br/>:8766 + 公网隧道"]
I3["Hot Restart<br/>不断任务换代码"]
I4["slurm H100 集群"]
I4["GPU Scout<br/>轮询空闲 H100 + SSH 直跑"]
end

SPAWN --> DG["🧑 Draft Human Gate<br/>批准骨架稿"] --> LOOP
Expand Down Expand Up @@ -105,9 +105,11 @@ flowchart TB

每一轮(round N)内部的固定节拍:

1. **Author Agent**(Claude,常驻 tmux,工作在自己的 worktree)收到本轮 Prompt:
1. **Author Agent**(Cursor Agent,当前默认 `gpt-5.6-sol-max-fast`,常驻 tmux,
工作在自己的 worktree)收到本轮 Prompt:
上一轮的评审报告 + 方法论技能(AR-AUTHOR)+ 图片技能菜单 + GPU 集群使用规范。
它做实验(提交 slurm GPU 任务)、改论文、重编译,最后写 `author.md` 作为完成信号。
它做实验(读取 GPU Scout 后直接 SSH 到空闲 H100)、改论文、重编译,
最后写 `author.md` 作为完成信号。
2. **Readiness Gate(确定性代码)**:编译必须干净;不允许任何 `\ARnum`/TODO/`??` 占位;
各章节实质完整;page-one 总览图必须存在;所有被引用的图文件存在;引用无悬空。
不合格 → 列出失败清单原样打回 Author,本轮重做,不消耗评审。
Expand All @@ -128,7 +130,8 @@ flowchart TB
- `AR-STUDIO.md` / `AR-AUTHOR.md` / `AR-REVIEWER.md`:三种角色的完整方法论;
- `figures/teaser-figure-1..4`、`results-figure-1..2`、`checkbib`:画图与查引用的具体做法
(从纯代码矢量图到 AI 生成再到混合方案,多风格可选,作者按需取用);
- `GPU-RESOURCES.md`:集群使用规范(禁止登录节点跑模型、sbatch 模板、防 GPU 被占的 requeue 守卫);
- `GPU-RESOURCES.md`:集群使用规范(禁止登录节点跑模型;读取 GPU Scout 的实时空闲清单,
直接 SSH 到 compute node 并用 `CUDA_VISIBLE_DEVICES` 启动作业);
- `paper-rebuttal/SKILL.md`、`paper-rebuttal-delivery/SKILL.md`:rebuttal 起草与终稿交付的方法论。

## 5. 模块四:Rebuttal Factory(两级结构 + 双人工 Gate)
Expand Down Expand Up @@ -167,7 +170,9 @@ flowchart TB
Agent 无感知;
- **监控循环**:`delivery_monitor.py` 等看门狗把"Agent 完成 → 校验 → 验收 → 喂回失败报告"的
节拍自动化,出结果或卡死才通知人;
- **slurm H100 集群**:实验全部走 sbatch;登录节点只做聚合和画图。
- **GPU Scout + H100 集群**:守护进程每分钟 SSH 各 compute node 读取真实
`nvidia-smi`,发布空闲 `node:gpu` 清单;Agent 不走不可靠的 Slurm 排队,
而是二次确认显存后直接 SSH 启动作业。登录节点只做聚合和画图。

---

Expand Down Expand Up @@ -206,3 +211,68 @@ Web/编排 loom/web.py
论文实例 <factory-root>/.RUD/<paper-slug>/{ar.json, rounds/, work/manuscript/main.pdf}
Rebuttal 实例 <paper-dir>/rebuttal-output/{state.json, responses/, delivery/attempts/<run>/deliverables/}
```

---

## 10. 对外项目介绍与当前状态(2026-08-17)

### 10.1 可以怎样向另一个团队介绍

**Loom Auto Research** 是一个面向长周期科研任务的 Agent 编排系统。它不是让一个
聊天模型一次性“写论文”,而是把科研过程拆成可恢复、可审计的状态机:

1. 从会议往届获奖论文、oral、热点和研究者已有能力中生成可证伪的选题;
2. 每个选题孵化为隔离的代码与 LaTeX 工作区,由长期运行的 Cursor Agent 做实验和写作;
3. Python Readiness Gate 先拦截编译错误、占位符、缺图和虚假完成;
4. GPT、Claude、Grok 三个独立 reviewer 只读编译后的 PDF,按最低分推动下一轮修改;
5. 稳定后进入 Delivered;投稿后还可进入 Rebuttal Factory,生成回复、修订稿和提交 bundle。

项目的核心技术价值在于:

- **可靠的长周期 Agent orchestration**:任务跨小时/天运行,进程、服务或会话重启后可从磁盘状态恢复;
- **确定性控制 + 模型创造力**:状态转换、门禁、页数、哈希和完成条件由代码控制,模型只负责研究内容;
- **跨模型 eval**:执行者与评审者隔离,三个模型只看最终 PDF,避免作者自评;
- **真实计算闭环**:Agent 自己写实验代码,GPU Scout 分配实际空闲 H100,结果再写回论文;
- **artifact-level verification**:批准绑定 PDF/文本哈希,任何修改都会令旧批准失效;
- **human-in-the-loop**:系统可以全自动运行,但保留关键 Gate 和实时 tmux 面板供人检查或介入。

### 10.2 当前实现和实跑规模

| 项目状态 | 当前情况 |
|---|---|
| 产品形态 | Research Factory、Paper 工作台、Rebuttal Factory、Web UI/API、实时 tmux 面板 |
| 会议支持 | ICLR、NeurIPS、ICML、COLM、WACV;WACV 支持 Algorithms/Applications/Datasets track |
| 当前实验 | 同时运行 8 篇 WACV 2027 + 8 篇 WSDM 2027 paper |
| 当前结果 | 截至本次快照,4/16 已 Delivered;其余处于第 1–7 轮 Author/Reviewer 循环 |
| 实时状态 | `docs/notes/zhizhou/WACV_WSDM_PAPER_PROGRESS.md` 每分钟按实际状态更新 |
| Rebuttal 验证 | 已在两篇 WACV rebuttal package 上跑通修订稿、一页回复、supplement、重编译和三模型图片验收 |
| 运行基础设施 | 16 个长期 Agent pane、两套 autopilot、GPU Scout、可热重启的 8766 Web 服务 |

这仍是一个研究原型,而不是“科研已被完全自动化”的结论。目前最重要的下一步是:
系统化比较自动生成论文与人工基线的科学质量、减少 agent 过度扩展实验范围、提高 reviewer
评分与人类专家评分的一致性,并把当前针对单个研究者的能力画像产品化为可复用 profile。

### 10.3 可以直接发送的英文消息草稿

> Hi [Name] — I wanted to ask whether there might be an opportunity to intern
> with your team during the fall semester.
>
> I have been building **Loom Auto Research**, an agentic system for long-horizon
> research workflows. It turns a venue and research direction into concrete
> hypotheses, runs real experiments on GPUs, writes and compiles papers, and
> iterates through an independent GPT/Claude/Grok PDF-review panel. A
> deterministic state machine controls readiness checks, recovery, human gates,
> and artifact hashes, so the system can run for days without treating an LLM's
> claim of completion as ground truth. I also built a related rebuttal pipeline
> that produces revised papers, one-page responses, supplements, and validated
> submission artifacts.
>
> In the current evaluation, Loom is running 16 concurrent WACV/WSDM research
> projects; four have reached the Delivered stage and the rest are progressing
> through automated author/reviewer rounds. The project has given me hands-on
> experience with reliable agent orchestration, long-horizon task recovery,
> multi-model evaluation, GPU execution, and human-in-the-loop system design.
>
> I think this work may overlap with your team's interests in [team area]. Would
> you be open to a short conversation about whether I could join your team as a
> fall intern? I would be happy to share a demo and the system design.
121 changes: 121 additions & 0 deletions docs/notes/wsdm2027/wsdm 3 paper.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# wsdm 3 paper

This file contains three WSDM papers. The titles, abstracts, and subject areas
are plain-text, copy-paste-ready CMT registration fields.

- Enter authors, conflicts, and other administrative metadata separately.
- Internal automated reviews are included only for reference; do not paste
them into CMT.

## Registration Summary

| ID | Status | Final round | Primary subject area |
|---|---|---:|---|
| `wsdm-05` | Delivered | 6 | Web Mining and Content Analysis → Web recommender systems and algorithms |
| `wsdm-11` | Delivered | 5 | Foundation Models and Agentic Systems → Evaluation and benchmarking of foundation models in search/mining |
| `wsdm-12` | Delivered | 8 | Web Mining and Content Analysis → Web recommender systems and algorithms |

## 1. wsdm-05

**Status:** Delivered, Round 6

### Title

When Does One Step Suffice? A Four-Gate Audit of Diffusion Recommendation Distillation

### Subject Areas

- **Primary:** Web Mining and Content Analysis → Web recommender systems and algorithms
- **Secondary:** Web Mining and Content Analysis → Scalable algorithms for mining web data, opinion mining and sentiment analysis
- **Secondary:** Web Search → Search benchmarking and evaluation
- **Secondary:** Privacy, Fairness, Interpretability → Model and algorithm transparency

### Abstract

A one-call deployment claim bundles four different statements: the teacher is competent, iterative sampling helps, the student preserves the teacher, and the result has serving value. We introduce a four-gate audit that tests these claims separately and scopes every decision to a checkpoint and protocol.

Using a common full-catalog harness over MovieLens-1M and Steam, together with a native Amazon Beauty DiffuRec reproduction, we compare diffusion teachers, iterative sampling, one-pass controls, and endpoint regression under matched evaluation. Utility-tuned SASRec outperforms the audited teachers in all 16 metric-level comparisons, and DDIM-1 outperforms the multi-step endpoint in 15 of 16; the remaining comparison is inconclusive. Endpoint regression sometimes preserves aggregate utility under simultaneous noninferiority tests, yet exact teacher top-10 set identity never exceeds 20.5%. The contribution is a falsifiable deployment contract and checkpoint-level evidence, not a broad claim that one step or many steps universally wins.

### Internal Automated Review — Do Not Paste into CMT

Round 6 panel score: **4/10 · soundness 2/4 · weak reject**.

| Reviewer | Rating | Soundness | Presentation | Contribution | Recommendation |
|---|---:|---:|---:|---:|---|
| GPT-5.6 Sol | 5/10 | 3/4 | 3/4 | 2/4 | Weak reject |
| Claude Fable 5 | 5/10 | 3/4 | 3/4 | 2/4 | Borderline |
| Cursor Grok 4.5 | 4/10 | 2/4 | 2/4 | 2/4 | Weak reject |

## 2. wsdm-11

**Status:** Delivered, Round 5

### Title

Query-Term Repetition Repels LLM Selectors from Weak Result Cards: A Controlled Audit

### Subject Areas

- **Primary:** Foundation Models and Agentic Systems → Evaluation and benchmarking of foundation models in search/mining
- **Secondary:** Web Search → Search user behavior and log analysis; Search user interfaces and interaction
- **Secondary:** Foundation Models and Agentic Systems → Retrieval, indexing, and ranking with foundation models
- **Secondary:** Web Search → Query analysis and query processing
- **Secondary:** Privacy, Fairness, Interpretability → Model and algorithm transparency

### Abstract

Implicit feedback is useful only when selection remains aligned with landing-page relevance. We audit one proposed surface intervention for deterministic LLM selectors: replacing generic metadata with repeated query terms in weak result cards. In paired result-card displays, lower-relevance cards contain either one copy or repeated copies of the same query terms, while answer content, landing page, judgment, card length, unique term set, topic, rank schedule, and all higher-relevance cards remain fixed.

Across four confirmatory selectors, repetition reduces false-choice rates by 2.3–6.2 percentage points, with Holm-adjusted p < .001 in every case. A short warning about repeated query words produces no detectable interaction with this effect and therefore does not explain an earlier hardened-prompt contrast. Exact frequency does not explain an earlier synthetic same-query effect that also changed topic and plausibility. We make no human-click claim, and we treat an earlier answer-rewrite audit without independent regrades as conditional evidence only.

### Internal Automated Review — Do Not Paste into CMT

Round 5 panel score: **4/10 · soundness 3/4 · weak reject**.

| Reviewer | Rating | Soundness | Presentation | Contribution | Recommendation |
|---|---:|---:|---:|---:|---|
| GPT-5.6 Sol | 5/10 | 2/4 | 3/4 | 2/4 | Weak reject |
| Claude Fable 5 | 5/10 | 3/4 | 2/4 | 2/4 | Weak reject |
| Cursor Grok 4.5 | 4/10 | 3/4 | 3/4 | 2/4 | Weak reject |

## 3. wsdm-12

**Status:** Delivered, Round 8

### Title

How Small Can You Go? Spectral Bounds for Recommendation Subsets

### Subject Areas

- **Primary:** Web Mining and Content Analysis → Web recommender systems and algorithms
- **Secondary:** Web Mining and Content Analysis → Large-scale graph analysis
- **Secondary:** Web Mining and Content Analysis → Scalable algorithms for mining web data, opinion mining and sentiment analysis
- **Secondary:** Privacy, Fairness, Interpretability → Model and algorithm transparency

### Abstract

How many original-identity interactions are necessary—and how many are actually sufficient—to preserve a collaborative-filtering propagation subspace? We separate these questions. For a source graph's rank-r normalized-biadjacency frame, every unweighted edge subset incurs joint projector loss at least the source leverage mass outside its retained user and item coordinates. Requiring an identifiable cutoff adds a component-multiplicity floor. The combined floor is asymptotically attainable on a block-complete family.

Real recommendation graphs are different. We audit three public graphs at ranks 2, 4, and 8. At rank eight, the necessary floors retain at most 2.1% of edges, while the first observed identifiable witnesses require at least 33%. A connectivity-preserving construction also remains far above the floor. The theorem therefore rules out ultra-small subsets but does not predict the attainable projector-collapse budget on these graphs. We report a lower-to-upper interval rather than call the necessary floor tight. The result applies to unweighted same-identity subsets, not synthetic identities, reweighted sparsifiers, arbitrary finite codes, or ranking utility.

### Internal Automated Review — Do Not Paste into CMT

Round 8 panel score: **4/10 · soundness 2/4 · weak reject**.

| Reviewer | Rating | Soundness | Presentation | Contribution | Recommendation |
|---|---:|---:|---:|---:|---|
| GPT-5.6 Sol | 5/10 | 3/4 | 3/4 | 2/4 | Weak reject |
| Claude Fable 5 | 5/10 | 3/4 | 3/4 | 2/4 | Weak reject |
| Cursor Grok 4.5 | 4/10 | 2/4 | 2/4 | 2/4 | Weak reject |

## CMT Registration Checklist

For each paper:

- Copy the title exactly from the `Title` field.
- Copy both abstract paragraphs into the CMT abstract field.
- Select the listed primary subject area first, then the suggested secondary
areas that CMT permits.
- Enter authors, affiliations, conflicts, and contact information separately.
- Do not paste the internal automated-review section into CMT.
Loading