Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 24 additions & 2 deletions docs/codex-pip-reverse-engineering.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
---
doc_id: codex-pip-reverse-engineering
title: "Codex picture-in-picture reverse engineering"
language: en
source_language: en
implementation_status: current
document_status: current
translation_status: source-only
last_verified: 2026-09-11
owners:
- maka-backend
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
Expand Down Expand Up @@ -235,5 +247,15 @@ It needs no accessibility and no unlocked screen — it drives Electron windows
only — which makes it the one real-machine check that keeps working when the
rest cannot run.

The physics and the anchor scoring are covered exactly, without a desktop, in
`apps/desktop/src/main/__tests__/computer-use-pip-motion.test.ts`.
The physics and the anchor scoring were covered without a desktop in
`apps/desktop/src/main/__tests__/computer-use-pip-motion.test.ts`; #2478
deleted that test, which only ends the coverage it provided — nothing has
taken its place. The module under test is untouched: `pip-motion.ts` is
byte-identical across the #3293 cursor replacement (implemented in #3456),
which rebuilt the agent cursor engine and its glyphs — including the PiP
glyph — without replacing the PiP window's motion constants. The dragging
and settling springs and the throw factors transcribed above are still
defined there and still drive `pip-window.ts`. Cursor provenance is tracked
in computer-use-cursor-provenance.md, and it concerns the agent cursor only;
the PiP window's motion model is a separate surface with no provenance
change to record.
12 changes: 12 additions & 0 deletions docs/computer-use-cursor-provenance.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
---
doc_id: computer-use-cursor-provenance
title: "Computer Use cursor provenance and independent replacement"
language: en
source_language: en
implementation_status: current
document_status: current
translation_status: source-only
last_verified: 2026-09-11
owners:
- maka-backend
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
Expand Down
12 changes: 12 additions & 0 deletions docs/computer-use-evidence-classes.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
---
doc_id: computer-use-evidence-classes
title: "Computer Use Evidence Classes"
language: en
source_language: en
implementation_status: current
document_status: current
translation_status: source-only
last_verified: 2026-09-11
owners:
- maka-backend
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
Expand Down
25 changes: 20 additions & 5 deletions docs/computer-use-foundation-contract.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
---
doc_id: computer-use-foundation-contract
title: "Maka Computer Use Foundation Contract"
language: zh-CN
source_language: zh-CN
implementation_status: current
document_status: current
translation_status: source-only
last_verified: 2026-09-11
owners:
- maka-backend
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
Expand Down Expand Up @@ -76,7 +88,7 @@

7. Approval and privacy
- approval 是 app capability gate,不是 active observation 或 action freshness 证明。
- Maka 采用分级短 lease:metadata read、screenshot read、pointer mutation、keyboard mutation、semantic mutation 分离;目标、action class、observation 或 session generation 变化时重新授权。
- Maka 采用分级短 lease:`metadata_read`、`screenshot_read`、`keyboard_mutation`、`semantic_mutation` 四类(见 `packages/core/src/computer-use.ts`),分离;目标、action class、observation 或 session generation 变化时重新授权。
- approval 至少标明 action class 与目标 app/window;敏感应用、secure/password field 和不支持的目的地 fail closed。
- screenshot、typed text、coordinate、raw AX label/value、window title、secret 和 raw page content 默认不进入持久 session log、telemetry 或 evaluation report。
- 上传截图前验证 model vision capability,并满足对应用户/provider consent policy。
Expand Down Expand Up @@ -111,11 +123,14 @@ Maka 自己的 Electron renderer 也是 Computer Use 的目标。它不能依赖
1. TypeScript 与 Storybook 构建保证语义属性、文案契约和非默认状态 fixture 能随产品 API
一起演进。不要重新引入基于正则的 JSX 源码扫描器;它无法可靠理解组件语义,主干已用
真实 AX 验证取代这类检查。
2. `scripts/ax-tree-audit.mjs` 是 Storybook 与 Electron E2E 共用的测试侧 AX 规则源,
2. `scripts/ax-tree-audit.mjs` 是 Storybook smoke 的测试侧 AX 规则源(当前唯一消费者,不对 Electron 运行)
拒绝无名或同一语义作用域内歧义的可操作 node、多 `main`、无名 dialog,以及缺少
checked/selected/expanded/value 的状态控件。
3. `apps/desktop/e2e/accessibility-coverage.spec.ts` 读取真实 Electron Chromium AX tree,
从运行时设置导航枚举所有设置页,并覆盖模块页、全局弹窗、会话页和 7 个工作栏面板。
3. Electron 侧的广域设置导航 AX 清单(原
`apps/desktop/e2e/accessibility-coverage.spec.ts`)已随 #4803 **退役而非迁
移**:#4803 明确不声称存在等价覆盖,且该缺口是有意接受的。目前没有任何
层对运行中的 Electron 应用做路由清单枚举;Electron E2E 仅保留修订、
WorkHub 与草稿焦点等边界内 journey。
4. `scripts/storybook-visual-smoke.mjs` 对 Storybook 全目录读取 AX tree,执行 `play`
函数到最终态,并验证 modal 焦点、隐藏/惰性 surface 和关键 Computer Use story
inventory;名字含 `narrow` 的故事必须在窄视口运行。独立 WebContentsView 由
Expand Down Expand Up @@ -251,7 +266,7 @@ JSON/YAML 不比"一元素一行 + 缩进"省:实测分别是它的 3.5 倍和

任务是「把窗口挪到左边」。移动窗口只能拖标题栏,拖标题栏只能用坐标动作,而坐标动作
要求目标像素属于目标窗口——Computer Use 驱动的是用户没在看的窗口,后台启动的窗口
必然压在 z-order 底部,于是必然被遮挡。**这个任务没有解**:协议里没有窗口管理动词
必然压在 z-order 底部,于是必然被遮挡。**这个任务当时没有解**:记录时协议里没有窗口管理动词(`window_action` 的 move/resize/minimize 是 #1952 之后才加入严格动作集的)
而「移动窗口」也不是任何控件的 AX 动作。

把拒绝语句写清楚之后,模型确实读懂了「这条路不通」,于是去试别的路——而别的路也不
Expand Down
12 changes: 12 additions & 0 deletions docs/computer-use-host-events-contract.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
---
doc_id: computer-use-host-events-contract
title: "Computer Use Host Events Contract"
language: en
source_language: en
implementation_status: current
document_status: current
translation_status: source-only
last_verified: 2026-09-11
owners:
- maka-backend
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
Expand Down
12 changes: 12 additions & 0 deletions docs/computer-use-model-loop-foundation.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
---
doc_id: computer-use-model-loop-foundation
title: "Computer Use Model-Loop Foundation"
language: en
source_language: en
implementation_status: current
document_status: current
translation_status: source-only
last_verified: 2026-09-11
owners:
- maka-backend
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
Expand Down
14 changes: 13 additions & 1 deletion docs/computer-use-provenance.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
---
doc_id: computer-use-provenance
title: "Computer Use provenance"
language: en
source_language: en
implementation_status: current
document_status: current
translation_status: source-only
last_verified: 2026-09-11
owners:
- maka-backend
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
Expand Down Expand Up @@ -125,7 +137,7 @@ artifact, retained facts, and Maka-authored divergences are recorded in

| Binary-recovered or observed fact | Informed area |
|---|---|
| Exact cursor geometry, center hotspot, motion configuration, close-enough thresholds, path measurement, and core scoring weights | `apps/desktop/src/renderer/computer-use-overlay/engine/cursor-engine.ts` |
| Historical (removed by #3293): cursor geometry, center hotspot, motion configuration, close-enough thresholds, path measurement, and core scoring weights | Formerly `apps/desktop/src/renderer/computer-use-overlay/engine/cursor-engine.ts`; the #3293 replacement removed every transcribed value (the hotspot is now the glyph tip, not the center) |
| Overlay level policy — an occluded target raises the cursor rather than hiding it | same file, and `apps/desktop/src/main/computer-use/cursor-overlay-window.ts` |
| The observation text shape | `packages/runtime/src/computer-use-tools.ts`, corroborated by the archived capture in §2 |
| OOP WebContent targeting, retained-element unique refetch, and renderer-generation fencing | Reimplemented in the pinned `maka-cu` source; this repository records the exact source commit and binary digest in `apps/desktop/bundled-tools.json` |
Expand Down
14 changes: 13 additions & 1 deletion docs/computer-use-provider-evidence.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
---
doc_id: computer-use-provider-evidence
title: "Computer Use Provider Evidence"
language: en
source_language: en
implementation_status: current
document_status: current
translation_status: source-only
last_verified: 2026-09-11
owners:
- maka-backend
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
Expand Down Expand Up @@ -38,7 +50,7 @@ provider transports, or execution backends.

## Report Contract

Reports separate three evidence classes:
Reports separate four evidence classes:

- `real-runtime`: a live provider model used the production Maka runtime;
- `fault-injection`: a live provider and Runtime exercised a named injected
Expand Down
31 changes: 23 additions & 8 deletions docs/computer-use-ui-coverage.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
---
doc_id: computer-use-ui-coverage
title: "Computer Use Semantic UI Coverage"
language: en
source_language: en
implementation_status: current
document_status: current
translation_status: source-only
last_verified: 2026-09-11
owners:
- maka-backend
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
Expand Down Expand Up @@ -35,7 +47,7 @@ are not measured here; they require separate accessibility testing.

| Surface | Runtime states | Semantic actions and effects |
|---|---|---|
| Settings | Every current top-level navigation entry is enumerated from the running UI; provider list/detail/catalog/add, subagent editor, memory populated, permission diagnostics, import empty/ready, usage variants, Daily Review selector and narrow states are in Storybook | Navigation exposes `aria-current`; focused nested stories exercise dialogs, disclosures, selectors and editors |
| Settings | Top-level navigation entries are covered through the Storybook settings-pages stories; provider list/detail/catalog/add, subagent editor, memory populated, permission diagnostics, import empty/ready, usage variants, Daily Review selector and narrow states are in Storybook (the Electron-side dynamic enumeration was retired by #4803) | Navigation exposes `aria-current`; focused nested stories exercise dialogs, disclosures, selectors and editors |
| Extensions | Skills empty/installed/bundled/update/disabled/narrow/inspector; MCP setup/marketplace/configured/inspector/editor/failure/narrow | Page selection, inspector/editor opening and actionable-node identity |
| Scheduled work | Empty/configured/long/narrow/task inspector; Daily Review loading/error/refreshing/report | Task selection, dialog focus, selector state and report actions |
| Conversation shell | New task, settled conversation, streaming, permission wait, native conversation, modes, context and inline completion | Composer submit effect, unique per-task and per-turn actions, current regions and workbar selection |
Expand All @@ -46,9 +58,11 @@ are not measured here; they require separate accessibility testing.

The Storybook catalog is exhaustive for its source-defined entries and the
smoke runner carries a required Computer Use story manifest for critical
runtime boundaries. The Electron accessibility test dynamically enumerates
settings navigation, then covers modules, global overlays, conversations and
all workbar entry points.
runtime boundaries. The Electron-side broad route inventory (a spec that
dynamically enumerated settings navigation, then modules, global overlays,
conversations and all workbar entry points) was retired by #4803 rather than
relocated; the surviving Electron journeys cover revision, WorkHub and
draft-focus boundaries only.

## AX completion gates

Expand All @@ -65,10 +79,11 @@ Every measured final state fails on:
- focus inside an inert or `aria-hidden` surface;
- a visible modal dialog that does not own focus.

Critical stories and Electron journeys additionally assert action-specific
effects. Overlay journeys wait for the specifically named prior dialog to
close before opening and auditing the next one. Transport success or a generic
dialog match alone is not accepted.
Critical stories additionally assert action-specific effects. (An Electron
variant that waited for the specifically named prior dialog to close before
auditing the next one was part of the retired route-inventory spec; the
surviving journeys do not carry it.) Transport success or a generic dialog
match alone is not accepted.

Repeated message and answer actions use a bounded excerpt of visible text plus
a stable human-readable timestamp. Opaque storage IDs remain machine data and
Expand Down