Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 40 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,8 +219,9 @@ retain window-scoped pixels and coordinates for Flow's remote visual compiler.
**Status: implemented, with display-free unit coverage on every
supported operating system.** The production release gate also requires live
window capture, input injection, movement, resize, video verification, and no
skipped tests on interactive macOS and Windows runners. A customer RDP or
Citrix deployment still requires task- and environment-specific qualification.
skipped tests on interactive macOS and Windows runners. Linux X11 has a separate
opt-in live window check. A customer RDP or Citrix deployment still requires
task- and environment-specific qualification.

By default the recorder captures the full screen. Window-scoped mode records
ONE window in that window's own pixel space. This is the mode built for
Expand All @@ -240,18 +241,20 @@ with Recorder(
input("Perform the task, then press Enter...")
```

`owner` matches the application (macOS: window owner name; Windows: process
executable name) and `title` optionally disambiguates among its windows; both
are case-insensitive substrings, mirroring how `openadapt-flow`'s
remote-display backend identifies the same window at replay time. The
selectors can also be set via config/environment
`owner` matches the application. macOS uses the window owner name. Windows and
Linux use the process executable name. `title` optionally disambiguates among
the application's windows. Both selectors are case-insensitive substrings,
matching how `openadapt-flow` identifies the same window at replay time. The
selectors can also be set through config or environment
(`RECORD_WINDOW_OWNER` / `RECORD_WINDOW_TITLE`).

In this mode:

- **Frames are the target window's pixels.** macOS captures the window's own
buffer (`CGWindowListCreateImage`, the identical call flow's replay uses);
Windows grabs the window's screen region, so keep the window unoccluded.
Linux X11 reads an XComposite named-window pixmap. It doesn't use a root
screenshot, so another window cannot replace the target pixels. Windows
grabs the window's screen region, so keep the window unoccluded.
- **Input coordinates are translated at capture time** into the captured
frame's pixel space (`pixel = (global_point - window_origin) * scale`, the
exact inverse of the replay mapping). Input outside the window records
Expand All @@ -262,6 +265,13 @@ In this mode:
(`CaptureSession.window_capture`), and the window is re-resolved every
frame with bounds changes recorded as window events, a bounds timeline
converters can use to be exact even when the window moves.
- **Each frame has one ordered geometry identity.** Current window captures
store the frame and its window event under the same source ordinal. An action
stores a later ordinal and names the exact earlier pair it used. The pair
binds the process start identity, display topology, bounds, scale, fixed
viewport, and geometry generation. Recorder shutdown retains one final frame
after input has stopped, so a consumer can select an exact after-action frame
by ordinal instead of by nearest timestamp.
- **Window movement and resize are supported.** The first frame fixes the
encoded video size. Later source frames scale to fit and letterbox into that
viewport. Input uses the exact current bounds and content rectangle. No frame
Expand All @@ -273,10 +283,24 @@ In this mode:
lost window, capture failure, or unexpected output-frame size fails the
session instead of producing complete-looking media with an evidence gap.

Linux window mode requires an X11 session with EWMH and XComposite. Capture
won't start window mode in a native Wayland or XWayland-only session. A future
Wayland producer must bind the portal-selected window, its pixel stream, and
event-time coordinates before it can replace this refusal.

Note for converters: window-mode coordinates are already in captured-frame
pixels (`coordinate_space == "window_pixels"`); do not rescale them by
`pixel_ratio`.

After every producer and writer has stopped, `Recorder` verifies the database
and writes `capture-artifact-manifest.json` plus `capture-terminal.json`. The
terminal binds the complete artifact inventory, event counts, final source
ordinal, capture session identity, and completion time. A current native
consumer should use `CaptureSession.load_verified()`. It checks the seal, copies
the exact artifacts into a private snapshot, and opens the copied database in
SQLite immutable read-only mode. A current window capture without this seal is
incomplete and must not be compiled as native geometry evidence.

## Multiple monitors

Full-screen mode records the complete virtual desktop reported by MSS, not
Expand Down Expand Up @@ -415,6 +439,14 @@ permissions:
uv run pytest -m slow
```

The Linux X11 window check also requires an explicit target:

```bash
OPENADAPT_CAPTURE_LINUX_WINDOW_QUALIFICATION=1 \
OPENADAPT_WINDOW_SMOKE_OWNER=citrix \
uv run pytest tests/test_window_capture_linux.py -m slow
```

## License

[MIT](LICENSE)
51 changes: 45 additions & 6 deletions docs/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,22 @@ One recording has these stages:
2. Resolve the initial native-window or virtual-desktop coordinate scope.
3. Create the per-capture SQLite database and media staging path.
4. Observe native input and screen frames on separate workers.
5. Put all observed events onto one timestamped processing queue.
6. Associate actionable input with the preceding screen observation and
optional structural observation.
7. Stream RGB frames to a separately provisioned FFmpeg process.
8. Close, verify, and atomically promote the MP4. Retain an incomplete partial
5. Reserve each observation in one ordered source journal before any optional
structural lookup. A failed reservation fails the session; later events
cannot pass it.
6. For a native window, enqueue each frame and its window geometry as one
source-ordinal pair. Publish that geometry to input observers only after the
pair enters the journal.
7. Bind each actionable input to the last published frame pair and optional
structural observation. Retain one ordinal-later frame after input stops.
8. Stream RGB frames to a separately provisioned FFmpeg process.
9. Close, verify, and atomically promote the MP4. Retain an incomplete partial
file on an encoder failure and never report it as complete media.
9. Post-process raw input into the public action view.
10. Post-process raw input into the public action view. A merged action keeps
its terminal primitive's source binding and refuses mixed geometry epochs.
11. Reconcile committed rows with producer counts, verify the v2 frame/action
relations, inventory every immutable artifact, and write the completion
seal.

A worker failure stops the session and propagates through the recording
boundary. A frame whose size violates the fixed stream contract is an error. It
Expand Down Expand Up @@ -91,6 +100,32 @@ window changed size.
Input outside the selected window remains out of range. Capture does not clamp
it into a valid-looking target coordinate.

Each current window frame carries a process-bound window identity, display
topology digest, and geometry epoch digest. The window and screenshot rows use
the same source ordinal. An action uses a later ordinal and names the exact pair
that supplied its coordinates. Capture refuses process replacement, topology
drift, off-screen state, mixed generations, or a missing pair.

## Completion and consumer boundary

A recorder session becomes complete only after all producers and writers have
stopped and the database has passed its integrity and relationship checks.
Capture then writes two create-only files:

- `capture-artifact-manifest.json` inventories every immutable regular file by
relative path, size, and SHA-256 digest.
- `capture-terminal.json` binds that manifest, the source session identity,
event counts, last source ordinal, and completion interval.

Both files use canonical JSON and domain-separated digests. Mutable local
control state is not part of the artifact inventory.

`CaptureSession.load_verified()` checks the complete inventory before it opens
the database. It copies the verified files to a private temporary directory and
opens the copied database with SQLite `mode=ro&immutable=1`. It never migrates
or writes the source capture. A current v2 window session without the terminal
and manifest is incomplete evidence.

## Native input

Capture records these primitive event classes:
Expand All @@ -104,6 +139,10 @@ Platform observers have one ordered callback contract. They identify injected
events when the operating system provides that information. Capture can exclude
its own injected qualification events from a normal session. It refuses an
incomplete observer startup instead of reporting partial coverage as complete.
On macOS, an active pass-through session tap holds downstream event delivery
only while Capture commits a clean frame. This requires both Input Monitoring
and Accessibility permission. The interactive macOS qualification proves that
an annotated downstream event cannot pass that cut before the frame commit.

The post-processing layer merges primitive events into higher-level actions.
The compiler remains responsible for refusing action forms that its selected
Expand Down
Loading