Skip to content

fix(click): resolve window id when web content runs in a helper process - #4

Open
caipira113 wants to merge 1 commit into
andelf:mainfrom
caipiralink:fix/window-id-helper-process
Open

fix(click): resolve window id when web content runs in a helper process#4
caipira113 wants to merge 1 commit into
andelf:mainfrom
caipiralink:fix/window-id-helper-process

Conversation

@caipira113

@caipira113 caipira113 commented Aug 5, 2026

Copy link
Copy Markdown

Problem

click, dblclick, and scroll --strategy cg-pid silently do nothing on Microsoft Teams. The event is posted, coordinates are correct, and no error is reported, but the app does not react. A real cursor click at the very same point works.

The cause is a mismatch between the process the event is posted to and the window it is tagged with.

Teams renders web content in a separate helper process:

wid=11607  pid=84794 (Microsoft Teams)          onscreen=true    <- the window on screen
wid=11610  pid=84807 (Microsoft Teams WebView)  onscreen=false   <- what AX reports

_AXUIElementGetWindow is not wrong here: an element inside the page really does belong to the WebView process's window. But mouse_click_bg posts to the application pid while tagging fields 91/92 with that helper window id, so hit-testing finds no match and the event is dropped.

Change

Added resolve_event_window_id, used by all three cg-pid call sites. It keeps the id from _AXUIElementGetWindow when that window is owned by the target pid, which is the common case, and otherwise matches the owning AX window's frame against the on-screen window list to find the right one.

Two supporting helpers in accessibility.rs:

  • window_belongs_to_pid — ownership check against CGWindowListCopyWindowInfo.
  • window_id_for_frame — frame match, restricted to layer 0 and skipping fully transparent windows. Teams keeps a zero-alpha overlay over its title bar that would otherwise be a candidate.

AX frames and kCGWindowBounds share a top-left origin, so they compare directly; a 2pt tolerance absorbs rounding.

Verification

Tested on macOS 27.0 (build 26A5388g, arm64), each command run with Finder frontmost.

Teams (helper-process app) — was broken, now works

$ axcli --pid 84794 click 'AXRow:has-text("Nayul Kim") >> nth=1'
debug: window 11610 is not owned by pid 84794; using 11607 matched by frame
cg-pid click pid=84794 wid=11607 screen=(2006,-494) local=(208,931)
→ window title becomes "채팅 | Nayul Kim | Microsoft Teams"

Scroll is fixed by the same path, confirmed visually: the message list moved from the latest message to content from several months earlier, and focus stayed on Finder throughout.

No regression on apps that were already fine

App Shape Reported wid Fallback used
Calculator native AppKit 12733 no
Windows App single-process, fullscreen 11642 no
Microsoft Teams web content in helper process 11610 (wrong owner) yes → 11607

Calculator still accumulates digits from background clicks, so the AppKit path is untouched.

cargo test passes (54 tests) and cargo clippy reports no new warnings.

Note on focus

Background clicks on Teams still bring it forward, but that is the app's own behaviour rather than an artifact of the event path: --strategy ax, which posts no mouse events at all, raises it identically, while clicking a non-interactive element (a heading) leaves focus alone. Scrolling never raises it.

@caipira113
caipira113 force-pushed the fix/window-id-helper-process branch from ee41fca to 1415b49 Compare August 5, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant