feat: click-xy / dblclick-xy — screen-coordinate clicks for non-AX UIs - #1
Open
maxesisnclaw wants to merge 1 commit into
Open
feat: click-xy / dblclick-xy — screen-coordinate clicks for non-AX UIs#1maxesisnclaw wants to merge 1 commit into
maxesisnclaw wants to merge 1 commit into
Conversation
Adds two new subcommands that bypass AX selectors and click directly at
(X, Y) screen coordinates via CGEventPostToPid. Targets self-drawn /
non-AX UIs (canvases, custom-rendered controls, proprietary financial
clients) that 'click <selector>' can't reach.
Window resolution:
1. --window <CGWindowID> if provided.
2. The visible AXWindow whose frame contains (X, Y).
3. First visible AXWindow of the app.
Strategies:
- auto (default) / cg-pid: SWaveAX recipe via mouse_click_bg /
mouse_dblclick_bg — background-safe, no focus steal, no cursor jump.
- cg / hid: global CGEventPost(HIDEventTap), equivalent to
'axcli mouse click X Y'. Pair with --activate for foreground.
Verified on 东方财富 Mac client (com.emmac.mac) — clicks on self-drawn
trade-panel buttons (无 AX 元素) succeed while the app is fully background
behind Brave Browser. Brave stays frontmost, no cursor movement, sell
tab toggles via cg-pid.
Implementation drafted with AI assistance, manually reviewed and tested.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation. Some macOS apps draw UI controls outside the Accessibility tree — proprietary financial / trading clients, canvas-based widgets, custom-rendered toolbars.
click <selector>can't reach them. The existingmouse click X Yroute works but goes throughCGEventPost(HIDEventTap), which steals focus and moves the system cursor.click-xy X Yexposes the existing SWaveAX background-click recipe (already implemented inmouse_click_bg, but previously only reachable via AX selectors) at raw coordinates.Surface.
axcli click-xy <X> <Y> [--strategy auto|cg|hid|cg-pid] [--window <ID>] [--activate]. Default strategyauto → cg-pid, identical toclick <selector>'s default.--windowlets callers override window selection; without it, the tool picks the visibleAXWindowwhose frame contains (X, Y), falling back to the first visible window.dblclick-xymirrors the same API.Validated on a real-world AppKit app with custom-drawn buttons that have zero AX exposure. Click delivered to the target window while the app was occluded behind another; foreground app and cursor position unchanged.
AI assistance. Implementation drafted with Claude (Opus 4.7) assistance, reviewed and tested by hand. No
cargo clippyregressions on new code.