From 0425726b2b3eebf6b1b3f5343410acbbf9a8e418 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Thu, 20 Aug 2026 19:42:49 +0200 Subject: [PATCH] feat(ios): split raw and regular snapshot projections behind one capture hint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The private-AX backend interpreted `--raw` as the regular projection: it folded the viewport and scroll clips and dropped sub-pixel decorations before returning, so a raw capture that recovered onto it answered with viewport-pruned nodes labeled raw (#1797 D4). Nothing related the two copies of that decision. Presentation now exposes the two projections it always implied. `presentRegular` folds visibility, eligibility, scope and scroll hints; `presentRaw` is the acquired tree, normalized, narrowed only by a scope or depth the request asked for. Acquisition reads one derived `CaptureHint` instead of the request itself, so what a capture may skip is stated once, beside the reason skipping it keeps the projection complete: scope and its relative depth never narrow, raw depth does (raw depth is traversal depth), and the raw projection never carries `interactiveOnly` — `--raw -i` is the acquired tree. Two structural rules replace the hand-synchronized ones. The raw plan is derived from `SnapshotBackendKind.supportsRawProjection`, so the query sweep — an interactive element query with no hierarchy to return — cannot be planned for a raw request. And presentation compares the requested projection with the hint the acquisition was captured under, dropping that tier with a structured `IOS_SNAPSHOT_PROJECTION_MISMATCH` failure rather than presenting it under the requested label. Declared residue: a regular `--depth` request still cuts the traversal at that depth while regular presentation emits collapsed depth, so a node that would present within the limit can be dropped. The cut is what keeps `--depth 1` probes cheap; making it complete is the open visible-depth frontier obligation. Refs #1797 (migration step 3, raw-projection delta). --- .github/workflows/ios.yml | 6 + CHANGELOG.md | 1 + CONTEXT.md | 5 + .../RunnerTests+AXSnapshotFallback.swift | 43 ++-- .../RunnerTests+FlatSnapshotFiltering.swift | 22 +- .../RunnerTests+PrivateAXPresentation.swift | 226 ++++++++++++++---- .../RunnerTests+Snapshot.swift | 43 ++-- .../RunnerTests+SnapshotCapturePlan.swift | 80 ++++++- .../RunnerTests+SnapshotPresentation.swift | 136 ++++++++--- ...unnerTests+SnapshotPresentationTests.swift | 116 +++++++-- .../adr/0004-ios-snapshot-backend-strategy.md | 25 ++ website/docs/docs/commands.md | 6 +- 12 files changed, 564 insertions(+), 145 deletions(-) diff --git a/.github/workflows/ios.yml b/.github/workflows/ios.yml index 02a4c2276..d5ef9b542 100644 --- a/.github/workflows/ios.yml +++ b/.github/workflows/ios.yml @@ -151,12 +151,18 @@ jobs: -only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationPreservesCurrentWireShape \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationOwnsBackendNeutralEligibility \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationOwnsScopeAndRelativeDepth \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPresentationRefusesAnAcquisitionCapturedForTheOtherProjection \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testCaptureHintIsTheOnlyAcquisitionViewOfARequest \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotScopePolicyMatchesGoldenParityTable \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testFlatSnapshotProjectionMatchesElementReverseScrollCapture \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXDepthLimitedRequiresEveryFrontierResolved \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testDeepExtensionCountsMissedFrontiers \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPreferredPrivateAXBackendPlansAsPenalized \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRawDiagnosticPlanCarriesOnlyBackendsThatCanServeRaw \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testProjectionMismatchFailureIsStructuredAndNotAnAxFailure \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXRegularPresentationProjectsToViewportAndKeepsScrollHint \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXRawProjectionKeepsEveryAcquiredNode \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXRawProjectionAppliesRequestedTraversalDepth \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXPresentationKeepsOffscreenSubtreeExcludedWhenChildFramesAreClamped \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXGeometrylessSemanticsAreNeverActionableOrScrollContexts \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXAcquisitionDoesNotInterpretScope \ diff --git a/CHANGELOG.md b/CHANGELOG.md index aaa9d4e98..34212bc4b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ - `--udid` with `--platform android` (and `--serial` with an Apple platform) now fails as the flag mistake it is — `INVALID_ARGS` naming the right flag — instead of reaching device resolution and answering `No Apple device with UDID emulator-5580` for an explicitly Android request. `--udid` addresses Apple devices, `--serial` addresses Android and HarmonyOS; matching pairs and requests that name no platform are unchanged. - Breaking (`--session-lock strip`): a device selector that names a different device than the bound session is no longer silently discarded. `strip` exists to drop redundant platform/scope selectors; when it also dropped `--udid`/`--serial`/`--device`, the command kept running against the *bound* device instead of the one the caller named — a wrong-device action that looks like a success. Such a request now fails with `INVALID_ARGS` under both `reject` and `strip`, and the error carries the two identities structurally (`requestedDevice`, `boundDevice`) plus a hint offering the two real recoveries: close the bound session if the requested device is intended, or remove the selector if the bound device is. The hint no longer suggests `--session-lock strip` for an identity conflict, since following that advice is what produced the wrong-device run. Scope-only stripping (`--platform`, `--target`, `--ios-simulator-device-set`, `--android-device-allowlist`) is unchanged. - iOS regular snapshots now apply one backend-neutral eligibility rule after every capture backend: a node survives when its accessibility type is interactive or it carries a non-empty label, identifier, or value. This removes the tree backend's extra "hittable non-Other" membership path and drops unlabeled decorative nodes consistently; labeled images, identifier-only nodes, and value-only nodes still survive. Raw snapshot membership is unchanged. +- iOS `snapshot --raw` is now the acquired accessibility tree on every backend that can serve it (#1797). A raw request that recovered onto the private-AX backend — the route an app whose XCTest tree capture fails takes — returned the *regular* projection's viewport-pruned nodes labeled raw: everything scrolled out of the viewport, and every sub-pixel decoration, was missing from the one view whose purpose is showing what the pruned view hid. Raw now keeps every node the backend serialized, at traversal depth, and `--depth` still narrows it (for raw, presented depth *is* traversal depth). Two structural rules replace the hand-synchronized ones: the raw capture plan is derived from each backend's declared ability to serve raw, so the interactive query sweep — which has no hierarchy to return — cannot be planned for a raw request; and presentation refuses an acquisition captured for the other projection instead of relabeling it, dropping that tier with a structured failure. Breaking in the same direction: `snapshot --raw -i` now returns the acquired tree instead of an interactive-filtered one — `-i` narrows the regular projection, and the pair used to produce a third membership rule that differed per backend. Regular and `-i` output is unchanged. Backends now read one derived capture hint rather than the request itself, so what a capture is allowed to skip is stated once, next to the proof that skipping it keeps the projection complete. - Android `snapshot --raw` is now the acquired accessibility tree (#1832 C3): the three regular-projection pruners — nodes Android marks invisible, stale application windows, and covered same-window surfaces — no longer run at parse time, so `--raw` keeps everything the helper serialized (normalization only) and pruned content is recoverable for diagnosis. Regular and `-i` output is unchanged (property-checked identical to the previous implementation over 12,000 random tree × projection pairs, hidden-content hints included). Also: Android blocking-dialog recovery now reads the same daemon presentation an agent's `snapshot` sees instead of a hand-rolled subset, and acts on its occlusion result — a stale "App isn't responding" surface left under the foreground one no longer triggers recovery, and a covered "Close app" is never tapped ahead of the visible one; the Android freshness route signature no longer keys on `role`/`selected`, fields the Android backend never carries; and the Android helper's declared fidelity residues (no `checked`/`checkable`/`selected`/`long-clickable`, 5000-node cap before scoping, API-level cache-reset divergence) are recorded in `CONTEXT.md`. - `agent-device mcp` now carries its own usage guidance, so MCP-only clients (Codex CLI, Cursor, custom agents) no longer depend on a separately installed skill (#1833). The handshake `instructions` — returned by both `server/discover` and, newly, the legacy `initialize` — is a compact (< 2 KB, the Claude Code truncation limit) workflow card: start with `open {app, foreground: true}` instead of probing, act with `settle: true` and continue from the diff, verify with `wait`/`is`/`get`/`find`, copy `@refs` byte-for-byte, recover from sparse/AX-unavailable, follow error hints, `close`. A new MCP-only `help` tool serves the full guides on demand: no `topic` returns the CLI's decision card; `topic` returns `agent-device help ` verbatim (workflow, gestures, scripting, tv, macos, web, remote, debugging, …, or any tool name for its complete flag reference), prefixed with the one-line CLI→tool-property mapping. `help` is router-owned rather than a command descriptor, so it appears in `tools/list` only — not in the CLI, Node client, or `batch` — and its description tells the model it is not a startup step. Legacy `initialize` gains the optional `instructions` field; no other legacy field changes. - Android `snapshot --scope` (and every selector command's `--scope`, e.g. `press "Save" --scope Panel`) now resolves scope exactly once, inside the Android projection, under the shared scope specification: the scope root is the first node **in document order** whose label, value, or identifier contains the scope text (case-insensitive) **and whose subtree still has content in the projection you asked for**, the result is that subtree re-rooted at depth 0, and no match returns an empty snapshot (#1832). That second clause is what makes `snapshot -i --scope panel` return the button inside a structural container `-i` drops, and stops a decorative heading that happens to match from emptying the snapshot. Before, Android ran two passes with contradictory rules — a breadth-first platform match that fell back to the full tree on a miss, then the daemon's document-order pass — so a shallower later container could win over an earlier match, and an interaction capture whose scope reached only the daemon layer was silently unscoped. `--depth` under `--scope` counts from the scope root, filtering the depths the response prints (a node shown at depth 0 is never hidden by `--depth 0`), and ancestor context above the scope root (a clickable row, a list) still shapes `-i` membership inside it. The rule is pinned by `contracts/fixtures/snapshot-scope-policy.json`, the same golden table the iOS runner will consume (#1797). Also new: `androidSnapshot.occlusionScanUnavailable: true` discloses an API 23 capture, where the helper cannot report `drawing-order` and covered same-window surfaces are therefore not pruned. diff --git a/CONTEXT.md b/CONTEXT.md index df378b2b8..eee240082 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -190,6 +190,11 @@ One backend attempt's raw accessibility nodes and attempt-level capture facts. **Presentation options**: The policy input controlling how one snapshot acquisition becomes a public projection. +**Capture hint**: +The acquisition-facing view of a snapshot request, derived once from presentation options. It names +the projection a backend must serve and may narrow acquisition only where that backend can prove the +narrowing complete. + **Snapshot eligibility**: Membership in a presented snapshot projection, independent of whether a node is currently hittable. diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift index 958cc93de..357a378e5 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift @@ -105,14 +105,14 @@ extension RunnerTests { func privateAXSnapshotAcquisition( app: XCUIApplication, - options: PresentationOptions, + hint: CaptureHint, deadline: Date = .distantFuture ) -> SnapshotAcquisition? { #if os(iOS) && targetEnvironment(simulator) - let requestedDepth = options.depth ?? 64 + let requestedDepth = hint.depth ?? 64 // An explicit --depth request is honored as asked: no accepted-depth // memory, no frontier extension past it. - let exactDepthRequested = options.depth != nil + let exactDepthRequested = hint.depth != nil let rememberedDepth = exactDepthRequested ? nil @@ -135,12 +135,16 @@ extension RunnerTests { NSLog("AGENT_DEVICE_RUNNER_PRIVATE_AX_SNAPSHOT_BUDGET_EXHAUSTED depth=%ld", depth) break } + // Declared residue (#1797): the bridge caps the tree at 5000 nodes while serializing, + // BEFORE either projection exists, so a raw capture of a huge screen is bounded rather + // than failing the way the tree backend's own raw cap does. The cap is disclosed as + // `truncated`, and it applied to the acquired tree before this projection split too. response = RunnerAXSnapshotBridge.snapshotTree( for: app, maxDepth: depth, maxNodes: Self.privateAXSnapshotMaxNodes, deepExtensionCallLimit: exactDepthRequested ? 0 : Self.privateAXDeepExtensionCallLimit, - customActionLimit: options.customActions ? Self.privateAXCustomActionLimit : 0, + customActionLimit: hint.customActions ? Self.privateAXCustomActionLimit : 0, deadline: deadline ) if response["ok"] as? Bool == true { @@ -175,9 +179,9 @@ extension RunnerTests { let rootFrame = privateAXRect(root["frame"]) let viewport = privateAXSnapshotViewport(app: app, rootFrame: rootFrame) - let nodes = privateAXPresentation( + let nodes = privateAXAcquisition( rawRoot: root, - options: options, + hint: hint, viewport: viewport ) if nodes.count <= 1 { @@ -203,6 +207,7 @@ extension RunnerTests { deepExtension?[RunnerAXSnapshotDeepExtensionNodesAddedKey] as? Int ?? 0 ) return SnapshotAcquisition( + hint: hint, nodes: nodes, truncated: (response["truncated"] as? Bool) == true, effectiveDepth: depthLimited ? effectiveDepth : nil, @@ -635,9 +640,10 @@ extension RunnerTests { ], ], ] - let nodes = privateAXPresentation( + let nodes = privateAXAcquisition( rawRoot: tree, - options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false), + hint: CaptureHint( + projection: .regular, depth: nil, interactiveOnly: false, customActions: false), viewport: CGRect(x: 0, y: 0, width: 390, height: 844) ) @@ -660,13 +666,17 @@ extension RunnerTests { ["type": 9, "label": "unrelated sibling", "children": []], ], ] - let nodes = privateAXPresentation( + // Scope never reaches acquisition: the hint derived for a scoped request carries no scope, + // and the backend has no way to interpret one. + let nodes = privateAXAcquisition( rawRoot: tree, - options: PresentationOptions( - interactiveOnly: false, - depth: nil, - scope: "homeScreen", - raw: false + hint: SnapshotPresentation.captureHint( + for: PresentationOptions( + interactiveOnly: false, + depth: nil, + scope: "homeScreen", + raw: false + ) ), viewport: .infinite ) @@ -742,9 +752,10 @@ extension RunnerTests { ] ], ] - let nodes = privateAXPresentation( + let nodes = privateAXAcquisition( rawRoot: tree, - options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false), + hint: CaptureHint( + projection: .regular, depth: nil, interactiveOnly: true, customActions: false), viewport: CGRect(x: 0, y: 0, width: 390, height: 844) ) diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift index f53966532..9d0eff5b9 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift @@ -108,16 +108,18 @@ extension RunnerTests { ) } + /// Regular-projection acquisition gate. The raw projection never reaches it: raw acquisition is + /// the acquired tree, so it has no visibility filter to consult (#1797 D4). func flatSnapshotFilterDecision( _ node: FlatSnapshotFilterNode, - options: PresentationOptions, + hint: CaptureHint, visibilityPolicy: FlatSnapshotVisibilityPolicy ) -> FlatSnapshotFilterDecision { let include: Bool if node.isRoot { include = true } else if !node.visible - && (options.interactiveOnly || visibilityPolicy == .viewportProjected) + && (hint.interactiveOnly || visibilityPolicy == .viewportProjected) { include = false } else { @@ -285,6 +287,10 @@ extension RunnerTests { } func testFlatSnapshotFilterDecisionMatrixCoversOptions() { + func hint(interactiveOnly: Bool) -> CaptureHint { + CaptureHint( + projection: .regular, depth: nil, interactiveOnly: interactiveOnly, customActions: false) + } let visibleContent = FlatSnapshotFilterNode( isRoot: false, visible: true @@ -305,42 +311,42 @@ extension RunnerTests { XCTAssertTrue( flatSnapshotFilterDecision( visibleContent, - options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false), + hint: hint(interactiveOnly: false), visibilityPolicy: .interactiveOnly ).include ) XCTAssertFalse( flatSnapshotFilterDecision( hiddenInteractive, - options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false), + hint: hint(interactiveOnly: true), visibilityPolicy: .interactiveOnly ).include ) XCTAssertFalse( flatSnapshotFilterDecision( hiddenInteractive, - options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false), + hint: hint(interactiveOnly: false), visibilityPolicy: .viewportProjected ).include ) XCTAssertTrue( flatSnapshotFilterDecision( hiddenInteractive, - options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false), + hint: hint(interactiveOnly: false), visibilityPolicy: .interactiveOnly ).include ) XCTAssertTrue( flatSnapshotFilterDecision( hiddenRoot, - options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false), + hint: hint(interactiveOnly: false), visibilityPolicy: .viewportProjected ).include ) XCTAssertTrue( flatSnapshotFilterDecision( decorative, - options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false), + hint: hint(interactiveOnly: false), visibilityPolicy: .interactiveOnly ).include ) diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift index 8182aa13d..bf4e7591e 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift @@ -1,41 +1,85 @@ import XCTest +/// Reported facts for one private-AX element, read once so both projections describe a node the +/// same way. Only membership and clip-derived state differ between them. +struct PrivateAXFields { + let rect: CGRect + let label: String + let identifier: String + let value: String + let rawType: Int + let elementType: XCUIElement.ElementType? + let enabled: Bool + let focused: Bool? + let selected: Bool? + let actions: [String]? + let children: [[String: Any]] + + var hasSemanticContent: Bool { [label, identifier, value].contains { !$0.isEmpty } } + var hasFrame: Bool { !rect.isNull && !rect.isEmpty } +} + extension RunnerTests { private static let privateAXProjectionTolerance: CGFloat = 1 - func privateAXPresentation(rawRoot: [String: Any], options: PresentationOptions, viewport: CGRect) + /// Acquisition entry point for the private AX backend. + /// + /// The regular projection folds the viewport and scroll clips and records scroll hints; the raw + /// projection is the acquired tree, so it drops nothing and keeps traversal depth (#1797 D4 — a + /// recovered `snapshot --raw` used to return viewport-pruned nodes labeled raw). + func privateAXAcquisition(rawRoot: [String: Any], hint: CaptureHint, viewport: CGRect) -> [RawAXNode] { var nodes: [RawAXNode] = [] - var hints: [Int: (above: Bool, below: Bool)] = [:] - appendPrivateAXNode(rawRoot, to: &nodes, hints: &hints, options: options, viewport: viewport, - depth: 0, parentIndex: nil, scrollContext: nil, - projectionCursor: .root) - return applyHiddenContentHints(hints, to: nodes) + switch hint.projection { + case .raw: + appendPrivateAXRawNode(rawRoot, to: &nodes, hint: hint, viewport: viewport, + depth: 0, parentIndex: nil) + return nodes + case .regular: + var hints: [Int: (above: Bool, below: Bool)] = [:] + appendPrivateAXNode(rawRoot, to: &nodes, hints: &hints, hint: hint, viewport: viewport, + depth: 0, parentIndex: nil, scrollContext: nil, + projectionCursor: .root) + return applyHiddenContentHints(hints, to: nodes) + } + } + + /// Raw projection: every serialized node, in traversal order, at traversal depth. Depth is the + /// one narrowing a raw acquisition can prove complete, because raw depth *is* traversal depth. + private func appendPrivateAXRawNode(_ raw: [String: Any], to nodes: inout [RawAXNode], + hint: CaptureHint, viewport: CGRect, depth: Int, parentIndex: Int?) + { + if let limit = hint.depth, depth > limit { return } + let fields = privateAXFields(raw) + // No clip fold outside the regular projection: hittability stays the reported geometric fact. + let onScreen = fields.hasFrame && isVisibleInViewport(fields.rect, viewport) + let index = nodes.count + nodes.append( + privateAXNode(fields, index: index, depth: depth, parentIndex: parentIndex, onScreen: onScreen) + ) + for child in fields.children { + appendPrivateAXRawNode(child, to: &nodes, hint: hint, viewport: viewport, + depth: depth + 1, parentIndex: index) + } } private func appendPrivateAXNode(_ raw: [String: Any], to nodes: inout [RawAXNode], - hints: inout [Int: (above: Bool, below: Bool)], options: PresentationOptions, viewport: CGRect, + hints: inout [Int: (above: Bool, below: Bool)], hint: CaptureHint, viewport: CGRect, depth: Int, parentIndex: Int?, scrollContext: (index: Int, rect: CGRect)?, projectionCursor: FlatSnapshotProjectionCursor) { - if let limit = options.depth, depth > limit { return } - let rect = privateAXRect(raw["frame"]) - let label = privateAXPresentationString(raw["label"]) - let identifier = privateAXPresentationString(raw["identifier"]) - let value = privateAXPresentationString(raw["value"]) - let rawType = privateAXPresentationInt(raw["type"]) ?? 0 - let enabled = privateAXPresentationBool(raw["enabled"]) ?? true - let children = raw["children"] as? [[String: Any]] ?? [] - let hasSemanticContent = [label, identifier, value].contains(where: { !$0.isEmpty }) - let elementType = flatSnapshotElementType(rawElementType: rawType) - let hasFrame = !rect.isNull && !rect.isEmpty + if let limit = hint.depth, depth > limit { return } + let fields = privateAXFields(raw) + let rect = fields.rect + let children = fields.children + let elementType = fields.elementType let negligibleDecoration = parentIndex != nil - && !hasSemanticContent - && (!hasFrame + && !fields.hasSemanticContent + && (!fields.hasFrame || rect.width <= Self.privateAXProjectionTolerance || rect.height <= Self.privateAXProjectionTolerance) - let onScreen = hasFrame + let onScreen = fields.hasFrame && isVisibleInRegularSnapshot( rect, viewport: viewport, @@ -52,7 +96,7 @@ extension RunnerTests { let presentationVisible = projection.presentationVisible && !negligibleDecoration let decision = flatSnapshotFilterDecision( FlatSnapshotFilterNode(isRoot: parentIndex == nil, visible: presentationVisible), - options: options, visibilityPolicy: .viewportProjected) + hint: hint, visibilityPolicy: .viewportProjected) let include = decision.include if let hiddenFrame = projectionTransition.hiddenContentFrame, let scrollContext { @@ -62,15 +106,10 @@ extension RunnerTests { let currentIndex: Int? if include { currentIndex = nodes.count - let typeName = elementType.map(elementTypeName) ?? "Element(\(rawType))" - nodes.append(RawAXNode(index: nodes.count, type: typeName, - label: label.isEmpty ? nil : label, identifier: identifier.isEmpty ? nil : identifier, - value: value.isEmpty ? nil : value, rect: snapshotRect(from: rect), enabled: enabled, - focused: privateAXPresentationBool(raw["focused"]) == true ? true : nil, - selected: privateAXPresentationBool(raw["selected"]) == true ? true : nil, - hittable: onScreen && enabled && privateAXInteractiveCandidate(rawElementType: rawType), - depth: depth, parentIndex: parentIndex, hiddenContentAbove: nil, hiddenContentBelow: nil, - actions: raw["actions"] as? [String])) + nodes.append( + privateAXNode( + fields, index: nodes.count, depth: depth, parentIndex: parentIndex, onScreen: onScreen) + ) } else { currentIndex = parentIndex } let nextScrollContext: (index: Int, rect: CGRect)? @@ -84,13 +123,46 @@ extension RunnerTests { ) ?? scrollContext } else { nextScrollContext = scrollContext } for child in children { - appendPrivateAXNode(child, to: &nodes, hints: &hints, options: options, viewport: viewport, + appendPrivateAXNode(child, to: &nodes, hints: &hints, hint: hint, viewport: viewport, depth: depth + 1, parentIndex: currentIndex, scrollContext: nextScrollContext, projectionCursor: projection.descendants) } } + private func privateAXFields(_ raw: [String: Any]) -> PrivateAXFields { + let rawType = privateAXPresentationInt(raw["type"]) ?? 0 + return PrivateAXFields( + rect: privateAXRect(raw["frame"]), + label: privateAXPresentationString(raw["label"]), + identifier: privateAXPresentationString(raw["identifier"]), + value: privateAXPresentationString(raw["value"]), + rawType: rawType, + elementType: flatSnapshotElementType(rawElementType: rawType), + enabled: privateAXPresentationBool(raw["enabled"]) ?? true, + focused: privateAXPresentationBool(raw["focused"]) == true ? true : nil, + selected: privateAXPresentationBool(raw["selected"]) == true ? true : nil, + actions: raw["actions"] as? [String], + children: raw["children"] as? [[String: Any]] ?? [] + ) + } + + private func privateAXNode(_ fields: PrivateAXFields, index: Int, depth: Int, parentIndex: Int?, + onScreen: Bool) -> RawAXNode + { + RawAXNode(index: index, + type: fields.elementType.map(elementTypeName) ?? "Element(\(fields.rawType))", + label: fields.label.isEmpty ? nil : fields.label, + identifier: fields.identifier.isEmpty ? nil : fields.identifier, + value: fields.value.isEmpty ? nil : fields.value, + rect: snapshotRect(from: fields.rect), enabled: fields.enabled, + focused: fields.focused, selected: fields.selected, + hittable: onScreen && fields.enabled + && privateAXInteractiveCandidate(rawElementType: fields.rawType), + depth: depth, parentIndex: parentIndex, hiddenContentAbove: nil, hiddenContentBelow: nil, + actions: fields.actions) + } + private func privateAXPresentationString(_ value: Any?) -> String { guard let value else { return "" } return (value as? String ?? String(describing: value)) @@ -106,18 +178,33 @@ extension RunnerTests { #if AGENT_DEVICE_RUNNER_UNIT_TESTS extension RunnerTests { - func testPrivateAXRegularPresentationProjectsToViewportAndKeepsScrollHint() { - func frame(_ x: Double, _ y: Double, _ width: Double, _ height: Double) -> [String: Any] { - ["x": x, "y": y, "width": width, "height": height] - } - let root: [String: Any] = ["type": Int(XCUIElement.ElementType.application.rawValue), + fileprivate static func privateAXFrame( + _ x: Double, _ y: Double, _ width: Double, _ height: Double + ) -> [String: Any] { + ["x": x, "y": y, "width": width, "height": height] + } + + /// A scroll container whose second row is scrolled out of the viewport, plus an unlabeled + /// decoration: the shapes the regular projection folds away and the raw projection must keep. + fileprivate static var privateAXScrolledFixture: [String: Any] { + let frame = privateAXFrame + return ["type": Int(XCUIElement.ElementType.application.rawValue), "label": "Element", "frame": frame(0, 0, 402, 874), "children": [[ "type": Int(XCUIElement.ElementType.scrollView.rawValue), "frame": frame(0, 96, 402, 700), "actions": ["Scroll down"], - "children": [["type": Int(XCUIElement.ElementType.button.rawValue), "label": "Profile picture", "frame": frame(16, 120, 44, 44)], - ["type": Int(XCUIElement.ElementType.button.rawValue), "label": "Theme", "frame": frame(16, 900, 360, 44)]]]]] - let nodes = privateAXPresentation(rawRoot: root, - options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false), + "children": [ + ["type": Int(XCUIElement.ElementType.button.rawValue), "label": "Profile picture", + "frame": frame(16, 120, 44, 44), + "children": [["type": Int(XCUIElement.ElementType.image.rawValue), + "frame": frame(16, 120, 1, 1)]]], + ["type": Int(XCUIElement.ElementType.button.rawValue), "label": "Theme", + "frame": frame(16, 900, 360, 44)]]]]] + } + + func testPrivateAXRegularPresentationProjectsToViewportAndKeepsScrollHint() { + let nodes = privateAXAcquisition(rawRoot: Self.privateAXScrolledFixture, + hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: false, + customActions: false), viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) XCTAssertEqual(nodes.compactMap(\.label), ["Element", "Profile picture"]) let scrollView = nodes.first { $0.type == "ScrollView" } @@ -125,10 +212,51 @@ extension RunnerTests { XCTAssertEqual(scrollView?.actions, ["Scroll down"]) } - func testPrivateAXPresentationKeepsOffscreenSubtreeExcludedWhenChildFramesAreClamped() { - func frame(_ x: Double, _ y: Double, _ width: Double, _ height: Double) -> [String: Any] { - ["x": x, "y": y, "width": width, "height": height] + /// #1797 D4: the raw projection is the acquired tree. The offscreen row and the sub-pixel + /// decoration the regular projection folds away are both present, at traversal depth, and every + /// regular node still appears — `regular ⊆ raw` on the same capture. + func testPrivateAXRawProjectionKeepsEveryAcquiredNode() { + let viewport = CGRect(x: 0, y: 0, width: 402, height: 874) + let root = Self.privateAXScrolledFixture + let regular = privateAXAcquisition(rawRoot: root, + hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: true, + customActions: false), + viewport: viewport) + let raw = privateAXAcquisition(rawRoot: root, + hint: CaptureHint(projection: .raw, depth: nil, interactiveOnly: false, customActions: false), + viewport: viewport) + + XCTAssertEqual(raw.map(\.type), ["Application", "ScrollView", "Button", "Image", "Button"]) + XCTAssertEqual(raw.map(\.depth), [0, 1, 2, 3, 2]) + XCTAssertEqual(raw.map(\.parentIndex), [nil, 0, 1, 2, 1]) + XCTAssertEqual(raw.compactMap(\.label), ["Element", "Profile picture", "Theme"]) + // The offscreen row is a reported fact in raw, and reported facts do not become hittable + // just because the projection kept them. + XCTAssertEqual(raw.last?.hittable, false) + XCTAssertTrue(raw.allSatisfy { $0.hiddenContentAbove == nil && $0.hiddenContentBelow == nil }) + + let rawKeys = Set(raw.map { "\($0.type)-\($0.label ?? "")-\($0.rect.y)" }) + for node in regular { + XCTAssertTrue( + rawKeys.contains("\(node.type)-\(node.label ?? "")-\(node.rect.y)"), + "regular node \(node.type)/\(node.label ?? "") is missing from the raw projection" + ) } + XCTAssertGreaterThan(raw.count, regular.count) + } + + /// Raw depth is traversal depth, so a raw `--depth` request is the one narrowing this backend + /// can prove complete. + func testPrivateAXRawProjectionAppliesRequestedTraversalDepth() { + let raw = privateAXAcquisition(rawRoot: Self.privateAXScrolledFixture, + hint: CaptureHint(projection: .raw, depth: 2, interactiveOnly: false, customActions: false), + viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + XCTAssertEqual(raw.map(\.type), ["Application", "ScrollView", "Button", "Button"]) + XCTAssertEqual(raw.map(\.depth), [0, 1, 2, 2]) + } + + func testPrivateAXPresentationKeepsOffscreenSubtreeExcludedWhenChildFramesAreClamped() { + let frame = Self.privateAXFrame let root: [String: Any] = ["type": Int(XCUIElement.ElementType.application.rawValue), "label": "Element", "frame": frame(0, 0, 402, 874), "children": [[ "type": Int(XCUIElement.ElementType.table.rawValue), "frame": frame(0, 96, 402, 700), @@ -139,8 +267,9 @@ extension RunnerTests { ["type": Int(XCUIElement.ElementType.switch.rawValue), "label": "Theme", "frame": frame(340, 96, 46, 44)]]]]]]] - let nodes = privateAXPresentation(rawRoot: root, - options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false), + let nodes = privateAXAcquisition(rawRoot: root, + hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: false, + customActions: false), viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) XCTAssertEqual(nodes.compactMap(\.label), ["Element"]) @@ -155,8 +284,9 @@ extension RunnerTests { "label": "Settings semantics", "frame": zero, "children": [[ "type": Int(XCUIElement.ElementType.button.rawValue), "label": "Theme", "frame": zero], ["type": Int(XCUIElement.ElementType.other.rawValue), "frame": zero]]]]] - let nodes = privateAXPresentation(rawRoot: root, - options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false), + let nodes = privateAXAcquisition(rawRoot: root, + hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: true, + customActions: false), viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) XCTAssertEqual(nodes.compactMap(\.label), ["Element", "Settings semantics", "Theme"]) XCTAssertEqual(nodes.filter { $0.index != 0 }.map(\.hittable), [false, false]) diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift index 0eea6ceb1..428b4ef3d 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift @@ -133,7 +133,7 @@ extension RunnerTests { func recursiveTreeSnapshotAcquisition( context: SnapshotTraversalContext, - options: PresentationOptions + hint: CaptureHint ) -> SnapshotAcquisition { var cachedDescendantElements: [XCUIElement]? func collapsedTabDescendants() -> [XCUIElement] { @@ -205,7 +205,7 @@ extension RunnerTests { let parentIndex = entry.parentIndex let nearestScrollAnchor = entry.nearestScrollAnchor let projectionCursor = entry.projectionCursor - if let limit = options.depth, depth > limit { continue } + if let limit = hint.depth, depth > limit { continue } let evaluation = evaluateSnapshot(snapshot, in: context) let intersectsViewportAndScrollClip = isVisibleInRegularSnapshot( @@ -245,7 +245,7 @@ extension RunnerTests { label: evaluation.label, identifier: evaluation.identifier, valueText: evaluation.valueText, - options: options, + hint: hint, hittable: evaluation.hittable, visible: projection.presentationVisible, regularSnapshot: true @@ -316,6 +316,7 @@ extension RunnerTests { } return SnapshotAcquisition( + hint: hint, nodes: applyHiddenContentHints(hiddenContentHintsByNodeIndex, to: nodes), truncated: false, effectiveDepth: nil @@ -421,12 +422,12 @@ extension RunnerTests { func rawTreeSnapshotAcquisition( context: SnapshotTraversalContext, - options: PresentationOptions + hint: CaptureHint ) throws -> SnapshotAcquisition { var nodes: [RawAXNode] = [] func walk(_ snapshot: XCUIElementSnapshot, depth: Int, parentIndex: Int?) throws { - if let limit = options.depth, depth > limit { return } + if let limit = hint.depth, depth > limit { return } let evaluation = evaluateSnapshot(snapshot, in: context) let include = shouldAcquireSnapshotNode( @@ -434,7 +435,7 @@ extension RunnerTests { label: evaluation.label, identifier: evaluation.identifier, valueText: evaluation.valueText, - options: options, + hint: hint, hittable: evaluation.hittable, visible: evaluation.visible ) @@ -461,24 +462,24 @@ extension RunnerTests { } try walk(context.rootSnapshot, depth: 0, parentIndex: nil) - return SnapshotAcquisition(nodes: nodes, truncated: false, effectiveDepth: nil) + return SnapshotAcquisition(hint: hint, nodes: nodes, truncated: false, effectiveDepth: nil) } func querySweepSnapshotAcquisition( app: XCUIApplication, - options: PresentationOptions, + hint: CaptureHint, planDeadline: Date = .distantFuture ) -> SnapshotAcquisition { var nodes: [RawAXNode] = [ interactiveRootNode(rect: .zero) ] - if options.depth == 0 { - return SnapshotAcquisition(nodes: nodes, truncated: false, effectiveDepth: nil) + if hint.depth == 0 { + return SnapshotAcquisition(hint: hint, nodes: nodes, truncated: false, effectiveDepth: nil) } // Bounded by both its own sweep budget and the umbrella capture-plan deadline, so a // chained recovery tier can never push the plan past the main-thread watchdog (#1105). - let sweepDeadline = options.interactiveOnly + let sweepDeadline = hint.interactiveOnly ? Date().addingTimeInterval(Self.flatInteractiveFallbackBudget) : Date.distantFuture let deadline = min(sweepDeadline, planDeadline) @@ -498,7 +499,7 @@ extension RunnerTests { index: 0, parentIndex: 0, viewport: viewport, - options: options + hint: hint ) else { continue } @@ -544,7 +545,7 @@ extension RunnerTests { ) ) } - return SnapshotAcquisition(nodes: nodes, truncated: truncated, effectiveDepth: nil) + return SnapshotAcquisition(hint: hint, nodes: nodes, truncated: truncated, effectiveDepth: nil) } func snapshotAccessibilityUnavailable(failure: SnapshotCaptureFailure) -> DataPayload { @@ -886,7 +887,7 @@ extension RunnerTests { label: String, identifier: String, valueText: String?, - options: PresentationOptions, + hint: CaptureHint, hittable: Bool, visible: Bool, regularSnapshot: Bool = false @@ -900,14 +901,14 @@ extension RunnerTests { #endif if regularSnapshot { #if os(macOS) - if options.interactiveOnly && !visible && type != .application { + if hint.interactiveOnly && !visible && type != .application { return false } #endif if type == .application || type == .window { return true } return visible } - if options.interactiveOnly { + if hint.interactiveOnly { if isScrollableContainer(snapshot, visible: visible) { return true } if interactiveTypes.contains(type) { return true } if hittable && type != .other { return true } @@ -938,7 +939,7 @@ extension RunnerTests { func makeSnapshotTraversalContext( app: XCUIApplication, - options: PresentationOptions, + hint: CaptureHint, captureDeadline: Date = .distantFuture, treeCaptureSliceBudgetOverride: TimeInterval? = nil ) throws -> SnapshotTraversalContext? { @@ -963,7 +964,7 @@ extension RunnerTests { viewport: viewport, flatSnapshots: flatSnapshots, snapshotRanges: snapshotRanges, - maxDepth: options.depth ?? Int.max + maxDepth: hint.depth ?? Int.max ) } @@ -1576,7 +1577,7 @@ extension RunnerTests { index: Int, parentIndex: Int?, viewport: CGRect, - options: PresentationOptions + hint: CaptureHint ) -> RawAXNode? { var node: RawAXNode? let exceptionMessage = RunnerObjCExceptionCatcher.catchException({ @@ -1584,7 +1585,7 @@ extension RunnerTests { let frame = element.frame if frame.isNull || frame.isEmpty { return } let visible = isVisibleInViewport(frame, viewport) - if options.interactiveOnly && !visible { return } + if hint.interactiveOnly && !visible { return } #if os(macOS) if !visible { return } #endif @@ -1600,7 +1601,7 @@ extension RunnerTests { ) if !flatSnapshotFilterDecision( filterNode, - options: options, + hint: hint, visibilityPolicy: .interactiveOnly ).include { return diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotCapturePlan.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotCapturePlan.swift index 3dc3eab3c..06f78b767 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotCapturePlan.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotCapturePlan.swift @@ -55,6 +55,19 @@ enum SnapshotBackendKind: String, CaseIterable { } } + /// The raw projection is the acquired tree, so only a backend that enumerates a hierarchy can + /// serve it. The query sweep answers an interactive element query: it has no hierarchy to + /// return, and planning it for `--raw` is exactly how a raw request gets answered with regular + /// membership (#1797 D4). + var supportsRawProjection: Bool { + switch self { + case .recursiveTree, .privateAX: + return true + case .querySweep: + return false + } + } + var isAvailableOnCurrentPlatform: Bool { switch self { case .recursiveTree, .querySweep: @@ -118,7 +131,10 @@ extension RunnerTests { // MARK: Plan definitions static let regularVisiblePlan: [SnapshotBackendKind] = [.recursiveTree, .querySweep, .privateAX] - static let rawDiagnosticPlan: [SnapshotBackendKind] = [.recursiveTree, .privateAX] + /// Derived from the backend trait rather than hand-listed: a backend that cannot serve the raw + /// projection drops out of the raw plan by construction, and a new one joins it by declaring the + /// trait instead of by someone remembering this line. + static let rawDiagnosticPlan: [SnapshotBackendKind] = regularVisiblePlan.filter(\.supportsRawProjection) // MARK: XCTest accessibility channel penalty (cross-attempt memory, #1105/#1156) // @@ -459,14 +475,14 @@ extension RunnerTests { deadline: Date, treeCaptureSliceBudgetOverride: TimeInterval? ) throws -> SnapshotBackendCapture? { - let acquisitionOptions = SnapshotPresentation.conservativeAcquisitionOptions(for: options) + let hint = SnapshotPresentation.captureHint(for: options) let acquisition: SnapshotAcquisition? switch kind { case .recursiveTree: guard let context = try makeSnapshotTraversalContext( app: app, - options: acquisitionOptions, + hint: hint, captureDeadline: deadline, treeCaptureSliceBudgetOverride: treeCaptureSliceBudgetOverride ) @@ -478,9 +494,9 @@ extension RunnerTests { timeout: min(treeCaptureSliceBudget, max(0.5, deadline.timeIntervalSinceNow)), timeoutError: snapshotMainThreadTimeoutError("processing tree snapshot") ) { - acquisitionOptions.raw - ? try self.rawTreeSnapshotAcquisition(context: context, options: acquisitionOptions) - : self.recursiveTreeSnapshotAcquisition(context: context, options: acquisitionOptions) + hint.isRaw + ? try self.rawTreeSnapshotAcquisition(context: context, hint: hint) + : self.recursiveTreeSnapshotAcquisition(context: context, hint: hint) } case .querySweep: acquisition = try runMainThreadWork( @@ -490,19 +506,41 @@ extension RunnerTests { ) { self.querySweepSnapshotAcquisition( app: app, - options: acquisitionOptions, + hint: hint, planDeadline: deadline ) } case .privateAX: acquisition = privateAXSnapshotAcquisition( app: app, - options: acquisitionOptions, + hint: hint, deadline: deadline ) } guard let acquisition else { return nil } - return SnapshotPresentation.present(acquisition, options: options) + guard let capture = SnapshotPresentation.present(acquisition, options: options) else { + throw Self.snapshotProjectionMismatchFailure( + kind, + requested: hint.projection, + acquired: acquisition.hint.projection + ) + } + return capture + } + + /// A backend that answers a request with the other projection loses its tier and says why, so + /// the miss lands in the quality verdict instead of shipping as a correct-looking capture. + static func snapshotProjectionMismatchFailure( + _ kind: SnapshotBackendKind, + requested: CaptureHint.Projection, + acquired: CaptureHint.Projection + ) -> SnapshotCaptureFailure { + SnapshotCaptureFailure( + code: "IOS_SNAPSHOT_PROJECTION_MISMATCH", + message: + "the \(kind.rawValue) backend returned a \(acquired.rawValue) capture for a \(requested.rawValue) snapshot request", + hint: "This is a runner bug: report it with the failing command and the app under test." + ) } // MARK: Quality classifier (the single source of "is this snapshot degraded") @@ -820,6 +858,30 @@ extension RunnerTests { XCTAssertEqual(Self.xcTestChannelStateFirstFailure(.boundedXCTestProbe)?.code, "budget") } + /// The raw plan is derived from what each backend can actually serve, not from a second + /// hand-maintained list. Non-vacuity: flipping `querySweep.supportsRawProjection` to true adds it + /// to the plan and fails the first two assertions — which is exactly the shape of #1797 D4, a + /// `--raw` request answered by a backend that has no hierarchy to return. + func testRawDiagnosticPlanCarriesOnlyBackendsThatCanServeRaw() { + XCTAssertEqual(Self.rawDiagnosticPlan, [.recursiveTree, .privateAX]) + XCTAssertEqual( + SnapshotBackendKind.allCases.filter { !$0.supportsRawProjection }, [.querySweep]) + XCTAssertTrue(Self.rawDiagnosticPlan.allSatisfy(\.supportsRawProjection)) + // Tree-first error propagation is the raw plan's other contract (ADR 0004). + XCTAssertEqual(Self.rawDiagnosticPlan.first, .recursiveTree) + } + + /// A projection mismatch is a runner bug, not an accessibility failure: it must not take the + /// AX-failure terminal route (rethrow / fail-closed), just drop its tier with a named reason. + func testProjectionMismatchFailureIsStructuredAndNotAnAxFailure() { + let failure = Self.snapshotProjectionMismatchFailure( + .querySweep, requested: .raw, acquired: .regular) + XCTAssertEqual(failure.code, "IOS_SNAPSHOT_PROJECTION_MISMATCH") + XCTAssertTrue(failure.message.contains("queries")) + XCTAssertTrue(failure.message.contains("raw")) + XCTAssertFalse(Self.isAxSnapshotFailure(failure)) + } + /// #1634 P2: the decoded wire field must reach capture options and its /// applicable plan. A pinned REGULAR capture defers to privateAX-first; the /// RAW diagnostic plan is never rerouted by the pin — raw keeps tree-first diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift index 7edba90d5..fc6884a2d 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift @@ -22,12 +22,49 @@ struct RawAXNode { var actions: [String]? = nil } +/// The acquisition-facing view of a snapshot request, derived once by +/// `SnapshotPresentation.captureHint(for:)`. +/// +/// Backends read a hint, never `PresentationOptions`: presentation owns interpretation, and a hint +/// may narrow acquisition only where the backend can prove the narrowing complete for the requested +/// projection (#1797 conservatism). Everything else a hint carries is budget and ordering. +struct CaptureHint { + /// Which projection this acquisition must serve. Backends that cannot serve one are not planned + /// for it (`SnapshotBackendKind.supportsRawProjection`), and presentation refuses an acquisition + /// captured for the other projection rather than relabeling it, so a `--raw` request can never be + /// answered with regular-projection membership (#1797 D4). + enum Projection: String { + case regular + case raw + } + + let projection: Projection + /// Traversal-depth budget. + /// + /// Declared residue (#1797): complete for the raw projection, whose presented depth *is* + /// traversal depth. Regular presentation emits collapsed depth, so cutting the traversal at this + /// limit can still drop a node that would have presented within it — the open "visible-depth + /// frontier completeness" obligation, kept as-is here because the cut is also what keeps + /// `--depth 1` probes cheap. + let depth: Int? + /// Regular-projection acquisition budget. The raw projection is the acquired tree, so it never + /// carries this: `--raw -i` returns everything the backend serialized. + let interactiveOnly: Bool + let customActions: Bool + + var isRaw: Bool { projection == .raw } +} + /// One backend attempt after acquisition and its current backend-specific interpretation. /// -/// Step 2 makes this the only input accepted by snapshot presentation. Later #1797 steps move the -/// interpretation that still precedes this value into `SnapshotPresentation` without changing the -/// capture-plan seam. +/// It is the only input snapshot presentation accepts, and it carries the hint it was captured +/// under so the two sides of the seam cannot disagree about which projection this is. Later #1797 +/// steps move the interpretation that still precedes this value — the clip fold and hittability — +/// into `SnapshotPresentation` without changing the capture-plan seam. struct SnapshotAcquisition { + /// The hint this acquisition was captured under. Presentation compares it with the requested + /// projection instead of trusting the backend's label. + let hint: CaptureHint let nodes: [RawAXNode] let truncated: Bool let effectiveDepth: Int? @@ -108,18 +145,76 @@ enum SnapshotPresentation { /// The capture plan's single presentation route for every snapshot backend. /// - /// Eligibility is the first intentional step-3 semantic delta: regular presentation keeps a - /// root carrier plus nodes with an interactive type or non-empty semantic content. Raw snapshots - /// retain their acquired membership until the dedicated raw-projection migration lands. + /// Refuses an acquisition captured for the other projection instead of presenting it under the + /// requested label: a backend that ignores `--raw` loses its tier and the plan advances, rather + /// than returning regular-projection membership called raw (#1797 D4). static func present( _ acquisition: SnapshotAcquisition, options: PresentationOptions + ) -> SnapshotBackendCapture? { + let requested = captureHint(for: options) + guard acquisition.hint.projection == requested.projection else { + NSLog( + "AGENT_DEVICE_RUNNER_SNAPSHOT_PROJECTION_MISMATCH requested=%@ acquired=%@", + requested.projection.rawValue, + acquisition.hint.projection.rawValue + ) + return nil + } + switch requested.projection { + case .regular: + return presentRegular(acquisition, options: options) + case .raw: + return presentRaw(acquisition, options: options) + } + } + + /// Visible projection: eligibility (an interactive type or non-empty semantic content, below the + /// root carrier), scope, and the acquired scroll hints. The only interpreter of what a screen + /// currently shows. + static func presentRegular( + _ acquisition: SnapshotAcquisition, + options: PresentationOptions + ) -> SnapshotBackendCapture { + project(acquisition, options: options, projection: .regular) + } + + /// Diagnostic projection: the acquired tree, normalized. Scope and depth apply when explicitly + /// requested; membership is never narrowed, so `interactive ⊆ regular ⊆ raw` holds for every + /// backend (ADR 0004's raw contract). + static func presentRaw( + _ acquisition: SnapshotAcquisition, + options: PresentationOptions ) -> SnapshotBackendCapture { - let scopedRawNodes = applyScope(to: acquisition.nodes, options: options) - let nodes = presentedNodes(from: scopedRawNodes, options: options) + project(acquisition, options: options, projection: .raw) + } + + /// Derives the one acquisition-facing view of a request, so no backend re-reads + /// `PresentationOptions` and reaches its own conclusion about what to capture. + /// + /// Scope re-roots the presented tree and depth counts from that root, so a scoped request + /// narrows neither: acquire broad, select once in presentation. + static func captureHint(for options: PresentationOptions) -> CaptureHint { + let scoped = SnapshotScopePolicy.isActive(options.scope) + let projection: CaptureHint.Projection = options.raw ? .raw : .regular + return CaptureHint( + projection: projection, + depth: scoped ? nil : options.depth, + interactiveOnly: projection == .raw ? false : options.interactiveOnly, + customActions: options.customActions + ) + } + + private static func project( + _ acquisition: SnapshotAcquisition, + options: PresentationOptions, + projection: CaptureHint.Projection + ) -> SnapshotBackendCapture { + let scopedRawNodes = applyScope(to: acquisition.nodes, options: options, projection: projection) + let nodes = presentedNodes(from: scopedRawNodes, projection: projection) let qualityPayload: DataPayload? = SnapshotScopePolicy.isActive(options.scope) ? DataPayload( - nodes: presentedNodes(from: acquisition.nodes, options: options), + nodes: presentedNodes(from: acquisition.nodes, projection: projection), truncated: acquisition.truncated ) : nil @@ -134,20 +229,6 @@ enum SnapshotPresentation { ) } - /// Scope and depth cannot safely narrow acquisition until a backend proves its hint complete. - /// Acquire the broad tree, then apply both relative to the selected presentation subtree. - static func conservativeAcquisitionOptions(for options: PresentationOptions) -> PresentationOptions { - guard SnapshotScopePolicy.isActive(options.scope) else { return options } - return PresentationOptions( - interactiveOnly: options.interactiveOnly, - depth: nil, - scope: nil, - raw: options.raw, - preferredBackend: options.preferredBackend, - customActions: options.customActions - ) - } - /// Explicit carve-out for selector queries and system-modal reads that intentionally return one /// already-resolved element instead of traversing a snapshot backend. static func singleElementRead(_ node: RawAXNode) -> PresentedNode { @@ -156,9 +237,9 @@ enum SnapshotPresentation { private static func presentedNodes( from rawNodes: [RawAXNode], - options: PresentationOptions + projection: CaptureHint.Projection ) -> [PresentedNode] { - if options.raw { + if projection == .raw { return rawNodes.map(PresentedNode.init(presenting:)) } @@ -192,7 +273,8 @@ enum SnapshotPresentation { private static func applyScope( to rawNodes: [RawAXNode], - options: PresentationOptions + options: PresentationOptions, + projection: CaptureHint.Projection ) -> [RawAXNode] { switch SnapshotScopePolicy.select( fromPreorder: rawNodes, @@ -200,7 +282,7 @@ enum SnapshotPresentation { depth: \.depth, semanticValues: { [$0.label, $0.identifier, $0.value] }, subtreeContributes: { range in - options.raw || rawNodes[range].contains(where: isEligibleForRegularPresentation) + projection == .raw || rawNodes[range].contains(where: isEligibleForRegularPresentation) } ) { case .unscoped: diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift index 15136d79f..ae2a85893 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift @@ -25,8 +25,10 @@ extension RunnerTests { let encoder = JSONEncoder() encoder.outputFormatting = [.sortedKeys] - let capture = SnapshotPresentation.present( + let capture = try XCTUnwrap(SnapshotPresentation.present( SnapshotAcquisition( + hint: CaptureHint( + projection: .raw, depth: nil, interactiveOnly: false, customActions: false), nodes: [raw], truncated: true, effectiveDepth: 4, @@ -43,7 +45,7 @@ extension RunnerTests { scope: nil, raw: true ) - ) + )) let nodes = try XCTUnwrap(capture.payload.nodes) let encoded = try encoder.encode(nodes) @@ -102,14 +104,20 @@ extension RunnerTests { node(10, type: "StaticText", label: "Reparented content", depth: 2, parentIndex: 9), ] let regular = try XCTUnwrap( - SnapshotPresentation.present( - SnapshotAcquisition(nodes: acquired, truncated: false, effectiveDepth: nil), + SnapshotPresentation.presentRegular( + SnapshotAcquisition( + hint: CaptureHint( + projection: .regular, depth: nil, interactiveOnly: false, customActions: false), + nodes: acquired, truncated: false, effectiveDepth: nil), options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false) ).payload.nodes ) let interactive = try XCTUnwrap( - SnapshotPresentation.present( - SnapshotAcquisition(nodes: acquired, truncated: false, effectiveDepth: nil), + SnapshotPresentation.presentRegular( + SnapshotAcquisition( + hint: CaptureHint( + projection: .regular, depth: nil, interactiveOnly: true, customActions: false), + nodes: acquired, truncated: false, effectiveDepth: nil), options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false) ).payload.nodes ) @@ -127,8 +135,11 @@ extension RunnerTests { XCTAssertEqual(regular.compactMap(\.value), ["3 items"]) let raw = try XCTUnwrap( - SnapshotPresentation.present( - SnapshotAcquisition(nodes: acquired, truncated: false, effectiveDepth: nil), + SnapshotPresentation.presentRaw( + SnapshotAcquisition( + hint: CaptureHint( + projection: .raw, depth: nil, interactiveOnly: false, customActions: false), + nodes: acquired, truncated: false, effectiveDepth: nil), options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: true) ).payload.nodes ) @@ -166,6 +177,8 @@ extension RunnerTests { } let acquisition = SnapshotAcquisition( + hint: CaptureHint( + projection: .regular, depth: nil, interactiveOnly: true, customActions: false), nodes: [ node(0, type: "Application", label: "App", depth: 0, parentIndex: nil), node(1, type: "Button", label: "Earlier sibling", depth: 1, parentIndex: 0), @@ -184,7 +197,7 @@ extension RunnerTests { scope: " SCOPE-ROOT ", raw: false ) - let capture = SnapshotPresentation.present(acquisition, options: options) + let capture = try XCTUnwrap(SnapshotPresentation.present(acquisition, options: options)) let nodes = try XCTUnwrap(capture.payload.nodes) XCTAssertEqual(nodes.map(\.label), [nil, "Child"]) @@ -195,8 +208,14 @@ extension RunnerTests { XCTAssertEqual(capture.qualityPayload?.nodes?.count, 6) let raw = try XCTUnwrap( - SnapshotPresentation.present( - acquisition, + SnapshotPresentation.presentRaw( + SnapshotAcquisition( + hint: CaptureHint( + projection: .raw, depth: nil, interactiveOnly: false, customActions: false), + nodes: acquisition.nodes, + truncated: false, + effectiveDepth: nil + ), options: PresentationOptions( interactiveOnly: true, depth: 1, @@ -208,12 +227,12 @@ extension RunnerTests { XCTAssertEqual(raw.map(\.type), ["Other", "StaticText", "Image"]) XCTAssertEqual(raw.map(\.depth), [0, 1, 1]) - let hint = SnapshotPresentation.conservativeAcquisitionOptions(for: options) - XCTAssertNil(hint.scope) + let hint = SnapshotPresentation.captureHint(for: options) XCTAssertNil(hint.depth) XCTAssertTrue(hint.interactiveOnly) + XCTAssertEqual(hint.projection, .regular) - let missing = SnapshotPresentation.present( + let missing = try XCTUnwrap(SnapshotPresentation.present( acquisition, options: PresentationOptions( interactiveOnly: true, @@ -221,9 +240,76 @@ extension RunnerTests { scope: "missing", raw: false ) - ) + )) XCTAssertEqual(missing.payload.nodes?.count, 0) XCTAssertNil(RunnerTests.sparsePayloadReason(try XCTUnwrap(missing.qualityPayload))) } + + /// #1797 D4: a backend that answers a `--raw` request with a regular capture (or the reverse) + /// loses its tier instead of having its output relabeled. Non-vacuity: dropping the projection + /// guard makes both `XCTAssertNil` assertions fail, and the raw projection then returns the + /// regular capture's four nodes under the raw label. + func testPresentationRefusesAnAcquisitionCapturedForTheOtherProjection() throws { + func node(_ index: Int, type: String, label: String?, parentIndex: Int?) -> RawAXNode { + RawAXNode( + index: index, type: type, label: label, identifier: nil, value: nil, + rect: SnapshotRect(x: 0, y: Double(index * 20), width: 100, height: 20), + enabled: true, focused: nil, selected: nil, hittable: false, + depth: parentIndex == nil ? 0 : 1, parentIndex: parentIndex, + hiddenContentAbove: nil, hiddenContentBelow: nil + ) + } + let nodes = [ + node(0, type: "Application", label: "App", parentIndex: nil), + node(1, type: "Button", label: "Continue", parentIndex: 0), + ] + let rawRequest = PresentationOptions( + interactiveOnly: false, depth: nil, scope: nil, raw: true) + let regularRequest = PresentationOptions( + interactiveOnly: false, depth: nil, scope: nil, raw: false) + + let regularAcquisition = SnapshotAcquisition( + hint: SnapshotPresentation.captureHint(for: regularRequest), + nodes: nodes, truncated: false, effectiveDepth: nil) + let rawAcquisition = SnapshotAcquisition( + hint: SnapshotPresentation.captureHint(for: rawRequest), + nodes: nodes, truncated: false, effectiveDepth: nil) + + XCTAssertNil(SnapshotPresentation.present(regularAcquisition, options: rawRequest)) + XCTAssertNil(SnapshotPresentation.present(rawAcquisition, options: regularRequest)) + XCTAssertEqual( + SnapshotPresentation.present(rawAcquisition, options: rawRequest)?.payload.nodes?.count, 2) + XCTAssertEqual( + SnapshotPresentation.present(regularAcquisition, options: regularRequest)? + .payload.nodes?.count, 2) + } + + /// The one derivation every backend reads. Non-vacuity: returning the request's own depth for a + /// scoped capture, or keeping `interactiveOnly` on the raw projection, each fails one assertion. + func testCaptureHintIsTheOnlyAcquisitionViewOfARequest() { + let scoped = SnapshotPresentation.captureHint( + for: PresentationOptions( + interactiveOnly: true, depth: 2, scope: "Settings", raw: false)) + // Scope re-roots the tree and depth counts from that root: neither can narrow acquisition. + XCTAssertNil(scoped.depth) + XCTAssertEqual(scoped.projection, .regular) + + let depthOnly = SnapshotPresentation.captureHint( + for: PresentationOptions(interactiveOnly: true, depth: 2, scope: nil, raw: false)) + XCTAssertEqual(depthOnly.depth, 2) + + // The raw projection is the acquired tree, so `--raw -i` never narrows acquisition either. + let raw = SnapshotPresentation.captureHint( + for: PresentationOptions(interactiveOnly: true, depth: 3, scope: nil, raw: true)) + XCTAssertEqual(raw.projection, .raw) + XCTAssertFalse(raw.interactiveOnly) + XCTAssertEqual(raw.depth, 3) + XCTAssertTrue(raw.isRaw) + + let actions = SnapshotPresentation.captureHint( + for: PresentationOptions( + interactiveOnly: false, depth: nil, scope: nil, raw: false, customActions: true)) + XCTAssertTrue(actions.customActions) + } } #endif diff --git a/docs/adr/0004-ios-snapshot-backend-strategy.md b/docs/adr/0004-ios-snapshot-backend-strategy.md index 58457d69a..4d6a72302 100644 --- a/docs/adr/0004-ios-snapshot-backend-strategy.md +++ b/docs/adr/0004-ios-snapshot-backend-strategy.md @@ -124,6 +124,31 @@ Scoped iOS acquisition stays broad (including when depth is requested) until an narrowing hint complete. The daemon never reapplies scope after the wire; Android selects its root inside its TypeScript presentation and desktop surface runtimes retain their platform projection. +The third semantic layer splits presentation into two projections and gives acquisition one input. +`SnapshotPresentation.captureHint` derives a `CaptureHint` from the request; backends read the hint, +never `PresentationOptions`. A hint names the projection the acquisition must serve and may narrow +acquisition only where the backend can prove the narrowing complete for that projection: scope and +its relative depth never narrow, raw depth does (raw depth *is* traversal depth), and the raw +projection never carries `interactiveOnly`. `presentRegular` folds visibility, eligibility, scope, +and scroll hints; `presentRaw` is the acquired tree, normalized, with scope and depth applied only +when the request asked for them — so `interactive ⊆ regular ⊆ raw` holds per backend rather than per +backend implementation. `snapshot --raw -i` therefore returns the acquired tree instead of an +interactive-filtered one. + +Two structural rules keep a backend from answering a request with the other projection, the shape +that let a recovered `snapshot --raw` return viewport-pruned nodes labeled raw: the raw diagnostic +plan is derived from `SnapshotBackendKind.supportsRawProjection` rather than hand-listed, so a +backend with no hierarchy to return (the query sweep) cannot be planned for raw; and presentation +compares the requested projection with the hint the acquisition was captured under, dropping that +tier with a structured `IOS_SNAPSHOT_PROJECTION_MISMATCH` failure instead of presenting it under the +requested label. + +Declared residue: a regular-projection `--depth` request still cuts the traversal at that depth, +while regular presentation emits collapsed depth. A node whose presented depth would be within the +limit can therefore be dropped when structural wrappers put it deeper in the raw tree. The cut is +what keeps `--depth 1` probes cheap; making it complete is the outstanding visible-depth frontier +obligation (#1797), not a property of the current output. + When adding new iOS snapshot behavior, maintainers should first decide which strategy owns it. If a change tries to make regular snapshots fast by dropping visible controls behind a node budget, or tries to make raw snapshots safe by silently truncating, it is probably crossing strategy diff --git a/website/docs/docs/commands.md b/website/docs/docs/commands.md index 66f6a932d..d3abce7e1 100644 --- a/website/docs/docs/commands.md +++ b/website/docs/docs/commands.md @@ -325,7 +325,11 @@ agent-device get text @e1 agent-device get attrs @e1 ``` -- iOS snapshots use XCTest on simulators and physical devices. +- iOS snapshots use XCTest on simulators and physical devices. iOS `--raw` is the acquired tree on + whichever backend serves the capture: it keeps offscreen nodes, decorations, and structural + wrappers the default and `-i` views fold away, so a recovered raw capture shows the same hierarchy + a healthy one does. `--depth` still applies to raw (it counts traversal depth there), while `-i` + narrows the default projection only — `--raw -i` returns the acquired tree. - Android snapshots require the bundled Android snapshot helper. The first snapshot verifies and installs the helper APK if it is missing or outdated. Local ADB-backed sessions keep the helper process warm over an `adb forward` socket and report `androidSnapshot.helperTransport` as