Skip to content

feat(ios): split raw and regular snapshot projections behind one capture hint (#1797) - #1926

Merged
thymikee merged 1 commit into
mainfrom
claude/agent-device-1797-a3d0e0
Aug 20, 2026
Merged

feat(ios): split raw and regular snapshot projections behind one capture hint (#1797)#1926
thymikee merged 1 commit into
mainfrom
claude/agent-device-1797-a3d0e0

Conversation

@thymikee

Copy link
Copy Markdown
Member

Summary

Next semantic delta of #1797 (migration step 3, after eligibility #1850 and scope #1855): the two projections presentation always implied, and one derived input for acquisition.

D4, the bug this closes. The private-AX backend interpreted --raw as the regular projection. appendPrivateAXNode folded the viewport and scroll clips and dropped sub-pixel decorations during acquisition, so a snapshot --raw that recovered onto that backend — the route an app whose XCTest tree capture fails takes — returned viewport-pruned nodes labeled raw. The one view whose purpose is showing what the pruned view hid was itself pruned, by a second copy of a decision nothing related to the first.

What replaces it. SnapshotPresentation now exposes presentRegular (clip-folded eligibility, scope, scroll hints) and presentRaw (the acquired tree, normalized, narrowed only by a scope or depth the request asked for). Acquisition backends no longer read PresentationOptions; they read a CaptureHint derived once by SnapshotPresentation.captureHint, so what a capture may skip is stated in one place beside the reason skipping it keeps the projection complete:

hint field narrows acquisition? why
projection selects the acquisition walk raw acquisition has no visibility filter to consult
depth raw: yes · regular: yes, declared residue raw depth is traversal depth; regular presents collapsed depth
depth under --scope never depth counts from a scope root presentation picks
interactiveOnly regular only the raw projection is the acquired tree
customActions budget only per-element AX round trips

Two structural rules replace the hand-synchronized ones. The raw plan is now derived from SnapshotBackendKind.supportsRawProjection instead of a second hand-maintained list, so the query sweep — an interactive element query with no hierarchy to return — cannot be planned for a raw request. And presentation compares the requested projection against the hint the acquisition was captured under, dropping that tier with a structured IOS_SNAPSHOT_PROJECTION_MISMATCH failure (classified as a runner bug, deliberately not an AX failure, so it cannot take the rethrow/fail-closed terminal route) rather than presenting a capture under a label it does not deserve.

Behavior changes

  • snapshot --raw recovered onto private-AX returns the acquired tree: offscreen and clipped subtrees, structural wrappers, and decorations are all present, at traversal depth. --depth still narrows it.
  • Breaking: snapshot --raw -i returns the acquired tree instead of an interactive-filtered one. -i narrows the regular projection; combined with --raw it used to produce a third membership rule that differed per backend (the tree backend filtered, private-AX did not) — exactly the divergence class this issue exists to remove. Same direction as Android's C3 change in refactor(android): raw is the acquired tree; one presentation for dialog recovery; residues declared (#1832 C3) #1865.
  • Regular and -i output are unchanged on every backend.

Evidence

Unit (17 targeted runner tests green, iPhone 16 Pro / iOS 26.2), 6 of them new, added to the iOS PR lane list; pnpm check:xctest-selection green (165 declared, 0 unreachable).

Non-vacuity, each proven by rebuilding the runner with the production code mutated:

  • Route the raw projection back through the regular walk → testPrivateAXRawProjectionKeepsEveryAcquiredNode and testPrivateAXRawProjectionAppliesRequestedTraversalDepth fail (["Application", "ScrollView", "Button"] vs ["Application", "ScrollView", "Button", "Image", "Button"]); both regular private-AX tests still pass.
  • Remove the projection guard → testPresentationRefusesAnAcquisitionCapturedForTheOtherProjection fails, and the failure message shows the raw request being answered with the regular capture's two nodes: the D4 shape, reproduced.
  • Flip querySweep.supportsRawProjection to truetestRawDiagnosticPlanCarriesOnlyBackendsThatCanServeRaw fails with queries in the raw plan.

Live differential, Settings on iPhone 16 Pro / iOS 26.2, this branch's runner vs a runner built from main, both with the raw plan pinned to private-AX so the recovery tier serves --raw deterministically (scaffolding, not in the diff), each variant on its own state dir and daemon:

  • Regular projections unchanged, compared on full node signatures (type, label, identifier, value, rect, depth, parentIndex, hittable): tree-backed snapshot 73 = 73 identical, tree-backed snapshot -i 18 = 18 identical; private-AX snapshot -i --actions 20 = 20 identical, private-AX snapshot --actions on a scrolled screen 59 = 59 identical.
  • snapshot --raw on private-AX: 158 → 167 nodes on two independent runs, 0 nodes lost. All 9 gained nodes are unlabeled (named nodes 20 → 20, unlabeled 112 → 121) — the 1pt separators and structural wrappers the regular projection folds away. Settings' private-AX tree is mostly on screen, so this is the small end of the delta; the clipped-subtree case is what the fixtures pin.
  • One private-AX --actions pair differed (57 vs 58 nodes) because a Settings follow-up banner ("Ready for Apple Intelligence") appeared in the later run and shifted every row by 35pt — device state, not projection: the same pair's -i capture is identical, and the earlier same-state pair is identical too.

Sessions closed, daemons stopped, throwaway simulator shut down.

Repo gates: pnpm check:affected --run green ("all runnable checks passed"), pnpm check:xctest-selection green, oxfmt/oxlint clean.

One lane I could not run locally: the macOS host lane never starts on this machine — AgentDeviceRunnerUITests-Runner … Early unexpected exit, operation never finished bootstrapping (Test crashed with signal kill before establishing connection), 0 tests executed. A runner built from main fails identically on the same machine, so it is environmental, not this change; the same tests run in the iOS simulator lane above and in CI.

Docs

ADR 0004 gains the acquire/present third layer (two projections, the hint's conservatism table, the two structural rules) and the declared depth residue; CONTEXT.md gains capture hint; CHANGELOG and website/docs/docs/commands.md describe iOS --raw as the acquired tree next to Android's.

What #1797 still needs after this

  1. Clip fold into presentation — both backends still fold viewport/scroll clips during acquisition (shared helpers, one call site each).
  2. Geometry carrier (effective rect on the wire) with the rect-consumer inventory.
  3. Occlusion out of runner hittable with the hittable-consumer inventory — 118 production references today.
  4. Step-4 enforcement: choke-point invariant, tree pin + penalty-clear hook, per-PR fixture differentials, nightly live differentials, parity ratchets.
  5. The visible-depth frontier obligation declared here: a regular --depth request still cuts the traversal, while regular presentation emits collapsed depth.

Refs #1797.

…ure hint

The private-AX backend interpreted `--raw` as the regular projection: it folded
the viewport and scroll clips and dropped sub-pixel decorations before returning,
so a raw capture that recovered onto it answered with viewport-pruned nodes
labeled raw (#1797 D4). Nothing related the two copies of that decision.

Presentation now exposes the two projections it always implied. `presentRegular`
folds visibility, eligibility, scope and scroll hints; `presentRaw` is the
acquired tree, normalized, narrowed only by a scope or depth the request asked
for. Acquisition reads one derived `CaptureHint` instead of the request itself,
so what a capture may skip is stated once, beside the reason skipping it keeps
the projection complete: scope and its relative depth never narrow, raw depth
does (raw depth is traversal depth), and the raw projection never carries
`interactiveOnly` — `--raw -i` is the acquired tree.

Two structural rules replace the hand-synchronized ones. The raw plan is derived
from `SnapshotBackendKind.supportsRawProjection`, so the query sweep — an
interactive element query with no hierarchy to return — cannot be planned for a
raw request. And presentation compares the requested projection with the hint
the acquisition was captured under, dropping that tier with a structured
`IOS_SNAPSHOT_PROJECTION_MISMATCH` failure rather than presenting it under the
requested label.

Declared residue: a regular `--depth` request still cuts the traversal at that
depth while regular presentation emits collapsed depth, so a node that would
present within the limit can be dropped. The cut is what keeps `--depth 1`
probes cheap; making it complete is the open visible-depth frontier obligation.

Refs #1797 (migration step 3, raw-projection delta).
@github-actions

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 2.34 MB 2.34 MB -429 B
JS gzip 769.6 kB 769.4 kB -205 B
npm tarball 895.4 kB 897.0 kB +1.6 kB
npm unpacked 3.12 MB 3.12 MB +2.8 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.4 ms 27.4 ms +0.1 ms
CLI --help 70.0 ms 69.5 ms -0.6 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/registry.js -125 B -51 B
dist/src/script-source-bundle.js -28 B -42 B
dist/src/dispatch.js -19 B -13 B
dist/src/internal/daemon.js 0 B +1 B
dist/src/session2.js 0 B +1 B

@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-20 19:45 UTC

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed the acquire/present D4 delta at 0425726. The projection is now carried through a single capture hint, raw plans only select hierarchy-capable backends, and a backend cannot relabel a regular capture as raw. The targeted XCTest regressions, live private-AX differential, and all required CI lanes are green. Merge-ready.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Aug 20, 2026
@thymikee
thymikee merged commit 4137e42 into main Aug 20, 2026
31 checks passed
@thymikee
thymikee deleted the claude/agent-device-1797-a3d0e0 branch August 20, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant