Skip to content

feat(ios): publish effective snapshot geometry - #1931

Merged
thymikee merged 1 commit into
claude/1797-clip-foldfrom
claude/1797-effective-geometry
Aug 21, 2026
Merged

feat(ios): publish effective snapshot geometry#1931
thymikee merged 1 commit into
claude/1797-clip-foldfrom
claude/1797-effective-geometry

Conversation

@thymikee

@thymikee thymikee commented Aug 21, 2026

Copy link
Copy Markdown
Member

Summary

Stacked on PR #1929, targeting claude/1797-clip-fold at e4b19df6c.

This PR completes the next smallest #1797 architecture unit: the effective-geometry carrier and
the rect-consumer inventory/migration. It does not add the separate occlusion/actionability
enforcement slice.

  • Acquisition keeps RawAXNode.rect as the backend-reported frame.
  • Regular presentation now carries a SnapshotPresentationNode with both the raw node and its
    presentation-owned effectiveRect.
  • The regular wire projection writes effectiveRect through the existing PresentedNode.rect
    field. Raw snapshots and direct single-element reads retain reported geometry.
  • Scope, reindexing, hidden-content hints, and collapsed-depth projection preserve the carrier.
  • The only policy branch is the declared SnapshotVisibilityFold.Policy: iOS's
    cursorProjected policy supplies ancestor scroll clips; plainViewport does not. No backend
    name or acquisition implementation is inspected by the geometry code.

Rect inventory

Consumer class Owners reviewed Classification / migration
Acquisition facts and runner-internal geometry RunnerTests+Snapshot.swift, the acquisition half of RunnerTests+PrivateAXPresentation.swift, and raw-node builders in RunnerTests+Interaction.swift, RunnerTests+SystemModal.swift, and RunnerTests+CommandJournal.swift Reported geometry. These continue to use RawAXNode.rect for traversal, roots, deduplication, and direct reads.
Presentation and payload assembly RunnerTests+SnapshotPresentation.swift, RunnerTests+SnapshotVisibilityFold.swift, RunnerTests+SnapshotCapturePlan.swift Effective geometry for regular projection; raw geometry for raw/direct-read paths. The typed carrier is the single migration point.
iOS daemon presentation src/daemon/snapshot-presentation/ios/*.ts (rows, noise, action-shelf, actions, scroll, transitions, and web) Existing wire rect consumers; no parallel field or adapter. They now receive effective geometry automatically.
Occlusion, targeting, touch-point, and overlays src/snapshot/snapshot-occlusion.ts, src/core/interaction-targeting.ts, src/core/interaction-touch-point.ts, src/core/react-native-overlay.ts, src/daemon/screenshot-overlay*.ts, and src/daemon/session-target-evidence.ts Effective geometry. These are post-wire consumers and remain unchanged.
Freshness, unchanged snapshots, selectors, Maestro, find/interaction, replay, and diff evidence src/commands/capture/runtime/snapshot-unchanged.ts, src/commands/interaction/runtime/*, packages/selectors/src/internal/*, packages/maestro/src/internal/*, src/daemon/handlers/*, src/daemon/adapters/maestro/*, and src/screenshot-diff/* Effective geometry from the existing rect contract; no consumer-specific migration was needed.

The complete rect search was classified before changing the carrier. No downstream wire consumer
was changed because rect remains the wire field and now has the intended regular-projection
meaning.

#1797 checklist status

Complete in this PR:

  • Migration step 3's effective-geometry carrier.
  • Exhaustive reported-vs-effective rect inventory and migration at the presentation choke point.
  • Raw reported geometry and the existing wire shape remain compatible.

Still separate / remaining:

  • The independent hittability/actionability and occlusion-ownership slice.
  • The visible-depth frontier completeness residue.
  • Step 4 structural enforcement, backend-construction gates, differential/conformance proof, and
    remaining cross-runtime parity work.

Validation

  • pnpm build
  • pnpm check:xctest-selection
  • pnpm check:affected --run (all runnable checks passed; GitHub-authoritative native/device lanes
    remain delegated to CI)
  • Fresh pnpm build:xcuitest:ios and pnpm build:xcuitest:macos test builds passed.
  • Final iOS Simulator focused run: 9 tests passed, covering effective intersection, fully clipped
    geometry, plain-vs-cursor policy, regular fold behavior, raw/regular wire compatibility, and
    scope/depth.
  • Explicit red evidence: temporarily bypassing the carrier (node.effectiveRect
    node.raw.rect) made testRegularPresentationPublishesEffectiveRectWhileRawKeepsReportedFrame
    fail with 100.0 vs 52.0; the carrier path was restored before commit.
  • A focused macOS host execution attempt stalled before XCTest output at the host automation
    boundary, so this PR claims the macOS compilation result but not a host execution pass.

Live full-production-route iOS geometry evidence

Captured at this PR's exact head 428b8c9b62b30d202c06b5a3b568cb03045d4851 after PR #1929's
Coverage lane passed. Stacked base #1929 remains a readiness dependency for this stack until its
Coverage lane is green; that dependency is now green for this evidence.

The existing booted iPhone 17 Pro (F7D6F9A4-4FCC-4DD7-AC0B-3280C9319CB9) ran the shipped
examples/test-app Catalog screen. No simulator was created or deleted. The source and runner
paths were rebuilt to defeat daemon/runner staleness:

  • pnpm build
  • pnpm clean:daemon
  • AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH=/private/tmp/agent-device-1931-ios-live-428b8c9b6 pnpm build:xcuitest:ios
  • Runner artifact: /private/tmp/agent-device-1931-ios-live-428b8c9b6/Build/Products/AgentDeviceRunner_AgentDeviceRunnerUITests_iphonesimulator26.2-arm64.xctestrun

The purpose-specific session was geometry-live-428b8c9b6, with isolated state at
/private/tmp/agent-device-1931-geometry.SBL95K/state. The public CLI route was:

pnpm ad open com.callstack.agentdevicelab --platform ios --udid F7D6F9A4-4FCC-4DD7-AC0B-3280C9319CB9 --session geometry-live-428b8c9b6 --state-dir /private/tmp/agent-device-1931-geometry.SBL95K/state --relaunch --foreground --json
pnpm ad press @e26 --platform ios --udid F7D6F9A4-4FCC-4DD7-AC0B-3280C9319CB9 --session geometry-live-428b8c9b6 --state-dir /private/tmp/agent-device-1931-geometry.SBL95K/state --settle --json
node bin/agent-device.mjs scroll down --platform ios --udid F7D6F9A4-4FCC-4DD7-AC0B-3280C9319CB9 --session geometry-live-428b8c9b6 --state-dir /private/tmp/agent-device-1931-geometry.SBL95K/state --settle --json
node bin/agent-device.mjs snapshot --platform ios --udid F7D6F9A4-4FCC-4DD7-AC0B-3280C9319CB9 --session geometry-live-428b8c9b6 --state-dir /private/tmp/agent-device-1931-geometry.SBL95K/state --force-full --json > /private/tmp/agent-device-1931-geometry.SBL95K/regular.json
node bin/agent-device.mjs snapshot --raw --platform ios --udid F7D6F9A4-4FCC-4DD7-AC0B-3280C9319CB9 --session geometry-live-428b8c9b6 --state-dir /private/tmp/agent-device-1931-geometry.SBL95K/state --force-full --json > /private/tmp/agent-device-1931-geometry.SBL95K/raw.json

Both snapshots returned structured success: true output for
com.callstack.agentdevicelab with snapshotQuality.state: "healthy" and
snapshotQuality.backend: "tree". The regular artifact reports visibility.partial: true,
45 nodes, and scroll-hidden-above/scroll-hidden-below; the raw artifact reports
visibility.partial: false, 279 nodes, and no visibility reasons. The isolated
events.ndjson records the regular snapshot as flags: {platform: "ios"} and the raw snapshot
as flags: {platform: "ios", raw: true}; runner.log records both commands as accepted and
completed successfully. This proves the projection through CLI → daemon → runner → wire rather
than through a helper or XCTest-only path.

For the deliberately clipped Bakery descendant in the same post-scroll state:

Wire projection x y width height
Regular effective rect 34.333334 62.000000 75.333328 24.333314
Raw reported rect 34.333334 48.666658 75.333328 37.666656

The raw rect spans y=48.666658..86.333314; the regular rect spans
y=62.000000..86.333314. The regular wire output therefore carries the effective top clip and
reduced height, while raw output preserves the original reported geometry.

Cleanup completed with close --session geometry-live-428b8c9b6; the isolated session list was
empty afterward. Scoped daemon cleanup removed only this verification daemon and retained runner;
no verification-owned processes remained. The iPhone 17 Pro stayed Booted as it was before the
run, and the JSON/log artifacts remain at /private/tmp/agent-device-1931-geometry.SBL95K/.
No source files changed, so the exact head remains 428b8c9b62b30d202c06b5a3b568cb03045d4851.

No app-control-bench files or unrelated worktrees/devices were touched. Nothing is merged.

@github-actions

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 2.34 MB 2.34 MB 0 B
JS gzip 770.2 kB 770.2 kB 0 B
npm tarball 897.9 kB 898.5 kB +604 B
npm unpacked 3.13 MB 3.13 MB +2.6 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.9 ms 28.9 ms +1.1 ms
CLI --help 70.3 ms 70.8 ms +0.5 ms

Top changed chunks: no changes in the largest emitted chunks.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed exact head 428b8c9: the implementation is code-clean. Regular presentation now carries one effective rectangle through the visibility fold and scope/reindex path, while raw/direct projections retain reported geometry; the existing wire shape is unchanged. Exact-head CI is green and Size is +2.6 kB unpacked. One merge-readiness proof is still missing: the current simulator evidence executes fabricated RawAXNode/CGRect unit cases, not the shipped agent-device snapshot route. Please attach an exact-head iOS simulator/device run with a partially clipped element showing regular wire rect = viewport/ancestor intersection and raw/direct rect = reported frame, including the target and command/artifact. This stack also cannot land before base #1929's failed Coverage lane is rerun or otherwise resolved.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Aug 21, 2026
@thymikee
thymikee merged commit af96c66 into main Aug 21, 2026
28 checks passed
@thymikee
thymikee deleted the claude/1797-effective-geometry branch August 21, 2026 09:27
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-21 09:27 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant