feat(fidelity): anchored selectors, unified target rule, geometric card isolation

Fixes the two dogfooding failures in cli-got28: components could not be
reliably isolated, and notes were associated with the wrong code.

F1 — anchored, code-locating selectors:
- SelectorEngine.generate anchors a non-identified target to its nearest
  identified ancestor and scopes indices within that component
  (#Settings.Models >> @save / text="…" / AXButton[n-in-scope]); component-local
  indices survive unrelated same-role tree changes.
- Emit text= steps; add Selector.fromPath so a #<slash/path> that resolves to
  nothing is never emitted.

F2 — deterministic component isolation:
- One pure, unit-tested AnnotationTargetRule (deepest actionable, else deepest
  meaningful; never window/chrome/ghost) shared by all three sources (AX,
  NSView-tree, iOS UIView), replacing the macOS "deepest meaningful" vs iOS
  "nearest identified" split. A click inside a button binds to the button, not
  its glyph.
- ComponentLadderSource + widen affordance to step selection up to the
  enclosing component. Docs (DECISIONS/README/spike/PARITY) corrected.

F3 — deterministic note->code mapping:
- AnnotationNote carries component / elementRole / elementText / unseeded,
  surfaced in markdown + JSON.

Geometric card isolation (dogfood follow-up):
- VirgilHUD's .axCardSurface seeds a card's id on a .background leaf that is a
  SIBLING of the content, not an ancestor, so pure-ancestry anchoring/widening
  could never reach the card. componentLadder now resolves enclosing components
  by frame containment (ancestor chain + each ancestor's direct children); the
  note component field and widening use it. The hover highlight shows a name tag
  so an element vs its enclosing card are distinguishable at a glance.

74 tests, incl. anchored-selector / mutation-stability / ambiguity / target-rule
/ widening / geometric-ladder / payload suites.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: angusbezzina

DECISIONS.md

@@ -9,13 +9,67 @@ Resolves the open decisions from the plan (planning/annotkit in the cli repo). P
 | Repo location | Standalone repo at `~/Development/annotkit` | VirgilHUD consumes it during dev via a local SwiftPM path dependency; public remote created by the maintainer. |
 | Versioning | SemVer, 0.x pre-1.0 | Breaking changes allowed while 0.x; 1.0 marks a stable public API. |
 | Default element source | Accessibility hierarchy | The only strategy that surfaces SwiftUI `accessibilityIdentifier` values. |
+| Annotation target rule | Deepest actionable, else deepest meaningful; anchor the selector to the nearest identifier | One rule on both platforms. Supersedes the earlier macOS "deepest meaningful" and iOS "nearest identified" split. See below. |
 | Opt-in element source | View tree (NSView/UIView) | Surfaces concrete view class names; richer for AppKit/UIKit hosts. Collapses to hosting views in pure SwiftUI. |
 | `pathname` mapping | Host-supplied route, inferred fallback | A native app has no URL routes; the host sets a route, else infer from the key window title or identifier. |
 | Overlay coverage | Primary screen (MVP) | The overlay covers the primary display; SwiftUI-local points map to AX screen coordinates there. Full multi-display placement is deferred (cli-a99qm.4.2). |
 | Dev-only gating | `#if DEBUG` default + env override | On in DEBUG unless `ANNOTKIT_DISABLE`; off in release unless `ANNOTKIT_ENABLE`. Mirrors VirgilHUD `InspectMode`. |
 | MCP bridge in v1 | Deferred to F6 (optional) | The file and clipboard sinks cover the agent loop; the MCP/HTTP bridge is an optional later target, not part of the 1.0 critical path. |
 | Concurrency | Swift 6 language mode, strict | Public `Element`/`CapturedImage`/`AnnotationNote` are `Sendable`; `ElementSource`/`Annotation` are `@MainActor`. |
 
+## Annotation target rule (cli-got28.2)
+
+A click must isolate the component the user meant and produce a selector that
+locates its code. One rule, both platforms:
+
+1. Hit-test to the deepest node at the point, then walk its ancestor chain.
+2. **Target = the deepest ACTIONABLE control** in the chain (button, link,
+   checkbox, popup, slider, menu item, or anything exposing `AXPress`). A click
+   anywhere inside a button binds to the button, not the static-text glyph that
+   happens to be its deepest descendant.
+3. Else **target = the deepest MEANINGFUL element** — one carrying an identifier,
+   a label, or a displayed value. A standalone `Text` inside a card is annotated
+   in its own right (the card is not actionable, so it does not swallow the
+   text); the selector engine then anchors it to the card's identifier.
+4. Never a target: the window, the application, window chrome (traffic lights),
+   or a structural, unidentified, content-less group that spans (nearly) the
+   whole window (an `NSHostingView` root `AXGroup` — the window in disguise).
+5. When the point hits nothing annotatable (decoration, dividers, padding beyond
+   any frame), a `RegionAnchorSource` anchors the click to the nearest meaningful
+   element as a REGION note rather than dropping it.
+
+Why not "prefer the deepest *identified* ancestor"? Because seeding is partial: a
+button may be seeded but a standalone text inside a seeded card is usually not.
+Preferring the identified container would collapse every click inside a card onto
+the card and lose the specific element. Preferring the actionable/meaningful leaf
+and letting the SELECTOR anchor to the nearest identifier (`#Card >> text="…"`)
+keeps both the specificity and the code-locating anchor. Selection *widening*
+(cli-got28.2.3) exists for the times the user does want the enclosing component.
+
+This replaces the earlier asymmetry (macOS "deepest meaningful", iOS "nearest
+identified") documented in `docs/spike-ax-pointquery.md`, now corrected. The pure
+decision lives in `AnnotationTargetRule` and is unit-tested independent of AX.
+
+### Component containment is GEOMETRIC, not tree-ancestry (cli-got28.2)
+
+"Which component encloses this element" is resolved by frame containment, not by
+walking the AX parent chain. A SwiftUI card seeded with `.axCardSurface` (the
+dominant VirgilHUD pattern) is a clear `Color.clear` background leaf carrying the
+identifier — and `.background` makes it a **sibling of the card's content, not an
+ancestor**. So the card's identifier never appears in the content's ancestor
+chain, and pure-ancestry anchoring/widening cannot reach it (it only worked for
+`.accessibilityElement(children: .contain)` containers, which *are* ancestors).
+
+The macOS `componentLadder` therefore collects every identified element whose
+frame **encloses the point** and is larger than the target, smallest-first —
+scanning the ancestor chain plus each ancestor's direct children (where those
+background surfaces live), so it reaches sibling card surfaces without a full
+snapshot. This ladder drives selection widening and the note's `component` field.
+Selector *anchoring* (`#Card >> …`) still requires a true ancestor because the
+`>>` operator is descendant-based; when the component is a sibling surface the
+selector may be positional or text-based while the `component` field still names
+the card, so the note locates the right code either way.
+
 ## IP hygiene (carried into the F7 legal gate)
 
 - Do not copy original Agentation source (PolyForm Shield 1.0.0, non-compete). Only the `AGENTATION_NOTES.md` file format is reused, reimplemented clean-room.

PARITY.md

@@ -10,7 +10,9 @@ row; each asymmetry is closed by code or has a tracked mitigation.
 | Opt-in element source | NSView tree | (the default already walks the view tree) | macOS adds a view-tree opt-in alongside its AX default; iOS only needs the one source |
 | `accessibilityIdentifier` | via AX | via `UIView.accessibilityIdentifier` | both surface SwiftUI identifiers |
 | Concrete view class name | via NSView opt-in source | via the default UIView walk | none |
-| Hit test | `AXUIElementCopyElementAtPosition` + NSView `hitTest` | `UIView.hitTest(_:with:)` | iOS has no global AX point query; uses view hitTest. Tracked: F5.2 |
+| Hit test primitive | `AXUIElementCopyElementAtPosition` + NSView `hitTest` | `UIView.hitTest(_:with:)` | iOS has no global AX point query; uses view hitTest. Tracked: F5.2 |
+| Annotation target rule | shared `AnnotationTargetRule` over an AX candidate chain | shared `AnnotationTargetRule` over a UIView candidate chain | none — both build a `[TargetCandidate]` chain and apply the SAME rule (deepest actionable, else deepest meaningful). Closes the earlier split (macOS "deepest meaningful" vs iOS "nearest identified"), cli-got28.2 |
+| Component widening | `ComponentLadderSource` (AX chain) | `ComponentLadderSource` (UIView chain) | none — same ladder (target, then enclosing identified components) |
 | Coordinate space | Cocoa bottom-left to AX top-left flip | UIKit top-left native | iOS needs no flip; shared `ScreenSpace` used only on macOS |
 | Screenshot | ScreenCaptureKit / `cacheDisplay` | `UIGraphicsImageRenderer` + `drawHierarchy` | both capture own hierarchy only; no cross-window or secure overlays |
 | Overlay host | resizing `NSPanel` (toolbar corner idle, full screen annotating) | pass-through `UIWindow` | both interactive; selection via the shared SwiftUI catcher, not a global monitor |

README.md

@@ -40,8 +40,13 @@ Annotation.install(sink: ClipboardSink(format: .json))
 
 - **macOS** queries the app's own accessibility tree (the only strategy that
   surfaces SwiftUI `accessibilityIdentifier` values); **iOS** walks the UIView
-  hierarchy. Both resolve a tap to the nearest identified element and generate a
-  selector that round-trips a resolver.
+  hierarchy. Both resolve a tap with the same rule: the **deepest actionable
+  control** wins (a click inside a button binds to the button, not its label
+  glyph), else the **deepest meaningful element** (a standalone text/label/value
+  leaf is annotated in its own right). The generated selector then **anchors** a
+  non-identified target to its nearest seeded `accessibilityIdentifier`
+  (`#Settings.Models >> @Save`), so it round-trips a resolver and points an agent
+  at the right component's code. See `DECISIONS.md`.
 - Notes are written in the `AGENTATION_NOTES.md` format that the
   `process-agentation-notes` skill consumes, or copied to the clipboard.
 

Sources/AnnotKit/AnnotationSink.swift

@@ -35,6 +35,23 @@ public struct AnnotationNote: Sendable, Hashable, Identifiable, Codable {
     public var selector: String
     /// The human-readable Element Path (ancestor chain).
     public var elementPath: String
+    /// The seeded component the note is scoped to: the deepest
+    /// `accessibilityIdentifier` in the target's ancestry (the target's own id
+    /// when it is seeded, else its nearest seeded ancestor). This is the string an
+    /// agent greps to find the component's Swift view; nil when nothing in the
+    /// ancestry is seeded. Optional and additive — old files decode with nil.
+    public var component: String?
+    /// The bound element's accessibility role (`AXButton`, `AXStaticText`, …), so
+    /// an agent knows what kind of view to look for. Optional/additive.
+    public var elementRole: String?
+    /// The bound element's displayed text/value, if any — a direct grep target
+    /// (e.g. the button title or the `Text` string). Optional/additive.
+    public var elementText: String?
+    /// True when the bound element has NO `accessibilityIdentifier` of its own, so
+    /// the selector had to anchor to an ancestor (``component``) or go positional.
+    /// It marks a miss the user can turn into a seeding task rather than a silent
+    /// misattribution. nil on notes captured before this field existed.
+    public var unseeded: Bool?
     public var selectedText: String?
     public var comment: String
     public var screenshot: CapturedImage?
@@ -61,6 +78,7 @@ public struct AnnotationNote: Sendable, Hashable, Identifiable, Codable {
     /// unchanged.
     private enum CodingKeys: String, CodingKey {
         case id, route, selector, elementPath, selectedText, comment, screenshot, timestamp
+        case component, elementRole, elementText, unseeded
         case regionOffset
     }
 
@@ -69,6 +87,10 @@ public struct AnnotationNote: Sendable, Hashable, Identifiable, Codable {
         route: String? = nil,
         selector: String,
         elementPath: String,
+        component: String? = nil,
+        elementRole: String? = nil,
+        elementText: String? = nil,
+        unseeded: Bool? = nil,
         selectedText: String? = nil,
         comment: String,
         screenshot: CapturedImage? = nil,
@@ -80,6 +102,10 @@ public struct AnnotationNote: Sendable, Hashable, Identifiable, Codable {
         self.route = route
         self.selector = selector
         self.elementPath = elementPath
+        self.component = component
+        self.elementRole = elementRole
+        self.elementText = elementText
+        self.unseeded = unseeded
         self.selectedText = selectedText
         self.comment = comment
         self.screenshot = screenshot

Sources/AnnotKit/AnnotationTargetRule.swift

@@ -0,0 +1,98 @@
+import Foundation
+
+/// The annotation-relevant facts about one node in a hit's ancestor chain, read
+/// once by a platform adapter so the target rule stays pure — testable without
+/// the AX API or UIKit. Field names mirror ``SelectorMatchable`` where they
+/// overlap.
+public struct TargetCandidate: Sendable, Hashable {
+    public let role: String
+    public let identifier: String
+    public let label: String
+    public let value: String
+    /// Exposes a press/edit action, or is one of the actionable roles.
+    public let isActionable: Bool
+    /// Window chrome (the traffic lights): a real, actionable button, but the
+    /// WINDOW's chrome, not app content — its selector locates no app code, so it
+    /// is never an annotation target.
+    public let isChrome: Bool
+    /// The window or application container. A background click must never resolve
+    /// to the whole window/app.
+    public let isContainerRoot: Bool
+    /// A structural, unidentified, content-less group spanning (nearly) the whole
+    /// window — an `NSHostingView` root `AXGroup`, the window in disguise.
+    public let isWindowGhost: Bool
+
+    public init(
+        role: String,
+        identifier: String = "",
+        label: String = "",
+        value: String = "",
+        isActionable: Bool = false,
+        isChrome: Bool = false,
+        isContainerRoot: Bool = false,
+        isWindowGhost: Bool = false
+    ) {
+        self.role = role
+        self.identifier = identifier
+        self.label = label
+        self.value = value
+        self.isActionable = isActionable
+        self.isChrome = isChrome
+        self.isContainerRoot = isContainerRoot
+        self.isWindowGhost = isWindowGhost
+    }
+
+    /// An actionable control eligible to be a target (not chrome, not the
+    /// window/app root).
+    var isEligibleActionable: Bool {
+        isActionable && !isChrome && !isContainerRoot
+    }
+
+    /// A meaningful element eligible to be a target: it carries an identifier, a
+    /// label, a displayed value, or an action, and is not chrome, a container
+    /// root, or a window-spanning ghost group.
+    var isEligibleMeaningful: Bool {
+        guard !isChrome, !isContainerRoot, !isWindowGhost else { return false }
+        return !identifier.isEmpty || !label.isEmpty || !value.isEmpty || isActionable
+    }
+}
+
+/// The unified annotation target rule (see DECISIONS.md → "Annotation target
+/// rule"). Given the hit's ancestor chain, pick which node the annotation binds
+/// to. Pure and platform-independent: both the macOS AX adapter and the iOS
+/// view-tree adapter build a `[TargetCandidate]` chain and call ``targetIndex``,
+/// so they resolve a click identically and the rule is unit-tested on its own.
+public enum AnnotationTargetRule {
+    /// Index in the root-first `chain` of the annotation target, or nil when
+    /// nothing at the point is annotatable.
+    ///
+    /// The deepest ACTIONABLE control wins — a click inside a button binds to the
+    /// button, not the static-text glyph that is its deepest descendant. Failing
+    /// that, the deepest MEANINGFUL element — a standalone text/label/value leaf
+    /// is annotated in its own right, and the selector engine anchors it to its
+    /// nearest identified ancestor.
+    public static func targetIndex(in chain: [TargetCandidate]) -> Int? {
+        for i in chain.indices.reversed() where chain[i].isEligibleActionable {
+            return i
+        }
+        for i in chain.indices.reversed() where chain[i].isEligibleMeaningful {
+            return i
+        }
+        return nil
+    }
+
+    /// The component-widening ladder for `chain`: the target itself first, then
+    /// each strictly-shallower identified ancestor, broadest last. Drives
+    /// selection widening (a click-again steps up to the enclosing component).
+    /// Empty when nothing is annotatable.
+    public static func wideningLadder(in chain: [TargetCandidate]) -> [Int] {
+        guard let target = targetIndex(in: chain) else { return [] }
+        var ladder = [target]
+        for i in stride(from: target - 1, through: 0, by: -1) {
+            let c = chain[i]
+            if c.isContainerRoot { break } // stop at the window; it is never a target
+            if !c.identifier.isEmpty, !c.isWindowGhost { ladder.append(i) }
+        }
+        return ladder
+    }
+}

Sources/AnnotKit/ElementSource.swift

@@ -19,9 +19,14 @@ public protocol ElementSource {
     /// Snapshot every inspectable window and its element tree.
     func snapshot() -> [WindowSnapshot]
 
-    /// Resolve a screen point (AX top-left coordinates) to the deepest element
-    /// under it, walked up to the nearest ancestor that carries a stable
-    /// identity. Returns nil when nothing resolves.
+    /// Resolve a screen point (AX top-left coordinates) to the annotation target
+    /// under it, per the unified target rule (see DECISIONS.md): the deepest
+    /// ACTIONABLE control at the point, else the deepest MEANINGFUL element
+    /// (identifier / label / value / action), never the window, application,
+    /// chrome, or a structural window-spanning group. Returns nil when nothing
+    /// resolves (a `RegionAnchorSource` can then anchor the click to a nearby
+    /// element instead of dropping it). The returned element's ``Element/path``
+    /// carries its identified ancestors, which the selector engine anchors to.
     func hitTest(_ point: CGPoint) -> Element?
 
     /// Generate the most-stable, round-tripping selector string for an element.
@@ -41,3 +46,15 @@ public protocol ElementSource {
 public protocol RegionAnchorSource {
     func regionAnchor(at point: CGPoint) -> Element?
 }
+
+/// Optional element-source capability: the component-widening ladder at a point —
+/// the annotation target first (same element ``ElementSource/hitTest(_:)``
+/// returns), then each enclosing identified component, broadest last. Lets the
+/// session step a selection UP to a parent component for a coarser-grained note
+/// (a click-again / widen affordance while the composer is open) without
+/// re-hit-testing. Sources that cannot offer it simply don't conform; the session
+/// then disables widening.
+@MainActor
+public protocol ComponentLadderSource {
+    func componentLadder(at point: CGPoint) -> [Element]
+}