Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions .abcd/development/intents/drafts/itd-6-terminal-capture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
id: itd-6
slug: terminal-capture
spec_id: null
kind: null
suggested_kind: null
reclassification_history: []
builds_on: []
severity: major
---

# Think Aloud While You Work at the Terminal

## Press Release

> **Testimony captures think-aloud sessions against a command-line tool.** A terminal session recorded with asciinema arrives as a `.cast` stream — a timestamped record of every command typed and every line of output — which `merge` normalises onto the same session clock as the narration, exactly as it does a browser's interaction events. A tester's spoken "I have no idea what that error is telling me" lands next to the command that produced it and the text it printed, and the anchor greps straight back to the source that emitted it.
>
> "Half of what we build is command-line, and it was the half I could never study properly," said Bob, application developer. "I'd watch someone stumble over a flag and then have nothing to show for it but a memory. Now the stumble, the command, and their exact words all sit on one line of the timeline."

## Why This Matters

The pipeline's evidence model assumes an interaction stream that can be joined to speech and then resolved to code. For browser targets rrweb supplies it; for a terminal, asciinema's `.cast` v2 format already *is* one — a timestamped event stream needing no new instrumentation in the tool under test. Without it, CLI sessions degrade to the keyframe fallback, which is the weakest and most expensive evidence channel, for the target type that would grep back to source most reliably of all.

This is also a dependency hole in the recorded plan. The codebase-mapping intent's acceptance criteria already assume a cast stream exists — "**Given** a finding from a CLI session… the anchor is resolved from the command or output text in the cast stream" — while `record`'s intent explicitly fences asciinema wrappers out of scope. This intent is the capability that criterion rests on, and Testimony being a CLI itself makes it the first tool available to study.

## What's In Scope

- Capturing a terminal session as an asciinema `.cast` v2 stream alongside the audio narration, anchored to the session `t0`.
- Normalising `.cast` records into `interactions.jsonl` so `merge` and `report` consume them unchanged — the timeline schema does not learn a new source type.
- An interaction `kind` vocabulary for the terminal: at minimum the command entered and the output emitted, carrying the text that later serves as the mapping anchor.
- Documenting the terminal target in the session-directory reference alongside the existing browser path.

## What's Out of Scope

- Instrumenting the tool under test — asciinema records the terminal, not the program, and that is the point.
- Resolving cast-stream anchors to source locations; that is the codebase-mapping step (itd-3), which this intent unblocks rather than performs.
- Native GUI application capture (itd-7) and third-party website capture (itd-4).
- Replaying a `.cast` stream as video; the stream is evidence and analysis input, not a playback surface.

## Acceptance Criteria

- **Given** a terminal session recorded with asciinema alongside narration, **when** `merge` runs, **then** `timeline.jsonl` interleaves the spoken utterances and the cast stream's commands and output on one session-relative clock.
- **Given** a merged terminal session, **when** `report` runs, **then** each utterance renders with the commands and output that fall inside its join window.
- **Given** a cast record and a spoken utterance at the same instant, **when** the timeline is built, **then** both carry session-relative times derived from the same `t0`, with no separate clock for the terminal stream.

## Open Questions

- Does asciinema's own clock need the spoken-marker correction, or is starting it from `record` sufficient to anchor it to `t0`?
- Should output be captured verbatim or truncated per record? Full output makes a better mapping anchor but a much larger `interactions.jsonl`, and a single command can emit more than the JSONL line limit.
- Does a TUI (as opposed to a line-oriented CLI) produce a usable cast stream, or does redraw traffic swamp the signal?

## Audit Notes

_Empty. Populated by intent-fidelity-reviewer when intent moves to shipped/._
54 changes: 54 additions & 0 deletions .abcd/development/intents/drafts/itd-7-native-app-capture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
id: itd-7
slug: native-app-capture
spec_id: null
kind: null
suggested_kind: null
reclassification_history: []
builds_on: []
severity: minor
---

# Study the Mac and iPhone Apps, Not Just the Web Ones

## Press Release

> **Testimony captures think-aloud sessions against native macOS and iOS applications.** Screen capture comes from ScreenCaptureKit on the Mac and from `simctl` on the iOS simulator or a cabled device, with narration recorded on the same clock as always. Native apps expose no interaction stream, so the transcript carries the semantic load and keyframes extracted at utterance timestamps supply the referents — the same fallback channel reference capture already relies on. Later, a debug build can log its own events and upgrade a native session to the full evidence model.
>
> "The Mac app is the one people actually touch, and it was the one I had the least evidence about," said Alice, usability researcher. "Now a native session produces the same report as a browser session — a bit less precise about which control they hit, but their words and the screen at that second are there."

## Why This Matters

macOS is the primary target and a macOS app layer is planned, so the tool would otherwise be unable to study the platform it is built for. Native capture is deliberately the weaker evidence story: without instrumentation there is no selector, no route, and no event to join against, so the join narrows to "what was on screen when they said this". Recording that honestly — as a keyframe-anchored session that yields findings but not `code_refs` — is better than either pretending native sessions are equivalent to instrumented ones or leaving the platform unstudied.

Separating this from terminal capture (itd-6) keeps the two evidence models from being conflated: a `.cast` stream is a real interaction stream that greps to source, while a native session in phase 1 is video plus voice. They deserve different acceptance bars.

## What's In Scope

- macOS capture via ScreenCaptureKit, and iOS capture via `xcrun simctl io booted recordVideo` for the simulator and a cabled-device path.
- Keyframe extraction at utterance timestamps as the referent channel, reusing the mechanism reference capture defines rather than inventing a second one.
- A session that merges and reports with no interaction stream present — a transcript-only timeline is a first-class outcome, not a degraded error.
- Recording in the manifest that a session is keyframe-anchored, so downstream steps and readers know which evidence channel backs a finding.

## What's Out of Scope

- Debug-build event instrumentation (`NSApplication.sendEvent` / `UIWindow.sendEvent` overrides, gesture-recogniser logging) — noted as the later upgrade path, not built here.
- Codebase mapping from native sessions; without accessibility identifiers there is no reliable anchor, and itd-3 already fences this out.
- Capturing third-party native applications, which is reference capture (itd-4) under a different rubric and different copyright footing.
- The macOS app layer that wraps the CLI core — a delivery surface, not a capture target.

## Acceptance Criteria

- **Given** a macOS application session recorded with narration and no interaction stream, **when** `merge` and `report` run, **then** a transcript-only timeline and report are produced without error.
- **Given** an utterance whose referent is ambiguous in text alone, **when** a keyframe is requested at that timestamp, **then** a frame extracted from the session recording is available as that finding's evidence.
- **Given** a keyframe-anchored session, **when** findings are produced, **then** each records that its evidence channel is a keyframe rather than an interaction event.

## Open Questions

- Is the keyframe path sufficient in practice, or does the absence of an event stream make native findings too vague to act on? This is the open question the architecture note poses, and only real sessions will answer it.
- Does iOS device capture over a cable share a clock with the Mac reliably enough for the join window, or does it need the spoken-marker correction?
- Should accessibility identifiers, where an app already sets them, be harvested as a partial anchor — a middle path between no instrumentation and a debug build?

## Audit Notes

_Empty. Populated by intent-fidelity-reviewer when intent moves to shipped/._
54 changes: 54 additions & 0 deletions .abcd/development/intents/drafts/itd-8-local-analysis.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
id: itd-8
slug: local-analysis
spec_id: null
kind: null
suggested_kind: null
reclassification_history: []
builds_on: []
severity: major
---

# Analysis That Never Leaves the Machine

## Press Release

> **Testimony can run its analysis pass entirely on local hardware.** A flag on the analysis surface points the same versioned rubric at a locally hosted model instead of a cloud one, producing the same `findings.jsonl` — same schema, same typing, same `status: unverified` birth state — with no session text leaving the machine. Voice and screen were already local; with this, so is every derived artefact, and the privacy boundary moves from "only derived text leaves" to "nothing leaves".
>
> "The ethics application asks where the data goes, and 'nowhere' is a much shorter answer than a paragraph about derived text," said Carol, session moderator. "Being able to say the whole pipeline runs on one machine in a locked office is what got the protocol through."

## Why This Matters

The local-processing boundary is the pipeline's strongest privacy property and, as the ethics constraints record, the strongest card in a research-ethics application. Today that boundary sits between raw media and derived text: audio and video stay put, but transcript and events reach a cloud model. For most sessions that is a defensible line. For a protocol that will not permit it — or a participant population where it cannot be justified — the whole pipeline becomes unusable at the last step, after all the local capture and transcription work is already done.

Making this a flag on the existing surface rather than a separate mode is deliberate. One analysis code path, one rubric, one findings schema means a local-analysed session is comparable with a cloud-analysed one, and the retained human verdicts measure both against the same bar. The operator chooses per session; the artefacts do not fork.

## What's In Scope

- A flag on the analysis surface selecting a locally hosted model backend, with the cloud path remaining the default.
- Identical output contract: the same rubric version, the same findings schema, and `status: unverified` on every machine-generated finding regardless of backend.
- Recording which backend and model produced a session's findings, so the provenance of a verdict is auditable and the two backends' precision can be compared over time.
- A stated, documented quality floor below which a local backend is not fit for the second-coder role.

## What's Out of Scope

- Local transcription, which is already local by design and is not part of this choice.
- Shipping, vendoring, or managing a local model — the flag points at a backend the operator has already stood up.
- Guaranteeing parity of finding quality between backends; the retained verdicts measure the gap rather than hiding it.
- A local backend for the codebase-mapping step (itd-3), which is a separate agentic surface.

## Acceptance Criteria

- **Given** a merged timeline and a local backend configured, **when** the analysis pass runs with the local flag, **then** `findings.jsonl` is produced under the same rubric and schema as a cloud run, with no session text sent off the machine.
- **Given** a findings file produced by either backend, **when** it is read, **then** the backend and model that produced it are recorded alongside the findings.
- **Given** the local flag is not passed, **when** the analysis pass runs, **then** behaviour is unchanged from today.

## Open Questions

- What is the acceptable quality floor for a local model in the second-coder role, and how is it measured — retained verdict precision against a cloud-analysed control set?
- Can a local model sustain the two-pass structure (segment coding then session-level synthesis), or does synthesis degrade first and need a smaller chunk size?
- Does the local path need its own rubric phrasing to survive a smaller context window, and if so, does that break comparability with cloud runs?

## Audit Notes

_Empty. Populated by intent-fidelity-reviewer when intent moves to shipped/._
54 changes: 54 additions & 0 deletions .abcd/development/intents/drafts/itd-9-regression-test-drafting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
id: itd-9
slug: regression-test-drafting
spec_id: null
kind: null
suggested_kind: null
reclassification_history: []
builds_on: []
severity: minor
---

# A Confirmed Bug Arrives With Its Test Already Written

## Press Release

> **Testimony drafts a regression test case from every confirmed finding.** Once a human has flipped a finding to `confirmed`, the drafting step turns it into a proposed test case: the steps reconstructed from the event window, the behaviour the participant expected, the behaviour they got, and their own words as the rationale. The draft goes to a person to accept, edit, or reject — nothing is written into a suite automatically. A session stops being a report that ages and becomes a test that keeps the bug from coming back.
>
> "A finding tells me something broke once; a test tells me it stays fixed," said Bob, application developer. "Getting the steps and the expected behaviour handed to me in the participant's own framing is most of the work of writing the test, and it's the part I always got subtly wrong from memory."

## Why This Matters

The pipeline's value decays at the end: a confirmed finding becomes an issue, the issue gets fixed, and the evidence that motivated it is never exercised again. Meanwhile the reproduction steps — which are the expensive, error-prone part of writing a regression test — are already sitting in the timeline as a precise event window with a spoken account of what the person expected. Drafting the test while that evidence is intact converts a one-off observation into a standing guarantee.

Keeping the human in the loop mirrors the stance the rest of the pipeline takes. Findings are unverified until a person confirms them; a test drafted from a finding is likewise a proposal, not a commit. The step is deliberately downstream of verification, so only evidence a human already vouched for can become a test.

## What's In Scope

- Drafting a test case from a `confirmed` finding: reproduction steps derived from the finding's event window, expected versus observed behaviour, and the participant quote as rationale.
- A human accept / edit / reject pass over each drafted test, with the decision retained as the verdicts already are.
- Linking the drafted test back to its source finding and session, so a failing test leads to the evidence that motivated it.
- Emitting the draft in a form the docs-as-code manual test records can hold, so a session becomes evidence attached to a test run.

## What's Out of Scope

- Generating executable test code for the application under test; that needs its test framework and conventions, and is a further step.
- Running, filing, or committing tests automatically — the output is a draft for review.
- Drafting from unverified or rejected findings; only human-confirmed findings are eligible.
- Reference-capture findings (itd-4), which are design preferences rather than defects and have nothing to regress against.

## Acceptance Criteria

- **Given** a finding whose status is `confirmed`, **when** the drafting step runs, **then** a test case draft is produced containing reproduction steps from the event window, the expected and observed behaviour, and the participant's quote.
- **Given** a finding whose status is `unverified` or `rejected`, **when** the drafting step runs, **then** no test case is drafted for it.
- **Given** a drafted test case, **when** a human accepts or rejects it, **then** the decision is retained and the draft remains linked to its source finding and session.

## Open Questions

- Does the event window alone yield reproduction steps a developer can follow, or does a reliable repro need the keyframe channel as well?
- Where do accepted drafts live — the application's own repository, the docs-as-code test plan, or alongside the session? The architecture note leaves this open.
- Should a drafted test carry the finding's severity through, so triage order survives the hand-off?

## Audit Notes

_Empty. Populated by intent-fidelity-reviewer when intent moves to shipped/._
Loading