Spike: captured device transcripts (Tier A) and a stateful virtual device (Tier B) for e2e-shaped testing without hardware

[thymikee]

Follow-up to the oracle-negation spike (docs/agents/oracle-negation-spike.md) and the fake-provider test conversions on #1679. Parking the design here — no scheduled owner yet; this is the "someday" track for running e2e-shaped tests in environments with no adb/xcrun (cloud sandboxes, fast PR lanes).

Problem

The provider-scenario lane runs the real daemon → router → handlers against fake providers, but every fixture in it is a hand-authored belief about what simctl/adb/the XCUITest runner actually print. Nothing detects drift when an OS or Xcode release changes an output shape. Meanwhile the real-device lanes (Android emulator on ubuntu-latest, iOS simulator on macOS runners) produce authentic outputs nightly — and throw them away.

Tier A — record/replay transcripts with provenance (small; do first)

The provider surfaces are already narrow JSON boundaries (AppleRunnerProvider.runCommand, AppleToolProvider, AndroidAdbProvider), and createRecordingAppleToolProvider in test/integration/provider-scenarios/providers.ts is named for exactly this.

• Add a capture mode to the live lanes: wrap the real providers with recorders during the nightly iOS/Android e2e scenarios; serialize (command, request, response) transcripts.
• Store under test/fixtures/transcripts/<scenario>.json with provenance: OS version, Xcode/SDK version, device model, commit, capture date — the same shape as the replay-compat corpus and scripts/check-replay-compat-provenance.ts.
• A cloud lane replays frozen transcripts through the full daemon (the existing provider-scenario harness already does this mechanically via createProviderTranscript).
• Drift check: nightly re-capture diffs against the committed transcript; a diff fails the nightly, never the PR gate.
• Cheap extension once transcripts exist: fault-injection sweeps — "inject failure at call N" per recorded scenario exercises the errorTaxonomy guarantee across real command sequences.

Tier B — stateful virtual device (the differentiator)

Transcripts replay only recorded sequences. To run arbitrary replay scripts (ad test examples/test-app/replays) without hardware, promote the scenario worlds (android-world.ts already carries shellState/appState) into a reusable fake device:

• Screens = captured snapshot trees (sourced from Tier A captures, so fidelity is inherited, not hand-maintained).
• A transition table: tap/fill/open/back move between screens; interactions mutate state the next snapshot reflects.
• Registered as a device the daemon can resolve (e.g. --platform android --udid virtual-…), so the public CLI drives it end-to-end.
• Payoffs: full replay corpus on every PR in seconds; agents can exercise app flows in cloud sandboxes where emulators can't run (no KVM); deterministic e2e-shaped tests for features that today only the nightly hardware lanes cover.

Sequencing

1. Tier A capture on one scenario per platform + provenance check (unblocks everything else).
2. Frozen-transcript replay lane in CI.
3. Nightly drift diff.
4. Tier B virtual device seeded from captured screens; start with the fixture app's main flows.

Related

• test: measure oracle liveness suite-wide; pin the one dead-path oracle #1679 — fake-provider conversions this builds on (withFakeAdb, withFakeAppleTool)
• docs/agents/oracle-negation-spike.md — follow-ups list, item 3
• ADR 0001 (provider-first integration scenarios) — this is that direction extended to captured rather than authored fixtures