Infrastructure workstream

[JasonVMo]

Summary

Make desktop UI behavior of the agentic component library provable rather than
merely demonstrable, and make its package-cost claims measurable. Deliver one
portable desktop test driver, deterministic Storybook end-to-end pipelines, a
Win32 Paper Storybook endpoint, and bundle-size and dependency regression gates
for agentic components and theming.

The completed workstream provides:

1. a single supported desktop test driver that runs one unchanged spec on multiple
   desktop endpoints with explicit application ownership and truthful reporting;
2. Storybook-driven end-to-end pipelines that run locally and in CI, with a clear
   separation between deterministic pass/fail gates and optional visual evidence;
   and
3. a Win32 (Paper) Storybook endpoint that coexists with the existing macOS and
   Windows Fabric endpoints without destabilizing either; and
4. repeatable consumer-bundle and package-dependency budgets that prove optional
   theming code stays optional and agentic components preserve their declared
   source boundary.

Observed current baseline

Storybook application

Observed. apps/storybook is an on-device Storybook
for React Native v10 app running in liteMode, loading stories from
packages/agentic/components/src/**/*.stories.?(ts|tsx) and
packages/native/Callout/src/**/*.stories.?(ts|tsx)
(apps/storybook/src/main.ts). There are 23
*.stories.tsx files under packages/agentic/components/src plus
packages/native/Callout/src/Callout.stories.tsx.

Observed. Two native endpoints exist, both hosted by react-native-test-app:
macOS (react-native-macos, CocoaPods, pods:macos / macos:build) and Windows
(react-native-windows 0.81 New Architecture / Fabric, generated with
install-windows-test-app --use-fabric). Both are declared in
apps/storybook/react-native.config.js
and apps/storybook/app.json, whose resources
list only macos and windows bundles.

Observed. Agent/automation surface already exists:
storybook-server.cjs exposes a
WebSocket channel, an MCP endpoint, and REST control endpoints (GET /index.json,
POST /select-story-sync/<storyId>, POST /send-event) on 127.0.0.1:7007;
scripts/storybook-control.cjs
wraps them (list, select, args, smoke).

Observed. The only automated on-device assertion suite today is Windows-only:
windows-tests/storybook-smoke.test.cjs
run by jest.windows.config.cjs
under testEnvironment: '@react-native-windows/automation' with WinAppDriver. It
covers the three stories listed in
scripts/smoke-stories.json
(components-button--default, primitives-icon--default,
primitives-callout--default) with testID visibility and status-string
assertions, and - since
4777e54f
(PR #4208) - a
focus-movement test on components-button--overview plus ten
"focuses ... without terminating the app" regression cases (nine parameterized
across Tag, Accordion, Tab, ListboxItem, Checkbox, MenuItem, ListItem, Radio, and
Switch, plus the interactive Card). It writes JSON artifacts under the ignored
artifacts/windows path.
Observed: there is no equivalent macOS harness in apps/storybook; the
scripts/ directory contains only Windows PowerShell session scripts and the
Storybook control CLIs.

Existing E2E infrastructure

Observed. apps/E2E is an Appium + WebdriverIO + Jasmine
suite for the legacy FluentTester apps, with one WDIO config per platform
(wdio.conf.android.js, wdio.conf.ios.js, wdio.conf.macos.js,
wdio.conf.win32.js, wdio.conf.windows.js), Appium driver install scripts
(e2eprep:<platform>), and run scripts (e2etest:<platform>). Selector convention
is testID on Win32/Windows/macOS/iOS and accessibilityLabel on Android
(apps/E2E/README.md). Failure screenshots go to
errorShots. It targets apps/fluent-tester and apps/win32, not apps/storybook.

Observed. apps/E2E/wdio.conf.win32.js
automates the Win32 Paper app by pointing the Appium windows driver at
@office-iss/rex-win32's ReactTest.exe with
appium:appArguments of the form
basePath <apps/win32/dist> plugin defaultplugin bundle index.win32 component FluentTester.
This is direct precedent that a rex-win32-hosted app is automatable with the same
driver stack the Storybook Windows harness uses.

CI

Observed. .github/workflows/pr.yml defines
JS PR (ubuntu-latest), Android PR (ubuntu-latest), macOS PR (macos-26),
iOS PR (macos-26), Windows PR (windows-latest), Win32 PR
(windows-latest), plus changeset, pack, and link-check jobs. E2E runs today in
macOS PR (yarn e2eprep:macos / yarn e2etest:macos), Windows PR
(launchE2EWindows.ps1 -enableDumpFiles), and Win32 PR (yarn e2eprep:win32 /
yarn e2etest:win32). Android and iOS E2E steps are present but commented out.
Windows and Win32 jobs install WinAppDriver 1.1 via msiexec. Failure artifacts are
uploaded with actions/upload-artifact from apps/E2E/reports and
apps/E2E/errorShots.

Observed. No workflow under .github/workflows/ or .ado/ references
apps/storybook at all: grep -rn "storybook" .github/workflows/ .ado/ returns no
matches. .ado/azure-pipelines.publish.yml is build-pack + ESRP publish on the
Azure-Pipelines-1ESPT-ExDShared pool and runs no UI tests.

Observed. No visual-regression service (Percy, Chromatic, image diffing) exists
anywhere in the repository. Screenshots are captured only as failure evidence.

Test driver branch

Observed. origin/user/jasonvmo/test-driver (tip
8f971021)
adds @fluentui-react-native/desktop-driver at packages/agentic/test-driver:
89 files, +12,361 / -28 lines against merge base
29f00b32.
origin/main is at
51fab45c,
so the branch is behind main and still uses the pre-repath
packages/agentic-components layout. Details are in
tasks/test-driver.md.

Observed - main has moved past a branch blocker. The branch's
NEXT-STEPS.md still names an unresolved agentic Button crash as blocking the
shared Windows Storybook suite. That text is a historical branch observation.
origin/main now contains
4777e54f
"Fix windows crash in components"
(PR #4208), which
adds the FocusVisual primitive
(packages/agentic/components/src/primitives/focus-visual)
and migrates eleven components onto it. Inferred: rebasing the branch onto
current main therefore integrates the fix rather than requiring a new diagnosis or
defect-filing task; what remains is a Windows rerun proving the unchanged suite is
unblocked.

Win32

Observed. @office-iss/react-native-win32 is present in the workspace at
resolved version 0.81.8 and is consumed by
apps/win32 (@fluentui-react-native/tester-win32), which
bundles with rnx-cli bundle to dist/index.win32.bundle and launches through the
prebuilt @office-iss/rex-win32@0.81.1 host (ReactTest.exe). react-native-test-app
5.3.3 as resolved in this workspace ships android, common, example, ios,
macos, visionos, and windows directories and no win32 directory, so the
Storybook app's current host cannot produce a Win32 endpoint. Details are in
tasks/win32-storybook.md.

Beta versus production

This workstream uses the following definitions for beta and production.

Beta (this workstream) means: a contributor or an agent can run a deterministic
desktop story-test suite on the supported endpoints, on demand, and a required CI
job proves at least one desktop endpoint stays green on every pull request.
Coverage may be narrow, one platform may be manual, and visual evidence may be
entirely manual, but every green result must be truthful - no fake-backend pass
can be mistaken for a real run, and no post-readiness crash can report success.
The component and design packages also have reproducible package-cost baselines,
advisory pull-request comparisons, and enforced dependency and optional-module
reachability rules.

Production (this workstream) means: every supported desktop endpoint
(macOS, Windows Fabric, Win32 Paper) has a required, reproducible CI gate with
published artifacts, a documented flake and quarantine policy, a documented
prerequisite and compatibility matrix, a published and externally consumable driver
package, and an owner for the interactive-desktop runner capacity the gates depend
on. Component and theming bundle-size budgets are also required gates, with
reviewed baseline updates and diagnostics that identify the modules behind a
regression.

Staged roadmap

Stage	Goal	Tasks
Stage 0 - Decisions	Approve driver and Win32 boundaries, CI rollout, legacy-harness disposition, bundle scenarios, metrics, and budget-update policy	Decision sections in all eight tasks, especially Smoke Harness Disposition
Stage 1 - Foundations	Land the driver and establish bundle-size fixtures and baselines before component or theming changes need to prove their impact	test-driver, bundle-size-and-dependency-budgets phase 1
Stage 2 - Beta	Prove the driver on real hardware, phase Storybook E2E from local to CI, add the Win32 endpoint, accessibility assertions, visual-capture evaluation, and advisory package-cost comparisons	test-driver-release-readiness, storybook-e2e phases 0-2, win32-storybook phases 1-2, bundle-size-and-dependency-budgets phase 2, story-accessibility-assertions, Stage 2 evaluation in windows-composition-screenshot
Stage 3 - Production	Require all endpoint, accessibility, package-cost, and artifact gates; resolve legacy harness ownership; assign ongoing owners	storybook-e2e phases 3-4, win32-storybook phase 3, release gate in test-driver-release-readiness, bundle-size-and-dependency-budgets phase 3, smoke-harness-disposition, windows-composition-screenshot, story-accessibility-assertions

Tasks

Task	Stage	Summary
Land the desktop test driver	Stage 1 (beta requirement)	Rebase, path-migrate, and merge @fluentui-react-native/desktop-driver from origin/user/jasonvmo/test-driver onto main: package layout, public API surface, backend selection, WDIO integration, Storybook manifest generation and loopback run service, on-device controls, and repository build/lint/test wiring.
Prove the desktop driver on real platforms and pass its release gate	Stage 2-3 (beta and production)	Execute the branch's own Priority 0 verification on macOS and Windows, complete the Priority 1 reporting and artifact-integrity work so no failure surface can report a false green, rerun the unchanged shared Windows suite on top of the already-landed FocusVisual fix, and satisfy the seven-item release gate.
Add Storybook E2E test pipelines	Stage 2-3 (beta and production)	Turn the Storybook app into an end-to-end tested surface: story-test authoring conventions, a deterministic local suite per endpoint, phased CI jobs starting with a non-interactive gate and escalating to interactive desktop gates, artifact publication, flake and quarantine policy, and an explicit boundary between deterministic gates and optional visual evidence.
Add a Win32 Paper Storybook endpoint	Stage 2-3 (beta and production)	Add an @office-iss/react-native-win32 (Paper) endpoint to apps/storybook hosted by @office-iss/rex-win32, reusing the existing story set, without disturbing the macOS and Windows Fabric endpoints, and wire it into the desktop driver and CI.
Add bundle-size and dependency regression gates	Stage 1-3 (beta and production)	Establish deterministic consumer-bundle scenarios and package dependency budgets for @fluentui-react-native/components and @fluentui-react-native/design, record current baselines, prove optional theming code is excluded unless imported, and fail CI when an unapproved dependency or size regression exceeds its checked-in budget.
Decide the disposition of the Windows Jest smoke harness	Stage 0 and Stage 3	Define parity, choose retirement, permanent retention, or conditional retention with expiry, preserve all focus-crash regression coverage, and resolve the WinAppDriver version, port, command, and CI ownership conflicts.
Deliver a reliable Windows Composition screenshot path	Stage 2-3 (beta and production)	Evaluate real WinAppDriver capture fidelity, add a window-scoped host fallback only if required, detect locked sessions, and publish redacted screenshots as evidence-only artifacts with explicit capture provenance.
Add accessibility assertions to Storybook story tests	Stage 2-3 (beta and production)	Extend the portable story-test schema and desktop driver with normalized role, name, state, and keyboard-reachability assertions on Windows and macOS, including fake-backend coverage, report fields, and an initial component cohort.

Dependency and CI rollout ordering

1. Record package-cost baselines before changing the packages.
   bundle-size-and-dependency-budgets
   phase 1 can start independently and should land before the component catalog
   grows or the theming consolidation changes module reachability. Later
   component and theming tasks supply the before-and-after comparisons.
2. Land the driver first. test-driver has no
   dependency on the bundle task. Storybook E2E and the Win32 endpoint consume
   its config factory, manifest format, and artifact contract. Landing it behind
   the existing
   @react-native-windows/automation smoke harness (rather than replacing it)
   keeps the current Windows validation path intact.
3. Local determinism before CI. storybook-e2e
   phase 0-1 (fake backend in the existing JS PR job, then a documented local
   real-platform run) must pass before any interactive desktop CI job is proposed,
   because a fake-backend pass is explicitly not proof of native behavior.
4. Real-platform proof before required gates.
   test-driver-release-readiness
   Priority 0 and Priority 1 gate the promotion of any desktop job from
   non-required to required. Inferred: without the Priority 1 reporting work,
   a required job could report green after an application crash.
5. Windows Fabric CI before Win32 CI. The Windows Fabric endpoint already has
   a working generate/build/register/launch chain in apps/storybook; the Win32
   endpoint does not exist yet. Prove the CI shape once on Windows Fabric, then
   reuse it for Win32, which is Inferred to be cheaper in CI because the
   rex-win32 host is a prebuilt binary and needs no MSBuild step.
6. macOS gate last or in parallel with Win32. macOS has never been verified
   for the driver (Observed in the branch's NEXT-STEPS.md), and its
   prerequisites - Xcode, WebDriverAgentMac build cache, accessibility permission,
   automation mode, logged-in GUI session - are the least proven on hosted runners.
7. Accessibility schema before interactive rollout.
   story-accessibility-assertions
   depends on the driver and Storybook manifest, but its schema, fake backend,
   and normalization contract should land before Phase 2 interactive CI so new
   stories carry accessibility assertions from their first real-platform run.
8. Parity before smoke-harness retirement.
   smoke-harness-disposition can record
   the decision at Stage 0, but the Jest path cannot be removed until the
   portable suite preserves all focus-crash coverage and reports application
   termination truthfully.
9. Evaluate native screenshots before building a fallback.
   windows-composition-screenshot
   shares the real Windows session used by release-readiness verification. A
   host capture path is implemented only if that evaluation proves WinAppDriver
   output inadequate.

Beta exit criteria

• @fluentui-react-native/desktop-driver is merged to main, builds under the
  unified tsc -b graph, and its package tests run in yarn lage test.
• apps/storybook exposes declared scripts to generate the story-test manifest
  and run it against the fake backend and at least one real desktop endpoint.
• At least three agentic components beyond Button declare story tests
  (inline plan or linked spec) and pass on at least one real desktop endpoint.
• One CI job runs the deterministic non-interactive portion of the suite
  (fake backend and manifest generation/validation) on every pull request.
• One interactive desktop CI job runs the real suite on a single endpoint,
  publishes run.json, junit.xml, and events.ndjson as artifacts, and is
  allowed to be non-required while its stability is measured.
• The Win32 Storybook endpoint bundles and launches locally with a documented
  command, and its story index matches the macOS and Windows Fabric endpoints.
• Every known false-green path listed in the branch's Priority 1 work is either
  fixed or documented as a known limitation in the package documentation.
• Deterministic component, Flex-theme, and package-archive scenarios produce
  checked-in baselines and advisory comparisons on every pull request.
• The component dependency audit rejects unused production dependencies and
  enforces the source boundary documented by the package.
• The portable story-test schema can assert role, accessible name, state,
  and keyboard reachability through the fake backend and at least one real
  desktop endpoint.

Production exit criteria

• macOS, Windows Fabric, and Win32 Paper each have a required CI gate running
  the same shared story-test manifest, with matching manifest digests.
• A documented flake policy exists: retry budget, quarantine mechanism,
  quarantine expiry, and an owner who reviews quarantined tests.
• Artifact retention, redaction, and review rules are documented and enforced
  for every gate, and no artifact directory is committed.
• Verified prerequisite and compatibility ranges are published for each backend
  (OS versions, driver versions, Xcode/WinAppDriver versions, Node version).
• The driver package is published with the required changeset and validated by
  at least one consumer that does not have a globally installed Appium driver.
• Interactive-desktop runner capacity (hosted or self-hosted) has a named owner
  and a documented failure/escalation path.
• Bundle-size budgets are required CI gates, and baseline updates require a
  reviewed rationale rather than being rewritten automatically.
• A Flex-only consumer excludes legacy Fluent construction and optional
  runtime color utilities, while opt-in scenarios prove each capability is
  included when explicitly imported.
• The Windows Jest smoke harness is retired or has a durable owner, expiry,
  version contract, and serialized execution model; all focus-crash tests
  remain required somewhere.
• Windows Fabric runs publish window-scoped visual evidence with capture
  provenance and locked-session diagnostics, without making pixels a
  pass/fail gate.
• Portable accessibility assertions are required on the initial component
  cohort for Windows and macOS.

Suggested Additions

These are not part of the current scope. They are decision-gated and
intentionally have no task files.

1. Re-enable iOS and Android E2E in pr.yml. Both are present but commented
   out; re-enabling is orthogonal to desktop work but affects the same CI budget.
2. Self-hosted interactive desktop runner pool. Observed: Windows UI
   automation requires an interactive, unlocked desktop, and clicks use synthetic
   input. Whether GitHub-hosted windows-latest satisfies this reliably at the
   required scale is an open question worth an explicit evaluation.
3. Storybook portable-stories / web test runner for non-native coverage. A
   cheap deterministic layer for logic-only story assertions that does not need a
   desktop session.
4. Backend migration decision (NovaWindows / Appium 4 / private Appium host).
   Observed: the branch declares NovaWindows unsupported and keeps Appium 4
   hosting as an open decision; both become urgent if WinAppDriver support lapses.
5. Publish @fluentui-react-native/desktop-driver outside the monorepo with
   its own documentation site entry and versioned compatibility matrix.
6. Add an apps/win32 Storybook-equivalent for the legacy FluentTester story
   set, if Win32 coverage should extend beyond agentic components.

Sources

Sources reflect the repository and linked branches as of 2026-08-20.

• apps/storybook/README.md,
  apps/storybook/AGENTS.md,
  apps/storybook/package.json - Storybook app
  contract, endpoints, and declared scripts.
• apps/storybook/src/main.ts,
  apps/storybook/metro.config.js,
  apps/storybook/react-native.config.js,
  apps/storybook/app.json,
  apps/storybook/.gitignore.
• apps/storybook/jest.windows.config.cjs,
  apps/storybook/windows-tests/storybook-smoke.test.cjs,
  apps/storybook/scripts/smoke-stories.json,
  apps/storybook/storybook-server.cjs.
• apps/E2E/README.md,
  apps/E2E/package.json,
  apps/E2E/wdio.conf.win32.js,
  apps/E2E/wdio.conf.windows.js,
  apps/E2E/wdio.conf.macos.js.
• .github/workflows/pr.yml,
  .ado/azure-pipelines.publish.yml.
• apps/win32/package.json,
  apps/win32/README.md,
  apps/win32/metro.config.js.
• packages/agentic/components/package.json,
  packages/native/Callout/package.json,
  packages/native/Callout/src/CalloutNativeComponent.ts,
  packages/native/Callout/react-native.config.cjs.
• origin/user/jasonvmo/test-driver at
  8f971021:
  README.md,
  DESIGN.md,
  USAGE.md,
  NEXT-STEPS.md,
  package.json,
  src/types.ts,
  apps/storybook/wdio.conf.ts.
• Installed packages inspected in the workspace: @office-iss/rex-win32@0.81.1,
  @office-iss/react-native-win32@0.81.8, react-native-test-app@5.3.3.
• Storybook for React Native and
  react-native-test-app as
  linked from apps/storybook/README.md.