Skip to content

Make the iOS E2E job tolerate staging and simulator flakiness - #81

Merged
0x7f merged 2 commits into
mainfrom
e2e-ios-flakiness
Oct 6, 2026
Merged

0x7f merged 2 commits into
mainfrom
e2e-ios-flakiness

Conversation

@0x7f

@0x7f 0x7f commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Summary

The iOS E2E suite from #80 runs against live staging on GitHub-hosted runners. Most of its failures so far came from there rather than from the code:

  • a 502 from staging's OIDC discovery document;
  • staging's first-layer page loading past the layer's 8 s timeout, after which the gate fails open;
  • Maestro's iOS driver not starting within its startup timeout.

This PR makes the job tolerate those failures while keeping them visible:

  • Retry once. A failed flow runs again; every flow does if Maestro stops before reporting any of them. Flows start from a clean slate (clearKeychain and a clearState launch), so a rerun is safe. A pass on retry turns the job green with a warning annotation naming the flows, and still uploads the debug output, including Metro's log, so the flake can be looked into.
  • Longer page load for the layer. The default-config job gives the layer's page 30 s to load, through a new EXPO_PUBLIC_LAYER_PAGE_LOAD_TIMEOUT_MS override in the example's Config.ts. These flows test the gate's behaviour, not staging's latency, and the fail-open path stays covered by cmp-timeout-fail-open.
  • Early simulator boot. The simulator boots at the start of the job, so its first boot happens while the app builds. Maestro's driver gets 300 s instead of 240 s to start.
  • Pinned versions. Maestro (2.11.0, the version CI has been running) and the simulator (iPhone 16 Pro, iOS 18.5, the one it has been picking) are pinned in the job's env. A Maestro release or a runner image update can no longer change the job between two runs of the same commit.

It also fixes a leak. Maestro writes the staging account's email and password in plain text into its debug output (commands.json, maestro.log). GitHub masks secrets in job logs but not inside uploaded artifacts, and this repository's artifacts are public. Before the upload, both values are now replaced with [redacted], and any file that still contains one is dropped. Artifacts uploaded before this change still contain the credentials, so the accounts' passwords need rotating.

Testing

  • The redaction was tested on fake debug output: regex metacharacters in the password are handled, a binary file that contains a secret is dropped, and empty secrets (as on fork PRs) are a no-op.

  • The retry logic was tested with a stubbed maestro in four cases: first attempt passes; one flow fails and then passes; Maestro writes no report and everything passes on retry; a flow fails twice. It exits and annotates correctly in each.

  • Maestro 2.11.0's JUnit report and its behaviour of continuing after a failed flow were checked on a local simulator.

  • EXPO_PUBLIC_LAYER_PAGE_LOAD_TIMEOUT_MS was confirmed to reach the dev bundle served by Metro.

  • The real runs are this PR's CI.

0x7f added 2 commits October 6, 2026 11:59
The flows run against live staging on hosted runners, and most failures
so far came from there rather than from the code: a 502 from staging's
OIDC discovery, its first-layer page loading past the layer's 8s timeout,
and Maestro's iOS driver not starting in time.

- A failed flow runs once more, and every flow does if Maestro stops
  before reporting any. Flows start from a clean slate, so a rerun is
  safe. A pass on retry still uploads the debug output, Metro's log
  included, and leaves a warning annotation naming the flows.
- The default-config job gives the layer's page 30s to load, through a
  new EXPO_PUBLIC_LAYER_PAGE_LOAD_TIMEOUT_MS override in the example.
- The simulator boots at the start of the job, so its first boot happens
  while the app builds, and Maestro's driver gets 300s instead of 240s
  to start.
- Maestro (2.11.0) and the simulator (iPhone 16 Pro, iOS 18.5) are
  pinned, so a Maestro release or a runner image update can't change the
  job between runs.
Maestro writes the variables passed with -e, and the text it types, in
plain text into its debug output: commands.json and maestro.log carry the
staging account's email and password. GitHub masks secrets in job logs but
not inside uploaded artifacts, and this repository's artifacts are public.

Before the upload, both values are replaced with [redacted] in every
non-image file, and any file that still contains one is dropped.
@0x7f
0x7f merged commit c6aef04 into main Oct 6, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant