Repository navigation
Make the iOS E2E job tolerate staging and simulator flakiness - #81
Merged
Merged
Conversation
The flows run against live staging on hosted runners, and most failures so far came from there rather than from the code: a 502 from staging's OIDC discovery, its first-layer page loading past the layer's 8s timeout, and Maestro's iOS driver not starting in time. - A failed flow runs once more, and every flow does if Maestro stops before reporting any. Flows start from a clean slate, so a rerun is safe. A pass on retry still uploads the debug output, Metro's log included, and leaves a warning annotation naming the flows. - The default-config job gives the layer's page 30s to load, through a new EXPO_PUBLIC_LAYER_PAGE_LOAD_TIMEOUT_MS override in the example. - The simulator boots at the start of the job, so its first boot happens while the app builds, and Maestro's driver gets 300s instead of 240s to start. - Maestro (2.11.0) and the simulator (iPhone 16 Pro, iOS 18.5) are pinned, so a Maestro release or a runner image update can't change the job between runs.
Maestro writes the variables passed with -e, and the text it types, in plain text into its debug output: commands.json and maestro.log carry the staging account's email and password. GitHub masks secrets in job logs but not inside uploaded artifacts, and this repository's artifacts are public. Before the upload, both values are replaced with [redacted] in every non-image file, and any file that still contains one is dropped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The iOS E2E suite from #80 runs against live staging on GitHub-hosted runners. Most of its failures so far came from there rather than from the code:
This PR makes the job tolerate those failures while keeping them visible:
clearKeychainand aclearStatelaunch), so a rerun is safe. A pass on retry turns the job green with a warning annotation naming the flows, and still uploads the debug output, including Metro's log, so the flake can be looked into.default-configjob gives the layer's page 30 s to load, through a newEXPO_PUBLIC_LAYER_PAGE_LOAD_TIMEOUT_MSoverride in the example'sConfig.ts. These flows test the gate's behaviour, not staging's latency, and the fail-open path stays covered bycmp-timeout-fail-open.2.11.0, the version CI has been running) and the simulator (iPhone 16 Pro, iOS 18.5, the one it has been picking) are pinned in the job'senv. A Maestro release or a runner image update can no longer change the job between two runs of the same commit.It also fixes a leak. Maestro writes the staging account's email and password in plain text into its debug output (
commands.json,maestro.log). GitHub masks secrets in job logs but not inside uploaded artifacts, and this repository's artifacts are public. Before the upload, both values are now replaced with[redacted], and any file that still contains one is dropped. Artifacts uploaded before this change still contain the credentials, so the accounts' passwords need rotating.Testing
The redaction was tested on fake debug output: regex metacharacters in the password are handled, a binary file that contains a secret is dropped, and empty secrets (as on fork PRs) are a no-op.
The retry logic was tested with a stubbed
maestroin four cases: first attempt passes; one flow fails and then passes; Maestro writes no report and everything passes on retry; a flow fails twice. It exits and annotates correctly in each.Maestro 2.11.0's JUnit report and its behaviour of continuing after a failed flow were checked on a local simulator.
EXPO_PUBLIC_LAYER_PAGE_LOAD_TIMEOUT_MSwas confirmed to reach the dev bundle served by Metro.The real runs are this PR's CI.