Skip to content

feat(tests): add step timing tracking to E2E tests#48

Open
abueide wants to merge 3 commits intomainfrom
feat/process-compose-timing-logs
Open

feat(tests): add step timing tracking to E2E tests#48
abueide wants to merge 3 commits intomainfrom
feat/process-compose-timing-logs

Conversation

@abueide
Copy link
Copy Markdown
Contributor

@abueide abueide commented Apr 29, 2026

Summary

Adds timing instrumentation to the E2E test framework so we can see exactly how long each step takes in CI logs.

Problem

Currently, E2E tests show pass/fail but no timing information. When a test takes 37 minutes (React Native iOS), we don't know where the time is being spent without manual analysis of logs.

Solution

Test Framework Changes

New Functions:

  • e2e_step_start(step_name) - Records start timestamp
  • e2e_step_end(step_name, status, [reason]) - Records end timestamp, calculates duration

Enhanced Reporting:

  • e2e_report_steps() now displays duration for each step:
    ✓ PASS: build-node (23s)
    ✓ PASS: install-pods (45s)
    ✓ PASS: build-ios (312s)
    ✓ PASS: deploy (8s)
    ✗ FAIL: verify-app (14s)
      Reason: App crashed during soak period
    

Removed Functions:

  • e2e_step_pass() / e2e_step_fail() - Replaced by start/end pattern for timing

React Native iOS E2E Instrumented Steps

Updated 5 critical steps in test-suite-ios-e2e.yaml:

  • build-node - npm install timing
  • install-pods - CocoaPods install timing
  • build-ios - xcodebuild timing (longest step)
  • deploy - App deployment timing
  • verify-app - Liveness check + soak period timing

Usage Pattern

processes:
  my-step:
    command: |
      set -e
      _step="my-step"
      
      # Source test framework
      export REPO_ROOT="$(cd ../.. && pwd)"
      . "$REPO_ROOT/plugins/tests/test-framework.sh"
      
      # Record start time
      e2e_step_start "$_step"
      
      # Trap errors for timing
      trap 'e2e_step_end "$_step" "fail" "Error: $?"' ERR
      
      # ... do work ...
      
      # Record success with timing
      e2e_step_end "$_step" "pass"

Benefits

  1. 🔍 Better observability - Immediately see where time is spent
  2. ⚡ Identify bottlenecks - Spot slow steps at a glance
  3. 📊 Performance tracking - Baseline for future optimizations
  4. 🐛 Better debugging - Know which step failed AND how long it ran
  5. 📈 Regression detection - Can detect when steps get slower over time

Example CI Output

Before:

✓ PASS: build-node
✓ PASS: install-pods
✓ PASS: build-ios
✓ PASS: deploy
✓ PASS: verify-app

After:

✓ PASS: build-node (23s)
✓ PASS: install-pods (45s)
✓ PASS: build-ios (312s)    ← 😱 This is the bottleneck!
✓ PASS: deploy (8s)
✓ PASS: verify-app (14s)

Next Steps

  • Add timing to Android E2E tests
  • Add timing to standalone iOS/Android tests
  • Add timing to other React Native test suites
  • Consider adding total time summary at end

Testing

Will validate in CI when this PR runs. Timing should appear in test output logs.


🤖 Generated with Claude Code

abueide added 3 commits April 29, 2026 15:55
Add timing instrumentation to E2E test framework:

**Test Framework Changes:**
- Add e2e_step_start() to record step start time
- Add e2e_step_end() to record step end time and calculate duration
- Update e2e_report_steps() to display duration for each step
- Remove e2e_step_pass/fail() - replaced by e2e_step_start/end pattern

**React Native iOS E2E Updates:**
- Instrument build-node, install-pods, build-ios, deploy, verify-app steps
- Source test-framework.sh in each instrumented step
- Use trap to record failures with timing

**Output Format:**
- Pass: ✓ PASS: step-name (42s)
- Fail: ✗ FAIL: step-name (15s)
       Reason: error message

**Benefits:**
- CI logs now show exactly how long each step takes
- Easier to identify performance bottlenecks
- Better debugging when tests fail
- Foundation for performance regression detection

Example output:
  ✓ PASS: build-node (23s)
  ✓ PASS: install-pods (45s)
  ✓ PASS: build-ios (312s)
  ✓ PASS: deploy (8s)
  ✓ PASS: verify-app (14s)
When steps like install-pods and build-ios use 'cd ios', the relative
'reports/steps' path writes to the wrong location (ios/reports/steps).

Fixed by:
- Make e2e_step_start/end use absolute paths based on REPO_ROOT
- Remove unnecessary _step_dir variables from YAML
- Status files now written to correct location even after cd

This fixes the 'Step never executed' failures for install-pods and build-ios.
The warning 'IOS_DEVELOPER_DIR not set' was showing even when
auto-detection succeeded (by setting DEVELOPER_DIR).

Changed to only warn when BOTH variables are unset, meaning
auto-detection actually failed.

Before:
  ⚠️  [WARN] IOS_DEVELOPER_DIR not set  (always shown)
  ✅ [OK] iOS environment ready

After (auto-detection succeeds):
  ✅ [OK] iOS environment ready  (no warning)

After (auto-detection fails):
  ⚠️  [WARN] Could not detect Xcode developer directory
         Set IOS_DEVELOPER_DIR or run: xcode-select -p
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant