fix runtime heartbeat API retries

[Pigbibi]

What changed

• retry transient GitHub API responses (429, 500, 502, 503, and 504)
• retry temporary network failures with bounded exponential backoff
• honor numeric Retry-After values with a 30-second cap
• preserve immediate failures for non-transient authorization and request errors
• add focused coverage for recovery, non-retryable errors, and retry exhaustion

Why

The Runtime Heartbeat failed when GitHub returned a transient HTTP 503 while listing workflow runs. The request had no retry handling, so a short GitHub service interruption caused a false heartbeat failure even though Runtime itself remained healthy.

Validation

• python3 -m unittest discover -s tests -p test_runtime_workflow_heartbeat.py
• uvx ruff check scripts/runtime_workflow_heartbeat.py tests/test_runtime_workflow_heartbeat.py
• git diff --check

[github-actions]

🤖 Codex PR Review

✅ Merge allowed: 1 finding(s) reported but none are blocking

ℹ️ Other Findings

1. 🟡 [MEDIUM] Reliability in scripts/runtime_workflow_heartbeat.py

The new URLError branch retries every urllib.error.URLError as if it were temporary. _github_request() is the heartbeat's GitHub API entry point, so permanent local failures such as TLS certificate verification errors, unsupported proxy configuration, or other non-recoverable transport setup errors will now incur three backoff sleeps before failing. That slows failure detection for each heartbeat run and can push the job past its expected timing budget even though the error is not transient. (line 76)

Suggestion: Only retry URLError instances whose reason is known to be temporary (for example timeouts, connection resets, or transient DNS failures). Let certificate, configuration, and other permanent transport errors fail immediately.

---

Review by Codex PR Review bot • PR

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fe398aa821

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Retry direct socket timeouts from urlopen

When the GitHub endpoint stalls past the explicit 20-second timeout, CPython's urlopen can raise TimeoutError directly rather than wrapping it in URLError. That exception bypasses both retry handlers here, so a transient read/connect timeout still terminates the heartbeat immediately and produces the same false alert this retry logic is meant to avoid.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Retry GitHub rate-limit 403 responses

GitHub can report primary or secondary rate-limit exhaustion with HTTP 403 (distinguishable from authorization failures by rate-limit/Retry-After headers), but this status is treated as permanently non-retryable. In that rate-limit case the heartbeat fails without an alert message instead of waiting and retrying, even though the API request will become valid again.

Useful? React with 👍 / 👎.