Skip to content

fix: downgrade transient railway token-refresh failures from ERROR to WARN and continue with valid token #SUPERLOG - #440

Open
superlog-app[bot] wants to merge 1 commit into
mainfrom
superlog/fix-railway-token-refresh-warn
Open

fix: downgrade transient railway token-refresh failures from ERROR to WARN and continue with valid token #SUPERLOG#440
superlog-app[bot] wants to merge 1 commit into
mainfrom
superlog/fix-railway-token-refresh-warn

Conversation

@superlog-app

@superlog-app superlog-app Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

When Railway's OAuth token endpoint returns a transient server_error, the worker was logging it as ERROR (triggering an incident) and skipping the entire installation for the current pass — even though the access token is still valid for up to 5 more minutes.

Root cause

The token-refresh block fires when the token has less than TOKEN_REFRESH_MARGIN_MS (5 minutes) remaining. At that point accessToken already holds the valid stored token. On a transient Railway server error the old code returned early, abandoning the still-valid token for the whole pass. Concretely: error: "server_error" is RFC 6749's transient-server-error code — analogous to the status_500 / status_503 values that railway grant refresh failed and railway inventory read failed already handle as WARN with graceful fallback.

Remediation

  • On a failed refresh, check whether the stored token has already expired:
    • Not yet expired → log WARN ("continuing with current token"), fall through using installation.accessToken; next pass will retry the refresh.
    • Already expired → log WARN ("token expired — skipping"), return as before; downstream Railway calls would fail anyway.
  • Move the accessToken = refreshed.accessToken / saveTokens(...) lines into an else branch so they only execute on a successful refresh.
  • Added two new tests in puller.test.ts covering both branches; all 8 tests pass.

Incident: yeasty-skunk — ERROR: railway token refresh failed


Was this PR helpful? Leave feedback — goes straight to the Superlog team.


Summary by cubic

Downgraded transient Railway token refresh failures to WARN and continue using the current valid token, avoiding false incidents and skipped pulls. If the token is already expired, we still skip but log WARN.

  • Bug Fixes
    • On refresh server_error with a not-yet-expired token, log WARN and proceed with installation.accessToken.
    • If the token is expired, log WARN and skip; no ERROR logs.
    • Only update accessToken/refreshToken on successful refresh.
    • Added tests covering both paths in apps/worker/src/railway/puller.test.ts.

Written for commit 6cf6234. Summary will update on new commits.

Review in cubic

… WARN and continue with valid token #SUPERLOG

Delivery-Id: 0b9e27c13d580ca8e548a1e9b65ab3d8c7e2edd876feb6ef31d7f107b8fa0142
Delivery-Base: main

@superlog-app superlog-app Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Observability review

  • 1 warning

{ installation_id: installation.id, error: refreshed.error },
"railway token refresh failed",
tokenExpired
? "railway token refresh failed and token expired — skipping"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logs · warning — Record refresh error on the active span when falling through with a valid token

Add span.recordException(new Error(refreshed.error)) (without setting error status, since the pass succeeds) so that traces for passes that used a degraded token are distinguishable from fully healthy passes; without it an operator cannot tell from a trace alone that the token was not refreshed.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant