fix: downgrade transient railway token-refresh failures from ERROR to WARN and continue with valid token #SUPERLOG - #440
Open
superlog-app[bot] wants to merge 1 commit into
Conversation
… WARN and continue with valid token #SUPERLOG Delivery-Id: 0b9e27c13d580ca8e548a1e9b65ab3d8c7e2edd876feb6ef31d7f107b8fa0142 Delivery-Base: main
| { installation_id: installation.id, error: refreshed.error }, | ||
| "railway token refresh failed", | ||
| tokenExpired | ||
| ? "railway token refresh failed and token expired — skipping" |
Contributor
Author
There was a problem hiding this comment.
logs · warning — Record refresh error on the active span when falling through with a valid token
Add span.recordException(new Error(refreshed.error)) (without setting error status, since the pass succeeds) so that traces for passes that used a degraded token are distinguishable from fully healthy passes; without it an operator cannot tell from a trace alone that the token was not refreshed.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When Railway's OAuth token endpoint returns a transient
server_error, the worker was logging it as ERROR (triggering an incident) and skipping the entire installation for the current pass — even though the access token is still valid for up to 5 more minutes.Root cause
The token-refresh block fires when the token has less than
TOKEN_REFRESH_MARGIN_MS(5 minutes) remaining. At that pointaccessTokenalready holds the valid stored token. On a transient Railway server error the old code returned early, abandoning the still-valid token for the whole pass. Concretely:error: "server_error"is RFC 6749's transient-server-error code — analogous to thestatus_500/status_503values thatrailway grant refresh failedandrailway inventory read failedalready handle as WARN with graceful fallback.Remediation
installation.accessToken; next pass will retry the refresh.accessToken = refreshed.accessToken/saveTokens(...)lines into anelsebranch so they only execute on a successful refresh.puller.test.tscovering both branches; all 8 tests pass.Incident: yeasty-skunk — ERROR: railway token refresh failed
Was this PR helpful? Leave feedback — goes straight to the Superlog team.
Summary by cubic
Downgraded transient Railway token refresh failures to WARN and continue using the current valid token, avoiding false incidents and skipped pulls. If the token is already expired, we still skip but log WARN.
server_errorwith a not-yet-expired token, log WARN and proceed withinstallation.accessToken.accessToken/refreshTokenon successful refresh.apps/worker/src/railway/puller.test.ts.Written for commit 6cf6234. Summary will update on new commits.