Skip to content

@actions/cache: a failed cache download leaks its segment timer and stalls the job for up to 10 minutes #2470

Description

@brunoborges

Describe the bug

promiseWithTimeout in packages/cache/src/internal/downloadUtils.ts clears its timeout inside a .then, which only runs when the raced promise fulfils:

https://github.com/actions/toolkit/blob/main/packages/cache/src/internal/downloadUtils.ts#L448-L461

return Promise.race([promise, timeoutPromise]).then(result => {
  clearTimeout(timeoutHandle)
  return result
})

When the download promise rejects, the onFulfilled callback is skipped, clearTimeout is never called, and the armed setTimeout keeps Node's event loop alive. The action has finished all of its work, but the process cannot exit until the timer fires.

The 'timeout' and success paths are both fine — they fulfil, so they clear correctly. Only rejection leaks.

Both call sites are affected:

Call site Timeout Hang after work completes
downloadCacheStorageSDK options.segmentTimeoutInMs || 3600000, default 600000 up to 10 minutes
downloadSegmentRetry 30000 up to ~30s, and it leaks one timer per attempt inside the 5-retry loop

To Reproduce

The trigger is any rejection from the download — most commonly the cache service dropping the blob mid-transfer. This repro uses the function verbatim, so it needs no cache service:

// leak.mjs — promiseWithTimeout, copied verbatim
const promiseWithTimeout = async (timeoutMs, promise) => {
  let timeoutHandle;
  const timeoutPromise = new Promise(resolve => {
    timeoutHandle = setTimeout(() => resolve('timeout'), timeoutMs);
  });
  return Promise.race([promise, timeoutPromise]).then(result => {
    clearTimeout(timeoutHandle);
    return result;
  });
};

try {
  await promiseWithTimeout(600000, Promise.reject(new Error('The specified blob does not exist.')));
} catch (error) {
  console.log(`caught: ${error.message}`);
}
console.log('all work finished; nothing left to do');
console.log('pending timers:', process.getActiveResourcesInfo().filter(r => r === 'Timeout').length);
$ timeout 20 node leak.mjs
caught: The specified blob does not exist.
all work finished; nothing left to do
pending timers: 1
$ echo $?
124          # still hanging, killed at 20s

Changing the rejection to Promise.resolve(Buffer.alloc(1)) prints pending timers: 0 and exits in 2ms.

Observed in a real workflow

actions/setup-java (which depends on @actions/cache ^6.2.0) on a hosted ubuntu-24.04 runner. The same action version, in the same job, on the same runner, took 0s in another slot — so the only variable was the blob failure:

05:09:35  Received 100663296 of 161701622 (62.3%), 67.9 MBs/sec
05:09:35  ##[warning]Failed to restore: The specified blob does not exist.
05:09:35  maven cache is not found          <- action finished its work here
05:13:07  ##[error]The operation was canceled.   <- 3.5 min of silence, cancelled manually

The step had consumed 214 seconds of runner time doing nothing. Left alone it would have idled the full 10 minutes. Because the failure is already handled and logged as a warning, there is no output during the wait and nothing indicating why the step is still running — it looks like the consuming action has hung.

Expected behavior

Once the download settles, the timer should be cleared regardless of outcome, and the process should be free to exit.

Suggested fix

.finally() runs on both settlement paths:

return Promise.race([promise, timeoutPromise]).finally(() => {
  clearTimeout(timeoutHandle)
})

.finally() passes the fulfilment value through and re-raises the rejection, so the observable behaviour of the function is otherwise unchanged. Alternatively, timeoutHandle.unref() would stop the timer from holding the event loop open, though it would leave the timer itself uncleared.

Additional context

  • @actions/cache 6.2.0 (current latest); present on main as of this writing.
  • Node 20 and 24, Linux.
  • Impact is wasted billable runner minutes on every transient cache-service blob failure, multiplied across every action that restores a cache. The failure is transient and already handled correctly — only the idle afterwards is the problem.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions