Gateway's interceptor credential expires after a forward clock jump or suspend; interceptor-bound RPCs fail closed until the next refresh #5044

User Story

I use OpenShell v0.1.5 as the sandbox runtime of a self-hosted agent platform I am building (Fort): one gateway as a systemd service on Ubuntu 24.04, Docker driver, a supervisor middleware and gateway interceptors on Unix sockets. I hit this while testing that setup on two VMs.

Problem Statement

With an authenticated gateway interceptor (Unix socket, extension auth on), every interceptor-bound RPC fails closed for tens of minutes with PERMISSION_DENIED "gateway interceptor '' failed closed: interceptor transport error: … 'The request does not have valid authentication credentials', message: "extension bearer token has expired"". The interceptor never receives these calls: the gateway itself refuses to send a credential it considers expired (BearerTokenSlot::authorization_at, crates/openshell-extension-core/src/auth.rs:86-100).

Observed: the gateway mints its interceptor credential with a 3600 s lifetime and refreshes it 2880 s (4/5) later, measured on the monotonic clock, while expiry is checked against the wall clock. When the wall clock moves ahead of the monotonic clock by more than the remaining 720 s, the credential expires before the next refresh. We reproduced this with a forward clock step: right after a refresh, with NTP off, date -s "+45 min"; 14 minutes later every interceptor-bound RPC failed, and it kept failing for 32 minutes, until the refresh fired exactly 2880 s of monotonic time after the previous one. A guest suspend (ACPI S3) opens the same gap: after a 6 min suspend CLOCK_MONOTONIC was 353 s behind CLOCK_BOOTTIME (we did not run a suspend long enough to expire the credential). We first saw the failure without a deliberate step, twice in one day on a laptop-hosted VM (26 min until a gateway restart, and 21–43 min recovering on its own).

Impact / Why This Matters

During the outage CreateSandbox, DeleteSandbox, UpdateConfig and provider writes all fail; the gateway's own SnapshotProviderProfiles calls fail too, so StartSandbox fails ("Docker supervisor exited before becoming ready") and leaves the sandbox in Error until a Stop + Start after recovery. Running sandboxes, their model calls through credential placeholders and StopSandbox keep working. Policy tightening and credential rotation cannot reach running sandboxes until it clears. Today the only immediate fix is restarting the gateway, which restarts every sandbox.

Acceptance Criteria

• A forward wall-clock step or a VM suspend/resume does not make interceptor-bound RPCs fail with "extension bearer token has expired".
• If the credential is found expired at send time, the gateway obtains a fresh one instead of failing the call.

Reproduction Steps

1. Run the gateway with an authenticated interceptor (Unix socket) and wait until the gateway has refreshed its interceptor credential once.
2. timedatectl set-ntp false; date -s "+45 min".
3. Every minute, call an interceptor-bound RPC (e.g. DeleteSandbox of a missing name with allowMissing: false).
4. About 12 minutes after the last refresh, every call returns PERMISSION_DENIED "… extension bearer token has expired", until roughly 48 minutes after that refresh.

Environment

• OpenShell: v0.1.5 (bdb956f), gateway release tarball, run as a systemd service (non-root openshell user)
• OS: Ubuntu 24.04 (arm64 on a Lima VM, kernel 6.8; x86_64 on a GCP VM, kernel 7.0.0-1014-gcp), Docker 29.9.2
• Deployment: single gateway, Docker compute driver, supervisor middleware on HTTPS loopback, two gateway interceptors on Unix sockets (extension auth on)

Logs

gateway interceptor 'arcology' failed closed: interceptor transport error: code: 'The request does not have valid authentication credentials', message: "extension bearer token has expired"
provider profile source 'interceptor/arcology' snapshot failed: ... SnapshotProviderProfiles failed for 'arcology': code: 'The request does not have valid authentication credentials'
start sandbox failed: Docker supervisor exited before becoming ready