Skip to content

[Bug]: INACTIVITY_TIMEOUT never stops a Kubernetes workspace pod #1698

Closed
Closed
[Bug]: INACTIVITY_TIMEOUT never stops a Kubernetes workspace pod#1698
Labels
bugSomething isn't working

Description

@wsmoak
wsmoak
opened on Oct 25, 2026

Before submitting

  • I have searched existing issues to confirm this is not a duplicate
  • I am running the latest version of Devsy

What happened?

With the kubernetes provider, INACTIVITY_TIMEOUT does not stop an idle workspace. The
timeout fires, but it only ends a secondary daemon process. The pod's PID 1 keeps
running, so the pod stays Running indefinitely.

The docs (inactivity-timeout.mdx) say that for Docker, Podman, Kubernetes and
Microsandbox, when nobody has connected for the timeout the process "ends the
container's main process". Since #1565 they say the daemon "exits", which is accurate,
but the option still has no useful effect on Kubernetes.

The kubernetes provider.yaml wires INACTIVITY_TIMEOUT into two places, and neither
stops the pod:

  • agent.containerInactivityTimeout becomes ContainerTimeout, which only starts the
    secondary daemon described below.
  • agent.kubernetes.inactivityTimeout (ProviderKubernetesDriverConfig.InactivityTimeout,
    pkg/provider/provider.go) is resolved in pkg/options/resolve.go and never read in Go.
    The Kubernetes driver was apparently meant to act on it, but doesn't. This was also
    flagged during review of docs(developing-in-workspaces): correct inactivity timeout page #1565.

Cause of the first path:

  1. The container entrypoint (DefaultEntrypoint, pkg/devcontainer/single.go) execs
    devsy internal agent container daemon as PID 1 with no --timeout. Its config
    comes from the daemon-config Secret / DEVSY_WORKSPACE_DAEMON_CONFIG, and
    BuildWorkspaceDaemonConfig (pkg/daemon/agent/daemon.go) never sets Timeout.
    PID 1 therefore runs no timeout monitor.
  2. During setup, startContainerDaemon (cmd/internal/agentcontainer/setup.go) sees
    ContainerTimeout set and starts a second daemon through
    command.StartBackgroundOnce("devsy.daemon", ...) with --timeout <value>. Its
    PID-file check can't see PID 1, which wasn't started that way.
  3. The second daemon's runTimeoutMonitor (cmd/internal/agentcontainer/daemon.go)
    checks the mtime of /tmp/devsy.activity every 10 s. When it's older than the
    timeout, it returns an error and Run calls os.Exit(1).
  4. Nothing signals PID 1 or deletes the pod. There is no Kill(1 / FindProcess(1 in
    the code, and the Kubernetes driver never reads inactivityTimeout (it's only
    resolved in pkg/options/resolve.go).

What did you expect to happen instead?

After INACTIVITY_TIMEOUT of no connections, the workspace stops: the pod is no longer
Running, as it is after devsy workspace stop.

Steps to reproduce

  1. devsy provider add kubernetes
  2. devsy provider set kubernetes -o KUBERNETES_CONTEXT=docker-desktop -o KUBERNETES_NAMESPACE=devsy-p3test -o INACTIVITY_TIMEOUT=1m -o DISK_SIZE=1Gi
  3. Create the workspace source folder with the devcontainer config shown below:
    mkdir -p p3test/.devcontainer && echo '{ "image": "alpine:3.20" }' > p3test/.devcontainer/devcontainer.json
  4. devsy workspace up ./p3test --id p3test --provider kubernetes --ide none
  5. Don't connect. Wait more than 1 minute.
  6. kubectl -n devsy-p3test get pods shows the pod still Running.

devcontainer.json

{ "image": "alpine:3.20" }

Error output / logs

# right after up
PID 1   /usr/local/bin/devsy internal agent container daemon
PID 74  /usr/local/bin/devsy internal agent container daemon --timeout 1m --shutdown-action stopContainer

# 2m49s after up, no connections
NAME                     READY   STATUS    RESTARTS   AGE
devsy-default-p3-73b80   1/1     Running   0          2m49s
PID 1   /usr/local/bin/devsy internal agent container daemon      # PID 74 is gone

How often does this happen?

Every time

Operating system

macOS

Linux distribution (if applicable)

N/A (the pod image is alpine:3.20)

Architecture

ARM64

Desktop app or CLI?

CLI only

Devsy version

v1.24.0

Devsy provider

Kubernetes

Provider version

v1.0.0

Screenshots

No response

Anything else?

Possible fixes:

  • Put Timeout in the config BuildWorkspaceDaemonConfig produces, so the entrypoint
    daemon runs the monitor itself, and skip the second daemon.
  • Or have the secondary daemon signal PID 1 (syscall.Kill(1, SIGTERM)) on timeout.
  • Or have the Kubernetes driver act on the timeout and delete the pod, as
    StopDevContainer (pkg/driver/kubernetes/driver.go) does for workspace stop.
    This would give the currently unused InactivityTimeout field a purpose. The
    provider's own option description says it "will automatically stop the pod after the
    inactivity period".

If the fix ends in an exiting PID 1, the workspace can't be resumed with workspace up
until a related issue "terminated pod isn't recreated" is fixed. Deleting the pod avoids that.

Workaround: stop idle workspaces explicitly with devsy workspace stop <id>, by hand or
from a scheduled job.

Related, closest existing issue: #1238 (AWS machines, closed).

Activity

  1. wsmoak
    added
    bugSomething isn't working
    on Oct 25, 2026
  2. skevetter
    added 5 commits that reference this issue on Oct 25, 2026

    fix: satisfy lint in kubernetes inactivity e2e (#1698)

    835a4a7

    test: avoid shadowing builtin real in lifecycle test (#1698)

    5d5ef32

    test: cover waitPodDeleted absent and timeout paths (#1698)

    71d152b

    fix: make Kubernetes inactivity lifecycle fail safe (#1698)

    c7a46bb

    fix: stop idle Kubernetes workspaces and recreate terminal pods (#1698)…

    b864424

  3. skevetter commented on Oct 31, 2026

    @skevetter
    skevetter
    on Oct 31, 2026 – with ChatGPT Codex Connector
    Contributor

    Triage result: fixed for the reported Kubernetes layout in v1.25.0-beta.6 and later prereleases

    Reviewed on 30 October 2026 against main at 9bc12ee041ca854c6f81de95fd8c6e49d7cc028c.

    PR #1701 merged the inactivity-lifecycle fix as b86442407a9ad9854213f08931236d470d783b77. It is included in v1.25.0-beta.6, confirmed by commit ancestry, and in the currently published v1.25.0-beta.8. The latest stable release remains v1.24.0, which contains the reported defect.

    The PR body's earlier “keep draft” wording is stale. GitHub records the PR as merged. The source below is the merged implementation, rather than a proposal to implement the same work again.

    The reported cause is now addressed

    Original failure Merged behavior
    The main container daemon received no timeout. BuildWorkspaceDaemonConfig serializes the resolved timeout, shutdown action and managed-lifecycle ownership into the main daemon's configuration.
    Setup started a second timeout daemon, whose exit did not end the pod. startContainerDaemon returns immediately for a managed lifecycle. The existing main daemon owns the timer.
    The timeout could race setup, hooks or sessions. Container lifecycle registration acknowledges protected setup before work and keeps activity registered for live setup/hook connections. The managed lifecycle loop begins unarmed until setup permits the idle timer.
    Timeout exit left the main process running. daemonExitCode treats inactivity expiry as normal completion. Ending the managed main process ends the workspace container.
    A terminal pod could not resume. The Kubernetes pod lifecycle replaces a terminal pod on the next ordinary up, preserving its PVC. This also fixes #1699.
    The Kubernetes-specific timeout field was unused. effectiveTimeout and stored-policy refresh resolve the canonical container timeout and deprecated Kubernetes alias and apply current policy on resume.

    Supported behavior and upgrade path

    The expected idle-stop result is a terminal pod. The pod object can remain visible as Succeeded or Failed until the next up or explicit stop. It does not have to disappear from kubectl get pods to be stopped.

    Use a supported single-workspace-container layout with the managed Devsy entrypoint. Positive timeouts reject extra regular containers that would keep the pod alive, external image-main entrypoints and Dockerless execution. Normal init containers and native sidecars have separate topology handling. Empty/zero timeout, shutdownAction: none, and the current invocation's --disable-daemon disable inactivity stop. The updated documentation defines these cases, activity tracking and PVC permissions.

    An existing pod that is still running an older lifecycle owner needs one explicit stop/start after upgrading. Updating an agent file does not replace the process that already owns the pod. The source returns a specific migration error that instructs devsy workspace stop followed by devsy workspace up. The workspace volume is kept; changes only in the container's writable filesystem do not survive replacement.

    Existing verification and final customer check

    The committed E2E suite exercises idle stop/resume, preserved PVC and file contents across two cycles, a live tracked SSH session, a long post-attach hook, disabled policy, an explicitly terminated pod and layout cases. The final-head Kubernetes inactivity CI job, general Kubernetes job and restricted-security job all completed successfully. This does not establish every cluster's webhook/security behavior.

    For this reporter's original Alpine/default-entrypoint case, the closure check is:

    1. Use beta.6 or later with the matching agent and set INACTIVITY_TIMEOUT=1m.
    2. If the pod predates the fix, perform the one-time stop/start migration.
    3. Write a sentinel on the workspace PVC, disconnect all tracked SSH/IDE sessions and wait beyond the timeout. Observe through kubectl; repeated SSH probes themselves count as activity.
    4. Confirm the pod leaves Running, its container has a terminal state and it does not restart itself.
    5. Run ordinary workspace up; confirm a new Pod UID, unchanged PVC/workspace UID and preserved sentinel.
    6. Hold a tracked SSH session open longer than the timeout. It must remain running, then receive a fresh idle interval after disconnection.

    No new implementation ticket is needed for this report. Resolve it as fixed in the identified prerelease, or retain it only for reporter/release confirmation. Any newly reproduced failure under an unsupported or different pod layout should include that topology and be scoped separately instead of reopening the original secondary-daemon design.

    This review inspected source, release ancestry and recorded CI results; it did not run a new Kubernetes reproduction.

  4. skevetter
    closed this as completedon Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions