Proposal: Automatically terminate idle Kubernetes Computing Units

[yrenat]

Background

When running Texera on Kubernetes, computing units (CUs) remain alive even after they become idle. Although keeping idle CUs can reduce startup latency for subsequent workflow executions, they continue consuming cluster resources unnecessarily when they remain unused for an extended period.

As a result, clusters may gradually accumulate idle computing units, reducing the amount of resources available for active workloads.

Proposal

Issue #5362 proposes introducing an automatic cleanup mechanism for idle computing units.

The basic idea is to periodically check the status of running CUs and terminate those that have been idle for longer than a configurable timeout. This allows inactive resources to be reclaimed automatically while still giving users the benefit of reusing recently active computing units.

The timeout should be configurable so that different deployments can choose an appropriate balance between resource utilization and startup latency.

Initial Scope

For this proposal, the implementation focuses on:

• adding a configurable idle timeout;
• periodically checking whether a computing unit has been idle beyond the configured threshold;
• automatically terminating eligible idle computing units;
• ensuring active computing units are unaffected.

This proposal intentionally focuses on basic idle cleanup. More advanced policies (for example, different timeout strategies or discussions based on frontend heartbeat signals) can be discussed separately if needed.

Questions for Discussion

• Does periodically terminating idle computing units align with the intended Kubernetes resource management strategy?
• Is a single configurable timeout sufficient, or should different types of computing units have different timeout policies?
• Are there scenarios where idle computing units should intentionally remain alive and therefore be excluded from automatic cleanup?

I have prepared an implementation of this proposal for reference in PR #6046.

[chenlica]

@yrenat Thanks for creating the discussion. Before we dive into the details, I wonder whether this problem is common in Kubernetes. If so, are there any existing solutions to this problem? Please do some research and report your findings.

[yrenat]

@chenlica
Based on my findings, this resource-waste problem is common. There are several generic solutions on "system" level, where they have a global control of what/how many resources are needed, but they can seldom address our specific problem where we need to control a specific pod/CU. The following are some of my findings, regarding garbage resource collection.

1. HPA / custom metrics
   This is the native idea from Kubernetes. It automatically changes the replica count of a scalable Kubernetes workload, such as a Deployment, ReplicaSet, ReplicationController, or StatefulSet, based on metrics like CPU, memory, or custom/external metrics. In simpler words, it can automatically scale up/down the number of pods/resources invovled. However, a Texera CU is a pod-specific per-user application resource with a DB record, access control, resource settings, and a corresponding runtime Pod (not just a pod). Also, “idle” is another Texera-specific, idea, which cannot be natively well-described by Kubernetes. Besides, HPA cannot fully shutdown the system. It has to have at least one pod running to keep track of all the computation resources.
2. KEDA
   KEDA (Kubernetes Event-driven Autoscaling) is designed for event-driven autoscaling and can scale workloads down to zero when event sources become inactive. This is a good solution for queue/event-driven services. However, using KEDA directly would require Texera to expose CU activity as an external/custom metric or event source, which will create multi layers of interference with Texera. It also would not automatically update Texera’s CU database state unless Texera adds extra integration.
3. Knative Serving
   Knative supports scale-to-zero for HTTP request/traffic-driven services. This is useful for serverless HTTP workloads, but a lot of Texera resources/commands are not handled in HTTP format. Adopting Knative would be a larger architectural change and still may not capture Texera’s definition of CU idleness. Besides, Knative also does not focus on pod/CU level logistics.

To sum up, for the current issue, I think the most suitable first solution is to implement our tailored idle detection inside Texera’s Computing Unit Managing Service or a Texera-side scheduled cleanup component. The cleanup should use Texera’s own CU termination logic rather than directly deleting Pods, so that both the Kubernetes Pod and the Texera DB state remain consistent.

[kunwp1]

Hi @yrenat We need this feature and thank you for working on this task! I did some research before and concluded that Kubernetes does not natively support this feature. I agree we need to have our own way mechanism to terminate idle CUs.

What's missing in both this discussion post and your PR is how you defined what "idle" CU means and the architectural design of how computing unit manager monitors all the CUs. Can you be more specific on those parts? Please include some visual diagrams if you can. We need to agree on the design before reviewing the PR.

[chenlica]

@yrenat Thanks for reporting the findings. I am wondering: do other systems have something similar to "CU"? If not, what do they use? If the "Computing Unit (CU)" concept in Texera unique, we want to know its pros and cons compared to other systems.