Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ jobs:
go-version-file: go.mod
- name: Test (envtest)
run: make test
- name: Test startup without Argo Rollouts
run: ARGO_ROLLOUTS_CRD= go test ./internal/controller -ginkgo.focus='Optional Rollout API'

build:
runs-on: ubuntu-latest
Expand Down
4 changes: 3 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ help: ## Display this help.
.PHONY: manifests
manifests: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
"$(CONTROLLER_GEN)" rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cp config/crd/bases/apps.understudy.sh_understudies.yaml charts/understudy/crds/apps.understudy.sh_understudies.yaml

.PHONY: generate
generate: controller-gen ## Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
Expand All @@ -60,8 +61,9 @@ vet: ## Run go vet against code.
go vet ./...

.PHONY: test
ARGO_ROLLOUTS_CRD ?= $(shell go list -m -f '{{.Dir}}' github.com/argoproj/argo-rollouts)/manifests/crds/rollout-crd.yaml
test: manifests generate fmt vet setup-envtest ## Run tests.
KUBEBUILDER_ASSETS="$(shell "$(ENVTEST)" use $(ENVTEST_K8S_VERSION) --bin-dir "$(LOCALBIN)" -p path)" go test $$(go list ./... | grep -v /e2e) -coverprofile cover.out
ARGO_ROLLOUTS_CRD="$(ARGO_ROLLOUTS_CRD)" KUBEBUILDER_ASSETS="$(shell "$(ENVTEST)" use $(ENVTEST_K8S_VERSION) --bin-dir "$(LOCALBIN)" -p path)" go test $$(go list ./... | grep -v /e2e) -coverprofile cover.out

# TODO(user): To use a different vendor for e2e tests, modify the setup under 'tests/e2e'.
# The default setup assumes Kind is pre-installed and builds/loads the Manager Docker image locally.
Expand Down
47 changes: 42 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@
<a href="LICENSE"><img alt="License" src="https://img.shields.io/github/license/kylan11/understudy?style=for-the-badge"></a>
<a href="https://github.com/kylan11/understudy/stargazers"><img alt="Stars" src="https://img.shields.io/github/stars/kylan11/understudy?style=for-the-badge&color=e3b341"></a>
</p>
<p>
<a href="https://faun.dev/toolbox/understudy/"><img alt="FAUN.dev score: 74/100" src="https://faun.dev/toolbox/understudy/badge.svg?kind=score"></a>
<a href="https://faun.dev/toolbox/understudy/"><img alt="Code: Open source" src="https://faun.dev/toolbox/understudy/badge.svg?kind=code"></a>
<a href="https://faun.dev/toolbox/understudy/"><img alt="License: Apache-2.0" src="https://faun.dev/toolbox/understudy/badge.svg?kind=license"></a>
</p>
<img src="docs/assets/terminal-demo.svg" width="720" alt="Terminal simulation: the pod in production receives a spot interruption notice, Understudy holds the eviction and starts a stand-in pod that goes from Pending to Running; only then does the original terminate. The probe records 764 requests, 1 failed.">
</div>

Expand All @@ -27,9 +32,9 @@ protection against unannounced hardware death. That last one is what a second
replica is for, and nothing changes it.

In theatre the understudy waits in the wings and steps in when the lead exits.
This operator does the same for a single-replica Deployment: when a node is
going away, it brings up a stand-in pod, waits until that pod is genuinely
serving traffic, and only then lets the original leave. The second pod exists
This operator does the same for a single-replica Deployment or Argo Rollout:
when a node is going away, it brings up a stand-in pod, waits until that pod is
genuinely serving traffic, and only then lets the original leave. The second pod exists
for the couple of minutes a disruption is actually in flight.

```yaml
Expand Down Expand Up @@ -177,7 +182,8 @@ someone runs them on GKE and AKS.

| Field | Default | What it does |
|---|---|---|
| `targetRef.name` | required | The Deployment to protect, in the same namespace |
| `targetRef.kind` | `Deployment` | `Deployment` (`apps/v1`) or `Rollout` (`argoproj.io/v1alpha1`) |
| `targetRef.name` | required | The workload to protect, in the same namespace |
| `surgeReplicas` | 1 | How many stand-ins to add |
| `minSurgeTimeSeconds` | 60 | How long this workload realistically takes to start serving. If a termination deadline is nearer than this, the eviction is released immediately instead of being held |
| `readinessDeadlineSeconds` | 600 | How long to hold an eviction with no progress before giving up and letting it through |
Expand Down Expand Up @@ -207,6 +213,37 @@ If you run on spot without the sentinel, set `readinessDeadlineSeconds` below
the platform's notice window, otherwise the operator can hold an eviction
against a deadline it cannot see.

### Argo Rollouts

Protect an Argo Rollout by setting `targetRef.kind: Rollout`:

```yaml
apiVersion: apps.understudy.sh/v1alpha1
kind: Understudy
metadata:
name: tenant-worker
spec:
targetRef:
kind: Rollout
name: tenant-worker
holdMode: always
```

Both canary and blue/green strategies are supported once fully promoted.
Understudy scales the Rollout's `spec.replicas`; Argo Rollouts continues to
manage its ReplicaSets and traffic routing. Only ready pods from the stable
revision (or the blue/green active revision) count as stand-ins. Ready preview
pods and old revisions cannot release a production pod's eviction hold.

During an update, pause, or abort, Understudy stands down and admits evictions.
If a release begins during an Understudy surge, it restores the original replica
count and relinquishes the hold. Progressive delivery remains Argo's responsibility.

Argo Rollouts must already be installed. Its API is detected at operator startup;
restart Understudy if you install the Rollout CRD later. Clusters without Argo
Rollouts continue to support Deployments. As with Deployments, another reconciler
(such as an HPA or GitOps tool) must not overwrite the temporary replica increase.

## How it works

Briefly: answer the eviction at the door with HTTP 429, surge when its node is
Expand Down Expand Up @@ -272,7 +309,7 @@ including single-replica StatefulSets, and always will be.
## Development

```sh
make test # envtest suite, downloads Kubernetes binaries on first run
make test # envtest suite, including the upstream Argo Rollout CRD
make lint
make build
```
Expand Down
22 changes: 22 additions & 0 deletions UPGRADING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,28 @@

Version-to-version notes. Anything not mentioned here needs no action.

## 0.4.2 to 0.4.3

Argo Rollouts are now supported with `targetRef.kind: Rollout`. Both canary and
blue/green strategies are protected once fully promoted. Install Argo Rollouts
before starting Understudy, or restart Understudy after installing its CRD.
Existing Deployment targets need no configuration changes.

Upgrade the chart to install the Rollout permissions alongside the new image:

```sh
helm upgrade understudy oci://ghcr.io/kylan11/charts/understudy \
--namespace understudy-system --version 0.4.3
```

If you manage RBAC yourself (`rbac.create: false`), grant `get`, `list`, `watch`,
`update`, and `patch` on `rollouts` in the `argoproj.io` API group.

The eviction webhook now stands down together with the controller during a
target's release. Rollouts that are updating, paused, or aborted are not held;
preview pods cannot count as production replacements. See
[Argo Rollouts](README.md#argo-rollouts) for the full behavior.

## 0.4.0 to 0.4.1

`hostageMode` is now `holdMode`. The old spelling described a budget that
Expand Down
1 change: 1 addition & 0 deletions api/v1alpha1/understudy_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ const (
)

type TargetReference struct {
// Kind is Deployment (apps/v1) or Rollout (argoproj.io/v1alpha1).
// +kubebuilder:default=Deployment
// +optional
Kind string `json:"kind,omitempty"`
Expand Down
22 changes: 11 additions & 11 deletions charts/understudy/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ description: >-
downtime from announced node disruptions (drains, upgrades, consolidation,
spot reclaims) without a standing second replica.
type: application
version: 0.4.2
appVersion: "0.4.2"
version: 0.4.3
appVersion: "0.4.3"
kubeVersion: ">=1.34.0-0"
keywords:
- availability
Expand All @@ -16,6 +16,7 @@ keywords:
- node-drain
- karpenter
- cluster-autoscaler
- argo-rollouts
home: https://github.com/kylan11/understudy
sources:
- https://github.com/kylan11/understudy
Expand All @@ -40,16 +41,16 @@ annotations:
url: https://github.com/kylan11/understudy/issues
artifacthub.io/images: |
- name: understudy
image: ghcr.io/kylan11/understudy:0.4.2
image: ghcr.io/kylan11/understudy:0.4.3
artifacthub.io/crds: |
- kind: Understudy
version: v1alpha1
name: understudies.apps.understudy.sh
displayName: Understudy
description: >-
Protects a single-replica Deployment by bringing up a stand-in pod when
its node is doomed and holding the eviction until the stand-in serves
traffic.
Protects a single-replica Deployment or Argo Rollout by bringing up a
stand-in pod when its node is doomed and holding the eviction until
the stand-in serves traffic.
artifacthub.io/crdsExamples: |
- apiVersion: apps.understudy.sh/v1alpha1
kind: Understudy
Expand All @@ -63,8 +64,7 @@ annotations:
minSurgeTimeSeconds: 60
readinessDeadlineSeconds: 600
artifacthub.io/changes: |
- kind: security
description: golang.org/x/net, golang.org/x/text and google.golang.org/grpc updated past CVEs reachable from the probe client and the webhook server
links:
- name: Report
url: https://github.com/kylan11/understudy/issues/3
- kind: added
description: Protect fully promoted Argo Rollouts with canary and blue/green strategies using targetRef.kind Rollout
- kind: fixed
description: Admit evictions while standing down for progressive delivery and exclude preview revisions from Rollout replacement readiness
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ spec:
properties:
kind:
default: Deployment
description: Kind is Deployment (apps/v1) or Rollout (argoproj.io/v1alpha1).
type: string
name:
minLength: 1
Expand Down
3 changes: 3 additions & 0 deletions charts/understudy/templates/rbac.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["argoproj.io"]
resources: ["rollouts"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["policy"]
resources: ["poddisruptionbudgets"]
verbs: ["get", "list", "delete"]
Expand Down
2 changes: 2 additions & 0 deletions cmd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import (
"strings"
"time"

rolloutsv1alpha1 "github.com/argoproj/argo-rollouts/pkg/apis/rollouts/v1alpha1"
_ "k8s.io/client-go/plugin/pkg/client/auth"

"k8s.io/apimachinery/pkg/runtime"
Expand Down Expand Up @@ -53,6 +54,7 @@ func init() {
utilruntime.Must(clientgoscheme.AddToScheme(scheme))

utilruntime.Must(appsv1alpha1.AddToScheme(scheme))
utilruntime.Must(rolloutsv1alpha1.AddToScheme(scheme))
// +kubebuilder:scaffold:scheme
}

Expand Down
1 change: 1 addition & 0 deletions config/crd/bases/apps.understudy.sh_understudies.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ spec:
properties:
kind:
default: Deployment
description: Kind is Deployment (apps/v1) or Rollout (argoproj.io/v1alpha1).
type: string
name:
minLength: 1
Expand Down
10 changes: 10 additions & 0 deletions config/rbac/role.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,16 @@ rules:
- get
- patch
- update
- apiGroups:
- argoproj.io
resources:
- rollouts
verbs:
- get
- list
- patch
- update
- watch
- apiGroups:
- events.k8s.io
resources:
Expand Down
24 changes: 19 additions & 5 deletions docs/how-it-works.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# How Understudy works

Understudy protects a single-replica Deployment during announced node
disruptions without keeping a second replica running all the time.
Understudy protects a single-replica Deployment or Argo Rollout during announced
node disruptions without keeping a second replica running all the time.

## The mechanism

Expand All @@ -16,15 +16,15 @@ select a node without being blocked by a budget pre-check.
**2. Record the node as doomed.** Cordon, taint, eviction and optional cloud
sentinel adapters normalize their observations into a `NodeDoom`.

**3. Surge.** The controller increases the Deployment replica count and marks
**3. Surge.** The controller increases the target workload's replica count and marks
the old pod with a negative `controller.kubernetes.io/pod-deletion-cost`.

**4. Wait for traffic readiness.** A stand-in counts only when it is Ready, is
not deleting, and runs on neither the victim node nor another doomed node.
Readiness gates therefore include load balancer registration. Once enough
healthy stand-ins exist, the webhook admits the drainer's next retry.

**5. Scale back.** After the old pod leaves, the Deployment returns to its
**5. Scale back.** After the old pod leaves, the workload returns to its
original replica count. The deletion cost makes the old pod the preferred
scale-down target.

Expand All @@ -49,6 +49,20 @@ sequenceDiagram
U->>W: replicas back to 1
```

## Argo Rollouts

For Argo Rollouts, the same mechanism applies to fully promoted canary and
blue/green strategies. Pod selection additionally checks the
`rollouts-pod-template-hash` against `status.stableRS`, or against
`status.blueGreen.activeSelector` for blue/green. Preview pods cannot substitute
for production capacity. During progressive delivery, pauses, or aborts, the
controller and webhook stand down together. A release that starts during a surge
restores the saved replica count and clears deletion-cost hints.

The operator uses Argo's native replica scaling behavior and does not manage its
Services, traffic weights, analysis runs, or promotion steps. The Rollout watch
is registered only when the API is discovered at startup.

## Signals

The core consumes one provider-neutral value:
Expand Down Expand Up @@ -113,7 +127,7 @@ flowchart TD

| Phase | Meaning |
|---|---|
| `Idle` | No disruption is active; matching evictions are protected |
| `Idle` | No surge is active; check `Protected` for eviction protection (false during progressive delivery) |
| `Surging` | A doomed node was seen and the stand-in is not ready |
| `Releasing` | A healthy stand-in is ready and eviction retries are admitted |
| `Relaxed` | The hold was released without a ready stand-in |
Expand Down
11 changes: 11 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,16 @@
# Roadmap

## v0.4.3 - Argo Rollouts

Fully promoted canary and blue/green Rollouts can now be protected using
`targetRef.kind: Rollout`. Replica scaling uses Argo's native controller, and
only the serving revision counts toward replacement readiness. Progressive
delivery, pauses, and aborts cause the controller and webhook to stand down.
Argo Rollouts remains optional and is detected at operator startup.

Validated with Argo Rollouts v1.10.1 on an isolated Kubernetes 1.36.4 Kind
cluster, including eviction holds, scale-back, HTTP serving, and paused releases.

## v0.4.1 - holdMode

`hostageMode` is renamed to `holdMode`, the old field still accepted and
Expand Down
8 changes: 5 additions & 3 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
module github.com/kylan11/understudy

go 1.26.0
go 1.26.1

require (
github.com/argoproj/argo-rollouts v1.10.1
github.com/onsi/ginkgo/v2 v2.27.4
github.com/onsi/gomega v1.39.0
github.com/prometheus/client_golang v1.23.2
Expand Down Expand Up @@ -34,6 +35,7 @@ require (
github.com/go-openapi/jsonreference v0.20.2 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/google/cel-go v0.26.0 // indirect
github.com/google/gnostic-models v0.7.0 // indirect
github.com/google/go-cmp v0.7.0 // indirect
Expand All @@ -50,7 +52,7 @@ require (
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.19.2 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/spf13/cobra v1.10.2 // indirect
github.com/spf13/pflag v1.0.9 // indirect
github.com/stoewer/go-strcase v1.3.0 // indirect
Expand All @@ -66,7 +68,7 @@ require (
go.opentelemetry.io/proto/otlp v1.9.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.1 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 // indirect
golang.org/x/mod v0.38.0 // indirect
Expand Down
Loading
Loading