Repository navigation
Procedure for shutting down a VKS cluster and sample scripts - #16
Conversation
…he user that they have completed the prerequisites. Cleaned up comments.
…hutdown and restart script requires it and does not have a default.
🛑 Legal Compliance Check FailedHi @dsu-igeek, thank you for your contribution! To merge this Pull Request, you must sign our CLA. Note: Even if you signed off your commits locally (using 1. Read the Document: Click here to read the CLA ⏳ Processing Schedule: |
|
I have read the CLA Document and I hereby sign the CLA for this and all future contributions. |
|
|
||
| # Shutdown of a VKS Workload Cluster | ||
|
|
||
| Shutdown of a VKS Workload Cluster is the process of powering off all of the Control Plane and Worker Node VMs. |
There was a problem hiding this comment.
thought: What happens if the Supervisor or VKS is upgraded, or someone edits the Cluster, during the shutdown? Perhaps it'd be useful to include a section to tell readers not to perform some set of operations during the outage.
| ORIGINAL_CONTROL_PLANE_REPLICAS=$( | ||
| kubectl get cluster "$CLUSTER_NAME" -n "$KUBENAMESPACE" \ | ||
| -o jsonpath='{.spec.topology.controlPlane.replicas}' | ||
| ) | ||
| echo "Original number of control plane replicas: ${ORIGINAL_CONTROL_PLANE_REPLICAS}" | ||
| echo "Note this number - restart-vks-cluster.sh requires it: ./restart-vks-cluster.sh ${CLUSTER_NAME} ${ORIGINAL_CONTROL_PLANE_REPLICAS}" |
There was a problem hiding this comment.
suggestion: Would it make sense to persist this (e.g., as an annotation on the Cluster, written if not already present) and have restart-vks-cluster.sh read it (and then clear it)? Printing to the terminal is probably fine, but this might reduce opportunity for mistakes (e.g., if terminal output is lost between shutdown and restart) and reduce the inputs required to restart.
| if [[ -n "$in_progress" ]]; then | ||
| echo "Cluster ${CLUSTER_NAME} has operation(s) in progress (${in_progress}) - aborting" >&2 | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
question: Do we need to either document that this flow only works on VKS versions that include Cluster API v1beta2 or fallback to reading from .status.v1beta2.conditions?
| confirm_prerequisites | ||
|
|
||
| # Verifies every required env var is set and usable | ||
| check_env() { |
There was a problem hiding this comment.
question: Would it be better to do the prechecking before we prompt the user to confirm they've completed the prerequisites?
| # shellcheck disable=SC2034 # out_ref is a nameref: assignment writes to the caller's array | ||
| get_provider_ids_array() { | ||
| local selector="$1" | ||
| local -n out_ref=$2 |
There was a problem hiding this comment.
question: Does local -n create any compatibility constraints? (Do we need to check for a sufficiently new bash version at the beginning of the script?)
| fi | ||
|
|
||
| echo "Verifying access to the Supervisor cluster (\$KUBECONFIG)..." | ||
| if ! kubectl get namespace "$KUBENAMESPACE" >/dev/null; then |
There was a problem hiding this comment.
question: Should we explicitly check for the necessary tools (kubectl, govc, jq, etc.) so we can give clear errors if any are missing?
| # grep -v exits 1 (no non-matching lines) when every pod is healthy - | ||
| # that's the success case, not an error, so `|| true` keeps set -e from | ||
| # treating it as one. | ||
| bad_pods=$(KUBECONFIG="$WORKLOAD_KUBECONFIG" kubectl get pods -A --no-headers 2>/dev/null | grep -viE "running|completed" || true) |
There was a problem hiding this comment.
suggestion: grep -viE "running|completed" treats a pod as healthy if the line contains either word anywhere (e.g., even in a pod name). Since this is the final "all good" signal for the restart, could we check readiness more explicitly, e.g. kubectl wait --for=condition=Ready pod --all -A or something along those lines.
| KCP_NAME=$( | ||
| kubectl get kubeadmcontrolplane -n "$KUBENAMESPACE" \ | ||
| -l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}" \ | ||
| -o jsonpath='{.items[0].metadata.name}' | ||
| ) | ||
|
|
||
| if [[ -z "$KCP_NAME" ]]; then | ||
| echo "Could not find KubeadmControlPlane for cluster '${CLUSTER_NAME}' in namespace '${KUBENAMESPACE}'" >&2 |
There was a problem hiding this comment.
nitpick: Seems like anything that causes kubectl to exit with an error code (e.g., .items not containing an element) is going to cause set -e to exit before we get to the friendly error message. Maybe -o name avoids this?
| echo "$(date -u +%H:%M:%S) - still waiting on unhealthy pods:" | ||
| echo "$bad_pods" | ||
| sleep 10 | ||
| done |
There was a problem hiding this comment.
suggestion: Since the user is told to save PDB copies as a prerequisite, would it be useful to include a closing reminder ("Remember to restore any Pod Disruption Budgets you removed before shutdown")?
Documents the procedure for shutting down a VKS cluster. Also contains sample scripts that will do much of the procedure.