feat: add support for dynamic KRaft quorum scaling by razvan · Pull Request #1010 · stackabletech/kafka-operator

razvan · 2026-08-18T12:51:47Z

Description

See CHANGELOG for a high level view of everything that changed.

✅ : OKD integration tests work. I ran them many times but there was always one or two tests that failed due to flakiness. I added some cleanup steps so that kuttl doesn't timeout during namespace cleanups.

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

Controllers now run a quorum-manager sidecar that admits itself into the KRaft voter set on startup (add-controller) and removes itself before termination (remove-controller via preStop), so controller role groups can be scaled up/down on a running cluster without a full rolling restart. controller.quorum.bootstrap.servers now points at each controller role group's headless Service DNS name instead of individual pod addresses, keeping container commands stable across replica changes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Controllers get a startupProbe (plain TCP, generous failure threshold for slow metadata-log replay on boot), a plain-TCP livenessProbe, and a readinessProbe that checks the node's Raft state via its metrics endpoint instead of a bare TCP check, so a controller stuck rejoining the quorum is correctly reported as not ready. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Controller pods now start/scale sequentially (OrderedReady) instead of in parallel, since the quorum-manager sidecar's admission flow assumes one voter joins at a time. Brokers are unaffected and keep Parallel. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Scaling controllers to 0 replicas while brokers keep running is now rejected at validation time with an actionable error, instead of failing much later and confusingly while building the broker's ConfigMap. Scaling controllers and brokers to 0 together (a coordinated whole-cluster stop) is still allowed and now actually builds, since downstream resource builders no longer assume a non-empty controller quorum whenever KRaft mode is active. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…ns tests Enable the previously version-gated scale-up/down steps (Kafka 3.7 no longer needs special-casing), assert quorum voter counts via kafka-metadata-quorum.sh after each scale, and add a final step scaling both controllers and brokers to 0 to exercise the whole-cluster-stop path before namespace teardown. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Update the KRaft controller usage guide for scale-up/down support, record the design spec and implementation plan, add the CHANGELOG entries for this branch's changes, and ignore .worktrees/ for local worktree checkouts. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

razvan · 2026-08-18T15:53:34Z

The failed test was due to namespace deletion timeout

--- FAIL: kuttl (2409.35s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/logging_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false (134.90s)
        --- FAIL: kuttl/harness/smoke-kraft_kafka-kraft-4.2.1_openshift-false (634.74s)
        --- PASS: kuttl/harness/smoke_kafka-3.9.2_zookeeper-3.9.5_use-client-tls-false_openshift-false (81.56s)
        --- PASS: kuttl/harness/upgrade_upgrade_old-3.9.2_upgrade_new-4.2.1_use-client-tls-false_use-client-auth-tls-false_openshift-false (120.71s)
        --- PASS: kuttl/harness/kerberos_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-external-unstable (113.64s)
        --- PASS: kuttl/harness/tls_kafka-3.9.2_zookeeper-latest-3.9.5_use-client-tls-false_use-client-auth-tls-false_openshift-false (33.64s)
        --- PASS: kuttl/harness/cluster-operation_kafka-latest-3.9.2_zookeeper-latest-3.9.5_openshift-false (41.49s)
        --- PASS: kuttl/harness/configuration_kafka-latest-3.9.2_openshift-false (11.45s)
        --- PASS: kuttl/harness/operations-kraft_kafka-kraft-4.2.1_openshift-false (1079.48s)
        --- PASS: kuttl/harness/opa_kafka-latest-3.9.2_zookeeper-latest-3.9.5_opa-latest-1.16.2_use-opa-tls-false_openshift-false_krb5-1.21.1 (128.33s)
        --- PASS: kuttl/harness/delete-rolegroup_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false (29.40s)
FAIL
ERROR:root:kuttl failed

…ter-membership

maltesander · 2026-08-25T06:09:31Z

+/// Only adding or removing a whole role group changes this list.
 pub(crate) fn kraft_controllers(pod_descriptors: &[KafkaPodDescriptor]) -> Vec<String> {
-    pod_descriptors
+    let mut role_group_addresses: Vec<String> = pod_descriptors


Why not use a BTreeSet here? No dedup or sort required in the end?

maltesander · 2026-08-25T10:15:48Z

+  with `kafka-storage.sh format --standalone`, bootstrapping a single-node quorum by itself.
+  Every other controller formats with `--no-initial-controllers` and joins purely through the sidecar's `add-controller` call.
+  Brokers always format with `--no-initial-controllers` too; they are never voters.
+  the current replica count.


Seems lost!

Suggested change

the current replica count.

maltesander · 2026-08-25T10:16:31Z

+  The property `controller.quorum.bootstrap.servers` now contains the headless service names
+  of all controller role groups instead of individual peer host names. This prevevents the
+  restart controller from restarting all pods in the quorum when a new one is added/deleted.
+  The controller `StatefulSet` is now scaled using `OrderedBy` instead of the `Parallel` strategy


Suggested change

The controller `StatefulSet` is now scaled using `OrderedBy` instead of the `Parallel` strategy

The controller `StatefulSet` is now scaled using `OrderedReady` instead of the `Parallel` strategy

maltesander · 2026-08-25T10:16:51Z

+  On termination, a new `preStop` hook on the controller container (`kafka`) removes the pod from
+  the voter list before shutdown.
+  The property `controller.quorum.bootstrap.servers` now contains the headless service names
+  of all controller role groups instead of individual peer host names. This prevevents the


Suggested change

of all controller role groups instead of individual peer host names. This prevevents the

of all controller role groups instead of individual peer host names. This prevents the

maltesander · 2026-08-27T07:35:59Z

+        "-c".to_string(),
+        format!(
+            "timeout 2 bash -c 'cat < /dev/null > /dev/tcp/localhost/{client_port}' || exit 1\n\
+             state=$(curl -s --max-time 2 localhost:{metrics_port}/metrics | grep -oE 'kafka_server_raft_metrics_current_state\\{{state=\"[a-z]+\"\\}}' | grep -oE '\"[a-z]+\"' | tr -d '\"')\n\


The quorum-manager loop logs Could not determine local Raft state. Missing the trailing comma.

Suggested change

state=$(curl -s --max-time 2 localhost:{metrics_port}/metrics | grep -oE 'kafka_server_raft_metrics_current_state\\{{state=\"[a-z]+\"\\}}' | grep -oE '\"[a-z]+\"' | tr -d '\"')\n\

state=$(curl -s --max-time 2 localhost:{metrics_port}/metrics | grep -oE 'kafka_server_raft_metrics_current_state\\{{state=\"[a-z]+\",?\\}}' | grep -oE '\"[a-z]+\"' | tr -d '\"')\n\

maltesander · 2026-08-27T08:24:05Z

+          && cat /tmp/{controller_properties_file} {admin_client_config} > {add_controller_config}; then
+          echo "Starting KRaft voter admission loop against bootstrap servers: $BOOTSTRAP_SERVERS"
+          while true; do
+            state=$(curl -s --max-time 5 --connect-timeout 2 localhost:{metrics_port}/metrics | grep -oE 'kafka_server_raft_metrics_current_state\{{state="[a-z]+"\}}' | grep -oE '"[a-z]+"' | tr -d '"')


Suggested change

state=$(curl -s --max-time 5 --connect-timeout 2 localhost:{metrics_port}/metrics | grep -oE 'kafka_server_raft_metrics_current_state\{{state="[a-z]+"\}}' | grep -oE '"[a-z]+"' | tr -d '"')

state=$(curl -s --max-time 5 --connect-timeout 2 localhost:{metrics_port}/metrics | grep -oE 'kafka_server_raft_metrics_current_state\{{state="[a-z]+",?\}}' | grep -oE '"[a-z]+"' | tr -d '"')

maltesander · 2026-08-27T11:08:54Z

+  image:
+{% if test_scenario['values']['kafka-kraft'].find(",") > 0 %}
+    custom: "{{ test_scenario['values']['kafka-kraft'].split(',')[1] }}"
+    productVersion: "{{ test_scenario['values']['kafka-kraft'].split(',')[0] }}"
+{% else %}
+    productVersion: "{{ test_scenario['values']['kafka-kraft'] }}"
+{% endif %}
+    pullPolicy: IfNotPresent
+  clusterConfig:
+    metadataManager: kraft
+{% if lookup('env', 'VECTOR_AGGREGATOR') %}
+    vectorAggregatorConfigMapName: vector-aggregator-discovery
+{% endif %}
+  brokers:
+    config:
+      logging:
+        enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
+    roleGroups:
+      default:
+        replicas: 3
+  controllers:
+    config:
+      logging:
+        enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
+    roleGroups:
+      default:
+        replicas: 3


Should not be required.

Suggested change

image:

{% if test_scenario['values']['kafka-kraft'].find(",") > 0 %}

custom: "{{ test_scenario['values']['kafka-kraft'].split(',')[1] }}"

productVersion: "{{ test_scenario['values']['kafka-kraft'].split(',')[0] }}"

{% else %}

productVersion: "{{ test_scenario['values']['kafka-kraft'] }}"

{% endif %}

pullPolicy: IfNotPresent

clusterConfig:

metadataManager: kraft

{% if lookup('env', 'VECTOR_AGGREGATOR') %}

vectorAggregatorConfigMapName: vector-aggregator-discovery

{% endif %}

brokers:

config:

logging:

enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}

roleGroups:

default:

replicas: 3

controllers:

config:

logging:

enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}

roleGroups:

default:

replicas: 3

maltesander · 2026-08-27T11:16:14Z

+        set -uo pipefail
+        {derive_pod_index}
+        [ -n "$POD_INDEX" ] || exit 0
+        {export_replica_id}
+        {extract_bootstrap_servers}
+        DEADLINE=$((SECONDS + {deadline_seconds}))
+        finished=false
+        while [ "$SECONDS" -lt "$DEADLINE" ]; do
+          describe=$(timeout --kill-after={cli_kill_after} {cli_timeout} {binary} --bootstrap-controller "$BOOTSTRAP_SERVERS" --command-config {config} describe --replication 2>/dev/null)
+          if [ -n "$describe" ]; then
+            voters=$(echo "$describe" | tail -n +2 | awk '$NF == "Leader" || $NF == "Follower"')
+            total_voters=$(echo "$voters" | grep -c .)
+            if [ "$total_voters" -gt 0 ]; then
+              remaining_after_removal=$(( total_voters - 1 ))
+              if [ "$remaining_after_removal" -ge 1 ]; then
+                directory_id=$(echo "$voters" | awk -v id="$REPLICA_ID" '$1 == id {{ print $2 }}')
+                if [ -n "$directory_id" ]; then
+                  echo "Removing self (node $REPLICA_ID, directory $directory_id) from the voter set..."
+                  if timeout --kill-after={cli_kill_after} {cli_timeout} {binary} --bootstrap-controller "$BOOTSTRAP_SERVERS" --command-config {config} remove-controller \
+                    --controller-id "$REPLICA_ID" --controller-directory-id "$directory_id"; then
+                    finished=true
+                  else
+                    echo "remove-controller attempt failed, will retry if time remains"
+                  fi
+                else
+                  echo "Could not find own node $REPLICA_ID among current voters (already removed?), nothing to do"
+                  finished=true
+                fi
+              else
+                echo "Removing self would leave zero voters, skipping (this can't become safe later during my own termination -- nothing else will add a voter for me)"
+                finished=true
+              fi
+              [ "$finished" = true ] && break
+            else
+              echo "Could not identify any voters in the describe output (unrecognized format), skipping removal for safety and retrying..."
+            fi
+          fi


This has a level of complexity and nesting (6 levels) we should not use in code like this. It is not really tested as well. I would try to early exit (resolve the nesting), add a couple of comments or extract a script from this. I would use formatdoc! as well rather than rawstring.

razvan and others added 6 commits August 18, 2026 14:24

razvan self-assigned this Aug 18, 2026

Cleanup inline comments and documentation

fc21e2e

razvan marked this pull request as draft August 18, 2026 15:57

razvan added 3 commits August 19, 2026 14:03

Remove the custom bash trap functions

d02b920

Comments abd documentation cleanups

62adbff

Merge remote-tracking branch 'origin/main' into feat/kraft-dynamic-vo…

95839be

…ter-membership

razvan mentioned this pull request Aug 20, 2026

fix: kraft kerberos support #999

Draft

20 tasks

razvan added 2 commits August 20, 2026 10:04

Cleanup CHANGELOG and KRaft docs.

c4cb3a5

Remove the kcat-prober container and consolidate container probes

4b42def

razvan marked this pull request as ready for review August 20, 2026 11:10

razvan added 2 commits August 20, 2026 14:30

Cleanups

0596a0a

Stop Kafka to prevent kuttl namespace deletion timeouts

ed3180b

razvan added this to Stackable Engineering Aug 21, 2026

razvan moved this to Development: Waiting for Review in Stackable Engineering Aug 21, 2026

maltesander self-requested a review September 1, 2026 11:40

maltesander moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Sep 1, 2026

maltesander requested changes Sep 1, 2026

View reviewed changes

Merge branch 'main' into feat/kraft-dynamic-voter-membership

6276e1f

Navigation Menu

Uh oh!

Uh oh!

feat: add support for dynamic KRaft quorum scaling - #1010

feat: add support for dynamic KRaft quorum scaling#1010
razvan wants to merge 15 commits into
mainfrom
feat/kraft-dynamic-voter-membership

razvan commented Aug 18, 2026 •

edited

Loading

Uh oh!

razvan commented Aug 18, 2026

Uh oh!

maltesander Aug 25, 2026

Uh oh!

maltesander Aug 25, 2026

Uh oh!

maltesander Aug 25, 2026

Uh oh!

maltesander Aug 25, 2026

Uh oh!

maltesander Aug 27, 2026

Uh oh!

maltesander Aug 27, 2026

Uh oh!

maltesander Aug 27, 2026

Uh oh!

maltesander Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	The controller `StatefulSet` is now scaled using `OrderedBy` instead of the `Parallel` strategy
	The controller `StatefulSet` is now scaled using `OrderedReady` instead of the `Parallel` strategy

	of all controller role groups instead of individual peer host names. This prevevents the
	of all controller role groups instead of individual peer host names. This prevents the

	state=$(curl -s --max-time 2 localhost:{metrics_port}/metrics \| grep -oE 'kafka_server_raft_metrics_current_state\\{{state=\"[a-z]+\"\\}}' \| grep -oE '\"[a-z]+\"' \| tr -d '\"')\n\
	state=$(curl -s --max-time 2 localhost:{metrics_port}/metrics \| grep -oE 'kafka_server_raft_metrics_current_state\\{{state=\"[a-z]+\",?\\}}' \| grep -oE '\"[a-z]+\"' \| tr -d '\"')\n\

	state=$(curl -s --max-time 5 --connect-timeout 2 localhost:{metrics_port}/metrics \| grep -oE 'kafka_server_raft_metrics_current_state\{{state="[a-z]+"\}}' \| grep -oE '"[a-z]+"' \| tr -d '"')
	state=$(curl -s --max-time 5 --connect-timeout 2 localhost:{metrics_port}/metrics \| grep -oE 'kafka_server_raft_metrics_current_state\{{state="[a-z]+",?\}}' \| grep -oE '"[a-z]+"' \| tr -d '"')

Uh oh!

Uh oh!

Conversation

razvan commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

razvan commented Aug 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

razvan commented Aug 18, 2026 •

edited

Loading