Reduction of event publishing blocking time when Apache Kafka is unavailable

[erikbocks]

Description

Apache CloudStack offers integrations with several messaging services, one of which is Apache Kafka. During service configuration on the platform, properties are defined that will be passed to the producer, which is responsible for publishing events. Among these properties, the most important are:

• bootstrap.servers: List of IP addresses and ports of the brokers (services responsible for receiving messages and serving them to consumers);
• topic: Message storage unit;
• key.serializer and value.serializer: classes responsible for serializing and deserializing the bytes managed by the brokers.

Currently, the only properties managed by Apache CloudStack are topic, key.serializer, and value.serializer. The topic property has a default value of cloudstack, and the <key|value>.serializer properties have the value org.apache.kafka.common.serialization.StringSerializer. Other properties can be defined in the kafka.producer.properties file, and these are passed automatically.

However, it was noticed that if one of the brokers present in bootstrap.server is unavailable, message publishing is blocked until a timeout occurs. This blockage culminates in the exhaustion of the available processing thread pool, which in turn causes several other errors. Upon analyzing the message publishing flow, it was observed that the error with the unavailable broker is caused by a search for Kafka cluster metadata, which occurs during the first publication. This search is not asynchronous and aims to return data such as the number of available brokers, topics, and cluster partitions, which are cached and used in message sending. Because the configured broker is unavailable, the thread is stuck for 60000 ms (1 minute).

This value is defined by the max.block.ms setting, which has a default value of 60000 ms, and is responsible for defining the maximum time that the metadata retrieval from the cluster can take and how long the producer should wait for free space in the buffer to write the message.

To prevent errors like this from occurring, a default timeout of 2500 ms (2.5s) has been defined if the property has not been set in the configuration file. This is a palliative change, in order to prevent the pool exhaustion. In the future, I pretend to find another way to prevent the error without having to change the max.block.ms property.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Screenshots (if appropriate):

How Has This Been Tested?

I created a Kafka cluster following the official guide (see Kafka Docker image), and validated that even using the new default value of the max.block.ms configuration, ACS still could communicate correctly with Kafka.

After that, I configured a local Kafka cluster with three nodes, and configured them into ACS. Then, I shut one of the cluster nodes down, and executed an operation that generates an event. When ACS tried to reach the unavailable node, it waited for 2.5 seconds and then move on to the next one.

[Copilot]

🔵 Needs a closer look

Kafka timeout failures must be caught or made explicitly best-effort before approval.

Pull request overview

Reduces Kafka event publishing delays when brokers are unavailable by defaulting max.block.ms to 2500 ms.

Changes:

• Adds a 2.5-second default producer timeout.
• Preserves explicitly configured values.

File summaries

File	Summary	Finding
plugins/event-bus/kafka/src/main/java/org/apache/cloudstack/mom/kafka/KafkaEventBus.java	Applies the default Kafka producer timeout.	Moderate (1 vote): producer timeouts may escape publish() and fail CloudStack operations.

Review details

Suppressed comments (1)

plugins/event-bus/kafka/src/main/java/org/apache/cloudstack/mom/kafka/KafkaEventBus.java:77

• This shortens the wait but does not make publishing continue after the timeout: when metadata is unavailable, KafkaProducer.send() throws an unchecked TimeoutException after max.block.ms, while EventDistributorImpl only catches EventBusException. Consequently, an unavailable broker can still escape eventDistributor.publish() and fail the CloudStack operation after 2.5 seconds; catch/wrap the producer failure in publish() (or otherwise make this failure explicitly best-effort) so the new timeout does not turn broker unavailability into request failures.

            if (!props.containsKey("max.block.ms")) {
                props.put("max.block.ms", DEFAULT_MAX_BLOCK_MS);
            }

• Files reviewed: 1/1 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.91%. Comparing base (602d9ec) to head (3abcc15).

Files with missing lines	Patch %	Lines
...org/apache/cloudstack/mom/kafka/KafkaEventBus.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14209      +/-   ##
============================================
- Coverage     19.91%   19.91%   -0.01%     
  Complexity    20200    20200              
============================================
  Files          6373     6373              
  Lines        577230   577232       +2     
  Branches      70696    70697       +1     
============================================
- Hits         114974   114964      -10     
- Misses       449690   449706      +16     
+ Partials      12566    12562       -4     

Flag	Coverage Δ
uitests	3.71% <ø> (ø)
unittests	21.18% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.