Emit estimated tasks per datastream as an INFO-level metric - #1048
Open
harshOSS wants to merge 1 commit into
Open
Emit estimated tasks per datastream as an INFO-level metric#1048harshOSS wants to merge 1 commit into
harshOSS wants to merge 1 commit into
Conversation
Add a per-datastream `estimatedNumTasks` gauge in StickyPartitionAssignmentStrategy that reports the estimated number of tasks calculated for each stream during elastic task assignment. Unlike the existing `numTasksCappedByMaxTasks` meter, which is only emitted when the estimate exceeds maxTasks, this gauge is emitted for every stream and records the uncapped estimate. An accompanying INFO log surfaces the same value per datastream. The metric is emitted from validateNumTasksAgainstMaxTasks, the common choke point shared by StickyPartitionAssignmentStrategy and LoadBasedPartitionAssignmentStrategy, so both strategies are covered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a per-datastream
estimatedNumTasksgauge metric (with an accompanying INFO log) that reports the estimated number of tasks calculated for each stream during elastic task assignment.numTasksCappedByMaxTasksmeter, which fires only when the estimate exceeds the datastream'smaxTasks. There was no visibility into the estimated task count for streams that were not capped.numTasks(which is capped atmaxTasks), and understand scaling behavior.StickyPartitionAssignmentStrategy.validateNumTasksAgainstMaxTasks()— the common code path shared by bothStickyPartitionAssignmentStrategy(partition-count-based estimate) andLoadBasedPartitionAssignmentStrategy(throughput/load-based estimate) — so both strategies are covered. The gauge is registered/declared/unregistered alongside the existing per-datastream gauges (numTasks,actualPartitionsPerTask).Metric name:
StickyPartitionAssignmentStrategy.<datastreamTaskPrefix>.estimatedNumTasksTesting Done
testElasticTaskPartitionAssignmentCreatesAtMostMaxTasksto assert the new gauge reports the uncapped estimate (13) whilenumTasksreports the capped actual (5)TestStickyPartitionAssignmentStrategy(15 tests) andTestLoadBasedPartitionAssignmentStrategypass under JDK 17checkstyleMainandcheckstyleTestpass🤖 Generated with GitHub Copilot CLI