[improve][broker] Optimize fine-grained concurrency control for BucketDelayedDeliveryTracker

[Denovo1998]

Fixes #24603

Main Issue: #24600

PIP: #xyz

Motivation

• Reduce lock contention and fix thread-safety issues in BucketDelayedDeliveryTracker.
• Remove blocking I/O from the publish path.
• Decouple the tracker from dispatcher for easier testing and benchmarking.

Modifications

• Concurrency
  • Replace StampedLock and synchronized with ReentrantReadWriteLock (read/write separation).
  • addMessage uses “read-lock check → write-lock modify” to minimize write lock time.
• Asynchronous snapshots
  • Seal-and-swap the current bucket and persist snapshots on a single-thread executor.
  • Ensure only one snapshot is created at a time.
• Scheduling and timer
  • getScheduledMessages triggers async loading of next snapshot segment instead of blocking.
  • Reworked timer handling with a dedicated lock and scheduleImmediateRun; no synchronization on dispatcher.
• Decoupling
  • Introduce DelayedDeliveryContext:
    • DispatcherDelayedDeliveryContext (production).
    • NoopDelayedDeliveryContext (tests/benchmarks).
  • Trackers and factories accept dispatcher or context; added constructors for name+cursor.
• Merging and lifecycle
  • asyncMergeBucketSnapshot uses write lock; merges bitmaps; removes old buckets; improved stats/logging.
  • clear() and close() under write lock; wait for snapshot futures outside locks; shutdown snapshot executor.
• Tests and benchmarks
  • Expanded unit and thread-safety tests; made MockManagedCursor public.
  • New JMH suite (BucketDelayedDeliveryTrackerBenchmark) and MockBucketSnapshotStorage; removed old simple benchmark.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: Denovo1998#10

[Denovo1998]

@codelipenghui @lhotari @coderzc @Apurva007
How does this copy on write method look, is my direction correct?
There is still much work to be done, and I will convert this PR to a draft later.

[lhotari]

@codelipenghui @lhotari @coderzc @Apurva007
How does this copy on write method look, is my direction correct?
There is still much work to be done, and I will convert this PR to a draft later.

Without benchmarking, it will be hard to validate assumptions. I'd suggest adding JMH benchmarks so that performance could be compared.

JMH benchmarks can be added to microbench module: https://github.com/apache/pulsar/tree/master/microbench

[Denovo1998]

OK. I will add JMH benchmarks first.

[lhotari]

Is this still relevant?

[Denovo1998]

@lhotari Recently, there have been many changes about delaying messages. I will start to re-study this pr next week.

[Denovo1998]

@lhotari
I have modified a version locally, and I think I may need to wait for these two pr to merge before continuing.

#26401

#26420

Let‘s finish the review and merge of these two pr first.