Skip to content

[fix][broker] PIP-468: don't let namespace consumers auto-recreate deleted scalable topics#26066

Open
merlimat wants to merge 1 commit into
apache:masterfrom
merlimat:mmerli/scalable-ns-consumer-no-recreate
Open

[fix][broker] PIP-468: don't let namespace consumers auto-recreate deleted scalable topics#26066
merlimat wants to merge 1 commit into
apache:masterfrom
merlimat:mmerli/scalable-ns-consumer-no-recreate

Conversation

@merlimat

Copy link
Copy Markdown
Contributor

Motivation

PIP-468 scalable topics auto-create on lookup when broker/namespace policy allows. A V5 namespace
(multi-topic) consumer opens a per-topic DAG-watch for each scalable topic matching its property
filters. When a topic is deleted, the namespace watcher notifies the consumer, which stops the
per-topic consumer for it — but a per-topic watch that reconnects during the deletion teardown
window would re-run the lookup and, with auto-creation enabled, resurrect the just-deleted
topic
. Relying on the close-vs-reconnect race to avoid this is not a guarantee.

A namespace consumer should only ever attach to topics that already exist (the watcher tells it
which); it must never create one.

Modifications

  • Add an optional create_if_missing field (default true) to CommandScalableTopicLookup. The
    default preserves the auto-create-on-lookup behavior for explicit single-topic producers/consumers
    and for older clients that don't set the field.
  • DagWatchClient carries the flag; the V5 namespace consumer (MultiTopicQueueConsumer) opens its
    per-topic watches with create_if_missing=false.
  • Broker DagWatchSession honors it: a topic:// lookup of a missing topic with
    create_if_missing=false fails not-found before consulting policy or creating anything — so a
    deleted topic can never be resurrected by a reconnecting namespace-consumer watch, deterministically
    rather than by winning a race.

Scope: this gates the lookup path used by the queue namespace consumer. The stream namespace
consumer uses a different subscribe path (ScalableTopicSubscribe → controller) that does not go
through auto-create-on-lookup.

Verifying this change

  • DagWatchSessionTest.testStartDoesNotAutoCreateWhenCallerOptsOut — broker-side gating: with
    create_if_missing=false, a missing topic:// lookup fails not-found and never consults policy or
    creates the topic.
  • V5NamespaceConsumerTopicDeletionTest (new) — namespace consumer with property filters: only
    matching topics attach; deleting a matched topic stops its per-topic consumer; the deleted topic
    stays gone across auto-discovery cycles (broker auto-creation enabled, so this would fail if the
    consumer resurrected it).
  • Regression: single-topic auto-create (V5ScalableTopicAutoCreateTest) and the multi-topic
    consumers (V5MultiTopicQueueConsumerTest, V5MultiTopicStreamConsumerTest,
    V5ScalableTopicsWatcherTest) are unaffected.

…leted scalable topics

PIP-468 scalable topics auto-create on lookup when broker/namespace policy allows. A V5
namespace (multi-topic) consumer opens a per-topic DAG-watch for each matching topic; when a
topic is deleted the namespace watcher tells the consumer to drop it, but a per-topic watch
that reconnects during the teardown window would re-run the lookup and, with auto-creation
enabled, resurrect the just-deleted topic. Relying on the close-vs-reconnect race is not a
guarantee.

Make it deterministic with a create-if-missing flag on the per-topic connect:

- Add optional 'create_if_missing' (default true) to CommandScalableTopicLookup. The default
  preserves auto-create-on-lookup for explicit single-topic producers/consumers and for older
  clients that don't set it.
- DagWatchClient carries the flag; the namespace consumer (MultiTopicQueueConsumer) opens its
  per-topic watches with create_if_missing=false.
- DagWatchSession honors it: a topic:// lookup of a missing topic with create_if_missing=false
  fails not-found before consulting policy or creating, so a deleted topic is never resurrected
  by a reconnecting namespace-consumer watch.

Scope: gates the lookup path used by the queue namespace consumer. The stream namespace
consumer uses a different subscribe path that does not go through auto-create-on-lookup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant