From 11a4a13f3c559fc4fab9d2e84221c768921d078f Mon Sep 17 00:00:00 2001 From: Matteo Merli Date: Tue, 1 Sep 2026 17:08:33 -0700 Subject: [PATCH] =?UTF-8?q?[improve][pip]=20PIP-494:=20Scalable=20Topics?= =?UTF-8?q?=20Client=20Specification=20=E2=80=94=20authoritative=20referen?= =?UTF-8?q?ce=20and=20change=20process?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Establish the Scalable Topics Client Specification (spec/scalable-topics/) as the authoritative, language-neutral reference for the client API, its semantics, and the protocol-level interactions — the base document for any client SDK adding scalable-topics support — and define the process that governs it: normative changes only via a PIP whose spec edits land with it, the Java V5 client as reference implementation, MAJOR.MINOR versioning tied to the Pulsar LTS cadence, Stable/Experimental/Deprecated tiers with an LTS-anchored deprecation/removal mechanism, and conformance obligations for new SDKs. The current work-in-progress draft is adopted as version 1.0. Assisted-by: Claude Code (Fable 5.1) --- pip/pip-494.md | 233 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 233 insertions(+) create mode 100644 pip/pip-494.md diff --git a/pip/pip-494.md b/pip/pip-494.md new file mode 100644 index 0000000000000..1b5b738173bd3 --- /dev/null +++ b/pip/pip-494.md @@ -0,0 +1,233 @@ +# PIP-494: Scalable Topics Client Specification — authoritative reference and change process + +*Sub-PIP of [PIP-460: Scalable Topics](pip-460.md)* + +> **Draft.** This PIP establishes a **process**; it deliberately does not restate the specification's +> content, which lives in the repository at `spec/scalable-topics/`. The current work-in-progress +> specification — +> — will be +> adopted as **version 1.0** of the specification. It reflects the current client API and +> semantics as designed in the scalable-topics PIPs ([PIP-460](pip-460.md), [PIP-468](pip-468.md), +> [PIP-483](pip-483.md), [PIP-486](pip-486.md)) and as implemented on the `master` branch. + +## Motivation + +Scalable topics ([PIP-460](pip-460.md) and its sub-PIPs — the controller [PIP-468](pip-468.md), auto +split/merge [PIP-483](pip-483.md), key-shared consumption [PIP-486](pip-486.md)) introduce a client model +that is materially different from classic Pulsar topics: no application-visible partitions, a dynamic +per-cluster segment layout the client must track, client-side key routing, three distinct consumer +modes with different ordering/acknowledgment contracts, a checkpoint position type, a controller-driven +consumer-assignment protocol, and a family of new binary-protocol commands. + +Today the only complete description of that client behavior is the **Java V5 client's source code**, with +the design rationale scattered across the PIPs. That is not a workable basis for the ecosystem: + +- **Other-language SDKs** (Go, Python, C++, Rust, Node, .NET) need a precise, language-neutral definition + of the API contract, the required client-side mechanisms, and the exact wire interactions — not a Java + implementation to reverse-engineer. +- **Interoperability** depends on every client routing keys identically, applying layouts and assignments + with the same staleness rules, and honoring the same ordering and acknowledgment semantics. Without a + normative reference, divergent clients are inevitable and hard to diagnose. +- **PIPs are design proposals, not contracts.** They explain *why* and record decisions, but they are + point-in-time documents that are never updated after acceptance. The living behavior needs a living, + normative document. +- **Stability and deprecation** need defined rules so SDK authors know what they can rely on across + Pulsar releases. + +This PIP establishes a **Scalable Topics Client Specification** as the single authoritative description +of the client API, its semantics, and its protocol-level interactions — the base document for anyone +implementing scalable-topics support in a Pulsar client SDK — and defines the **process** by which that +specification is maintained and changed. + +## Background knowledge + +**PIPs vs. a specification.** A PIP proposes and ratifies a change; once merged it is a historical +record of a decision. A specification is a *current* normative statement of behavior that implementations +conform to, and it evolves as behavior evolves. Mature cross-implementation ecosystems keep both: the +OpenTelemetry project, for example, maintains a versioned specification with explicit stability tiers and +a formal change process, separate from the proposals that motivated each change. This PIP adopts that +model for scalable topics. + +**Reference implementation.** The Java V5 client (`pulsar-client-api-v5`, `pulsar-client-v5`) is the +reference implementation of scalable-topics client behavior. The specification is derived from it and +verified against it, but the specification — not the Java code — is the source of truth for conformance. + +## Goals + +### In Scope + +- Establish `spec/scalable-topics/` in the `apache/pulsar` repository as the **authoritative, + language-neutral specification** for scalable-topics clients: the API contract, its semantics, the + required client-side mechanisms, and the wire-protocol interactions. +- Define the specification as the **required base for any new client SDK** adding scalable-topics + support, and the reference against which SDK **conformance** is judged. +- Define the **change process**: how normative changes to the specification are proposed, reviewed, and + landed, and how the specification is kept in sync with the reference implementation. +- Define the specification's **versioning, stability tiers, and deprecation/removal rules**, aligned with + the Apache Pulsar release (LTS) cadence. + +### Out of Scope + +- The specification's content itself. It is a separate deliverable (already drafted) and is not + restated here; this PIP governs it, it does not define client behavior. +- Client behavior for classic (non-scalable) topics — the specification covers scalable topics and their + migration bridge from classic topics only. +- The mechanics of publishing the specification to the user-facing documentation site + (`apache/pulsar-site`); that is a follow-up task once the process is in place. + +## High Level Design + +### What is established + +1. **Location and status.** The specification lives at `spec/scalable-topics/` in `apache/pulsar`, as a + set of Markdown documents with a top-level index (`README.md`). It is a first-class artifact of the + project, maintained alongside the code it describes. + +2. **Authority.** For scalable-topics client behavior, the specification is **normative**. Where an + implementation (including the Java reference client) and the specification disagree, the + specification is the source of truth and the implementation is the bug — unless a PIP changes the + specification. + +3. **Audience and use.** The specification is the base document for implementing a scalable-topics + client in any language. It separates the transport-agnostic **API contract** (what an application + observes: operations, inputs, guarantees, errors) from the **implementation requirements** (what a + client must do internally: routing, layout tracking, fan-out, retries) and from the **wire protocol** + (the exact command exchanges, with sequence diagrams), so an implementer can build against the + contract and bind it to the Pulsar protocol. + +4. **Conformance.** The specification defines what a conformant client MUST implement (the core + producer, the three consumer modes, transactions) and which capabilities are optional. An SDK claims + scalable-topics support by conforming to the specification's conformance document. + +### How it is governed + +The specification is **not itself a PIP**. It is a living document maintained under the change process +below, which ties every normative change to a PIP and keeps the document in lockstep with the reference +implementation. + +## Detailed Design + +### 1. Change process + +- **Every normative change to the specification MUST be made through a PIP.** A "normative change" is + any change to a requirement (MUST/SHOULD/MAY), a guarantee, an operation's contract, or a wire + interaction. The PIP carries the design and rationale; reviewers evaluate the proposed behavior and the + specification edits **together**. +- **Specification edits land with the PIP.** When a PIP that changes scalable-topics client behavior is + accepted, the corresponding edits to `spec/scalable-topics/` are merged together with it — normally in + the **same pull request** as the implementation — so the contract and its decision record never + drift. A PIP that changes client-observable behavior is not complete until its specification edits are + merged. +- **Non-normative edits** (typos, wording clarifications that change no requirement, added examples or + diagrams, cross-reference fixes) do not require a PIP and follow the normal pull-request review. +- **PIPs remain the rationale and decision record**; the specification is the normative contract kept + continuously in sync with the ratified PIPs. A reader who wants to know *what* a client must do reads + the specification; a reader who wants to know *why* follows its references to the PIPs. + +### 2. Relationship to the reference implementation + +- The Java V5 client is the **reference implementation**. The specification's Stable content describes + behavior that is **implemented in a released broker and the reference client, or ratified via an + accepted PIP and committed to ship**; an SDK built to it interoperates with current and announced + brokers. +- A change to the reference implementation that alters client-observable behavior MUST be accompanied + by the matching specification change (via §1). Conversely, the specification MUST NOT describe + behavior the reference implementation does not (or will not, per an accepted PIP) exhibit. +- The specification includes a **Java reference mapping** appendix that maps each language-neutral + operation and type to the Java V5 surface, so the reference implementation and the specification can + be checked against each other and other SDKs can align their naming. + +### 3. Versioning, aligned with Pulsar LTS releases + +- The specification carries its own version of the form `MAJOR.MINOR`, declared in its index. +- **MAJOR** increments on a backward-incompatible change to a *Stable* requirement and is **tied to the + Apache Pulsar LTS cadence**: each new MAJOR corresponds to a new Pulsar LTS release. Backward-incompatible + changes and removals of deprecated features therefore land only at an LTS boundary, never in an interim + release. +- **MINOR** increments on backward-compatible additions or clarifications and may occur in any release. +- The work-in-progress draft is currently at `0.x` (**Draft**). Acceptance of this PIP adopts it as + **`1.0`**, from which point the stability guarantees below take full effect. Any later MAJOR + transition is a normative change made through the change process (§1). + +### 4. Stability tiers and deprecation + +Every feature in the specification carries a stability tier expressing its change contract: + +- **Stable** — will not change incompatibly without a MAJOR bump; conformant clients MUST implement all + Stable, non-optional requirements. +- **Experimental** — may change in any way, including removal, in a MINOR version; clients MAY implement + it, and omitting it never makes a client non-conformant. Used for designs that are documentable but not + yet settled. +- **Deprecated** — Stable but scheduled for removal; retained for compatibility, not to be newly adopted. + +**Deprecation and removal mechanism.** A feature enters Deprecated only through the change process (a +PIP) that records the reason, the replacement (or that none exists), and the earliest version at which +removal is permitted. While Deprecated, brokers MUST keep honoring it, clients SHOULD migrate, and +implementations SHOULD emit a non-fatal deprecation signal. A Deprecated Stable feature MUST NOT be +removed before the next MAJOR version **and** MUST remain available at least until the next Pulsar LTS +release, guaranteeing an LTS-to-LTS upgrade path in which the feature and its replacement coexist. + +### 5. Scope of the specification + +- **In scope:** everything a client SDK must implement for scalable topics — topic identity, the data + model (message, message identifier, checkpoint, schema), the producer and the three consumer modes, + transactions, error semantics, the client-side mechanisms, the complete wire protocol, conformance + criteria, and compatibility with classic topics (the migration bridge). +- **Out of scope:** broker-internal behavior (controller, split/merge execution, storage) except where + observable by a client; and designs whose specification is not yet settled (currently geo-replication + of scalable topics), which enter the specification — initially as Experimental — only once ratified. + +### 6. Obligations on new client SDKs + +An SDK adding scalable-topics support: + +- MUST implement the specification's core (Stable, non-optional) requirements and MAY implement optional + capabilities, each fully per its section, gating on broker support where the specification requires. +- MUST use the specification — not the Java client's source — as its reference, and SHOULD map its + idiomatic API to the specification's language-neutral operations (as the Java mapping appendix does for + Java), so cross-SDK behavior stays comparable. +- SHOULD contribute clarifications back through the change process when the specification is found + ambiguous or incomplete, rather than resolving the ambiguity privately. + +## Public-facing changes + +- **New repository artifact:** `spec/scalable-topics/` — the specification document set. The + work-in-progress draft linked above is adopted as **version 1.0**: it reflects the current client + API and semantics as designed in the scalable-topics PIPs and as implemented on `master`; the + stability guarantees of §3–§4 apply from that version. +- **Process:** the change process in §1 applies to every subsequent PIP that touches scalable-topics + client behavior, and to pull requests changing the reference implementation's client-observable + behavior. +- No code, wire-protocol, configuration, API, or metrics changes are introduced by this PIP. + +## Backward & Forward Compatibility + +Not applicable to this PIP itself: it introduces a document and a process, not a behavior change. The +compatibility guarantees the specification *makes* (protocol extensibility by addition, cross-version +restorability of serialized identifiers and checkpoints, the LTS-anchored deprecation window) are defined +within the specification and are governed by §3–§4 of this PIP. + +## Alternatives + +- **Keep the PIPs as the only description.** Rejected: PIPs are point-in-time proposals that are never + updated after acceptance; they cannot serve as a living, normative contract, and they describe design + rationale rather than exact client obligations. +- **Make the specification itself a PIP.** Rejected: a PIP is ratified once and then frozen, whereas the + specification must evolve continuously with the implementation. Governing the specification *through* + PIPs (§1) captures the review rigor without freezing the document. +- **Per-language documentation only (e.g. Javadoc as the reference).** Rejected: language-specific + documentation cannot define language-neutral contracts, wire interactions, or conformance, and it + invites divergence between SDKs. +- **Publish the specification only on the documentation site, not in the repository.** Rejected for the + source of truth: keeping it in `apache/pulsar` lets specification edits merge in the same pull request + as the implementation change (§1). Mirroring to the docs site is a follow-up. + +## Links + + +* Specification (work in progress): + + — will live at `spec/scalable-topics/` in `apache/pulsar` (index: `README.md`) +* Related: [PIP-460](pip-460.md), [PIP-468](pip-468.md), [PIP-483](pip-483.md), [PIP-486](pip-486.md) +* Mailing List discussion thread: +* Mailing List voting thread: