apache · merlimat · Sep 2, 2026
diff --git a/pip/pip-494.md b/pip/pip-494.md
@@ -0,0 +1,233 @@
+# PIP-494: Scalable Topics Client Specification — authoritative reference and change process
+
+*Sub-PIP of [PIP-460: Scalable Topics](pip-460.md)*
+
+> **Draft.** This PIP establishes a **process**; it deliberately does not restate the specification's
+> content, which lives in the repository at `spec/scalable-topics/`. The current work-in-progress
+> specification —
+> <https://github.com/merlimat/pulsar/tree/mmerli/scalable-topics-spec/spec/scalable-topics> — will be
+> adopted as **version 1.0** of the specification. It reflects the current client API and
+> semantics as designed in the scalable-topics PIPs ([PIP-460](pip-460.md), [PIP-468](pip-468.md),
+> [PIP-483](pip-483.md), [PIP-486](pip-486.md)) and as implemented on the `master` branch.
+
+## Motivation
+
+Scalable topics ([PIP-460](pip-460.md) and its sub-PIPs — the controller [PIP-468](pip-468.md), auto
+split/merge [PIP-483](pip-483.md), key-shared consumption [PIP-486](pip-486.md)) introduce a client model
+that is materially different from classic Pulsar topics: no application-visible partitions, a dynamic
+per-cluster segment layout the client must track, client-side key routing, three distinct consumer
+modes with different ordering/acknowledgment contracts, a checkpoint position type, a controller-driven
+consumer-assignment protocol, and a family of new binary-protocol commands.
+
+Today the only complete description of that client behavior is the **Java V5 client's source code**, with
+the design rationale scattered across the PIPs. That is not a workable basis for the ecosystem:
+
+- **Other-language SDKs** (Go, Python, C++, Rust, Node, .NET) need a precise, language-neutral definition
+  of the API contract, the required client-side mechanisms, and the exact wire interactions — not a Java
+  implementation to reverse-engineer.
+- **Interoperability** depends on every client routing keys identically, applying layouts and assignments
+  with the same staleness rules, and honoring the same ordering and acknowledgment semantics. Without a
+  normative reference, divergent clients are inevitable and hard to diagnose.
+- **PIPs are design proposals, not contracts.** They explain *why* and record decisions, but they are
+  point-in-time documents that are never updated after acceptance. The living behavior needs a living,
+  normative document.
+- **Stability and deprecation** need defined rules so SDK authors know what they can rely on across
+  Pulsar releases.
+
+This PIP establishes a **Scalable Topics Client Specification** as the single authoritative description
+of the client API, its semantics, and its protocol-level interactions — the base document for anyone
+implementing scalable-topics support in a Pulsar client SDK — and defines the **process** by which that
+specification is maintained and changed.
+
+## Background knowledge
+
+**PIPs vs. a specification.** A PIP proposes and ratifies a change; once merged it is a historical
+record of a decision. A specification is a *current* normative statement of behavior that implementations
+conform to, and it evolves as behavior evolves. Mature cross-implementation ecosystems keep both: the
+OpenTelemetry project, for example, maintains a versioned specification with explicit stability tiers and
+a formal change process, separate from the proposals that motivated each change. This PIP adopts that
+model for scalable topics.
+
+**Reference implementation.** The Java V5 client (`pulsar-client-api-v5`, `pulsar-client-v5`) is the
+reference implementation of scalable-topics client behavior. The specification is derived from it and
+verified against it, but the specification — not the Java code — is the source of truth for conformance.
+
+## Goals
+
+### In Scope
+
+- Establish `spec/scalable-topics/` in the `apache/pulsar` repository as the **authoritative,
+  language-neutral specification** for scalable-topics clients: the API contract, its semantics, the
+  required client-side mechanisms, and the wire-protocol interactions.
+- Define the specification as the **required base for any new client SDK** adding scalable-topics
+  support, and the reference against which SDK **conformance** is judged.
+- Define the **change process**: how normative changes to the specification are proposed, reviewed, and
+  landed, and how the specification is kept in sync with the reference implementation.
+- Define the specification's **versioning, stability tiers, and deprecation/removal rules**, aligned with
+  the Apache Pulsar release (LTS) cadence.
+
+### Out of Scope
+
+- The specification's content itself. It is a separate deliverable (already drafted) and is not
+  restated here; this PIP governs it, it does not define client behavior.
+- Client behavior for classic (non-scalable) topics — the specification covers scalable topics and their
+  migration bridge from classic topics only.
+- The mechanics of publishing the specification to the user-facing documentation site
+  (`apache/pulsar-site`); that is a follow-up task once the process is in place.
+
+## High Level Design
+
+### What is established
+
+1. **Location and status.** The specification lives at `spec/scalable-topics/` in `apache/pulsar`, as a
+   set of Markdown documents with a top-level index (`README.md`). It is a first-class artifact of the
+   project, maintained alongside the code it describes.
+
+2. **Authority.** For scalable-topics client behavior, the specification is **normative**. Where an
+   implementation (including the Java reference client) and the specification disagree, the
+   specification is the source of truth and the implementation is the bug — unless a PIP changes the
+   specification.
+
+3. **Audience and use.** The specification is the base document for implementing a scalable-topics
+   client in any language. It separates the transport-agnostic **API contract** (what an application
+   observes: operations, inputs, guarantees, errors) from the **implementation requirements** (what a
+   client must do internally: routing, layout tracking, fan-out, retries) and from the **wire protocol**
+   (the exact command exchanges, with sequence diagrams), so an implementer can build against the
+   contract and bind it to the Pulsar protocol.
+
+4. **Conformance.** The specification defines what a conformant client MUST implement (the core
+   producer, the three consumer modes, transactions) and which capabilities are optional. An SDK claims
+   scalable-topics support by conforming to the specification's conformance document.
+
+### How it is governed
+
+The specification is **not itself a PIP**. It is a living document maintained under the change process
+below, which ties every normative change to a PIP and keeps the document in lockstep with the reference
+implementation.
+
+## Detailed Design
+
+### 1. Change process
+
+- **Every normative change to the specification MUST be made through a PIP.** A "normative change" is
+  any change to a requirement (MUST/SHOULD/MAY), a guarantee, an operation's contract, or a wire
+  interaction. The PIP carries the design and rationale; reviewers evaluate the proposed behavior and the
+  specification edits **together**.
+- **Specification edits land with the PIP.** When a PIP that changes scalable-topics client behavior is
+  accepted, the corresponding edits to `spec/scalable-topics/` are merged together with it — normally in
+  the **same pull request** as the implementation — so the contract and its decision record never
+  drift. A PIP that changes client-observable behavior is not complete until its specification edits are
+  merged.
+- **Non-normative edits** (typos, wording clarifications that change no requirement, added examples or
+  diagrams, cross-reference fixes) do not require a PIP and follow the normal pull-request review.
+- **PIPs remain the rationale and decision record**; the specification is the normative contract kept
+  continuously in sync with the ratified PIPs. A reader who wants to know *what* a client must do reads
+  the specification; a reader who wants to know *why* follows its references to the PIPs.
+
+### 2. Relationship to the reference implementation
+
+- The Java V5 client is the **reference implementation**. The specification's Stable content describes
+  behavior that is **implemented in a released broker and the reference client, or ratified via an
+  accepted PIP and committed to ship**; an SDK built to it interoperates with current and announced
+  brokers.
+- A change to the reference implementation that alters client-observable behavior MUST be accompanied
+  by the matching specification change (via §1). Conversely, the specification MUST NOT describe
+  behavior the reference implementation does not (or will not, per an accepted PIP) exhibit.
+- The specification includes a **Java reference mapping** appendix that maps each language-neutral
+  operation and type to the Java V5 surface, so the reference implementation and the specification can
+  be checked against each other and other SDKs can align their naming.
+
+### 3. Versioning, aligned with Pulsar LTS releases
+
+- The specification carries its own version of the form `MAJOR.MINOR`, declared in its index.
+- **MAJOR** increments on a backward-incompatible change to a *Stable* requirement and is **tied to the
+  Apache Pulsar LTS cadence**: each new MAJOR corresponds to a new Pulsar LTS release. Backward-incompatible
+  changes and removals of deprecated features therefore land only at an LTS boundary, never in an interim
+  release.
+- **MINOR** increments on backward-compatible additions or clarifications and may occur in any release.
+- The work-in-progress draft is currently at `0.x` (**Draft**). Acceptance of this PIP adopts it as
+  **`1.0`**, from which point the stability guarantees below take full effect. Any later MAJOR
+  transition is a normative change made through the change process (§1).
+
+### 4. Stability tiers and deprecation
+
+Every feature in the specification carries a stability tier expressing its change contract:
+
+- **Stable** — will not change incompatibly without a MAJOR bump; conformant clients MUST implement all
+  Stable, non-optional requirements.
+- **Experimental** — may change in any way, including removal, in a MINOR version; clients MAY implement
+  it, and omitting it never makes a client non-conformant. Used for designs that are documentable but not
+  yet settled.
+- **Deprecated** — Stable but scheduled for removal; retained for compatibility, not to be newly adopted.
+
+**Deprecation and removal mechanism.** A feature enters Deprecated only through the change process (a
+PIP) that records the reason, the replacement (or that none exists), and the earliest version at which
+removal is permitted. While Deprecated, brokers MUST keep honoring it, clients SHOULD migrate, and
+implementations SHOULD emit a non-fatal deprecation signal. A Deprecated Stable feature MUST NOT be
+removed before the next MAJOR version **and** MUST remain available at least until the next Pulsar LTS
+release, guaranteeing an LTS-to-LTS upgrade path in which the feature and its replacement coexist.
+
+### 5. Scope of the specification
+
+- **In scope:** everything a client SDK must implement for scalable topics — topic identity, the data
+  model (message, message identifier, checkpoint, schema), the producer and the three consumer modes,
+  transactions, error semantics, the client-side mechanisms, the complete wire protocol, conformance
+  criteria, and compatibility with classic topics (the migration bridge).
+- **Out of scope:** broker-internal behavior (controller, split/merge execution, storage) except where
+  observable by a client; and designs whose specification is not yet settled (currently geo-replication
+  of scalable topics), which enter the specification — initially as Experimental — only once ratified.
+
+### 6. Obligations on new client SDKs
+
+An SDK adding scalable-topics support:
+
+- MUST implement the specification's core (Stable, non-optional) requirements and MAY implement optional
+  capabilities, each fully per its section, gating on broker support where the specification requires.
+- MUST use the specification — not the Java client's source — as its reference, and SHOULD map its
+  idiomatic API to the specification's language-neutral operations (as the Java mapping appendix does for
+  Java), so cross-SDK behavior stays comparable.
+- SHOULD contribute clarifications back through the change process when the specification is found
+  ambiguous or incomplete, rather than resolving the ambiguity privately.
+
+## Public-facing changes
+
+- **New repository artifact:** `spec/scalable-topics/` — the specification document set. The
+  work-in-progress draft linked above is adopted as **version 1.0**: it reflects the current client
+  API and semantics as designed in the scalable-topics PIPs and as implemented on `master`; the
+  stability guarantees of §3–§4 apply from that version.
+- **Process:** the change process in §1 applies to every subsequent PIP that touches scalable-topics
+  client behavior, and to pull requests changing the reference implementation's client-observable
+  behavior.
+- No code, wire-protocol, configuration, API, or metrics changes are introduced by this PIP.
+
+## Backward & Forward Compatibility
+
+Not applicable to this PIP itself: it introduces a document and a process, not a behavior change. The
+compatibility guarantees the specification *makes* (protocol extensibility by addition, cross-version
+restorability of serialized identifiers and checkpoints, the LTS-anchored deprecation window) are defined
+within the specification and are governed by §3–§4 of this PIP.
+
+## Alternatives
+
+- **Keep the PIPs as the only description.** Rejected: PIPs are point-in-time proposals that are never
+  updated after acceptance; they cannot serve as a living, normative contract, and they describe design
+  rationale rather than exact client obligations.
+- **Make the specification itself a PIP.** Rejected: a PIP is ratified once and then frozen, whereas the
+  specification must evolve continuously with the implementation. Governing the specification *through*
+  PIPs (§1) captures the review rigor without freezing the document.
+- **Per-language documentation only (e.g. Javadoc as the reference).** Rejected: language-specific
+  documentation cannot define language-neutral contracts, wire interactions, or conformance, and it
+  invites divergence between SDKs.
+- **Publish the specification only on the documentation site, not in the repository.** Rejected for the
+  source of truth: keeping it in `apache/pulsar` lets specification edits merge in the same pull request
+  as the implementation change (§1). Mirroring to the docs site is a follow-up.
+
+## Links
+
+<!-- Updated afterwards -->
+* Specification (work in progress):
+  <https://github.com/merlimat/pulsar/tree/mmerli/scalable-topics-spec/spec/scalable-topics>
+  — will live at `spec/scalable-topics/` in `apache/pulsar` (index: `README.md`)
+* Related: [PIP-460](pip-460.md), [PIP-468](pip-468.md), [PIP-483](pip-483.md), [PIP-486](pip-486.md)
+* Mailing List discussion thread:
+* Mailing List voting thread: