Skip to content

feat(schema): Protobuf decode and encode for Schema Registry topics - #8

Merged
ritikbansod merged 1 commit into
mainfrom
feat/protobuf
Sep 28, 2026
Merged

ritikbansod merged 1 commit into
mainfrom
feat/protobuf

Conversation

@ritikbansod

Copy link
Copy Markdown
Owner

Implements F4 — Protobuf codecs, the last high-impact feature from the deep analysis: PROTOBUF subjects were the one Schema Registry gap — the code literally threw "Protobuf encoding arrives in phase 2b — not supported yet" while every comparable console decodes proto. Stacked on #7 (both touch pom.xml) — merge #1 → #2 → #3 → #4 → #5 → #6 → #7 → this.

How it works

  • Decode (browser, live tail, /decode): after the 5-byte Confluent header, the payload's message-index prefix (1-based varints terminated by 0x00 — Confluent's documented protobuf wire shape) is stripped and the message parsed against the .proto compiled from the registry, then rendered as proto3 JSON with original field names and defaults made explicit. Defenses: nested index prefixes (>1 level) are reported as unsupported, and a payload that fails with a prefix is retried without one for non-Confluent producers; unparseable bytes come back as a reported error, never a throw.
  • Encode (/encode, feeding the produce form): the JSON payload is validated against the descriptor (unknown fields rejected with the offending name) and emitted as Confluent wire bytes with the index prefix — asserted byte-for-byte in tests. Multi-message .proto files pick the message from the wire shape of the subject's declaration order, an envelope messageName, or the new optional messageName request field.
  • Compilation: .proto text is parsed in-process by kafka-protobuf-provider 8.3.2 (Confluent Community License) — the same library Confluent's own deserializer uses — resolved from a new packages.confluent.io Maven repository in the pom. Compiled schemas are TTL-cached (5 min) like Avro; the cache is keyed per registry URL and schema id.

Verification

  • 5 new integration tests against a mocked registry: hand-built Confluent wire fixture decodes to proto3 JSON (order_id/amount/paid preserved); encode emits 0x00 + id + 01 00 + message exactly (index prefix asserted byte-for-byte); produced wire bytes survive produce-and-browse with schema id attached; unknown JSON fields rejected; unparseable messages reported not thrown.
  • Full suite: 63/63 green.

Honest limitations (documented)

  • Nested-message index prefixes (subjects whose wire bytes carry multi-level indexes) are reported as unsupported rather than mis-decoded.
  • Verified against the documented Confluent wire shape and a round-trip; a live Confluent Cloud/Strimzi registry round-trip is the remaining acceptance step for a release.
  • kafka-protobuf-provider is Confluent Community License — a runtime dependency only; no Confluent code is copied into Kview.

PROTOBUF subjects now work end to end alongside Avro and JSON Schema:

- decode: browser, live tail and /decode render protobuf payloads as proto3
  JSON with original field names; the Confluent message-index prefix
  (1-based varints, 0x00-terminated) is stripped, nested prefixes are
  reported as unsupported, and a defensive fallback parses payloads without
  a prefix
- encode: /encode takes the subject's .proto, validates the JSON payload
  against the descriptor (unknown fields rejected), and emits Confluent wire
  bytes with the index prefix; multi-message .proto files pick the message
  via the wire prefix or the new optional messageName on the encode request
- .proto text from the registry is compiled in-process by
  kafka-protobuf-provider (Confluent Community License, resolved from
  packages.confluent.io — new Maven repository in the pom); compiled
  schemas are TTL-cached like Avro schemas
- tests: 5 integration tests against a mocked registry — hand-built
  Confluent wire fixture decodes to proto3 JSON, encode round-trips with the
  index prefix asserted byte-for-byte, produced messages survive
  produce-and-browse, unknown fields rejected, unparseable messages reported
  instead of thrown. Full suite 63/63.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant