Repository navigation
feat(schema): Protobuf decode and encode for Schema Registry topics - #8
Merged
Merged
Conversation
PROTOBUF subjects now work end to end alongside Avro and JSON Schema: - decode: browser, live tail and /decode render protobuf payloads as proto3 JSON with original field names; the Confluent message-index prefix (1-based varints, 0x00-terminated) is stripped, nested prefixes are reported as unsupported, and a defensive fallback parses payloads without a prefix - encode: /encode takes the subject's .proto, validates the JSON payload against the descriptor (unknown fields rejected), and emits Confluent wire bytes with the index prefix; multi-message .proto files pick the message via the wire prefix or the new optional messageName on the encode request - .proto text from the registry is compiled in-process by kafka-protobuf-provider (Confluent Community License, resolved from packages.confluent.io — new Maven repository in the pom); compiled schemas are TTL-cached like Avro schemas - tests: 5 integration tests against a mocked registry — hand-built Confluent wire fixture decodes to proto3 JSON, encode round-trips with the index prefix asserted byte-for-byte, produced messages survive produce-and-browse, unknown fields rejected, unparseable messages reported instead of thrown. Full suite 63/63.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements F4 — Protobuf codecs, the last high-impact feature from the deep analysis: PROTOBUF subjects were the one Schema Registry gap — the code literally threw "Protobuf encoding arrives in phase 2b — not supported yet" while every comparable console decodes proto. Stacked on #7 (both touch
pom.xml) — merge #1 → #2 → #3 → #4 → #5 → #6 → #7 → this.How it works
/decode): after the 5-byte Confluent header, the payload's message-index prefix (1-based varints terminated by0x00— Confluent's documented protobuf wire shape) is stripped and the message parsed against the.protocompiled from the registry, then rendered as proto3 JSON with original field names and defaults made explicit. Defenses: nested index prefixes (>1 level) are reported as unsupported, and a payload that fails with a prefix is retried without one for non-Confluent producers; unparseable bytes come back as a reported error, never a throw./encode, feeding the produce form): the JSON payload is validated against the descriptor (unknown fields rejected with the offending name) and emitted as Confluent wire bytes with the index prefix — asserted byte-for-byte in tests. Multi-message.protofiles pick the message from the wire shape of the subject's declaration order, an envelopemessageName, or the new optionalmessageNamerequest field..prototext is parsed in-process bykafka-protobuf-provider8.3.2 (Confluent Community License) — the same library Confluent's own deserializer uses — resolved from a newpackages.confluent.ioMaven repository in the pom. Compiled schemas are TTL-cached (5 min) like Avro; the cache is keyed per registry URL and schema id.Verification
order_id/amount/paidpreserved); encode emits0x00 + id + 01 00 + messageexactly (index prefix asserted byte-for-byte); produced wire bytes survive produce-and-browse with schema id attached; unknown JSON fields rejected; unparseable messages reported not thrown.Honest limitations (documented)
kafka-protobuf-provideris Confluent Community License — a runtime dependency only; no Confluent code is copied into Kview.