[Umbrella] Kafka Protocols Compatibility - Basic Produce

[gyang94]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Apache Fluss needs a first Kafka protocol compatibility milestone that allows standard Kafka producers to discover Fluss tables and write records without changing the Kafka client.

The target workflow is DDL-first: users create and manage tables and schemas through Fluss DDL, then produce JSON records through the Kafka protocol. Schema-aware conversion maps records into the pre-defined table schema and existing Fluss storage format. Kafka CreateTopics, DeleteTopics, and automatic table creation are not part of this milestone.

This umbrella is issue-driven: define a focused child issue, link its implementation PR, review it, and then move to the next capability. The foundation and Basic Produce tasks below follow dependency order. The previous four-PR approach has been retired and is not part of the current dependency chain.

Sub-tasks

Only scoped child issues are listed here. Each issue owns its scope, acceptance criteria, dependencies, and implementation PR. An issue is complete after the reviewed implementation meets those criteria and is merged, not merely after a PR is opened.

Foundation

• [kafka] Add request and transport framework #4264 — PR01: Request and transport framework.
• [kafka] Implement registry-backed ApiVersions API #4265 — PR02: Registry-backed ApiVersions API.
• [kafka] Define DDL table mapping for Kafka compatibility #4275 — PR03: DDL table mapping contract.
• [kafka] Implement Metadata API for pre-created Fluss tables #4266 — PR04: Metadata API for pre-created Fluss tables.

Review and dependency order: #4264 → #4265 → #4275 → #4266.

Basic Produce

• [kafka] Handle basic Produce requests independently of storage #4281 — PR05: Produce protocol handling.
• [kafka] Transcode mapped raw and string records to Arrow #4282 — PR06: Raw/string record conversion.
• [kafka] Integrate Produce with native Fluss append #4283 — PR07: Native append integration.
• [kafka] Cover Produce lifecycle and replication regressions #4298 — PR08: Produce lifecycle and replication regressions.

Core changes (independent review and merge)

• [server] complete delayed fetch operations after produce to reduce fetch latency #3455 — Complete delayed fetch after produce.

Planned follow-ups

Create additional child issues progressively after the scoped foundation and Basic Produce tasks. These are directions for later issue scoping, not a commitment to reuse the closed implementation PRs.

Capability

• SASL/PLAIN authentication: handshake, per-connection authentication state, and credential validation.
• Schema-aware JSON Produce: table-metadata-based record mapping, JSON decoding, validation, and conversion to Fluss rows.

Delivery

• End-to-end validation of DDL → Metadata → Kafka producer → Fluss readback, including schema handling, acknowledgements, authentication, and failure cases.
• Configuration and compatibility documentation, supported API/version boundaries, DDL/JSON examples, and Fluss RPC/server regression coverage.

Future Work

Outside this Basic Produce milestone:

• Idempotent and transactional Produce.
• Bounded Produce admission and flow control.
• Kafka consumer and Fetch compatibility.
• Further authoritative ISR propagation and publisher/cache hardening beyond Metadata discovery.

Review workflow

1. Create a child issue with a bounded scope, acceptance criteria, and explicit dependencies.
2. Link a focused implementation PR to that child issue; do not close the umbrella from an individual PR.
3. Review Kafka feature issues in dependency order. Keep dependent PRs as drafts and use incremental comparisons while prerequisites are unmerged.
4. As prerequisites merge, rebase dependent branches onto the updated main and validate again. Record completion through the child issues, then scope the next unit.

Willingness to contribute

• I'm willing to submit a PR!

[dstolis]

Hi @gyang94 , I'd like to help with Kafka compatibility, starting with the Produce path. Is there a child issue I could pick up, and could I help with testing as the PRs land?