Add Korvet documentation (latest)

content/integrate/korvet/_index.md

@@ -0,0 +1,54 @@
+---
+Title: Korvet
+alwaysopen: false
+categories:
+- docs
+- integrate
+- korvet
+description: null
+group: service
+hideListLinks: false
+linkTitle: Korvet
+summary: Korvet provides a Kafka-compatible API backed by Redis Streams, so you
+  can use existing Kafka clients and tools with Redis as the storage engine.
+type: integration
+weight: 1
+---
+
+Korvet is a Kafka-compatible streaming service backed by Redis Streams.
+
+Korvet is developed by Redis Field Engineering.
+To report bugs, request features, or receive assistance, please [file an issue](https://github.com/redis-field-engineering/korvet-dist/issues).
+
+## Overview
+
+Korvet provides a Kafka-compatible API backed by Redis Streams:
+
+- **Kafka compatibility**: Use existing Kafka clients and tools
+- **Redis Streams**: High-performance, durable message storage
+- **Consumer groups**: Coordinated consumption with offset tracking
+- **Low latency**: Sub-millisecond read/write performance
+
+## Key Features
+
+- **Kafka Protocol Support**: Compatible with Kafka clients (produce, consume, consumer groups)
+- **Redis Streams**: High-performance storage with built-in persistence
+- **Tiered Storage**: Optionally archive sealed segments to Apache Iceberg tables on object storage for cost-efficient long-term retention
+- **Consumer Groups**: Full support for coordinated consumption and offset management
+- **Admin API**: Create/delete topics, configure retention, describe cluster
+- **Production-ready**: Built-in metrics, health checks, and observability
+
+## Use Cases
+
+- **Kafka alternative**: Lightweight Kafka-compatible streaming on Redis
+- **Kafka migration**: Gradual migration from Kafka to Redis-based streaming
+- **Low-latency streaming**: Sub-millisecond message delivery
+- **Simplified operations**: Single Redis instance instead of Kafka cluster
+
+## License
+
+Korvet is licensed under the [Business Source License 1.1](https://github.com/redis-field-engineering/korvet-dist/blob/main/LICENSE).
+
+Production use is permitted only with Redis Community Edition, Redis Cloud, or Redis Software.
+Non-production use (development, testing) is unrestricted.
+The license converts to MIT four years after each version's publication.

content/integrate/korvet/concepts.md

@@ -0,0 +1,97 @@
+---
+Title: Concepts & Architecture
+alwaysopen: false
+categories:
+- docs
+- integrate
+- korvet
+description: How Korvet maps Kafka topics, offsets, and consumer groups onto Redis
+  Streams primitives.
+linkTitle: Concepts & Architecture
+weight: 20
+---
+
+Korvet is a Kafka-compatible streaming service backed by Redis Streams. This page explains
+the core concepts and the high-level architecture: how Kafka topics, offsets, and consumer groups map onto
+Redis primitives. If you only need to run or use Korvet, the [Get Started]({{< relref "/integrate/korvet/quick-start" >}})
+and [Using the Kafka API]({{< relref "/integrate/korvet/kafka-api" >}}) sections are enough. For the request-by-request implementation
+details, see Under the Hood.
+
+## Architecture
+
+Korvet exposes a Kafka-compatible broker interface while persisting data in Redis-backed
+storage. At a high level:
+
+- Kafka protocol requests are terminated by the broker.
+- Kafka topics and partitions map onto Redis Streams.
+- Kafka offsets are translated to and from Redis Stream entry IDs.
+- Consumer-group behavior combines broker-side coordination with Redis-native delivery primitives.
+
+{{< image filename="images/korvet/architecture.svg" alt="Architecture Overview" >}}
+
+### Kafka to Redis Model
+
+| Kafka Concept | How Korvet Implements It |
+|---|---|
+| Topic partition | A Redis Stream |
+| Message record (key, value, headers, timestamp) | A single Redis Stream entry |
+| Message offset | Encoded from the Redis Stream entry ID — no offset table to maintain |
+| Consumer group | A Redis Streams consumer group |
+| Committed offsets and topic metadata | Tracked in Redis |
+
+For the exact keys and structures, see Redis Data Structures.
+
+### Design Principles
+
+- **Compatibility**: Maintain full Kafka protocol compatibility so existing clients and tools work unchanged.
+- **Statelessness**: Offsets are computed from Redis entry IDs rather than stored in side tables (see [Topics, Partitions, and Offsets](#topics-partitions-and-offsets)).
+- **Atomicity**: Use Redis transactions (`MULTI`/`EXEC`) and Lua scripts for atomic operations.
+- **Performance**: Leverage pipelining, connection pooling, and caching for high throughput.
+
+## Topics, Partitions, and Offsets
+
+Like Kafka, Korvet organizes messages into topics and partitions:
+
+- **Topic**: A logical stream of messages (e.g., `orders`, `events`).
+- **Partition**: A topic is divided into partitions for parallelism; each partition maps to a Redis Stream.
+- **Offset**: Each message has a unique, monotonically increasing offset within its partition.
+
+Offsets are **stateless**: rather than maintaining a side table, Korvet encodes the Kafka
+offset directly from the Redis Stream entry ID (`{timestamp}-{sequence}`). This makes offset conversion an
+O(1) computation in both directions and requires no extra storage. The number of bits reserved for the
+sequence is tunable per topic (`offset-sequence-bits`) to trade write throughput against batch coherence;
+see [the configuration reference]({{< relref "/integrate/korvet/reference/configuration" >}}) for defaults and limits, and
+Offset Encoding for the exact formula.
+
+## Consumer Groups
+
+Korvet implements Kafka consumer groups on top of Redis Streams native consumer groups:
+
+- **Coordination**: The broker implements the Kafka group coordinator protocol (join, sync, heartbeat, rebalance).
+- **Delivery**: Redis Streams consumer groups (`XREADGROUP`) handle per-consumer delivery state.
+- **Offset management**: Explicit Kafka commits are tracked in a separate committed-offset store, so consumer
+  progress survives restarts.
+- **Membership**: Active group membership (members, assignments, generation) is held in broker memory. After a
+  broker restart, groups with committed offsets remain visible to admin APIs in the `Empty` state until their
+  clients rejoin.
+
+## Message Format
+
+Messages follow the Kafka record format:
+
+- **Key**: Optional message key (byte array).
+- **Value**: Message payload (byte array).
+- **Headers**: Optional key-value metadata.
+- **Timestamp**: Message timestamp.
+
+Each Kafka record is stored as a single Redis Stream entry whose body breaks the record out into separate,
+directly-readable `value`, `key`, `headers`, and `timestamp` fields, so a non-Kafka client can read the
+payload straight from the stream (for example with `XRANGE`). A field is omitted when its component is
+absent. See Redis Data Structures for the exact
+layout.
+
+## Next Steps
+
+- [Using the Kafka API]({{< relref "/integrate/korvet/kafka-api" >}}) — produce, consume, and manage topics
+- [Tiered Storage]({{< relref "/integrate/korvet/storage" >}}) — local and remote tiers, and how data moves between them
+- Under the Hood — request workflows, protocol mapping, and internals

content/integrate/korvet/kafka-api/_index.md

@@ -0,0 +1,53 @@
+---
+Title: Kafka API
+alwaysopen: false
+categories:
+- docs
+- integrate
+- korvet
+description: Use existing Kafka clients and tools with Korvet's implementation of
+  the Kafka protocol.
+group: service
+hideListLinks: false
+linkTitle: Using the Kafka API
+summary: Korvet provides a Kafka-compatible API backed by Redis Streams, so you
+  can use existing Kafka clients and tools with Redis as the storage engine.
+type: integration
+weight: 30
+---
+
+Korvet implements the Kafka protocol, allowing you to use existing Kafka clients and tools.
+
+## Supported Operations
+
+- **Produce**: Send messages to topics
+- **Fetch**: Read messages from topics
+- **Consumer Groups**: Coordinate multiple consumers with offset tracking
+- **Topic Management**: Create and list topics
+
+## Client Compatibility
+
+Korvet is compatible with standard Kafka clients:
+
+- **Java**: kafka-clients library
+- **Python**: kafka-python, confluent-kafka-python
+- **Go**: sarama, confluent-kafka-go
+- **Node.js**: kafkajs, node-rdkafka
+- **Command-line**: kafka-console-producer, kafka-console-consumer
+- **Databricks**: Spark Structured Streaming (see [Databricks Integration]({{< relref "/integrate/korvet/kafka-api/databricks" >}}))
+
+## Connection
+
+Connect to Korvet using the standard Kafka bootstrap server configuration:
+
+```properties
+bootstrap.servers=localhost:9092
+```
+
+## Next Steps
+
+- [Producing messages]({{< relref "/integrate/korvet/kafka-api/produce" >}})
+- [Consuming messages]({{< relref "/integrate/korvet/kafka-api/consume" >}})
+- [Topic management]({{< relref "/integrate/korvet/kafka-api/topics" >}})
+- [Databricks integration]({{< relref "/integrate/korvet/kafka-api/databricks" >}})
+- [Kafka compatibility details]({{< relref "/integrate/korvet/kafka-api/compatibility" >}})

content/integrate/korvet/kafka-api/compatibility.md

@@ -0,0 +1,115 @@
+---
+Title: Kafka Compatibility
+alwaysopen: false
+categories:
+- docs
+- integrate
+- korvet
+description: Korvet implements a subset of the Kafka protocol for compatibility with
+  existing clients and tools.
+linkTitle: Kafka Compatibility
+weight: 60
+---
+
+Korvet implements a subset of the Kafka protocol for compatibility with existing clients and tools.
+
+## Supported APIs
+
+| API | Status | Notes |
+|---|---|---|
+| Produce | ✅ Supported | Send messages to topics |
+| Fetch | ✅ Supported | Read messages from topics |
+| Metadata | ✅ Supported | Topic and partition information |
+| ApiVersions | ✅ Supported | Protocol version negotiation |
+| Consumer Groups | ✅ Supported | JoinGroup, SyncGroup, Heartbeat, LeaveGroup, OffsetCommit, OffsetFetch |
+| Transactions | ❌ Not planned | Use Redis transactions instead |
+| Admin API | ✅ Supported | CreateTopics, DeleteTopics, DescribeConfigs, AlterConfigs, IncrementalAlterConfigs, DescribeCluster, ListGroups, DescribeGroups, DeleteGroups |
+| Idempotent Producers | ✅ Supported | InitProducerId API for idempotent producer support (non-transactional) |
+
+## Kafka Version Compatibility
+
+Korvet uses the Apache Kafka client library version 3.9.2 and is compatible with Kafka clients from version 2.8.0 and later.
+
+### Client Compatibility
+
+- **Minimum supported client version**: 2.8.0
+- **Recommended client version**: 3.9.x
+- **Kafka client library**: 3.9.2
+
+Kafka clients are backward compatible, so newer clients (3.x, 4.x) can connect to Korvet without issues.
+
+### Protocol Features
+
+Korvet implements Kafka protocol features equivalent to Kafka 2.8.0+, including:
+
+- Produce API (the version range advertised by the bundled kafka-clients 3.9.2 library)
+- Fetch API (v0-v12; capped at v12 for stability)
+- Consumer Group Protocol (JoinGroup, SyncGroup, Heartbeat, LeaveGroup)
+- Offset Management (OffsetCommit, OffsetFetch)
+- Topic Administration (CreateTopics, DeleteTopics)
+- Metadata API
+
+## Compression
+
+Korvet supports all Kafka compression types:
+
+- **NONE**: No compression (default)
+- **GZIP**: Good compression ratio, higher CPU usage
+- **SNAPPY**: Balanced compression and speed
+- **LZ4**: Fast compression, lower CPU usage
+- **ZSTD**: Best compression ratio, moderate CPU usage
+
+### How Compression Works
+
+Korvet implements server-side compression:
+
+1. **Producer side**: Kafka clients can send compressed or uncompressed batches. Korvet automatically decompresses incoming batches into individual records before storing them.
+
+2. **Consumer side**: When consumers fetch messages, Korvet compresses the response based on the topic's `compression.type` configuration (not the producer's compression setting).
+
+3. **At rest**: The producer's Kafka batch compression is not retained — each record is stored as its own Redis Stream entry. The storage backend may then apply its own configurable at-rest compression to each record's `value` field (`korvet.storage.local.compression.codec`, default `none`), independently of the Kafka `compression.type` used on the wire.
+
+### Configuring Compression
+
+Compression is configured per-topic using the `compression.type` setting:
+
+```bash
+# Set compression for a topic (requires Admin API support)
+kafka-configs --bootstrap-server localhost:9092 \
+  --entity-type topics \
+  --entity-name my-topic \
+  --alter \
+  --add-config compression.type=lz4
+```
+
+The default compression type is `NONE`.
+
+### Benefits
+
+- **Network bandwidth**: Compression reduces the amount of data transferred between Korvet and consumers
+- **Flexibility**: Different topics can use different compression algorithms based on their data characteristics
+- **Compatibility**: Works transparently with all Kafka clients
+
+## Limitations
+
+- **Replication factor**: Always 1 (Redis provides persistence)
+- **Transactions**: Not supported
+- **Exactly-once semantics**: Not supported (at-least-once delivery)
+- **Consumer group membership**: Held in broker memory. Committed offsets are durable in Redis, so after a broker restart `ListGroups` and `DescribeGroups` report groups with committed offsets in the `Empty` state (with no member details) until clients rejoin.
+
+## Client Configuration
+
+Most Kafka client configurations work with Korvet. Some settings are ignored:
+
+- `acks`: Always treated as `acks=1`
+- `replication.factor`: Ignored (always 1)
+- `min.insync.replicas`: Ignored
+
+## Testing Compatibility
+
+You can test Korvet with your existing Kafka applications by simply changing the `bootstrap.servers` configuration to point to Korvet.
+
+## Next Steps
+
+- [Producing messages]({{< relref "/integrate/korvet/kafka-api/produce" >}})
+- [Consuming messages]({{< relref "/integrate/korvet/kafka-api/consume" >}})