From 73fe71f489d72d3fadfe76f013571dcf8ae629c8 Mon Sep 17 00:00:00 2001 From: Josh Rotenberg Date: Wed, 26 Aug 2026 13:03:33 -0700 Subject: [PATCH 1/4] Add Korvet product skeleton under integrate/korvet Korvet is a Kafka-compatible streaming service backed by Redis Streams, developed by Redis Field Engineering. This adds the product landing page, the reference section index, and the two architecture images, as the skeleton for a 1:1 port of the existing Korvet docs (release 0.19) from redis-field-engineering.github.io/korvet. --- content/integrate/korvet/_index.md | 54 ++++++++++ content/integrate/korvet/reference/_index.md | 14 +++ static/images/korvet/architecture.svg | 102 +++++++++++++++++++ static/images/korvet/tiered-storage.svg | 56 ++++++++++ 4 files changed, 226 insertions(+) create mode 100644 content/integrate/korvet/_index.md create mode 100644 content/integrate/korvet/reference/_index.md create mode 100644 static/images/korvet/architecture.svg create mode 100644 static/images/korvet/tiered-storage.svg diff --git a/content/integrate/korvet/_index.md b/content/integrate/korvet/_index.md new file mode 100644 index 0000000000..f250ac0203 --- /dev/null +++ b/content/integrate/korvet/_index.md @@ -0,0 +1,54 @@ +--- +Title: Korvet +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Korvet is a Kafka-compatible streaming service backed by Redis Streams. +group: service +hideListLinks: false +linkTitle: Korvet +summary: Korvet provides a Kafka-compatible API backed by Redis Streams, so you + can use existing Kafka clients and tools with Redis as the storage engine. +type: integration +weight: 1 +--- + +Korvet is a Kafka-compatible streaming service backed by Redis Streams. + +Korvet is developed by Redis Field Engineering. +To report bugs, request features, or receive assistance, please [file an issue](https://github.com/redis-field-engineering/korvet-dist/issues). + +## Overview + +Korvet provides a Kafka-compatible API backed by Redis Streams: + +- **Kafka compatibility**: Use existing Kafka clients and tools +- **Redis Streams**: High-performance, durable message storage +- **Consumer groups**: Coordinated consumption with offset tracking +- **Low latency**: Sub-millisecond read/write performance + +## Key Features + +- **Kafka Protocol Support**: Compatible with Kafka clients (produce, consume, consumer groups) +- **Redis Streams**: High-performance storage with built-in persistence +- **Tiered Storage**: Optionally archive sealed segments to Apache Iceberg tables on object storage for cost-efficient long-term retention +- **Consumer Groups**: Full support for coordinated consumption and offset management +- **Admin API**: Create/delete topics, configure retention, describe cluster +- **Production-ready**: Built-in metrics, health checks, and observability + +## Use Cases + +- **Kafka alternative**: Lightweight Kafka-compatible streaming on Redis +- **Kafka migration**: Gradual migration from Kafka to Redis-based streaming +- **Low-latency streaming**: Sub-millisecond message delivery +- **Simplified operations**: Single Redis instance instead of Kafka cluster + +## License + +Korvet is licensed under the [Business Source License 1.1](https://github.com/redis-field-engineering/korvet-dist/blob/main/LICENSE). + +Production use is permitted only with Redis Community Edition, Redis Cloud, or Redis Software. +Non-production use (development, testing) is unrestricted. +The license converts to MIT four years after each version's publication. diff --git a/content/integrate/korvet/reference/_index.md b/content/integrate/korvet/reference/_index.md new file mode 100644 index 0000000000..bda2c098a2 --- /dev/null +++ b/content/integrate/korvet/reference/_index.md @@ -0,0 +1,14 @@ +--- +Title: Reference +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Reference documentation for Korvet configuration, APIs, and metrics. +hideListLinks: false +linkTitle: Reference +weight: 60 +--- + +Reference documentation for Korvet configuration, APIs, and metrics. diff --git a/static/images/korvet/architecture.svg b/static/images/korvet/architecture.svg new file mode 100644 index 0000000000..41974b9317 --- /dev/null +++ b/static/images/korvet/architecture.svg @@ -0,0 +1,102 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Kafka Producer + + Kafka Consumer + + Kafka Admin Client + + + + + + + + + + Korvet Broker + + Kafka Protocol Handler + + Topic Registry + + Storage + + + + + + + + Redis + + + Metadata + + + Streams + + + + Object Store + + + + + + + local + + + remote + diff --git a/static/images/korvet/tiered-storage.svg b/static/images/korvet/tiered-storage.svg new file mode 100644 index 0000000000..ae574bc3f0 --- /dev/null +++ b/static/images/korvet/tiered-storage.svg @@ -0,0 +1,56 @@ + + + + + + + + + + + + A partition is a sequence of segments, oldest to newest + + + + storage worker seals & offloads sealed segments + + + + + + + + + Segment 0 + offloaded + Segment 1 + offloaded + Segment 2 + sealed + Segment 3 + open · current writes + + + + + + + + Remote tier + Apache Iceberg on object storage + + + Local tier + Redis Streams + + + + producer appends (XADD) + + + The broker serves reads across both tiers — consumers never see the boundary. + + + Redis-only topics (the default) keep a single stream per partition, with no segments. + From 0452f49e2d6f057dcdec0aa1373b5449c73a766d Mon Sep 17 00:00:00 2001 From: Josh Rotenberg Date: Wed, 26 Aug 2026 13:17:27 -0700 Subject: [PATCH 2/4] Port Korvet documentation from the Antora site (0.19) 1:1 format port of the published Korvet docs (release 0.19) from Antora/AsciiDoc to Hugo Markdown: Get Started, Concepts & Architecture, Using the Kafka API, Tiered Storage, Operations, and Reference (configuration, API, metrics, FAQ). Prose is carried over verbatim; only syntax changed (frontmatter, relrefs, admonition shortcodes, fenced code blocks, Markdown tables, inlined metrics partials). Not ported, per the migration plan: the 'Under the Hood' internals section (stays in the korvet repo); links to it are de-linked. The standalone generated configuration page is folded into the Configuration reference page, matching the published nav. --- content/integrate/korvet/concepts.md | 97 +++ content/integrate/korvet/kafka-api/_index.md | 49 ++ .../korvet/kafka-api/compatibility.md | 115 +++ content/integrate/korvet/kafka-api/consume.md | 83 +++ .../integrate/korvet/kafka-api/databricks.md | 200 +++++ content/integrate/korvet/kafka-api/produce.md | 80 ++ .../korvet/kafka-api/schema-registry.md | 41 ++ content/integrate/korvet/kafka-api/topics.md | 383 ++++++++++ content/integrate/korvet/operations/_index.md | 39 + .../integrate/korvet/operations/admin-api.md | 107 +++ .../korvet/operations/authentication.md | 450 +++++++++++ .../integrate/korvet/operations/benchmarks.md | 453 ++++++++++++ .../integrate/korvet/operations/deployment.md | 582 +++++++++++++++ .../integrate/korvet/operations/kubernetes.md | 435 +++++++++++ .../integrate/korvet/operations/logging.md | 186 +++++ .../integrate/korvet/operations/migration.md | 324 ++++++++ .../integrate/korvet/operations/monitoring.md | 225 ++++++ .../korvet/operations/production-tuning.md | 67 ++ .../integrate/korvet/operations/resilience.md | 130 ++++ .../korvet/operations/troubleshooting.md | 372 ++++++++++ content/integrate/korvet/operations/web-ui.md | 151 ++++ .../integrate/korvet/quick-start/_index.md | 47 ++ .../korvet/quick-start/configuration.md | 349 +++++++++ content/integrate/korvet/quick-start/demo.md | 96 +++ .../integrate/korvet/quick-start/docker.md | 147 ++++ .../korvet/quick-start/hello-world.md | 90 +++ .../integrate/korvet/quick-start/install.md | 100 +++ content/integrate/korvet/reference/api.md | 192 +++++ .../korvet/reference/configuration.md | 696 ++++++++++++++++++ content/integrate/korvet/reference/faq.md | 203 +++++ .../korvet/reference/metrics/_index.md | 44 ++ .../korvet/reference/metrics/application.md | 29 + .../korvet/reference/metrics/broker.md | 198 +++++ .../korvet/reference/metrics/contracts.md | 73 ++ .../korvet/reference/metrics/mapper.md | 72 ++ .../korvet/reference/metrics/redis-client.md | 44 ++ .../reference/metrics/storage-worker.md | 34 + .../korvet/reference/metrics/storage.md | 386 ++++++++++ content/integrate/korvet/storage/_index.md | 129 ++++ .../korvet/storage/migrate-to-segmented.md | 76 ++ .../integrate/korvet/storage/redis-streams.md | 279 +++++++ .../korvet/storage/remote-storage.md | 226 ++++++ 42 files changed, 8079 insertions(+) create mode 100644 content/integrate/korvet/concepts.md create mode 100644 content/integrate/korvet/kafka-api/_index.md create mode 100644 content/integrate/korvet/kafka-api/compatibility.md create mode 100644 content/integrate/korvet/kafka-api/consume.md create mode 100644 content/integrate/korvet/kafka-api/databricks.md create mode 100644 content/integrate/korvet/kafka-api/produce.md create mode 100644 content/integrate/korvet/kafka-api/schema-registry.md create mode 100644 content/integrate/korvet/kafka-api/topics.md create mode 100644 content/integrate/korvet/operations/_index.md create mode 100644 content/integrate/korvet/operations/admin-api.md create mode 100644 content/integrate/korvet/operations/authentication.md create mode 100644 content/integrate/korvet/operations/benchmarks.md create mode 100644 content/integrate/korvet/operations/deployment.md create mode 100644 content/integrate/korvet/operations/kubernetes.md create mode 100644 content/integrate/korvet/operations/logging.md create mode 100644 content/integrate/korvet/operations/migration.md create mode 100644 content/integrate/korvet/operations/monitoring.md create mode 100644 content/integrate/korvet/operations/production-tuning.md create mode 100644 content/integrate/korvet/operations/resilience.md create mode 100644 content/integrate/korvet/operations/troubleshooting.md create mode 100644 content/integrate/korvet/operations/web-ui.md create mode 100644 content/integrate/korvet/quick-start/_index.md create mode 100644 content/integrate/korvet/quick-start/configuration.md create mode 100644 content/integrate/korvet/quick-start/demo.md create mode 100644 content/integrate/korvet/quick-start/docker.md create mode 100644 content/integrate/korvet/quick-start/hello-world.md create mode 100644 content/integrate/korvet/quick-start/install.md create mode 100644 content/integrate/korvet/reference/api.md create mode 100644 content/integrate/korvet/reference/configuration.md create mode 100644 content/integrate/korvet/reference/faq.md create mode 100644 content/integrate/korvet/reference/metrics/_index.md create mode 100644 content/integrate/korvet/reference/metrics/application.md create mode 100644 content/integrate/korvet/reference/metrics/broker.md create mode 100644 content/integrate/korvet/reference/metrics/contracts.md create mode 100644 content/integrate/korvet/reference/metrics/mapper.md create mode 100644 content/integrate/korvet/reference/metrics/redis-client.md create mode 100644 content/integrate/korvet/reference/metrics/storage-worker.md create mode 100644 content/integrate/korvet/reference/metrics/storage.md create mode 100644 content/integrate/korvet/storage/_index.md create mode 100644 content/integrate/korvet/storage/migrate-to-segmented.md create mode 100644 content/integrate/korvet/storage/redis-streams.md create mode 100644 content/integrate/korvet/storage/remote-storage.md diff --git a/content/integrate/korvet/concepts.md b/content/integrate/korvet/concepts.md new file mode 100644 index 0000000000..c9fa1c9ed4 --- /dev/null +++ b/content/integrate/korvet/concepts.md @@ -0,0 +1,97 @@ +--- +Title: Concepts & Architecture +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: How Korvet maps Kafka topics, offsets, and consumer groups onto Redis + Streams primitives. +linkTitle: Concepts & Architecture +weight: 20 +--- + +Korvet is a Kafka-compatible streaming service backed by Redis Streams. This page explains +the core concepts and the high-level architecture: how Kafka topics, offsets, and consumer groups map onto +Redis primitives. If you only need to run or use Korvet, the [Get Started]({{< relref "/integrate/korvet/quick-start" >}}) +and [Using the Kafka API]({{< relref "/integrate/korvet/kafka-api" >}}) sections are enough. For the request-by-request implementation +details, see Under the Hood. + +## Architecture + +Korvet exposes a Kafka-compatible broker interface while persisting data in Redis-backed +storage. At a high level: + +- Kafka protocol requests are terminated by the broker. +- Kafka topics and partitions map onto Redis Streams. +- Kafka offsets are translated to and from Redis Stream entry IDs. +- Consumer-group behavior combines broker-side coordination with Redis-native delivery primitives. + +{{< image filename="images/korvet/architecture.svg" alt="Architecture Overview" >}} + +### Kafka to Redis Model + +| Kafka Concept | How Korvet Implements It | +|---|---| +| Topic partition | A Redis Stream | +| Message record (key, value, headers, timestamp) | A single Redis Stream entry | +| Message offset | Encoded from the Redis Stream entry ID — no offset table to maintain | +| Consumer group | A Redis Streams consumer group | +| Committed offsets and topic metadata | Tracked in Redis | + +For the exact keys and structures, see Redis Data Structures. + +### Design Principles + +- **Compatibility**: Maintain full Kafka protocol compatibility so existing clients and tools work unchanged. +- **Statelessness**: Offsets are computed from Redis entry IDs rather than stored in side tables (see [Topics, Partitions, and Offsets](#topics-partitions-and-offsets)). +- **Atomicity**: Use Redis transactions (`MULTI`/`EXEC`) and Lua scripts for atomic operations. +- **Performance**: Leverage pipelining, connection pooling, and caching for high throughput. + +## Topics, Partitions, and Offsets + +Like Kafka, Korvet organizes messages into topics and partitions: + +- **Topic**: A logical stream of messages (e.g., `orders`, `events`). +- **Partition**: A topic is divided into partitions for parallelism; each partition maps to a Redis Stream. +- **Offset**: Each message has a unique, monotonically increasing offset within its partition. + +Offsets are **stateless**: rather than maintaining a side table, Korvet encodes the Kafka +offset directly from the Redis Stream entry ID (`{timestamp}-{sequence}`). This makes offset conversion an +O(1) computation in both directions and requires no extra storage. The number of bits reserved for the +sequence is tunable per topic (`offset-sequence-bits`) to trade write throughput against batch coherence; +see [the configuration reference]({{< relref "/integrate/korvet/reference/configuration" >}}) for defaults and limits, and +Offset Encoding for the exact formula. + +## Consumer Groups + +Korvet implements Kafka consumer groups on top of Redis Streams native consumer groups: + +- **Coordination**: The broker implements the Kafka group coordinator protocol (join, sync, heartbeat, rebalance). +- **Delivery**: Redis Streams consumer groups (`XREADGROUP`) handle per-consumer delivery state. +- **Offset management**: Explicit Kafka commits are tracked in a separate committed-offset store, so consumer + progress survives restarts. +- **Membership**: Active group membership (members, assignments, generation) is held in broker memory. After a + broker restart, groups with committed offsets remain visible to admin APIs in the `Empty` state until their + clients rejoin. + +## Message Format + +Messages follow the Kafka record format: + +- **Key**: Optional message key (byte array). +- **Value**: Message payload (byte array). +- **Headers**: Optional key-value metadata. +- **Timestamp**: Message timestamp. + +Each Kafka record is stored as a single Redis Stream entry whose body breaks the record out into separate, +directly-readable `value`, `key`, `headers`, and `timestamp` fields, so a non-Kafka client can read the +payload straight from the stream (for example with `XRANGE`). A field is omitted when its component is +absent. See Redis Data Structures for the exact +layout. + +## Next Steps + +- [Using the Kafka API]({{< relref "/integrate/korvet/kafka-api" >}}) — produce, consume, and manage topics +- [Tiered Storage]({{< relref "/integrate/korvet/storage" >}}) — local and remote tiers, and how data moves between them +- Under the Hood — request workflows, protocol mapping, and internals diff --git a/content/integrate/korvet/kafka-api/_index.md b/content/integrate/korvet/kafka-api/_index.md new file mode 100644 index 0000000000..cda291a5ce --- /dev/null +++ b/content/integrate/korvet/kafka-api/_index.md @@ -0,0 +1,49 @@ +--- +Title: Kafka API +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Use existing Kafka clients and tools with Korvet's implementation of + the Kafka protocol. +hideListLinks: false +linkTitle: Using the Kafka API +weight: 30 +--- + +Korvet implements the Kafka protocol, allowing you to use existing Kafka clients and tools. + +## Supported Operations + +- **Produce**: Send messages to topics +- **Fetch**: Read messages from topics +- **Consumer Groups**: Coordinate multiple consumers with offset tracking +- **Topic Management**: Create and list topics + +## Client Compatibility + +Korvet is compatible with standard Kafka clients: + +- **Java**: kafka-clients library +- **Python**: kafka-python, confluent-kafka-python +- **Go**: sarama, confluent-kafka-go +- **Node.js**: kafkajs, node-rdkafka +- **Command-line**: kafka-console-producer, kafka-console-consumer +- **Databricks**: Spark Structured Streaming (see [Databricks Integration]({{< relref "/integrate/korvet/kafka-api/databricks" >}})) + +## Connection + +Connect to Korvet using the standard Kafka bootstrap server configuration: + +```properties +bootstrap.servers=localhost:9092 +``` + +## Next Steps + +- [Producing messages]({{< relref "/integrate/korvet/kafka-api/produce" >}}) +- [Consuming messages]({{< relref "/integrate/korvet/kafka-api/consume" >}}) +- [Topic management]({{< relref "/integrate/korvet/kafka-api/topics" >}}) +- [Databricks integration]({{< relref "/integrate/korvet/kafka-api/databricks" >}}) +- [Kafka compatibility details]({{< relref "/integrate/korvet/kafka-api/compatibility" >}}) diff --git a/content/integrate/korvet/kafka-api/compatibility.md b/content/integrate/korvet/kafka-api/compatibility.md new file mode 100644 index 0000000000..19a0f639b3 --- /dev/null +++ b/content/integrate/korvet/kafka-api/compatibility.md @@ -0,0 +1,115 @@ +--- +Title: Kafka Compatibility +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Korvet implements a subset of the Kafka protocol for compatibility with + existing clients and tools. +linkTitle: Kafka Compatibility +weight: 60 +--- + +Korvet implements a subset of the Kafka protocol for compatibility with existing clients and tools. + +## Supported APIs + +| API | Status | Notes | +|---|---|---| +| Produce | ✅ Supported | Send messages to topics | +| Fetch | ✅ Supported | Read messages from topics | +| Metadata | ✅ Supported | Topic and partition information | +| ApiVersions | ✅ Supported | Protocol version negotiation | +| Consumer Groups | ✅ Supported | JoinGroup, SyncGroup, Heartbeat, LeaveGroup, OffsetCommit, OffsetFetch | +| Transactions | ❌ Not planned | Use Redis transactions instead | +| Admin API | ✅ Supported | CreateTopics, DeleteTopics, DescribeConfigs, AlterConfigs, IncrementalAlterConfigs, DescribeCluster, ListGroups, DescribeGroups, DeleteGroups | +| Idempotent Producers | ✅ Supported | InitProducerId API for idempotent producer support (non-transactional) | + +## Kafka Version Compatibility + +Korvet uses the Apache Kafka client library version 3.9.2 and is compatible with Kafka clients from version 2.8.0 and later. + +### Client Compatibility + +- **Minimum supported client version**: 2.8.0 +- **Recommended client version**: 3.9.x +- **Kafka client library**: 3.9.2 + +Kafka clients are backward compatible, so newer clients (3.x, 4.x) can connect to Korvet without issues. + +### Protocol Features + +Korvet implements Kafka protocol features equivalent to Kafka 2.8.0+, including: + +- Produce API (the version range advertised by the bundled kafka-clients 3.9.2 library) +- Fetch API (v0-v12; capped at v12 for stability) +- Consumer Group Protocol (JoinGroup, SyncGroup, Heartbeat, LeaveGroup) +- Offset Management (OffsetCommit, OffsetFetch) +- Topic Administration (CreateTopics, DeleteTopics) +- Metadata API + +## Compression + +Korvet supports all Kafka compression types: + +- **NONE**: No compression (default) +- **GZIP**: Good compression ratio, higher CPU usage +- **SNAPPY**: Balanced compression and speed +- **LZ4**: Fast compression, lower CPU usage +- **ZSTD**: Best compression ratio, moderate CPU usage + +### How Compression Works + +Korvet implements server-side compression: + +1. **Producer side**: Kafka clients can send compressed or uncompressed batches. Korvet automatically decompresses incoming batches into individual records before storing them. + +2. **Consumer side**: When consumers fetch messages, Korvet compresses the response based on the topic's `compression.type` configuration (not the producer's compression setting). + +3. **At rest**: The producer's Kafka batch compression is not retained — each record is stored as its own Redis Stream entry. The storage backend may then apply its own configurable at-rest compression to each record's `value` field (`korvet.storage.local.compression.codec`, default `none`), independently of the Kafka `compression.type` used on the wire. + +### Configuring Compression + +Compression is configured per-topic using the `compression.type` setting: + +```bash +# Set compression for a topic (requires Admin API support) +kafka-configs --bootstrap-server localhost:9092 \ + --entity-type topics \ + --entity-name my-topic \ + --alter \ + --add-config compression.type=lz4 +``` + +The default compression type is `NONE`. + +### Benefits + +- **Network bandwidth**: Compression reduces the amount of data transferred between Korvet and consumers +- **Flexibility**: Different topics can use different compression algorithms based on their data characteristics +- **Compatibility**: Works transparently with all Kafka clients + +## Limitations + +- **Replication factor**: Always 1 (Redis provides persistence) +- **Transactions**: Not supported +- **Exactly-once semantics**: Not supported (at-least-once delivery) +- **Consumer group membership**: Held in broker memory. Committed offsets are durable in Redis, so after a broker restart `ListGroups` and `DescribeGroups` report groups with committed offsets in the `Empty` state (with no member details) until clients rejoin. + +## Client Configuration + +Most Kafka client configurations work with Korvet. Some settings are ignored: + +- `acks`: Always treated as `acks=1` +- `replication.factor`: Ignored (always 1) +- `min.insync.replicas`: Ignored + +## Testing Compatibility + +You can test Korvet with your existing Kafka applications by simply changing the `bootstrap.servers` configuration to point to Korvet. + +## Next Steps + +- [Producing messages]({{< relref "/integrate/korvet/kafka-api/produce" >}}) +- [Consuming messages]({{< relref "/integrate/korvet/kafka-api/consume" >}}) diff --git a/content/integrate/korvet/kafka-api/consume.md b/content/integrate/korvet/kafka-api/consume.md new file mode 100644 index 0000000000..18200f9d3f --- /dev/null +++ b/content/integrate/korvet/kafka-api/consume.md @@ -0,0 +1,83 @@ +--- +Title: Consuming Messages +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: How to consume messages from Korvet using Kafka clients. +linkTitle: Consuming Messages +weight: 20 +--- + +This guide shows how to consume messages from Korvet using Kafka clients. + +## Using kafka-console-consumer + +The simplest way to consume messages: + +```bash +kafka-console-consumer --bootstrap-server localhost:9092 \ + --topic my-topic \ + --from-beginning +``` + +## Java Consumer + +```java +Properties props = new Properties(); +props.put("bootstrap.servers", "localhost:9092"); +props.put("group.id", "my-consumer-group"); +props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); +props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); + +KafkaConsumer consumer = new KafkaConsumer<>(props); +consumer.subscribe(Collections.singletonList("my-topic")); + +while (true) { + ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); + for (ConsumerRecord record : records) { + System.out.printf("offset = %d, key = %s, value = %s%n", + record.offset(), record.key(), record.value()); + } +} +``` + +## Python Consumer + +```python +from kafka import KafkaConsumer + +consumer = KafkaConsumer( + 'my-topic', + bootstrap_servers='localhost:9092', + group_id='my-consumer-group', + auto_offset_reset='earliest' +) + +for message in consumer: + print(f"Offset: {message.offset}, Value: {message.value}") +``` + +## Offset Management + +Korvet tracks consumer offsets to ensure messages are not lost or duplicated: + +- **Auto-commit**: Offsets are automatically committed periodically +- **Manual commit**: You can control when offsets are committed + +## Reading from Specific Offset + +You can start reading from a specific offset: + +```bash +kafka-console-consumer --bootstrap-server localhost:9092 \ + --topic my-topic \ + --partition 0 \ + --offset 100 +``` + +## Next Steps + +- [Producing messages]({{< relref "/integrate/korvet/kafka-api/produce" >}}) +- [Topic management]({{< relref "/integrate/korvet/kafka-api/topics" >}}) diff --git a/content/integrate/korvet/kafka-api/databricks.md b/content/integrate/korvet/kafka-api/databricks.md new file mode 100644 index 0000000000..3397fb6079 --- /dev/null +++ b/content/integrate/korvet/kafka-api/databricks.md @@ -0,0 +1,200 @@ +--- +Title: Databricks Integration +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Integrate Korvet with Databricks Spark Structured Streaming using the + standard Kafka connector. +linkTitle: Databricks Integration +weight: 50 +--- + +This guide covers integrating Korvet with Databricks Spark Structured Streaming. + +## Overview + +Databricks can connect to Korvet using the standard Kafka connector for Spark Structured Streaming. This allows you to: + +- Stream data from Kafka topics into Databricks for processing +- Write processed data back to Kafka topics +- Use Databricks for stream processing, Delta Lake sinks, or analytics on top of Korvet topics + +## Prerequisites + +- Korvet server running and accessible from Databricks +- Network connectivity between Databricks and Korvet (see [Network Requirements](#network-requirements)) +- Kafka topic created in Korvet + +## Basic Usage + +### Reading from Kafka + +```python +# Read from a Kafka topic +df = spark.readStream \ + .format("kafka") \ + .option("kafka.bootstrap.servers", ":9092") \ + .option("subscribe", "my-topic") \ + .option("startingOffsets", "latest") \ + .load() + +# Parse the value column (Kafka messages are binary) +from pyspark.sql.functions import col, from_json +from pyspark.sql.types import StructType, StructField, StringType + +schema = StructType([ + StructField("id", StringType()), + StructField("name", StringType()), + StructField("timestamp", StringType()) +]) + +parsed_df = df.select( + col("key").cast("string"), + from_json(col("value").cast("string"), schema).alias("data"), + col("topic"), + col("partition"), + col("offset"), + col("timestamp") +).select("key", "data.*", "topic", "partition", "offset", "timestamp") + +# Display the stream +display(parsed_df) +``` + +### Writing to Kafka + +```python +# Write to a Kafka topic +query = parsed_df \ + .selectExpr("key", "to_json(struct(*)) AS value") \ + .writeStream \ + .format("kafka") \ + .option("kafka.bootstrap.servers", ":9092") \ + .option("topic", "output-topic") \ + .option("checkpointLocation", "/tmp/checkpoint") \ + .start() + +query.awaitTermination() +``` + +## Configuration + +### Recommended Settings + +```python +df = spark.readStream \ + .format("kafka") \ + .option("kafka.bootstrap.servers", ":9092") \ + .option("subscribe", "my-topic") \ + .option("startingOffsets", "latest") \ + .option("maxOffsetsPerTrigger", 10000) \ # Limit records per micro-batch + .option("kafka.session.timeout.ms", "30000") \ # 30 seconds + .option("kafka.request.timeout.ms", "60000") \ # 60 seconds + .load() +``` + +### Starting From a Timestamp + +Spark can also start from a time boundary instead of only `earliest` or `latest`. +This is useful when you want to avoid replaying an entire topic while still +starting near a known point in time. + +```python +df = spark.readStream \ + .format("kafka") \ + .option("kafka.bootstrap.servers", ":9092") \ + .option("subscribe", "my-topic") \ + .option("startingTimestamp", "1711324800000") \ # epoch millis (UTC) + .load() +``` + +For per-partition control, use `startingOffsetsByTimestamp`: + +```python +df = spark.readStream \ + .format("kafka") \ + .option("kafka.bootstrap.servers", ":9092") \ + .option("subscribe", "my-topic") \ + .option("startingOffsetsByTimestamp", '{"my-topic":{"0":1711324800000,"1":1711328400000}}') \ + .load() +``` + +### Performance Tuning + +For high-throughput scenarios: + +```python +df = spark.readStream \ + .format("kafka") \ + .option("kafka.bootstrap.servers", ":9092") \ + .option("subscribe", "my-topic") \ + .option("maxOffsetsPerTrigger", 100000) \ # Process more records per batch + .option("minPartitions", 8) \ # Create more Spark partitions + .load() +``` + +## Network Requirements + +Databricks must be able to reach Korvet on port 9092. Common deployment scenarios: + +### Same VPC + +If Databricks and Korvet are in the same AWS VPC: + +1. Ensure security groups allow traffic on port 9092 +2. Use Korvet's private IP or internal load balancer hostname + +### Different VPCs + +If Databricks and Korvet are in different VPCs: + +1. Set up VPC peering between the VPCs +2. Update route tables to allow traffic +3. Update security groups to allow port 9092 + +### Internet-Facing Load Balancer + +If Korvet is behind an internet-facing AWS load balancer: + +1. Configure security group to allow Databricks IP ranges on port 9092 +2. See [Databricks IP ranges](https://docs.databricks.com/resources/supported-regions.html) +3. Configure Korvet's advertised address: + + ```bash + export KORVET_BROKER_ADVERTISED_HOST= + export KORVET_BROKER_ADVERTISED_PORT=9092 + ``` + +## Troubleshooting + +### Timeout Error + +If you see: + +``` +TimeoutException: Timed out waiting for a node assignment. Call: describeTopics +``` + +This indicates a network connectivity issue. See [Databricks Troubleshooting]({{< relref "/integrate/korvet/operations/troubleshooting#databricks-spark-structured-streaming-timeout" >}}). + +**Quick diagnosis**: + +```python +import socket +socket.create_connection(("", 9092), timeout=10) +``` + +If this times out, Databricks cannot reach Korvet. Check: + +- Security groups +- Network ACLs +- VPC peering (if applicable) +- Load balancer configuration + +## Next Steps + +- [Consuming Messages]({{< relref "/integrate/korvet/kafka-api/consume" >}}) +- [Troubleshooting]({{< relref "/integrate/korvet/operations/troubleshooting" >}}) +- [Configuration Reference]({{< relref "/integrate/korvet/reference/configuration" >}}) diff --git a/content/integrate/korvet/kafka-api/produce.md b/content/integrate/korvet/kafka-api/produce.md new file mode 100644 index 0000000000..7e6888085c --- /dev/null +++ b/content/integrate/korvet/kafka-api/produce.md @@ -0,0 +1,80 @@ +--- +Title: Producing Messages +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: How to produce messages to Korvet using Kafka clients. +linkTitle: Producing Messages +weight: 10 +--- + +This guide shows how to produce messages to Korvet using Kafka clients. + +## Using kafka-console-producer + +The simplest way to produce messages: + +```bash +kafka-console-producer --bootstrap-server localhost:9092 --topic my-topic +``` + +Type messages and press Enter to send each one. + +## Java Producer + +```java +Properties props = new Properties(); +props.put("bootstrap.servers", "localhost:9092"); +props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); +props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); + +KafkaProducer producer = new KafkaProducer<>(props); + +ProducerRecord record = + new ProducerRecord<>("my-topic", "key", "value"); + +producer.send(record, (metadata, exception) -> { + if (exception == null) { + System.out.println("Sent to partition " + metadata.partition() + + " at offset " + metadata.offset()); + } else { + exception.printStackTrace(); + } +}); + +producer.close(); +``` + +## Python Producer + +```python +from kafka import KafkaProducer + +producer = KafkaProducer(bootstrap_servers='localhost:9092') + +producer.send('my-topic', b'Hello, Korvet!') +producer.flush() +``` + +## Message Format + +Messages consist of: + +- **Key** (optional): Used for partitioning +- **Value**: The message payload +- **Headers** (optional): Key-value metadata +- **Timestamp**: Automatically set if not provided + +## Partitioning + +Messages are distributed across partitions based on: + +- **Key hash**: If a key is provided, messages with the same key go to the same partition +- **Round-robin**: If no key is provided, messages are distributed evenly + +## Next Steps + +- [Consuming messages]({{< relref "/integrate/korvet/kafka-api/consume" >}}) +- [Topic management]({{< relref "/integrate/korvet/kafka-api/topics" >}}) diff --git a/content/integrate/korvet/kafka-api/schema-registry.md b/content/integrate/korvet/kafka-api/schema-registry.md new file mode 100644 index 0000000000..fe05d3daff --- /dev/null +++ b/content/integrate/korvet/kafka-api/schema-registry.md @@ -0,0 +1,41 @@ +--- +Title: Schema Registry +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Korvet exposes a Confluent Schema Registry-compatible REST API for registering + and looking up Avro, Protobuf, and JSON Schema subjects. +linkTitle: Schema Registry +weight: 40 +--- + +Korvet exposes a Confluent Schema Registry-compatible REST API on the existing Spring Boot HTTP port when `korvet.schema-registry.enabled=true`. +Kafka clients should configure `schema.registry.url` to the Korvet HTTP base URL and continue using Confluent serializers and deserializers. + +The registry supports registering and looking up Avro, Protobuf, and JSON Schema subjects, managing global and subject compatibility levels, and checking compatibility for new schemas. +The Kafka broker preserves Confluent-encoded key and value bytes unchanged. +When `korvet.schema-registry.validate-produce=true`, Korvet validates produced records that belong to registered `-key` or `-value` subjects and stores schema identity metadata alongside the archived payload. + +```yaml +korvet: + schema-registry: + enabled: true + default-compatibility: BACKWARD + validate-produce: true +``` + +Supported v1 endpoints include: + +- `GET /subjects` +- `GET /subjects/{subject}/versions` +- `GET /subjects/{subject}/versions/{version}` +- `GET /schemas/ids/{id}` +- `POST /subjects/{subject}/versions` +- `POST /subjects/{subject}` +- `POST /compatibility/subjects/{subject}/versions/{version}` +- `GET /config`, `PUT /config` +- `GET /config/{subject}`, `PUT /config/{subject}` + +Deletes, modes, exporters, contexts, and ACLs are intentionally out of scope for the first implementation. diff --git a/content/integrate/korvet/kafka-api/topics.md b/content/integrate/korvet/kafka-api/topics.md new file mode 100644 index 0000000000..df874fb4bf --- /dev/null +++ b/content/integrate/korvet/kafka-api/topics.md @@ -0,0 +1,383 @@ +--- +Title: Topic Management +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: How to create and manage topics in Korvet. +linkTitle: Topic Management +weight: 30 +--- + +This guide covers creating and managing topics in Korvet. + +{{< note >}} +Topic defaults — including auto-create — are configured via the pattern list under `korvet.topics`. Each entry's `name` is a glob pattern matched against topic names; entries are evaluated in declared order and combined first-match-wins per field. +{{< /note >}} + +## Creating Topics + +### Automatic Topic Creation + +By default, topics are not automatically created when you first produce to them or request metadata for them. +Auto-creation is configured per pattern under `korvet.topics`: + +```yaml +korvet: + topics: + - name: "*" + auto-create: false # Enable/disable automatic topic creation (default: false) + partitions: 1 # Default partitions for auto-created topics (default: 1) +``` + +When auto-creation is disabled, you must explicitly create topics before using them. + +### Explicit Topic Creation + +You can create topics explicitly with standard Kafka tooling or with the bundled `korvet` CLI. + +#### Using `korvet topics` + +`korvet topics` mirrors `kafka-topics` syntax. It is a thin wrapper over the Kafka AdminClient +and passes `--config key=value` pairs through verbatim. The only Korvet-specific topic config is +`offset.sequence.bits`; all other accepted keys are standard Kafka topic configs such as `retention.ms` +and `segment.ms`. + +```bash +korvet topics --bootstrap-server localhost:9092 \ + --create \ + --topic my-topic \ + --partitions 3 \ + --config retention.ms=604800000 \ + --config offset.sequence.bits=14 \ + --config segment.ms=3600000 +``` + +{{< note >}} +The `korvet topics create` command does not accept `--replication-factor`, as Korvet uses Redis for storage and replication. The upstream `kafka-topics` tool still requires it (see below). +{{< /note >}} + +#### Using `kafka-topics` + +Upstream Kafka CLI tooling works for standard Kafka topic configs: + +```bash +kafka-topics --bootstrap-server localhost:9092 \ + --create \ + --topic my-topic \ + --partitions 3 \ + --replication-factor 1 +``` + +#### Creating Topics with the `offset.sequence.bits` Configuration + +{{< warning >}} +The upstream **`kafka-topics` CLI** performs client-side validation and rejects the Korvet-specific `offset.sequence.bits` config with an `Unknown topic config name` error. + +Use `korvet topics` (a thin AdminClient wrapper that does not validate config names client-side) when you want to set `offset.sequence.bits`, or use the Kafka AdminClient API directly. +{{< /warning >}} + +To set `offset.sequence.bits` with upstream Kafka tooling, use the Kafka AdminClient API, which does **not** perform client-side validation: + +```java +Properties props = new Properties(); +props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); + +try (AdminClient admin = AdminClient.create(props)) { + NewTopic topic = new NewTopic("my-topic", 3, (short) 1); + topic.configs(Map.of( + "offset.sequence.bits", "14", + "retention.ms", "604800000" + )); + admin.createTopics(List.of(topic)).all().get(); +} +``` + +**Accepted topic configurations:** + +The broker accepts only the following topic config keys. Any other key (including `value.type` and `storage.compression`) is rejected with `INVALID_CONFIG` ("Unknown or unsupported topic config"). + +- `retention.ms` - Total time-based retention in milliseconds (across all tiers) +- `retention.bytes` - Total size-based retention in bytes +- `segment.ms` - Duration of each local stream bucket in milliseconds. Must be positive and less than the effective local retention window. Newly created topics default to `86400000` (1 day); the default is applied only when it fits within the effective retention (otherwise the topic rolls on `segment.bytes` alone). Compacted topics keep a single-stream layout and are not segmented. +- `segment.bytes` - Size of each local stream bucket in bytes. Newly created topics default to `134217728` (128 MiB). Compacted topics keep a single-stream layout and are not segmented. Adding `segment.bytes` (or `segment.ms`) to a legacy single-stream topic migrates it to the segmented layout in place — see [Migrating a Single-Stream Topic to Segmented Storage]({{< relref "/integrate/korvet/storage/migrate-to-segmented" >}}). +- `compression.type` - Compression for Kafka fetch responses (`none`, `gzip`, `snappy`, `lz4`, `zstd`). Default: `none` +- `cleanup.policy` - `delete` (retention-based trimming, the default), `compact` (key-based log compaction), or `compact,delete` (both). See [Log Compaction](#log-compaction). +- `message.timestamp.type` - Which timestamp Korvet reports for records on this topic: `CreateTime` (default) reports the producer-supplied creation time; `LogAppendTime` reports the broker append time (the millisecond component of the Redis Stream entry ID). See [Record Timestamps](#record-timestamps). +- `log.append.timestamp.header` - When `true`, every record in fetch responses gets an additional `korvet.log.append.timestamp.ms` header carrying the broker append time, leaving the record's own timestamp untouched. Default: `false`. Available since v0.18.0. See [Record Timestamps](#record-timestamps). +- `offset.sequence.bits` - Bits reserved for the per-millisecond sequence component in Korvet offsets. Range: `1`-`16`. Default: `14`. Settable only at topic creation; cannot be altered. +- `storage.compression.type` - Codec for compressing the record value at rest in Redis (`none`, `gzip`, `snappy`, `lz4`, `zstd`). Unset inherits the server-level `korvet.storage.local.compression.codec` (default `none`). Use `zstd` for JSON/log storage efficiency, `snappy` when CPU cost matters more, and `none` when non-Kafka clients must read values directly with `XRANGE`. Settable only at topic creation; cannot be altered. + +**Tiered storage configurations** (when remote storage is enabled at server level): + +- `remote.storage.enable` - Enable tiered storage for this topic (Kafka KIP-405). Default: `false` +- `local.retention.ms` - Time to keep in the local tier before Redis data expires. `-2` = use `retention.ms` (Kafka KIP-405) +- `local.retention.bytes` - Size to keep in the local tier before Redis trimming falls back to `retention.bytes`. `-2` = use `retention.bytes` (Kafka KIP-405) + +{{< note >}} +At-rest compression of the record value is set per topic via `storage.compression.type` (falling back to the server-level `korvet.storage.local.compression.codec`, default `none`). This is distinct from the Kafka-facing `compression.type`, which only affects fetch-response compression. Prefer `zstd` for JSON/log topics and `snappy` for CPU-sensitive topics. +{{< /note >}} + +## Log Compaction + +Topics created or altered with `cleanup.policy=compact` (or `compact,delete`) are compacted by key: a background pass on the storage worker periodically deletes every record that has been superseded by a newer record with the same key, keeping only the latest record per key. This supports keyed-state topics such as Schema Registry journals and Kafka Connect config/offset topics, which rebuild their state by replaying a compacted topic. + +Because Korvet derives Kafka offsets from Redis stream entry IDs rather than positions, compaction deletes superseded entries in place (`XDEL`): surviving records keep their original offsets, and consumers simply observe offset gaps — the same behavior as Kafka compaction. + +Semantics: + +- `compact`-only topics ignore `retention.ms`/`retention.bytes`: the head of the log is never trimmed, only superseded keys are removed. `compact,delete` applies both compaction and retention trimming. +- Records produced to a compacted topic must have a key; unkeyed records are rejected with `INVALID_RECORD` (standard Kafka behavior). +- Tombstones (records with a null value) are retained as the latest record for their key so replaying consumers observe deletions. Tombstone purging (`delete.retention.ms`) is not implemented; tombstones are kept indefinitely. +- `min.compaction.lag.ms`/`max.compaction.lag.ms` are not supported; compaction runs at the storage worker's tick interval. Records appended while a compaction pass is running are left for the next pass. +- `cleanup.policy=compact` cannot be combined with `remote.storage.enable=true`: compaction is not supported on tiered topics, and the combination is rejected at create/alter time. + +## Record Timestamps + +Every record Korvet stores carries a millisecond timestamp, persisted in the `timestamp` field of its Redis Stream entry. Two per-topic settings control how that timestamp is sourced and reported. + +### `message.timestamp.type` + +- `CreateTime` (default) — the timestamp is the producer-supplied creation time taken from the `ProducerRecord`. It is stored verbatim and returned unchanged to consumers. +- `LogAppendTime` — Korvet reports the broker append time instead: the millisecond component of the Redis Stream entry ID assigned when the record was written. Fetched records carry this value as their timestamp, and produce responses report it in `log_append_time`. + +This mirrors Kafka's `message.timestamp.type`. It changes which value consumers see as **the** record timestamp. + +### `log.append.timestamp.header` + +When set to `true`, Korvet adds a header named `korvet.log.append.timestamp.ms` to every record returned in fetch responses. Its value is the broker append time (the Redis Stream entry ID's millisecond component) as an ASCII-decimal string. + +This is additive and independent of `message.timestamp.type`: + +- The record's own `timestamp` is left untouched — consumers that ignore the header see no change. +- It lets you expose log-append time **alongside** the producer creation time, without switching the topic to `LogAppendTime`. + +The setting defaults to `false` and is available since v0.18.0. + +### Consuming the log-append header + +Enable the header on the topic first: + +```bash +korvet topics --bootstrap-server localhost:9092 \ + --alter --topic logs \ + --config log.append.timestamp.header=true +``` + +The header value is the broker append time in epoch milliseconds, encoded as an ASCII-decimal string. Consumers must request headers and parse the bytes to a `long`. + +#### Java Kafka consumer + +```java +import org.apache.kafka.common.header.Header; +import java.nio.charset.StandardCharsets; + +for (ConsumerRecord record : records) { + Header header = record.headers().lastHeader("korvet.log.append.timestamp.ms"); + if (header != null) { + long appendTimeMs = Long.parseLong(new String(header.value(), StandardCharsets.US_ASCII)); + // record.timestamp() is still the producer CreateTime; + // appendTimeMs is when Korvet wrote it to Redis. + } +} +``` + +#### Python (kafka-python) + +```python +consumer = KafkaConsumer("logs", bootstrap_servers="localhost:9092") +for msg in consumer: + headers = dict(msg.headers) # list of (key, value-bytes) tuples + raw = headers.get("korvet.log.append.timestamp.ms") + append_time_ms = int(raw.decode("ascii")) if raw else None +``` + +#### Spark Structured Streaming + +Spark exposes Kafka headers as an `array>` column, but only when `includeHeaders` is enabled on the source. Pick out the header by key, cast its bytes to a string, then to a `bigint`, and convert to a timestamp: + +```python +kafka_df = ( + spark.readStream.format("kafka") + .option("kafka.bootstrap.servers", "korvet:9092") + .option("subscribe", "logs") + .option("includeHeaders", "true") # required to read headers + .option("startingOffsets", "earliest") + .load() +) + +events = kafka_df.select( + col("value").cast("string").alias("value"), + col("timestamp").alias("producer_create_time"), # CreateTime from the record + col("headers"), +).withColumn( + "korvet_append_time", + expr( + "timestamp_millis(CAST(get(transform(" + "filter(headers, h -> h.key = 'korvet.log.append.timestamp.ms'), " + "h -> CAST(h.value AS STRING)), 0) AS BIGINT))" + ), +).drop("headers") +``` + +`filter(...)` selects the matching header, `transform(...)` decodes its binary value to a string, `get(..., 0)` takes the first match, and `timestamp_millis(...)` turns the epoch-millis `bigint` into a Spark `timestamp`. With both `producer_create_time` and `korvet_append_time` in hand you can compute ingest latency, e.g. `unix_millis(korvet_append_time) - unix_millis(producer_create_time)`. + +{{< tip >}} +A complete, runnable pipeline (Logstash → Korvet → Spark → Delta/S3) that uses this exact expression to measure end-to-end latency lives in `samples/logstash-spark-s3/spark_consumer.py`. +{{< /tip >}} + +## Listing Topics + +List all topics: + +```bash +kafka-topics --bootstrap-server localhost:9092 --list +``` + +## Describing Topics + +Get details about a topic: + +```bash +kafka-topics --bootstrap-server localhost:9092 \ + --describe \ + --topic my-topic +``` + +## Deleting Topics + +Delete a topic: + +```bash +kafka-topics --bootstrap-server localhost:9092 \ + --delete \ + --topic my-topic +``` + +## Altering Topic Configuration + +Topics can be configured with: + +- **Partitions**: Number of partitions for parallelism (set during creation only) +- **Retention**: Time-based (`retention.ms`) and size-based (`retention.bytes`) retention policies +- **Protocol Compression**: Compression for Kafka fetch responses (`compression.type`) +- **At-Rest Compression**: Codec for the record value stored in Redis (`storage.compression.type`, create-time only) +- **Offset Encoding**: Per-topic offset sequence width (`offset.sequence.bits`, create-time only) +- **Bucketing**: Time-bucketed local streams (`segment.ms`, `segment.bytes`) + +### Using kafka-configs CLI + +Use `korvet topics --alter` or `kafka-configs` to alter topic configurations. + +```bash +korvet topics --bootstrap-server localhost:9092 \ + --alter \ + --topic my-topic \ + --config retention.ms=604800000 \ + --config segment.ms=1800000 +``` + +```bash +kafka-configs --bootstrap-server localhost:9092 \ + --entity-type topics \ + --entity-name my-topic \ + --alter \ + --add-config retention.ms=604800000,compression.type=lz4 +``` + +{{< note >}} +`offset.sequence.bits` cannot be altered after topic creation; it is settable only at creation time. +{{< /note >}} + +### Using AdminClient API + +Alternatively, use the AdminClient API: + +```java +ConfigResource topicResource = new ConfigResource(ConfigResource.Type.TOPIC, "my-topic"); +List ops = List.of( + new AlterConfigOp(new ConfigEntry("retention.ms", "604800000"), AlterConfigOp.OpType.SET), + new AlterConfigOp(new ConfigEntry("compression.type", "lz4"), AlterConfigOp.OpType.SET) +); +admin.incrementalAlterConfigs(Map.of(topicResource, ops)).all().get(); +``` + +See [Redis Streams storage]({{< relref "/integrate/korvet/storage/redis-streams" >}}) for details on how records are stored. + +### Describing Topic Configuration + +View current topic configuration using `korvet topics --describe` or `kafka-configs --describe`: + +```bash +korvet topics --bootstrap-server localhost:9092 \ + --describe \ + --topic my-topic +``` + +```bash +kafka-configs --bootstrap-server localhost:9092 \ + --entity-type topics \ + --entity-name my-topic \ + --describe +``` + +**Protocol compression types** (`compression.type`): + +- `none` - No compression (default) +- `gzip` - Good compression ratio, higher CPU usage +- `snappy` - Balanced compression and speed +- `lz4` - Fast compression, lower CPU usage +- `zstd` - Best compression ratio, moderate CPU usage + +{{< note >}} +At-rest compression in Redis is set per topic via `storage.compression.type` (create-time only), falling back to the server-level `korvet.storage.local.compression.codec` (default `none`). It is independent of the Kafka-facing `compression.type`, which only affects fetch-response compression. Prefer `zstd` for JSON/log topics, `snappy` for CPU-sensitive topics, and `none` when non-Kafka clients must read values directly from Redis. +{{< /note >}} + +See [Compression]({{< relref "/integrate/korvet/kafka-api/compatibility#compression" >}}) for more details on protocol compression. + +## Tiered Storage Configuration + +When tiered storage is enabled at the server level, you can configure per-topic retention policies to control when data moves between tiers. + +### Configuring Tiered Storage with AdminClient API + +Use the AdminClient API to configure tiered storage (since `kafka-configs --alter` is not supported): + +```java +NewTopic topic = new NewTopic("my-topic", 3, (short) 1); +topic.configs(Map.of( + "remote.storage.enable", "true", + "retention.ms", "31536000000", // 1 year total + "local.retention.ms", "86400000" // 1 day in local tier +)); +admin.createTopics(List.of(topic)).all().get(); +``` + +This configures: + +- **Local tier**: 1 day (`local.retention.ms=86400000`) +- **Remote tier**: ~364 days (implicit: `retention.ms - local.retention.ms`) +- **Total retention**: 1 year (`retention.ms=31536000000`) + +### Tiered Storage Configuration Reference + +| Configuration | Default | Description | +|---|---|---| +| `remote.storage.enable` | `false` | Enable tiered storage for this topic (Kafka KIP-405) | +| `local.retention.ms` | `-2` | Time to keep in the local tier before Redis data expires. `-2` = use total `retention.ms` | +| `local.retention.bytes` | `-2` | Size to keep in the local tier before Redis trimming falls back to `retention.bytes`. `-2` = use total `retention.bytes` | +| `retention.ms` | `604800000` | Total retention across all tiers (7 days default) | + +{{< note >}} +Remote tier retention is implicit and calculated as `retention.ms - local.retention.ms`. Data is deleted after the total `retention.ms` period. +{{< /note >}} + +See [Remote Storage]({{< relref "/integrate/korvet/storage/remote-storage" >}}) for server-level tiered storage configuration. + +## Next Steps + +- [Producing messages]({{< relref "/integrate/korvet/kafka-api/produce" >}}) +- [Consuming messages]({{< relref "/integrate/korvet/kafka-api/consume" >}}) +- [Redis Streams storage]({{< relref "/integrate/korvet/storage/redis-streams" >}}) +- [Remote Storage (Parquet)]({{< relref "/integrate/korvet/storage/remote-storage" >}}) diff --git a/content/integrate/korvet/operations/_index.md b/content/integrate/korvet/operations/_index.md new file mode 100644 index 0000000000..f74cc82c8e --- /dev/null +++ b/content/integrate/korvet/operations/_index.md @@ -0,0 +1,39 @@ +--- +Title: Operations +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Deploy, monitor, and operate Korvet in production. +hideListLinks: false +linkTitle: Operations +weight: 50 +--- + +This section covers deploying, monitoring, and operating Korvet in production. + +## In This Section + +- [Deployment]({{< relref "/integrate/korvet/operations/deployment" >}}) - Deploy Korvet in production +- [Production Tuning]({{< relref "/integrate/korvet/operations/production-tuning" >}}) - Size the storage connection pool and tune for produce load +- [Kubernetes]({{< relref "/integrate/korvet/operations/kubernetes" >}}) - Run Korvet on Kubernetes +- [Authentication]({{< relref "/integrate/korvet/operations/authentication" >}}) - Secure access with SASL authentication +- [Monitoring]({{< relref "/integrate/korvet/operations/monitoring" >}}) - Monitor health and performance +- [Logging]({{< relref "/integrate/korvet/operations/logging" >}}) - Configure logging and troubleshooting +- [Admin API]({{< relref "/integrate/korvet/operations/admin-api" >}}) - Manage topics, retention, and the cluster +- [Benchmarks]({{< relref "/integrate/korvet/operations/benchmarks" >}}) - Performance benchmarks and how to run them +- [Troubleshooting]({{< relref "/integrate/korvet/operations/troubleshooting" >}}) - Common issues and solutions + +## Production Checklist + +Before deploying to production: + +- [ ] Configure external Redis instance +- [ ] Enable SASL authentication +- [ ] Enable TLS for Kafka protocol +- [ ] Set up monitoring and alerting +- [ ] Configure log aggregation +- [ ] Plan capacity and scaling +- [ ] Test failover scenarios +- [ ] Document runbook procedures diff --git a/content/integrate/korvet/operations/admin-api.md b/content/integrate/korvet/operations/admin-api.md new file mode 100644 index 0000000000..02f5ae3242 --- /dev/null +++ b/content/integrate/korvet/operations/admin-api.md @@ -0,0 +1,107 @@ +--- +Title: Admin API +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: HTTP Admin API for topic administration, consumer-group inspection, + monitoring snapshots, and Kafka SASL credential management. +linkTitle: Admin API +weight: 100 +--- + +Korvet exposes an HTTP Admin API under `/api/v1` for topic administration, consumer-group inspection, monitoring snapshots, and Kafka SASL credential management. + +The Admin API is protected with HTTP Basic authentication. + +## Authentication + +A bootstrap admin credential is created automatically on first startup. +Configure it with: + +```yaml +korvet: + admin: + username: admin + password: admin + bootstrap: true +``` + +Change the default password before exposing the API outside a local development environment. + +## Kafka SASL Credentials + +The credentials API manages Kafka SASL service accounts — the usernames and passwords that Kafka clients use to authenticate against the broker. + +### Supported mechanisms + +- `SCRAM-SHA-256` (default) — recommended for most deployments. +- `PLAIN` — requires TLS to be enabled on the broker (`korvet.broker.tls=true`). + +### Endpoints + +- `POST /api/v1/credentials` — create a credential. +- `GET /api/v1/credentials` — list all credentials. +- `GET /api/v1/credentials/{username}` — get one credential. +- `PUT /api/v1/credentials/{username}` — rotate the password or change the mechanism. +- `DELETE /api/v1/credentials/{username}` — delete a credential. + +### Create a credential + +```http +POST /api/v1/credentials +Content-Type: application/json + +{ + "username": "kafka-client-1", + "password": "secret123", + "mechanism": "SCRAM-SHA-256" +} +``` + +`mechanism` is optional and defaults to `SCRAM-SHA-256`. +`username` must be 3–64 characters and contain only letters, digits, `-`, `_`, or `.`. +`password` must be 8–128 characters with no whitespace. + +### Rotate a password + +```http +PUT /api/v1/credentials/kafka-client-1 +Content-Type: application/json + +{ "password": "newSecret456" } +``` + +Omitting `mechanism` keeps the credential's existing mechanism. + +## Topics + +- `POST /api/v1/topics` creates a topic. +- `GET /api/v1/topics` lists topics. +- `GET /api/v1/topics/{name}` returns one topic. +- `GET /api/v1/topics/{name}/partitions` returns current per-partition stream and offset stats. +- `PUT /api/v1/topics/{name}` replaces explicit topic configuration. +- `DELETE /api/v1/topics/{name}` deletes a topic. + +## Consumer Groups + +- `GET /api/v1/consumer-groups` lists consumer groups. +- `GET /api/v1/consumer-groups/{groupId}` returns group state and members. + +## Monitoring + +- `GET /api/v1/health` returns a stable Admin API health schema backed by Spring Boot health. +- `GET /api/v1/metrics` returns a curated metrics snapshot. +- `GET /api/v1/storage-stats` returns remote storage diagnostics when available. +- `GET /api/v1/storage/offload-jobs` returns the current segment-derived offload queue view. In + Phase 1, this endpoint reports `pending`, `running`, and limited `done` rows from existing segment + state only; it does not expose durable job history, retries, failures, or cancellation. The optional + query parameters are `status` (`pending`, `running`, or `done`; repeatable), `offset` (zero-based row + offset, default `0`), and `limit` (default `100`, maximum `500`). + +## OpenAPI + +OpenAPI JSON is available at `/v3/api-docs`. + +Swagger UI is available at `/swagger-ui.html`. diff --git a/content/integrate/korvet/operations/authentication.md b/content/integrate/korvet/operations/authentication.md new file mode 100644 index 0000000000..5f18d5c410 --- /dev/null +++ b/content/integrate/korvet/operations/authentication.md @@ -0,0 +1,450 @@ +--- +Title: Authentication +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Configure SASL authentication to secure access to Korvet. +linkTitle: Authentication +weight: 40 +--- + +This guide covers configuring SASL authentication to secure access to Korvet. + +## Overview + +Korvet supports SASL (Simple Authentication and Security Layer) authentication to control access to the Kafka protocol endpoint. When enabled, clients must authenticate before producing or consuming messages. + +### Supported Mechanisms + +- **SASL/SCRAM-SHA-256** - Challenge-response authentication using stored SCRAM keys. This is the default mechanism. +- **SASL/PLAIN** - Username and password authentication. Must be opted in explicitly and requires TLS (see below). + +By default, only `SCRAM-SHA-256` is advertised to clients. The advertised mechanisms are controlled by `korvet.broker.sasl.mechanisms` (default `SCRAM-SHA-256`). + +## Enabling Authentication + +### Configuration + +Enable SASL authentication in your Korvet configuration: + +```yaml +korvet: + broker: + sasl: + enabled: true + mechanisms: + - SCRAM-SHA-256 +``` + +Or using environment variables: + +```bash +KORVET_BROKER_SASL_ENABLED=true +KORVET_BROKER_SASL_MECHANISMS=SCRAM-SHA-256 +``` + +### Enabling PLAIN + +`PLAIN` is not advertised by default. To use it, add it explicitly to `korvet.broker.sasl.mechanisms`. Because `PLAIN` transmits credentials without encryption, Korvet requires TLS to be enabled when `PLAIN` is advertised. Starting the broker with `PLAIN` advertised while `korvet.broker.tls=false` fails validation at startup with: + +``` +korvet.broker.sasl: PLAIN mechanism requires korvet.broker.tls=true +``` + +Clients using `PLAIN` must therefore connect with the `SASL_SSL` security protocol, not `SASL_PLAINTEXT`. + +```yaml +korvet: + broker: + tls: true + sasl: + enabled: true + mechanisms: + - SCRAM-SHA-256 + - PLAIN +``` + +## Managing Credentials + +### Credential Storage + +Credentials are stored in Redis using secure PBKDF2 password hashing: + +- **Algorithm**: PBKDF2WithHmacSHA256 +- **Iterations**: 10,000 +- **Salt**: 128-bit random per credential +- **Hash**: 256-bit output + +### Creating User Credentials + +Use the Korvet admin API or Redis CLI to create user credentials. The admin API +(see [Deployment]({{< relref "/integrate/korvet/operations/deployment" >}}) for enabling it) exposes +credential management over HTTP; the Redis CLI approach below is shown for +direct access. + +#### Using Redis CLI + +```bash +# Store a SCRAM-SHA-256 credential for user "alice" in tenant "tenant1" +redis-cli HSET korvet:broker:credentials:alice \ + mechanism SCRAM-SHA-256 \ + password_hash \ + server_key \ + salt \ + iterations 10000 \ + tenant_id tenant1 +``` + +For SCRAM, `password_hash` stores the Base64-encoded StoredKey and `server_key` +stores the Base64-encoded ServerKey. Do not store the salted password. + +To create a `PLAIN` credential (only usable when `PLAIN` is advertised and TLS +is enabled), use `mechanism PLAIN` with `password_hash`, `salt`, and +`iterations` fields. + +#### Programmatic Creation + +```java +import com.redis.korvet.broker.redis.RedisCredentialStore; + +// Create credential store +RedisCredentialStore credentialStore = + new RedisCredentialStore(redisClient, "korvet"); +PasswordHasher passwordHasher = new PasswordHasher(); + +// Hash the password +PasswordHasher.HashedPassword hashed = + passwordHasher.hashPassword("secret-password"); + +// Store the credential +StoredCredential credential = StoredCredential.builder() + .username("alice") + .mechanism("PLAIN") + .passwordHash(hashed.getHash()) + .salt(hashed.getSalt()) + .iterations(hashed.getIterations()) + .tenantId("tenant1") + .build(); + +credentialStore.storeCredential(credential); +``` + +### Updating Credentials + +To update a user's password, store a new credential with the same username: + +```java +// Hash new password +PasswordHasher.HashedPassword newHashed = + passwordHasher.hashPassword("new-password"); + +// Update credential +StoredCredential updated = StoredCredential.builder() + .username("alice") + .mechanism("PLAIN") + .passwordHash(newHashed.getHash()) + .salt(newHashed.getSalt()) + .iterations(newHashed.getIterations()) + .tenantId("tenant1") + .build(); + +credentialStore.storeCredential(updated); +``` + +### Deleting Credentials + +```java +credentialStore.deleteCredential("alice"); +``` + +Or using Redis CLI: + +```bash +redis-cli DEL korvet:broker:credentials:alice +``` + +## Client Configuration + +### Kafka Producer + +```java +Properties props = new Properties(); +props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); +props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, + StringSerializer.class.getName()); +props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, + StringSerializer.class.getName()); + +// SASL configuration (default SCRAM-SHA-256 mechanism) +props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT"); +props.put(SaslConfigs.SASL_MECHANISM, "SCRAM-SHA-256"); +props.put(SaslConfigs.SASL_JAAS_CONFIG, + "org.apache.kafka.common.security.scram.ScramLoginModule required " + + "username=\"alice\" " + + "password=\"secret-password\";"); + +KafkaProducer producer = new KafkaProducer<>(props); +``` + +For `PLAIN` clients, use the `SASL_SSL` security protocol (PLAIN requires TLS) and Kafka's PLAIN login module: + +```java +props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL"); +props.put(SaslConfigs.SASL_MECHANISM, "PLAIN"); +props.put(SaslConfigs.SASL_JAAS_CONFIG, + "org.apache.kafka.common.security.plain.PlainLoginModule required " + + "username=\"alice\" " + + "password=\"secret-password\";"); +``` + +### Kafka Consumer + +```java +Properties props = new Properties(); +props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); +props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-group"); +props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, + StringDeserializer.class.getName()); +props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, + StringDeserializer.class.getName()); + +// SASL configuration (default SCRAM-SHA-256 mechanism) +props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT"); +props.put(SaslConfigs.SASL_MECHANISM, "SCRAM-SHA-256"); +props.put(SaslConfigs.SASL_JAAS_CONFIG, + "org.apache.kafka.common.security.scram.ScramLoginModule required " + + "username=\"alice\" " + + "password=\"secret-password\";"); + +KafkaConsumer consumer = new KafkaConsumer<>(props); +``` + +For `PLAIN`, set `security.protocol=SASL_SSL`, `sasl.mechanism=PLAIN`, and use the `PlainLoginModule` (PLAIN requires TLS). + +### Command Line Tools + +```bash +# kafka-console-producer +kafka-console-producer \ + --bootstrap-server localhost:9092 \ + --topic test \ + --producer-property security.protocol=SASL_PLAINTEXT \ + --producer-property sasl.mechanism=SCRAM-SHA-256 \ + --producer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="alice" password="secret-password";' +``` + +```bash +# kafka-console-consumer +kafka-console-consumer \ + --bootstrap-server localhost:9092 \ + --topic test \ + --from-beginning \ + --consumer-property security.protocol=SASL_PLAINTEXT \ + --consumer-property sasl.mechanism=SCRAM-SHA-256 \ + --consumer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="alice" password="secret-password";' +``` + +## Topic Authorization (ACLs) + +SASL authentication alone only prevents unauthenticated access — once connected, every +authenticated principal can produce to and consume from any topic. Topic ACLs add per-user +authorization on top of authentication. + +### Enabling ACL Enforcement + +```yaml +korvet: + broker: + sasl: + enabled: true + acl: + enabled: true +``` + +`korvet.broker.acl.enabled` requires `korvet.broker.sasl.enabled=true`; enabling ACLs without +SASL fails validation at startup. + +### Authorization Model + +ACL rules are allow-only grants of the form `(principal, topic, operation)`: + +- **principal** - the SASL username +- **topic** - a topic name, or `*` to grant the operation on all topics +- **operation** - `READ` (fetch) or `WRITE` (produce) + +When ACL enforcement is enabled, a principal may only perform operations it has been granted; +everything else — including topics with no rules at all — is denied with the standard Kafka +`TOPIC_AUTHORIZATION_FAILED` error, which Kafka clients surface as `TopicAuthorizationException`. + +Rules are stored in Redis as one JSON document per principal at +`{namespace}:broker:acls:{username}`. The principal's policy is resolved once per connection at +authentication time, so rule changes apply to connections established afterwards. + +### Managing ACL Rules + +ACL rules are managed through the admin REST API: + +```bash +# Grant alice WRITE on orders +curl -u admin:admin-password -X POST http://localhost:8080/api/v1/acls \ + -H 'Content-Type: application/json' \ + -d '{"principal":"alice","topic":"orders","operation":"WRITE"}' + +# Grant bob READ on all topics +curl -u admin:admin-password -X POST http://localhost:8080/api/v1/acls \ + -H 'Content-Type: application/json' \ + -d '{"principal":"bob","topic":"*","operation":"READ"}' + +# List alice's rules +curl -u admin:admin-password http://localhost:8080/api/v1/acls/alice + +# Delete a rule +curl -u admin:admin-password -X DELETE \ + 'http://localhost:8080/api/v1/acls/alice?topic=orders&operation=WRITE' +``` + +## Multi-Tenancy + +Each credential is associated with a tenant ID. When a client authenticates, the tenant ID is attached to the connection and can be used for: + +- **Data isolation** - Separate topics and consumer groups per tenant +- **Resource quotas** - Limit resources per tenant +- **Access control** - Restrict access to tenant-specific resources + +### Tenant Mapping + +```java +// User "alice" belongs to "tenant1" +StoredCredential credential = StoredCredential.builder() + .username("alice") + .tenantId("tenant1") + // ... other fields + .build(); + +// User "bob" belongs to "tenant2" +StoredCredential credential2 = StoredCredential.builder() + .username("bob") + .tenantId("tenant2") + // ... other fields + .build(); +``` + +## Security Best Practices + +### Password Security + +- Use strong, randomly generated passwords +- Rotate passwords regularly +- Never commit passwords to version control +- Use environment variables or secret management systems + +### Network Security + +SASL/PLAIN transmits credentials in base64 encoding (not encrypted). Korvet therefore requires TLS whenever `PLAIN` is advertised, so `PLAIN` clients must use the `SASL_SSL` security protocol. SCRAM-SHA-256 does not send the password and can be used over `SASL_PLAINTEXT`, though TLS is still recommended in production: + +- **Use TLS** - Required for `PLAIN` (`SASL_SSL`); recommended for SCRAM +- **Network isolation** - Deploy in private networks +- **Firewall rules** - Restrict access to Korvet port + +### Credential Management + +- **Principle of least privilege** - Create separate credentials per application +- **Audit access** - Monitor authentication attempts +- **Revoke unused credentials** - Delete credentials for decommissioned applications + +## Troubleshooting + +### Authentication Failures + +#### Invalid Credentials + +```log +ERROR Authentication failed for user 'alice': Invalid password +``` + +**Solution**: Verify the username and password are correct. + +#### User Not Found + +```log +ERROR Authentication failed for user 'bob': User not found +``` + +**Solution**: Create the credential using the credential store. + +#### Mechanism Not Supported + +```log +ERROR Unsupported SASL mechanism: SCRAM-SHA-512 +``` + +**Solution**: Use a supported mechanism (`PLAIN` or `SCRAM-SHA-256`). + +### Connection Issues + +#### Client Configuration + +Verify the client is configured with: + +- `security.protocol=SASL_PLAINTEXT` for SCRAM-SHA-256, or `SASL_SSL` for PLAIN (PLAIN requires TLS) +- `sasl.mechanism=SCRAM-SHA-256` (default) or `sasl.mechanism=PLAIN` +- Correct JAAS configuration with username and password + +#### Server Configuration + +Verify SASL is enabled in Korvet: + +```bash +# Check environment variable +echo $KORVET_BROKER_SASL_ENABLED + +# Should output: true +``` + +### Debugging + +Enable debug logging for authentication: + +```yaml +logging: + level: + com.redis.korvet.broker.auth: DEBUG + com.redis.korvet.broker.kafka.SaslHandshakeHandler: DEBUG + com.redis.korvet.broker.kafka.SaslAuthenticateHandler: DEBUG +``` + +## Migration Guide + +### Enabling Authentication on Existing Deployment + +{{< warning >}} +Enabling authentication will break existing unauthenticated clients. +{{< /warning >}} + +1. Create credentials for all existing applications +2. Update client configurations with SASL settings +3. Test authentication with a subset of clients +4. Enable SASL in Korvet configuration +5. Monitor for authentication failures +6. Update remaining clients + +### Disabling Authentication + +To disable authentication: + +```yaml +korvet: + broker: + sasl: + enabled: false +``` + +Clients can then connect without authentication. + +## See Also + +- [Deployment Guide]({{< relref "/integrate/korvet/operations/deployment" >}}) +- [Configuration Reference]({{< relref "/integrate/korvet/reference/configuration" >}}) +- [Troubleshooting]({{< relref "/integrate/korvet/operations/troubleshooting" >}}) diff --git a/content/integrate/korvet/operations/benchmarks.md b/content/integrate/korvet/operations/benchmarks.md new file mode 100644 index 0000000000..01b17ce773 --- /dev/null +++ b/content/integrate/korvet/operations/benchmarks.md @@ -0,0 +1,453 @@ +--- +Title: Benchmarks +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Performance benchmarks for Korvet with Redis Enterprise as the storage + backend. +linkTitle: Benchmarks +weight: 110 +--- + +This page documents performance benchmarks for Korvet with Redis Enterprise as the storage backend. + +## Optimal Configuration Benchmark + +This benchmark demonstrates the best throughput configuration for Korvet with Redis Enterprise. + +### Test Environment + +- **Redis Enterprise**: 16 shards, running locally +- **Korvet**: Single instance (macOS, Apple Silicon) +- **Kafka Tools**: `kafka-producer-perf-test` from Apache Kafka +- **Topic Configuration**: 16 partitions (1× the number of shards) +- **Record Size**: 1 KB (1024 bytes) +- **Total Messages**: 8,000,000 (1,000,000 per producer) + +### Configuration + +| Parameter | Value | +|---|---| +| Producers | 8 | +| Batch Size | 1000 messages (1.07 MB) | +| Redis Connection Pool Size | 8 | +| Acks | 1 | +| Compression | none | +| Linger | 0ms | + +### Performance Results + +| Metric | Value | +|---|---| +| Aggregate Throughput | 380,952 records/sec | +| Throughput (MB/sec) | 372.02 MB/sec | +| Total Messages | 8,000,000 | +| Duration | 21 seconds | +| Average Latency (range) | 249-411 ms | +| 95th Percentile Latency (range) | 657-2136 ms | + +### Korvet Resource Usage + +| Metric | Value | +|---|---| +| Process CPU | 2.35% | +| System CPU | 20.51% | +| JVM Memory Used | 244.13 MB | + +### Redis Enterprise Metrics + +| Metric | Value | +|---|---| +| Total CPU (all 16 shards) | 79% | +| Per-Shard CPU | 2-8% | +| Data per Shard | 112-172 MB | +| Data Distribution | Even across all shards | + +### Key Findings + +- **High throughput with low CPU usage**: Achieved 372 MB/sec with only 2.35% Korvet CPU usage +- **Excellent scalability headroom**: Both Korvet and Redis Enterprise operating well below capacity +- **Even load distribution**: Data and CPU load distributed evenly across all 16 Redis shards +- **Optimal batch size**: 1000 messages per batch provided the best balance of throughput and latency + +### Running This Benchmark + +To reproduce this benchmark, use the provided benchmark script from the [korvet-dist](https://github.com/redis-field-engineering/korvet-dist) repository: + +```bash +git clone https://github.com/redis-field-engineering/korvet-dist.git +cd korvet-dist/samples/benchmark/scripts +./run-comprehensive-benchmark.sh +``` + +The script will: + +1. Start Korvet with the specified Redis pool size +2. Create a topic with 16 partitions +3. Run 8 concurrent producers, each sending 1,000,000 messages +4. Collect metrics from Korvet (via actuator) and Redis Enterprise (via API) +5. Generate a detailed report with throughput, latency, and resource usage + +Results are saved to `/tmp/korvet-benchmark-/`. + +## Single Shard Benchmark + +This benchmark demonstrates Korvet performance with a single Redis shard, providing a baseline for comparison with multi-shard configurations. + +### Test Environment + +- **Redis Enterprise**: 1 shard (~1 GB maxmemory), running locally +- **Korvet**: Single instance (macOS, Apple Silicon) +- **Kafka Tools**: `kafka-producer-perf-test` from Apache Kafka +- **Topic Configuration**: 1 partition (matching the single shard) +- **Record Size**: 1 KB (1024 bytes) + +### Configuration + +| Parameter | Value | +|---|---| +| Producers | 1 (baseline) / 8 (concurrent) | +| Batch Size | 1000 messages (1.07 MB) | +| Redis Connection Pool Size | 16 | +| Acks | 1 | +| Compression | none | +| Linger | 0ms | + +### Performance Results + +#### Single Producer (Baseline) + +| Metric | Value | +|---|---| +| Throughput | 151,860 records/sec | +| Throughput (MB/sec) | 148.30 MB/sec | +| Total Messages | 200,000 | +| Average Latency | 160 ms | +| P99 Latency | 329 ms | + +#### 8 Concurrent Producers + +| Metric | Value | +|---|---| +| Aggregate Throughput | 168,641 records/sec | +| Throughput (MB/sec) | 164.68 MB/sec | +| Total Messages | 674,564 | +| Duration | 4.58 seconds | +| Average Latency (range) | 495-724 ms | +| Memory Used | 776.47 MB | + +### Comparison: 16 Shards vs 1 Shard + +| Metric | 16 Shards | 1 Shard | Ratio | +|---|---|---|---| +| Database Memory | ~16 GB | ~1 GB | 16× | +| Topic Partitions | 16 | 1 | 16× | +| Throughput (rec/s) | 380,952 | 168,641 | 2.26× | +| Throughput (MB/s) | 372.02 | 164.68 | 2.26× | +| Per-shard throughput | 23,809 | 168,641 | 0.14× | + +### Key Findings + +- **Single shard achieves ~44% of 16-shard aggregate throughput**: 168,641 vs 380,952 records/sec +- **Higher per-shard efficiency with fewer shards**: A single shard processes 168,641 rec/s vs 23,809 rec/s per shard in the 16-shard setup +- **Memory efficiency**: ~1.15 KB per message in Redis Streams (776 MB for 674,564 messages) +- **Single producer baseline**: 151,860 rec/s provides a clean baseline without concurrency overhead + +## Remote Storage Archival Benchmark + +This benchmark measures the throughput of archiving sealed Redis stream segments to Apache Iceberg tables on S3. + +### Test Environment + +- **EC2 Instance**: c5.2xlarge (8 vCPU, 16GB RAM) in us-west-1 +- **S3 Bucket**: Same region (us-west-1) for optimal network performance +- **Redis**: Docker container on same instance +- **Message Size**: ~100 bytes (binary payload) +- **Compression**: Iceberg default Parquet compression + +### Single Stream Results + +Archiving from a single Redis Stream to S3: + +| Messages | Archive Time | Throughput | Parquet Files | +|---|---|---|---| +| 1,000,000 | 31.3s | 31,970 msg/s | 100 @ 186ms avg | + +### Multi-Stream Results (4 Partitions) + +Archiving from 4 Redis Streams in parallel to S3: + +| Messages | Archive Time | Throughput | Parquet Files | +|---|---|---|---| +| 1,000,000 | 12.5s | 80,239 msg/s | 100 @ 212ms avg | +| 4,000,000 | 34.7s | 115,347 msg/s | 400 @ 192ms avg | + +### Scaling Summary + +| Configuration | Throughput | vs Single Stream | +|---|---|---| +| 1 stream | 32k msg/s | baseline | +| 4 streams (1M messages) | 80k msg/s | 2.5× | +| 4 streams (4M messages) | 115k msg/s | 3.6× | + +### Key Findings + +- **Single stream peaks at ~32k msg/s**: Bottleneck is S3 PUT latency for Parquet files +- **Near-linear scaling with streams**: 4 streams achieves 115k msg/s (3.6× single stream) +- **Parquet writes average ~190ms**: Same-region S3 provides consistent low latency +- **Excellent compression**: SNAPPY on this payload achieves ~50:1 compression ratio (~2 bytes/message stored) +- **Same-region S3 is critical**: Cross-region throughput drops ~50% + +### Storage Efficiency + +| Metric | Value | +|---|---| +| Messages archived | 4,000,000 | +| S3 objects created | 400 (one Parquet per sealed segment) | +| Total S3 storage | ~8 MB | +| Bytes per message | ~2 bytes (after SNAPPY compression) | +| Compression ratio | ~50:1 | + +### Archival Configuration + +The storage worker was configured with: + +```yaml +korvet: + storage: + remote: + path: s3://your-bucket/korvet + s3: + region: us-west-1 + worker: + enabled: true +``` + +## Redis Flex (Auto-Tiering) Benchmark + +This benchmark evaluates Korvet performance with Redis Flex (Auto-Tiering), which uses NVMe flash storage to extend Redis capacity beyond RAM. + +### Test Environment + +- **Redis Enterprise**: 1× i4i.xlarge (4 vCPU, 32GB RAM, 937GB NVMe) +- **Database Config**: 100GB capacity, 10GB RAM (10% ratio), 8 shards +- **Korvet Client**: c7i.4xlarge (16 vCPU, 32GB RAM) +- **Kafka Tools**: `kafka-producer-perf-test` from Apache Kafka +- **Record Size**: 1 KB (1024 bytes) +- **Region**: us-west-2 (all instances in same VPC) + +### Test Configuration + +| Parameter | Value | +|---|---| +| Instance Type (Redis) | i4i.xlarge (NVMe-backed) | +| Instance Type (Client) | c7i.4xlarge | +| Shards | 8 (1.25GB RAM per shard) | +| Redis Pool Size | 256 | +| Producer Batch Size | 128KB (`batch.size=131072`) | +| Linger | 5ms (`linger.ms=5`) | +| Acks | 1 | + +### Performance Results + +| Metric | Korvet → Redis Flex | Direct Redis (XADD) | +|---|---|---| +| Peak Throughput | 150,784 rec/s (147 MB/s) | 130,690 rec/s (128 MB/s) | +| Sustained Throughput | 110,000 rec/s (107 MB/s) | 103,000 rec/s (100 MB/s) | +| Average Latency | 201 ms | < 1 ms | +| P99 Latency | 510 ms | 28 ms | + +### Data Structure Comparison + +We compared Redis Streams (XADD) vs simple key-value (SET) operations on Redis Flex: + +| Operation | Throughput | Notes | +|---|---|---| +| SET (1KB values) | 153,000 ops/sec | Simple key-value, flash-friendly | +| XADD (Streams, 1KB payload) | 103,000 ops/sec | Stream data structure overhead | +| Korvet → XADD | 110-150k rec/sec | Near-native XADD performance | + +### Key Findings + +- **Korvet matches native Redis Streams performance**: Korvet achieved 110-150k rec/sec, matching or exceeding direct XADD benchmarks +- **Flash eviction is the bottleneck for sustained writes**: RAM fills faster than NVMe can drain at very high throughput +- **Larger RAM buffers help**: 8 shards (1.25GB RAM/shard) outperformed 48 shards (208MB RAM/shard) by avoiding OOM errors +- **Client instance sizing matters**: Upgraded from t3.medium (2 vCPU) to c7i.4xlarge (16 vCPU) to eliminate client-side bottleneck + +### OOM Behavior + +At sustained throughput above ~150k rec/sec with 1KB payloads, Redis Flex may return OOM errors when the RAM buffer fills faster than flash eviction can drain. This is inherent to Redis Streams on flash storage, not specific to Korvet. + +| Scenario | Throughput | Result | +|---|---|---| +| Burst (1M records) | 150k rec/s | ✅ Success | +| Sustained (2M+ records) | 150k rec/s | ⚠️ OOM after ~1.3M records | +| Sustained (unlimited) | 110k rec/s | ✅ Success | + +**Mitigation**: For sustained high-throughput workloads on Redis Flex: + +- Use fewer shards with larger RAM buffers (e.g., 8 shards vs 48) +- Increase RAM-to-disk ratio (e.g., 15-20% instead of 10%) +- Throttle producer throughput to ~100k rec/sec per instance +- Use multiple Redis Flex clusters for horizontal scaling + +### Sizing Recommendations for Redis Flex + +| Workload | Shards | RAM per Shard | +|---|---|---| +| Light (< 50k rec/s) | 4 | 2.5GB | +| Medium (50-100k rec/s) | 8 | 1.25GB+ | +| Heavy (100k+ rec/s) | 8-16 | 1GB+ (with throttling) | + +## Configuration Recommendations + +For optimal throughput: + +- **Batch size**: Use 1000 messages per batch for best balance of throughput and latency +- **Producers**: 8 concurrent producers provides excellent throughput with manageable latency +- **Redis pool size**: Match pool size to number of producers (8) for optimal connection utilization +- **Partitions**: Use 1-2× the number of Redis shards (16 partitions for 16 shards) +- **Redis shards**: Match the number of shards to available CPU cores +- **Rebalance delay**: Configure `korvet.broker.rebalance-delay` appropriately (default 3s) to allow all consumers to join before rebalancing +- **Replication**: Disable replication for write-heavy workloads (if durability requirements allow) + +## Running Your Own Benchmarks + +### Using the Benchmark Script + +The [korvet-dist](https://github.com/redis-field-engineering/korvet-dist) repository contains a script to run benchmarks with various configurations. + +```bash +git clone https://github.com/redis-field-engineering/korvet-dist.git +cd korvet-dist/samples/benchmark/scripts +./run-comprehensive-benchmark.sh +``` + +#### Configuration Options + +Edit the script to customize benchmark parameters: + +```bash +# Test parameters +TOPIC="benchmark-test" +PARTITIONS=16 +RECORD_SIZE=1024 +NUM_RECORDS=1000000 + +# Parameter arrays +PRODUCERS=(8) # Number of concurrent producers +BATCH_SIZES=(1000) # Messages per batch +POOL_SIZES=(8) # Redis connection pool size +``` + +#### What the Script Does + +1. **Starts Korvet** with the specified Redis pool size +2. **Flushes Redis** to ensure clean state +3. **Creates topic** with specified number of partitions +4. **Runs producers** using `kafka-producer-perf-test` +5. **Collects metrics**: + - Korvet CPU and memory (via Spring Boot Actuator at port 8080) + - Redis Enterprise CPU and memory (via REST API at port 9443) + - Producer throughput and latency +6. **Generates report** with detailed results + +#### Output + +Results are saved to `/tmp/korvet-benchmark-/`: + +- `SUMMARY.txt`: Summary table of all test results +- `producers-_batch-msg_pool-

.txt`: Detailed results for each test + +Example summary output: + +``` +Producers Batch(msg) Pool Total Msgs Duration(s) Throughput(rec/s) Throughput(MB/s) +8 1000 8 8000000 21 380952 372.02 +``` + +### Running Storage Tier Benchmarks + +Korvet ships JUnit-based micro-benchmarks alongside its integration tests. These +run under the Gradle `integrationTest` task (not `test`) and use Testcontainers, +so they require a running Docker engine. + +#### Tiered read coordinator benchmark + +`RedisTieredGroupReadCoordinatorBenchmark` in the `korvet-storage-tiered-redis` +module measures group-read coordination throughput across a range of batch sizes +against a Redis container. It is gated behind the `korvet.benchmark.pel` system +property so it is skipped during normal test runs: + +```bash +./gradlew :korvet-storage-tiered-redis:integrationTest \ + --tests "RedisTieredGroupReadCoordinatorBenchmark" \ + -Dkorvet.benchmark.pel=true +``` + +{{< note >}} +The remote/S3 archival and end-to-end remote-read numbers reported above +were produced with purpose-built harnesses on EC2. The reusable benchmark +scaffolding for capturing results — `BenchmarkConfig`, `ScenarioResult`, and +`BenchmarkResult` — lives in `korvet-server/src/integrationTest` under +`com.redis.korvet.benchmark`. A packaged, repeatable S3/remote-read benchmark +entry point is planned; the throughput figures in this page should be treated as +illustrative of what the storage tier can achieve rather than as a turnkey test +you can run as-is. +{{< /note >}} + +#### Result scaffolding + +The `BenchmarkResult` type captures version, Git commit, and environment metadata +alongside per-scenario `ScenarioResult` entries, which makes results +comparable across releases. A captured result is shaped like this: + +```json +{ + "korvetVersion": "0.5.0-ea1", + "timestamp": "2026-03-30T17:30:00Z", + "gitCommit": "abc1234", + "environment": { + "javaVersion": "25", + "osName": "Linux", + "availableProcessors": 4, + "maxMemoryMb": 4096 + }, + "config": { + "messageCount": 10000, + "recordSizeBytes": 1024, + "partitions": 1, + "iterations": 3, + "batchSizes": [100, 500, 1000] + }, + "scenarios": [ + { + "type": "STANDALONE_CONSUMER", + "batchSize": 100, + "avgLatencyMs": 250, + "p95LatencyMs": 320, + "throughputMsgPerSec": 400.0 + } + ] +} +``` + +#### Interpreting Results + +- **Standalone vs Consumer Group overhead**: Consumer group reads include additional coordination (JoinGroup, SyncGroup, OffsetFetch) which adds latency +- **ListOffsets Earliest latency**: High values indicate slow remote-tier metadata lookups (Parquet footer reads against the manifest's oldest REMOTE segment) +- **Throughput scaling**: If throughput doesn't scale linearly with batch size, there may be per-request overhead dominating +- **P95 vs Avg**: Large gaps indicate object-store tail latencies or GC pauses + +#### Timeout Recommendations + +Based on production experience with S3-backed remote storage, configure consumer timeouts appropriately: + +```java +// For remote storage reads, increase timeouts +props.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, "60000"); // 60s +props.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, "120000"); // 2min +props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, "30000"); // 30s +``` diff --git a/content/integrate/korvet/operations/deployment.md b/content/integrate/korvet/operations/deployment.md new file mode 100644 index 0000000000..4464aa9148 --- /dev/null +++ b/content/integrate/korvet/operations/deployment.md @@ -0,0 +1,582 @@ +--- +Title: Deployment +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Deploy Korvet in production environments. +linkTitle: Deployment +weight: 10 +--- + +This guide covers deploying Korvet in production environments. + +## Deployment shapes + +A Korvet JVM runs the **broker** (Kafka listener and group coordinator) and the +leader-locked **storage worker** by default. The storage worker rolls +eligible segments, offloads sealed segments, and enforces local and remote +retention. + +Disable a component explicitly only when a deployment needs a single-role JVM. + +| Shape | Configuration | Components started | +|---|---|---| +| All-in-one (default) | no component flags required | broker + storage worker | +| Broker-only pod | `KORVET_STORAGE_WORKER_ENABLED=false` | broker | +| Storage-worker pod | `KORVET_BROKER_ENABLED=false` | storage worker | +| Single-role pod | set one of `KORVET_BROKER_ENABLED` / `KORVET_STORAGE_WORKER_ENABLED` to `false` | the selected role | +| Custom | set `KORVET_BROKER_ENABLED` and/or `KORVET_STORAGE_WORKER_ENABLED` | the components whose flag is `true` | + +`korvet.storage.remote.path` makes the cold tier available. Without it, Korvet +runs local-only on Redis Streams. + +## Docker Deployment + +### Single Instance + +```bash +docker run -d \ + --name korvet \ + -p 9092:9092 \ + -e JAVA_OPTS="-Xms2g -Xmx2g -XX:MaxDirectMemorySize=512m" \ + -e KORVET_REDIS_URI=redis://redis.example.com:6379 \ + -e KORVET_REDIS_USERNAME=default \ + -e KORVET_REDIS_PASSWORD=${REDIS_PASSWORD} \ + redisfield/korvet:latest server +``` + +Use `JAVA_OPTS` to pass heap settings and additional JVM flags to the container at startup. + +{{< warning >}} +Tune JVM memory through `JAVA_OPTS`, **not** `JAVA_TOOL_OPTIONS`. The broker bakes +`-XX:MaxDirectMemorySize=512m` into its default launch arguments, and those defaults are placed +**after** `JAVA_TOOL_OPTIONS` on the command line — so a `MaxDirectMemorySize` set via +`JAVA_TOOL_OPTIONS` is silently overridden. Only `JAVA_OPTS` (applied last) overrides the baked +default. + +Direct buffer memory lives outside the heap, so size the container so that +`memory limit >= -Xmx + MaxDirectMemorySize + ~512m` native/metaspace/stack overhead. With +`-Xmx2g` and `MaxDirectMemorySize=512m`, use a container memory limit of at least `4Gi`. Increase +`MaxDirectMemorySize` for high fan-in workloads (many concurrent consumers, e.g. Spark/Flink) and +raise the container limit to match. +{{< /warning >}} + +### Docker Compose + +```yaml +services: + redis: + image: redis:8.6 + command: redis-server --requirepass ${REDIS_PASSWORD} + ports: + - "6379:6379" + + korvet: + image: redisfield/korvet:latest + command: server + ports: + - "9092:9092" + environment: + JAVA_OPTS: "-Xms2g -Xmx2g -XX:MaxDirectMemorySize=512m" + KORVET_REDIS_URI: redis://redis:6379 + KORVET_REDIS_PASSWORD: ${REDIS_PASSWORD} + depends_on: + - redis +``` + +## Kubernetes Deployment + +For single-broker or simple deployments, use a standard Deployment. For multi-broker clusters with proper broker discovery, see [Multi-Broker Deployment](#multi-broker-deployment). + +### Simple Deployment (Single Broker) + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: korvet +spec: + replicas: 1 # Single broker + selector: + matchLabels: + app: korvet + template: + metadata: + labels: + app: korvet + spec: + containers: + - name: korvet + image: redisfield/korvet:latest + args: ["server"] + ports: + - containerPort: 9092 + env: + - name: KORVET_REDIS_URI + value: redis://redis-service:6379 + - name: KORVET_BROKER_HOST + value: 0.0.0.0 + - name: KORVET_BROKER_PORT + value: "9092" + resources: + requests: + memory: "512Mi" + cpu: "500m" + limits: + memory: "2Gi" + cpu: "2000m" + livenessProbe: + httpGet: + path: /actuator/health/liveness + port: 8080 + initialDelaySeconds: 30 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /actuator/health/readiness + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 5 +``` + +### Service Manifest + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: korvet-service +spec: + selector: + app: korvet + ports: + - protocol: TCP + port: 9092 + targetPort: 9092 + type: LoadBalancer +``` + +{{< note >}} +For multi-broker deployments with high availability, use a StatefulSet instead. See [Kubernetes StatefulSet](#kubernetes-statefulset). +{{< /note >}} + +## Multi-Broker Deployment + +Korvet supports multi-broker deployments where multiple Korvet instances share the same Redis backend. This provides high availability and load distribution while maintaining full data consistency. + +### Architecture Overview + +In a multi-broker deployment: + +- All brokers connect to the same Redis instance or cluster +- Brokers automatically discover each other via the **Broker Registry** stored in Redis +- Kafka clients can connect to any broker and receive metadata about all brokers in the cluster +- Messages produced to one broker are immediately available from any other broker + +``` + ┌─────────────────┐ + │ Load Balancer │ + │ (TCP/9092) │ + └────────┬────────┘ + │ + ┌────────────────────┼────────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌───────────────┐ ┌───────────────┐ +│ Korvet 0 │ │ Korvet 1 │ │ Korvet 2 │ +│ id=0 │ │ id=1 │ │ id=2 │ +│ port=9092 │ │ port=9092 │ │ port=9092 │ +└───────┬───────┘ └───────┬───────┘ └───────┬───────┘ + │ │ │ + └───────────────────┼───────────────────┘ + │ + ▼ + ┌───────────────────────┐ + │ Redis │ + │ (Streams + Registry) │ + └───────────────────────┘ +``` + +### Broker Configuration + +Each broker requires a unique `id` and proper network configuration: + +```yaml +korvet: + namespace: korvet # Must be the same across all brokers + + broker: + id: 0 # Unique ID for this broker (0, 1, 2, etc.) + host: 0.0.0.0 # Listen on all interfaces + port: 9092 # Kafka protocol port + advertised-host: korvet-0.korvet.default.svc.cluster.local # Hostname clients use + advertised-port: 9092 # Port clients use + + redis: + uri: redis://redis:6379 # Same Redis for all brokers +``` + +{{< warning >}} +All brokers in a cluster **must** use the same `korvet.namespace` and connect to the **same Redis** instance. +{{< /warning >}} + +### Broker Discovery + +Korvet uses a **Broker Registry** stored in Redis for automatic broker discovery: + +- Each broker registers itself on startup with its ID, host, and port +- Brokers send periodic heartbeats (every 10 seconds by default) +- Entries are considered stale 30 seconds after their last heartbeat and are then ignored +- Kafka clients receive all live registered brokers in metadata responses + +Redis keys used by the registry: + +``` +korvet:broker:nodes # Single hash: field = broker id, value = host/port/rack + heartbeat timestamp +``` + +{{< note >}} +There is no Redis TTL on the key. Staleness is determined client-side by comparing each +entry's heartbeat timestamp against the 30-second threshold. +{{< /note >}} + +### Docker Compose Example + +```yaml +services: + redis: + image: redis:8.6 + ports: + - "6379:6379" + + korvet-0: + image: redisfield/korvet:latest + command: server + ports: + - "9092:9092" + environment: + KORVET_BROKER_ID: 0 + KORVET_BROKER_HOST: 0.0.0.0 + KORVET_BROKER_ADVERTISED_HOST: localhost + KORVET_BROKER_ADVERTISED_PORT: 9092 + KORVET_REDIS_URI: redis://redis:6379 + + korvet-1: + image: redisfield/korvet:latest + command: server + ports: + - "9093:9092" + environment: + KORVET_BROKER_ID: 1 + KORVET_BROKER_HOST: 0.0.0.0 + KORVET_BROKER_ADVERTISED_HOST: localhost + KORVET_BROKER_ADVERTISED_PORT: 9093 + KORVET_REDIS_URI: redis://redis:6379 + + korvet-2: + image: redisfield/korvet:latest + command: server + ports: + - "9094:9092" + environment: + KORVET_BROKER_ID: 2 + KORVET_BROKER_HOST: 0.0.0.0 + KORVET_BROKER_ADVERTISED_HOST: localhost + KORVET_BROKER_ADVERTISED_PORT: 9094 + KORVET_REDIS_URI: redis://redis:6379 +``` + +Clients can connect using multiple bootstrap servers: + +```bash +kafka-console-producer --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --topic test +``` + +### Kubernetes StatefulSet + +For Kubernetes, use a StatefulSet to ensure each broker gets a unique, stable identity. The StatefulSet provides: + +- **Stable network identity**: Each pod gets a predictable DNS name (`korvet-0`, `korvet-1`, etc.) +- **Ordered deployment**: Pods are created sequentially, ensuring broker registration order +- **Stable storage**: PersistentVolumeClaims are retained across pod restarts (if needed) + +#### Complete Kubernetes Manifests + +```yaml +# ConfigMap for shared configuration +apiVersion: v1 +kind: ConfigMap +metadata: + name: korvet-common +data: + KORVET_BROKER_HOST: "0.0.0.0" + KORVET_BROKER_PORT: "9092" + KORVET_NAMESPACE: "korvet" + KORVET_BROKER_REBALANCE_DELAY: "5s" # Allow time for consumers to join in K8s +--- +# Headless service for StatefulSet DNS +apiVersion: v1 +kind: Service +metadata: + name: korvet + labels: + app: korvet +spec: + clusterIP: None + selector: + app: korvet + ports: + - port: 9092 + name: kafka + - port: 8080 + name: actuator +--- +# LoadBalancer service for external access +apiVersion: v1 +kind: Service +metadata: + name: korvet-lb +spec: + type: LoadBalancer + selector: + app: korvet + ports: + - port: 9092 + targetPort: 9092 + name: kafka +--- +# StatefulSet +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: korvet +spec: + serviceName: korvet + replicas: 3 + podManagementPolicy: Parallel # Start all pods simultaneously + selector: + matchLabels: + app: korvet + template: + metadata: + labels: + app: korvet + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "8080" + prometheus.io/path: "/actuator/prometheus" + spec: + terminationGracePeriodSeconds: 30 + initContainers: + # Extract broker ID from pod name (korvet-0 -> 0, korvet-1 -> 1, etc.) + - name: init-broker-id + image: busybox:1.36 + command: + - sh + - -c + - | + ORDINAL=${HOSTNAME##*-} + echo "KORVET_BROKER_ID=${ORDINAL}" > /config/broker.env + echo "KORVET_BROKER_ADVERTISED_HOST=${HOSTNAME}.korvet.${NAMESPACE}.svc.cluster.local" >> /config/broker.env + echo "Broker ID: ${ORDINAL}, Advertised Host: ${HOSTNAME}.korvet.${NAMESPACE}.svc.cluster.local" + env: + - name: NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + volumeMounts: + - name: config-volume + mountPath: /config + containers: + - name: korvet + image: redisfield/korvet:latest + command: + - sh + - -c + - | + # Source the broker-specific config + export $(cat /config/broker.env | xargs) + # Start the application + exec java --sun-misc-unsafe-memory-access=allow -jar /app/korvet.jar + ports: + - containerPort: 9092 + name: kafka + - containerPort: 8080 + name: actuator + envFrom: + - configMapRef: + name: korvet-common + - secretRef: + name: korvet-redis-credentials + optional: true + env: + - name: KORVET_REDIS_URI + value: redis://redis:6379 + - name: KORVET_BROKER_ADVERTISED_PORT + value: "9092" + # JVM tuning for containers. This manifest launches `java -jar` directly (no distribution + # start script), so there are no baked applicationDefaultJvmArgs to override and JAVA_OPTS + # would not be expanded by the command above — JAVA_TOOL_OPTIONS is auto-applied by the JVM + # and is the right place here. Keep -Xmx + MaxDirectMemorySize below the container limit. + - name: JAVA_TOOL_OPTIONS + value: "-Xms2g -Xmx2g -XX:+UseG1GC -XX:MaxDirectMemorySize=512m --sun-misc-unsafe-memory-access=allow" + resources: + requests: + memory: "2Gi" + cpu: "500m" + limits: + memory: "4Gi" + cpu: "2000m" + volumeMounts: + - name: config-volume + mountPath: /config + livenessProbe: + httpGet: + path: /actuator/health/liveness + port: 8080 + initialDelaySeconds: 30 + periodSeconds: 10 + failureThreshold: 3 + readinessProbe: + httpGet: + path: /actuator/health/readiness + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 5 + failureThreshold: 3 + startupProbe: + httpGet: + path: /actuator/health/liveness + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 5 + failureThreshold: 30 + volumes: + - name: config-volume + emptyDir: {} +``` + +#### Redis Credentials Secret + +If Redis requires authentication, create a secret: + +```bash +kubectl create secret generic korvet-redis-credentials \ + --from-literal=KORVET_REDIS_PASSWORD=your-password +``` + +#### Scaling the Cluster + +Scale up or down with: + +```bash +# Scale to 5 brokers +kubectl scale statefulset korvet --replicas=5 + +# Scale down to 3 brokers +kubectl scale statefulset korvet --replicas=3 +``` + +When scaling down, brokers are removed in reverse order (highest ID first). The broker registry stops advertising an entry once its last heartbeat is older than 30 seconds. + +#### Pod Disruption Budget + +For high availability, configure a PodDisruptionBudget: + +```yaml +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: korvet-pdb +spec: + minAvailable: 2 + selector: + matchLabels: + app: korvet +``` + +#### External Access + +For clients outside the Kubernetes cluster, you have several options: + +**Option 1: LoadBalancer Service** (shown above) + +Clients connect to the LoadBalancer IP. All traffic is distributed across brokers. + +**Option 2: NodePort Service** + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: korvet-nodeport +spec: + type: NodePort + selector: + app: korvet + ports: + - port: 9092 + targetPort: 9092 + nodePort: 30092 +``` + +**Option 3: Ingress with TCP support** (e.g., NGINX Ingress Controller) + +Configure TCP services in the ingress controller's ConfigMap. + +#### Monitoring in Kubernetes + +Korvet exposes Prometheus metrics at `/actuator/prometheus`. With the annotations in the StatefulSet, Prometheus will automatically scrape metrics. + +For Grafana dashboards, query metrics like: + +- `korvet_broker_produce_seconds_count` - Total produce requests +- `korvet_broker_fetch_seconds_count` - Total fetch requests +- `korvet_broker_request_seconds_count` - Kafka API request rate, by `api_key` and `result` +- `korvet_broker_backpressure_connections` - Connections currently under backpressure + +### Consumer Group Coordination + +In multi-broker deployments, consumer group coordination is handled specially: + +- The **group coordinator** is selected using consistent hashing based on the group ID +- Clients are directed to the correct coordinator via the `FindCoordinator` response +- All consumer group state is stored in Redis, so any broker can serve offset commits/fetches + +### Best Practices + +- **Broker IDs**: Use sequential IDs starting from 0 (0, 1, 2, ...). Each broker must have a unique ID. +- **Advertised Listeners**: Always configure `advertised-host` and `advertised-port` to the address clients should use to connect. This is especially important in Docker/Kubernetes where internal and external addresses differ. +- **Load Balancing**: Use a TCP load balancer (not HTTP) in front of your brokers. Any load balancing strategy works since all brokers serve the same data. +- **Bootstrap Servers**: Configure Kafka clients with multiple bootstrap servers for fault tolerance: + + ```properties + bootstrap.servers=korvet-0:9092,korvet-1:9092,korvet-2:9092 + ``` + +- **Redis High Availability**: For production, use a highly available Redis deployment, such as Redis Enterprise, to ensure the storage layer is also highly available. + +## High Availability + +For production deployments: + +- **Multiple instances**: Run 3+ Korvet instances for redundancy +- **Redis HA**: Use a highly available Redis deployment, such as Redis Enterprise, for HA +- **Health checks**: Configure liveness and readiness probes +- **Graceful shutdown**: Allow time for in-flight requests to complete + +## Scaling + +Korvet can be scaled horizontally: + +- **Stateless**: Each instance shares state via Redis +- **Load balancing**: Use any TCP load balancer +- **Add brokers**: Simply start new instances with unique broker IDs + +## Next Steps + +- [Monitoring]({{< relref "/integrate/korvet/operations/monitoring" >}}) +- [Logging]({{< relref "/integrate/korvet/operations/logging" >}}) +- [Configuration]({{< relref "/integrate/korvet/quick-start/configuration" >}}) diff --git a/content/integrate/korvet/operations/kubernetes.md b/content/integrate/korvet/operations/kubernetes.md new file mode 100644 index 0000000000..022c62bc30 --- /dev/null +++ b/content/integrate/korvet/operations/kubernetes.md @@ -0,0 +1,435 @@ +--- +Title: Kubernetes Deployment +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Step-by-step instructions for deploying Korvet on Kubernetes. +linkTitle: Kubernetes +weight: 30 +--- + +This guide provides step-by-step instructions for deploying Korvet on Kubernetes. + +## Prerequisites + +- Kubernetes cluster (1.24+) +- `kubectl` configured to access your cluster +- Redis instance accessible from the cluster (Redis Enterprise or standalone Redis) +- Helm 3.x (optional, for Helm-based deployment) + +## Quick Start + +### 1. Create Namespace + +```bash +kubectl create namespace korvet +``` + +### 2. Create Secret for Redis Credentials + +```bash +kubectl create secret generic korvet-redis-credentials \ + --namespace korvet \ + --from-literal=password=your-redis-password +``` + +### 3. Apply Kubernetes Manifests + +Save the manifests below to a file and apply them: + +```bash +kubectl apply -f korvet.yaml +``` + +## Kubernetes Manifests + +### ConfigMap + +Store non-sensitive configuration in a ConfigMap: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: korvet-common + namespace: korvet +data: + KORVET_BROKER_HOST: "0.0.0.0" + KORVET_BROKER_PORT: "9092" + KORVET_BROKER_ID: "0" + KORVET_NAMESPACE: "korvet" + # Pattern-based topic defaults (index 0 acts as catch-all) + KORVET_TOPICS_0_NAME: "*" + KORVET_TOPICS_0_AUTO_CREATE: "true" + KORVET_TOPICS_0_PARTITIONS: "3" + KORVET_REDIS_URI: "redis://redis-service:6379" + # Advertised host/port - the address clients use to connect back to Korvet + # Set this to the external service hostname or load balancer address + KORVET_BROKER_ADVERTISED_HOST: "korvet.example.com" + KORVET_BROKER_ADVERTISED_PORT: "9092" +``` + +{{< warning >}} +The `KORVET_BROKER_ADVERTISED_HOST` and `KORVET_BROKER_ADVERTISED_PORT` settings are critical for Kafka clients. These values are returned in Metadata responses and tell clients where to connect. Set these to the external hostname/IP that clients will use to reach Korvet (e.g., your LoadBalancer address or Ingress hostname). +{{< /warning >}} + +### Secret + +Store sensitive configuration in a Secret: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: korvet-redis-credentials + namespace: korvet +type: Opaque +stringData: + password: "your-redis-password" + # For remote storage with static S3 credentials (optional): + # aws-access-key-id: "AKIAIOSFODNN7EXAMPLE" + # aws-secret-access-key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" +``` + +### Deployment + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: korvet + namespace: korvet + labels: + app: korvet +spec: + replicas: 1 + selector: + matchLabels: + app: korvet + template: + metadata: + labels: + app: korvet + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "8080" + prometheus.io/path: "/actuator/prometheus" + spec: + containers: + - name: korvet + image: redisfield/korvet:latest + args: ["server"] + ports: + - name: kafka + containerPort: 9092 + protocol: TCP + - name: management + containerPort: 8080 + protocol: TCP + envFrom: + - configMapRef: + name: korvet-common + env: + - name: KORVET_REDIS_PASSWORD + valueFrom: + secretKeyRef: + name: korvet-redis-credentials + key: password + resources: + requests: + memory: "512Mi" + cpu: "500m" + limits: + memory: "2Gi" + cpu: "2000m" + livenessProbe: + httpGet: + path: /actuator/health/liveness + port: management + initialDelaySeconds: 30 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + readinessProbe: + httpGet: + path: /actuator/health/readiness + port: management + initialDelaySeconds: 10 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 3 + startupProbe: + httpGet: + path: /actuator/health/liveness + port: management + initialDelaySeconds: 10 + periodSeconds: 5 + failureThreshold: 30 + terminationGracePeriodSeconds: 30 +``` + +{{< warning >}} +This Quick Start runs a **single broker** (`replicas: 1`). Every broker in a cluster needs a unique `KORVET_BROKER_ID`, which a plain Deployment cannot provide because all pods share the same ConfigMap. To run multiple brokers, use a StatefulSet that derives a per-pod ID from the pod ordinal. See [Kubernetes StatefulSet]({{< relref "/integrate/korvet/operations/deployment#kubernetes-statefulset" >}}) in the deployment guide. +{{< /warning >}} + +### Service + +Expose Korvet within the cluster: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: korvet + namespace: korvet + labels: + app: korvet +spec: + selector: + app: korvet + ports: + - name: kafka + port: 9092 + targetPort: kafka + protocol: TCP + - name: management + port: 8080 + targetPort: management + protocol: TCP + type: ClusterIP +``` + +For external access, use a LoadBalancer or Ingress (see [External Access](#external-access)). + +## External Access + +### LoadBalancer Service + +For cloud environments with LoadBalancer support: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: korvet-external + namespace: korvet +spec: + selector: + app: korvet + ports: + - name: kafka + port: 9092 + targetPort: kafka + type: LoadBalancer +``` + +### NodePort Service + +For on-premises or development environments: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: korvet-nodeport + namespace: korvet +spec: + selector: + app: korvet + ports: + - name: kafka + port: 9092 + targetPort: kafka + nodePort: 30092 + type: NodePort +``` + +## Horizontal Pod Autoscaler + +{{< warning >}} +Do not autoscale the single-broker Deployment above. Each broker needs a unique `KORVET_BROKER_ID`, so horizontal scaling only works on the StatefulSet-based multi-broker setup. Run the StatefulSet from [the deployment guide]({{< relref "/integrate/korvet/operations/deployment#kubernetes-statefulset" >}}) first, then target it with the HPA below. +{{< /warning >}} + +Scale the StatefulSet based on CPU utilization: + +```yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: korvet-hpa + namespace: korvet +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: StatefulSet + name: korvet + minReplicas: 3 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 +``` + +## Pod Disruption Budget + +Ensure high availability during cluster maintenance: + +```yaml +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: korvet-pdb + namespace: korvet +spec: + minAvailable: 2 + selector: + matchLabels: + app: korvet +``` + +## TLS Configuration + +### Create TLS Secret + +```bash +kubectl create secret tls korvet-tls \ + --namespace korvet \ + --cert=server.crt \ + --key=server.key +``` + +### Mount TLS in Deployment + +Add the following to the Deployment spec: + +```yaml +spec: + containers: + - name: korvet + env: + - name: KORVET_BROKER_TLS + value: "true" + - name: KORVET_BROKER_CERT_FILE + value: "/etc/korvet/tls/tls.crt" + - name: KORVET_BROKER_KEY_FILE + value: "/etc/korvet/tls/tls.key" + volumeMounts: + - name: tls-certs + mountPath: /etc/korvet/tls + readOnly: true + volumes: + - name: tls-certs + secret: + secretName: korvet-tls +``` + +## Remote Storage with S3 + +For tiered storage to an Apache Iceberg table on S3. + +### Using IAM Roles for Service Accounts (IRSA) + +On EKS, use IRSA for secure S3 access: + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: korvet + namespace: korvet + annotations: + eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/KorvetS3Role +``` + +Add to Deployment: + +```yaml +spec: + serviceAccountName: korvet + containers: + - name: korvet + env: + - name: KORVET_STORAGE_REMOTE_PATH + value: "s3://my-bucket/korvet" + - name: KORVET_STORAGE_REMOTE_S3_REGION + value: "us-east-1" +``` + +### Using Static Credentials + +```yaml +env: +- name: KORVET_STORAGE_REMOTE_PATH + value: "s3://my-bucket/korvet" +- name: KORVET_STORAGE_REMOTE_S3_REGION + value: "us-east-1" +- name: KORVET_STORAGE_REMOTE_S3_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: korvet-redis-credentials + key: aws-access-key-id +- name: KORVET_STORAGE_REMOTE_S3_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: korvet-redis-credentials + key: aws-secret-access-key +``` + +## Monitoring with Prometheus + +Korvet exposes Prometheus metrics at `/actuator/prometheus`. + +### ServiceMonitor (for Prometheus Operator) + +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: korvet + namespace: korvet +spec: + selector: + matchLabels: + app: korvet + endpoints: + - port: management + path: /actuator/prometheus + interval: 15s +``` + +## Verification + +### Check Deployment Status + +```bash +kubectl get pods -n korvet +kubectl get svc -n korvet +``` + +### Test Connectivity + +```bash +# Port-forward for local testing +kubectl port-forward -n korvet svc/korvet 9092:9092 + +# Test with kafka-console-producer +kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test +``` + +### Check Logs + +```bash +kubectl logs -n korvet -l app=korvet --tail=100 +``` + +## Next Steps + +- [Monitoring]({{< relref "/integrate/korvet/operations/monitoring" >}}) +- [Logging]({{< relref "/integrate/korvet/operations/logging" >}}) +- [Configuration Reference]({{< relref "/integrate/korvet/quick-start/configuration" >}}) diff --git a/content/integrate/korvet/operations/logging.md b/content/integrate/korvet/operations/logging.md new file mode 100644 index 0000000000..608a205370 --- /dev/null +++ b/content/integrate/korvet/operations/logging.md @@ -0,0 +1,186 @@ +--- +Title: Logging +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Korvet uses Spring Boot's built-in structured logging support with Logback. +linkTitle: Logging +weight: 80 +--- + +Korvet uses Spring Boot's built-in structured logging support with Logback. +Console logs can be emitted in Logstash-compatible JSON when enabled explicitly. + +## Log Format + +Enable structured console logging in `application.yml` with: + +```yaml +logging: + structured: + format: + console: logstash +``` + +A log line looks like this: + +```json +{ + "@timestamp": "2024-01-15T10:30:45.123Z", + "@version": "1", + "message": "Kafka protocol server started successfully on 0.0.0.0:9092 (TLS: false)", + "logger_name": "com.redis.korvet.broker.KorvetBroker", + "thread_name": "main", + "level": "INFO", + "level_value": 20000 +} +``` + +## Log Levels + +Configure log levels via environment variables: + +```bash +# Root level +export LOGGING_LEVEL_ROOT=INFO + +# Korvet components (default) +export LOGGING_LEVEL_COM_REDIS_KORVET=INFO + +# Spring framework +export LOGGING_LEVEL_ORG_SPRINGFRAMEWORK=WARN + +# Redis client +export LOGGING_LEVEL_IO_LETTUCE=INFO +``` + +Or in `application.yml`: + +```yaml +logging: + structured: + format: + console: logstash + level: + root: INFO + com.redis.korvet: INFO + org.springframework: WARN + io.lettuce: INFO +``` + +## Structured Logging + +Korvet guarantees the following request-scoped MDC fields when the broker request context is available: + +- **correlationId**: Kafka request correlation ID +- **clientId**: Kafka client ID +- **apiKey**: Kafka API key name +- **apiVersion**: Kafka API version + +Spring Boot's Logstash formatter adds MDC key/value pairs directly to the JSON object. +Operation-specific values such as topic, partition, and offset continue to appear in message text unless a component explicitly adds them as structured key/value pairs. + +## Plain Text In Tests And CLI + +Test logback configurations remain plain-text for readability. +The operational CLI also keeps its normal stdout command output. Server runtime logs remain plain text unless structured JSON console logging is enabled explicitly. + +## Log Aggregation + +### Filebeat + +Ship logs to Elasticsearch: + +```yaml +filebeat.inputs: +- type: container + paths: + - '/var/lib/docker/containers/*/*.log' + +output.elasticsearch: + hosts: ["elasticsearch:9200"] +``` + +### Fluentd + +``` + + @type tail + path /var/log/korvet/*.log + pos_file /var/log/korvet/korvet.log.pos + tag korvet + + @type json + + + + + @type elasticsearch + host elasticsearch + port 9200 + index_name korvet + +``` + +## Troubleshooting Logs + +Korvet defaults to `INFO` logging. Raise specific components to `DEBUG` only while troubleshooting: + +```bash +export LOGGING_LEVEL_COM_REDIS_KORVET_BROKER=DEBUG +export LOGGING_LEVEL_COM_REDIS_KORVET_STORAGE_TIERED=DEBUG +export LOGGING_LEVEL_COM_REDIS_KORVET_STORAGE_REDIS=DEBUG +export LOGGING_LEVEL_COM_REDIS_KORVET=DEBUG +``` + +## Runtime Log Level Changes With Actuator + +For Spring Boot applications, the usual way to change logger levels without restarting the process is the Actuator `loggers` endpoint. The same approach works in Kubernetes as long as the application includes Actuator and exposes `/actuator/loggers` on the management port. + +Korvet already includes `spring-boot-starter-actuator`, but the `loggers` endpoint must be exposed before you can use it remotely. Add `loggers` to `management.endpoints.web.exposure.include`: + +```yaml +management: + endpoints: + web: + exposure: + include: health,info,prometheus,metrics,loggers +``` + +Update a specific logger at runtime with a `POST` request: + +```json +{"configuredLevel":"DEBUG"} +``` + +Example: + +```bash +curl -X POST http://:8080/actuator/loggers/com.redis.korvet.storage.tiered \ + -H 'Content-Type: application/json' \ + -d '{"configuredLevel":"DEBUG"}' +``` + +For Korvet troubleshooting, this is a good way to enable `DEBUG` only for targeted packages such as: + +- `com.redis.korvet.broker` +- `com.redis.korvet.storage.tiered` +- `com.redis.korvet.storage.redis` + +In Kubernetes, operators commonly use `kubectl port-forward` to reach the management port without exposing the endpoint publicly: + +```bash +kubectl port-forward pod/ 8080:8080 +curl -X POST http://127.0.0.1:8080/actuator/loggers/com.redis.korvet.storage.redis \ + -H 'Content-Type: application/json' \ + -d '{"configuredLevel":"DEBUG"}' +``` + +This change is runtime-only and usually does not survive a pod restart unless you also update the application configuration. If the endpoint is not exposed, or is protected by network policy or security controls, Spring Boot will not let you change logger levels remotely. + +## Next Steps + +- [Monitoring]({{< relref "/integrate/korvet/operations/monitoring" >}}) +- [Troubleshooting]({{< relref "/integrate/korvet/operations/troubleshooting" >}}) diff --git a/content/integrate/korvet/operations/migration.md b/content/integrate/korvet/operations/migration.md new file mode 100644 index 0000000000..65316e89ac --- /dev/null +++ b/content/integrate/korvet/operations/migration.md @@ -0,0 +1,324 @@ +--- +Title: Migrating +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Upgrade an existing Korvet 0.12.5 deployment to the current version. +linkTitle: Migrating +weight: 90 +--- + +This page describes how to upgrade an existing Korvet 0.12.5 deployment to the current version. +The upgrade involves three kinds of changes: + +- Renamed configuration options (`KORVET_SERVER_*` is now `KORVET_BROKER_*`). +- A new Redis key layout for metadata and stream storage. +- A one-time, offline data migration performed with the `korvet migrate` command. + +{{< warning >}} +Only 0.12.5 is supported as a full migration source. The `migrate` command detects the source layout and refuses to run against any other version, with one exception: against a deployment that already uses the current layout it converts leftover per-partition committed-offset keys, as described in [Upgrading from 0.13–0.16](#upgrading-from-013016). +{{< /warning >}} + +## Upgrading from 0.13–0.16 + +Two breaking changes apply when upgrading a deployment that already uses the current key layout (Korvet 0.13 through 0.16): + +- **The Redis database must provide the Search capability**, alongside JSON. The broker creates a search index over committed-offset group hashes at startup and fails to start with a clear error if Search is missing. On Redis Enterprise, enable the Search module on the database before upgrading. +- **Committed offsets moved from one String per group and partition (`{namespace}:broker:commit:{topic:partition:group}`) to one Hash per group (`{namespace}:broker:commit:{group}`).** The broker does not read the old keys; run `korvet migrate --execute` once (with the broker offline) to convert them. The conversion is idempotent and re-running it converges. If the broker finds leftover old-format keys at startup it logs a prominent warning directing to the migrate command, and consumer groups appear to have no committed offsets until the conversion has run. + +```console +$ korvet migrate -u redis://localhost:6379 --execute +Detected Korvet Redis layout: +Target Korvet version: +Migration executed. +MIGRATE committed offsets korvet:broker:commit: (6 offsets) +``` + +Committed-offset keys whose topic no longer exists cannot be converted unambiguously and are skipped with a `SKIP` line; they are ignored by the broker and can be deleted manually. + +## Configuration Changes + +The configuration model was reorganized between 0.12.5 and the current version. +The most visible change is that the Kafka listener moved from `korvet.server.*` to `korvet.broker.*` (environment variables: `KORVET_SERVER_*` to `KORVET_BROKER_*`), but several other areas were renamed, restructured, or removed. +Review every `korvet.*` property and environment variable in your deployment against the tables below; see [Configuration]({{< relref "/integrate/korvet/reference/configuration" >}}) for the full current reference with defaults. + +The tables list properties in their dotted form. The environment-variable form follows Spring Boot's relaxed binding in both versions: uppercase the property name and replace `.` and `-` with `_`. For example, `korvet.broker.max-request-bytes` is set with `KORVET_BROKER_MAX_REQUEST_BYTES`, and the same rule yields the 0.12.5 name on the left-hand side of each row. Sections below call out the cases where the resulting environment variable is not a mechanical rename. + +### Broker Listener (`korvet.server.*` -> `korvet.broker.*`) + +| 0.12.5 | Current | Notes | +|---|---|---| +| `korvet.server.keyspace` | `korvet.namespace` | Moved to the root `korvet` prefix. Default `korvet` unchanged. Env var: `KORVET_SERVER_KEYSPACE` -> `KORVET_NAMESPACE`. | +| `korvet.server.broker-id` | `korvet.broker.id` | | +| `korvet.server.host` | `korvet.broker.host` | | +| `korvet.server.port` | `korvet.broker.port` | | +| `korvet.server.advertised-host` | `korvet.broker.advertised-host` | | +| `korvet.server.advertised-port` | `korvet.broker.advertised-port` | | +| `korvet.server.boss-threads` | `korvet.broker.boss-threads` | | +| `korvet.server.worker-threads` | `korvet.broker.worker-threads` | | +| `korvet.server.max-request-size` | `korvet.broker.max-request-bytes` | Renamed. | +| `korvet.server.partition-max-bytes` | `korvet.broker.fetch-partition-max-bytes` | Renamed. | +| `korvet.server.fetch-max-wait` | `korvet.broker.fetch-max-wait` | | +| `korvet.server.max-pending-bytes` | `korvet.broker.max-pending-bytes` | | +| `korvet.server.resume-pending-bytes` | `korvet.broker.resume-pending-bytes` | | +| `korvet.server.rebalance-delay` | `korvet.broker.rebalance-delay` | | +| `korvet.server.tls` | `korvet.broker.tls` | | +| `korvet.server.cert-file` | `korvet.broker.cert-file` | | +| `korvet.server.key-file` | `korvet.broker.key-file` | | +| `korvet.server.key-password` | `korvet.broker.key-password` | | +| `korvet.server.trust-cert-file` | `korvet.broker.trust-cert-file` | | +| `korvet.server.client-auth-required` | `korvet.broker.client-auth-required` | | +| `korvet.server.bucket-index-search-limit` | *removed* | No longer user-configurable. | + +For environment variables, the mapping is mechanical: `KORVET_SERVER_MAX_REQUEST_SIZE` becomes `KORVET_BROKER_MAX_REQUEST_BYTES`, `KORVET_SERVER_ADVERTISED_HOST` becomes `KORVET_BROKER_ADVERTISED_HOST`, and so on. + +### Redis Connection (`korvet.redis.*`) + +| 0.12.5 | Current | Notes | +|---|---|---| +| `korvet.redis.uri`, `host`, `port`, `username`, `password`, `cluster` | unchanged | | +| `korvet.redis.timeout` | `korvet.redis.timeout` | Now has an explicit default of `1m`. | +| `korvet.redis.io-thread-pool-size` | `korvet.redis.io-threads` | Renamed. | +| `korvet.redis.pool-size` | `korvet.redis.pool.size` | Now nested under `pool`. | +| `korvet.redis.pool-max-wait` | `korvet.redis.pool.max-wait` | Now nested under `pool`. Default changed from `10s` to `3s`. | +| `korvet.redis.metadata-pool-size` | *removed* | The per-purpose connection pools (metadata, archival source, committed offsets) were consolidated into the single `korvet.redis.pool`. A separate Redis client for message storage can be configured with `korvet.storage.local.redis.*` instead. | +| `korvet.redis.archival-source-pool-size` | *removed* | | +| `korvet.redis.committed-offset-pool-size` | *removed* | | +| `korvet.redis.archival-source-pool-max-wait` | *removed* | | +| `korvet.redis.committed-offset-pool-max-wait` | *removed* | | +| `korvet.redis.metrics.*` | unchanged | | + +Note that because `.` and `-` both map to `_`, the environment variables for the pool settings are unchanged despite the nesting: `KORVET_REDIS_POOL_SIZE` and `KORVET_REDIS_POOL_MAX_WAIT` keep working. The renamed I/O thread setting becomes `KORVET_REDIS_IO_THREAD_POOL_SIZE` -> `KORVET_REDIS_IO_THREADS`. + +### Topic Configuration (`korvet.topics.*`) + +In 0.12.5, `korvet.topics.*` was a single, flat set of defaults applied to all topics. +It is now a **list** of glob patterns evaluated first-match-wins, with the same flat per-topic setting names inside each list entry. +A flat 0.12.5 default block translates to one catch-all `name: "*"` entry: + +```yaml +# 0.12.5 +korvet: + topics: + auto-create: true + partitions: 3 + retention-time: 7d + retention-bytes: -1 + +# Current +korvet: + topics: + - name: "*" + auto-create: true + partitions: 3 + retention-time: 7d + retention-bytes: -1 +``` + +| 0.12.5 | Current | Notes | +|---|---|---| +| `korvet.topics.auto-create` | `korvet.topics[*].auto-create` | Now per-pattern. | +| `korvet.topics.partitions` | `korvet.topics[*].partitions` | | +| `korvet.topics.compression` | `korvet.topics[*].compression` | | +| `korvet.topics.offset-sequence-bits` | `korvet.topics[*].offset-sequence-bits` | | +| `korvet.topics.retention-time` | `korvet.topics[*].retention-time` | Now per-pattern. | +| `korvet.topics.retention-bytes` | `korvet.topics[*].retention-bytes` | Now per-pattern. | +| `korvet.topics.local-retention-time` | `korvet.topics[*].local-retention-time` | Now per-pattern. | +| `korvet.topics.local-retention-bytes` | `korvet.topics[*].local-retention-bytes` | Now per-pattern. | +| `korvet.topics.remote-storage-enabled` | `korvet.topics[*].remote-storage-enabled` | | +| `korvet.topics.bucket-duration` | `korvet.topics[*].segment-time` | Buckets were replaced by segments; a `segment-bytes` size threshold is also available. | +| `korvet.topics.storage-compression` | *removed* | At-rest value compression is now configured server-wide with `korvet.storage.local.compression.codec` (default `none`). This codec must match the `--storage-compression` option used during migration (see below). | +| `korvet.topics.value-type` | *removed* | The legacy RAW/JSON/AUTO value layout no longer exists; all records use the new envelope format. | +| `korvet.topics.average-message-bytes` | *removed* | No longer user-configurable; average message size is measured at runtime. | +| `korvet.topics.approximate-trimming` | *removed* | | + +In environment-variable form, the list entries are addressed by index, and each entry needs a `NAME`. The flat 0.12.5 variables translate to a `_0_` catch-all entry: + +```console +# 0.12.5 +export KORVET_TOPICS_AUTO_CREATE=true +export KORVET_TOPICS_PARTITIONS=3 +export KORVET_TOPICS_RETENTION_TIME=7d +export KORVET_TOPICS_RETENTION_BYTES=-1 + +# Current +export KORVET_TOPICS_0_NAME='*' +export KORVET_TOPICS_0_AUTO_CREATE=true +export KORVET_TOPICS_0_PARTITIONS=3 +export KORVET_TOPICS_0_RETENTION_TIME=7d +export KORVET_TOPICS_0_RETENTION_BYTES=-1 +``` + +### Storage (`korvet.storage.*`) + +In 0.12.5 the only storage configuration was the remote (cold) tier under `korvet.storage.remote.*`, backed by Delta Lake. +The current version splits storage configuration into three areas: + +- `korvet.storage.local.*` — *new*: the Redis (hot) tier, including the at-rest compression codec and an optional dedicated Redis client for message storage (see [New Configuration Areas](#new-configuration-areas)). +- `korvet.storage.worker.*` — *new*: the background worker that handles retention and segment tiering, replacing the 0.12.5 archiver. +- `korvet.storage.remote.*` — the cold tier, which moved from Delta Lake to Apache Iceberg: + +| 0.12.5 | Current | Notes | +|---|---|---| +| `korvet.storage.remote.path` | `korvet.storage.remote.path` | Unchanged, but now points at an Iceberg warehouse. | +| `korvet.storage.remote.s3.region`, `endpoint`, `access-key-id`, `secret-access-key` | unchanged | A new `path-style-access` option is available for S3-compatible stores such as MinIO. | +| `korvet.storage.remote.s3.credentials.type` | *removed* | Credential resolution is now automatic. | +| `korvet.storage.remote.archiver.*` | *removed* | Archiver internals are no longer exposed; the storage worker (`korvet.storage.worker.*`) manages tiering. | +| `korvet.storage.remote.writer.*` | *removed* | Replaced by Iceberg writer settings `korvet.storage.remote.iceberg.target-file-size` and `row-group-size`. | +| `korvet.storage.remote.index.*` | *removed* | The cold index is no longer user-configurable. | + +### New Configuration Areas + +These did not exist in 0.12.5; defaults are generally sensible, but review them as part of the upgrade: + +- `korvet.broker.enabled`, `korvet.broker.response-queue-timeout`, `korvet.broker.produce-timeout`, `korvet.broker.rebalance-threads` — new listener tuning options. +- `korvet.broker.sasl.*` — SASL authentication (PLAIN, SCRAM-SHA-256). +- `korvet.broker.metrics.*` — consumer-offset gauge publishing. +- `korvet.redis.circuit-breaker.*` — circuit breaker for stream operations (enabled by default). +- `korvet.storage.local.compression.codec` — server-wide at-rest compression codec (default `none`). +- `korvet.storage.local.redis.*` — optional separate Redis client/pool for message storage. +- `korvet.storage.worker.*` — background worker for retention and segment management. +- `korvet.schema-registry.*` — embedded schema registry. +- `korvet.admin.*` — Admin API bootstrap credentials. The defaults are `admin`/`admin`; **change these in production**. + +## Redis Key Layout Changes + +The current version uses a different Redis key layout than 0.12.5, for both metadata and message storage. +This is why a data migration is required: a current server cannot read 0.12.5 keys directly. + +### Metadata Keys + +| Area | 0.12.5 | Current | +|---|---|---| +| Topic registry | `{namespace}:topics` (Set) + one `{namespace}:topic:{topic}` Hash per topic + `{namespace}:topic-ids` Hash | `{namespace}:topics` (single JSON document containing all topics and their config) | +| Broker registry | `{namespace}:brokers` (Set) + one `{namespace}:broker:{id}` Hash per broker | `{namespace}:broker:nodes` (single JSON document) | +| Credentials | `{namespace}:credentials:{username}` (Hash, fields `passwordHash`, `serverKey`, `tenantId`) | `{namespace}:broker:credentials:{username}` (Hash, fields renamed to `password_hash`, `server_key`, `tenant_id`) | +| Committed offsets | `{namespace}:commit:{namespace}:stream:{topic}:{partition}:{group}` (String) | `{namespace}:broker:commit:{group}` (one Hash per consumer group: a `group` field holding the group ID plus one `{topic}:{partition}` field per committed offset), listed through the `{namespace}:broker:commit-idx` search index | + +### Stream Storage Keys + +Messages are still stored in Redis Streams, but both the key names and the per-entry field layout changed. + +**Keys:** + +- 0.12.5: `{namespace}:stream:{topic}:{partition}` +- Current: `{namespace}:storage:local:{topic}:{partition}` + +**Stream entry fields:** + +In 0.12.5, each record was stored as multiple stream fields: the record key under `__key`, the value either under `__value` (raw) or flattened into one field per top-level JSON attribute, and each header under `__header.`. + +The current format stores each record as a fixed envelope: + +| Field | Content | +|---|---| +| `value` | Record value bytes, stored with the configured at-rest codec (`none` by default, i.e. uncompressed) | +| `key` | Record key bytes, verbatim | +| `headers` | All record headers encoded into one self-delimiting blob | +| `timestamp` | Kafka record timestamp as a decimal string | + +Stream entry IDs (and therefore Kafka offsets) are preserved by the migration. + +{{< note >}} +0.12.5 did not store the producer's record timestamp, so migrated records carry no timestamp (`-1`) rather than a fabricated one. +{{< /note >}} + +## The `korvet migrate` Command + +The `migrate` command reads a 0.12.5 keyspace and rewrites it in place into the current layout. +It migrates, in order: + +1. **Topic registry** — converts the topic Set and per-topic Hashes into the new JSON document, and pins each topic's `storageCompressionType` to the codec used during migration. +2. **Broker registry** — converts the broker Set and per-broker Hashes into the `broker:nodes` JSON document. +3. **Credentials** — moves each credential Hash to the new key and renames its fields. +4. **Committed offsets** — rewrites each per-partition committed-offset String into the owning group's Hash, preserving values. +5. **Local streams** — rewrites every `{topic}:{partition}` stream into the new location, converting each entry from the legacy field layout to the new envelope format (decompressing legacy values where needed and re-compressing with the chosen codec), preserving stream entry IDs. + +By default the command is a **dry run**: it prints what it would migrate without writing anything. +Pass `--execute` to apply the migration. +Successfully migrated source keys are deleted unless `--keep-source` is given. + +### Options + +| Option | Default | Description | +|---|---|---| +| `-u`, `--uri` | *(required)* | Redis connection URI of the deployment to migrate. | +| `--namespace` | `korvet` | Korvet keyspace/namespace used by the legacy deployment. | +| `--execute` | off | Apply the migration. Without this option the command only prints a dry run. | +| `--replace` | off | Replace existing destination keys. Without it, the command skips any area whose destination key already exists (useful when re-running after a partial migration). | +| `--keep-source` | off | Keep legacy source keys after successful migration. By default, migrated source keys are deleted. | +| `--storage-compression` | `none` | At-rest codec to write migrated stream values with. Must match the target server's `korvet.storage.local.compression.codec`. Values: `none`, `gzip`, `snappy`, `lz4`, `zstd`. | +| `--transfers` | `1` | Number of streams (topic-partitions) to migrate in parallel. Each transfer rewrites one stream, in order, on its own Redis connection. Increase this to speed up migrations with many topic-partitions. | + +The command also accepts the standard Redis connection options (TLS, credentials, timeouts) shared by all `korvet` commands; run `korvet migrate --help` for the full list. + +### Output and Exit Code + +The command prints the detected source layout, the target version, and one line per migrated (`MIGRATE`), skipped (`SKIP`), or failed (`ERROR`) item: + +```console +$ korvet migrate -u redis://localhost:6379 --execute +Detected Korvet Redis layout: 0.12.5 +Target Korvet version: 0.19 +Migration executed. +MIGRATE topic registry korvet:topics (3 topics) +MIGRATE broker registry korvet:broker:nodes (1 brokers) +MIGRATE credentials korvet:broker:credentials: (2 credentials) +MIGRATE committed offsets korvet:broker:commit: (6 offsets) +MIGRATE local streams korvet:storage:local: (9 streams) +``` + +The exit code is `0` on success (including dry runs) and `1` if any error was reported. + +## Step-by-Step Migration + +The migration must be performed offline: no 0.12.5 broker may be writing to Redis while data is being rewritten, and the new broker must not start until the migration has completed. + +1. **Stop producers and consumers.** Drain or pause all Kafka clients so no data is lost while the broker is down. + +2. **Shut down the 0.12.5 broker(s).** Stop every Korvet server process or container that uses this Redis database. + +3. **Back up the Redis database.** Take an RDB snapshot (or use your usual backup mechanism) so you can roll back if needed: + + ```console + redis-cli -u redis://localhost:6379 BGSAVE + ``` + +4. **Run a dry run.** Inspect what will be migrated without changing anything: + + ```console + korvet migrate -u redis://localhost:6379 + ``` + + Review the `MIGRATE`/`SKIP` lines. If you used a custom namespace, pass `--namespace `. + +5. **Execute the migration:** + + ```console + korvet migrate -u redis://localhost:6379 --execute + ``` + + For large deployments with many topic-partitions, parallelize stream migration: + + ```console + korvet migrate -u redis://localhost:6379 --execute --transfers 8 + ``` + + If you want to keep the legacy keys around until you have verified the new deployment, add `--keep-source` (note that this temporarily doubles the memory used by stream data). + + Verify that the command exits with code `0` and reports no `ERROR` lines. If it fails partway, fix the cause and re-run with `--execute --replace` to overwrite partially written destinations. + +6. **Update the broker configuration.** Apply all renames from [Configuration Changes](#configuration-changes) above: `korvet.server.*` to `korvet.broker.*` (including `korvet.server.keyspace` to `korvet.namespace`), the `korvet.redis` pool restructure, and the flat `korvet.topics.*` block to a pattern list. Remove any settings listed as removed. If you passed a non-default `--storage-compression`, set `korvet.storage.local.compression.codec` to the same value. + +7. **Start the new broker version.** Deploy and start the current Korvet release against the same Redis database. + +8. **Verify.** Check that the broker starts cleanly, then validate with a Kafka client: list topics, consume existing messages from an old topic, produce and consume a new message, and confirm consumer groups resume from their committed offsets. + +9. **Resume traffic.** Re-enable producers and consumers. Once you are satisfied, you can delete the legacy keys if you used `--keep-source`, and remove the backup per your retention policy. + +## Rollback + +If verification fails, stop the new broker, restore the Redis backup taken in step 3, and restart the 0.12.5 deployment. +Because the migration runs offline, no new data is produced between backup and verification, so the restore is lossless. diff --git a/content/integrate/korvet/operations/monitoring.md b/content/integrate/korvet/operations/monitoring.md new file mode 100644 index 0000000000..c8917baccd --- /dev/null +++ b/content/integrate/korvet/operations/monitoring.md @@ -0,0 +1,225 @@ +--- +Title: Monitoring +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Monitor Korvet through Spring Boot Actuator and Micrometer. +linkTitle: Monitoring +weight: 60 +--- + +Korvet provides comprehensive monitoring through Spring Boot Actuator and Micrometer. + +## Health Checks + +Korvet exposes health check endpoints: + +```bash +# Overall health +curl http://localhost:8080/actuator/health + +# Liveness probe (for Kubernetes) +curl http://localhost:8080/actuator/health/liveness + +# Readiness probe (for Kubernetes) +curl http://localhost:8080/actuator/health/readiness +``` + +{{< note >}} +The `liveness` and `readiness` health groups are only exposed when probes are enabled. Spring Boot enables them automatically when it detects a Kubernetes environment, or you can enable them explicitly with `management.endpoint.health.probes.enabled=true`. On a plain local run without this setting, the `/actuator/health/liveness` and `/actuator/health/readiness` sub-paths are not available; use `/actuator/health` instead. +{{< /note >}} + +## Metrics + +Metrics are exposed in Prometheus format: + +```bash +curl http://localhost:8080/actuator/prometheus +``` + +### Available Metrics + +#### Korvet Custom Metrics + +- **korvet.broker.produce**: Produce request latency histogram (tag: `topic`) +- **korvet.broker.fetch**: Fetch request latency histogram (tag: `topic`) +- **korvet.broker.request**: Kafka API request latency histogram (tags: `api_key`, `result`) +- **korvet.broker.frame_size**: Kafka frame size distribution (tags: `direction`, `api_key`) +- **korvet.broker.up**: Broker lifecycle gauge +- **korvet.broker.failures**: Broker failure counter (tags: `operation`, `error_type`). Authentication failures arrive here with `operation=auth`. +- **korvet.storage.read**, **korvet.storage.write**, **korvet.storage.ack**: Cross-tier storage verb latency timers (tags: `tier`, `result`; read also carries `mode=stream|group`) +- **korvet.storage.read.messages**, **korvet.storage.write.messages**: Cross-tier messages-per-call distribution summaries (tag: `tier`; read.messages also carries `mode`) +- **korvet.storage.archive**, **korvet.storage.archive.segments**, **korvet.storage.archive.bytes**, **korvet.storage.archive.failures**, **korvet.storage.archive.lag.\***: Cold-tier offload latency, throughput, failures, and sealed-segment backlog. +- **korvet.storage.local.pool.acquire** / **korvet.storage.local.pool.pending**: Local Redis connection-pool meters (acquire timer carries `result=success|timeout|error`) +- **korvet.storage.worker.up**, **korvet.storage.worker.failures**: Storage worker liveness gauge and failure counter (failures tagged `phase`, `error_type`). The worker's storage I/O flows through the shared `korvet.storage.{read,write,ack}` timers tagged `tier=remote`. + +See [Metrics Reference]({{< relref "/integrate/korvet/reference/metrics" >}}) for the full module-by-module catalog. + +#### JVM and System Metrics + +Standard JVM and system metrics from Micrometer: + +- **jvm.memory.used**: JVM memory used +- **jvm.memory.max**: JVM maximum memory +- **jvm.gc.pause**: Garbage collection pause time +- **jvm.threads.live**: Live threads +- **process.cpu.usage**: Process CPU usage +- **system.cpu.usage**: System CPU usage +- **system.load.average.1m**: System load average + +## Prometheus Configuration + +Add Korvet to your Prometheus scrape config: + +```yaml +scrape_configs: + - job_name: 'korvet' + static_configs: + - targets: ['korvet:8080'] + metrics_path: '/actuator/prometheus' +``` + +## Example Prometheus Queries + +See [Metrics Reference]({{< relref "/integrate/korvet/reference/metrics" >}}) for the full metric and tag vocabulary used below. + +**Produce latency (p99) per topic**: + +```promql +histogram_quantile(0.99, sum by (topic, le) (rate(korvet_broker_produce_seconds_bucket[5m]))) +``` + +**Fetch latency (p95) per topic**: + +```promql +histogram_quantile(0.95, sum by (topic, le) (rate(korvet_broker_fetch_seconds_bucket[5m]))) +``` + +**Broker request rate by API key and result**: + +```promql +sum by (api_key, result) (rate(korvet_broker_request_seconds_count[5m])) +``` + +**Local Redis pool timeout rate**: + +```promql +rate(korvet_storage_local_pool_acquire_seconds_count{result="timeout"}[5m]) +``` + +**Read mix by tier**: + +```promql +sum by (tier, result) (rate(korvet_storage_read_seconds_count[5m])) +``` + +**Broker failure rate by operation and error type**: + +```promql +sum by (operation, error_type) (rate(korvet_broker_failures_total[5m])) +``` + +**Redis command latency (p99) by command**: + +```promql +histogram_quantile(0.99, sum by (command, le) (rate(lettuce_command_completion_seconds_bucket[5m]))) +``` + +## Monitoring Stack Setup + +A complete monitoring stack with Prometheus and Grafana is available in the [korvet-dist observability sample](https://github.com/redis-field-engineering/korvet-dist/tree/main/samples/observability) directory. + +### Quick Start with Docker Compose + +The easiest way to set up monitoring is to include the observability stack in your `docker-compose.yml`: + +```yaml +include: + - path/to/korvet-dist/samples/observability/monitoring.yml + +services: + redis: + image: redis:8.6 + ports: + - "6379:6379" + + korvet: + image: redisfield/korvet:latest + command: server + ports: + - "9092:9092" + - "8080:8080" + environment: + - KORVET_REDIS_HOST=redis + - KORVET_REDIS_METRICS_ENABLED=true + depends_on: + - redis +``` + +This automatically adds: + +- **Prometheus** on port 9090 - Metrics collection and storage +- **Grafana** on port 3000 - Pre-configured dashboard for Korvet metrics + +### Accessing the Dashboard + +1. Start your services: + + ```bash + docker compose up -d + ``` + +2. Open Grafana at http://localhost:3000 +3. The Korvet dashboard loads automatically (no login required) + +### Dashboard Features + +The pre-built Grafana dashboard visualizes: + +- **Message Rates**: Real-time produce/fetch rates using `irate()` for instant metrics +- **Latency Percentiles**: P50, P95, P99 for produce and fetch operations +- **Throughput**: Ingress and egress bytes/sec +- **Redis Metrics**: Command rates, latency percentiles (when `korvet.redis.metrics.enabled=true`) +- **JVM Metrics**: Heap memory, GC pauses, thread count +- **System Metrics**: CPU usage, load average, disk space + +The dashboard defaults to a 5-minute time range with 5-second auto-refresh for real-time monitoring. + +### Standalone Setup + +For production deployments, see the [observability README](https://github.com/redis-field-engineering/korvet-dist/tree/main/samples/observability/README.adoc) for: + +- Prometheus configuration examples +- Grafana datasource and dashboard provisioning +- Customizing the dashboard + +## Alerting + +Set up alerts for: + +### Korvet-Specific Alerts + +- **High produce latency**: p99 > 100ms +- **High fetch latency**: p99 > 50ms +- **Broker request error spike**: `result=error` share of `korvet.broker.request` > 1% of requests +- **Redis pool contention**: sustained `korvet.storage.local.pool.pending` growth or `korvet.storage.local.pool.acquire{result="timeout"}` rate +- **No produce activity**: No messages produced in 5 minutes (if expected) + +### System Alerts + +- **Memory pressure**: JVM heap > 80% +- **High GC activity**: Frequent or long GC pauses +- **High CPU usage**: Process CPU > 80% + +## Logging + +See [Logging]({{< relref "/integrate/korvet/operations/logging" >}}) for log-based monitoring. + +## Next Steps + +- [Logging]({{< relref "/integrate/korvet/operations/logging" >}}) +- [Troubleshooting]({{< relref "/integrate/korvet/operations/troubleshooting" >}}) +- [Metrics Reference]({{< relref "/integrate/korvet/reference/metrics" >}}) +- [Metrics Contract]({{< relref "/integrate/korvet/reference/metrics/contracts" >}}) diff --git a/content/integrate/korvet/operations/production-tuning.md b/content/integrate/korvet/operations/production-tuning.md new file mode 100644 index 0000000000..bb38c51e4d --- /dev/null +++ b/content/integrate/korvet/operations/production-tuning.md @@ -0,0 +1,67 @@ +--- +Title: Production Tuning +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Guidance for sizing and tuning Korvet for production produce/consume + workloads. +linkTitle: Production Tuning +weight: 20 +--- + +Guidance for sizing and tuning Korvet for production produce/consume workloads. The defaults are chosen to be safe for small deployments; high-volume or multi-datasource workloads typically need the storage connection pool sized to their partition fan-out. + +## Storage connection pool sizing + +Korvet writes every record to Redis through a bounded connection pool (`korvet.redis.pool`, or the storage-tier override `korvet.storage.local.redis.pool`). The broker serializes storage writes *per partition*, and each in-flight partition write holds a borrowed connection for the duration of its `XADD`. Connection demand therefore scales with the number of *distinct partitions being produced to concurrently* — not with the raw request rate. + +A round-robin producer spreads a single topic across all of its partitions, so the connections in simultaneous demand are roughly: + +``` +sum(partitions) across all topics actively produced to + + headroom for metadata reads sharing the same pool (ListOffsets / getStreamInfo) +``` + +When demand exceeds the pool size, writes queue until they exceed `korvet.redis.pool.max-wait` (default 3s) and fail. Before Korvet 0.17.x these failures surfaced to the client as the non-retriable `UNKNOWN_SERVER_ERROR`, causing producers such as Logstash to *drop* records; they are now surfaced as the retriable `REQUEST_TIMED_OUT` so clients retry instead. Either way, a pool sized below the partition fan-out caps throughput and adds latency, so size it correctly. + +**Sizing rule** + +{{< note >}} +Set `korvet.redis.pool.size` to at least the sum of partitions across all topics produced to concurrently, plus headroom for metadata reads. For example, a round-robin producer writing to two 8-partition topics drives 16 concurrent partition writes — well above the historical default of 8. +{{< /note >}} + +The default pool size is *32*, which comfortably covers a couple of 8-partition topics with metadata headroom. Raise it for larger fan-outs: + +```bash +# e.g. ~50 partitions of concurrent produce fan-out + metadata headroom +export KORVET_REDIS_POOL_SIZE=64 +export KORVET_REDIS_POOL_MAX_WAIT=3s # block time before an acquire fails +export KORVET_REDIS_IO_THREADS=8 # Lettuce event-loop threads +``` + +| Setting | Env var | Notes | +|---|---|---| +| `korvet.redis.pool.size` | `KORVET_REDIS_POOL_SIZE` | Max pooled connections. Floor = sum of concurrent partition writes + metadata headroom. Default 32. | +| `korvet.redis.pool.max-wait` | `KORVET_REDIS_POOL_MAX_WAIT` | Time a write blocks waiting to borrow a connection before failing. Default 3s. | +| `korvet.redis.io-threads` | `KORVET_REDIS_IO_THREADS` | Lettuce event-loop thread pool. Raise alongside the pool on high-core hosts. | + +When the message-storage tier uses a dedicated Redis (`korvet.storage.local.redis.*`), size *that* pool to the produce fan-out; `korvet.redis.pool` then carries only metadata and registry traffic. Override fields are sparse — anything unset inherits from `korvet.redis.pool`. + +### Confirm before raising + +More connections do not help if Redis itself is the bottleneck. Before scaling the pool, confirm the Redis backend has CPU and latency headroom — otherwise additional connections just move the queue from the pool to Redis. + +### Metrics to watch + +Watch these while tuning (see [Storage metrics]({{< relref "/integrate/korvet/reference/metrics/storage" >}})): + +- `korvet.storage.local.pool.pending` — waiters queued for a connection. Sustained non-zero values mean the pool is undersized for the load. +- `korvet.storage.local.pool.acquire` — acquisition latency by result. A rising `timeout` result count is the direct signal that produces are failing on pool acquisition. + +## Next Steps + +- [Monitoring]({{< relref "/integrate/korvet/operations/monitoring" >}}) +- [Troubleshooting]({{< relref "/integrate/korvet/operations/troubleshooting" >}}) +- [Configuration Reference]({{< relref "/integrate/korvet/reference/configuration" >}}) diff --git a/content/integrate/korvet/operations/resilience.md b/content/integrate/korvet/operations/resilience.md new file mode 100644 index 0000000000..fd8f050df2 --- /dev/null +++ b/content/integrate/korvet/operations/resilience.md @@ -0,0 +1,130 @@ +--- +Title: Resilience & Chaos Testing +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: How Korvet behaves under infrastructure failures, with the automated chaos + test suite that validates this behaviour and an operator runbook. +linkTitle: Resilience & Chaos Testing +weight: 70 +--- + +How Korvet behaves under infrastructure failures, the automated chaos +test suite that validates this behaviour, and a runbook for operators responding to +common failure scenarios. + +## Resilience model + +Korvet is a stateless broker: all durable state (topics, messages, consumer offsets, +producer IDs) lives in Redis. A Korvet process holds only in-flight request state and +cached metadata. Two consequences follow: + +- **Broker restarts are cheap and lossless.** A restarted broker rebuilds its view from + Redis, so any record that was acknowledged before the restart remains readable. +- **Korvet's availability tracks Redis.** When Redis is unreachable, Korvet cannot durably + write or read, so it fails affected requests rather than acknowledging data it cannot + persist. This is by design: an acknowledgement always means the data is in Redis. + +## Failure modes + +| Failure | Broker behaviour | Recovery | +|---|---|---| +| Redis connection dropped | In-flight and new produce/fetch requests fail; the broker does not acknowledge writes it cannot persist. No partial or phantom acknowledgements. | Lettuce auto-reconnects when Redis returns. The next request succeeds with no operator action. Acknowledged data is intact. | +| Network latency to Redis | Request latency increases roughly in proportion to the added round-trip cost; throughput drops. No errors while latency stays under client timeouts. | Latency returns to normal when the network recovers. No operator action. | +| Network partition from Redis | Equivalent to a connection drop for the duration of the partition: affected requests fail. | Broker reconnects automatically when the partition heals and resumes serving traffic. | +| Broker (pod) restart | Connected clients see their connection drop and reconnect to the broker (or another broker behind the load balancer), retrying in-flight requests. | Restarted broker serves all data acknowledged before the restart. No data loss. | +| Remote storage (S3) outage during tiering | Local (Redis) tier is unaffected: produce and recent-data fetch continue. Offload of cold segments retries with backoff; reads of already-offloaded segments fail until S3 returns. | Offload resumes when S3 recovers; no data is lost because segments are only deleted from the local tier after a successful offload. | + +## Automated chaos test suite + +`ChaosEngineeringIntegrationTest` (module `korvet-test`) injects failures into the network +path between the broker and Redis using +[Toxiproxy](https://github.com/Shopify/toxiproxy). Redis runs on a private Docker network and +all broker traffic is routed through Toxiproxy, so a test can sever the connection, add +latency, or simulate a partition while the broker keeps running. Redis state is never +destroyed, which lets every scenario assert that acknowledged data survives the failure. + +| Scenario | Assertion | +|---|---| +| Redis connection cut during produce | Sends during the outage fail fast (graceful degradation); every acknowledged record stays readable; sends after recovery succeed. | +| Redis connection cut during consume | The consumer does not crash — polls return no records during the outage — and reads every record after recovery. | +| Latency injection | Added round-trip latency is observable on a produce cycle; the broker returns to normal once the latency is removed. | +| Network partition then heal | The broker reconnects after the partition heals and serves a full produce/consume cycle. | +| Broker restart | Data acknowledged before the restart is still readable from a freshly started broker. | + +The chaos tests are tagged `@Tag("chaos")`. Because they sleep through simulated outages they +are slower than regular integration tests, so they are excluded from the default +`integrationTest` suite and run in a dedicated `chaosTest` task (and its own CI workflow). + +Run the suite: + +```bash +./gradlew chaosTest +``` + +The suite requires a Docker daemon (Testcontainers pulls the Redis and Toxiproxy images). + +### Scenarios validated outside this suite + +Two scenarios from the resilience story need an environment Testcontainers cannot provide and +are exercised by end-to-end suites instead: + +- **S3 503 errors during tiering** — covered by the remote-storage integration tests against a + MinIO/S3 endpoint (`korvet-storage-tiered-iceberg`). The offload path retries on transient + 5xx responses and only removes a local segment after a confirmed offload. +- **Kubernetes pod eviction** — covered by deploying to a cluster and deleting the broker pod + while a client produces. Because state is in Redis, the client reconnects (to the rescheduled + pod or another replica) and no acknowledged data is lost. + +## Runbook + +### Redis is unreachable + +**Symptoms**: produce/fetch requests fail or time out; broker logs show Lettuce reconnect +attempts; `redis_client` metrics show connection errors. + +1. Confirm Redis health directly: `redis-cli -h -p ping`. +2. Check network reachability from the broker host to Redis (firewall, security groups, DNS). +3. If Redis is up and reachable, the broker reconnects automatically — no restart needed. + Verify recovery by producing a test record. +4. If Redis is down, restore it. Korvet resumes serving as soon as the connection is + re-established. Acknowledged data is intact. + +### Elevated produce/fetch latency + +**Symptoms**: client-observed latency rises; broker-to-Redis round-trip metrics increase. + +1. Inspect network latency between the broker and Redis. +2. Check Redis-side load (slow commands, CPU, memory pressure). +3. Latency clears on its own once the network or Redis recovers; no Korvet action is required. + If it persists, scale Redis or move the broker closer to Redis (same AZ/region). + +### Network partition between broker and Redis + +**Symptoms**: sustained request failures with no Redis-side errors; broker cannot reach Redis. + +1. Treat as "Redis is unreachable" above — the failure mode and recovery are identical. +2. The broker reconnects automatically when the partition heals; confirm with a test produce. + +### Broker (pod) restart + +**Symptoms**: clients briefly disconnect and reconnect; in-flight requests retry. + +1. No data action required — durable state is in Redis. +2. After restart, verify the broker is healthy (`/actuator/health`) and serving by producing + and consuming a test record. +3. For zero-disruption restarts, run multiple replicas behind a load balancer so clients + fail over while one pod restarts (see [Kubernetes]({{< relref "/integrate/korvet/operations/kubernetes" >}})). + +### Remote storage (S3) outage + +**Symptoms**: cold-segment offload stalls; reads of already-offloaded (cold) data fail; recent +data still produces and consumes normally. + +1. Confirm the S3 endpoint/credentials and bucket reachability. +2. Recent data on the local (Redis) tier is unaffected — produce and recent reads continue. +3. Offload resumes automatically when S3 recovers. No data is lost: segments are removed from + the local tier only after a successful offload. +4. Watch the storage-worker metrics for offload backlog draining once S3 is healthy. diff --git a/content/integrate/korvet/operations/troubleshooting.md b/content/integrate/korvet/operations/troubleshooting.md new file mode 100644 index 0000000000..7e106077d5 --- /dev/null +++ b/content/integrate/korvet/operations/troubleshooting.md @@ -0,0 +1,372 @@ +--- +Title: Troubleshooting +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Common issues and solutions for Korvet. +linkTitle: Troubleshooting +weight: 120 +--- + +Common issues and solutions for Korvet. + +## Connection Issues + +### Cannot connect to Korvet + +**Symptoms**: Kafka clients timeout or fail to connect + +**Solutions**: + +1. Check Korvet is running: + + ```bash + curl http://localhost:8080/actuator/health + ``` + +2. Verify port is accessible: + + ```bash + telnet localhost 9092 + ``` + +3. Check firewall rules allow port 9092 + +### Cannot connect to Redis + +**Symptoms**: Korvet fails to start or logs Redis connection errors + +**Solutions**: + +1. Verify Redis is running: + + ```bash + redis-cli ping + ``` + +2. Check Redis connection settings: + + ```bash + export KORVET_REDIS_URI=redis://localhost:6379 + export KORVET_REDIS_USERNAME=default + export KORVET_REDIS_PASSWORD=secret + ``` + +3. Test Redis connectivity: + + ```bash + redis-cli -h localhost -p 6379 ping + ``` + +## Performance Issues + +### High latency + +**Symptoms**: Slow produce/fetch operations + +**Solutions**: + +1. Check Redis latency: + + ```bash + redis-cli --latency + ``` + +2. Monitor JVM garbage collection: + + ```bash + curl http://localhost:8080/actuator/metrics/jvm.gc.pause + ``` + +3. Increase JVM heap size: + + ```bash + export JAVA_OPTS="-Xmx2g -Xms2g" + ``` + +### Low throughput + +**Symptoms**: Cannot achieve expected messages per second + +**Solutions**: + +1. Enable pipelining in Redis client +2. Increase batch size for produce operations +3. Size the storage connection pool to your partition fan-out — see [Production Tuning]({{< relref "/integrate/korvet/operations/production-tuning" >}}) +4. Scale horizontally (add more Korvet instances) +5. Scale the Redis backend for better performance + +## Data Issues + +### Messages not appearing + +**Symptoms**: Produced messages don't show up when consuming + +**Solutions**: + +1. Check topic exists: + + ```bash + kafka-topics --bootstrap-server localhost:9092 --list + ``` + +2. Verify partition assignment +3. Check consumer is reading from correct offset +4. Enable debug logging: + + ```bash + export LOGGING_LEVEL_COM_REDIS_KORVET=DEBUG + ``` + +### Duplicate messages + +**Symptoms**: Same message consumed multiple times + +**Solutions**: + +1. Check consumer offset management +2. Verify consumer group configuration +3. Ensure proper error handling in consumer + +## Resource Issues + +### Out of memory + +**Symptoms**: JVM crashes with OutOfMemoryError + +**Solutions**: + +1. Increase heap size: + + ```bash + export JAVA_OPTS="-Xmx4g -Xms4g" + ``` + +2. Check for memory leaks in metrics +3. Reduce message batch sizes +4. Enable GC logging for analysis + +### Redis storage full + +**Symptoms**: Cannot produce new messages + +**Solutions**: + +1. Check Redis memory usage: + + ```bash + redis-cli info memory + ``` + +2. Use Redis `XTRIM` to manually trim old messages +3. Reduce retention period for topics +4. Increase Redis memory limit + +## Integration Issues + +### Databricks Spark Structured Streaming Timeout + +**Symptoms**: Databricks Spark Structured Streaming fails with timeout error: + +``` +org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeTopics +``` + +**Root Cause**: Network connectivity issue between Databricks and Korvet, typically when Korvet is behind an AWS load balancer. + +**Diagnosis**: + +1. Test TCP connectivity from Databricks notebook: + + ```python + import socket + socket.create_connection(("", 9092), timeout=10) + ``` + + If this times out, Databricks cannot reach Korvet. + +2. Check if producers work from the same Databricks workspace: + + If producers work but Spark Structured Streaming doesn't, the issue is likely network-related. + +3. Determine where working producers are running: + + - Same Databricks workspace? + - EC2 instances in Korvet's VPC? + - Different network? + +**Solutions**: + +**If Korvet is behind an AWS ELB/NLB:** + +1. **Check ELB scheme** (internal vs internet-facing): + + ```bash + aws elbv2 describe-load-balancers \ + --region \ + --query 'LoadBalancers[?DNSName==``].[Scheme,Type,VpcId]' + ``` + +2. **If ELB is internet-facing**, update security group to allow Databricks: + + ```bash + # Add inbound rule for port 9092 from Databricks IP ranges + aws ec2 authorize-security-group-ingress \ + --group-id \ + --protocol tcp \ + --port 9092 \ + --cidr + ``` + + See [Databricks IP ranges](https://docs.databricks.com/resources/supported-regions.html) for your region. + +3. **If ELB is internal**, set up VPC peering: + + ```bash + # Create VPC peering between Databricks VPC and Korvet VPC + aws ec2 create-vpc-peering-connection \ + --vpc-id \ + --peer-vpc-id + ``` + + Then update route tables and security groups to allow traffic. + +4. **Configure Korvet's advertised address** to use the ELB hostname: + + ```bash + export KORVET_BROKER_ADVERTISED_HOST= + export KORVET_BROKER_ADVERTISED_PORT=9092 + ``` + +**Verification**: + +After fixing network connectivity, test from Databricks: + +```python +# Test connectivity +import socket +socket.create_connection(("", 9092), timeout=10) + +# Test Spark Structured Streaming +df = spark.readStream \ + .format("kafka") \ + .option("kafka.bootstrap.servers", ":9092") \ + .option("subscribe", "") \ + .option("startingOffsets", "latest") \ + .load() + +df.printSchema() # Should not timeout +``` + +**Additional Notes**: + +- Producers may work while consumers fail because producers don't need to call `describeTopics()` during initialization +- The error occurs when Spark's `KafkaAdminClient` tries to describe topics but cannot connect to the broker +- This is a network connectivity issue, not a Korvet bug or configuration issue + +### Databricks Consumer Not Receiving Messages + +**Symptoms**: Databricks Spark Structured Streaming consumer connects successfully but receives no messages, even with `startingOffsets = "earliest"`. Korvet logs show only metadata and ListOffsets requests, but no Fetch requests. + +**Root Cause**: Databricks has saved checkpoint data from a previous run that contains committed offsets. The consumer resumes from these saved offsets instead of starting from "earliest". + +**Diagnosis**: + +1. Check Korvet logs for Fetch requests: + + ```bash + # Look for FetchHandler log entries + grep "Handling fetch" korvet.log + ``` + + If you only see `MetadataHandler` and `ListOffsetsHandler` entries but no `FetchHandler` entries, the consumer is not fetching. + +2. Verify checkpoint location in Databricks: + + ```python + # Check if checkpoint directory exists + dbutils.fs.ls("/path/to/checkpoint/location") + ``` + +**Solutions**: + +**Option 1: Clear checkpoint data (recommended for testing)** + +Delete the checkpoint directory to force the consumer to start fresh: + +```python +# Remove checkpoint directory +dbutils.fs.rm("/path/to/checkpoint/location", recurse=True) + +# Restart the streaming query +df = spark.readStream \ + .format("kafka") \ + .option("kafka.bootstrap.servers", ":9092") \ + .option("subscribe", "") \ + .option("startingOffsets", "earliest") \ + .load() +``` + +**Option 2: Use a different checkpoint location** + +Specify a new checkpoint location for the streaming query: + +```python +query = df.writeStream \ + .format("console") \ + .option("checkpointLocation", "/new/checkpoint/location") \ + .start() +``` + +**Option 3: Reset offsets in Korvet (if using consumer groups)** + +If using consumer groups, reset the committed offsets: + +```bash +# Use kafka-consumer-groups tool to reset offsets +kafka-consumer-groups --bootstrap-server :9092 \ + --group \ + --topic \ + --reset-offsets --to-earliest \ + --execute +``` + +**Verification**: + +After clearing checkpoints, verify messages are being consumed: + +```python +# Start streaming query with console output +query = df.writeStream \ + .format("console") \ + .option("truncate", False) \ + .start() + +# Check Korvet logs for Fetch requests +# You should now see: +# DEBUG c.r.korvet.broker.kafka.FetchHandler - Handling fetch: topics=1, partitions=... +``` + +**Additional Notes**: + +- Databricks Structured Streaming automatically saves checkpoint data to ensure exactly-once processing +- The `startingOffsets` option only applies when there is no checkpoint data +- Once checkpoint data exists, the consumer always resumes from the saved offsets +- This is expected Spark Structured Streaming behavior, not a Korvet issue +- For production use, manage checkpoint locations carefully to avoid data loss + +## Getting Help + +If you can't resolve the issue: + +1. Check logs with debug level enabled +2. Collect metrics from `/actuator/prometheus` +3. File an issue at https://github.com/redis-field-engineering/korvet-dist/issues + +## Next Steps + +- [Monitoring]({{< relref "/integrate/korvet/operations/monitoring" >}}) +- [Logging]({{< relref "/integrate/korvet/operations/logging" >}}) +- [FAQ]({{< relref "/integrate/korvet/reference/faq" >}}) diff --git a/content/integrate/korvet/operations/web-ui.md b/content/integrate/korvet/operations/web-ui.md new file mode 100644 index 0000000000..40497805ea --- /dev/null +++ b/content/integrate/korvet/operations/web-ui.md @@ -0,0 +1,151 @@ +--- +Title: Web UI +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Korvet includes a web-based administration console for managing topics, + monitoring consumer groups, and inspecting storage health. +linkTitle: Web UI +weight: 50 +--- + +Korvet includes a web-based administration console for managing topics, monitoring consumer groups, and inspecting storage health. + +## Accessing the Web UI + +The web UI is available at `http://localhost:8080` by default. + +```bash +# Start the server +korvet server + +# Open http://localhost:8080 in your browser +# (macOS: open, Linux: xdg-open) +open http://localhost:8080 +``` + +## Authentication + +The web UI uses HTTP Basic authentication. Configure the admin credentials: + +```yaml +korvet: + admin: + username: admin + password: admin + bootstrap: true +``` + +When `bootstrap` is `true` (the default), Korvet creates the configured admin user on startup if it does not already exist. Set it to `false` once you manage admin credentials yourself. + +Change the default password before exposing the UI outside a local development environment. + +For local demos and development, you can disable authentication: + +```bash +korvet server --korvet.admin.security-enabled=false +``` + +When authentication is disabled, the UI skips the login screen and all API endpoints are publicly accessible. + +{{< warning >}} +Only disable authentication in trusted local environments. The `korvet demo` command uses this mode but binds to loopback (`127.0.0.1`) to keep the unauthenticated UI and broker accessible only locally. +{{< /warning >}} + +## Dashboard Pages + +### Topics + +Browse and manage Kafka topics: + +- List all topics with partition counts +- View topic configuration (retention policies, segment settings) +- Inspect messages and their content +- Monitor topic metrics and throughput + +### Consumer Groups + +Monitor consumer group health and activity: + +- List all consumer groups +- View group members and their partition assignments +- Track committed offsets and consumer lag +- Monitor consumption progress per partition + +### Brokers + +View broker health and configuration: + +- Broker status and connectivity +- Kafka listener addresses +- Version and build information +- Cluster metadata + +### Storage + +The Storage Control Plane provides comprehensive visibility into tiered storage health and operations across four tabs: + +#### Redis Streams (Local Tier) + +Monitor the local Redis Streams storage: + +- Redis connectivity, latency, and health metrics +- Total key count and memory usage +- Per-topic/partition stream health +- Latency breakdown by Redis command +- Connection pool statistics + +#### Remote Storage + +Track the Iceberg cold-tier archival: + +- Remote storage enabled status +- Topics using remote tier +- Offload throughput and backlog +- Per-topic archival statistics + +#### Segments & Retention + +Review retention policies and segment management: + +- Local and total retention policies per topic (time and size) +- Segment rolling configuration (`segment.ms`, `segment.bytes`) +- Policy inheritance and overrides + +#### Offload Jobs + +Monitor segment archival jobs: + +{{< note >}} +The Offload Jobs tab displays placeholder data in the current release. Full job tracking will be added in a future version. +{{< /note >}} + +### Security + +Manage Kafka SASL credentials for client authentication: + +- Create, update, and delete credentials +- Support for SCRAM-SHA-256 and PLAIN mechanisms +- Credential rotation and password management + +## Configuration + +Customize the web UI port and binding: + +```yaml +server: + port: 8080 + address: 0.0.0.0 # Bind to all interfaces (use 127.0.0.1 for local-only) + +korvet: + admin: + security-enabled: true # Set to false to disable authentication +``` + +## Next Steps + +- [Admin API]({{< relref "/integrate/korvet/operations/admin-api" >}}) for programmatic access +- [Monitoring]({{< relref "/integrate/korvet/operations/monitoring" >}}) for metrics and health checks +- [Authentication]({{< relref "/integrate/korvet/operations/authentication" >}}) for production security diff --git a/content/integrate/korvet/quick-start/_index.md b/content/integrate/korvet/quick-start/_index.md new file mode 100644 index 0000000000..b73dc255a3 --- /dev/null +++ b/content/integrate/korvet/quick-start/_index.md @@ -0,0 +1,47 @@ +--- +Title: Get Started +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Install Korvet, run the built-in demo, and connect your own Kafka client. +hideListLinks: false +linkTitle: Get Started +weight: 10 +--- + +The fastest way to see Korvet in action is the built-in demo: +one command starts everything and streams live data you can explore in the web +UI. From there, the "Hello World" walkthrough connects your own Kafka client. + +We recommend going through this section in order: + +1. [**Install**]({{< relref "/integrate/korvet/quick-start/install" >}}) Korvet. +2. [**Run the demo**]({{< relref "/integrate/korvet/quick-start/demo" >}}) to see the broker, web UI, tiered + storage, and a live producer/consumer working together with zero configuration. +3. [**Hello World**]({{< relref "/integrate/korvet/quick-start/hello-world" >}}) to create a topic and produce + and consume your first records with a standard Kafka client. + +## Quickest path + +If you just want to see it run and you have Docker available: + +```bash +korvet demo +``` + +This resolves a module-enabled Redis (reusing one that is already running, +otherwise starting a local `redis-server`, otherwise a `redis:8` Docker +container), starts the Korvet broker and web UI, creates a +tiered demo topic, and streams synthetic e-commerce events while a consumer +group reads them. Everything is torn down cleanly on `Ctrl-C`. + +See [Run the demo]({{< relref "/integrate/korvet/quick-start/demo" >}}) for the full walkthrough. + +## Next steps + +- [Running with Docker]({{< relref "/integrate/korvet/quick-start/docker" >}}) +- [Configuration guide]({{< relref "/integrate/korvet/quick-start/configuration" >}}) +- [Concepts & Architecture]({{< relref "/integrate/korvet/concepts" >}}) +- [Using the Kafka API]({{< relref "/integrate/korvet/kafka-api" >}}) diff --git a/content/integrate/korvet/quick-start/configuration.md b/content/integrate/korvet/quick-start/configuration.md new file mode 100644 index 0000000000..7410e3470d --- /dev/null +++ b/content/integrate/korvet/quick-start/configuration.md @@ -0,0 +1,349 @@ +--- +Title: Configuration +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Configure Korvet using Spring Boot's standard configuration mechanisms. +linkTitle: Configuration +weight: 50 +--- + +Korvet is configured using Spring Boot's standard configuration mechanisms. + +## Configuration Files + +Configuration can be provided via: + +- `application.yml` or `application.properties` +- Environment variables +- Command-line arguments + +The examples on this page use YAML, but every property can equally be set as an environment variable (uppercase, `.` and `-` replaced with `_`) or a command-line argument (`--korvet.broker.port=9092`). See [Environment Variables](#environment-variables) below for the env-var equivalents of each example. + +## Basic Configuration + +```yaml +korvet: + broker: + port: 9092 + host: 0.0.0.0 + redis: + uri: redis://localhost:6379 +``` + +## Redis Configuration + +Configure the Redis connection: + +```yaml +korvet: + redis: + uri: redis://localhost:6379 + username: default + password: ${REDIS_PASSWORD} +``` + +You can also embed credentials in the URI: + +```yaml +korvet: + redis: + uri: redis://username:password@redis.example.com:6379 +``` + +For TLS/SSL connections, use the `rediss://` scheme: + +```yaml +korvet: + redis: + uri: rediss://redis.example.com:6379 + username: default + password: ${REDIS_PASSWORD} +``` + +For clustered Redis deployments: + +```yaml +korvet: + redis: + uri: redis://node1:6379,node2:6379,node3:6379 + cluster: true + username: default + password: ${REDIS_PASSWORD} +``` + +## Topic Configuration + +Configure topic behavior using a list of glob patterns. Patterns are evaluated in declared order and combined first-match-wins per field, so place more specific patterns ahead of a `*` catch-all. + +```yaml +korvet: + topics: + - name: "order.*" # Override defaults for topics matching this pattern + retention-time: 1h + - name: "*" # Catch-all defaults + auto-create: false # Automatically create matching topics when they don't exist + partitions: 1 # Default number of partitions + retention-time: 7d # Default retention +``` + +When auto-create is disabled, topics must be created explicitly using the Kafka AdminClient or command-line tools before they can be used. + +As environment variables, the list entries are addressed by index in declared order: the example above translates to `KORVET_TOPICS_0_NAME='order.*'`, `KORVET_TOPICS_0_RETENTION_TIME=1h`, `KORVET_TOPICS_1_NAME='*'`, `KORVET_TOPICS_1_AUTO_CREATE=false`, and so on. + +## Offset Encoding Configuration + +Korvet uses a stateless encoding scheme to convert Redis Stream entry IDs (timestamp-sequence pairs) into Kafka offsets. +The `offset-sequence-bits` setting controls how many bits are allocated for the sequence number portion of the offset. + +```yaml +korvet: + topics: + - name: "*" + offset-sequence-bits: 14 # Default: 14 bits +``` + +### Understanding Offset Encoding + +Redis Stream entry IDs have the format `{timestamp}-{sequence}` (e.g., `1732896000000-0`). +Korvet encodes these into Kafka offsets using bit-packing: + +``` +Kafka offset = (timestamp << offset-sequence-bits) | sequence +``` + +### When to Change This Setting + +The default value of **14 bits** is suitable for most use cases, supporting up to **~16 million messages/second** per partition. + +You should consider changing this setting if: + +| Scenario | Recommended Value | Maximum Throughput | +|---|---|---| +| **High throughput** (up to 16M msg/sec) | `14` (default) | ~16 million messages/second | +| **Very high throughput** (>16M msg/sec) | `16` | ~65 million messages/second | +| **Standard throughput** (up to 1M msg/sec, smaller offset deltas) | `10` | ~1 million messages/second | +| **Low throughput** (<100K msg/sec, smallest offset deltas) | `8` | ~256,000 messages/second | + +### Trade-offs + +**Lower values (8-10 bits):** + +- ✅ Smaller offset deltas between messages +- ✅ Better compatibility with Kafka clients that have offset delta limits +- ❌ Lower maximum throughput per partition + +**Higher values (14-16 bits):** + +- ✅ Higher maximum throughput per partition +- ❌ Larger offset deltas between messages from different milliseconds +- ❌ May exceed Kafka's maximum offset delta limit (Integer.MAX_VALUE) for messages far apart in time + +### Maximum Offset Delta Constraint + +Kafka has a constraint that the offset delta between consecutive messages in a fetch response cannot exceed `Integer.MAX_VALUE` (2,147,483,647). + +With different sequence bit settings, messages from different milliseconds have different offset deltas: + +| Sequence Bits | Offset Delta/ms | Max Time Span | +|---|---|---| +| 8 bits | 256 | ~97 days | +| 10 bits | 1,024 | ~24 days | +| 14 bits (default) | 16,384 | ~36 hours | +| 16 bits | 65,536 | ~9 hours | + +{{< warning >}} +At the default 14 bits (or higher), ensure your consumers fetch messages regularly to avoid exceeding the offset delta limit when messages span more than a few hours. +{{< /warning >}} + +### Example Configurations + +For a high-throughput logging system: + +```yaml +korvet: + topics: + - name: "*" + offset-sequence-bits: 14 # Support up to ~16 million messages/second +``` + +For a low-throughput event system with long retention: + +```yaml +korvet: + topics: + - name: "*" + offset-sequence-bits: 8 # Lower offset delta, supports longer time spans (~256K msg/sec max) +``` + +## Consumer Group Configuration + +Configure consumer group rebalancing behavior: + +```yaml +korvet: + broker: + rebalance-delay: 3s # Delay before completing rebalance (default: 3s) +``` + +The `rebalance-delay` setting controls how long Korvet waits before completing a consumer group rebalance to allow more members to join. +Increase this value for larger consumer groups, especially in Kubernetes environments where pods may start at different times. + +## TLS Configuration + +Enable TLS for the Kafka protocol endpoint: + +```yaml +korvet: + broker: + tls: true + cert-file: /path/to/server.crt + key-file: /path/to/server.key + key-password: ${KEY_PASSWORD} +``` + +## Remote Storage Configuration + +Enable tiered storage to archive sealed segments as Parquet files on S3. The cold tier is configured with `korvet.storage.remote.path`; S3 connection options live under `korvet.storage.remote.s3`. + +```yaml +korvet: + storage: + remote: + path: s3://my-bucket/korvet + s3: + region: us-east-1 +``` + +{{< note >}} +`korvet.storage.remote.path` configures the remote tier. The leader-locked storage worker is enabled by default and rolls eligible segments, offloads sealed segments, and enforces local and remote retention. Topics are archived only when they also have `remote.storage.enable=true`. +{{< /note >}} + +For static credentials: + +```yaml +korvet: + storage: + remote: + path: s3://my-bucket/korvet + s3: + region: us-east-1 + access-key-id: ${AWS_ACCESS_KEY_ID} + secret-access-key: ${AWS_SECRET_ACCESS_KEY} +``` + +For LocalStack or MinIO: + +```yaml +korvet: + storage: + remote: + path: s3://my-bucket/korvet + s3: + region: us-east-1 + endpoint: http://localhost:4566 + path-style-access: true + access-key-id: test + secret-access-key: test +``` + +{{< note >}} +Remote storage is optional. If `korvet.storage.remote.path` is not set, all reads are served from Redis Streams. +{{< /note >}} + +See [Remote Storage]({{< relref "/integrate/korvet/storage/remote-storage" >}}) for complete configuration options and per-topic retention settings. + +## Redis Metrics Configuration + +Korvet can optionally enable Lettuce command latency metrics to track Redis operation performance. + +{{< note >}} +These metrics are disabled by default to minimize overhead. Enable them when you need detailed Redis performance monitoring. +{{< /note >}} + +```yaml +korvet: + redis: + metrics: + enabled: true # Enable Lettuce command metrics (default: false) + histogram: false # Enable histogram buckets for percentiles (default: false) + local-distinction: false # Track per connection vs per host (default: false) + max-latency: 5m # Maximum expected latency (default: 5 minutes) + min-latency: 1ms # Minimum expected latency (default: 1 millisecond) +``` + +**Configuration Properties**: + +- `enabled`: Enable Lettuce command latency metrics (default: `false`) +- `histogram`: Enable histogram buckets for aggregable percentile approximations (default: `false`) +- `local-distinction`: Track metrics per connection instead of per host/port (default: `false`) +- `max-latency`: Maximum expected latency for histogram buckets (default: `5m`, only applies when `histogram` is enabled) +- `min-latency`: Minimum expected latency for histogram buckets (default: `1ms`, only applies when `histogram` is enabled) + +When enabled, Lettuce will publish two timer metrics: + +- `lettuce.command.firstresponse` - Time to first response from Redis +- `lettuce.command.completion` - Time to complete Redis command + +Each metric includes tags: `command` (e.g., `XADD`, `XREAD`), `local` (if `local-distinction` enabled), and `remote` (Redis server address). + +For more details, see [Lettuce Redis Client Metrics]({{< relref "/integrate/korvet/reference/metrics/redis-client" >}}). + +## Environment Variables + +All configuration can be set via environment variables using Spring Boot's relaxed binding. +Property paths are converted to uppercase with underscores: + +```bash +# Broker configuration +export KORVET_BROKER_HOST=0.0.0.0 +export KORVET_BROKER_PORT=9092 +export KORVET_BROKER_ID=0 + +# Consumer groups +export KORVET_BROKER_REBALANCE_DELAY=3s + +# TLS for the Kafka endpoint +export KORVET_BROKER_TLS=true +export KORVET_BROKER_CERT_FILE=/path/to/server.crt +export KORVET_BROKER_KEY_FILE=/path/to/server.key +export KORVET_BROKER_KEY_PASSWORD=secret + +# Redis configuration +export KORVET_REDIS_URI=redis://redis.example.com:6379 +export KORVET_REDIS_USERNAME=default +export KORVET_REDIS_PASSWORD=secret +export KORVET_REDIS_CLUSTER=false + +# Redis metrics (optional) +export KORVET_REDIS_METRICS_ENABLED=false +export KORVET_REDIS_METRICS_HISTOGRAM=false +export KORVET_REDIS_METRICS_LOCAL_DISTINCTION=false +export KORVET_REDIS_METRICS_MAX_LATENCY=5m +export KORVET_REDIS_METRICS_MIN_LATENCY=1ms + +# Topic configuration (pattern-based; entries are addressed by index, +# evaluated in order — index 0 is the catch-all here) +export KORVET_TOPICS_0_NAME='*' +export KORVET_TOPICS_0_AUTO_CREATE=true +export KORVET_TOPICS_0_PARTITIONS=1 +export KORVET_TOPICS_0_RETENTION_TIME=7d +export KORVET_TOPICS_0_OFFSET_SEQUENCE_BITS=14 + +# Remote storage configuration (optional) +export KORVET_STORAGE_REMOTE_PATH=s3://my-bucket/korvet +export KORVET_STORAGE_REMOTE_S3_REGION=us-east-1 +# For static credentials, also set: +# export KORVET_STORAGE_REMOTE_S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE +# export KORVET_STORAGE_REMOTE_S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +# For S3-compatible stores such as LocalStack or MinIO, also set: +# export KORVET_STORAGE_REMOTE_S3_ENDPOINT=http://localhost:4566 +# export KORVET_STORAGE_REMOTE_S3_PATH_STYLE_ACCESS=true +``` + +## Next Steps + +- [Complete configuration reference]({{< relref "/integrate/korvet/reference/configuration" >}}) +- [Deployment guide]({{< relref "/integrate/korvet/operations/deployment" >}}) diff --git a/content/integrate/korvet/quick-start/demo.md b/content/integrate/korvet/quick-start/demo.md new file mode 100644 index 0000000000..e019881c43 --- /dev/null +++ b/content/integrate/korvet/quick-start/demo.md @@ -0,0 +1,96 @@ +--- +Title: Run the Demo +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: The korvet demo command starts a complete, self-contained Korvet showcase + with a single command. +linkTitle: Run the Demo +weight: 20 +--- + +The `korvet demo` command starts a complete, self-contained Korvet +showcase with a single command. It is the fastest way to see how the broker, +web UI, and tiered storage work together. + +## Prerequisites + +- [Install Korvet]({{< relref "/integrate/korvet/quick-start/install" >}}) — on macOS and + Linux, `brew install redis/tap/korvet`. +- A way for the demo to reach a **module-enabled** Redis 8+ — one with the + RediSearch (`FT.*`) and RedisJSON (`JSON.*`) commands. Any of: a module-enabled + Redis already running, a module-enabled `redis-server` binary on your `PATH` + (the Redis Open Source cask provides one; see + [Install]({{< relref "/integrate/korvet/quick-start/install" >}})), or Docker. The demo tries them in that + order. + +{{< note >}} +The Homebrew **core** `redis` formula ships without those modules, so a +`redis-server` from it is not sufficient — the demo would skip it and fall back +to Docker. For a Docker-free run, install the module-enabled cask +(`brew install --cask redis`). +{{< /note >}} + +## Start the demo + +```bash +brew install redis/tap/korvet +korvet demo +``` + +This single command: + +1. **Resolves Redis** — reuses a module-enabled Redis already running at + `redis://localhost:6379`, otherwise starts a throwaway local `redis-server` + (only if it is module-enabled), otherwise falls back to a `redis:8` Docker + container. +2. **Starts the broker and web UI** in-process, pointed at a local-filesystem + Iceberg cold tier under `/tmp/korvet-demo`. +3. **Creates a tiered demo topic** (`events`, 3 partitions) with a short + `segment.ms` so data reaches the cold tier within seconds. +4. **Streams synthetic JSON e-commerce events** into the topic while a consumer + group of two members reads them. + +On an interactive terminal the demo shows a full-screen live dashboard. When +output is piped or `--verbose` is set, it prints a plain walkthrough and streams +logs to the console instead. + +## What you'll see + +| | | +|---|---| +| Kafka bootstrap | `localhost:9092` — connect any Kafka client or app here | +| Web UI | `http://localhost:8080` — explore topics, messages, consumer groups, and lag | +| Iceberg cold tier | `/tmp/korvet-demo` — watch the warehouse fill as segments offload | + +In the web UI, explore: + +- The `events` topic and its JSON messages. +- The `demo-consumers` consumer group — its members, partition assignment, + committed offsets, and lag. +- The cold-tier warehouse filling up as sealed segments offload to Iceberg. + +## Useful options + +| | | +|---|---| +| `--records ` | Produce a fixed number of events, or `0` to stream continuously (default). | +| `--rate ` | Approximate produce rate. Default: `50`. | +| `--timeout ` | Auto-shutdown after a duration, e.g. `10m`. | +| `--kafka-port` / `--http-port` | Override the broker and UI ports (default `9092` / `8080`). | +| `--redis-uri ` | Reuse a Redis already running at this URI. | +| `--data-dir

` | Directory for Redis data and the Iceberg warehouse. Default: `/tmp/korvet-demo` (reset each run). | +| `--verbose` | Stream full broker and Kafka-client logs instead of the live dashboard. | + +## Stop the demo + +Press `Ctrl-C`. Redis, the broker, and the web UI all shut down cleanly. + +## Next steps + +- [Hello World]({{< relref "/integrate/korvet/quick-start/hello-world" >}}) — create a topic and produce + and consume records with your own Kafka client. +- [Concepts & Architecture]({{< relref "/integrate/korvet/concepts" >}}) +- [Tiered Storage]({{< relref "/integrate/korvet/storage" >}}) diff --git a/content/integrate/korvet/quick-start/docker.md b/content/integrate/korvet/quick-start/docker.md new file mode 100644 index 0000000000..31d183d42c --- /dev/null +++ b/content/integrate/korvet/quick-start/docker.md @@ -0,0 +1,147 @@ +--- +Title: Running with Docker +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: How to run Korvet using Docker, including Docker Compose and JVM tuning. +linkTitle: Running with Docker +weight: 40 +--- + +This guide shows how to run Korvet using Docker. + +## Quick Start + +Run Korvet with default configuration: + +```bash +docker run -p 9092:9092 redisfield/korvet:latest server +``` + +This starts Korvet on port 9092. An external Redis instance is required; the image does not include one. See [Using External Redis](#using-external-redis) to point Korvet at your Redis. + +The image also includes the Korvet operational CLI. Pass a command as the container argument: + +```bash +docker run --rm redisfield/korvet:latest topics --bootstrap-server host.docker.internal:9092 --list +docker run --rm redisfield/korvet:latest migrate -u redis://host.docker.internal:6379 +``` + +## Using External Redis + +To use an external Redis instance: + +```bash +docker run -p 9092:9092 \ + -e KORVET_REDIS_URI=redis://redis.example.com:6379 \ + redisfield/korvet:latest server +``` + +With authentication: + +```bash +docker run -p 9092:9092 \ + -e KORVET_REDIS_URI=redis://redis.example.com:6379 \ + -e KORVET_REDIS_USERNAME=default \ + -e KORVET_REDIS_PASSWORD=${REDIS_PASSWORD} \ + redisfield/korvet:latest server +``` + +## Docker Compose + +Create a `docker-compose.yml` file: + +```yaml +services: + redis: + image: redis:8.6 + ports: + - "6379:6379" + + korvet: + image: redisfield/korvet:latest + command: server + ports: + - "9092:9092" + environment: + KORVET_REDIS_URI: redis://redis:6379 + depends_on: + - redis +``` + +With Redis authentication: + +```yaml +services: + redis: + image: redis:8.6 + command: redis-server --requirepass ${REDIS_PASSWORD} + ports: + - "6379:6379" + + korvet: + image: redisfield/korvet:latest + command: server + ports: + - "9092:9092" + environment: + KORVET_REDIS_URI: redis://redis:6379 + KORVET_REDIS_PASSWORD: ${REDIS_PASSWORD} + depends_on: + - redis +``` + +Run with: + +```bash +docker compose up +``` + +## Configuration + +See [Configuration]({{< relref "/integrate/korvet/quick-start/configuration" >}}) for all available options. + +## JVM Tuning + +The Docker image accepts JVM options through the `JAVA_OPTS` environment variable. +Use this to increase heap size or apply additional JVM tuning flags. + +For example, to run with a 2 GiB heap: + +```bash +docker run -p 9092:9092 \ + -e JAVA_OPTS="-Xms2g -Xmx2g" \ + redisfield/korvet:latest server +``` + +### Direct Memory for High Concurrency + +The broker defaults to `-XX:MaxDirectMemorySize=512m`. For production deployments with many +concurrent connections (e.g., Databricks Spark, Flink), raise it further. Set it via `JAVA_OPTS` +(not `JAVA_TOOL_OPTIONS`, which the baked default overrides), and keep the container memory limit +above `-Xmx + MaxDirectMemorySize` plus native overhead: + +```bash +docker run -p 9092:9092 \ + -e JAVA_OPTS="-Xms2g -Xmx2g -XX:MaxDirectMemorySize=512m" \ + redisfield/korvet:latest server +``` + +Or in Docker Compose: + +```yaml +services: + korvet: + image: redisfield/korvet:latest + command: server + environment: + JAVA_OPTS: "-Xms2g -Xmx2g -XX:MaxDirectMemorySize=512m" + mem_limit: 4g +``` + +## Next Steps + +- [Using the Kafka API]({{< relref "/integrate/korvet/kafka-api" >}}) +- [Monitoring]({{< relref "/integrate/korvet/operations/monitoring" >}}) diff --git a/content/integrate/korvet/quick-start/hello-world.md b/content/integrate/korvet/quick-start/hello-world.md new file mode 100644 index 0000000000..57591faaf9 --- /dev/null +++ b/content/integrate/korvet/quick-start/hello-world.md @@ -0,0 +1,90 @@ +--- +Title: '"Hello World" for Apache Kafka' +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Start a Korvet broker, create a topic, and produce and consume your + first records with standard Kafka tooling. +linkTitle: Hello World +weight: 30 +--- + +This walkthrough starts a Korvet broker against your own Redis, +creates a topic, and produces and consumes your first records with standard +Kafka tooling. + +If you just want to watch Korvet run with data already flowing, +use [the demo]({{< relref "/integrate/korvet/quick-start/demo" >}}) instead — it provisions everything for +you. This page is the bring-your-own-client baseline. + +## Prerequisites + +- [Install Korvet]({{< relref "/integrate/korvet/quick-start/install" >}}) — on macOS and + Linux, `brew install redis/tap/korvet`. +- A module-enabled Redis 8+. The quickest option: + + ```bash + docker run -d --name korvet-redis -p 6379:6379 redis:8 + ``` + +- A Kafka client. The examples below use the `kafka-console-*` tools that ship + with Apache Kafka. + +## Step 1 — Start the broker + +```bash +korvet server +``` + +The broker starts on `localhost:9092` and connects to Redis at +`redis://localhost:6379` by default. To run it in a container instead: + +```bash +docker run -p 9092:9092 redisfield/korvet:latest server +``` + +## Step 2 — Create a topic + +Use the bundled CLI: + +```bash +korvet topics --create --bootstrap-server localhost:9092 --topic helloworld --partitions 3 +``` + +Confirm it was created: + +```bash +korvet topics --list --bootstrap-server localhost:9092 +``` + +## Step 3 — Produce records + +Korvet speaks the Kafka protocol, so any Kafka producer works: + +```bash +echo "hello world" | kafka-console-producer --bootstrap-server localhost:9092 --topic helloworld +``` + +## Step 4 — Consume records + +```bash +kafka-console-consumer --bootstrap-server localhost:9092 --topic helloworld --from-beginning +``` + +You should see `hello world` printed back. Press `Ctrl-C` to stop the consumer. + +## What you accomplished + +You started a Korvet broker backed by Redis, created a topic, +and produced and consumed records through it — using nothing but the standard +Kafka API. Any Kafka client, framework, or connector that targets +`localhost:9092` works the same way. + +## Next steps + +- [Using the Kafka API]({{< relref "/integrate/korvet/kafka-api" >}}) — produce, consume, and manage topics. +- [Schema Registry]({{< relref "/integrate/korvet/kafka-api/schema-registry" >}}) +- [Tiered Storage]({{< relref "/integrate/korvet/storage" >}}) +- [Deploying to production]({{< relref "/integrate/korvet/operations/deployment" >}}) diff --git a/content/integrate/korvet/quick-start/install.md b/content/integrate/korvet/quick-start/install.md new file mode 100644 index 0000000000..726a751f00 --- /dev/null +++ b/content/integrate/korvet/quick-start/install.md @@ -0,0 +1,100 @@ +--- +Title: Installation +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Different ways to install Korvet, including Homebrew, Docker, and the + distribution package. +linkTitle: Installation +weight: 10 +--- + +This guide covers different ways to install Korvet. + +## Homebrew + +On macOS and Linux, the easiest way to install Korvet is with +[Homebrew](https://brew.sh) from the Redis tap (this also installs `openjdk` for +the Java runtime): + +```bash +brew install redis/tap/korvet +``` + +Korvet connects to a Redis you provide — it does not bundle one. +It needs a module-enabled Redis 8.x with the RediSearch (`FT.*`) and +RedisJSON (`JSON.*`) commands, such as [Redis Cloud](https://redis.io/cloud/), Redis +Enterprise, the official `redis:8` Docker image, or the module-enabled Redis +Open Source cask: + +```bash +brew tap redis/redis +brew install --cask redis +``` + +{{< note >}} +The Homebrew **core** `redis` formula ships without those modules, so it is +not sufficient. +{{< /note >}} + +Once installed, point Korvet at your Redis and run it: + +```bash +korvet server +``` + +Or jump straight to the [demo]({{< relref "/integrate/korvet/quick-start/demo" >}}). + +## Docker + +```bash +docker run -p 9092:9092 redisfield/korvet:latest server +``` + +For configuration and production options, see [Docker deployment]({{< relref "/integrate/korvet/quick-start/docker" >}}). + +## Distribution Package + +Download the distribution package (`.tar` or `.zip`) from [Korvet Releases](https://github.com/redis-field-engineering/korvet-dist/releases). + +### Extract the Archive + +```bash +# For .tar +tar -xf korvet-.tar + +# For .zip +unzip korvet-.zip +``` + +### Run Korvet + +The distribution includes a startup script with all required JVM options: + +```bash +cd korvet- +./bin/korvet server +``` + +The same script also includes operational commands: + +```bash +./bin/korvet topics --bootstrap-server localhost:9092 --list +./bin/korvet migrate -u redis://localhost:6379 +``` + +Replace `` with the version you downloaded. This documentation is for `0.19`. + +## System Requirements + +- **Java**: 25 or later +- **Redis**: 8+ (the JSON and Search capabilities must be available; on Redis Enterprise, create the database with the JSON and Search modules enabled) +- **Memory**: Minimum 512MB RAM (2GB+ recommended for production) +- **Storage**: Depends on message volume and retention policy + +## Next Steps + +- [Running with Docker]({{< relref "/integrate/korvet/quick-start/docker" >}}) +- [Configuration guide]({{< relref "/integrate/korvet/quick-start/configuration" >}}) diff --git a/content/integrate/korvet/reference/api.md b/content/integrate/korvet/reference/api.md new file mode 100644 index 0000000000..e02ccdf49a --- /dev/null +++ b/content/integrate/korvet/reference/api.md @@ -0,0 +1,192 @@ +--- +Title: API Reference +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Documents the Kafka protocol APIs and Actuator REST endpoints supported + by Korvet. +linkTitle: API +weight: 20 +--- + +Korvet implements the Kafka protocol. This page documents the supported APIs. + +## Kafka Protocol APIs + +### Produce API + +Send messages to topics. + +**Request**: + +``` +ProduceRequest { + topic: string + partition: int32 + messages: [ + { + key: bytes + value: bytes + headers: [(string, bytes)] + timestamp: int64 + } + ] +} +``` + +**Response**: + +``` +ProduceResponse { + topic: string + partition: int32 + offset: int64 + timestamp: int64 +} +``` + +### Fetch API + +Read messages from topics. + +**Request**: + +``` +FetchRequest { + topics: [ + { + topic: string + partitions: [ + { + partition: int32 + offset: int64 + max_bytes: int32 + } + ] + } + ] + max_wait_ms: int32 + min_bytes: int32 +} +``` + +**Response**: + +``` +FetchResponse { + topics: [ + { + topic: string + partitions: [ + { + partition: int32 + high_watermark: int64 + messages: [ + { + offset: int64 + key: bytes + value: bytes + headers: [(string, bytes)] + timestamp: int64 + } + ] + } + ] + } + ] +} +``` + +### Metadata API + +Get topic and partition information. + +**Request**: + +``` +MetadataRequest { + topics: [string] # Empty for all topics +} +``` + +**Response**: + +``` +MetadataResponse { + brokers: [ + { + node_id: int32 + host: string + port: int32 + } + ] + topics: [ + { + name: string + partitions: [ + { + partition: int32 + leader: int32 + } + ] + } + ] +} +``` + +## REST API (Actuator) + +Spring Boot Actuator endpoints for monitoring and management. + +### Health Check + +```bash +GET /actuator/health +``` + +**Response**: + +```json +{ + "status": "UP", + "components": { + "redis": { + "status": "UP" + } + } +} +``` + +### Metrics + +```bash +GET /actuator/prometheus +``` + +Returns Prometheus-formatted metrics. + +### Info + +```bash +GET /actuator/info +``` + +**Response**: + +```json +{ + "build": { + "group": "com.redis", + "artifact": "korvet-server", + "version": "1.0.0", + "time": "2026-01-01T00:00:00.000Z" + } +} +``` + +## Next Steps + +- [Kafka API guide]({{< relref "/integrate/korvet/kafka-api" >}}) +- [Metrics reference]({{< relref "/integrate/korvet/reference/metrics" >}}) diff --git a/content/integrate/korvet/reference/configuration.md b/content/integrate/korvet/reference/configuration.md new file mode 100644 index 0000000000..4365d97aa4 --- /dev/null +++ b/content/integrate/korvet/reference/configuration.md @@ -0,0 +1,696 @@ +--- +Title: Configuration Reference +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Complete configuration reference for Korvet. +linkTitle: Configuration +weight: 10 +--- + +Complete configuration reference for Korvet. + +All `korvet` properties have sensible defaults. You can override them via +`application.yml`, environment variables (e.g. `KORVET_BROKER_PORT`), or +command-line arguments (e.g. `--korvet.broker.port=9092`). Unknown `korvet` +properties fail startup (strict binding) — a typo is caught immediately rather +than silently ignored. + +## General + +Top-level Korvet runtime settings: namespace and the entry points into each sub-system. + +#### `korvet.namespace` {#korvet-namespace} + +**Env var:** `KORVET_NAMESPACE` · **Type:** string · **Default:** `korvet` + +Logical namespace applied to all storage and registry state this Korvet instance owns. Must not contain whitespace or the `:` delimiter — colons are appended internally to compose sub-namespaces. + +## Admin + +HTTP Admin API bootstrap credentials. + +#### `korvet.admin.bootstrap` {#korvet-admin-bootstrap} + +**Env var:** `KORVET_ADMIN_BOOTSTRAP` · **Type:** boolean · **Default:** `true` + +Create the default admin credential at startup when it does not already exist. + +#### `korvet.admin.first-run-setup-enabled` {#korvet-admin-first-run-setup-enabled} + +**Env var:** `KORVET_ADMIN_FIRST_RUN_SETUP_ENABLED` · **Type:** boolean · **Default:** `true` + +Enable the unauthenticated first-run setup endpoint that creates the first admin user while the admin user store is empty. + +#### `korvet.admin.password` {#korvet-admin-password} + +**Env var:** `KORVET_ADMIN_PASSWORD` · **Type:** string · **Default:** `admin` + +Password for the bootstrapped admin credential. Change this before exposing the Admin API. + +#### `korvet.admin.security-enabled` {#korvet-admin-security-enabled} + +**Env var:** `KORVET_ADMIN_SECURITY_ENABLED` · **Type:** boolean · **Default:** `true` + +When false, the Admin API and UI are served without authentication: the security filter chains permit every request and the SPA skips its login screen. Intended for local, single-user contexts such as `korvet demo`. Defaults to true. + +#### `korvet.admin.username` {#korvet-admin-username} + +**Env var:** `KORVET_ADMIN_USERNAME` · **Type:** string · **Default:** `admin` + +Username for the bootstrapped admin credential. + +### Jwt + +JWT session settings for the Admin API cookie auth. + +#### `korvet.admin.jwt.expiry` {#korvet-admin-jwt-expiry} + +**Env var:** `KORVET_ADMIN_JWT_EXPIRY` · **Type:** duration · **Default:** `8h` + +Lifetime of an admin session JWT. Defaults to 8 hours. + +#### `korvet.admin.jwt.secret` {#korvet-admin-jwt-secret} + +**Env var:** `KORVET_ADMIN_JWT_SECRET` · **Type:** string + +HMAC-SHA256 signing secret for admin session JWTs. Must be at least 32 characters. If not set, a random key is generated at startup (all sessions are invalidated on restart). + +## Broker + +Kafka-wire broker listener and group coordinator: bind address, TLS, request limits, backpressure, and rebalance tuning. + +#### `korvet.broker.advertised-host` {#korvet-broker-advertised-host} + +**Env var:** `KORVET_BROKER_ADVERTISED_HOST` · **Type:** string + +Hostname advertised to clients via the Kafka metadata response. Leave unset (null) to fall back to `host`, or to `localhost` when `host` binds to all interfaces (`0.0.0.0`); blank/whitespace values are rejected. + +#### `korvet.broker.advertised-port` {#korvet-broker-advertised-port} + +**Env var:** `KORVET_BROKER_ADVERTISED_PORT` · **Type:** integer + +Port advertised to clients via the Kafka metadata response. Defaults to `port` when unset. + +#### `korvet.broker.boss-threads` {#korvet-broker-boss-threads} + +**Env var:** `KORVET_BROKER_BOSS_THREADS` · **Type:** integer · **Default:** `1` + +Netty boss (accept) thread count. + +#### `korvet.broker.cert-file` {#korvet-broker-cert-file} + +**Env var:** `KORVET_BROKER_CERT_FILE` · **Type:** file path + +PEM-encoded server certificate (or chain). Required when `tls=true`. + +#### `korvet.broker.client-auth-required` {#korvet-broker-client-auth-required} + +**Env var:** `KORVET_BROKER_CLIENT_AUTH_REQUIRED` · **Type:** boolean · **Default:** `false` + +Require clients to present a certificate (mTLS). + +#### `korvet.broker.enabled` {#korvet-broker-enabled} + +**Env var:** `KORVET_BROKER_ENABLED` · **Type:** boolean · **Default:** `true` + +Enable the broker listener. + +#### `korvet.broker.fetch-max-wait` {#korvet-broker-fetch-max-wait} + +**Env var:** `KORVET_BROKER_FETCH_MAX_WAIT` · **Type:** duration · **Default:** `500ms` + +Maximum time a fetch request will block waiting for `fetch.min.bytes` to be satisfied. + +#### `korvet.broker.fetch-partition-max-bytes` {#korvet-broker-fetch-partition-max-bytes} + +**Env var:** `KORVET_BROKER_FETCH_PARTITION_MAX_BYTES` · **Type:** data size · **Default:** `1MB` + +Default upper bound on per-partition bytes returned by a fetch. + +#### `korvet.broker.host` {#korvet-broker-host} + +**Env var:** `KORVET_BROKER_HOST` · **Type:** string · **Default:** `0.0.0.0` + +Listener host (interface to bind to). + +#### `korvet.broker.id` {#korvet-broker-id} + +**Env var:** `KORVET_BROKER_ID` · **Type:** integer · **Default:** `0` + +Numeric broker id advertised to Kafka clients. Must be unique across a multi-node deployment. + +#### `korvet.broker.key-file` {#korvet-broker-key-file} + +**Env var:** `KORVET_BROKER_KEY_FILE` · **Type:** file path + +PEM-encoded server private key. Required when `tls=true`. + +#### `korvet.broker.key-password` {#korvet-broker-key-password} + +**Env var:** `KORVET_BROKER_KEY_PASSWORD` · **Type:** secret + +Passphrase protecting `keyFile`, if encrypted. + +#### `korvet.broker.max-pending-bytes` {#korvet-broker-max-pending-bytes} + +**Env var:** `KORVET_BROKER_MAX_PENDING_BYTES` · **Type:** data size · **Default:** `100MB` + +Backpressure threshold: pause reads from the wire once this many bytes are pending outbound. + +#### `korvet.broker.max-request-bytes` {#korvet-broker-max-request-bytes} + +**Env var:** `KORVET_BROKER_MAX_REQUEST_BYTES` · **Type:** data size · **Default:** `100MB` + +Maximum size of a single inbound Kafka request. Requests larger than this are rejected. + +#### `korvet.broker.port` {#korvet-broker-port} + +**Env var:** `KORVET_BROKER_PORT` · **Type:** integer · **Default:** `9092` + +Listener TCP port. + +#### `korvet.broker.produce-timeout` {#korvet-broker-produce-timeout} + +**Env var:** `KORVET_BROKER_PRODUCE_TIMEOUT` · **Type:** duration · **Default:** `5s` + +Server-side cap on how long a single partition's storage write may run before it is surfaced as `REQUEST_TIMED_OUT`. Because produce responses are sent in request order per connection, a stalled write would otherwise hold the head of the in-order response queue (and every pipelined request behind it) for the client's full `timeoutMs` — collapsing throughput on the connection. The effective bound per partition is the smaller of this value and the client-supplied request timeout. Set to zero to disable the server-side cap. + +#### `korvet.broker.rebalance-delay` {#korvet-broker-rebalance-delay} + +**Env var:** `KORVET_BROKER_REBALANCE_DELAY` · **Type:** duration · **Default:** `3s` + +Grace period before triggering a consumer-group rebalance after a member joins or leaves. + +#### `korvet.broker.rebalance-threads` {#korvet-broker-rebalance-threads} + +**Env var:** `KORVET_BROKER_REBALANCE_THREADS` · **Type:** integer · **Default:** max(2, available CPU cores) + +Scheduler thread-pool size for the group coordinator. + +#### `korvet.broker.response-queue-timeout` {#korvet-broker-response-queue-timeout} + +**Env var:** `KORVET_BROKER_RESPONSE_QUEUE_TIMEOUT` · **Type:** duration · **Default:** `10s` + +Upper bound on how long a single request may hold the head of the per-connection, in-order response queue before it is failed with `REQUEST_TIMED_OUT`. Prevents a request whose handler never completes from stalling every later request on the connection. Fetch requests additionally get their `fetchMaxWait` long-poll budget on top of this. + +#### `korvet.broker.resume-pending-bytes` {#korvet-broker-resume-pending-bytes} + +**Env var:** `KORVET_BROKER_RESUME_PENDING_BYTES` · **Type:** data size · **Default:** `50MB` + +Backpressure release threshold: resume reads once pending bytes drop below this. Must be `<` `maxPendingBytes`. + +#### `korvet.broker.tls` {#korvet-broker-tls} + +**Env var:** `KORVET_BROKER_TLS` · **Type:** boolean · **Default:** `false` + +Enable TLS on the listener. When `true`, `certFile` and `keyFile` are required. + +#### `korvet.broker.trust-cert-file` {#korvet-broker-trust-cert-file} + +**Env var:** `KORVET_BROKER_TRUST_CERT_FILE` · **Type:** file path + +PEM-encoded CA trust store for verifying client certificates (mTLS). Required when `clientAuthRequired=true`. + +#### `korvet.broker.worker-threads` {#korvet-broker-worker-threads} + +**Env var:** `KORVET_BROKER_WORKER_THREADS` · **Type:** integer · **Default:** `0` + +Netty worker (IO) thread count. `0` lets Netty pick a default based on CPU count. + +### Acl + +#### `korvet.broker.acl.enabled` {#korvet-broker-acl-enabled} + +**Env var:** `KORVET_BROKER_ACL_ENABLED` · **Type:** boolean · **Default:** `false` + +Enable topic ACL enforcement. Requires SASL authentication to be enabled. + +### Metrics + +#### `korvet.broker.metrics.offset-cardinality-cap` {#korvet-broker-metrics-offset-cardinality-cap} + +**Env var:** `KORVET_BROKER_METRICS_OFFSET_CARDINALITY_CAP` · **Type:** integer · **Default:** `10000` + +Maximum number of `(topic, partition)` pairs published as offset gauges. When the live topic-partition count exceeds this cap, the publisher skips the refresh and logs a warning to prevent unbounded time-series cardinality. + +#### `korvet.broker.metrics.offset-refresh-interval` {#korvet-broker-metrics-offset-refresh-interval} + +**Env var:** `KORVET_BROKER_METRICS_OFFSET_REFRESH_INTERVAL` · **Type:** duration · **Default:** `15s` + +Interval between refreshes of per-topic-partition offset gauges (`korvet.broker.max_offset`, `korvet.broker.log_start_offset`). + +### Sasl + +#### `korvet.broker.sasl.enabled` {#korvet-broker-sasl-enabled} + +**Env var:** `KORVET_BROKER_SASL_ENABLED` · **Type:** boolean · **Default:** `false` + +Enable SASL authentication on the listener. + +#### `korvet.broker.sasl.mechanisms` {#korvet-broker-sasl-mechanisms} + +**Env var:** `KORVET_BROKER_SASL_MECHANISMS` · **Type:** list of string · **Default:** `SCRAM-SHA-256` + +SASL mechanisms advertised to clients. Must be a non-empty subset of `SUPPORTED`. Defaults to SCRAM-SHA-256 only — PLAIN must be opted in explicitly and requires TLS. + +## Redis + +Primary Redis client used by the broker, registries, and (unless overridden) storage. + +#### `korvet.redis.cluster` {#korvet-redis-cluster} + +**Env var:** `KORVET_REDIS_CLUSTER` · **Type:** boolean · **Default:** `false` + +Treat the target as a Redis Cluster (uses Lettuce `RedisClusterClient`). + +#### `korvet.redis.host` {#korvet-redis-host} + +**Env var:** `KORVET_REDIS_HOST` · **Type:** string · **Default:** `localhost` + +Redis host. Ignored when `uri` is set. + +#### `korvet.redis.io-threads` {#korvet-redis-io-threads} + +**Env var:** `KORVET_REDIS_IO_THREADS` · **Type:** integer · **Default:** available CPU cores + +Size of the Lettuce IO (event-loop) thread pool. + +#### `korvet.redis.password` {#korvet-redis-password} + +**Env var:** `KORVET_REDIS_PASSWORD` · **Type:** secret + +Password for AUTH. + +#### `korvet.redis.port` {#korvet-redis-port} + +**Env var:** `KORVET_REDIS_PORT` · **Type:** integer · **Default:** `6379` + +Redis port. Ignored when `uri` is set. + +#### `korvet.redis.timeout` {#korvet-redis-timeout} + +**Env var:** `KORVET_REDIS_TIMEOUT` · **Type:** duration · **Default:** `1m` + +Default per-command timeout. + +#### `korvet.redis.uri` {#korvet-redis-uri} + +**Env var:** `KORVET_REDIS_URI` · **Type:** string + +Redis URI (`redis://...` or `rediss://...`). When set, supersedes `host`/`port`/`username`/`password`. + +#### `korvet.redis.username` {#korvet-redis-username} + +**Env var:** `KORVET_REDIS_USERNAME` · **Type:** string + +Username for ACL authentication. Leave unset for password-only AUTH. + +### Circuit Breaker + +Fail-fast circuit breaker around Redis stream operations on the primary client. + +#### `korvet.redis.circuit-breaker.enabled` {#korvet-redis-circuit-breaker-enabled} + +**Env var:** `KORVET_REDIS_CIRCUIT_BREAKER_ENABLED` · **Type:** boolean · **Default:** `true` + +Enable the stream-operation circuit breaker. + +#### `korvet.redis.circuit-breaker.log-interval` {#korvet-redis-circuit-breaker-log-interval} + +**Env var:** `KORVET_REDIS_CIRCUIT_BREAKER_LOG_INTERVAL` · **Type:** duration · **Default:** `30s` + +Minimum interval between breaker-state log lines (rate-limit for repeated open events). + +#### `korvet.redis.circuit-breaker.open-duration` {#korvet-redis-circuit-breaker-open-duration} + +**Env var:** `KORVET_REDIS_CIRCUIT_BREAKER_OPEN_DURATION` · **Type:** duration · **Default:** `30s` + +How long the breaker stays open before allowing a probe call. + +### Metrics + +Client-side latency metrics published by Lettuce for the primary Redis client. + +#### `korvet.redis.metrics.enabled` {#korvet-redis-metrics-enabled} + +**Env var:** `KORVET_REDIS_METRICS_ENABLED` · **Type:** boolean · **Default:** `false` + +Enable command-latency metrics collection. + +#### `korvet.redis.metrics.histogram` {#korvet-redis-metrics-histogram} + +**Env var:** `KORVET_REDIS_METRICS_HISTOGRAM` · **Type:** boolean · **Default:** `false` + +Publish per-command histograms in addition to summary statistics. + +#### `korvet.redis.metrics.local-distinction` {#korvet-redis-metrics-local-distinction} + +**Env var:** `KORVET_REDIS_METRICS_LOCAL_DISTINCTION` · **Type:** boolean · **Default:** `false` + +Split metrics by local (client) socket address — useful in pooled deployments. + +#### `korvet.redis.metrics.max-latency` {#korvet-redis-metrics-max-latency} + +**Env var:** `KORVET_REDIS_METRICS_MAX_LATENCY` · **Type:** duration · **Default:** `5m` + +Upper bound used when bucketing latency samples. + +#### `korvet.redis.metrics.min-latency` {#korvet-redis-metrics-min-latency} + +**Env var:** `KORVET_REDIS_METRICS_MIN_LATENCY` · **Type:** duration · **Default:** `1ms` + +Lower bound used when bucketing latency samples. + +### Pool + +Connection-pool sizing for the primary Redis client. + +#### `korvet.redis.pool.max-wait` {#korvet-redis-pool-max-wait} + +**Env var:** `KORVET_REDIS_POOL_MAX_WAIT` · **Type:** duration · **Default:** `3s` + +Maximum time a caller will block waiting to borrow a connection from a saturated pool. + +#### `korvet.redis.pool.size` {#korvet-redis-pool-size} + +**Env var:** `KORVET_REDIS_POOL_SIZE` · **Type:** integer · **Default:** `32` + +Maximum number of pooled connections. Each in-flight per-partition produce write holds one connection for its `XADD`, so size this at least to the sum of partitions across all topics produced to concurrently, plus headroom for the metadata reads that share this pool. Too small a pool queues writes until acquisition times out and produces fail. + +## Schema Registry + +Embedded Confluent-compatible schema-registry HTTP endpoint. + +#### `korvet.schema-registry.default-compatibility` {#korvet-schema-registry-default-compatibility} + +**Env var:** `KORVET_SCHEMA_REGISTRY_DEFAULT_COMPATIBILITY` · **Type:** none, backward, backward_transitive, forward, forward_transitive, full, full_transitive · **Default:** `backward` + +Default compatibility level applied to newly-created subjects. + +#### `korvet.schema-registry.enabled` {#korvet-schema-registry-enabled} + +**Env var:** `KORVET_SCHEMA_REGISTRY_ENABLED` · **Type:** boolean · **Default:** `true` + +Enable the embedded schema-registry HTTP endpoint. + +#### `korvet.schema-registry.validate-produce` {#korvet-schema-registry-validate-produce} + +**Env var:** `KORVET_SCHEMA_REGISTRY_VALIDATE_PRODUCE` · **Type:** boolean · **Default:** `true` + +Reject produce requests whose payload doesn't validate against the subject's latest schema. + +## Storage + +Storage settings split by tier: a Redis-backed local tier and optional object-store remote tier. + +### Local + +Redis-backed local tier settings. + +#### Compression + +#### `korvet.storage.local.compression.codec` {#korvet-storage-local-compression-codec} + +**Env var:** `KORVET_STORAGE_LOCAL_COMPRESSION_CODEC` · **Type:** none, gzip, snappy, lz4, zstd · **Default:** `none` + +Default compression codec for values at rest in Redis, applied to topics that do not pin their own `storage.compression.type` at creation. `none` stores values uncompressed and directly readable; any other codec compresses the value into its standard frame format. Compression trades codec CPU for lower Redis I/O and memory usage. Use zstd for JSON/log storage efficiency, snappy when CPU cost matters more, and none when payloads must stay directly readable from Redis or are random/already compressed. Values: none, gzip, snappy, lz4, zstd. + +#### Redis + +Sparse overrides for the local storage Redis client. Each unset field inherits the corresponding value from `korvet.redis`. + +#### `korvet.storage.local.redis.cluster` {#korvet-storage-local-redis-cluster} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_CLUSTER` · **Type:** boolean + +Treat the storage Redis target as a Redis Cluster. + +#### `korvet.storage.local.redis.host` {#korvet-storage-local-redis-host} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_HOST` · **Type:** string + +Storage Redis host. Ignored when `uri` is set. + +#### `korvet.storage.local.redis.io-threads` {#korvet-storage-local-redis-io-threads} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_IO_THREADS` · **Type:** integer + +Storage Redis Lettuce IO thread-pool size. + +#### `korvet.storage.local.redis.password` {#korvet-storage-local-redis-password} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_PASSWORD` · **Type:** secret + +Storage Redis password. + +#### `korvet.storage.local.redis.port` {#korvet-storage-local-redis-port} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_PORT` · **Type:** integer + +Storage Redis port. Ignored when `uri` is set. + +#### `korvet.storage.local.redis.timeout` {#korvet-storage-local-redis-timeout} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_TIMEOUT` · **Type:** duration + +Storage Redis per-command timeout. + +#### `korvet.storage.local.redis.uri` {#korvet-storage-local-redis-uri} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_URI` · **Type:** string + +Storage Redis URI. When set, overrides the primary Redis URI for storage traffic. + +#### `korvet.storage.local.redis.username` {#korvet-storage-local-redis-username} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_USERNAME` · **Type:** string + +Storage Redis ACL username. + +##### Circuit Breaker + +Sparse circuit-breaker overrides for the local storage Redis client. + +#### `korvet.storage.local.redis.circuit-breaker.enabled` {#korvet-storage-local-redis-circuit-breaker-enabled} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_CIRCUIT_BREAKER_ENABLED` · **Type:** boolean + +Enable the storage Redis stream-operation circuit breaker. + +#### `korvet.storage.local.redis.circuit-breaker.log-interval` {#korvet-storage-local-redis-circuit-breaker-log-interval} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_CIRCUIT_BREAKER_LOG_INTERVAL` · **Type:** duration + +Minimum interval between storage Redis breaker-state log lines. + +#### `korvet.storage.local.redis.circuit-breaker.open-duration` {#korvet-storage-local-redis-circuit-breaker-open-duration} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_CIRCUIT_BREAKER_OPEN_DURATION` · **Type:** duration + +How long the storage Redis circuit breaker stays open before allowing a probe call. + +##### Metrics + +Sparse metrics overrides for the local storage Redis client. + +#### `korvet.storage.local.redis.metrics.enabled` {#korvet-storage-local-redis-metrics-enabled} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_METRICS_ENABLED` · **Type:** boolean + +Enable command-latency metrics for the storage Redis client. + +#### `korvet.storage.local.redis.metrics.histogram` {#korvet-storage-local-redis-metrics-histogram} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_METRICS_HISTOGRAM` · **Type:** boolean + +Publish per-command histograms for the storage Redis client. + +#### `korvet.storage.local.redis.metrics.local-distinction` {#korvet-storage-local-redis-metrics-local-distinction} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_METRICS_LOCAL_DISTINCTION` · **Type:** boolean + +Split storage Redis metrics by local socket address. + +#### `korvet.storage.local.redis.metrics.max-latency` {#korvet-storage-local-redis-metrics-max-latency} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_METRICS_MAX_LATENCY` · **Type:** duration + +Upper bound used when bucketing storage Redis latency samples. + +#### `korvet.storage.local.redis.metrics.min-latency` {#korvet-storage-local-redis-metrics-min-latency} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_METRICS_MIN_LATENCY` · **Type:** duration + +Lower bound used when bucketing storage Redis latency samples. + +##### Pool + +Sparse pool overrides for the local storage Redis client. + +#### `korvet.storage.local.redis.pool.max-wait` {#korvet-storage-local-redis-pool-max-wait} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_POOL_MAX_WAIT` · **Type:** duration + +Maximum time a caller will block waiting to borrow a storage Redis connection. + +#### `korvet.storage.local.redis.pool.size` {#korvet-storage-local-redis-pool-size} + +**Env var:** `KORVET_STORAGE_LOCAL_REDIS_POOL_SIZE` · **Type:** integer + +Maximum number of pooled storage Redis connections. + +#### Write + +#### `korvet.storage.local.write.connections` {#korvet-storage-local-write-connections} + +**Env var:** `KORVET_STORAGE_LOCAL_WRITE_CONNECTIONS` · **Type:** integer · **Default:** `8` + +Number of dedicated Redis connections the produce write path is sharded across, routed by stream key. Each Lettuce connection is pinned to a single event loop thread, so this bounds how many cores one broker's write path can use (issue #885). + +### Remote + +Optional object-store remote tier settings. + +#### `korvet.storage.remote.path` {#korvet-storage-remote-path} + +**Env var:** `KORVET_STORAGE_REMOTE_PATH` · **Type:** string + +Object-store path for the remote Iceberg table (e.g. `s3://bucket/korvet/cold`). Absent/blank disables the remote tier — Korvet runs local-only on Redis Streams. + +#### Iceberg + +#### `korvet.storage.remote.iceberg.row-group-size` {#korvet-storage-remote-iceberg-row-group-size} + +**Env var:** `KORVET_STORAGE_REMOTE_ICEBERG_ROW_GROUP_SIZE` · **Type:** data size · **Default:** `128MB` + +Parquet row-group size for Iceberg data files written by the remote segment store. Maps to Iceberg `write.parquet.row-group-size-bytes`. + +#### `korvet.storage.remote.iceberg.target-file-size` {#korvet-storage-remote-iceberg-target-file-size} + +**Env var:** `KORVET_STORAGE_REMOTE_ICEBERG_TARGET_FILE_SIZE` · **Type:** data size · **Default:** `128MB` + +Target size for Iceberg data files written by the remote segment store. Maps to Iceberg `write.target-file-size-bytes`. + +#### Metrics + +#### `korvet.storage.remote.metrics.common-tags` {#korvet-storage-remote-metrics-common-tags} + +**Env var:** `KORVET_STORAGE_REMOTE_METRICS_COMMON_TAGS` · **Type:** map · **Default:** `[:]` + +Extra tags attached to remote-tier Iceberg and object-storage SDK meters. + +#### `korvet.storage.remote.metrics.enabled` {#korvet-storage-remote-metrics-enabled} + +**Env var:** `KORVET_STORAGE_REMOTE_METRICS_ENABLED` · **Type:** boolean · **Default:** `true` + +Enable Iceberg and object-storage SDK metrics for the remote tier when a Micrometer `MeterRegistry` is available. + +#### S3 + +S3 connection settings for the remote tier object store. Only consulted when `korvet.storage.remote.path` starts with `s3://`. + +#### `korvet.storage.remote.s3.access-key-id` {#korvet-storage-remote-s3-access-key-id} + +**Env var:** `KORVET_STORAGE_REMOTE_S3_ACCESS_KEY_ID` · **Type:** string + +Access-key id. Maps to Iceberg `s3.access-key-id`. + +#### `korvet.storage.remote.s3.endpoint` {#korvet-storage-remote-s3-endpoint} + +**Env var:** `KORVET_STORAGE_REMOTE_S3_ENDPOINT` · **Type:** string + +Optional endpoint URL for non-AWS S3-compatible stores (e.g. MinIO). Maps to `s3.endpoint`. + +#### `korvet.storage.remote.s3.path-style-access` {#korvet-storage-remote-s3-path-style-access} + +**Env var:** `KORVET_STORAGE_REMOTE_S3_PATH_STYLE_ACCESS` · **Type:** boolean + +Use path-style addressing instead of the default virtual-hosted-style. Maps to `s3.path-style-access`. Required for most non-AWS S3 stores. + +#### `korvet.storage.remote.s3.region` {#korvet-storage-remote-s3-region} + +**Env var:** `KORVET_STORAGE_REMOTE_S3_REGION` · **Type:** string + +AWS region (e.g. `us-east-1`). Maps to Iceberg `client.region`. + +#### `korvet.storage.remote.s3.secret-access-key` {#korvet-storage-remote-s3-secret-access-key} + +**Env var:** `KORVET_STORAGE_REMOTE_S3_SECRET_ACCESS_KEY` · **Type:** secret + +Secret access key. Maps to `s3.secret-access-key`. + +### Worker + +Storage worker: leader-locked periodic task that rolls eligible segments, offloads sealed segments, and enforces local and remote retention. + +#### `korvet.storage.worker.enabled` {#korvet-storage-worker-enabled} + +**Env var:** `KORVET_STORAGE_WORKER_ENABLED` · **Type:** boolean · **Default:** `true` + +Enable the storage worker. The Redis leader lock ensures at most one enabled instance runs at a time across the cluster. + +#### `korvet.storage.worker.lease-duration` {#korvet-storage-worker-lease-duration} + +**Env var:** `KORVET_STORAGE_WORKER_LEASE_DURATION` · **Type:** duration · **Default:** `2m` + +Redis leader-lock lease duration. Must exceed `tickInterval` so a slow tick does not let the lease expire mid-flight. + +#### `korvet.storage.worker.tick-interval` {#korvet-storage-worker-tick-interval} + +**Env var:** `KORVET_STORAGE_WORKER_TICK_INTERVAL` · **Type:** duration · **Default:** `1m` + +Tick cadence for the storage worker loop. + +## Ui + +#### `korvet.ui.enabled` {#korvet-ui-enabled} + +**Env var:** `KORVET_UI_ENABLED` · **Type:** boolean · **Default:** `true` + +Whether the embedded SPA is served. Defaults to `true`. + +## Topic configuration patterns + +The `korvet.topics` list is order-sensitive: each entry is a glob pattern plus +optional field overrides, evaluated top-to-bottom with first-non-null-wins +semantics. Fields not set by any matching pattern fall back to the built-in +defaults. Per-topic admin-set overrides beat patterns. + +The per-element fields (`korvet.topics[n].partitions`, `compression`, etc.) +live on `TopicPattern` in korvet-server and are not enumerated in the tables +above. The full list: + +- `name` (required) — glob pattern matched against topic names. +- `auto-create` — whether unknown topics matching this pattern may be auto-created. +- `partitions`, `offset-sequence-bits` +- `compression`, `remote-storage-enabled` +- `retention-time`, `retention-bytes` (`*-bytes` accepts data sizes such as `10GB`) +- `local-retention-time`, `local-retention-bytes` (`*-bytes` accepts data sizes such as `512MB`) +- `segment-time`, `segment-bytes` (`*-bytes` accepts data sizes such as `64MB`) + +As environment variables, list entries are addressed by index: `korvet.topics[0].retention-time` becomes `KORVET_TOPICS_0_RETENTION_TIME`, `korvet.topics[1].auto-create` becomes `KORVET_TOPICS_1_AUTO_CREATE`, and so on. Each indexed entry needs at least `KORVET_TOPICS__NAME`. + +See [the configuration guide]({{< relref "/integrate/korvet/quick-start/configuration" >}}) for examples. + +## Environment variables + +Every property can be set via an environment variable using Spring Boot's +relaxed binding: uppercase the property name, replace `.` and `-` with `_`. +For example: + +| Property | Environment variable | +|---|---| +| `korvet.broker.max-request-bytes` | `KORVET_BROKER_MAX_REQUEST_BYTES` | +| `korvet.storage.remote.path` | `KORVET_STORAGE_REMOTE_PATH` | +| `korvet.storage.worker.enabled` | `KORVET_STORAGE_WORKER_ENABLED` | + +The exact env-var name for every property is listed in the tables above. For the `korvet.topics` pattern list, entries are addressed by index, e.g. `KORVET_TOPICS_0_NAME` and `KORVET_TOPICS_0_PARTITIONS`. + +## Related pages + +- [Getting Started — Configuration]({{< relref "/integrate/korvet/quick-start/configuration" >}}) +- [Remote Storage configuration & S3 tuning]({{< relref "/integrate/korvet/storage/remote-storage" >}}) +- [Deployment]({{< relref "/integrate/korvet/operations/deployment" >}}) diff --git a/content/integrate/korvet/reference/faq.md b/content/integrate/korvet/reference/faq.md new file mode 100644 index 0000000000..aa087b43c8 --- /dev/null +++ b/content/integrate/korvet/reference/faq.md @@ -0,0 +1,203 @@ +--- +Title: Frequently Asked Questions +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Common questions about Korvet. +linkTitle: FAQ +weight: 40 +--- + +Common questions about Korvet. + +## General + +### What is Korvet? + +Korvet is a Kafka-compatible streaming service backed by Redis Streams, with optional tiered storage to an Apache Iceberg table on an object store for long-term archival. + +### Why use Korvet instead of Kafka? + +- **Simpler operations**: No ZooKeeper, no partition rebalancing complexity +- **Redis integration**: Leverage existing Redis infrastructure and persistence +- **Low latency**: Sub-millisecond read/write performance with Redis Streams +- **Optional cost optimization**: Tiered storage to object-store Parquet for long-term retention + +### Is Korvet production-ready? + +Korvet is in active development. The core features are functional: + +- ✅ Kafka protocol implementation (Produce, Fetch, Consumer Groups, Admin API) +- ✅ Redis Streams storage with retention policies +- ✅ Async operations for high throughput +- ✅ Metrics and observability +- ✅ Built-in Parquet archival to object store (100k+ msg/s to S3) + +## Compatibility + +### Which Kafka clients work with Korvet? + +Any Kafka client that supports the Kafka protocol should work. Tested clients include: + +- Java: kafka-clients +- Python: kafka-python, confluent-kafka-python +- Go: sarama +- Node.js: kafkajs + +### What Kafka features are supported? + +Supported: + +- ✅ Produce API +- ✅ Fetch API +- ✅ Consumer Groups (JoinGroup, SyncGroup, Heartbeat, LeaveGroup, OffsetCommit, OffsetFetch) +- ✅ Topic metadata +- ✅ Admin API (CreateTopics, DeleteTopics, DescribeConfigs, DescribeCluster) +- ✅ Compression (GZIP, SNAPPY, LZ4, ZSTD) + +Not supported: + +- ❌ Transactions +- ❌ Exactly-once semantics (at-least-once delivery only) + +### Can I migrate from Kafka to Korvet? + +Yes, but with some considerations: + +- Transactions are not supported +- Exactly-once semantics are not supported (at-least-once only) +- You'll need to re-produce historical data or use a migration tool +- Consumer group offsets won't transfer automatically + +## Performance + +### What throughput can Korvet handle? + +Performance depends on your Redis instance and configuration. With async operations and pipelining: + +- **Produce**: 50,000+ messages/second per instance (tested up to 170k msg/sec) +- **Fetch**: 100,000+ messages/second per instance (tested up to 313k msg/sec) +- **Latency**: Sub-millisecond p50, 1-3ms p95, <100ms p99 + +See the [load-testing sample](https://github.com/redis-field-engineering/korvet-dist/tree/main/samples/load-testing) for benchmarking tools. + +### How do I scale Korvet? + +Korvet is stateless and can be scaled horizontally: + +1. Run multiple Korvet instances +2. Put a load balancer in front +3. Clients connect to any instance + +### What are the resource requirements? + +Minimum: + +- **CPU**: 1 core +- **Memory**: 512MB +- **Redis**: 8+ + +Recommended for production: + +- **CPU**: 2-4 cores +- **Memory**: 2-4GB +- **Redis**: Cluster or Enterprise for HA + +## Storage + +### How long can I keep data in Redis? + +As long as your Redis instance has capacity. Configure retention policies (the Kafka-compatible topic config keys `retention.ms` and `retention.bytes`, or the `retention-time`/`retention-bytes` keys on a `korvet.topics` pattern) to trim old messages. Trimming is enforced asynchronously by a leader-locked background storage worker (via Redis `XTRIM`), not at write time. + +### What happens when Redis is full? + +You have several options: + +1. **Increase Redis memory**: Scale up your Redis instance +2. **Configure retention**: Set `retention.ms` or `retention.bytes` to automatically trim old messages +3. **Enable tiered storage**: Configure optional Parquet archival to move old messages to object storage +4. **Redis eviction policies**: Configure Redis eviction (e.g., `allkeys-lru`) as a last resort + +### Can I query archived data directly from the object store? + +Yes. The remote tier is a standard Apache Iceberg table (Parquet data files plus Iceberg metadata and manifests), so external query engines such as Spark, Trino, Athena, and DuckDB can read it directly. Korvet clients also read archived data transparently through the normal Kafka fetch API — the broker reads from the remote tier on their behalf. + +See [Remote Storage]({{< relref "/integrate/korvet/storage/remote-storage" >}}) for configuration details. + +### Should I use JSON flattening or RAW storage with compression? + +It depends on your data structure and message size: + +**Use JSON flattening** (default for JSON objects): + +- Shallow JSON with many top-level fields (10+) +- Small messages (<1KB) +- When you need field-level access in Redis +- Varied field values (UUIDs, timestamps, user data) + +**Use RAW + storage compression**: + +- Deeply nested JSON (>2-3 levels) +- Large messages (>10KB) +- Repetitive data patterns (logs, telemetry) +- Binary formats (Protocol Buffers, Avro) + +Start with ZSTD for JSON/log storage efficiency. Use SNAPPY when CPU cost matters more than the last +few percentage points of compression. + +**Example:** For log messages with deep nesting, RAW+ZSTD can reduce Redis memory usage by 18x compared to JSON flattening. + +See [Value Mapper Selection]({{< relref "/integrate/korvet/storage/redis-streams" >}}#value-mapper-selection) for detailed guidance and performance comparisons. + +### How much memory can storage compression save? + +Based on production testing with 100KB messages: + +- **ZSTD**: Up to 51x compression (100KB → 2KB) +- **LZ4**: Up to 14x compression (100KB → 7KB) +- **GZIP**: Up to 26x compression (100KB → 4KB) +- **SNAPPY**: Up to 10x compression (100KB → 10KB) + +Actual compression ratios depend on your data. Repetitive data (logs, structured text) compresses better than random data (images, encrypted data). + +{{< tip >}} +Use ZSTD for JSON/log storage efficiency. Use SNAPPY for CPU-sensitive topics that still benefit from compression. +{{< /tip >}} + +## Operations + +### How do I monitor Korvet? + +Korvet exposes metrics via Prometheus and health checks via Spring Boot Actuator. See [Monitoring]({{< relref "/integrate/korvet/operations/monitoring" >}}). + +### How do I troubleshoot issues? + +1. Check logs (JSON format by default) +2. Review metrics in Prometheus +3. Check health endpoints +4. See [Troubleshooting guide]({{< relref "/integrate/korvet/operations/troubleshooting" >}}) + +### Can I run Korvet in Kubernetes? + +Yes! See [Deployment guide]({{< relref "/integrate/korvet/operations/deployment" >}}) for Kubernetes manifests. + +## Development + +### How do I contribute to Korvet? + +See the [GitHub repository](https://github.com/redis-field-engineering/korvet-dist) for contribution guidelines. + +### Where can I report bugs? + +File an issue at https://github.com/redis-field-engineering/korvet-dist/issues + +### Is there a roadmap? + +Check the GitHub issues and project boards for planned features. + +## Next Steps + +- [Getting started]({{< relref "/integrate/korvet/quick-start" >}}) +- [Troubleshooting]({{< relref "/integrate/korvet/operations/troubleshooting" >}}) diff --git a/content/integrate/korvet/reference/metrics/_index.md b/content/integrate/korvet/reference/metrics/_index.md new file mode 100644 index 0000000000..4abbb8eff0 --- /dev/null +++ b/content/integrate/korvet/reference/metrics/_index.md @@ -0,0 +1,44 @@ +--- +Title: Metrics +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Korvet exposes metrics in Prometheus format via Spring Boot Actuator. +hideListLinks: false +linkTitle: Metrics +weight: 30 +--- + +Korvet exposes metrics in Prometheus format via Spring Boot Actuator. + +The metrics reference is organized by module: + +- [Tags and families]({{< relref "/integrate/korvet/reference/metrics/contracts" >}}) +- [Application metrics]({{< relref "/integrate/korvet/reference/metrics/application" >}}) +- [Broker metrics]({{< relref "/integrate/korvet/reference/metrics/broker" >}}) +- [Mapper metrics]({{< relref "/integrate/korvet/reference/metrics/mapper" >}}) +- [Storage metrics]({{< relref "/integrate/korvet/reference/metrics/storage" >}}) (cross-tier verbs plus local/remote tier-specific meters) +- [Storage worker metrics]({{< relref "/integrate/korvet/reference/metrics/storage-worker" >}}) +- [Lettuce Redis client metrics]({{< relref "/integrate/korvet/reference/metrics/redis-client" >}}) + +See [Tags and families]({{< relref "/integrate/korvet/reference/metrics/contracts" >}}) for the tag vocabulary and top-level metric families. + +## Platform Metrics + +Micrometer also exports standard JVM and system metrics. + +| Metric | Description | +|---|---| +| `jvm.memory.used` | JVM memory used | +| `jvm.memory.max` | JVM maximum memory | +| `jvm.gc.pause` | Garbage collection pause time | +| `jvm.threads.live` | Live threads | +| `process.cpu.usage` | Process CPU usage | +| `system.cpu.usage` | System CPU usage | + +## Next Steps + +- [Monitoring guide]({{< relref "/integrate/korvet/operations/monitoring" >}}) — example Prometheus queries, dashboards, and alerts +- [API reference]({{< relref "/integrate/korvet/reference/api" >}}) diff --git a/content/integrate/korvet/reference/metrics/application.md b/content/integrate/korvet/reference/metrics/application.md new file mode 100644 index 0000000000..8ecf763a35 --- /dev/null +++ b/content/integrate/korvet/reference/metrics/application.md @@ -0,0 +1,29 @@ +--- +Title: Application Metrics +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Metrics that carry the running application's identity. +linkTitle: Application +weight: 20 +--- + +These metrics carry the running application's identity. + +## Metrics + +### Build + +Constant gauge (value `1`) carrying the running application's identity in its tag set. Tags: `name`, `version`. + +**Name**: `korvet.application.build` \ +**Type**: `gauge` + +**Tags** + +| Key | Description | +|---|---| +| `name` | Application name (typically `korvet`). | +| `version` | Application version as reported by Spring Boot `BuildProperties`; `unknown` when the build did not embed `build-info.properties`. | diff --git a/content/integrate/korvet/reference/metrics/broker.md b/content/integrate/korvet/reference/metrics/broker.md new file mode 100644 index 0000000000..c77bc6b398 --- /dev/null +++ b/content/integrate/korvet/reference/metrics/broker.md @@ -0,0 +1,198 @@ +--- +Title: Broker Metrics +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Metrics emitted by korvet-broker. +linkTitle: Broker +weight: 30 +--- + +These metrics are emitted by `korvet-broker`. + +## Metrics + +### Backpressure Connections + +Current number of broker connections with backpressure applied. + +**Name**: `korvet.broker.backpressure.connections` \ +**Type**: `gauge` + +### Backpressure Transitions + +Backpressure lifecycle transitions, tagged by `action=applied|released`. + +**Name**: `korvet.broker.backpressure.transitions` \ +**Type**: `counter` + +**Tags** + +| Key | Description | +|---|---| +| `action` | Backpressure action: `applied` or `released`. | + +### Connections + +Current number of active broker connections. + +**Name**: `korvet.broker.connections` \ +**Type**: `gauge` + +### Failures + +Number of broker failures, including lifecycle (start) and authentication failures (tagged `operation=auth`). + +**Name**: `korvet.broker.failures` \ +**Type**: `counter` + +**Tags** + +| Key | Description | +|---|---| +| `error_type` | Failure category, lower-cased. | +| `operation` | Logical operation name. | + +### Fetch Latency + +Fetch request latency. + +**Name**: `korvet.broker.fetch` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `topic` | Kafka topic name. | + +### Frame Size + +Broker Kafka frame size in bytes, by direction (request/response) and API key. + +**Name**: `korvet.broker.frame_size` \ +**Type**: `distribution summary` \ +**Base unit**: `bytes` + +**Tags** + +| Key | Description | +|---|---| +| `api_key` | Kafka API key (request type) on the broker, lower-cased. | +| `direction` | Frame direction: `request` or `response`. | + +### Log Start Offset + +Log start offset per `(topic, partition)`: earliest still-readable offset. Empty partitions report `0`. + +**Name**: `korvet.broker.log_start_offset` \ +**Type**: `gauge` + +**Tags** + +| Key | Description | +|---|---| +| `partition` | Kafka partition number, as a decimal string. | +| `topic` | Kafka topic name. | + +### Lossy Records + +Number of records affected by lossy offset mapping, by phase. + +**Name**: `korvet.broker.lossy_records` \ +**Type**: `counter` + +**Tags** + +| Key | Description | +|---|---| +| `phase` | Lifecycle phase, lower-cased. | +| `topic` | Kafka topic name. | + +### Max Offset + +High watermark per `(topic, partition)`: next offset to be assigned (i.e. latest visible offset + 1). Empty partitions report `0`. + +**Name**: `korvet.broker.max_offset` \ +**Type**: `gauge` + +**Tags** + +| Key | Description | +|---|---| +| `partition` | Kafka partition number, as a decimal string. | +| `topic` | Kafka topic name. | + +### Pending + +Current broker request bytes waiting on asynchronous completion. + +**Name**: `korvet.broker.pending` \ +**Type**: `gauge` \ +**Base unit**: `bytes` + +### Produce Latency + +Produce request latency. + +**Name**: `korvet.broker.produce` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `topic` | Kafka topic name. | + +### Produce Records + +Number of records successfully produced. + +**Name**: `korvet.broker.produce.records` \ +**Type**: `counter` \ +**Base unit**: `records` + +**Tags** + +| Key | Description | +|---|---| +| `topic` | Kafka topic name. | + +### Rebalances + +Number of consumer group rebalance lifecycle events. + +**Name**: `korvet.broker.rebalances` \ +**Type**: `counter` + +**Tags** + +| Key | Description | +|---|---| +| `phase` | Lifecycle phase, lower-cased. | + +### Request Latency + +Broker Kafka API request latency, observed per request. + +**Name**: `korvet.broker.request` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `api_key` | Kafka API key (request type) on the broker, lower-cased. | +| `result` | Outcome of the broker request, lower-cased. | + +### Up + +Whether the broker is currently running. + +**Name**: `korvet.broker.up` \ +**Type**: `gauge` diff --git a/content/integrate/korvet/reference/metrics/contracts.md b/content/integrate/korvet/reference/metrics/contracts.md new file mode 100644 index 0000000000..f07aeb995c --- /dev/null +++ b/content/integrate/korvet/reference/metrics/contracts.md @@ -0,0 +1,73 @@ +--- +Title: Tags and Families +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: The top-level metric families Korvet exports and the tag keys you can + expect to see on them. +linkTitle: Tags and Families +weight: 10 +--- + +This page lists the top-level metric families Korvet exports and the tag keys you can expect to see on them. + +## Metric Families + +Korvet-owned metric families: + +- `korvet.application.*` — Running application identity (build info gauge). +- `korvet.broker.*` — Kafka API request/response, frame sizes, backpressure, auth, rebalance, and broker lifecycle. +- `korvet.storage.*` — Cross-tier storage verbs (`read`, `write`, `ack`), tagged with `tier=local|remote`; read meters also carry `mode=stream|group`. +- `korvet.storage.local.*` — Local Redis connection-pool internals. +- `korvet.storage.worker.*` — Leader-locked storage worker liveness and failures. +- `korvet.mapper.*` — Record/offset mapping internals. + +Standard Micrometer JVM/system metrics and Lettuce Redis client metrics are also exported but are not Korvet-owned. + +## Tag Keys You Will See + +Korvet-owned metrics use only the following tag keys: + +- `name` +- `version` +- `topic` +- `partition` +- `operation` +- `result` +- `error_type` +- `phase` +- `action` +- `tier` +- `compression` +- `api_key` +- `direction` +- `mode` + +You can safely build dashboards and alerts around these labels. + +`partition` appears only on the broker offset gauges (`korvet.broker.max_offset`, +`korvet.broker.log_start_offset`). Its cardinality is bounded by the +per-topic partition count and capped by `korvet.broker.metrics.offset-cardinality-cap`. + +## Why Tag Keys Are Bare + +Tag keys are bare (`topic`, not `korvet_topic`) because Korvet's deployment model does not co-locate other Prometheus exporters that emit the same generic keys against unrelated metric families. Bare keys keep dashboards portable and queries readable. If you operate Korvet alongside an exporter that emits a colliding key on unrelated metrics, apply a Prometheus `metric_relabel_configs` rule at scrape time. + +## Tag Keys You Will Not See + +The following dimensions are intentionally omitted to keep time-series cardinality bounded, even when they exist in the underlying request or message: + +- `consumer` +- `consumer_group` +- `message_id` +- `offset` +- `partition_epoch` +- `request_id` +- `session_id` +- `stream` +- `transaction_id` +- `user_id` + +If you need per-consumer or per-request visibility, use logs or traces rather than metric labels. diff --git a/content/integrate/korvet/reference/metrics/mapper.md b/content/integrate/korvet/reference/metrics/mapper.md new file mode 100644 index 0000000000..4b078672e5 --- /dev/null +++ b/content/integrate/korvet/reference/metrics/mapper.md @@ -0,0 +1,72 @@ +--- +Title: Mapper Metrics +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Metrics emitted by korvet-mapper. +linkTitle: Mapper +weight: 40 +--- + +These metrics are emitted by `korvet-mapper`. + +## Metrics + +### Compression Ratio + +Compression ratio recorded by mapper compression operations. The distribution's `_count` also reports the compression event count per algorithm. + +**Name**: `korvet.mapper.compression_ratio` \ +**Type**: `distribution summary` + +**Tags** + +| Key | Description | +|---|---| +| `compression` | Compression algorithm (e.g. `none`, `gzip`, `snappy`, `lz4`, `zstd`). | + +### Failures + +Mapper failures by operation and exception class. + +**Name**: `korvet.mapper.failures` \ +**Type**: `counter` + +**Tags** + +| Key | Description | +|---|---| +| `error_type` | Exception class simple name normalised to lower case. | +| `operation` | Logical mapper operation name. | + +### Operation + +Mapper operation latency, observed per call. Tagged with the logical operation name and outcome ( `success`, `error`). Use the timer's `_count` for operation counts. + +**Name**: `korvet.mapper.operation` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | Logical mapper operation name. | +| `result` | Operation outcome (e.g. `success`, `error`). | + +### Payload Size + +Mapper payload sizes in bytes, by operation and phase. + +**Name**: `korvet.mapper.payload_size` \ +**Type**: `distribution summary` \ +**Base unit**: `bytes` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | Logical mapper operation name. | +| `phase` | Mapper phase (e.g. `input`, `output`). | diff --git a/content/integrate/korvet/reference/metrics/redis-client.md b/content/integrate/korvet/reference/metrics/redis-client.md new file mode 100644 index 0000000000..7de1661724 --- /dev/null +++ b/content/integrate/korvet/reference/metrics/redis-client.md @@ -0,0 +1,44 @@ +--- +Title: Lettuce Redis Client Metrics +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Korvet can optionally enable Lettuce command latency metrics to track + Redis operation performance. +linkTitle: Redis Client +weight: 70 +--- + +Korvet can optionally enable Lettuce command latency metrics to track Redis operation performance. + +{{< note >}} +These metrics are disabled by default. Enable them by setting `korvet.redis.metrics.enabled=true` in your configuration. +{{< /note >}} + +| Metric | Description | Tags | +|---|---|---| +| `lettuce.command.firstresponse` | Time to first response from Redis (timer) | `command`, `local`, `remote` | +| `lettuce.command.completion` | Time to complete Redis command (timer) | `command`, `local`, `remote` | + +**Configuration**: + +```yaml +korvet: + redis: + metrics: + enabled: true + histogram: false + local-distinction: false + max-latency: 5m + min-latency: 1ms +``` + +**Configuration Properties**: + +- `enabled` (boolean, default: `false`): Enable Lettuce command latency metrics +- `histogram` (boolean, default: `false`): Enable histogram buckets for aggregable percentile approximations +- `local-distinction` (boolean, default: `false`): Track metrics per connection instead of per host/port +- `max-latency` (duration, default: `5m`): Maximum expected latency for histogram buckets +- `min-latency` (duration, default: `1ms`): Minimum expected latency for histogram buckets diff --git a/content/integrate/korvet/reference/metrics/storage-worker.md b/content/integrate/korvet/reference/metrics/storage-worker.md new file mode 100644 index 0000000000..25399550bc --- /dev/null +++ b/content/integrate/korvet/reference/metrics/storage-worker.md @@ -0,0 +1,34 @@ +--- +Title: Storage Worker Metrics +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Metrics emitted by the leader-locked storage worker. +linkTitle: Storage Worker +weight: 60 +--- + +These metrics are emitted by the leader-locked storage worker, which rolls eligible segments, offloads sealed segments to the remote tier, and enforces local and remote retention. + +## Up + +Storage worker liveness. `1` while the worker is running in this JVM, `0` otherwise. Because the worker is leader-locked, at most one instance across the cluster reports `1` at a time. + +**Name**: `korvet.storage.worker.up` \ +**Type**: `gauge` + +## Failures + +Storage worker failures by lifecycle phase and error type. + +**Name**: `korvet.storage.worker.failures` \ +**Type**: `counter` + +**Tags** + +| Key | Description | +|---|---| +| `phase` | Lifecycle phase where the failure occurred (e.g. `start`). | +| `error_type` | Exception class simple name, or `none`. | diff --git a/content/integrate/korvet/reference/metrics/storage.md b/content/integrate/korvet/reference/metrics/storage.md new file mode 100644 index 0000000000..d48fcd3be9 --- /dev/null +++ b/content/integrate/korvet/reference/metrics/storage.md @@ -0,0 +1,386 @@ +--- +Title: Storage Metrics +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Metrics emitted by every Korvet storage tier under the korvet.storage.* + namespace. +linkTitle: Storage +weight: 50 +--- + +These metrics are emitted by every Korvet storage tier under the +`korvet.storage.*` namespace. Each verb (`read`, `write`, `ack`) +is recorded once per call and tagged with `tier=local|remote`, so the same +dashboard query works across the local Redis tier and the remote +Iceberg tier. The `read` timer additionally carries `mode=stream|group` to +distinguish standalone reads from consumer-group reads. + +## Cross-Tier Verbs + +### Ack + +Storage acknowledgement latency. + +**Name**: `korvet.storage.ack` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `result` | Operation outcome (e.g. `success`, `empty`, `error`, `timeout`). | +| `tier` | Storage tier ( `local` or `remote`). | + +### Archive + +Remote segment archive latency for offload attempts from local Redis segments to the remote tier. + +**Name**: `korvet.storage.archive` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `result` | Operation outcome (e.g. `success`, `empty`, `error`, `timeout`). | +| `tier` | Storage tier ( `local` or `remote`). | + +### Archive Bytes + +Remote bytes archived successfully. + +**Name**: `korvet.storage.archive.bytes` \ +**Type**: `counter` \ +**Base unit**: `bytes` + +**Tags** + +| Key | Description | +|---|---| +| `tier` | Storage tier ( `local` or `remote`). | + +### Archive Failures + +Remote segment archive failures by exception type. + +**Name**: `korvet.storage.archive.failures` \ +**Type**: `counter` + +**Tags** + +| Key | Description | +|---|---| +| `error_type` | Exception class simple name, or `none`. | +| `tier` | Storage tier ( `local` or `remote`). | + +### Archive Lag Oldest + +Age of the oldest sealed remote-enabled segment waiting for archive. + +**Name**: `korvet.storage.archive.lag.oldest` \ +**Type**: `gauge` \ +**Base unit**: `seconds` + +### Archive Lag Segments + +Sealed remote-enabled segments waiting for archive. + +**Name**: `korvet.storage.archive.lag.segments` \ +**Type**: `gauge` \ +**Base unit**: `segments` + +### Archive Segments + +Remote segments archived successfully. + +**Name**: `korvet.storage.archive.segments` \ +**Type**: `counter` \ +**Base unit**: `segments` + +**Tags** + +| Key | Description | +|---|---| +| `tier` | Storage tier ( `local` or `remote`). | + +### Read + +Storage read latency. Tagged by `mode=stream|group` to distinguish direct stream reads from consumer-group reads (XREADGROUP, pending claims, autoclaim). + +**Name**: `korvet.storage.read` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `mode` | Read mode ( `stream` for direct stream reads, `group` for consumer-group reads). | +| `result` | Operation outcome (e.g. `success`, `empty`, `error`, `timeout`). | +| `tier` | Storage tier ( `local` or `remote`). | + +### Read Messages + +Messages returned per read call. Tagged by `mode=stream|group`. + +**Name**: `korvet.storage.read.messages` \ +**Type**: `distribution summary` \ +**Base unit**: `messages` + +**Tags** + +| Key | Description | +|---|---| +| `mode` | Read mode ( `stream` for direct stream reads, `group` for consumer-group reads). | +| `tier` | Storage tier ( `local` or `remote`). | + +### Write + +Storage write latency (writes, deletes, group admin). + +**Name**: `korvet.storage.write` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `result` | Operation outcome (e.g. `success`, `empty`, `error`, `timeout`). | +| `tier` | Storage tier ( `local` or `remote`). | + +### Write Messages + +Messages written per write call. + +**Name**: `korvet.storage.write.messages` \ +**Type**: `distribution summary` \ +**Base unit**: `messages` + +**Tags** + +| Key | Description | +|---|---| +| `tier` | Storage tier ( `local` or `remote`). | + +## Local Tier (Redis) + +### Pool Acquire + +Local Redis pool acquisition latency by result. + +**Name**: `korvet.storage.local.pool.acquire` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `result` | Pool acquisition outcome ( `success`, `timeout`, `error`). | + +### Pool Pending + +Current number of pending waiters on the local Redis connection pool. + +**Name**: `korvet.storage.local.pool.pending` \ +**Type**: `gauge` + +## Remote Tier (Iceberg And S3) + +Remote-tier reads and writes flow through the shared cross-tier verbs above +with `tier=remote`. The cold tier also emits Iceberg scan/commit reports +and S3 SDK call metrics when `korvet.storage.remote.metrics.enabled=true` +and a Micrometer registry is available. + +### Commit Added Files + +Number of Iceberg data files added by each cold-tier commit. + +**Name**: `iceberg.commit.added.files` \ +**Type**: `distribution summary` \ +**Base unit**: `files` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | Iceberg operation, such as `scan`, `append`, or `delete`. | +| `table` | Fully qualified Iceberg table name. | + +### Commit Added Records + +Number of records added by each Iceberg cold-tier commit. + +**Name**: `iceberg.commit.added.records` \ +**Type**: `distribution summary` \ +**Base unit**: `records` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | Iceberg operation, such as `scan`, `append`, or `delete`. | +| `table` | Fully qualified Iceberg table name. | + +### Commit Attempts + +Number of attempts Iceberg needed for each cold-tier commit. + +**Name**: `iceberg.commit.attempts` \ +**Type**: `distribution summary` \ +**Base unit**: `attempts` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | Iceberg operation, such as `scan`, `append`, or `delete`. | +| `table` | Fully qualified Iceberg table name. | + +### Commit Duration + +Time spent committing Iceberg metadata changes for cold-tier writes and deletes. + +**Name**: `iceberg.commit.duration` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | Iceberg operation, such as `scan`, `append`, or `delete`. | +| `table` | Fully qualified Iceberg table name. | + +### Commit Removed Files + +Number of Iceberg data files removed by each cold-tier commit. + +**Name**: `iceberg.commit.removed.files` \ +**Type**: `distribution summary` \ +**Base unit**: `files` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | Iceberg operation, such as `scan`, `append`, or `delete`. | +| `table` | Fully qualified Iceberg table name. | + +### S3 Api Call Count + +AWS SDK S3 API call count tagged by operation and error outcome. + +**Name**: `aws.s3.api.call.count` \ +**Type**: `counter` + +**Tags** + +| Key | Description | +|---|---| +| `error` | Whether the AWS SDK call ended with an error. | +| `operation` | AWS SDK operation name. | + +### S3 Api Call Duration + +AWS SDK S3 API call duration for object-store operations issued by Iceberg. + +**Name**: `aws.s3.api.call.duration` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | AWS SDK operation name. | + +### S3 Api Call Retry Count + +AWS SDK retry count for S3 API calls issued by Iceberg. + +**Name**: `aws.s3.api.call.retry.count` \ +**Type**: `counter` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | AWS SDK operation name. | + +### S3 Http Client Acquire Duration + +Time the AWS SDK S3 HTTP client spent acquiring concurrency for a request. + +**Name**: `aws.s3.http.client.acquire.duration` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +### Scan Bytes Scanned + +Bytes in data files included in each Iceberg cold-tier scan result. + +**Name**: `iceberg.scan.bytes.scanned` \ +**Type**: `distribution summary` \ +**Base unit**: `bytes` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | Iceberg operation, such as `scan`, `append`, or `delete`. | +| `table` | Fully qualified Iceberg table name. | + +### Scan Files Scanned + +Number of data files included in each Iceberg cold-tier scan result. + +**Name**: `iceberg.scan.files.scanned` \ +**Type**: `distribution summary` \ +**Base unit**: `files` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | Iceberg operation, such as `scan`, `append`, or `delete`. | +| `table` | Fully qualified Iceberg table name. | + +### Scan Files Skipped + +Number of data files skipped by each Iceberg cold-tier scan. + +**Name**: `iceberg.scan.files.skipped` \ +**Type**: `distribution summary` \ +**Base unit**: `files` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | Iceberg operation, such as `scan`, `append`, or `delete`. | +| `table` | Fully qualified Iceberg table name. | + +### Scan Planning Duration + +Time spent by Iceberg planning a scan against the cold tier. + +**Name**: `iceberg.scan.planning.duration` \ +**Type**: `timer` \ +**Base unit**: `seconds` + +**Tags** + +| Key | Description | +|---|---| +| `operation` | Iceberg operation, such as `scan`, `append`, or `delete`. | +| `table` | Fully qualified Iceberg table name. | + +See [Storage Worker Metrics]({{< relref "/integrate/korvet/reference/metrics/storage-worker" >}}) for +offload and retention activity against the remote tier. diff --git a/content/integrate/korvet/storage/_index.md b/content/integrate/korvet/storage/_index.md new file mode 100644 index 0000000000..19e603b3fe --- /dev/null +++ b/content/integrate/korvet/storage/_index.md @@ -0,0 +1,129 @@ +--- +Title: Tiered Storage +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Korvet stores all messages in Redis Streams (the local tier) and can + optionally archive older data to Apache Iceberg tables on an object store (the + remote tier). +hideListLinks: false +linkTitle: Tiered Storage +weight: 40 +--- + +Korvet stores all messages in Redis Streams (the local tier) and can optionally archive older data to Apache Iceberg tables on an object store (the remote tier) for cost-efficient long-term retention. This section covers both tiers and how data moves between them. For the foundational storage model, see [Concepts & Architecture]({{< relref "/integrate/korvet/concepts" >}}); for the exact Redis key layout, see Redis Data Structures. + +## Storage Architecture + +Korvet supports two storage configurations: + +### Redis-Only Storage (Default) + +The default configuration uses Redis Streams exclusively: + +- **Primary storage**: All messages stored in Redis Streams +- **Persistence**: Redis AOF and RDB for durability +- **Consumer groups**: Built-in support for coordinated consumption +- **Performance**: Sub-millisecond read/write latency +- **Retention**: Configurable time and size-based retention (applied by the background storage worker via `XTRIM`) + +This is the recommended configuration for most use cases. + +### Tiered Storage (Local → Remote) + +For long-term data retention and cost optimization, Korvet can be configured with tiered storage. +Because a tiered partition is split into fixed-size **segments** (see [How It Works](#how-it-works)), the data +naturally spans three temperature tiers: + +- **Hot tier (RAM)**: The current **open** segment lives in Redis memory and receives all writes, giving sub-millisecond produce and fetch latency. +- **Warm tier (flash/disk)**: Sealed segments that are still within the local retention window. With [Redis Flex (Auto Tiering, formerly Redis on Flash)](https://redis.io/docs/latest/operate/rs/databases/auto-tiering/), Redis transparently keeps these colder sealed segments on SSD/flash while keeping the hot working set in RAM — no Korvet configuration required. Without Redis Flex, sealed segments simply remain in RAM until they are offloaded or expire. +- **Cold tier (object store)**: Offloaded sealed segments archived to Apache Iceberg tables (one per topic) on S3 (or any Iceberg-supported object store / Hadoop-compatible filesystem). + +Key points: + +- **Automatic archival**: The built-in storage worker continuously offloads sealed segments from the local tier (hot/warm) to the cold tier. +- **Transparent to Redis Flex**: Segmentation is what makes the hot/warm split work — each sealed segment is a separate Redis Stream key, so Redis Flex can demote whole segments to flash based on access patterns. +- **High throughput**: Achieves 100k+ messages/second to S3 with parallel streams. + +## How It Works + +### Redis-Only Storage + +1. **Produce**: Messages are written to Redis Streams using `XADD` +2. **Retention**: A background storage worker applies retention policies using `XTRIM` (`MAXLEN` for count/byte limits, `MINID` for time limits) +3. **Consume**: Consumers read messages using `XREAD` (standalone) or `XREADGROUP` (consumer groups) +4. **Persistence**: Redis handles durability through AOF/RDB snapshots + +### Tiered Storage + +A topic partition is treated as a sequence of fixed-size **segments**. Segmentation is not specific to +tiered storage: every newly created topic is segmented by default (see `segment.bytes` / `segment.ms` in +[Topics]({{< relref "/integrate/korvet/kafka-api/topics" >}})), regardless of whether `remote.storage.enable` is set. The only +exception is compacted topics, which keep a single-stream layout. In a Redis-only deployment the segments +simply stay in Redis; tiered storage adds offloading on top. + +With tiered storage, the newest segment (the **open** segment) lives in Redis and receives all writes; older +sealed segments are offloaded to the remote tier and dropped from Redis once local retention expires. Reads +are served transparently across all tiers, so a consumer never sees the tier boundary. + +{{< image filename="images/korvet/tiered-storage.svg" alt="How tiered storage works" >}} + +1. **Produce (hot)**: Messages are appended (`XADD`) to the open segment in Redis RAM (hot tier). +2. **Seal (warm)**: When the open segment reaches its configured size or age (`segment.bytes` / `segment.ms`), the storage worker seals it and opens the next one. `segment.bytes` is compared against the segment's **real** stored size (Redis `MEMORY USAGE`), not an estimate. Sealed segments remain in the local tier; with Redis Flex enabled, Redis can demote these colder segments from RAM to flash automatically (warm tier). +3. **Archive (cold)**: The leader-locked storage worker streams sealed segments from Redis into the topic's Apache Iceberg table on the remote object store (cold tier), then marks them offloaded. +4. **Consume**: Consumers read transparently across tiers — the broker reads from local Redis (RAM or flash) or the remote table depending on which segment holds the requested offset. +5. **Cleanup**: After `local.retention.ms`, offloaded segments are dropped from Redis; the total `retention.ms` governs when they are removed from the remote tier. + +{{< note >}} +The hot/warm split is handled entirely by Redis Flex and requires no Korvet configuration — Korvet only +distinguishes **local** (Redis) and **remote** (object store) tiers. Segmentation is what lets Redis Flex +operate per-segment: each sealed segment is its own stream key, so the colder ones can live on flash +while the open segment stays in RAM. +{{< /note >}} + +### Per-Topic Tiered Storage Configuration + +Tiered storage is controlled at the topic level using Kafka-compatible configuration: + +- `remote.storage.enable=true` - Enable tiered storage for a topic (Kafka KIP-405) +- `local.retention.ms` - Time to keep in local tier before moving to remote +- `retention.ms` - Total retention across all tiers + +Example: 1 hour local, 6 days remote (7 days total): + +```bash +kafka-configs --bootstrap-server localhost:9092 \ + --entity-type topics --entity-name my-topic --alter \ + --add-config remote.storage.enable=true,local.retention.ms=3600000,retention.ms=604800000 +``` + +See [Topic Configuration]({{< relref "/integrate/korvet/kafka-api/topics#tiered-storage-configuration" >}}) for full details. + +Legacy single-stream topics can be migrated to the segmented layout in place — see +[Migrating a Single-Stream Topic to Segmented Storage]({{< relref "/integrate/korvet/storage/migrate-to-segmented" >}}). + +## Storage Layout + +Each topic partition maps to a Redis Stream, and each Kafka record is stored as a single stream entry +whose body breaks the record out into separate, directly-readable `value`, `key`, `headers`, and +`timestamp` fields. When tiered storage is enabled, a partition is split into per-segment streams that +are offloaded to the remote tier as they seal. For the exact stream-key patterns, segment keying, and +record field layout, see Redis Data Structures. + +## Benefits + +- **Performance**: Sub-millisecond latency for local tier operations +- **Simplicity**: Redis-only mode requires no additional infrastructure +- **Reliability**: Redis persistence ensures data durability +- **Scalability**: Handle millions of messages per second +- **Cost optimization**: Optional remote tier reduces storage costs for long-term retention +- **Flexibility**: Choose between simplicity (Redis-only) and cost optimization (tiered) + +## Next Steps + +- [Redis Streams (Local Tier)]({{< relref "/integrate/korvet/storage/redis-streams" >}}) +- [Remote Storage (Apache Iceberg)]({{< relref "/integrate/korvet/storage/remote-storage" >}}) +- [Storage Configuration]({{< relref "/integrate/korvet/reference/configuration" >}}) +- [Monitoring]({{< relref "/integrate/korvet/operations/monitoring" >}}) diff --git a/content/integrate/korvet/storage/migrate-to-segmented.md b/content/integrate/korvet/storage/migrate-to-segmented.md new file mode 100644 index 0000000000..7f2be82205 --- /dev/null +++ b/content/integrate/korvet/storage/migrate-to-segmented.md @@ -0,0 +1,76 @@ +--- +Title: Migrating a Single-Stream Topic to Segmented Storage +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Migrate a legacy single-stream topic to the segmented storage layout in + place by adding a segment roll policy. +linkTitle: Migrating to Segmented Storage +weight: 30 +--- + +Topics created by older Korvet releases (before segmentation became the default), or +created with an unlimited `segment.bytes=-1` and no `segment.ms`, use the backward-compatible +**single-stream layout**: the whole partition lives in one Redis Stream with no roll policy. Adding a +segment roll policy migrates such a topic to the **segmented layout** in place — no data is copied, +moved, or re-keyed, and the topic stays online and readable throughout. + +{{< note >}} +Topics created by current releases are already segmented (they default to `segment.bytes=134217728` (128 MiB)), +so this migration applies only to legacy single-stream topics. Compacted topics (`cleanup.policy=compact`) +always keep the single-stream layout and cannot be migrated. +{{< /note >}} + +## Trigger the migration + +Add a `segment.bytes` (and/or `segment.ms`) roll policy to the topic: + +```bash +kafka-configs --bootstrap-server localhost:9092 \ + --entity-type topics --entity-name my-topic --alter \ + --add-config segment.bytes=536870912 +``` + +{{< warning >}} +`segment.ms` must be smaller than the topic's effective retention (a segment has to roll before its +data expires), so on a topic with short retention prefer `segment.bytes` alone. `segment.bytes` itself +has no such constraint and can be added to any non-compacted topic. + +`segment.bytes` is enforced by the storage worker on each tick (`storage.worker.tick-interval`), by +comparing the open segment's real `MEMORY USAGE` against the budget — not on every append. It is thus +a tick-cadence bound rather than a hard per-append cap: a burst that writes several multiples of the +budget within a single tick interval seals as one oversized segment. Steady and slow producers stay +near the configured size; if you need a tighter bound under bursty load, lower the tick interval. +{{< /warning >}} + +## What happens + +The migration is **lazy**: it is performed on the next produce to or consume from the topic, not at the +moment the config is altered. An idle topic stays single-stream until it is next accessed. On that first +access: + +1. **Adopt in place** — the existing single stream becomes **open segment 0**. Its Redis key is left + unchanged (the segment keeps addressing the original stream key), so adoption is an O(1) metadata + operation with no `RENAME`, `COPY`, or re-write — safe even for very large partitions and on Redis + Cluster. +2. **Seal** — once the adopted segment exceeds the new roll policy (which the pre-existing data + typically already does), the background storage worker seals it. New writes then roll into freshly + keyed segments (`my-topic:0:1`, `:2`, …). +3. **Reclaim** — retention is applied per segment from the head of the log. With remote storage enabled, + sealed segments are offloaded to the remote tier and their local copy is then dropped. In a + Redis-only deployment the storage worker trims the oldest sealed segment's head progressively as + entries age past `retention.ms`, reclaiming memory continuously rather than only when the whole + (initially very large) adopted segment finally expires; the segment is dropped whole once its newest + entry also expires. + +The end state is a fully segmented topic: the original single stream is gone and all data is addressed +through per-segment streams. No consumer offsets are invalidated by the layout change itself — only the +normal retention rules apply. + +{{< tip >}} +Memory from the adopted segment is reclaimed one head-trim per storage-worker tick +(`korvet.storage.worker.tick-interval`, default 1 minute). Lower the tick interval if you need finer-grained +reclamation during a migration of a large topic. +{{< /tip >}} diff --git a/content/integrate/korvet/storage/redis-streams.md b/content/integrate/korvet/storage/redis-streams.md new file mode 100644 index 0000000000..223cada1c6 --- /dev/null +++ b/content/integrate/korvet/storage/redis-streams.md @@ -0,0 +1,279 @@ +--- +Title: Redis Streams Storage +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Korvet uses Redis Streams as its primary storage layer for all messages. +linkTitle: Redis Streams (Local Tier) +weight: 10 +--- + +Korvet uses Redis Streams as its primary storage layer for all messages. + +## Why Redis Streams? + +- **Low latency**: Sub-millisecond read/write performance +- **Consumer groups**: Built-in support for coordinated consumption +- **Persistence**: AOF and RDB for durability +- **Scalability**: Handle millions of messages per second + +## Stream Structure + +In **Redis-only** mode (the default), each topic partition maps to a single Redis Stream. When **tiered +storage** is enabled for the topic (`remote.storage.enable=true`), the partition is instead split into +**segment** streams that seal and roll over as they fill. + +For the exact stream-key patterns, segment keying, and record field layout, see +Redis Data Structures. + +### Segments (tiered storage only) + +Segments apply only to topics with tiered storage enabled; Redis-only topics use the single +per-partition stream described above. + +A tiered partition starts with a single open segment (`segmentId` `0`). The storage worker seals the +open segment and opens the next one (`1`, `2`, ...) when the open segment reaches its message-count +limit or its configured age. Sealing into discrete segments enables: + +- **Efficient retention**: Drop entire sealed segments when data expires +- **Archival**: Offload sealed segments to remote storage without blocking writes to the open segment +- **Memory tiering with Redis Flex**: Because each segment is a separate stream key, [Redis Flex (Auto Tiering)](https://redis.io/docs/latest/operate/rs/databases/auto-tiering/) can keep the open segment hot in RAM while transparently demoting colder sealed segments to flash/SSD — a **warm** tier between RAM and the remote object store. This is handled by Redis and needs no Korvet configuration. + +## Write Behavior + +A few local-tier write specifics worth knowing: + +- The broker assigns each stream entry ID (a timestamp-based `timestamp-sequence` value); it is **not** the `*` auto-generated ID. +- The `XADD` call sets only the entry ID and carries no retention arguments — retention is applied separately by the storage worker (see [Retention Policies](#retention-policies)). +- Null versus empty (for example a `null`-value tombstone versus an empty value) is encoded by stream-field **presence**: a field is omitted when its component is absent and present-but-empty when the component is a zero-length array. + +The record field layout itself is documented in Redis Data Structures. + +## At-Rest Compression + +Korvet can compress the `value` field before writing it to Redis. The codec is set per topic via +`storage.compression.type`, falling back to the server-level `korvet.storage.local.compression.codec` +(default `none`). The default stores values uncompressed and directly readable by non-Kafka clients. +When a codec is used, the value is written as that codec's +**standard frame format** with no Korvet-specific marker prefixed (`gzip`, the +[LZ4 frame](https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md), the +[Snappy framing format](https://github.com/google/snappy/blob/main/framing_format.txt), or a `zstd` +frame), so a non-Kafka client can decompress the field with any stock decompressor for that codec — +knowing only the topic's codec. + +`storage.compression.type` is fixed at topic creation. Because the stored value carries no codec marker, +the effective codec is pinned at creation: a topic that does not set its own `storage.compression.type` +snapshots the server-level default into its config, so a later change to the server default never +reinterprets a topic's already-written values. + +Supported codecs are `none`, `gzip`, `snappy`, `lz4`, and `zstd`. Use `none` when values must stay +directly readable from Redis with `XRANGE` or when payloads are small, random, or already compressed. +For JSON, logs, telemetry, and other repetitive text payloads, start with `zstd` for the best storage +reduction. Use `snappy` when CPU headroom is tighter and you want the lowest codec cost with good +compression. + +Compression is a trade-off, not a free win: produce pays the compression cost before `XADD`, and +fetch pays the decompression cost after `XREAD`. For compressible payloads, smaller stored values +can more than offset that CPU cost because Redis writes and reads less data. + +The following local benchmark uses a batch of 1,000 JSON records, each 1 KiB before compression. +Write latency is `compress 1,000 records + one pipelined batch of 1,000 XADD commands`; read latency +is `XREAD COUNT 1000 + decompress 1,000 records`. Values are median batch latencies from a local +Redis instance and are intended as directional guidance, not end-to-end broker latency. +The compressed write and read paths are faster in this JSON workload because the reduced Redis I/O +more than pays for codec CPU time. + +| Codec | Stored size | Total write latency | Write change vs none | Total read latency | Read change vs none | +|---|---|---|---|---|---| +| `none` | 100% | 29 ms | baseline | 7 ms | baseline | +| `lz4` | 30% | 15 ms | 48% lower | 5 ms | 34% lower | +| `snappy` | 31% | 13 ms | 57% lower | 4 ms | 48% lower | +| `zstd` | 22% | 10 ms | 64% lower | 4 ms | 49% lower | +| `gzip` | 23% | 15 ms | 49% lower | 5 ms | 36% lower | + +For JSON-like payloads around this size, `snappy` is the lowest-CPU choice, while `zstd` stores the +least data and produced the lowest total write/read batch latency in this run. For small, random, or +already-compressed payloads, compression can add CPU cost without reducing Redis I/O enough to pay +for it; leave those topics uncompressed. + +{{< note >}} +At-rest compression is independent of Kafka protocol compression (`compression.type`). At-rest compression applies to data stored in Redis, while protocol compression applies to data in transit between Kafka clients and Korvet. +{{< /note >}} + +## Retention Policies + +Korvet enforces retention with a background storage worker that issues `XTRIM`, not at produce time: + +- **`retention.ms`**: Time-based retention (default: 7 days) +- **`retention.bytes`**: Size-based retention (default: unlimited) +- **`compression.type`**: Compression type for fetch responses (none, gzip, snappy, lz4, zstd) + +### How Retention Works + +A leader-locked background storage worker periodically trims each stream with `XTRIM`: + +- **Count-based** (`retention.bytes`): `XTRIM MAXLEN `. The byte limit is converted to a message count by dividing by the topic's measured average message size (falling back to 1024 bytes when no measurement exists yet). +- **Time-based** (`retention.ms`): `XTRIM MINID `, where `minTimestamp` is the current time minus `retention.ms`. + +```bash +# Count-based trim +XTRIM korvet:storage:local:my-topic:0 MAXLEN 1000 + +# Time-based trim (keep messages newer than minTimestamp) +XTRIM korvet:storage:local:my-topic:0 MINID 1234567890000 +``` + +{{< note >}} +- Retention is enforced by the background storage worker, not at produce time. `XADD` carries no retention arguments. +- `XTRIM` uses exact trimming (no `~` / `LIMIT`) for predictable retention behavior. +- Size-based retention (`retention.bytes`) is converted to a message count using the topic's measured average message size. +{{< /note >}} + +### Configuring Retention + +Set retention when creating topics via Kafka Admin API: + +**Example: Create topic with retention configuration** + +```java +Properties props = new Properties(); +props.put("bootstrap.servers", "localhost:9092"); + +AdminClient admin = AdminClient.create(props); +NewTopic topic = new NewTopic("my-topic", 3, (short) 1); +topic.configs(Map.of( + "retention.ms", "86400000", // 1 day + "retention.bytes", "1073741824", // 1 GB + "compression.type", "lz4" // Compress fetch responses with LZ4 +)); +admin.createTopics(List.of(topic)); +``` + +Or configure defaults in `application.yml` via a catch-all pattern: + +```yaml +korvet: + topics: + - name: "*" + retention-time: 7d + retention-bytes: 10GB + compression: lz4 +``` + +### Manual Trimming + +For manual stream management, use Redis's `XTRIM` command: + +```bash +# Trim by count (keep last 1000 messages) +XTRIM korvet:storage:local:my-topic:0 MAXLEN 1000 + +# Trim by age (keep messages newer than the given timestamp) +XTRIM korvet:storage:local:my-topic:0 MINID +``` + +## Performance Tuning + +### Async Operations + +Korvet uses asynchronous Redis operations for maximum throughput: + +- All Redis commands use Lettuce's async API (`RedisFuture`) +- Operations return `CompletableFuture` to avoid blocking +- Multiple operations execute in parallel +- Netty event loop threads remain non-blocking + +**Benefits:** + +- Higher throughput with fewer threads +- Better resource utilization +- Reduced latency under load + +### Pipelining + +Korvet automatically batches Redis operations using Lettuce's command pipelining: + +- Multiple commands are batched together +- `setAutoFlushCommands(false)` delays command execution +- `flushCommands()` sends all commands in a single network round-trip +- Significantly improves throughput for high-volume producers + +**Example:** Producing 1000 messages sends 1000 `XADD` commands in a single pipeline instead of 1000 round-trips. + +- Configurable pool size based on workload + +### Compression Types + +Korvet supports two types of compression: + +#### 1. At-Rest Compression + +Compresses the stored record value at rest in Redis Streams: + +- **Where**: Applied to the `value` field in Redis Streams +- **When**: At write time (produce) and read time (fetch) +- **Configuration**: Per topic via `storage.compression.type` (create-time only), falling back to the server-level `korvet.storage.local.compression.codec` (default `none`). The effective codec is pinned at topic creation. Set a topic to `none` to keep values directly readable by non-Kafka clients. +- **Use case**: Reduce Redis memory usage for large or repetitive data + +**How it works:** + +- Producer writes: The value is compressed before storing in Redis, as the topic codec's standard frame format with no Korvet-specific marker +- Consumer reads: The value is decompressed when fetching from Redis +- The codec is pinned per topic at creation, so the value field is always a plain standard frame a non-Kafka client can decompress with any stock decompressor for that codec +- Transparent to Kafka clients - they receive uncompressed data +- Independent of Kafka protocol compression + +**Benefits:** + +- Reduces Redis memory usage +- Lower storage costs +- Faster Redis persistence (smaller AOF/RDB files) +- No impact on Kafka client compatibility + +See [At-Rest Compression](#at-rest-compression) for configuration details. + +#### 2. Protocol Compression (Kafka Standard) + +Compresses Kafka protocol messages between clients and Korvet: + +- **Where**: Applied to Kafka Fetch/Produce request/response payloads +- **When**: During network transmission +- **Configuration**: `compression.type` topic config (none, gzip, snappy, lz4, zstd) +- **Use case**: Reduce network bandwidth between Kafka clients and Korvet + +**How it works:** + +- **Producer side**: Kafka clients can send compressed or uncompressed batches; Korvet decompresses them before storing +- **Consumer side**: Korvet compresses fetch responses based on topic's `compression.type` configuration +- **Storage**: Messages are stored uncompressed in Redis (unless at-rest compression is enabled) + +**Benefits:** + +- Reduces network bandwidth for fetch responses +- Transparent to clients - works with all Kafka clients +- Flexible per-topic configuration +- Standard Kafka feature + +#### Compression Comparison + +| Feature | At-Rest Compression | Protocol Compression | +|---|---|---| +| **Purpose** | Reduce Redis memory usage | Reduce network bandwidth | +| **Applied at** | Redis storage layer | Kafka protocol layer | +| **Configuration** | `korvet.storage.local.compression.codec` | `compression.type` | +| **Affects** | Redis memory, persistence | Network traffic | +| **Transparent to** | Kafka clients | Storage layer | +| **Recommended for** | JSON/log payloads with `zstd`; CPU-sensitive topics with `snappy` | High-throughput consumers | + +{{< tip >}} +You can use both types of compression together! For example, set `korvet.storage.local.compression.codec=zstd` to save Redis memory for JSON/log topics and `compression.type=lz4` for fast network compression. +{{< /tip >}} + +See [Compression]({{< relref "/integrate/korvet/kafka-api/compatibility#compression" >}}) for protocol compression configuration details. + +## Next Steps + +- [Monitoring]({{< relref "/integrate/korvet/operations/monitoring" >}}) +- [Configuration Reference]({{< relref "/integrate/korvet/reference/configuration" >}}) diff --git a/content/integrate/korvet/storage/remote-storage.md b/content/integrate/korvet/storage/remote-storage.md new file mode 100644 index 0000000000..13643bcf64 --- /dev/null +++ b/content/integrate/korvet/storage/remote-storage.md @@ -0,0 +1,226 @@ +--- +Title: Remote Storage (Apache Iceberg on object store) +alwaysopen: false +categories: +- docs +- integrate +- korvet +description: Korvet includes a built-in storage worker that offloads sealed Redis stream + segments to Apache Iceberg tables backed by Parquet files on S3. +linkTitle: Remote Storage (Iceberg) +weight: 20 +--- + +Korvet includes a built-in storage worker that offloads sealed Redis stream segments to Apache Iceberg tables backed by Parquet files on S3. + +{{< note >}} +In Korvet tiered storage terminology, the **local tier** uses Redis Streams and the **remote tier** is a set of Apache Iceberg tables, one per topic. Korvet uses local/remote in its configuration and APIs. +{{< /note >}} + +## Overview + +The remote tier holds one Apache Iceberg table per topic, named after the topic with separator characters (`.`, `_`, `:`) rewritten to `_` (topic `orders.created.v1` becomes table `orders_created_v1`). All partitions of a topic share its table. The storage worker: + +1. Reads sealed LOCAL segments from each topic-partition's segment metadata in Redis +2. Streams the segment contents from Redis (one page resident at a time) into an Iceberg data file +3. Appends and commits the data file to the Iceberg table, then marks the Redis segment as offloaded +4. Runs in-process as part of the Korvet server; there is no separate archive daemon + +Each topic's table is a standard Iceberg table: external query engines (Spark, Trino, Athena, DuckDB, ...) can read it directly. Iceberg manages its own metadata and manifest files under the table location. Each offloaded write commits as one Iceberg append; if the storage worker crashes before the commit, the Redis segment stays sealed-but-not-offloaded and the next scan re-streams the (still untouched) Redis segment. + +## Configuration + +Enable remote storage by setting `korvet.storage.remote.path` in your `application.yml`: + +```yaml +korvet: + storage: + remote: + path: s3://my-bucket/korvet + s3: + region: us-west-1 + worker: + tick-interval: 1m +``` + +For local development and tests, point the cold tier at a local directory instead of an object store. No `s3.*` settings are needed: + +```yaml +korvet: + storage: + remote: + path: file:///var/lib/korvet/cold +``` + +{{< note >}} +Setting `korvet.storage.remote.path` makes the cold tier available. The leader-locked storage worker is enabled by default and rolls eligible segments, offloads sealed segments, and enforces local and remote retention. Topics are archived only when they also have `remote.storage.enable=true`. +{{< /note >}} + +### Storage Properties + +| Property | Default | Description | +|---|---|---| +| `korvet.storage.remote.path` | *required* | Cold-tier root URI. Supports `s3://` for object storage and `file://` for a local-filesystem warehouse (useful for local development and tests without MinIO/S3). Iceberg resolves the `FileIO` from the URI scheme. | +| `korvet.storage.remote.s3.region` | *unset* | AWS region for the S3 object store. | +| `korvet.storage.remote.s3.endpoint` | *unset* | Optional endpoint URL for S3-compatible stores such as MinIO or LocalStack. | +| `korvet.storage.remote.s3.path-style-access` | *unset* | Use path-style addressing. Required for most non-AWS S3-compatible stores. | +| `korvet.storage.remote.s3.access-key-id` | *unset* | Static access-key id. Prefer IAM roles in production. | +| `korvet.storage.remote.s3.secret-access-key` | *unset* | Static secret access key. Prefer IAM roles in production. | + +### Maintenance Properties + +| Property | Default | Description | +|---|---|---| +| `korvet.storage.worker.enabled` | `true` | Enables the storage worker in this JVM. | +| `korvet.storage.worker.tick-interval` | `1m` | Tick cadence for the storage worker loop. | +| `korvet.storage.worker.lease-duration` | `2m` | Redis leader-lock lease duration. Must exceed `tick-interval`. | + +### Per-Topic Retention Configuration + +Control when data moves from the local tier to the remote tier using topic-level configuration: + +| Configuration | Default | Description | +|---|---|---| +| `remote.storage.enable` | `false` | Enable tiered storage for this topic (Kafka KIP-405 standard). | +| `local.retention.ms` | `-2` | Time to keep in the local tier. `-2` means use total `retention.ms` (Kafka KIP-405). | +| `local.retention.bytes` | `-2` | Size to keep in the local tier. `-2` means use total `retention.bytes` (Kafka KIP-405). | +| `retention.ms` | `604800000` (7 days) | Total retention across all tiers. | + +Remote-tier retention is implicit: `retention.ms - local.retention.ms`. + +**Example**: Keep 1 day local and the rest remote (1 year total): + +```bash +kafka-configs --bootstrap-server localhost:9092 \ + --entity-type topics --entity-name my-topic --alter \ + --add-config remote.storage.enable=true,retention.ms=31536000000,local.retention.ms=86400000 +``` + +## Storage Format + +### Table Layout + +The remote tier holds one Iceberg table per topic, under an Iceberg namespace named after +`korvet.namespace` (default `korvet`) so instances sharing a warehouse keep separate tables. +Each table is partitioned by `stream_key` +(identity) — so each topic partition's rows land in their own Iceberg partition — and sorted by +`message_ts`. Iceberg owns the on-disk layout under `korvet.storage.remote.path`: a `metadata/` +directory for table metadata and manifests, plus Parquet data files. Each offloaded segment is written +as one data file named: + +``` +segment--.parquet +``` + +The trailing UUID keeps each write unique so a retried offload never collides with a file left by a +prior failed attempt. + +### Iceberg Schema + +Each row is one message. The hot-tier stream entry is decoded into analytics-friendly columns so external +engines can query the key, value, headers, and timestamps directly. The Iceberg schema is: + +``` +required STRING stream_key; +required LONG segment_id; +required STRING message_id; +required LONG message_ts; +required LONG kafka_timestamp; +optional BINARY key; +optional BINARY value; +optional LIST>; +``` + +| Column | Type | Description | +|---|---|---| +| `stream_key` | STRING | Local stream key of the source partition (the table partition column). | +| `segment_id` | LONG | Sealed segment number the message came from. | +| `message_id` | STRING | Full Redis stream message id (e.g. `1708956789000-0`). | +| `message_ts` | LONG | Redis stream-id millisecond component (append-time ordering); the table sort key. | +| `kafka_timestamp` | LONG | Producer record timestamp in epoch milliseconds, or `-1` when none was preserved. | +| `key` | BINARY | Record key, decoded from the stream entry. Null when the record has no key. | +| `value` | BINARY | Record value, decoded from the stream entry. Null for a tombstone. | +| `headers` | LIST\ | Record headers, preserving order and duplicate keys. Each entry has a required `header_key` (STRING) and an optional `header_value` (BINARY). | + +### Compression + +Data files use Iceberg's default Parquet compression. Row-group and target file sizes are configurable via `korvet.storage.remote.iceberg.row-group-size` and `korvet.storage.remote.iceberg.target-file-size` (both default `128MB`). + +## AWS Authentication + +The cold tier writes through Iceberg's `S3FileIO`. Configure common S3 settings under `korvet.storage.remote.s3`; otherwise the AWS SDK default credential provider chain is used. + +### Credential providers + +Common production choices: + +| Provider | Use when | +|---|---| +| EC2, ECS, or EKS node roles | Leave static credentials unset and let the AWS SDK use instance metadata. | +| EKS IAM Roles for Service Accounts (IRSA) | Leave static credentials unset. `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE` are injected into the pod and picked up by the SDK. | +| Static access key + secret | Use `korvet.storage.remote.s3.access-key-id` and `korvet.storage.remote.s3.secret-access-key` for development or S3-compatible stores. | + +### IRSA (EKS IAM Roles for Service Accounts) + +```yaml +korvet: + storage: + remote: + path: s3://my-bucket/korvet + s3: + region: us-east-1 +``` + +When IRSA is configured on the cluster, `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE` are injected into the pod and picked up automatically. + +### IAM role (EC2/ECS/EKS nodes) + +```yaml +korvet: + storage: + remote: + path: s3://my-bucket/korvet + s3: + region: us-west-1 +``` + +### Static credentials (dev / MinIO) + +```yaml +korvet: + storage: + remote: + path: s3://my-bucket/korvet + s3: + region: us-west-1 + endpoint: http://localhost:9000 + path-style-access: true + access-key-id: minioadmin + secret-access-key: minioadmin +``` + +## Performance + +The storage worker achieves high throughput when archiving to same-region S3: + +| Configuration | Throughput | Notes | +|---|---|---| +| Single stream | ~32,000 msg/s | Baseline | +| 4 streams (parallel) | ~115,000 msg/s | Near-linear scaling | + +See [Remote Storage Benchmarks]({{< relref "/integrate/korvet/operations/benchmarks#remote-storage-archival-benchmark" >}}) for detailed results. + +### Performance tips + +- **Same-region S3**: deploy Korvet in the same AWS region as your bucket. +- **Multiple partitions**: archival parallelism is per (topic, partition); more partitions = more archive concurrency. +- **Segment size**: larger sealed segments produce larger Parquet files with better compression and fewer object-store operations. + +## Next Steps + +- [Storage Overview]({{< relref "/integrate/korvet/storage" >}}) +- [Redis Streams (Local Tier)]({{< relref "/integrate/korvet/storage/redis-streams" >}}) +- [Remote Storage Benchmarks]({{< relref "/integrate/korvet/operations/benchmarks#remote-storage-archival-benchmark" >}}) +- [Kubernetes Deployment]({{< relref "/integrate/korvet/operations/kubernetes" >}}) From c6d931453568d7262acaa25d0baf4234877a156d Mon Sep 17 00:00:00 2001 From: Josh Rotenberg Date: Wed, 26 Aug 2026 13:50:46 -0700 Subject: [PATCH 3/4] Add type/group/summary to Korvet section indexes Every section _index.md in an integrate product tree carries type: integration, group, and summary (see the RDI subtree) so nested section pages do not fall through to the integrate card-grid template, which requires a non-nil group on child sections and fails the build otherwise (integrate/list.html: index of $labelColors with nil). --- content/integrate/korvet/kafka-api/_index.md | 4 ++++ content/integrate/korvet/operations/_index.md | 4 ++++ content/integrate/korvet/quick-start/_index.md | 4 ++++ content/integrate/korvet/reference/_index.md | 4 ++++ content/integrate/korvet/reference/metrics/_index.md | 4 ++++ content/integrate/korvet/storage/_index.md | 4 ++++ 6 files changed, 24 insertions(+) diff --git a/content/integrate/korvet/kafka-api/_index.md b/content/integrate/korvet/kafka-api/_index.md index cda291a5ce..2b9fbdec50 100644 --- a/content/integrate/korvet/kafka-api/_index.md +++ b/content/integrate/korvet/kafka-api/_index.md @@ -7,8 +7,12 @@ categories: - korvet description: Use existing Kafka clients and tools with Korvet's implementation of the Kafka protocol. +group: service hideListLinks: false linkTitle: Using the Kafka API +summary: Korvet provides a Kafka-compatible API backed by Redis Streams, so you + can use existing Kafka clients and tools with Redis as the storage engine. +type: integration weight: 30 --- diff --git a/content/integrate/korvet/operations/_index.md b/content/integrate/korvet/operations/_index.md index f74cc82c8e..3864507181 100644 --- a/content/integrate/korvet/operations/_index.md +++ b/content/integrate/korvet/operations/_index.md @@ -6,8 +6,12 @@ categories: - integrate - korvet description: Deploy, monitor, and operate Korvet in production. +group: service hideListLinks: false linkTitle: Operations +summary: Korvet provides a Kafka-compatible API backed by Redis Streams, so you + can use existing Kafka clients and tools with Redis as the storage engine. +type: integration weight: 50 --- diff --git a/content/integrate/korvet/quick-start/_index.md b/content/integrate/korvet/quick-start/_index.md index b73dc255a3..f1976bfbca 100644 --- a/content/integrate/korvet/quick-start/_index.md +++ b/content/integrate/korvet/quick-start/_index.md @@ -6,8 +6,12 @@ categories: - integrate - korvet description: Install Korvet, run the built-in demo, and connect your own Kafka client. +group: service hideListLinks: false linkTitle: Get Started +summary: Korvet provides a Kafka-compatible API backed by Redis Streams, so you + can use existing Kafka clients and tools with Redis as the storage engine. +type: integration weight: 10 --- diff --git a/content/integrate/korvet/reference/_index.md b/content/integrate/korvet/reference/_index.md index bda2c098a2..38385cba37 100644 --- a/content/integrate/korvet/reference/_index.md +++ b/content/integrate/korvet/reference/_index.md @@ -6,8 +6,12 @@ categories: - integrate - korvet description: Reference documentation for Korvet configuration, APIs, and metrics. +group: service hideListLinks: false linkTitle: Reference +summary: Korvet provides a Kafka-compatible API backed by Redis Streams, so you + can use existing Kafka clients and tools with Redis as the storage engine. +type: integration weight: 60 --- diff --git a/content/integrate/korvet/reference/metrics/_index.md b/content/integrate/korvet/reference/metrics/_index.md index 4abbb8eff0..3c45892889 100644 --- a/content/integrate/korvet/reference/metrics/_index.md +++ b/content/integrate/korvet/reference/metrics/_index.md @@ -6,8 +6,12 @@ categories: - integrate - korvet description: Korvet exposes metrics in Prometheus format via Spring Boot Actuator. +group: service hideListLinks: false linkTitle: Metrics +summary: Korvet provides a Kafka-compatible API backed by Redis Streams, so you + can use existing Kafka clients and tools with Redis as the storage engine. +type: integration weight: 30 --- diff --git a/content/integrate/korvet/storage/_index.md b/content/integrate/korvet/storage/_index.md index 19e603b3fe..bbac1ecbdf 100644 --- a/content/integrate/korvet/storage/_index.md +++ b/content/integrate/korvet/storage/_index.md @@ -8,8 +8,12 @@ categories: description: Korvet stores all messages in Redis Streams (the local tier) and can optionally archive older data to Apache Iceberg tables on an object store (the remote tier). +group: service hideListLinks: false linkTitle: Tiered Storage +summary: Korvet provides a Kafka-compatible API backed by Redis Streams, so you + can use existing Kafka clients and tools with Redis as the storage engine. +type: integration weight: 40 --- From 2608df4bfcdff14af3b936b75d29517a6ce2ab4b Mon Sep 17 00:00:00 2001 From: Josh Rotenberg Date: Wed, 26 Aug 2026 14:32:40 -0700 Subject: [PATCH 4/4] Set null description on the Korvet landing page The theme renders the description above the body, and the body's first line is the same sentence; the RDI landing page uses a null description for the same reason. --- content/integrate/korvet/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/korvet/_index.md b/content/integrate/korvet/_index.md index f250ac0203..2dd61da13d 100644 --- a/content/integrate/korvet/_index.md +++ b/content/integrate/korvet/_index.md @@ -5,7 +5,7 @@ categories: - docs - integrate - korvet -description: Korvet is a Kafka-compatible streaming service backed by Redis Streams. +description: null group: service hideListLinks: false linkTitle: Korvet