Why · Verify · Run · Results · Docs

kafka-benchmarks compares Apache Kafka clients and produces sealed, checksummed evidence. It currently focuses on the public Kafkars producer against raw librdkafka.

Why

A benchmark here is more than a timing:

• Clients run as isolated processes.
• An independent verifier checks what reached Kafka.
• Crashes, timeouts, and interrupts still seal evidence.
• Results include validity, repetition, noise, and provenance.
• Sealed bundles are immutable.

Verify

git clone https://github.com/kafkars/benchmarks
cd benchmarks
cargo +1.96.0 install zcheck --version 0.0.2 --locked
cargo +1.96.0 install zrail --version 0.0.1 --locked
zcheck

zcheck is the review gate. It records one receipt across formatting, linting, tests, docs, schemas, zrail policy, detached-adapter policy, legacy tests, and provenance checks. No Kafka cluster or sibling checkout is required.

Run

Real-client runs need Docker. The native adapter uses the exact published RC.2 and its locked registry closure; these clean source checkouts provide strict provenance correspondence, not build inputs. The scripts build pinned librdkafka.

kafkars/
kafka-driver/
kafka-protocol/   # the kafkars/kafka-wire repository

Start the local three-broker cluster and run five paired attempts:

docker compose -f clusters/dev-compose/compose.yml up -d --wait

bootstrap=127.0.0.1:39092,127.0.0.1:39093,127.0.0.1:39094
scripts/bench-suite-acceptance "$bootstrap" 5

The script builds both adapters, generates machine-local configuration, runs the suite, verifies every bundle, and prints the report directory. Output stays under target/suite-acceptance/.

For one attempt:

scripts/bench-m0-acceptance

For a continuous producer soak, start the cluster with docker compose -p kafkars-endurance-local -f clusters/dev-compose/compose.yml up -d --wait, then run scripts/run-producer-endurance --seconds 3600 --compose-project kafkars-endurance-local --output target/endurance/smoke and scripts/verify-producer-endurance target/endurance/smoke. After that gate passes, use --seconds 86400 and a new output directory locally. The opt-in Actions workflow permits only one- or four-hour lifetimes, one job at a time; segmented runs do not establish continuous 24-hour endurance.

For the qualified source preview, select subject: qualified-candidate in that existing workflow; paired-suite: true adds five diagnostic paired attempts after the smoke passes. The lab rebuilds and checks the three exact package hashes from dda2545 qualification, leaves registry mode unchanged, and seals the source/adapter locks and observed binary proof externally. This does not describe the older published RC.2 or establish production/performance support.

Read the results

Suite reports lead with the conclusion:

Section	Meaning
Result	paired ratios, confidence intervals, and favorable/worse/unresolved outcomes
Scorecard	each client's median goodput, latency, CPU, and memory
Checks	whether the evidence and comparison rules passed
Run stability	how much subjects and ratios varied
Request economics	requests, batches, retries, timeouts, and wire efficiency

Every report states whether it may support a public claim. All current scenarios set claim_eligible = false; laptop and shared-runner numbers are diagnostic only.

Commands

Command	Purpose
benchctl resolve	show the exact experiment without touching Kafka
benchctl run	run one attempt and print its sealed bundle
benchctl suite	run paired, alternating repetitions
benchctl capacity	search an offered rate against declared objectives
benchctl pack	run a reviewed scenario manifest
benchctl report	render one sealed attempt
benchctl packet	verify generated prose against an analysis packet

Repository

Path	Purpose
crates/	schemas, control plane, verification, reporting, and the C shim
adapters/	standalone subject adapters
scenarios/	workloads and cadence packs
schemas/ and conformance/	versioned contracts and byte-exact vectors
docs/	design, evidence semantics, boundaries, and roadmap
scripts/	qualification tasks and benchmark entry points

Documentation

• Performance contract
• Evidence format
• Architecture
• Harness design
• Roadmap
• Deferred scenarios
• Zrail gaps

Boundaries

This repository is not a leaderboard, a Kafka client, or a benchmark of private internals. Results describe one experiment on one environment. Adapters use shipped public client APIs and never decide their own validity.

Status

Pre-0.1. The v2 engine supports paired suites, capacity search, bounded histograms, four-timestamp latency accounting, sealed reports, and exact public Kafkars request and batch counters. Nothing is published.

Project

Changelog · Contributing · Security · Releasing

License

Apache-2.0. See LICENSE and NOTICE.

Apache Kafka is a trademark of the Apache Software Foundation. This project is independent and is not endorsed by the Apache Software Foundation.