OpenRaft

Advanced Raft in 🦀 Rust on any async runtime. Please ⭐ on GitHub!

This project intends to improve raft as the next-generation consensus protocol for distributed data storage systems (SQL, NoSQL, KV, Streaming, Graph ... or maybe something more exotic).

Currently, openraft is the consensus engine of meta-service cluster in databend.

• 🚀 Get started:

  • OpenRaft guide is the best place to get started,
  • OpenRaft docs for more in-depth details,
  • OpenRaft FAQ explains some common questions.
  • OpenRaft on DeepWiki provides detailed architectural documentation to help understand OpenRaft internals.

• 💡 Example Applications:

  • Examples with OpenRaft 0.10 require OpenRaft 0.10.0-alpha.30 (alpha) on crates.io/openraft;
  • Examples with OpenRaft 0.9 require OpenRaft 0.9 on crates.io/openraft.

• 🙌 Questions?

  • Why not take a peek at our FAQ? You might find just what you need.
  • Wanna chat? Come hang out with us on Discord!
  • Or start a new discussion over on GitHub.
  • Or join our Feishu group.
  • And hey, if you're on WeChat, add us: drmingdrmer. Let's get the conversation started!

Whatever your style, we're here to support you. 🚀 Let's make something awesome together!

• OpenRaft is derived from async-raft with several bugs fixed: Fixed bugs.

Status

• The features are almost complete for building an application.
• Performance: Supports 33,000 writes/sec for a single writer and 5,615,000 writes/sec with batch writes. See: Performance
• Unit test coverage stands at 92%.
• The chaos test has not yet been completed, and further testing is needed to ensure the application's robustness and reliability.

API status

• OpenRaft API is not stable yet. Before 1.0.0, an upgrade may contain incompatible changes. Check our change-log. A commit message starts with a keyword to indicate the modification type of the commit:

  • DataChange: on-disk data types changes, which may require manual upgrade.
  • Change: if it introduces incompatible changes.
  • Feature: if it introduces compatible non-breaking new features.
  • Fix: if it just fixes a bug.

Versions

• Branch main has been under active development. The main branch is for the release-0.10. Latest alpha on crates.io: v0.10.0-alpha.30. The 0.10 line is in alpha — the API may still change before the final 0.10.0 release.

• Branch release-0.9: Latest: ( v0.9.0 | Change log ); Upgrade guide: ⬆️ 0.8 to 0.9; release-0.9 Won't accept new features but only bug fixes.

• Branch release-0.8: Latest: ( v0.8.8 | Change log ); Upgrade guide: ⬆️ 0.7 to 0.8, ⬆️ 0.8.3 to 0.8.4; release-0.8 Won't accept new features but only bug fixes.

• Branch release-0.7: Latest: ( v0.7.6 | Change log ); Upgrade guide: ⬆️ 0.6 to 0.7; release-0.7 Won't accept new features but only bug fixes.

• Branch release-0.6: Latest: ( v0.6.8 | Change log ); release-0.6 won't accept new features but only bug fixes.

Roadmap

• 2022-10-31 Extended joint membership
• 2023-02-14 Minimize confliction rate when electing; See: OpenRaft Vote design; Or use standard Raft leader-ID mode.
• 2023-04-26 Goal performance is 1,000,000 put/sec.
• Reduce the complexity of vote and pre-vote: get rid of pre-vote RPC;
• Support flexible quorum, e.g.: Hierarchical Quorums
• Consider introducing read-quorum and write-quorum, improve efficiency with a cluster with an even number of nodes.

Performance

The benchmark is focused on the OpenRaft framework itself and is run on a minimized store and network. This is NOT a real world application benchmark!!!

Single writes:

clients	put/s
4096	3,548,000
1024	3,006,000
256	1,808,000
64	912,000
1	33,000

Batch writes (4 entries per batch):

clients	put/s
4096	5,615,000

For benchmark detail, go to the ./benchmarks/minimal folder.

Features

What sets OpenRaft apart from a standard Raft implementation:

• Generalized membership change: extended joint membership changes an arbitrary set of nodes in a single operation; standard Raft's one-node-at-a-time change is a restricted special case.
• Fewer election conflicts: the redesigned Vote minimizes election conflict rate — a split vote does not force a new term; the standard Raft leader-ID mode is also supported.
• Async and event-driven: state transitions are driven by Raft events without periodic ticks; messages are batched, reaching millions of writes/sec in the framework benchmark.
• Fully pluggable: storage (RaftLogStorage, RaftStateMachine), networking (RaftNetworkV2, including application-defined snapshot transport), and every core type — node ID, node, term, vote, log entry — via RaftTypeConfig.
• Runtime-agnostic: tokio by default, compio and monoio via the AsyncRuntime trait, and a single-threaded mode that removes Send bounds.
• Deterministically tested: a turmoil-based simulation fuzzer steps the cluster tick by tick and verifies Raft invariants from the paper and the TLA+ spec after every tick.
• Observable: streaming metrics via watch channels, and tracing instrumentation with compile-time verbosity filtering.

Functionality:

• ✅ Leader election: by policy or manually (Trigger::elect()).
• ✅ Leader transfer: Trigger::transfer_leader().
• ✅ Pre-vote: avoid unnecessary term increments by enabling Config::enable_pre_vote.
• ✅ Non-voter(learner) Role: refer to add_learner().
• ✅ Log Compaction(snapshot of state machine): by policy or manually (Trigger::snapshot()).
• ✅ Snapshot replication.
• ✅ Dynamic Membership: using joint membership config change. Refer to dynamic membership
• ✅ Linearizable read: ensure_linearizable().
• ✅ Metrics: Raft::metrics(), Raft::data_metrics(), and Raft::server_metrics().
• ⛔️ Won't support: Single-step config change. Single-step membership change is a restricted subset of joint consensus that only allows changing one node at a time. Openraft uses the more general joint consensus approach which supports arbitrary membership changes in a single operation.
• ✅ Toggle heartbeat / election: RuntimeConfigHandle::heartbeat() / RuntimeConfigHandle::elect().
• ✅ Trigger snapshot / election manually: Trigger::snapshot() / Trigger::elect().
• ✅ Purge log by policy or manually: Trigger::purge_log().

Who uses it

• Databend - The Next-Gen Cloud [Data+AI] Analytics
• Walrus - Distributed message streaming engine with Raft-based metadata coordination.
• CnosDB - A cloud-native open source distributed time series database.
• RobustMQ - Next generation cloud-native converged message queue.
• RocketMQ-rust - Apache RocketMQ reimplemented in Rust.
• Hiqlite - Highly-available, embeddable, Raft-based SQLite with caching.
• Octopii - A distributed systems kernel for building replicated, fault-tolerant services.
• Renegade - On-chain dark pool for anonymous trading.
• Helyim - SeaweedFS implemented in pure Rust.
• Ahnlich - In-memory vector database with an AI proxy.
• raymondshe/matchengine-raft - A example to demonstrate how openraft persists snapshots/logs to disk.
• tsoracle - Distributed timestamp and sequence oracle serving strictly monotonic, gapless IDs.
• yuyang0/rrqlite - A rust implementation of rqlite.

📣 Using openraft in your project? We'd love to feature it here — open an issue to let us know, or skip the wait and submit a pull request directly.

Contributing

Check out the CONTRIBUTING.md guide for more details on getting started with contributing to this project.

Contributors

Thanks to everyone who contributes to OpenRaft:

License

OpenRaft is licensed under the terms of the MIT License or the Apache License 2.0, at your choosing.