Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
keywords: [disaster recovery, GreptimeDB, DR solutions, backup and restore, active-active failover, cross-region deployment, RTO, RPO]
description: Overview of disaster recovery (DR) solutions in GreptimeDB, including basic concepts, component architecture, and various DR solutions such as standalone, active-active failover, cross-region deployment, and backup & restore.
keywords: [disaster recovery, GreptimeDB, DR solutions, backup and restore, single-region deployment, region failover, active-active failover, cross-region deployment, RTO, RPO]
description: Overview of disaster recovery (DR) solutions in GreptimeDB, including basic concepts, component architecture, and various DR solutions such as standalone, single-region deployment in a single cluster, active-active failover, cross-region deployment, and backup & restore.
---

# Disaster Recovery
Expand Down Expand Up @@ -74,6 +74,35 @@ The node is not fully stateless, though: a standalone instance keeps its metadat

**RPO=0** and an **RTO in minutes** are the design targets of this topology. They hold as long as the Kafka cluster and the object storage both survive the failure you are planning for, the WAL covering unflushed writes is still present, and the metadata can be restored. Verify them with a failure drill against your own deployment.

### DR solution based on single-region deployment in a single cluster

![Single-region-single-cluster](/Single-region-single-cluster.svg)

Before a cluster spans regions, it has to survive the loss of a single node or of a single AZ inside one region. An AZ here is the same logical unit of disaster recovery used in the cross-region solutions: a data center, or a compartment of a data center. This topology is the common production baseline, and the cross-region solutions below are built on top of it.

The cluster lives in one region, with its roles spread over the AZs that region offers:

* **Frontend** is stateless. Run several replicas behind a load balancer so that a failed replica only affects the requests in flight on it. See [Role Replicas Configuration](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md#role-replicas-configuration).
* **Metasrv** runs as several replicas that elect a leader; losing a follower is transparent, and losing the leader costs a re-election. The metadata itself lives in an external backend — etcd, MySQL or PostgreSQL — that Metasrv does not replicate for you, so give that backend its own high availability and its own backups. See [Metadata Management](/user-guide/deployments-administration/manage-metadata/overview.md) and [Metadata Export & Import](/user-guide/deployments-administration/disaster-recovery/back-up-&-restore-meta-data.md).
* **Datanode** holds the regions. When one fails, [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) reopens its regions on the surviving Datanodes. It is **disabled by default**, and it requires [shared storage](/user-guide/deployments-administration/configuration.md#storage-options) together with remote WAL. Enabling it on local WAL with `allow_region_failover_on_local_wal=true` is possible but may lose data, because the WAL of the failed node stays on that node's disk.
* **Kafka** (remote WAL) and the **object storage** hold the state that has to outlive the node. Give Kafka a replication factor that tolerates the broker loss you are planning for, and spread brokers and Datanodes over the AZs instead of packing them into one.

Latencies:
- Same-region round trips only; neither writes nor replication pay a cross-region penalty

Supports High Availability:
- A single node is unavailable with almost the same performance, once its regions are reopened elsewhere
- A single AZ is unavailable with degraded performance, unless the surviving AZs were sized to absorb its share of the load
- The region itself is not covered; it is a single failure domain in this topology

This solution targets zero RPO and a minute-level RTO for a node or AZ failure. As with the other solutions, the numbers depend on conditions you have to check:

- Region Failover is **disabled by default** and must be enabled explicitly.
- The surviving Datanodes need spare capacity to carry the failed node's regions, otherwise failover only moves the overload.
- The recovery time is dominated by WAL replay: the more regions share one Kafka topic, the more redundant data has to be read before those regions serve again. Read [The recovery time of Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md#the-recovery-time-of-region-failover) for the model behind that.

Confirm the resulting RPO and RTO with a failure drill. If the region as a whole, the Kafka cluster or the object storage becomes unavailable, you need one of the solutions below, or regular backups kept in another region.

### DR solution based on Active-Active Failover

![Active-active failover](/active-active-failover.png)
Expand Down Expand Up @@ -126,6 +155,7 @@ By comparing these DR solutions, you can decide on the final option based on the
| DR solution | Error Tolerance Objective | RPO | RTO | TCO | Scenarios | Remote WAL & Object Storage | Notes |
| ------------- | ------------------------- | ----- | ----- | ----- | ---------------- | --------- | --------|
| DR solution for Standalone| Single-Region | Backup Interval | Minute or Hour level | Low | Low requirements for availability and reliability in small scenarios | Optional | |
| DR solution based on single-region deployment in a single cluster | Single-Region, node and AZ level | 0 | Minute level | Medium | The common production baseline for a cluster that lives in one region | Required | Region Failover is disabled by default |
| DR solution based on Active-Active Failover | Cross-Region | Depends on pending changes and the failure mode | Depends on external failover | Low | High requirements for availability and reliability in small-to-medium scenarios | Optional | Commercial feature |
| DR solution based on cross-region deployment in a single cluster| Multi-Regions | 0 | Minute level | High | High requirements for availability and reliability in medium-to-large scenarios | Required | |
| DR solution based on BR | Single-Region | Backup Interval | Minute or Hour level | Low | Acceptable requirements for availability and reliability | Optional | |
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
keywords: [灾难恢复, DR 解决方案, 备份与恢复, RTO, RPO, 组件架构, 双活互备, 跨区域部署, 数据恢复]
description: 介绍 GreptimeDB 的灾难恢复(DR)解决方案,包括基本概念、组件架构、不同的 DR 解决方案及其比较。
keywords: [灾难恢复, DR 解决方案, 备份与恢复, RTO, RPO, 组件架构, 单区域部署, Region Failover, 双活互备, 跨区域部署, 数据恢复]
description: 介绍 GreptimeDB 的灾难恢复(DR)解决方案,包括基本概念、组件架构、单集群单区域部署等不同的 DR 解决方案及其比较。
---

# 灾难恢复
Expand Down Expand Up @@ -83,6 +83,35 @@ GreptimeDB 将数据存储在对象存储(如 [AWS S3](https://docs.aws.amazon

RPO=0 和分钟级 RTO 是该拓扑的设计目标,成立需要三个前提:Kafka 集群与对象存储都在你要防范的故障中幸存、尚未 flush 的那部分写入所对应的 WAL 仍在、元数据能够恢复。请在自己的部署上通过故障演练验证。

### 基于单集群单区域部署的 DR 解决方案

![Single-region-single-cluster](/Single-region-single-cluster.svg)

在跨区域之前,集群首先要能扛住一个区域内单个节点或单个 AZ 的故障。这里的 AZ 与跨区域方案中的定义一致,是灾难恢复的逻辑单元:一个机房,或机房中的一部分。这套拓扑是常见的生产基线,下面的跨区域方案都建立在它之上。

集群部署在一个区域内,各角色分散在该区域的多个 AZ 上:

* **Frontend** 是无状态的。在负载均衡后面部署多个副本,某个副本挂掉只影响它上面正在处理的请求。参见[服务运行副本数配置](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md#服务运行副本数配置)。
* **Metasrv** 以多副本运行并选举 leader,丢一个 follower 对上层透明,丢掉 leader 的代价是一次重新选举。元数据本身存放在外部后端(etcd、MySQL 或 PostgreSQL),Metasrv 不会替你复制它,因此该后端需要自己的高可用和备份。参见[元数据管理](/user-guide/deployments-administration/manage-metadata/overview.md)与[元数据导出与导入](/user-guide/deployments-administration/disaster-recovery/back-up-&-restore-meta-data.md)。
* **Datanode** 承载 Region。某个 Datanode 故障时,[Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) 会把它的 Region 在存活节点上重新打开。该功能**默认关闭**,且要求集群使用[共享存储](/user-guide/deployments-administration/configuration.md#storage-options)配合 Remote WAL。在本地 WAL 上通过 `allow_region_failover_on_local_wal=true` 也能开启,但可能丢数据,因为故障节点的 WAL 仍留在它自己的磁盘上。
* **Kafka**(Remote WAL)和**对象存储**保存着必须比节点活得更久的状态。Kafka 的副本数要能容忍你所防范的 broker 故障,并把 broker 和 Datanode 分散到多个 AZ,而不是堆在同一个 AZ 里。

延迟:
- 只有区域内的往返;写入和复制都不需要付跨区域的延迟代价

支持的高可用能力:
- 单个节点不可用,其 Region 在别处重新打开后,性能几乎不受影响
- 单个 AZ 不可用,性能会下降,除非存活 AZ 在容量上就按接管它的负载来规划
- 区域本身不在覆盖范围内;在这套拓扑里,它是单一故障域

此解决方案面向节点或 AZ 故障场景下的零 RPO 和分钟级 RTO。和其他方案一样,这些指标取决于你需要逐项确认的前提:

- Region Failover **默认关闭**,需要显式开启。
- 存活的 Datanode 需要预留出接管故障节点 Region 的容量,否则故障转移只是把过载搬了个地方。
- 恢复时间主要由 WAL 回放决定:共享同一个 Kafka topic 的 Region 越多,这些 Region 重新提供服务前需要读取的冗余数据就越多。相关模型参见 [Region Failover 的恢复用时](/user-guide/deployments-administration/manage-data/region-failover.md)。

最终的 RPO 和 RTO 请通过故障演练确认。如果区域整体、Kafka 集群或对象存储不可用,仍然需要下面的方案之一,或者把备份定期存放到另一个区域。

### 基于双活互备的 DR 解决方案

![Active-active failover](/active-active-failover.png)
Expand Down Expand Up @@ -141,6 +170,7 @@ BR 进程持续定期将数据从 Cluster 1 备份到 Region 2。
| DR 解决方案 | 容错目标 | RPO | RTO | TCO | 场景 | 远程 WAL 和对象存储 | 备注 |
| ------------- | ------------------------- | ----- | ----- | ----- | ---------------- | --------- | --------|
| 独立模式的 DR 解决方案 | 单区域 | 备份间隔 | 分钟或小时级 | 低 | 小型场景中对可用性和可靠性要求较低 | 可选 | |
| 基于单集群单区域部署的 DR 解决方案 | 单区域,节点与 AZ 级 | 0 | 分钟级 | 中 | 部署在单个区域内的集群,常见的生产基线 | 必需 | Region Failover 默认关闭 |
| 基于双活互备的 DR 解决方案 | 跨区域 | 取决于待同步的数据变更和故障类型 | 取决于外部故障切换 | 低 | 中小型场景中对可用性和可靠性要求较高 | 可选 | 商业功能 |
| 基于单集群跨区域部署的 DR 解决方案 | 多区域 | 0 | 分钟级 | 高 | 中大型场景中对可用性和可靠性要求较高 | 必需 | |
| 基于 BR 的 DR 解决方案 | 单区域 | 备份间隔 | 分钟或小时级 | 低 | 可接受的可用性和可靠性要求 | 可选 | |
Expand Down
92 changes: 92 additions & 0 deletions static/Single-region-single-cluster.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading