diff --git a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/additional-topics.mdx b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/additional-topics.mdx index 9f23d553474e..1de62b0d1d50 100644 --- a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/additional-topics.mdx +++ b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/additional-topics.mdx @@ -43,6 +43,31 @@ configuration property. **_hbase.backup.logroll.timeout.millis_**: The amount of time (in milliseconds) to wait for RegionServers to execute a WAL rolling in the Master's procedure framework (default: 30000). +### Continuous backup and PITR properties + +**_hbase.backup.continuous.wal.dir_**: Filesystem path for continuously backed-up WALs. Required to enable +continuous backup and PITR. Must be an HDFS-compatible URI (for example, `hdfs://`, `s3a://`). + +**_hbase.backup.continuous.pitr.window.days_**: Maximum number of days in the past from which point-in-time +recovery is allowed (default: `30`). This is a cluster-level setting. + +**_hbase.backup.continuous.offset.update.interval.ms_**: The interval at which the replication source +shipper persists the replication offset for the continuous backup peer (default: `300000`, 5 minutes). +Before the offset is persisted, the continuous backup replication endpoint flushes its buffered WAL data +to the backup storage. This value is passed to the continuous backup replication peer when the peer is +created. + +**_hbase.backup.continuous.offset.update.size.threshold_**: The amount of data processed by the +replication shipper before the replication offset is persisted (default: `134217728`, 128 MB). Before +persistence, buffered WAL data is flushed to backup storage. This value is passed to the continuous +backup replication peer when the peer is created. + +**_hbase.replication.bulkload.enabled_**: When set to `true`, bulk-loaded HFiles are replicated through the +continuous backup replication peer. Required if tables under continuous backup use bulk load operations. +When enabled, bulk-load operations are represented in the WAL and the corresponding HFiles are copied to +the continuous backup storage. During PITR, applicable bulk-loaded HFiles are restored for the requested +time range. + ## Best Practices ### Formulate a restore strategy and test it. @@ -65,6 +90,28 @@ backup should be stored outside of the source cluster. To ensure complete datase with the option to restore baseline full backup. The full backup is the foundation of your dataset. Incremental backup data is applied on top of the full backup during the restore operation to return you to the point in time when backup was last taken. +### Understand the RPO/RTO tradeoff between continuous and incremental backup. + +Continuous backup and incremental backup serve different purposes. + +Continuous backup continuously copies WALs to external storage and therefore reduces Recovery Point +Objective (RPO). + +Incremental backup converts accumulated WALs into HFiles that can be restored more efficiently than +replaying a large number of WAL entries. Therefore, periodic incremental backups reduce PITR recovery +time (Recovery Time Objective, or RTO). + +A common strategy is to combine both: periodic full backups, scheduled incremental backups, and +continuous WAL backup enabled from the first full backup. + +### Combine continuous backup with scheduled incremental backups. + +For production environments that require low RPO and fast recovery, enable continuous backup on the +initial full backup (`--continuous-backup-enabled`) and continue running incremental backups on a +schedule. Continuous backup streams WALs to external storage in near real time, minimizing data loss +between backup jobs. Incremental backups convert accumulated WALs into HFiles, which serve as +checkpoints that reduce the amount of WAL replay required during PITR and lower RTO. + ### Define and use backup sets for groups of tables that are logical subsets of the entire dataset. You can group tables into an object called a backup set. A backup set can save time when you have a particular group of tables @@ -91,75 +138,70 @@ The HBase administration team uses backup sets to store data from a group of tab application called green. In this example, one table contains transaction records and the other contains customer details. The two tables need to be backed up and be recoverable as a group. -The admin team also wants to ensure daily backups occur automatically. - ![Tables Composing The Backup Set](/docs-images/backup-app-components.png) -The following is an outline of the steps and examples of commands that are used to backup the data for the _green_ application and -to recover the data later. All commands are run when logged in as HBase superuser. +The following workflow outlines the steps to back up and recover the _green_ application data. All +commands are run as the HBase superuser. Configure S3 authentication using the Hadoop filesystem +configuration appropriate for your environment (for example, instance profiles, access keys, or tokens). + +**Step 1: Configure the WAL directory** + +Add the WAL directory to `hbase-site.xml` and restart HBase: + +```xml + + hbase.backup.continuous.wal.dir + s3a://prodhbasebackups/wal-backup + +``` -- A backup set called _green_set_ is created as an alias for both the transactions table and the customer table. The backup set can - be used for all operations to avoid typing each table name. The backup set name is case-sensitive and should be formed with only - printable characters and without spaces. +**Step 2: Create a backup set** - ```bash - $ hbase backup set add green_set transactions - $ hbase backup set add green_set customer - ``` +```bash +$ hbase backup set add green_set transactions +$ hbase backup set add green_set customer +``` -- The first backup of green_set data must be a full backup. The following command example shows how credentials are passed to Amazon - S3 and specifies the file system with the s3a: prefix. +**Step 3: Create the first full backup with continuous backup** - ```bash - $ ACCESS_KEY=ABCDEFGHIJKLMNOPQRST - $ SECRET_KEY=123456789abcdefghijklmnopqrstuvwxyzABCD - $ sudo -u hbase hbase backup create full\ - s3a://$ACCESS_KEY:SECRET_KEY@prodhbasebackups/backups -s green_set - ``` +```bash +$ hbase backup create full s3a://prodhbasebackups/backups -s green_set --continuous-backup-enabled +``` -- Incremental backups should be run according to a schedule that ensures essential data recovery in the event of a catastrophe. At - this retail company, the HBase admin team decides that automated daily backups secures the data sufficiently. The team decides that - they can implement this by modifying an existing Cron job that is defined in `/etc/crontab`. Consequently, IT modifies the Cron job - by adding the following line: +**Step 4: Schedule incremental backups** - ```bash - @daily hbase hbase backup create incremental s3a://$ACCESS_KEY:$SECRET_KEY@prodhbasebackups/backups -s green_set - ``` +Add a daily cron job to run incremental backups: -- A catastrophic IT incident disables the production cluster that the green application uses. An HBase system administrator of the - backup cluster must restore the _green_set_ dataset to the point in time closest to the recovery objective. +```bash +@daily hbase backup create incremental s3a://prodhbasebackups/backups -s green_set +``` - - If the administrator of the backup HBase cluster has the backup ID with relevant details in accessible records, the following - search with the `hdfs dfs -ls` command and manually scanning the backup ID list can be bypassed. Consider continuously maintaining - and protecting a detailed log of backup IDs outside the production cluster in your environment. - +**Step 5: Incident occurs** - The HBase administrator runs the following command on the directory where backups are stored to print the list of successful backup - IDs on the console: +A catastrophic IT incident disables the production cluster that the green application uses. - ```bash - hdfs dfs -ls -t /prodhbasebackups/backups - ``` +**Step 6: Determine the recovery timestamp** -- The admin scans the list to see which backup was created at a date and time closest to the recovery objective. To do this, the - admin converts the calendar timestamp of the recovery point in time to Unix time because backup IDs are uniquely identified with - Unix time. The backup IDs are listed in reverse chronological order, meaning the most recent successful backup appears first. +The administrator identifies the Unix timestamp corresponding to the desired recovery point in time. - The admin notices that the following line in the command output corresponds with the _green_set_ backup that needs to be restored: +**Step 7: Run PITR** - ```bash - /prodhbasebackups/backups/backup_1467823988425` - ``` +Because continuous backup is enabled, the admin restores to the specific point in time rather than +being limited to the nearest scheduled backup: -- The admin restores green_set invoking the backup ID and the -overwrite option. The -overwrite option truncates all existing data - in the destination and populates the tables with data from the backup dataset. Without this flag, the backup data is appended to the - existing data in the destination. In this case, the admin decides to overwrite the data because it is corrupted. +```bash +$ hbase pitr -s green_set --to-datetime 1467823988425 -o +``` - ```bash - $ sudo -u hbase hbase restore -s green_set \ - s3a://$ACCESS_KEY:$SECRET_KEY@prodhbasebackups/backups backup_1467823988425 \ -overwrite - ``` +PITR selects the most recent valid backup before the target time, restores it, and replays WALs from +the WAL directory to bring the tables to the requested timestamp. The `-o` flag overwrites existing +data in the destination tables because the current data is corrupted. + +If recovery to a specific backup ID is sufficient, the standard restore command can be used instead: + +```bash +$ hbase restore s3a://prodhbasebackups/backups backup_1467823988425 -s green_set -o +``` ## Security of Backup Data @@ -201,14 +243,73 @@ HBase Bulk Load utility. You can only restore on a live HBase cluster because the data must be redistributed to complete the restore operation successfully. +## Technical Details of Continuous Backup and PITR + +Continuous backup uses a dedicated replication peer (`continuous_backup_replication_peer`) with a +custom replication endpoint (`ContinuousBackupReplicationEndpoint`) to stream WAL entries from the +source cluster to the WAL directory. WAL files are organized by date under the WAL directory: + +```text +/WALs// +/bulk-load-files/ +``` + +WALs are partitioned by date so PITR can efficiently select WAL files covering the requested recovery +interval. + +When a full backup is created with `--continuous-backup-enabled`, HBase performs the following steps: + +1. Sets the replication scope of every column family on the backed-up tables to GLOBAL. +2. Creates or updates the `continuous_backup_replication_peer` to include the specified tables. +3. Records the tables and their continuous backup start timestamps in the `hbase:backup` system table. +4. Takes and exports a full snapshot to the backup directory. + +From that point forward, WAL entries are continuously replicated to the WAL directory. The replication +endpoint groups WAL entries by day and periodically flushes data to avoid large partially-written +files and replication lag. + +### Incremental backups with continuous backup + +When incremental backups are run for tables that have continuous backup enabled, the backup process +reads WALs from the WAL directory instead of collecting them from the source cluster. This eliminates +the need to retain WALs on the source cluster between incremental backup runs and reduces pressure on +the source cluster's log management. The incremental backup command itself is unchanged; the system +detects whether tables are under continuous backup and selects the appropriate WAL source +automatically. + +### Point-in-Time Recovery process + +PITR restores a table to a specific timestamp using the following steps: + +1. **Select a backup**: Locate the most recent valid full or incremental backup completed before the + target time. The backup must have started after continuous backup was enabled for the table. +2. **Restore the backup**: Restore the selected backup image to the target table using the standard + restore process. +3. **Replay WALs**: Run a distributed WAL replay (MapReduce-based WALPlayer) over WAL files from the + WAL directory for the time range between the backup start time and the target timestamp. +4. **Re-apply bulk loads**: Bulk-loaded HFiles discovered in the WAL directory for the same time range + are bulk-loaded into the target table. + +If `--to-datetime` is not specified, PITR restores to the last known safe replication checkpoint, +which is the earliest timestamp across the active RegionServers for which WAL entries before that +point are known to have been persisted to the continuous backup WAL directory. + ## A Warning on File System Growth As a reminder, incremental backups are implemented via retaining the write-ahead logs which HBase primarily uses for data durability. Thus, to ensure that all data needing to be included in a backup is still available in the system, the HBase backup and restore feature -retains all write-ahead logs since the last backup until the next incremental backup is executed. +retains all write-ahead logs since the last backup until the next incremental backup is executed. With continuous backup enabled, +WALs are additionally replicated to the WAL directory. + +WALs in the continuous backup directory must be retained as long as they are required to support the +configured PITR window and the available backup chain. Backup deletion performs validation to prevent +deletion of a backup that would leave a table without a valid PITR recovery path, and removes WALs +that are no longer required. Like HBase Snapshots, this can have an expectedly large impact on the HDFS usage of HBase for high volume tables. Take care in enabling and using the backup and restore feature, specifically with a mind to removing backup sessions when they are not actively being used. +Running incremental backups on a regular schedule reduces WAL retention on the source cluster for tables under continuous backup, +because WALs are read from the WAL directory rather than from the source cluster. The only automated, upper-bound on retained write-ahead logs for backup and restore is based on the TTL of the `hbase:backup` system table which, as of the time this document is written, is infinite (backup table entries are never automatically deleted). This requires that administrators @@ -299,3 +400,7 @@ will be modifications that would necessitate breaking compatibility ([HBASE-1751 **Lack of global metrics for backup and restore** Individual backup and restore operations contain metrics about the amount of work the operation included, but there is no centralized location (e.g. the Master UI) which present information for consumption ([HBASE-16565](https://issues.apache.org/jira/browse/HBASE-16565)). + +**Single WAL directory per cluster** +Continuous backup and PITR support only one WAL directory per cluster, configured via +`hbase.backup.continuous.wal.dir`. All tables under continuous backup share this location. diff --git a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/administration.mdx b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/administration.mdx index dfd3c666904b..e123b80253e2 100644 --- a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/administration.mdx +++ b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/administration.mdx @@ -114,6 +114,12 @@ Comma-separated list of backup IDs to delete. **_-k \_** Deletes all backup images completed more than the specified number of days ago. +**_--force-delete_** (or **_-fd_**) +(Optional) Force deletion of a backup even when it is the only remaining full backup that enables +Point-in-Time Recovery (PITR) for one or more tables. Without this flag, deletion is rejected when +it would leave tables without a valid PITR recovery path. After a successful deletion, the system +attempts to clean up WAL files in the continuous backup store that are no longer needed for PITR. + These options are **mutually exclusive**. Only one of `-l` or `-k` may be used at a time. @@ -126,6 +132,12 @@ Delete specific backup images by ID: $ hbase backup delete -l backupId_1467823988425,backupId_1467824989999 ``` +Force-delete a backup that is required for PITR: + +```bash +$ hbase backup delete -l backupId_1467823988425 --force-delete +``` + Delete all backup images older than 30 days: ```bash @@ -144,6 +156,11 @@ $ hbase backup delete -k 30 Running another incremental backup immediately after may result in missing data in the backup image. (See [HBASE-28084](https://issues.apache.org/jira/browse/HBASE-28084)) +- For tables with continuous backup enabled, deleting the only remaining full backup that supports + PITR is blocked unless `--force-delete` is specified. If the last backup is force-deleted and no + full backups remain, the system removes continuous backup metadata, deletes associated WAL files, + and disables the `continuous_backup_replication_peer`. + ### Backup Repair Command diff --git a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/commands.mdx b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/commands.mdx index bdcc38b284fb..b0e41a3a57c5 100644 --- a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/commands.mdx +++ b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/commands.mdx @@ -75,6 +75,16 @@ are different, we should use -i option to skip checksum-checks. (Optional) Allows specification of the name of a YARN queue which the MapReduce job to create the backup should be executed in. This option is useful to prevent backup tasks from stealing resources away from other MapReduce jobs of high importance. +**_--continuous-backup-enabled_** (or **_-cb_**) +(Optional, full backups only) Enables continuous backup for the specified tables. On the first run with +this flag, HBase creates a full backup, registers the tables with a dedicated replication peer +(`continuous_backup_replication_peer`), and begins streaming WAL entries to the WAL directory +configured by `hbase.backup.continuous.wal.dir`. After continuous backup is enabled for a table, +subsequent full backups for that table do not require this flag. This flag cannot be used with +incremental backups. All tables in a single backup request must either all have continuous backup +enabled or none of them must; mixing tables with and without continuous backup in one request is +not allowed. + #### Example usage [#backup-restore-commands-creating-a-backup-image-example-usage] ```bash @@ -84,6 +94,16 @@ $ hbase backup create full hdfs://host5:9000/data/backup -t SALES2,SALES3 -w 3 This command creates a full backup image of two tables, SALES2 and SALES3, in the HDFS instance who NameNode is host5:9000 in the path _/data/backup_. The _-w_ option specifies that no more than three parallel works complete the operation. +To create a full backup with continuous backup enabled: + +```bash +$ hbase backup create full hdfs://host5:9000/data/backup -t SALES2 -cb +``` + +This command creates a full backup of `SALES2` and starts continuous WAL replication to the WAL +directory configured in `hbase.backup.continuous.wal.dir`. After this initial setup, run incremental +backups on a schedule to convert accumulated WALs into HFiles, which reduces recovery time during PITR. + ### Restoring a Backup Image Run the following command as an HBase superuser. You can only restore a backup on a running HBase cluster because the data must be @@ -144,6 +164,90 @@ This command restores two tables of an incremental backup image. In this example [HBASE-25707](https://issues.apache.org/jira/browse/HBASE-25707) +The `hbase restore` command restores a table to the state captured at a specific backup ID. To restore +to an arbitrary timestamp within the PITR window, use the `hbase pitr` command described below. + +### Point-in-Time Recovery (PITR) + +Point-in-Time Recovery restores tables to a specific timestamp by combining full or incremental +backups with WAL replay from the continuous backup store. PITR requires that continuous backup was +previously enabled for the tables being restored and that `hbase.backup.continuous.wal.dir` is +configured. + +Run the following command as an HBase superuser: + +```bash +hbase pitr [options] +``` + +For each table, PITR locates the most recent valid backup completed before the target time, restores +that backup, and replays WAL entries from the WAL directory to bring the table to the requested +timestamp. If no `--to-datetime` is specified, PITR restores to the last known safe replication +checkpoint. + +#### Named Command-Line Arguments + +**_-t \_** +A comma-separated list of source tables to restore. Mutually exclusive with the _-s_ option. + +**_-s \_** +Restore tables from a named backup set. Mutually exclusive with the _-t_ option. + +**_-q \_** +(Optional) YARN queue name for the MapReduce jobs used during WAL replay. + +**_-c_** +(Optional) Perform a dry-run. Validates that PITR can succeed without executing the restore. + +**_-m \_** +(Optional) A comma-separated list of destination tables. If omitted, tables are restored to their +original names. When provided, there must be an equal number of entries in the `-t` option. + +**_-o_** +(Optional) Overwrite the target table if it already exists. + +**_--to-datetime \_** (or **_-td \_**) +(Optional) Target timestamp for the restore, expressed as a Unix timestamp in milliseconds or seconds. +If omitted, PITR restores to the last known safe replication checkpoint. The requested time must fall +within the PITR window (`hbase.backup.continuous.pitr.window.days`) and must not exceed the +replication checkpoint. + + + If `--to-datetime` is omitted, PITR does not attempt to restore to the current wall-clock time. It + restores to the latest safe replication checkpoint. + + +**_--backup-path \_** (or **_-bp \_**) +(Optional) Restore from a custom backup location instead of using backup metadata from the +`hbase:backup` system table. This is useful when backup images have been moved to archival storage +that is not recorded in the system table. + +#### Example usage + +Restore `mytable` to a specific point in time: + +```bash +$ hbase pitr -t mytable --to-datetime 1710500000000 +``` + +Restore `mytable` into a new table `mytable_restored` at the last known safe replication checkpoint: + +```bash +$ hbase pitr -t mytable -m mytable_restored +``` + +Restore from a custom backup location: + +```bash +$ hbase pitr -t mytable --to-datetime 1710500000000 --backup-path s3a://archive/backups +``` + + + For best recovery performance, run periodic incremental backups on tables with continuous backup + enabled. Incremental backups convert WALs into HFiles, so PITR replays fewer WAL entries on top of + the most recent incremental checkpoint. + + ### Merging Incremental Backup Images This command can be used to merge two or more incremental backup images into a single incremental diff --git a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/index.mdx b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/index.mdx index 7a69147ee05c..5a37b9fed90c 100644 --- a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/index.mdx +++ b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/index.mdx @@ -1,4 +1,4 @@ --- title: "Backup and Restore" -description: "Comprehensive guide to HBase backup and restore capabilities for data protection, disaster recovery, and point-in-time recovery." +description: "Comprehensive guide to HBase backup and restore capabilities including full and incremental backups, continuous backup, point-in-time recovery (PITR), and disaster recovery." --- diff --git a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/overview.mdx b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/overview.mdx index c6faca7bfe2b..ae89c54628f0 100644 --- a/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/overview.mdx +++ b/hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/overview.mdx @@ -6,8 +6,7 @@ description: "Introduction to HBase backup and restore feature, including full a Backup and restore is a standard operation provided by many databases. An effective backup and restore strategy helps ensure that users can recover data in case of unexpected failures. The HBase backup and restore feature helps ensure that enterprises using HBase as a canonical data repository can recover from catastrophic -failures. Another important feature is the ability to restore the database to a particular -point-in-time, commonly referred to as a snapshot. +failures. The HBase backup and restore feature provides the ability to create full backups and incremental backups on tables in an HBase cluster. The full backup is the foundation on which incremental backups are applied @@ -17,6 +16,13 @@ the changes since the last backup and they also enable administrators to restore utilities also enable table-level data backup-and-recovery if you do not want to restore the entire dataset of the backup. +**Continuous backup** extends this model by continuously streaming Write-Ahead Logs (WALs) to external +storage using the HBase replication framework. This reduces the Recovery Point Objective (RPO) compared +with scheduled incremental backups alone, because WALs are replicated as they are produced rather than +waiting for the next backup job. **Point-in-Time Recovery (PITR)** builds on continuous backup by +restoring a table to an arbitrary timestamp within a configurable retention window, using a combination +of full backups, incremental backups, and replay of continuously backed-up WALs. + The backup and restore feature supplements the HBase Replication feature. While HBase replication is ideal for creating "hot" copies of the data (where the replicated data is immediately available for query), the backup and restore feature is ideal for creating "cold" copies of data (where a manual step must be taken to restore the system). @@ -47,6 +53,12 @@ system. - _A backup_: A logical unit of data and metadata which can restore a table to its state at a specific point in time. - _Full backup_: a type of backup which wholly encapsulates the contents of the table at a point in time. - _Incremental backup_: a type of backup which contains the changes in a table since a full backup. +- _Continuous backup_: A mode in which WAL entries are continuously replicated to external storage via a dedicated replication peer, enabling low-RPO recovery and PITR. +- _Point-in-Time Recovery (PITR)_: Restoring table data to a specific timestamp by applying a full or incremental backup and replaying WALs from the continuous backup store. +- _Backup directory_: The filesystem location where full and incremental backup images are stored. +- _WAL directory_: The filesystem location where continuously backed-up WALs are stored. Configured via `hbase.backup.continuous.wal.dir`. +- _PITR window_: The maximum duration, in days, for which point-in-time recovery is supported. Configured via `hbase.backup.continuous.pitr.window.days` (default: 30). +- _Replication checkpoint_: The earliest timestamp across the active RegionServers for which WAL entries before that point are known to have been persisted to the continuous backup WAL directory. PITR cannot restore beyond this checkpoint. - _Backup set_: A user-defined name which references one or more tables over which a backup can be executed. - _Backup ID_: A unique names which identifies one backup from the rest, e.g. `backupId_1467823988425` @@ -155,3 +167,46 @@ Add the following properties to hbase-site.xml and restart HBase if it is alread org.apache.hadoop.hbase.backup.BackupHFileCleaner,... ``` + +### Continuous backup and PITR configuration + +To enable continuous backup and point-in-time recovery, add the following properties to +`hbase-site.xml` in addition to the properties above: + +```xml + + hbase.backup.continuous.wal.dir + s3a://my-bucket/wal-backup + + Filesystem path for continuously backed-up WALs. Required for continuous backup and PITR. + Must be an HDFS-compatible URI (hdfs:, s3a:, etc.). + + + + hbase.backup.continuous.pitr.window.days + 30 + + Maximum number of days in the past from which point-in-time recovery is allowed (default: 30). + + +``` + +If your tables use bulk load operations and you need those files included in continuous backup, +also enable bulk load replication: + +```xml + + hbase.replication.bulkload.enabled + true + +``` + +When bulk-load replication is enabled, bulk-load operations are represented in the WAL and the +corresponding HFiles are copied to the continuous backup storage. During PITR, applicable bulk-loaded +HFiles are restored for the requested time range. + + + When continuous backup is enabled for a table, the replication scope of every column family on that + table is set to GLOBAL. This is required by the replication framework that streams WAL entries to + the WAL directory. +