HBASE-29309 Document continuous backup and PITR in backup-restore guide

hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/additional-topics.mdx

@@ -43,6 +43,31 @@ configuration property.
 **_hbase.backup.logroll.timeout.millis_**: The amount of time (in milliseconds) to wait for RegionServers to execute a WAL rolling
 in the Master's procedure framework (default: 30000).
 
+### Continuous backup and PITR properties
+
+**_hbase.backup.continuous.wal.dir_**: Filesystem path for continuously backed-up WALs. Required to enable
+continuous backup and PITR. Must be an HDFS-compatible URI (for example, `hdfs://`, `s3a://`).
+
+**_hbase.backup.continuous.pitr.window.days_**: Maximum number of days in the past from which point-in-time
+recovery is allowed (default: `30`). This is a cluster-level setting.
+
+**_hbase.backup.continuous.offset.update.interval.ms_**: The interval at which the replication source
+shipper persists the replication offset for the continuous backup peer (default: `300000`, 5 minutes).
+Before the offset is persisted, the continuous backup replication endpoint flushes its buffered WAL data
+to the backup storage. This value is passed to the continuous backup replication peer when the peer is
+created.
+
+**_hbase.backup.continuous.offset.update.size.threshold_**: The amount of data processed by the
+replication shipper before the replication offset is persisted (default: `134217728`, 128 MB). Before
+persistence, buffered WAL data is flushed to backup storage. This value is passed to the continuous
+backup replication peer when the peer is created.
+
+**_hbase.replication.bulkload.enabled_**: When set to `true`, bulk-loaded HFiles are replicated through the
+continuous backup replication peer. Required if tables under continuous backup use bulk load operations.
+When enabled, bulk-load operations are represented in the WAL and the corresponding HFiles are copied to
+the continuous backup storage. During PITR, applicable bulk-loaded HFiles are restored for the requested
+time range.
+
 ## Best Practices
 
 ### Formulate a restore strategy and test it.
@@ -65,6 +90,28 @@ backup should be stored outside of the source cluster. To ensure complete datase
 with the option to restore baseline full backup. The full backup is the foundation of your dataset. Incremental backup data
 is applied on top of the full backup during the restore operation to return you to the point in time when backup was last taken.
 
+### Understand the RPO/RTO tradeoff between continuous and incremental backup.
+
+Continuous backup and incremental backup serve different purposes.
+
+Continuous backup continuously copies WALs to external storage and therefore reduces Recovery Point
+Objective (RPO).
+
+Incremental backup converts accumulated WALs into HFiles that can be restored more efficiently than
+replaying a large number of WAL entries. Therefore, periodic incremental backups reduce PITR recovery
+time (Recovery Time Objective, or RTO).
+
+A common strategy is to combine both: periodic full backups, scheduled incremental backups, and
+continuous WAL backup enabled from the first full backup.
+
+### Combine continuous backup with scheduled incremental backups.
+
+For production environments that require low RPO and fast recovery, enable continuous backup on the
+initial full backup (`--continuous-backup-enabled`) and continue running incremental backups on a
+schedule. Continuous backup streams WALs to external storage in near real time, minimizing data loss
+between backup jobs. Incremental backups convert accumulated WALs into HFiles, which serve as
+checkpoints that reduce the amount of WAL replay required during PITR and lower RTO.
+
 ### Define and use backup sets for groups of tables that are logical subsets of the entire dataset.
 
 You can group tables into an object called a backup set. A backup set can save time when you have a particular group of tables
@@ -91,75 +138,70 @@ The HBase administration team uses backup sets to store data from a group of tab
 application called green. In this example, one table contains transaction records and the other contains customer details. The
 two tables need to be backed up and be recoverable as a group.
 
-The admin team also wants to ensure daily backups occur automatically.
-
 ![Tables Composing The Backup Set](/docs-images/backup-app-components.png)
 
-The following is an outline of the steps and examples of commands that are used to backup the data for the _green_ application and
-to recover the data later. All commands are run when logged in as HBase superuser.
+The following workflow outlines the steps to back up and recover the _green_ application data. All
+commands are run as the HBase superuser. Configure S3 authentication using the Hadoop filesystem
+configuration appropriate for your environment (for example, instance profiles, access keys, or tokens).
+
+**Step 1: Configure the WAL directory**
+
+Add the WAL directory to `hbase-site.xml` and restart HBase:
+
+```xml
+<property>
+  <name>hbase.backup.continuous.wal.dir</name>
+  <value>s3a://prodhbasebackups/wal-backup</value>
+</property>
+```
 
-- A backup set called _green_set_ is created as an alias for both the transactions table and the customer table. The backup set can
-  be used for all operations to avoid typing each table name. The backup set name is case-sensitive and should be formed with only
-  printable characters and without spaces.
+**Step 2: Create a backup set**
 
-      ```bash
-      $ hbase backup set add green_set transactions
-      $ hbase backup set add green_set customer
-      ```
+```bash
+$ hbase backup set add green_set transactions
+$ hbase backup set add green_set customer
+```
 
-- The first backup of green_set data must be a full backup. The following command example shows how credentials are passed to Amazon
-  S3 and specifies the file system with the s3a: prefix.
+**Step 3: Create the first full backup with continuous backup**
 
-      ```bash
-      $ ACCESS_KEY=ABCDEFGHIJKLMNOPQRST
-      $ SECRET_KEY=123456789abcdefghijklmnopqrstuvwxyzABCD
-      $ sudo -u hbase hbase backup create full\
-        s3a://$ACCESS_KEY:SECRET_KEY@prodhbasebackups/backups -s green_set
-      ```
+```bash
+$ hbase backup create full s3a://prodhbasebackups/backups -s green_set --continuous-backup-enabled
+```
 
-- Incremental backups should be run according to a schedule that ensures essential data recovery in the event of a catastrophe. At
-  this retail company, the HBase admin team decides that automated daily backups secures the data sufficiently. The team decides that
-  they can implement this by modifying an existing Cron job that is defined in `/etc/crontab`. Consequently, IT modifies the Cron job
-  by adding the following line:
+**Step 4: Schedule incremental backups**
 
-      ```bash
-      @daily hbase hbase backup create incremental s3a://$ACCESS_KEY:$SECRET_KEY@prodhbasebackups/backups -s green_set
-      ```
+Add a daily cron job to run incremental backups:
 
-- A catastrophic IT incident disables the production cluster that the green application uses. An HBase system administrator of the
-  backup cluster must restore the _green_set_ dataset to the point in time closest to the recovery objective.
+```bash
+@daily hbase backup create incremental s3a://prodhbasebackups/backups -s green_set
+```
 
-      <Callout type="info">
-      If the administrator of the backup HBase cluster has the backup ID with relevant details in accessible records, the following
-      search with the `hdfs dfs -ls` command and manually scanning the backup ID list can be bypassed. Consider continuously maintaining
-      and protecting a detailed log of backup IDs outside the production cluster in your environment.
-      </Callout>
+**Step 5: Incident occurs**
 
-      The HBase administrator runs the following command on the directory where backups are stored to print the list of successful backup
-      IDs on the console:
+A catastrophic IT incident disables the production cluster that the green application uses.
 
-      ```bash
-      hdfs dfs -ls -t /prodhbasebackups/backups
-      ```
+**Step 6: Determine the recovery timestamp**
 
-- The admin scans the list to see which backup was created at a date and time closest to the recovery objective. To do this, the
-  admin converts the calendar timestamp of the recovery point in time to Unix time because backup IDs are uniquely identified with
-  Unix time. The backup IDs are listed in reverse chronological order, meaning the most recent successful backup appears first.
+The administrator identifies the Unix timestamp corresponding to the desired recovery point in time.
 
-      The admin notices that the following line in the command output corresponds with the _green_set_ backup that needs to be restored:
+**Step 7: Run PITR**
 
-      ```bash
-      /prodhbasebackups/backups/backup_1467823988425`
-      ```
+Because continuous backup is enabled, the admin restores to the specific point in time rather than
+being limited to the nearest scheduled backup:
 
-- The admin restores green_set invoking the backup ID and the -overwrite option. The -overwrite option truncates all existing data
-  in the destination and populates the tables with data from the backup dataset. Without this flag, the backup data is appended to the
-  existing data in the destination. In this case, the admin decides to overwrite the data because it is corrupted.
+```bash
+$ hbase pitr -s green_set --to-datetime 1467823988425 -o
+```
 
-      ```bash
-      $ sudo -u hbase hbase restore -s green_set \
-        s3a://$ACCESS_KEY:$SECRET_KEY@prodhbasebackups/backups backup_1467823988425 \ -overwrite
-      ```
+PITR selects the most recent valid backup before the target time, restores it, and replays WALs from
+the WAL directory to bring the tables to the requested timestamp. The `-o` flag overwrites existing
+data in the destination tables because the current data is corrupted.
+
+If recovery to a specific backup ID is sufficient, the standard restore command can be used instead:
+
+```bash
+$ hbase restore s3a://prodhbasebackups/backups backup_1467823988425 -s green_set -o
+```
 
 ## Security of Backup Data
 
@@ -201,14 +243,73 @@ HBase Bulk Load utility.
 
 You can only restore on a live HBase cluster because the data must be redistributed to complete the restore operation successfully.
 
+## Technical Details of Continuous Backup and PITR
+
+Continuous backup uses a dedicated replication peer (`continuous_backup_replication_peer`) with a
+custom replication endpoint (`ContinuousBackupReplicationEndpoint`) to stream WAL entries from the
+source cluster to the WAL directory. WAL files are organized by date under the WAL directory:
+
+```text
+<wal-directory>/WALs/<YYYY-MM-DD>/<wal-file>
+<wal-directory>/bulk-load-files/<bulk-loaded-hfiles>
+```
+
+WALs are partitioned by date so PITR can efficiently select WAL files covering the requested recovery
+interval.
+
+When a full backup is created with `--continuous-backup-enabled`, HBase performs the following steps:
+
+1. Sets the replication scope of every column family on the backed-up tables to GLOBAL.
+2. Creates or updates the `continuous_backup_replication_peer` to include the specified tables.
+3. Records the tables and their continuous backup start timestamps in the `hbase:backup` system table.
+4. Takes and exports a full snapshot to the backup directory.
+
+From that point forward, WAL entries are continuously replicated to the WAL directory. The replication
+endpoint groups WAL entries by day and periodically flushes data to avoid large partially-written
+files and replication lag.
+
+### Incremental backups with continuous backup
+
+When incremental backups are run for tables that have continuous backup enabled, the backup process
+reads WALs from the WAL directory instead of collecting them from the source cluster. This eliminates
+the need to retain WALs on the source cluster between incremental backup runs and reduces pressure on
+the source cluster's log management. The incremental backup command itself is unchanged; the system
+detects whether tables are under continuous backup and selects the appropriate WAL source
+automatically.
+
+### Point-in-Time Recovery process
+
+PITR restores a table to a specific timestamp using the following steps:
+
+1. **Select a backup**: Locate the most recent valid full or incremental backup completed before the
+   target time. The backup must have started after continuous backup was enabled for the table.
+2. **Restore the backup**: Restore the selected backup image to the target table using the standard
+   restore process.
+3. **Replay WALs**: Run a distributed WAL replay (MapReduce-based WALPlayer) over WAL files from the
+   WAL directory for the time range between the backup start time and the target timestamp.
+4. **Re-apply bulk loads**: Bulk-loaded HFiles discovered in the WAL directory for the same time range
+   are bulk-loaded into the target table.
+
+If `--to-datetime` is not specified, PITR restores to the last known safe replication checkpoint,
+which is the earliest timestamp across the active RegionServers for which WAL entries before that
+point are known to have been persisted to the continuous backup WAL directory.
+
 ## A Warning on File System Growth
 
 As a reminder, incremental backups are implemented via retaining the write-ahead logs which HBase primarily uses for data durability.
 Thus, to ensure that all data needing to be included in a backup is still available in the system, the HBase backup and restore feature
-retains all write-ahead logs since the last backup until the next incremental backup is executed.
+retains all write-ahead logs since the last backup until the next incremental backup is executed. With continuous backup enabled,
+WALs are additionally replicated to the WAL directory.
+
+WALs in the continuous backup directory must be retained as long as they are required to support the
+configured PITR window and the available backup chain. Backup deletion performs validation to prevent
+deletion of a backup that would leave a table without a valid PITR recovery path, and removes WALs
+that are no longer required.
 
 Like HBase Snapshots, this can have an expectedly large impact on the HDFS usage of HBase for high volume tables. Take care in enabling
 and using the backup and restore feature, specifically with a mind to removing backup sessions when they are not actively being used.
+Running incremental backups on a regular schedule reduces WAL retention on the source cluster for tables under continuous backup,
+because WALs are read from the WAL directory rather than from the source cluster.
 
 The only automated, upper-bound on retained write-ahead logs for backup and restore is based on the TTL of the `hbase:backup` system table which,
 as of the time this document is written, is infinite (backup table entries are never automatically deleted). This requires that administrators
@@ -299,3 +400,7 @@ will be modifications that would necessitate breaking compatibility ([HBASE-1751
 **Lack of global metrics for backup and restore**  
 Individual backup and restore operations contain metrics about the amount of work the operation included, but there is no centralized location (e.g. the Master UI)
 which present information for consumption ([HBASE-16565](https://issues.apache.org/jira/browse/HBASE-16565)).
+
+**Single WAL directory per cluster**  
+Continuous backup and PITR support only one WAL directory per cluster, configured via
+`hbase.backup.continuous.wal.dir`. All tables under continuous backup share this location.

hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/backup-restore/administration.mdx

@@ -114,6 +114,12 @@ Comma-separated list of backup IDs to delete.
 **_-k \<days\>_**  
 Deletes all backup images completed more than the specified number of days ago.
 
+**_--force-delete_** (or **_-fd_**)  
+(Optional) Force deletion of a backup even when it is the only remaining full backup that enables
+Point-in-Time Recovery (PITR) for one or more tables. Without this flag, deletion is rejected when
+it would leave tables without a valid PITR recovery path. After a successful deletion, the system
+attempts to clean up WAL files in the continuous backup store that are no longer needed for PITR.
+
 <Callout type="info">
   These options are **mutually exclusive**. Only one of `-l` or `-k` may be used at a time.
 </Callout>
@@ -126,6 +132,12 @@ Delete specific backup images by ID:
 $ hbase backup delete -l backupId_1467823988425,backupId_1467824989999
 ```
 
+Force-delete a backup that is required for PITR:
+
+```bash
+$ hbase backup delete -l backupId_1467823988425 --force-delete
+```
+
 Delete all backup images older than 30 days:
 
 ```bash
@@ -144,6 +156,11 @@ $ hbase backup delete -k 30
   Running another incremental backup immediately after may result in missing data in the
   backup image. (See [HBASE-28084](https://issues.apache.org/jira/browse/HBASE-28084))
 
+- For tables with continuous backup enabled, deleting the only remaining full backup that supports
+  PITR is blocked unless `--force-delete` is specified. If the last backup is force-deleted and no
+  full backups remain, the system removes continuous backup metadata, deletes associated WAL files,
+  and disables the `continuous_backup_replication_peer`.
+
 </Callout>
 
 ### Backup Repair Command