Skip to content

docs(backups): document PostgreSQL point-in-time recovery (PITR) #629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Andrey Kolkov (androndo) merged 1 commit into from
Jul 24, 2026
+114 −0
Merged

docs(backups): document PostgreSQL point-in-time recovery (PITR) #629

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions content/en/docs/next/operations/services/backup-classes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,63 @@ spec:
name: orders-db
```

## Point-in-time recovery (PostgreSQL)

A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). Under the hood the CNPG barman-cloud plugin restores the newest base backup taken at/before that instant and replays archived WAL up to it, so the restored cluster reflects the database exactly as of `recoveryTime`; later writes are absent.

```yaml
apiVersion: backups.cozystack.io/v1alpha1
kind: RestoreJob
metadata:
name: orders-db-pitr
namespace: tenant-acme
spec:
backupRef:
name: orders-db-adhoc # a completed Backup of the source
targetApplicationRef: # omit for a destructive in-place restore
apiGroup: apps.cozystack.io
kind: Postgres
name: orders-db-copy # a pre-deployed, empty Postgres app
options:
recoveryTime: "2026-07-21T09:30:00Z"
```

A full, scripted example (write a marker, capture the timestamp, restore to it, assert what survived) lives in the monorepo at [`examples/backups/postgres/`](https://github.com/cozystack/cozystack/tree/main/examples/backups/postgres) — `45-restorejob-pitr.yaml`, driven by `run-all.sh`.

### The recoverable window

`recoveryTime` must fall inside the window the archive can reconstruct:

- **Earliest** — the completion of the oldest base backup still in the archive. You cannot recover to an instant before the first base backup: WAL replay always starts from a base backup, and retention (`barmanObjectStore.retentionPolicy`) eventually trims the oldest ones together with the WAL that predates them.
- **Latest** — the timestamp of the most recent WAL segment shipped to object storage. While the source runs with `backup.enabled: true` it archives continuously, so the latest restorable time trails "now" only by the archive lag (seconds for a healthy cluster). Once the source is deleted, the latest restorable time is frozen at whatever WAL made it to S3 before it went away.

A `recoveryTime` **past the latest archived WAL cannot converge**: PostgreSQL replays every available WAL, never reaches the target, exits with `FATAL: recovery ended before configured recovery target was reached`, and CNPG re-creates the recovery instance in a loop. Such a wedged restore is failed by the restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) — and the driver reads the recovery pod's log at that point, so instead of a generic timeout the RestoreJob ends `status.phase: Failed` with conditions `RecoveryConverged=False` and `Ready=False`, both reason `RecoveryTargetUnreachable`, and a message naming the target. The driver deliberately does **not** fail earlier off that FATAL: a *reachable* near-now target hits the identical FATAL transiently — often repeatedly, on a slow archiver — before it converges, so an earlier trip would reject a recoverable restore. To reject an unreachable target quickly, set a short `restoreTimeoutSeconds`; otherwise widen the window (take a fresh backup, or restore closer to now) or pick a time inside it.

Restoring to a **very recent** instant is safe as long as WAL archiving is current: the segment covering the target may not be in object storage yet and the same FATAL fires transiently, but because the driver only fails at the deadline (not on that FATAL), the recovery has the whole window to catch up — the next attempt promotes once the WAL ships and the cluster goes healthy, which completes the restore. Only a target that never becomes reachable within the deadline is failed. If archiving is badly behind (a stalled `archive_command`, an overloaded cluster) a near-now target can exceed even the default 30m window; restore to a point you can confirm is archived (e.g. at/before the most recent completed backup, per the discovery query below).

A `recoveryTime` **before the earliest base backup** fails differently: PostgreSQL cannot begin replay before the base backup it restored, so recovery never reaches a consistent state and never emits the "recovery ended before …" FATAL the driver classifies on. That case also surfaces at the deadline, but with the generic reason `RestoreFailed` rather than `RecoveryTargetUnreachable`. Choosing a `recoveryTime` at/after the oldest base backup's completion (below) avoids it.

### Discovering the earliest / latest restorable time

CNPG's `Cluster.status.firstRecoverabilityPoint` exists in the status schema but is not reliably populated under the barman-cloud plugin, so read the window from the backup catalog instead. Each completed base backup records its WAL range and timestamps on the underlying `cnpg.io/Backup`:

```bash
# Completed base backups, oldest first: STOP is the earliest instant that
# backup alone can restore to; the oldest STOP is the window's lower bound.
kubectl -n <ns> get backups.postgresql.cnpg.io \
--sort-by=.status.stoppedAt \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt,BEGINWAL:.status.beginWal,ENDWAL:.status.endWal
Comment on lines +247 to +249

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "File locations:"
git ls-files | rg 'backup-classes\.md$' || true

echo
echo "Relevant sections (next):"
sed -n '220,270p' content/en/docs/next/operations/services/backup-classes.md

echo
echo "Relevant sections (v1.6):"
sed -n '220,270p' content/en/docs/v1.6/operations/services/backup-classes.md

echo
echo "All kubectl backups.postgresql.cnpg.io commands:"
rg -n "kubectl .*backups\.postgresql\.cnpg\.io|status\.phase|stoppedAt" content/en/docs content/en/docs/next content/en/docs/v1.6 2>/dev/null || true

Repository: cozystack/website

Length of output: 28526

Add a completed-backup filter to the recovery-window query.

The query is introduced with “Each completed base backup records its WAL range…”, but it lists all cnpg.io/Backup statuses. Add a selector or post-filter for status.phase == "completed" before sorting/picking the oldest stoppedAt, and apply the same change in the versioned doc copy.

📍 Affects 2 files
  • content/en/docs/next/operations/services/backup-classes.md#L247-L249 (this comment)
  • content/en/docs/v1.6/operations/services/backup-classes.md#L247-L249
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@content/en/docs/next/operations/services/backup-classes.md` around lines 247
- 249, Update the recovery-window backup query in
content/en/docs/next/operations/services/backup-classes.md:247-249 and
content/en/docs/v1.6/operations/services/backup-classes.md:247-249 to filter for
status.phase == "completed" before sorting by stoppedAt and selecting the oldest
backup. Apply the same completed-backup filter in both document copies.


# The Cozystack Backup points at its underlying cnpg.io/Backup and S3 prefix.
kubectl -n <ns> get backup.backups.cozystack.io <name> -o jsonpath='{.spec.driverMetadata}'
```

The upper bound — the latest archived WAL — is whatever the source has shipped so far; with archiving healthy, any time up to a few seconds ago is safe. To confirm the archive is current, check that the source cluster's WAL is flowing to S3 (the `barman-cloud` sidecar logs an upload per segment) before restoring to a near-now target.

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Files:"
git ls-files | rg 'content/en/docs/(next|v1\.6)/operations/services/backup-classes\.md$' || true

for f in content/en/docs/next/operations/services/backup-classes.md content/en/docs/v1.6/operations/services/backup-classes.md; do
  if [ -f "$f" ]; then
    echo
    echo "=== $f ==="
    sed -n '220,275p' "$f" | cat -n | sed 's/^/  /'
  fi
done

echo
echo "Context search around 'up to a few seconds ago' and near-now:"
rg -n "up to a few seconds ago|near-now|archived WAL|latest archived WAL|barman-cloud|archives current|archive is current" content/en/docs/next content/en/docs/v1.6 -S || true

Repository: cozystack/website

Length of output: 26317

Avoid implying near-now restores are always safe.

Archive lag is not fixed; the preceding text already warns that if archiving is stalled a near-now target can exceed the restore deadline, and that only targets still reachable before the deadline are eventually reached. Keep the sentence conditional or narrow it to targets that can be confirmed as archived.

  • content/en/docs/next/operations/services/backup-classes.md#L255
  • content/en/docs/v1.6/operations/services/backup-classes.md#L255
📍 Affects 2 files
  • content/en/docs/next/operations/services/backup-classes.md#L255-L255 (this comment)
  • content/en/docs/v1.6/operations/services/backup-classes.md#L255-L255
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@content/en/docs/next/operations/services/backup-classes.md` at line 255, The
restore guidance around the upper-bound WAL description must not imply that
near-now targets are always safe. Update the sentence in
content/en/docs/next/operations/services/backup-classes.md:255 and
content/en/docs/v1.6/operations/services/backup-classes.md:255 to make near-now
restores conditional on confirming the required WAL has been archived and is
reachable before the restore deadline; preserve the existing barman-cloud
verification guidance.


### Idempotency under GitOps

An in-progress restore is safe to reconcile. The driver purges the target `Cluster` + PVCs exactly once per RestoreJob (guarded by the `TargetPurged` condition and a freshly-recovered check), suspends the target's HelmRelease across the purge so Flux cannot race the bootstrap swap, and resumes it once the recovery cluster is rendered. A Flux reconcile (or a controller restart) mid-restore therefore re-attaches to the recovering cluster rather than deleting it and starting over.

## See also

- [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}) — the tenant guide for database backups (BackupJob, Plan, RestoreJob).
Expand Down
57 changes: 57 additions & 0 deletions content/en/docs/v1.6/operations/services/backup-classes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,63 @@ spec:
name: orders-db
```

## Point-in-time recovery (PostgreSQL)

A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). Under the hood the CNPG barman-cloud plugin restores the newest base backup taken at/before that instant and replays archived WAL up to it, so the restored cluster reflects the database exactly as of `recoveryTime`; later writes are absent.

```yaml
apiVersion: backups.cozystack.io/v1alpha1
kind: RestoreJob
metadata:
name: orders-db-pitr
namespace: tenant-acme
spec:
backupRef:
name: orders-db-adhoc # a completed Backup of the source
targetApplicationRef: # omit for a destructive in-place restore
apiGroup: apps.cozystack.io
kind: Postgres
name: orders-db-copy # a pre-deployed, empty Postgres app
options:
recoveryTime: "2026-07-21T09:30:00Z"
```

A full, scripted example (write a marker, capture the timestamp, restore to it, assert what survived) lives in the monorepo at [`examples/backups/postgres/`](https://github.com/cozystack/cozystack/tree/main/examples/backups/postgres) — `45-restorejob-pitr.yaml`, driven by `run-all.sh`.

### The recoverable window

`recoveryTime` must fall inside the window the archive can reconstruct:

- **Earliest** — the completion of the oldest base backup still in the archive. You cannot recover to an instant before the first base backup: WAL replay always starts from a base backup, and retention (`barmanObjectStore.retentionPolicy`) eventually trims the oldest ones together with the WAL that predates them.
- **Latest** — the timestamp of the most recent WAL segment shipped to object storage. While the source runs with `backup.enabled: true` it archives continuously, so the latest restorable time trails "now" only by the archive lag (seconds for a healthy cluster). Once the source is deleted, the latest restorable time is frozen at whatever WAL made it to S3 before it went away.

A `recoveryTime` **past the latest archived WAL cannot converge**: PostgreSQL replays every available WAL, never reaches the target, exits with `FATAL: recovery ended before configured recovery target was reached`, and CNPG re-creates the recovery instance in a loop. Such a wedged restore is failed by the restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) — and the driver reads the recovery pod's log at that point, so instead of a generic timeout the RestoreJob ends `status.phase: Failed` with conditions `RecoveryConverged=False` and `Ready=False`, both reason `RecoveryTargetUnreachable`, and a message naming the target. The driver deliberately does **not** fail earlier off that FATAL: a *reachable* near-now target hits the identical FATAL transiently — often repeatedly, on a slow archiver — before it converges, so an earlier trip would reject a recoverable restore. To reject an unreachable target quickly, set a short `restoreTimeoutSeconds`; otherwise widen the window (take a fresh backup, or restore closer to now) or pick a time inside it.

Restoring to a **very recent** instant is safe as long as WAL archiving is current: the segment covering the target may not be in object storage yet and the same FATAL fires transiently, but because the driver only fails at the deadline (not on that FATAL), the recovery has the whole window to catch up — the next attempt promotes once the WAL ships and the cluster goes healthy, which completes the restore. Only a target that never becomes reachable within the deadline is failed. If archiving is badly behind (a stalled `archive_command`, an overloaded cluster) a near-now target can exceed even the default 30m window; restore to a point you can confirm is archived (e.g. at/before the most recent completed backup, per the discovery query below).

A `recoveryTime` **before the earliest base backup** fails differently: PostgreSQL cannot begin replay before the base backup it restored, so recovery never reaches a consistent state and never emits the "recovery ended before …" FATAL the driver classifies on. That case also surfaces at the deadline, but with the generic reason `RestoreFailed` rather than `RecoveryTargetUnreachable`. Choosing a `recoveryTime` at/after the oldest base backup's completion (below) avoids it.

### Discovering the earliest / latest restorable time

CNPG's `Cluster.status.firstRecoverabilityPoint` exists in the status schema but is not reliably populated under the barman-cloud plugin, so read the window from the backup catalog instead. Each completed base backup records its WAL range and timestamps on the underlying `cnpg.io/Backup`:

```bash
# Completed base backups, oldest first: STOP is the earliest instant that
# backup alone can restore to; the oldest STOP is the window's lower bound.
kubectl -n <ns> get backups.postgresql.cnpg.io \
--sort-by=.status.stoppedAt \
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt,BEGINWAL:.status.beginWal,ENDWAL:.status.endWal

# The Cozystack Backup points at its underlying cnpg.io/Backup and S3 prefix.
kubectl -n <ns> get backup.backups.cozystack.io <name> -o jsonpath='{.spec.driverMetadata}'
```

The upper bound — the latest archived WAL — is whatever the source has shipped so far; with archiving healthy, any time up to a few seconds ago is safe. To confirm the archive is current, check that the source cluster's WAL is flowing to S3 (the `barman-cloud` sidecar logs an upload per segment) before restoring to a near-now target.

### Idempotency under GitOps

An in-progress restore is safe to reconcile. The driver purges the target `Cluster` + PVCs exactly once per RestoreJob (guarded by the `TargetPurged` condition and a freshly-recovered check), suspends the target's HelmRelease across the purge so Flux cannot race the bootstrap swap, and resumes it once the recovery cluster is rendered. A Flux reconcile (or a controller restart) mid-restore therefore re-attaches to the recovering cluster rather than deleting it and starting over.

## See also

- [Application Backup and Recovery]({{% ref "/docs/v1.6/applications/backup-and-recovery" %}}) — the tenant guide for database backups (BackupJob, Plan, RestoreJob).
Expand Down
Loading