Add Disaster Recovery with OADP on ROSA HCP guide

[kmcolli]

Summary

• Adds a new end-to-end guide for disaster recovery on ROSA HCP using OADP, S3 Cross-Region Replication, EFS replication, and Route 53 DNS failover
• Covers full infrastructure setup (S3 buckets, EFS, IAM roles with IRSA, OADP operator, Route 53 health checks) and two DR scenarios:
  • Scenario 1 (Hot-to-Hot): Simulates primary failure by stopping EC2 worker instances, automatic DNS failover via Route 53 health checks, and automatic failback
  • Scenario 2 (Cold DR): Scales down DR cluster to save costs, then brings it up on-demand with OADP backup/restore and EFS replication for full recovery
• Deploys the Phoenix Mission Control demo app to demonstrate real-time telemetry, S3 data feeds, and a dashboard showing active/DR region status
• Includes cleanup steps for all AWS and OpenShift resources

Test plan

• Walked through all steps end-to-end on ROSA HCP 4.22 clusters in us-east-1 and us-east-2
• Verified Scenario 1: primary instance stop triggers Route 53 failover, app accessible on DR, automatic failback on instance restart
• Verified Scenario 2: cold DR cluster starts, OADP restore succeeds, EFS replication promotes, app serves on DR
• Verified cleanup steps remove all created resources

[aws-amplify-us-east-2]

This pull request is automatically being deployed by Amplify Hosting (learn more).

Access this pull request here: https://pr-982.dqokbbp7eqr35.amplifyapp.com

[kumuduh]

PR #982 Validation Report

Validated end-to-end on two ROSA HCP 4.22.4 clusters (khdemopub in us-west-2, khdemoeast in us-east-2) including both DR scenarios and Route 53 failover/failback.

Result: 1 Blocker, 14 Should-fix, 8 Nits

---

BLOCKER

B1. Indented closing code fence breaks rendering (line 591)
The closing code fence has 4 spaces of leading whitespace. Per CommonMark spec, a closing fence indented 4+ spaces is treated as content, not a delimiter. The code block opened at line 562 never closes, corrupting rendering of the entire second half of the document (Steps 4 through Cleanup, both DR scenarios).

Fix: Remove the 4 leading spaces so the line reads just ``` at column 0.

---

SHOULD-FIX (Technical Accuracy)

S11. Machine pool name workers does not exist on ROSA HCP (lines 1039, 1103)
The guide uses rosa edit machinepool workers. On ROSA HCP, machine pools are per-AZ and named workers-0, workers-1, workers-2. The command fails with: ERR: machine pool 'workers' does not exist for hosted cluster.

Fix: Replace with a loop:

for MP in $(rosa list machinepools -c $PRIMARY_CLUSTER_NAME -o json | jq -r '.[].id'); do
  rosa edit machinepool $MP --cluster $PRIMARY_CLUSTER_NAME --autorepair=false
done

S12. Single EFS mount target fails on multi-AZ clusters (lines 263-271, 960-967)
The guide creates one mount target per cluster using a single subnet. ROSA HCP clusters typically have workers across 3 AZs. Pods scheduled to an AZ without a mount target fail with: No matching mount target in the az. Verified: flight-recorder-0 pod failed until additional mount targets were created.

Fix: Create mount targets in all machinepool subnets:

for SUBNET in $(rosa list machinepools -c $PRIMARY_CLUSTER_NAME -o json | jq -r '.[].subnet'); do
  aws efs create-mount-target --region $PRIMARY_REGION \
    --file-system-id $PRIMARY_EFS --subnet-id $SUBNET \
    --security-groups $EFS_SG_PRIMARY
done

S13. Cleanup: Route 53 DNS records not deleted (lines 1292-1296)
The cleanup section only deletes the health check but does not delete the PRIMARY and SECONDARY failover CNAME records. Readers will be left with orphaned DNS records.

S14. Cleanup order: oc delete namespace before helm uninstall (lines 1209-1211)
Deleting the namespace removes all Helm-managed resources, making the subsequent helm uninstall redundant or erroring. Should be reversed: helm uninstall first, then delete the namespace.

---

SHOULD-FIX (Repo Compliance)

S1. Authors field is a single comma-separated string (line 6)
Hugo treats - Kevin Collins, Diana Sari, Kumudu Herath as one author. Each should be a separate list item.

S2. Raw blockquote should use alert shortcode (line 952)
> **Note:** EFS cross-region replicas... should use {{< alert >}}.

S3. Unformatted "Important" callout (lines 347-348)
Important: Skip the Create an EFS file system section renders as plain text. Wrap in {{< alert >}}.

---

SHOULD-FIX (Grammar)

S4. Comma splice (line 952)
AWS doesn't have a "promote" API, you have to break the replication link. -- use a semicolon.

S5. Misleading instruction (line 595)
The examples below show the primary cluster. contradicts the subsection which provides both primary and DR configs. Remove these sentences.

S6. Awkward sentence (line 1201)
DNS failover happens automatically via Route 53 health check, or update DNS manually. -- missing article "the", awkward declarative-to-imperative switch.

S7. "backup" should be "back up" (line 1092)
Image alt text: "Primary site backup and active" should be "back up and active" (verb, not noun).

---

SHOULD-FIX (Terminology / Formatting)

S8. Inconsistent capitalization (lines 15-16)
S3 with Cross-Region Replication vs. EFS with cross-region replication in the same list.

S9. "operator" not capitalized (line 1215)
Cleanup heading says "OADP operator" but rest of doc uses "OADP Operator".

S10. Double blank line (lines 869-870)
Extra blank line before "Request the certificate."

---

NITS (8)

#	Line(s)	Issue
N1	352-358, 952, 1057, 1061, 1076, 1092	Em dashes in prose (AGENTS.md prohibits)
N2	868	Missing comma before "so" joining independent clauses
N3	1096	Awkward: "The DR cluster has worker nodes stopped"
N4	236, 251, 282, etc.	"filesystem" should be "file system" per AWS style
N5	22	"Helm CLI" not backticked like oc and rosa
N6	952	"promote" should use backticks: `promote`
N7	105	us-east-1 conditional is correct but a comment would help
N8	1144	Cold DR reuses WORKER_IDS set in earlier section; won't work in new shell

---

Technical Walkthrough Results

Step	Status	Notes
Environment Variables	PASS	All commands resolve correctly
Step 1: S3 + CRR	PASS	4 buckets, versioning, replication all work
Step 2: EFS + Replication	PASS (with fix)	Must create mount targets in all AZ subnets
Step 3: IAM Roles	PASS	IRSA trust policies work correctly
Step 4: OADP Install	PASS	Operator, DPA, BSL Available on both clusters
Step 5: Demo App	PASS (with fix)	Runs after multi-AZ mount target fix
Step 6: Route 53	PASS	Health check, failover CNAMEs, TLS, custom domain
Scenario 1 (Hot-to-Hot)	PASS (with fix)	Failover and failback work; machinepool fix needed
Scenario 2 (Cold DR)	PASS (with fix)	Cold start, restore, recovery work; same fixes
Cleanup	PARTIAL	oc works; Route 53 record deletion missing

[kumuduh]

Additional Finding: S3 Cleanup fails on versioned buckets

Severity: Should-fix

The cleanup section uses:

aws s3 rb s3://$BUCKET --force

This fails with BucketNotEmpty because --force only removes current objects, not versioned objects or delete markers. Since Step 1 enables versioning (required for CRR), all 4 buckets will have version history that blocks deletion.

Fix: Purge all object versions and delete markers before removing the bucket:

for BUCKET in $APP_BUCKET_PRIMARY $APP_BUCKET_DR \
             $OADP_BUCKET_PRIMARY $OADP_BUCKET_DR; do
  aws s3api delete-objects --bucket $BUCKET --delete \
    "$(aws s3api list-object-versions --bucket $BUCKET \
       --query '{Objects: [Versions,DeleteMarkers][] | [].{Key:Key,VersionId:VersionId}}' \
       --output json)"
  aws s3 rb s3://$BUCKET --force
done

Verified: Hit this exact error during hands-on walkthrough cleanup.

[kumuduh]

Additional Finding: EFS Cleanup order is incomplete — fails on replication and async mount targets

Severity: Should-fix

The cleanup section deletes EFS resources but is missing two critical steps that cause failures:

1. Replication must be deleted first (from both sides)

Deleting an EFS filesystem that is in a replication (source or replica) fails with:

FileSystemInUse: You cannot delete file system because it is in a replication

The replication must be deleted from the source, and crucially, you must also wait for the replica side to fully release — otherwise the DR filesystem deletion also fails.

2. Mount target deletion is async — must wait before deleting filesystem

Deleting mount targets returns immediately, but the actual deletion takes 10-30 seconds. Calling delete-file-system right after fails with:

FileSystemInUse: The file system cannot be deleted because it has mount targets

Fix: The correct EFS cleanup order is replication → access points → mount targets (+ wait) → filesystem:

# 1. Delete replication (from source)
aws efs delete-replication-configuration \
  --source-file-system-id $PRIMARY_EFS --region $PRIMARY_REGION

# Wait for replication to clear on both sides
echo "Waiting for replication to be deleted..."
for EFS_ID_CHECK in $PRIMARY_EFS $DR_EFS; do
  REGION_CHECK=$PRIMARY_REGION
  if [ "$EFS_ID_CHECK" = "$DR_EFS" ]; then REGION_CHECK=$DR_REGION; fi
  while aws efs describe-replication-configurations \
    --file-system-id $EFS_ID_CHECK --region $REGION_CHECK 2>/dev/null | grep -q "DELETING"; do
    sleep 10
  done
done

# 2. Delete access points, mount targets, then filesystems
for EFS_ID in $PRIMARY_EFS $DR_EFS; do
  REGION=$PRIMARY_REGION
  if [ "$EFS_ID" = "$DR_EFS" ]; then REGION=$DR_REGION; fi

  for AP in $(aws efs describe-access-points \
    --file-system-id $EFS_ID --region $REGION \
    --query 'AccessPoints[*].AccessPointId' --output text); do
    aws efs delete-access-point --access-point-id $AP --region $REGION
  done

  for MT in $(aws efs describe-mount-targets \
    --file-system-id $EFS_ID --region $REGION \
    --query 'MountTargets[*].MountTargetId' --output text); do
    aws efs delete-mount-target --mount-target-id $MT --region $REGION
  done

  echo "Waiting for mount targets on $EFS_ID..."
  while [ "$(aws efs describe-mount-targets --file-system-id $EFS_ID \
    --region $REGION --query 'length(MountTargets)' --output text)" != "0" ]; do
    sleep 5
  done

  aws efs delete-file-system --file-system-id $EFS_ID --region $REGION
  echo "Deleted $EFS_ID"
done

Verified: Hit all three failures during hands-on walkthrough cleanup (replication block on both source and replica, mount target async race).