From 24206c92d596660f269ef40c2fc0884c3372a139 Mon Sep 17 00:00:00 2001 From: Dmitrii Andreev Date: Mon, 27 Apr 2026 13:46:59 -0500 Subject: [PATCH] HYPERFLEET-859 - docs: revise delete/update test cases for hard-delete design and terminology Align test case documents with the implemented hard-delete mechanism and current API terminology after recent feature merges. - Replace "Ready" with "Reconciled" as the primary convergence condition - Remove stale preconditions (hard-delete endpoints, mutation guard TODOs) - Add namespace cleanup verification steps (Step 5) to deletion tests - Add hard-delete mechanism note linking to architecture design doc - Add three new test cases: LIST soft-deleted clusters, cascade DELETE while nodepool already deleting, cascade DELETE during nodepool update - Fix recreate-same-name test to explicitly reuse cluster name via jq - Update cross-references and hard-delete descriptions across all files --- pkg/client/kubernetes/client.go | 2 +- test-design/testcases/delete-cluster.md | 614 ++++++++++++++---- test-design/testcases/delete-nodepool.md | 184 +++++- test-design/testcases/update-cluster.md | 319 ++++++--- .../testcases/update-delete-test-matrix.md | 64 +- test-design/testcases/update-nodepool.md | 116 ++-- 6 files changed, 977 insertions(+), 322 deletions(-) diff --git a/pkg/client/kubernetes/client.go b/pkg/client/kubernetes/client.go index 65bd09e..aa19a1a 100644 --- a/pkg/client/kubernetes/client.go +++ b/pkg/client/kubernetes/client.go @@ -112,7 +112,7 @@ func (c *Client) FetchNamespace(ctx context.Context, name string) (*corev1.Names ns, err := c.CoreV1().Namespaces().Get(ctx, name, metav1.GetOptions{}) if err != nil { if apierrors.IsNotFound(err) { - return nil, fmt.Errorf("namespace %s not found", name) + return nil, fmt.Errorf("namespace %s not found: %w", name, err) } return nil, fmt.Errorf("failed to get namespace %s: %w", name, err) } diff --git a/test-design/testcases/delete-cluster.md b/test-design/testcases/delete-cluster.md index e405602..a8cbb2d 100644 --- a/test-design/testcases/delete-cluster.md +++ b/test-design/testcases/delete-cluster.md @@ -10,19 +10,24 @@ 6. [Create nodepool under soft-deleted cluster returns 409 Conflict](#test-title-create-nodepool-under-soft-deleted-cluster-returns-409-conflict) 7. [DELETE non-existent cluster returns 404](#test-title-delete-non-existent-cluster-returns-404) 8. [Stuck deletion -- adapter unable to finalize prevents hard-delete](#test-title-stuck-deletion----adapter-unable-to-finalize-prevents-hard-delete) -9. [DELETE during initial creation before cluster reaches Ready](#test-title-delete-during-initial-creation-before-cluster-reaches-ready) -10. [Simultaneous DELETE requests produce a single tombstone](#test-title-simultaneous-delete-requests-produce-a-single-tombstone) +9. [DELETE during initial creation before cluster reaches Reconciled](#test-title-delete-during-initial-creation-before-cluster-reaches-reconciled) +10. [Simultaneous DELETE requests produce a single soft-delete record](#test-title-simultaneous-delete-requests-produce-a-single-soft-delete-record) 11. [Adapter treats externally-deleted K8s resources as finalized](#test-title-adapter-treats-externally-deleted-k8s-resources-as-finalized) 12. [DELETE during update reconciliation before adapters converge](#test-title-delete-during-update-reconciliation-before-adapters-converge) 13. [Recreate cluster with same name after hard-delete](#test-title-recreate-cluster-with-same-name-after-hard-delete) +14. [LIST returns soft-deleted clusters alongside active clusters](#test-title-list-returns-soft-deleted-clusters-alongside-active-clusters) +15. [Cascade DELETE on cluster while a child nodepool is already deleting](#test-title-cascade-delete-on-cluster-while-a-child-nodepool-is-already-deleting) +16. [Cascade DELETE on cluster while child nodepool is mid-update-reconciliation](#test-title-cascade-delete-on-cluster-while-child-nodepool-is-mid-update-reconciliation) --- +> **Hard-delete mechanism:** Hard-delete executes inline within the `POST /adapter_statuses` request that computes `Reconciled=True`. No separate endpoint or background process — test steps simply poll until GET returns 404. See [hard-delete-design.md](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/components/api-service/hard-delete-design.md). + ## Test Title: Cluster deletion happy path -- soft-delete through hard-delete ### Description -This test validates the complete cluster deletion lifecycle end-to-end. It verifies that when a DELETE request is sent for a cluster, the API sets `deleted_time` (soft-delete/tombstone), adapters detect the deletion and clean up their managed K8s resources reporting `Finalized=True`, the API computes `Reconciled=True` from adapter statuses, and the hard-delete mechanism permanently removes the cluster record from the database. +This test validates the complete cluster deletion lifecycle end-to-end. It verifies that when a DELETE request is sent for a cluster, the API sets `deleted_time` (soft-delete), adapters detect the deletion and clean up their managed K8s resources reporting `Finalized=True`, the API computes `Reconciled=True` from adapter statuses, hard-delete permanently removes the cluster record from the database, and the downstream K8s namespace is confirmed absent. --- @@ -43,14 +48,12 @@ This test validates the complete cluster deletion lifecycle end-to-end. It verif 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE and hard-delete endpoints are deployed and operational -5. Reconciled status aggregation is implemented --- ### Test Steps -#### Step 1: Create a cluster and wait for it to reach Ready state +#### Step 1: Create a cluster and wait for it to reach Reconciled state **Action:** - Submit a POST request to create a Cluster resource: @@ -59,14 +62,14 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ -d @testdata/payloads/clusters/cluster-request.json ``` -- Wait for the cluster to reach Ready state: +- Wait for the cluster to reach Reconciled state: ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` **Expected Result:** -- Cluster reaches `Ready` condition `status: "True"` and `Available` condition `status: "True"` -- `Reconciled` condition `status: "True"` at `observed_generation: 1` +- Cluster reaches `Reconciled` condition `status: "True"` with `observed_generation: 1` +- `Available` condition `status: "True"` - All required adapters report `Applied: True`, `Available: True`, `Health: True` #### Step 2: Send DELETE request to soft-delete the cluster @@ -111,25 +114,43 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` -- Continue polling until the cluster record is removed by hard-delete: +- Poll until cluster record is removed (hard-delete executes automatically when `Reconciled=True`): ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` **Expected Result:** - Cluster `Reconciled` condition transitions to `status: "True"` (all adapters confirmed cleanup) -- After hard-delete executes: GET returns HTTP 404 (Not Found) +- After hard-delete completes: GET returns HTTP 404 (Not Found) - Adapter statuses also return HTTP 404 or empty list **Note:** The window between `Reconciled=True` and hard-delete may be brief. If polling observes 404 directly without capturing `Reconciled=True`, this still confirms the full lifecycle completed successfully. +#### Step 5: Verify downstream K8s namespace is cleaned up + +**Action:** +- Poll using `h.GetNamespace()` within `Eventually()` until a `NotFound` error is returned: +```go +Eventually(func(g Gomega) { + _, err := h.GetNamespace(ctx, clusterID) + g.Expect(err).To(HaveOccurred()) + g.Expect(apierrors.IsNotFound(err)).To(BeTrue(), "expected NotFound from GetNamespace, got: %v", err) +}, h.Cfg.Timeouts.Adapter.Processing, h.Cfg.Polling.Interval).Should(Succeed()) +``` + +**Expected Result:** +- `h.GetNamespace()` returns a `NotFound` error — the namespace and all resources within it were deleted as part of adapter finalization +- No orphaned K8s resources remain from this HyperFleet cluster + +**Note:** Namespace deletion in K8s is asynchronous (namespace transitions through `Terminating` before removal). The `Eventually` wrapper prevents flaky failures when the namespace has not yet been fully garbage-collected at the time of the first check. Adapter `Finalized=True` confirms the adapter issued its K8s delete calls, but the namespace may still be terminating. + --- ## Test Title: Cluster deletion cascades to child nodepools ### Description -This test validates hierarchical deletion behavior. When a cluster is deleted, the API must cascade `deleted_time` to all child nodepools simultaneously. Each nodepool's adapters must independently confirm cleanup via `Finalized=True`. The hard-delete mechanism must remove subresource records (nodepools) before removing the parent resource (cluster). +This test validates hierarchical deletion behavior. When a cluster is deleted, the API must cascade `deleted_time` to all child nodepools simultaneously. Each nodepool's adapters must independently confirm cleanup via `Finalized=True`. The hard-delete mechanism must remove subresource records (nodepools) before removing the parent resource (cluster). After hard-delete, the cluster's downstream K8s namespace (which contained all nodepool resources) must be confirmed absent. --- @@ -150,14 +171,12 @@ This test validates hierarchical deletion behavior. When a cluster is deleted, t 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE and hard-delete endpoints are deployed and operational -5. Reconciled status aggregation is implemented --- ### Test Steps -#### Step 1: Create a cluster with two nodepools and wait for Ready state +#### Step 1: Create a cluster with two nodepools and wait for Reconciled state **Action:** - Create a cluster: @@ -177,11 +196,11 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools \ -H "Content-Type: application/json" \ -d @testdata/payloads/nodepools/nodepool-request.json ``` -- Wait for cluster and both nodepools to reach Ready state +- Wait for cluster and both nodepools to reach Reconciled state **Expected Result:** -- Cluster `Ready` condition `status: "True"` -- Both nodepools `Ready` condition `status: "True"` +- Cluster `Reconciled` condition `status: "True"` +- Both nodepools `Reconciled` condition `status: "True"` #### Step 2: Send DELETE request for the cluster (not individual nodepools) @@ -194,7 +213,7 @@ curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} **Expected Result:** - Response returns HTTP 202 (Accepted) with `deleted_time` set on the cluster -#### Step 3: Verify cascade -- all child nodepools have matching deleted_time +#### Step 3: Verify cascade -- all child nodepools have deleted_time set **Action:** - Retrieve each nodepool to verify cascade: @@ -206,8 +225,8 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepo ``` **Expected Result:** -- Both nodepools have `deleted_time` set -- The `deleted_time` values match the cluster's `deleted_time` (set simultaneously) +- Both nodepools have `deleted_time` set (non-nil, valid RFC3339 timestamp) +- The nodepool `deleted_time` values may differ slightly from the cluster `deleted_time`; exact equality is not required - Each nodepool's `generation` is incremented #### Step 4: Verify all adapters report Finalized=True and hard-delete completes @@ -219,7 +238,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepo curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id_2}/statuses curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses ``` -- Continue polling until hard-delete removes all records: +- Poll until all records are removed (hard-delete executes automatically when `Reconciled=True` — subresources first, then resource): ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id_1} curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id_2} @@ -233,13 +252,31 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} **Note:** The window between `Reconciled=True` and hard-delete may be brief. If polling observes 404 directly, this still confirms the full lifecycle completed successfully. +#### Step 5: Verify downstream K8s namespace is cleaned up + +**Action:** +- Poll using `h.GetNamespace()` within `Eventually()` until a `NotFound` error is returned: +```go +Eventually(func(g Gomega) { + _, err := h.GetNamespace(ctx, clusterID) + g.Expect(err).To(HaveOccurred()) + g.Expect(apierrors.IsNotFound(err)).To(BeTrue(), "expected NotFound from GetNamespace, got: %v", err) +}, h.Cfg.Timeouts.Adapter.Processing, h.Cfg.Polling.Interval).Should(Succeed()) +``` + +**Expected Result:** +- `h.GetNamespace()` returns a `NotFound` error — the namespace and all resources within it (cluster + both child nodepools) were deleted as part of adapter finalization +- No orphaned K8s resources remain + +**Note:** See test #1 Step 5 note on namespace deletion timing. + --- ## Test Title: Soft-deleted cluster remains visible via GET and LIST ### Description -This test validates that after a cluster is soft-deleted (tombstoned), it remains queryable via GET and LIST operations until the hard-delete completes. This allows monitoring the deletion progress and debugging stuck deletions. +This test validates that after a cluster is soft-deleted, it remains queryable via GET and LIST operations before hard-delete. The test uses a Sentinel fence (scale `sentinel-clusters` to 0) immediately after DELETE so the visibility window is deterministic and not dependent on reconciliation timing races. --- @@ -260,16 +297,15 @@ This test validates that after a cluster is soft-deleted (tombstoned), it remain 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE endpoint is deployed and operational --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready state +#### Step 1: Create a cluster and wait for Reconciled state **Action:** -- Create a cluster and wait for `Ready` condition `status: "True"`: +- Create a cluster and wait for `Reconciled` condition `status: "True"`: ```bash curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ @@ -277,53 +313,70 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ ``` **Expected Result:** -- Cluster reaches Ready state +- Cluster reaches `Reconciled` condition `status: "True"` with `observed_generation: 1` +- Cluster `generation` equals 1 -#### Step 2: Send DELETE request +#### Step 2: Send DELETE request to soft-delete the cluster **Action:** +- Send DELETE request: ```bash curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` +- Scale Sentinel for clusters to 0 replicas to freeze reconciliation while visibility assertions run: +```bash +kubectl scale deployment/sentinel-clusters -n hyperfleet --replicas=0 +kubectl rollout status deployment/sentinel-clusters -n hyperfleet --timeout=60s +``` **Expected Result:** - Response returns HTTP 202 (Accepted) with `deleted_time` set +- `deleted_time` is a valid RFC3339 timestamp +- `generation` is incremented to 2 +- Sentinel cluster reconciler is paused, preventing hard-delete progression during visibility checks -#### Step 3: Verify GET returns the soft-deleted cluster +#### Step 3: Verify GET observes the soft-deleted cluster before hard-delete **Action:** -- Retrieve the cluster immediately after soft-delete (before hard-delete completes): +- Poll GET with `Eventually` until the soft-deleted cluster is observed via HTTP 200 with `deleted_time` populated. While the Sentinel fence is active, HTTP 404 in this step is a failure (it means visibility was not proven). Use framework-configured polling/timeout values (for example, `500ms` interval and `10s` timeout): ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` **Expected Result:** -- Response returns HTTP 200 (OK) -- Response body includes the full cluster object -- `deleted_time` field is populated with a valid RFC3339 timestamp -- Cluster conditions reflect the current deletion progress +- At least one GET returns HTTP 200 (OK) with the cluster object present and `deleted_time` populated +- Cluster `generation` equals 2 +- This proves the cluster remains visible in soft-deleted state while reconciliation is paused +- HTTP 404 is not an acceptable success outcome for this visibility step + +**Note:** During this observation period, `Reconciled` is frequently `False` while adapters finalize generation 2, but it can transition quickly depending on system timing. -#### Step 4: Verify LIST includes the soft-deleted cluster +#### Step 4: Verify LIST includes the soft-deleted cluster before hard-delete completes **Action:** -- List all clusters: +- Poll LIST with `Eventually` until the deleted cluster appears with `deleted_time` set. Use framework-configured polling/timeout values (for example, `500ms` interval and `10s` timeout): ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters ``` **Expected Result:** -- Response returns HTTP 200 (OK) -- The soft-deleted cluster appears in the list +- At least one LIST includes the deleted cluster entry - The cluster entry has `deleted_time` populated +- This confirms LIST visibility of the soft-deleted resource while reconciliation is paused #### Step 5: Cleanup resources **Action:** -- The cluster is already soft-deleted. Wait for hard-delete to complete (poll until GET returns 404). -- The framework's `h.CleanupTestCluster()` helper handles this automatically in `AfterEach`. +- Scale Sentinel for clusters back to 1 replica to resume reconciliation: +```bash +kubectl scale deployment/sentinel-clusters -n hyperfleet --replicas=1 +kubectl rollout status deployment/sentinel-clusters -n hyperfleet --timeout=60s +``` +- If the cluster still exists after Step 4, continue polling until GET returns HTTP 404 (hard-delete executes automatically when `Reconciled=True` after Sentinel resumes) +- The framework's `h.CleanupTestCluster()` helper can also handle the remaining lifecycle in `AfterEach` **Expected Result:** -- Cluster is hard-deleted (GET returns 404) +- Cluster is eventually hard-deleted (GET returns HTTP 404) --- @@ -352,16 +405,15 @@ This test validates that calling DELETE on a cluster that has already been soft- 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE endpoint is deployed and operational --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready state +#### Step 1: Create a cluster and wait for Reconciled state **Action:** -- Create a cluster and wait for Ready: +- Create a cluster and wait for Reconciled: ```bash curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ @@ -369,7 +421,7 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ ``` **Expected Result:** -- Cluster reaches Ready state +- Cluster reaches `Reconciled: True` #### Step 2: Send first DELETE request @@ -399,7 +451,7 @@ curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} #### Step 4: Cleanup resources **Action:** -- The cluster is already soft-deleted. Wait for hard-delete to complete (poll until GET returns 404). +- The cluster is already soft-deleted. Poll until GET returns 404 (hard-delete executes automatically when `Reconciled=True`). - The framework's `h.CleanupTestCluster()` helper handles this automatically in `AfterEach`. **Expected Result:** @@ -413,21 +465,19 @@ curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} This test validates that the API rejects mutation requests (PATCH) to clusters that have been soft-deleted. Once a cluster has `deleted_time` set, no spec modifications should be allowed to prevent new generation events from triggering reconciliation while deletion cleanup is in progress. -**Note:** The PATCH request schema only accepts mutable fields (`spec`), so `deleted_time` cannot be cleared via PATCH. However, a PATCH on a tombstoned resource bumps `generation` (when spec changes), creating a mismatch (`observed_generation < generation`) that blocks hard-delete until all adapters re-process and report at the new generation. The adapter's `lifecycle.delete.when` check short-circuits spec application (no K8s resources are recreated), but the unnecessary round-trip through Sentinel, adapter, and status reporting delays hard-delete completion. A 409 guard at the API boundary prevents this distributed churn entirely. - -**Status note:** This test case requires the API to implement a mutation guard for tombstoned resources. Until then, PATCH will succeed on soft-deleted resources. +**Note:** The PATCH request schema only accepts mutable fields (`spec`), so `deleted_time` cannot be cleared via PATCH. However, a PATCH on a soft-deleted resource bumps `generation` (when spec changes), creating a mismatch (`observed_generation < generation`) that blocks hard-delete until all adapters re-process and report at the new generation. The adapter's `lifecycle.delete.when` check short-circuits spec application (no K8s resources are recreated), but the unnecessary round-trip through Sentinel, adapter, and status reporting delays hard-delete completion. A 409 guard at the API boundary prevents this distributed churn entirely. --- | **Field** | **Value** | |-----------|-----------| | **Pos/Neg** | Negative | -| **Priority** | Tier0 | +| **Priority** | Tier1 | | **Status** | Draft | | **Automation** | Not Automated | | **Version** | Post-MVP | | **Created** | 2026-04-15 | -| **Updated** | 2026-04-16 | +| **Updated** | 2026-04-28 | --- @@ -436,17 +486,15 @@ This test validates that the API rejects mutation requests (PATCH) to clusters t 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE and PATCH endpoints are deployed and operational -5. Mutation guard for tombstoned resources is implemented in the API --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready state +#### Step 1: Create a cluster and wait for Reconciled state **Action:** -- Create a cluster and wait for Ready: +- Create a cluster and wait for Reconciled: ```bash curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ @@ -454,7 +502,7 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ ``` **Expected Result:** -- Cluster reaches Ready state at `generation: 1` +- Cluster reaches Reconciled state at `generation: 1` #### Step 2: Send DELETE request to soft-delete the cluster @@ -497,7 +545,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} #### Step 5: Cleanup resources **Action:** -- The cluster is already soft-deleted. Wait for hard-delete to complete (poll until GET returns 404). +- The cluster is already soft-deleted. Poll until GET returns 404 (hard-delete executes automatically when `Reconciled=True`). - The framework's `h.CleanupTestCluster()` helper handles this automatically in `AfterEach`. **Expected Result:** @@ -511,8 +559,6 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} This test validates that creating new subresources (nodepools) under a soft-deleted cluster is rejected with 409 Conflict. This prevents new resources from being provisioned while the parent cluster is being cleaned up. -**Status note:** This test case requires the API to implement a mutation guard for tombstoned resources. Until then, POST will succeed on soft-deleted clusters, creating orphan nodepools that would be immediately cascaded for deletion. - --- | **Field** | **Value** | @@ -532,17 +578,15 @@ This test validates that creating new subresources (nodepools) under a soft-dele 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE endpoint is deployed and operational -5. Mutation guard for tombstoned resources is implemented in the API --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready state +#### Step 1: Create a cluster and wait for Reconciled state **Action:** -- Create a cluster and wait for Ready: +- Create a cluster and wait for Reconciled: ```bash curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ @@ -550,7 +594,7 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ ``` **Expected Result:** -- Cluster reaches Ready state +- Cluster reaches `Reconciled: True` #### Step 2: Send DELETE request to soft-delete the cluster @@ -584,12 +628,12 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools ``` **Expected Result:** -- Response returns an empty list (or only pre-existing nodepools, if any) +- Response returns an empty list #### Step 5: Cleanup resources **Action:** -- The cluster is already soft-deleted. Wait for hard-delete to complete (poll until GET returns 404). +- The cluster is already soft-deleted. Poll until GET returns 404 (hard-delete executes automatically when `Reconciled=True`). - The framework's `h.CleanupTestCluster()` helper handles this automatically in `AfterEach`. **Expected Result:** @@ -651,12 +695,12 @@ This test validates that when an adapter is unable to complete deletion cleanup | **Field** | **Value** | |-----------|-----------| | **Pos/Neg** | Negative | -| **Priority** | Tier1 | +| **Priority** | Tier2 | | **Status** | Draft | | **Automation** | Not Automated | | **Version** | Post-MVP | | **Created** | 2026-04-16 | -| **Updated** | 2026-04-16 | +| **Updated** | 2026-04-28 | --- @@ -665,27 +709,26 @@ This test validates that when an adapter is unable to complete deletion cleanup 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE and hard-delete endpoints are deployed and operational -5. A dedicated crash-adapter is available for deployment via Helm (same as used in "Cluster can reach correct status after adapter crash and recovery") +4. A dedicated crash-adapter is available for deployment via Helm (same as used in "Cluster can reach correct status after adapter crash and recovery") --- ### Test Steps -#### Step 1: Deploy crash-adapter and create a cluster, wait for Ready state +#### Step 1: Deploy crash-adapter and create a cluster, wait for Reconciled state **Action:** - Deploy a dedicated crash-adapter via Helm (`${ADAPTER_DEPLOYMENT_NAME}`), separate from the normal adapters used in other tests -- Create a cluster and wait for Ready: +- Create a cluster and wait for Reconciled: ```bash curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ -d @testdata/payloads/clusters/cluster-request.json ``` -- Wait for cluster to reach `Ready` condition `status: "True"` with all adapters (including crash-adapter) reporting `Applied: True` +- Wait for cluster to reach `Reconciled` condition `status: "True"` with all adapters (including crash-adapter) reporting `Applied: True` **Expected Result:** -- Cluster reaches Ready state +- Cluster reaches `Reconciled: True` - crash-adapter is present in adapter statuses with `Applied: True`, `Available: True`, `Health: True` #### Step 2: Scale down crash-adapter to simulate unavailability @@ -713,7 +756,7 @@ curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} #### Step 4: Wait and verify cluster remains stuck in soft-deleted state **Action:** -- Wait for a reasonable period (e.g., 2x the normal hard-delete timeout) to allow healthy adapters to finalize +- Wait for `2 * h.Cfg.Timeouts.Cluster.Ready` to allow healthy adapters to finalize - Poll cluster status periodically: ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} @@ -744,7 +787,7 @@ kubectl scale deployment/${ADAPTER_DEPLOYMENT_NAME} -n hyperfleet --replicas=1 ```bash kubectl rollout status deployment/${ADAPTER_DEPLOYMENT_NAME} -n hyperfleet --timeout=60s ``` -- Poll until the cluster is hard-deleted: +- Poll until cluster record is removed (hard-delete executes automatically when `Reconciled=True`): ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` @@ -775,11 +818,11 @@ kubectl delete namespace {cluster_id} --ignore-not-found --- -## Test Title: DELETE during initial creation before cluster reaches Ready +## Test Title: DELETE during initial creation before cluster reaches Reconciled ### Description -This test validates deletion behavior when a cluster is still mid-reconciliation (adapters have not yet reported `Applied=True`). The cluster is created and immediately deleted without waiting for Ready state. Adapters should detect the `deleted_time` tombstone regardless of their pre-deletion state and finalize cleanup. The system must not get stuck due to adapters having stale or incomplete status from the initial creation. +This test validates deletion behavior when a cluster is still mid-reconciliation (adapters have not yet reported `Applied=True`). The cluster is created and immediately deleted without waiting for Reconciled state. Adapters should detect `deleted_time` regardless of their pre-deletion state and finalize cleanup. The system must not get stuck due to adapters having stale or incomplete status from the initial creation. --- @@ -800,13 +843,12 @@ This test validates deletion behavior when a cluster is still mid-reconciliation 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE and hard-delete endpoints are deployed and operational --- ### Test Steps -#### Step 1: Create a cluster and immediately send DELETE without waiting for Ready +#### Step 1: Create a cluster and immediately send DELETE without waiting for Reconciled **Action:** - Create a cluster: @@ -815,7 +857,7 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ -d @testdata/payloads/clusters/cluster-request.json ``` -- Immediately send DELETE (do NOT wait for Ready or any adapter status): +- Immediately send DELETE (do NOT wait for Reconciled or any adapter status): ```bash curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` @@ -846,7 +888,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses **Expected Result:** - All required adapters eventually report `Finalized` condition `status: "True"` -- Adapters that had not yet reported `Applied=True` (stale `Applied=False` or no status at all) still detect the tombstone and finalize +- Adapters that had not yet reported `Applied=True` (stale `Applied=False` or no status at all) still detect the soft-delete and finalize - `observed_generation: 2` on all adapter statuses **Note:** Some adapters may have partially applied K8s resources from the initial creation before detecting `deleted_time`. The adapter's `lifecycle.delete.when` check runs before apply on subsequent reconciliation, so these partial resources should be cleaned up during finalization. @@ -854,7 +896,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses #### Step 3: Verify cluster is hard-deleted **Action:** -- Poll until the cluster record is removed: +- Poll until cluster record is removed (hard-delete executes automatically when `Reconciled=True`): ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` @@ -876,11 +918,11 @@ kubectl delete namespace {cluster_id} --ignore-not-found --- -## Test Title: Simultaneous DELETE requests produce a single tombstone +## Test Title: Simultaneous DELETE requests produce a single soft-delete record ### Description -This test validates that when multiple DELETE requests for the same cluster are issued in parallel (as opposed to sequentially), the API handles them idempotently at the server boundary. Exactly one tombstone is written (`deleted_time` is set once), `generation` is incremented exactly once, and the downstream reconciliation completes normally. This complements the sequential re-DELETE idempotency test by exercising the concurrency-safety property of the DELETE handler. +This test validates that when multiple DELETE requests for the same cluster are issued in parallel (as opposed to sequentially), the API handles them idempotently at the server boundary. Exactly one soft-delete is written (`deleted_time` is set once), `generation` is incremented exactly once, and the downstream reconciliation completes normally. This complements the sequential re-DELETE idempotency test by exercising the concurrency-safety property of the DELETE handler. --- @@ -901,13 +943,12 @@ This test validates that when multiple DELETE requests for the same cluster are 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE and hard-delete endpoints are deployed and operational --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready state +#### Step 1: Create a cluster and wait for Reconciled state **Action:** ```bash @@ -915,10 +956,10 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ -d @testdata/payloads/clusters/cluster-request.json ``` -- Wait for Ready and `Reconciled: True` at `generation: 1` +- Wait for `Reconciled: True` at `generation: 1` **Expected Result:** -- Cluster reaches `Ready: True`, `Reconciled: True`, `generation: 1` +- Cluster reaches `Reconciled: True`, `generation: 1` #### Step 2: Send multiple DELETE requests in parallel @@ -936,7 +977,7 @@ wait - Every request returns HTTP 200 or 202 -- no 5xx responses - No request returns 404 (all observe the resource as existing at least at time of handler entry) -#### Step 3: Verify exactly one tombstone was written +#### Step 3: Verify exactly one soft-delete was written **Action:** - Compare `deleted_time` and `generation` across all 5 response bodies: @@ -952,12 +993,12 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} - All 5 responses carry the **same** `deleted_time` value (single RFC3339 timestamp) - All 5 responses carry the **same** post-delete `generation` value - Server-side GET shows `generation` incremented by exactly 1 compared to Step 1 (i.e., equals 2), **not** by the number of parallel DELETE requests -- `deleted_time` is set exactly once (no tombstone churn) +- `deleted_time` is set exactly once (no duplicate soft-delete writes) #### Step 4: Verify deletion completes normally **Action:** -- Poll adapter statuses and cluster GET until hard-delete: +- Poll adapter statuses and cluster GET until hard-delete completes (executes automatically when `Reconciled=True`): ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} @@ -998,7 +1039,7 @@ This test validates the adapter-side "NotFound as success" semantics. When the m | **Automation** | Not Automated | | **Version** | Post-MVP | | **Created** | 2026-04-16 | -| **Updated** | 2026-04-16 | +| **Updated** | 2026-04-28 | --- @@ -1007,14 +1048,13 @@ This test validates the adapter-side "NotFound as success" semantics. When the m 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE and hard-delete endpoints are deployed and operational -5. The tester has `kubectl` credentials sufficient to delete the namespace created by the cluster adapters (to simulate external deletion) +4. The tester has `kubectl` credentials sufficient to delete the namespace created by the cluster adapters (to simulate external deletion) --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready state +#### Step 1: Create a cluster and wait for Reconciled state **Action:** ```bash @@ -1022,14 +1062,14 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ -d @testdata/payloads/clusters/cluster-request.json ``` -- Wait for cluster to reach `Ready: True`, `Reconciled: True`, `generation: 1` +- Wait for cluster to reach `Reconciled: True`, `generation: 1` - Confirm the managed K8s resources exist: ```bash kubectl get namespace {cluster_id} ``` **Expected Result:** -- Cluster is Ready; managed namespace exists +- Cluster is Reconciled; managed namespace exists #### Step 2: Externally delete the managed K8s resources (bypass the API) @@ -1045,7 +1085,7 @@ kubectl get namespace {cluster_id} **Expected Result:** - `kubectl get` returns `NotFound` (or confirms namespace is in `Terminating` then fully gone) -- Important: do **not** issue an API DELETE in this step -- the API still thinks the cluster is Ready +- Important: do **not** issue an API DELETE in this step -- the API still thinks the cluster is Reconciled #### Step 3: Send DELETE request through the API @@ -1067,7 +1107,9 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses **Expected Result:** - Each required adapter reports `Finalized` condition `status: "True"` with `observed_generation: 2` -- `reason` / `message` indicate the resources were already absent (e.g., `"ResourcesAlreadyAbsent"`, `"NotFoundTreatedAsSuccess"`) -- exact strings are implementation-defined, but must not indicate an error +- `Finalized` condition `reason`: assert present/non-nil first, then assert non-empty; do not assert on specific string values +- `Finalized` condition `message`: assert present/non-nil first, then assert non-empty; do not assert on specific string values +- `Finalized` condition `last_transition_time` is a valid RFC3339 timestamp - `Health` condition remains `status: "True"` (adapter itself is healthy; the NotFound was not an error) - No error-class log output is required from adapters; the NotFound path is the non-exceptional success path @@ -1101,7 +1143,7 @@ kubectl delete namespace {cluster_id} --ignore-not-found ### Description -This test validates the interaction between update and delete workflows. When a cluster is updated via PATCH and immediately deleted before adapters finish reconciling the update, the deletion workflow must take priority. Adapters receive the next event, detect `deleted_time`, and switch to cleanup mode instead of continuing update reconciliation. This is distinct from "DELETE during initial creation" (test #9) because adapters already have `Applied=True` from the previous generation and are mid-reconciliation for the new generation — a different code path in the adapter's lifecycle handler. +This test validates the interaction between update and delete workflows. When a cluster is updated via PATCH and immediately deleted before adapters finish reconciling the update, the deletion workflow must take priority. Adapters receive the next event, detect `deleted_time`, and switch to cleanup mode instead of continuing update reconciliation. This is distinct from [DELETE during initial creation](#test-title-delete-during-initial-creation-before-cluster-reaches-reconciled) (matrix #18) because adapters already have `Applied=True` from the previous generation and are mid-reconciliation for the new generation — a different code path in the adapter's lifecycle handler. --- @@ -1122,14 +1164,12 @@ This test validates the interaction between update and delete workflows. When a 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE and PATCH endpoints are deployed and operational -5. Reconciled status aggregation is implemented --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready state at generation 1 +#### Step 1: Create a cluster and wait for Reconciled state at generation 1 **Action:** - Create a cluster and wait for full convergence: @@ -1141,7 +1181,7 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ - Wait for `Reconciled` condition `status: "True"` at `generation: 1` **Expected Result:** -- Cluster reaches `Reconciled: True`, `Ready: True` at `generation: 1` +- Cluster reaches `Reconciled: True` at `generation: 1` - All adapters report `Applied: True`, `observed_generation: 1` #### Step 2: Send PATCH request (do NOT wait for reconciliation to complete) @@ -1188,7 +1228,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses #### Step 5: Verify cluster is hard-deleted **Action:** -- Poll until the cluster record is removed: +- Poll until cluster record is removed (hard-delete executes automatically when `Reconciled=True`): ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` @@ -1235,14 +1275,12 @@ This test validates that after a cluster is fully deleted (hard-deleted from the 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE and hard-delete endpoints are deployed and operational -5. Reconciled status aggregation is implemented --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready state +#### Step 1: Create a cluster and wait for Reconciled state **Action:** - Create a cluster using the standard payload (name is generated via `{{.Random}}` template): @@ -1255,7 +1293,7 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ - Record the `id` as `{first_cluster_id}` and the `name` as `{cluster_name}` **Expected Result:** -- Cluster reaches `Reconciled: True`, `Ready: True` +- Cluster reaches `Reconciled: True` - All adapters report `Applied: True` #### Step 2: Delete the cluster and wait for hard-delete to complete @@ -1265,7 +1303,7 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ ```bash curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{first_cluster_id} ``` -- Wait for hard-delete (poll until GET returns 404): +- Poll until GET returns 404 (hard-delete executes automatically when `Reconciled=True`): ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{first_cluster_id} ``` @@ -1280,9 +1318,10 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{first_cluster_id} **Action:** - Create a new cluster reusing `{cluster_name}` captured from Step 1's response: ```bash -curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ +jq --arg name "{cluster_name}" '.name = $name' testdata/payloads/clusters/cluster-request.json \ +| curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ - -d @testdata/payloads/clusters/cluster-request.json + -d @- ``` - Record the `id` as `{second_cluster_id}` @@ -1305,7 +1344,6 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{second_cluster_id}/statuses **Expected Result:** - Cluster `Reconciled` condition `status: "True"` at `generation: 1` -- `Ready` condition `status: "True"` - All adapters report `Applied: True`, `Available: True`, `Health: True` with `observed_generation: 1` - No adapter errors related to pre-existing resources, duplicate subscriptions, or namespace conflicts @@ -1325,7 +1363,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{first_cluster_id} ```bash curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{second_cluster_id} ``` -- Wait for hard-delete to complete (poll until GET returns 404). +- Poll until GET returns 404 (hard-delete executes automatically when `Reconciled=True`). - If cleanup fails, fall back to namespace deletion: ```bash kubectl delete namespace {second_cluster_id} --ignore-not-found @@ -1335,3 +1373,349 @@ kubectl delete namespace {second_cluster_id} --ignore-not-found - All test resources are cleaned up --- + +## Test Title: LIST returns soft-deleted clusters alongside active clusters + +### Description + +This test validates that soft-deleted clusters (with `deleted_time` set) remain visible in LIST responses alongside active clusters. The test uses a Sentinel fence (scale `sentinel-clusters` to 0) immediately after DELETE so the observation window is deterministic and not dependent on reconciliation races. + +--- + +| **Field** | **Value** | +|-----------|-----------| +| **Pos/Neg** | Positive | +| **Priority** | Tier1 | +| **Status** | Draft | +| **Automation** | Not Automated | +| **Version** | Post-MVP | +| **Created** | 2026-04-17 | +| **Updated** | 2026-04-28 | + +--- + +### Preconditions + +1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources +2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully +3. The adapters defined in testdata/adapter-configs are all deployed successfully + +--- + +### Test Steps + +#### Step 1: Create two clusters and wait for Reconciled state + +**Action:** +- Create two clusters: +```bash +curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ + -H "Content-Type: application/json" \ + -d @testdata/payloads/clusters/cluster-request.json +``` +```bash +curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ + -H "Content-Type: application/json" \ + -d @testdata/payloads/clusters/cluster-request.json +``` +- Wait for both to reach Reconciled state +- Record IDs as `{active_cluster_id}` and `{deleted_cluster_id}` + +**Expected Result:** +- Both clusters reach `Reconciled: True` + +#### Step 2: Soft-delete one cluster + +**Action:** +- Soft-delete one cluster: +```bash +curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{deleted_cluster_id} +``` +- Scale Sentinel for clusters to 0 replicas to freeze reconciliation while visibility assertions run: +```bash +kubectl scale deployment/sentinel-clusters -n hyperfleet --replicas=0 +kubectl rollout status deployment/sentinel-clusters -n hyperfleet --timeout=60s +``` + +**Expected Result:** +- Response returns HTTP 202 (Accepted) with `deleted_time` set +- `generation` on `{deleted_cluster_id}` is incremented to the post-delete generation +- Sentinel cluster reconciler is paused, preventing hard-delete progression during visibility checks + +#### Step 3: LIST all clusters and verify both appear before hard-delete completes + +**Action:** +- Poll LIST with `Eventually` until both the active and deleted clusters are present simultaneously. Use framework-configured polling/timeout values (for example, `500ms` interval and `10s` timeout): +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters +``` + +**Expected Result:** +- At least one LIST returns both `{active_cluster_id}` and `{deleted_cluster_id}` +- `{active_cluster_id}` has `deleted_time` as null/absent +- `{deleted_cluster_id}` has `deleted_time` set to a valid RFC3339 timestamp +- Both clusters have their full resource representation (conditions, spec, labels) + +#### Step 4: Verify GET for each cluster returns the expected state before hard-delete completes + +**Action:** +- Poll GET on `{deleted_cluster_id}` with `Eventually` until the soft-deleted cluster is observed via HTTP 200 with `deleted_time` populated. While the Sentinel fence is active, HTTP 404 in this step is a failure (it means visibility was not proven). Use framework-configured polling/timeout values (for example, `500ms` interval and `10s` timeout), then GET the active cluster: +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{active_cluster_id} +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{deleted_cluster_id} +``` + +**Expected Result:** +- Active cluster: HTTP 200, no `deleted_time`, `Reconciled: True` +- Deleted cluster: observed at least once as HTTP 200 with `deleted_time` set while reconciliation is paused +- The deleted cluster remains observable long enough to distinguish it from the active cluster +- HTTP 404 is not an acceptable success outcome for this visibility step + +#### Step 5: Cleanup resources + +**Action:** +- Scale Sentinel for clusters back to 1 replica to resume reconciliation: +```bash +kubectl scale deployment/sentinel-clusters -n hyperfleet --replicas=1 +kubectl rollout status deployment/sentinel-clusters -n hyperfleet --timeout=60s +``` +- Delete the active cluster: +```bash +curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{active_cluster_id} +``` +- If either cluster still exists after the assertions, poll until both return HTTP 404 (hard-delete executes automatically when `Reconciled=True`) +- The framework cleanup helpers can handle any remaining lifecycle in `AfterEach` + +**Expected Result:** +- Both clusters are eventually hard-deleted (GET returns HTTP 404) + +--- + +## Test Title: Cascade DELETE on cluster while a child nodepool is already deleting + +### Description + +This test validates the interaction between individual nodepool deletion and parent cluster cascade deletion. When a nodepool is already soft-deleted (has `deleted_time` set) and the parent cluster is subsequently deleted, the cascade must not overwrite the nodepool's existing `deleted_time`. The nodepool's original deletion timestamp and lifecycle must be preserved. Both the nodepool and cluster must eventually complete their deletion lifecycles. + +--- + +| **Field** | **Value** | +|-----------|-----------| +| **Pos/Neg** | Positive | +| **Priority** | Tier2 | +| **Status** | Draft | +| **Automation** | Not Automated | +| **Version** | Post-MVP | +| **Created** | 2026-04-17 | +| **Updated** | 2026-04-17 | + +--- + +### Preconditions + +1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources +2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully +3. The adapters defined in testdata/adapter-configs are all deployed successfully + +--- + +### Test Steps + +#### Step 1: Create a cluster with a nodepool and wait for Reconciled state + +**Action:** +- Create a cluster and one nodepool: +```bash +curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ + -H "Content-Type: application/json" \ + -d @testdata/payloads/clusters/cluster-request.json +``` +```bash +curl -X POST ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools \ + -H "Content-Type: application/json" \ + -d @testdata/payloads/nodepools/nodepool-request.json +``` +- Wait for both to reach Reconciled state + +**Expected Result:** +- Cluster and nodepool reach `Reconciled: True` + +#### Step 2: Soft-delete the nodepool first (do NOT wait for hard-delete) + +**Action:** +```bash +curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} +``` +- Record the nodepool's `deleted_time` as `{nodepool_original_deleted_time}` + +**Expected Result:** +- Response returns HTTP 202 (Accepted) with `deleted_time` set on the nodepool + +#### Step 3: Immediately delete the parent cluster + +**Action:** +- Without waiting for the nodepool deletion to complete: +```bash +curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} +``` + +**Expected Result:** +- Response returns HTTP 202 (Accepted) with `deleted_time` set on the cluster + +#### Step 4: Verify nodepool's deleted_time is preserved (not overwritten by cascade) + +**Action:** +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} +``` + +**Expected Result:** +- Nodepool `deleted_time` equals `{nodepool_original_deleted_time}` (preserved from Step 2, not overwritten by the cluster cascade) +- The cascade's `WHERE deleted_time IS NULL` guard should have skipped the already-deleted nodepool + +#### Step 5: Verify both resources complete deletion lifecycle + +**Action:** +- Poll until both are hard-deleted (hard-delete executes automatically when `Reconciled=True`): +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} +``` + +**Expected Result:** +- Nodepool adapters report `Finalized: True` +- Cluster adapters report `Finalized: True` +- Both return HTTP 404 after hard-delete completes + +#### Step 6: Cleanup resources + +**Action:** +- If the test failed before hard-delete, fall back to namespace deletion: +```bash +kubectl delete namespace {cluster_id} --ignore-not-found +``` + +**Expected Result:** +- All test resources are cleaned up + +--- + +## Test Title: Cascade DELETE on cluster while child nodepool is mid-update-reconciliation + +### Description + +This test validates the interaction between nodepool update reconciliation and parent cluster cascade deletion. When a nodepool has been updated via PATCH (generation incremented, adapters not yet reconciled) and the parent cluster is subsequently deleted, the cascade must set `deleted_time` on the nodepool. Nodepool adapters must detect the soft-delete and switch to deletion mode, abandoning the in-flight update reconciliation. This is distinct from [Cascade DELETE on cluster while a child nodepool is already deleting](#test-title-cascade-delete-on-cluster-while-a-child-nodepool-is-already-deleting) (matrix #26) because here the nodepool has a pending spec update — the adapter must prioritize deletion over update reconciliation at a generation that has been bumped by both the PATCH and the cascade DELETE. + +--- + +| **Field** | **Value** | +|-----------|-----------| +| **Pos/Neg** | Positive | +| **Priority** | Tier2 | +| **Status** | Draft | +| **Automation** | Not Automated | +| **Version** | Post-MVP | +| **Created** | 2026-04-17 | +| **Updated** | 2026-04-17 | + +--- + +### Preconditions + +1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources +2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully +3. The adapters defined in testdata/adapter-configs are all deployed successfully + +--- + +### Test Steps + +#### Step 1: Create a cluster with a nodepool and wait for Reconciled state + +**Action:** +- Create a cluster and one nodepool: +```bash +curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ + -H "Content-Type: application/json" \ + -d @testdata/payloads/clusters/cluster-request.json +``` +```bash +curl -X POST ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools \ + -H "Content-Type: application/json" \ + -d @testdata/payloads/nodepools/nodepool-request.json +``` +- Wait for both to reach Reconciled state at `generation: 1` + +**Expected Result:** +- Cluster and nodepool reach `Reconciled: True` at `generation: 1` +- All adapters report `observed_generation: 1` + +#### Step 2: PATCH the nodepool to trigger update reconciliation (do NOT wait for reconciliation) + +**Action:** +- Send a PATCH to the nodepool to bump its generation: +```bash +curl -X PATCH ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} \ + -H "Content-Type: application/json" \ + -d '{"spec": {"trigger-update": "true"}}' +``` + +**Expected Result:** +- Response returns HTTP 200 with nodepool `generation: 2` +- Nodepool adapters have not yet reconciled to generation 2 (update in flight) + +#### Step 3: Immediately DELETE the parent cluster before nodepool reconciliation completes + +**Action:** +- Without waiting for nodepool adapters to reconcile: +```bash +curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} +``` + +**Expected Result:** +- Response returns HTTP 202 (Accepted) with `deleted_time` set on the cluster +- Cluster `generation` incremented + +#### Step 4: Verify cascade sets deleted_time on the nodepool + +**Action:** +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} +``` + +**Expected Result:** +- Nodepool has `deleted_time` set (cascaded from parent cluster) +- Nodepool `generation` is incremented beyond 2 (bumped by both the PATCH and the cascade DELETE) + +#### Step 5: Verify all adapters finalize and both resources are hard-deleted + +**Action:** +- Poll adapter statuses for both nodepool and cluster: +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id}/statuses +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses +``` +- Poll until hard-delete completes (hard-delete executes automatically when `Reconciled=True`): +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} +``` + +**Expected Result:** +- Nodepool adapters report `Finalized: True` at the final generation (not the update generation) +- Nodepool adapters did not complete update reconciliation for generation 2 — they detected `deleted_time` and switched to cleanup mode +- Cluster adapters report `Finalized: True` +- Both nodepool and cluster return HTTP 404 after hard-delete + +#### Step 6: Cleanup resources + +**Action:** +- If the test failed before hard-delete, fall back to namespace deletion: +```bash +kubectl delete namespace {cluster_id} --ignore-not-found +``` + +**Expected Result:** +- All test resources are cleaned up + +--- diff --git a/test-design/testcases/delete-nodepool.md b/test-design/testcases/delete-nodepool.md index 80475d7..8a1b081 100644 --- a/test-design/testcases/delete-nodepool.md +++ b/test-design/testcases/delete-nodepool.md @@ -7,9 +7,12 @@ 3. [Re-DELETE on already-deleted nodepool is idempotent](#test-title-re-delete-on-already-deleted-nodepool-is-idempotent) 4. [DELETE non-existent nodepool returns 404](#test-title-delete-non-existent-nodepool-returns-404) 5. [PATCH to soft-deleted nodepool returns 409 Conflict](#test-title-patch-to-soft-deleted-nodepool-returns-409-conflict) +6. [Soft-deleted nodepool remains visible via GET and LIST](#test-title-soft-deleted-nodepool-remains-visible-via-get-and-list) --- +> **Hard-delete mechanism:** Hard-delete executes inline within the `POST /adapter_statuses` request that computes `Reconciled=True`. No separate endpoint or background process — test steps simply poll until GET returns 404. See [hard-delete-design.md](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/components/api-service/hard-delete-design.md). + ## Test Title: Nodepool deletion happy path -- soft-delete through hard-delete ### Description @@ -35,14 +38,12 @@ This test validates the complete nodepool deletion lifecycle. It verifies that w 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE and hard-delete endpoints are deployed and operational -5. Reconciled status aggregation is implemented --- ### Test Steps -#### Step 1: Create a cluster and a nodepool, wait for Ready state +#### Step 1: Create a cluster and a nodepool, wait for Reconciled state **Action:** - Create a cluster: @@ -57,11 +58,11 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools \ -H "Content-Type: application/json" \ -d @testdata/payloads/nodepools/nodepool-request.json ``` -- Wait for both cluster and nodepool to reach Ready state +- Wait for both cluster and nodepool to reach Reconciled state **Expected Result:** -- Cluster `Ready` condition `status: "True"` -- Nodepool `Ready` condition `status: "True"` +- Cluster `Reconciled` condition `status: "True"` +- Nodepool `Reconciled` condition `status: "True"` #### Step 2: Send DELETE request for the nodepool only @@ -93,7 +94,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepo #### Step 4: Verify nodepool reaches Reconciled=True and is hard-deleted **Action:** -- Poll nodepool status, then attempt GET after hard-delete: +- Poll nodepool status until hard-delete completes (executes automatically when `Reconciled=True`): ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} ``` @@ -112,7 +113,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} **Expected Result:** - Cluster does NOT have `deleted_time` set -- Cluster `Ready` condition remains `status: "True"` +- Cluster `Reconciled` condition remains `status: "True"` - Cluster `Available` condition remains `status: "True"` - Cluster is fully operational and unaffected by the nodepool deletion @@ -154,13 +155,12 @@ This test validates isolation between sibling nodepools during deletion. When on 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE endpoint is deployed and operational --- ### Test Steps -#### Step 1: Create a cluster with two nodepools and wait for Ready state +#### Step 1: Create a cluster with two nodepools and wait for Reconciled state **Action:** - Create a cluster and two nodepools (each call generates a unique name via `{{.Random}}` template): @@ -179,10 +179,10 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools \ -H "Content-Type: application/json" \ -d @testdata/payloads/nodepools/nodepool-request.json ``` -- Wait for all to reach Ready state +- Wait for all to reach Reconciled state **Expected Result:** -- Cluster and both nodepools reach `Ready` condition `status: "True"` +- Cluster and both nodepools reach `Reconciled` condition `status: "True"` #### Step 2: Delete one nodepool @@ -208,7 +208,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepo **Expected Result:** - Sibling nodepool does NOT have `deleted_time` set -- Sibling nodepool `Ready` condition remains `status: "True"` +- Sibling nodepool `Reconciled` condition remains `status: "True"` - Sibling nodepool adapter statuses are unchanged (`Applied: True`, `Available: True`, `Health: True`) #### Step 4: Verify parent cluster is unaffected @@ -220,7 +220,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} **Expected Result:** - Cluster does NOT have `deleted_time` set -- Cluster `Ready` condition remains `status: "True"` +- Cluster `Reconciled` condition remains `status: "True"` #### Step 5: Cleanup resources @@ -260,16 +260,15 @@ This test validates that calling DELETE on a nodepool that has already been soft 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE endpoint is deployed and operational --- ### Test Steps -#### Step 1: Create a cluster and nodepool, wait for Ready state +#### Step 1: Create a cluster and nodepool, wait for Reconciled state **Action:** -- Create a cluster and nodepool, wait for Ready: +- Create a cluster and nodepool, wait for Reconciled: ```bash curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ @@ -282,7 +281,7 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools \ ``` **Expected Result:** -- Cluster and nodepool reach Ready state +- Cluster and nodepool reach `Reconciled: True` #### Step 2: Send first DELETE request @@ -392,21 +391,19 @@ curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} This test validates that the API rejects mutation requests (PATCH) to nodepools that have been soft-deleted. Once a nodepool has `deleted_time` set, no spec modifications should be allowed to prevent new generation events from triggering reconciliation while deletion cleanup is in progress. -**Note:** Same mechanism as the cluster PATCH 409 test case — a PATCH on a tombstoned nodepool bumps `generation`, creating a mismatch that blocks hard-delete until adapters re-process at the new generation. The adapter won't recreate K8s resources (deletion check short-circuits apply), but the round-trip through Sentinel and adapter delays hard-delete. A 409 guard prevents this. - -**Status note:** This test case requires the API to implement a mutation guard for tombstoned resources. Until then, PATCH will succeed on soft-deleted nodepools. +**Note:** Same mechanism as the cluster PATCH 409 test case — a PATCH on a soft-deleted nodepool bumps `generation`, creating a mismatch that blocks hard-delete until adapters re-process at the new generation. The adapter won't recreate K8s resources (deletion check short-circuits apply), but the round-trip through Sentinel and adapter delays hard-delete. A 409 guard prevents this. --- | **Field** | **Value** | |-----------|-----------| | **Pos/Neg** | Negative | -| **Priority** | Tier0 | +| **Priority** | Tier1 | | **Status** | Draft | | **Automation** | Not Automated | | **Version** | Post-MVP | | **Created** | 2026-04-15 | -| **Updated** | 2026-04-16 | +| **Updated** | 2026-04-28 | --- @@ -415,14 +412,12 @@ This test validates that the API rejects mutation requests (PATCH) to nodepools 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. DELETE and PATCH endpoints are deployed and operational -5. Mutation guard for tombstoned resources is implemented in the API --- ### Test Steps -#### Step 1: Create a cluster and nodepool, wait for Ready state +#### Step 1: Create a cluster and nodepool, wait for Reconciled state **Action:** - Create a cluster and nodepool: @@ -436,10 +431,10 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools \ -H "Content-Type: application/json" \ -d @testdata/payloads/nodepools/nodepool-request.json ``` -- Wait for both to reach Ready state +- Wait for both to reach Reconciled state **Expected Result:** -- Cluster and nodepool reach `Ready` condition `status: "True"` +- Cluster and nodepool reach `Reconciled` condition `status: "True"` - Nodepool at `generation: 1` #### Step 2: Send DELETE request to soft-delete the nodepool @@ -492,3 +487,136 @@ curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} - Cluster and all associated resources are cleaned up --- + +## Test Title: Soft-deleted nodepool remains visible via GET and LIST + +### Description + +This test validates that after a nodepool is soft-deleted, it remains queryable via GET and LIST before hard-delete. The test uses a Sentinel fence (scale `sentinel-nodepools` to 0) immediately after DELETE so the visibility window is deterministic and not dependent on reconciliation timing races. + +--- + +| **Field** | **Value** | +|-----------|-----------| +| **Pos/Neg** | Positive | +| **Priority** | Tier1 | +| **Status** | Draft | +| **Automation** | Not Automated | +| **Version** | Post-MVP | +| **Created** | 2026-04-17 | +| **Updated** | 2026-04-28 | + +--- + +### Preconditions + +1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources +2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully +3. The adapters defined in testdata/adapter-configs are all deployed successfully + +--- + +### Test Steps + +#### Step 1: Create a cluster with two nodepools and wait for Reconciled state + +**Action:** +- Create a cluster and two nodepools: +```bash +curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ + -H "Content-Type: application/json" \ + -d @testdata/payloads/clusters/cluster-request.json +``` +```bash +curl -X POST ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools \ + -H "Content-Type: application/json" \ + -d @testdata/payloads/nodepools/nodepool-request.json +``` +```bash +curl -X POST ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools \ + -H "Content-Type: application/json" \ + -d @testdata/payloads/nodepools/nodepool-request.json +``` +- Wait for all to reach Reconciled state +- Record IDs as `{active_nodepool_id}` and `{deleted_nodepool_id}` + +**Expected Result:** +- Cluster and both nodepools reach `Reconciled: True` + +#### Step 2: Soft-delete one nodepool + +**Action:** +- Soft-delete one nodepool: +```bash +curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{deleted_nodepool_id} +``` +- Scale Sentinel for nodepools to 0 replicas to freeze reconciliation while visibility assertions run: +```bash +kubectl scale deployment/sentinel-nodepools -n hyperfleet --replicas=0 +kubectl rollout status deployment/sentinel-nodepools -n hyperfleet --timeout=60s +``` + +**Expected Result:** +- Response returns HTTP 202 (Accepted) with `deleted_time` set +- `generation` on `{deleted_nodepool_id}` is incremented to the post-delete generation +- Sentinel nodepool reconciler is paused, preventing hard-delete progression during visibility checks + +#### Step 3: Verify GET observes the soft-deleted nodepool before hard-delete + +**Action:** +- Poll GET with `Eventually` until the soft-deleted nodepool is observed via HTTP 200 with `deleted_time` populated. While the Sentinel fence is active, HTTP 404 in this step is a failure (it means visibility was not proven). Use framework-configured polling/timeout values (for example, `500ms` interval and `10s` timeout): +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{deleted_nodepool_id} +``` + +**Expected Result:** +- At least one GET returns HTTP 200 (OK) with the nodepool object present and `deleted_time` populated +- This proves the nodepool remains visible in soft-deleted state while reconciliation is paused +- HTTP 404 is not an acceptable success outcome for this visibility step + +**Note:** During this observation period, `Reconciled` is frequently `False` while adapters finalize the post-delete generation, but it can transition quickly depending on system timing. + +#### Step 4: Verify LIST includes both active and soft-deleted nodepools before hard-delete completes + +**Action:** +- Poll LIST with `Eventually` until both the active and deleted nodepools are present simultaneously. Use framework-configured polling/timeout values (for example, `500ms` interval and `10s` timeout): +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools +``` + +**Expected Result:** +- At least one LIST returns both `{active_nodepool_id}` and `{deleted_nodepool_id}` +- `{active_nodepool_id}` has `deleted_time` as null/absent +- `{deleted_nodepool_id}` has `deleted_time` set to a valid RFC3339 timestamp +- Both nodepools have their full resource representation (conditions, spec, labels) + +#### Step 5: Verify active nodepool is unaffected + +**Action:** +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{active_nodepool_id} +``` + +**Expected Result:** +- Active nodepool: HTTP 200, no `deleted_time`, `Reconciled: True` +- Active nodepool adapter statuses are unchanged + +#### Step 6: Cleanup resources + +**Action:** +- Scale Sentinel for nodepools back to 1 replica to resume reconciliation: +```bash +kubectl scale deployment/sentinel-nodepools -n hyperfleet --replicas=1 +kubectl rollout status deployment/sentinel-nodepools -n hyperfleet --timeout=60s +``` +- Delete the cluster (cascades to remaining nodepool): +```bash +curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} +``` +- If the deleted nodepool or parent cluster still exists after the assertions, poll until GET returns HTTP 404 (hard-delete executes automatically when `Reconciled=True`) +- The framework cleanup helpers can handle any remaining lifecycle in `AfterEach` + +**Expected Result:** +- Cluster and all nodepools are eventually hard-deleted (GET returns HTTP 404) + +--- diff --git a/test-design/testcases/update-cluster.md b/test-design/testcases/update-cluster.md index eaa7b9b..b242adc 100644 --- a/test-design/testcases/update-cluster.md +++ b/test-design/testcases/update-cluster.md @@ -5,7 +5,8 @@ 1. [Cluster update via PATCH triggers reconciliation and reaches Reconciled](#test-title-cluster-update-via-patch-triggers-reconciliation-and-reaches-reconciled) 2. [Adapter statuses transition during update reconciliation](#test-title-adapter-statuses-transition-during-update-reconciliation) 3. [Multiple rapid updates coalesce to latest generation](#test-title-multiple-rapid-updates-coalesce-to-latest-generation) -4. [PATCH with invalid payload is rejected without changing cluster state](#test-title-patch-with-invalid-payload-is-rejected-without-changing-cluster-state) +4. [Labels-only PATCH bumps generation and triggers reconciliation](#test-title-labels-only-patch-bumps-generation-and-triggers-reconciliation) +5. [No-op PATCH does not increment generation](#test-title-no-op-patch-does-not-increment-generation) --- @@ -13,7 +14,7 @@ ### Description -This test validates the cluster update lifecycle end-to-end. It verifies that when a PATCH request modifies a cluster's spec, the API increments the `generation`, Sentinel detects the generation change and publishes a reconciliation event, adapters reconcile to the new generation reporting updated `observed_generation`, and the cluster reaches `Reconciled=True` at the new generation. This confirms the complete update-reconciliation pipeline works correctly. +This test validates the cluster update lifecycle end-to-end. It verifies that when a PATCH request modifies a cluster's spec, the API increments the `generation`, Sentinel detects the generation change and publishes a reconciliation event, adapters reconcile to the new generation reporting updated `observed_generation`, and the cluster reaches `Reconciled=True` and `Available=True` at the new generation. This confirms the complete update-reconciliation pipeline works correctly. --- @@ -25,7 +26,7 @@ This test validates the cluster update lifecycle end-to-end. It verifies that wh | **Automation** | Not Automated | | **Version** | Post-MVP | | **Created** | 2026-04-15 | -| **Updated** | 2026-04-15 | +| **Updated** | 2026-04-28 | --- @@ -34,17 +35,15 @@ This test validates the cluster update lifecycle end-to-end. It verifies that wh 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. PATCH endpoint is deployed and operational -5. Reconciled status aggregation is implemented --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready state at generation 1 +#### Step 1: Create a cluster and wait for Reconciled and Available state at generation 1 **Action:** -- Create a cluster and wait for Ready: +- Create a cluster and wait for full convergence: ```bash curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ @@ -52,10 +51,11 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ ``` **Expected Result:** -- Cluster reaches `Ready` condition `status: "True"` and `Available` condition `status: "True"` - Cluster `generation` equals 1 -- `Reconciled` condition `status: "True"` with `observed_generation: 1` +- Cluster `Reconciled` condition `status: "True"` with `observed_generation: 1` +- Cluster `Available` condition `status: "True"` with `observed_generation: 1` - All required adapters report `observed_generation: 1` +- **Per-adapter conditions on cluster status**: each required adapter condition on the cluster resource has `status: "True"` #### Step 2: Send PATCH request to update the cluster spec @@ -90,7 +90,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses - **Adapter status metadata validation** (for each required adapter): - `last_report_time`: Updated to a timestamp after the PATCH request -#### Step 4: Verify cluster reaches Reconciled=True at new generation +#### Step 4: Verify cluster reaches Reconciled=True and Available=True at new generation **Action:** - Retrieve the cluster status: @@ -100,24 +100,12 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} **Expected Result:** - Cluster `Reconciled` condition `status: "True"` with `observed_generation: 2` -- Cluster `Ready` condition `status: "True"` -- Cluster `Available` condition `status: "True"` +- Cluster `Available` condition `status: "True"` with `observed_generation: 2` +- Cluster `id` is unchanged from Step 1 - `generation` equals 2 +- **Per-adapter conditions on cluster status**: each required adapter condition on the cluster resource has `status: "True"` -#### Step 5: Verify adapter statuses reflect the update - -**Action:** -- Retrieve adapter statuses to confirm all adapters reconciled the update: -```bash -curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses -``` - -**Expected Result:** -- All required adapters report `observed_generation: 2` -- All adapters report `Applied: True` (confirming managed K8s resources were updated) -- This implicitly confirms K8s resources (e.g., namespace annotation `hyperfleet.io/generation`) reflect the update - -#### Step 6: Cleanup resources +#### Step 5: Cleanup resources **Action:** ```bash @@ -133,7 +121,7 @@ curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ### Description -This test validates the intermediate status transitions during update reconciliation. When a cluster spec is updated, there is a window where adapters have not yet reconciled to the new generation. During this window, `Reconciled` should be `False` (indicating stale adapter statuses relative to the new generation). This test captures and validates these intermediate states. +This test validates the intermediate status transitions during update reconciliation. When a cluster spec is updated, there is a window where adapters have not yet reconciled to the new generation. During this window, `Reconciled` should be `False` (indicating stale adapter statuses relative to the new generation). To guarantee this window is observable, a dedicated crash-adapter is deployed and scaled to 0 before the PATCH. With a stuck adapter, `Reconciled` remains `False` indefinitely, allowing reliable assertion via `Consistently`. After verification, the adapter is restored and full convergence is confirmed. --- @@ -145,7 +133,7 @@ This test validates the intermediate status transitions during update reconcilia | **Automation** | Not Automated | | **Version** | Post-MVP | | **Created** | 2026-04-15 | -| **Updated** | 2026-04-15 | +| **Updated** | 2026-04-28 | --- @@ -154,16 +142,17 @@ This test validates the intermediate status transitions during update reconcilia 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. PATCH endpoint is deployed and operational -5. Reconciled status aggregation is implemented +4. A dedicated crash-adapter is available for deployment via Helm --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready and Reconciled at generation 1 +#### Step 1: Deploy crash-adapter and create a cluster, wait for Reconciled at generation 1 **Action:** +- Deploy a dedicated crash-adapter via Helm (`${ADAPTER_DEPLOYMENT_NAME}`), separate from the normal adapters +- Configure API required adapters to include crash-adapter - Create a cluster and wait for full convergence: ```bash curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ @@ -173,59 +162,79 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ **Expected Result:** - Cluster `Reconciled` condition `status: "True"` at `generation: 1` -- All adapters report `observed_generation: 1` +- All adapters (including crash-adapter) report `observed_generation: 1` -#### Step 2: Send PATCH request and immediately poll for intermediate state +#### Step 2: Scale down crash-adapter, then send PATCH request **Action:** +- Scale the crash-adapter deployment to 0 replicas: +```bash +kubectl scale deployment/${ADAPTER_DEPLOYMENT_NAME} -n hyperfleet --replicas=0 +``` +- Wait for the crash-adapter pod to terminate - Send PATCH to trigger generation increment: ```bash curl -X PATCH ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} \ -H "Content-Type: application/json" \ -d '{"spec": {"trigger-reconcile": "true"}}' ``` -- Immediately poll cluster status: + +**Expected Result:** +- Response returns HTTP 200 with `generation: 2` +- crash-adapter cannot reconcile to generation 2 (it is unavailable) + +#### Step 3: Verify Reconciled=False persists while crash-adapter is down + +**Action:** +- Poll cluster GET repeatedly over multiple polling intervals while crash-adapter remains down: ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` -- Immediately poll adapter statuses: +- Poll adapter statuses: ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses ``` **Expected Result:** -- Cluster `generation` is now 2 -- **Intermediate state** (captured if polled before adapters reconcile): - - `Reconciled` condition `status: "False"` (adapters have not yet reported at generation 2) - - Some or all adapters still report `observed_generation: 1` (stale relative to generation 2) +- Cluster `Reconciled` condition `status: "False"` persists over multiple polling cycles +- Healthy adapters report `observed_generation: 2` (they reconciled the update) +- crash-adapter either has no status entry or still reports `observed_generation: 1` (stale) -**Note:** This intermediate state may be very brief depending on adapter reconciliation speed. The test should poll immediately after PATCH to maximize the chance of capturing it. - -#### Step 3: Wait for full convergence and verify final state +#### Step 4: Restore crash-adapter and verify full convergence **Action:** -- Continue polling until all adapters report the new generation: +- Scale the crash-adapter back up: ```bash -curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses +kubectl scale deployment/${ADAPTER_DEPLOYMENT_NAME} -n hyperfleet --replicas=1 +``` +- Wait for crash-adapter to become ready: +```bash +kubectl rollout status deployment/${ADAPTER_DEPLOYMENT_NAME} -n hyperfleet --timeout=60s ``` +- Poll until cluster reaches `Reconciled: True`: ```bash curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` **Expected Result:** +- crash-adapter reconciles and reports `observed_generation: 2` - All adapters report `observed_generation: 2` -- Cluster `Reconciled` condition transitions to `status: "True"` -- Full state transition observed: `Reconciled: True (gen 1)` -> `Reconciled: False (gen 2 pending)` -> `Reconciled: True (gen 2)` +- Cluster `Reconciled` condition transitions to `status: "True"` with `observed_generation: 2` +- Full state transition confirmed: `Reconciled: True (gen 1)` -> `Reconciled: False (gen 2 pending)` -> `Reconciled: True (gen 2)` -#### Step 4: Cleanup resources +#### Step 5: Cleanup resources **Action:** +- Restore API required adapters to original config +- Uninstall crash-adapter Helm release +- Clean up Pub/Sub subscription ```bash curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` **Expected Result:** - Cluster and all associated resources are cleaned up +- crash-adapter deployment is removed --- @@ -254,17 +263,15 @@ This test validates that when multiple PATCH requests are sent in rapid successi 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. PATCH endpoint is deployed and operational -5. Reconciled status aggregation is implemented --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready at generation 1 +#### Step 1: Create a cluster and wait for Reconciled at generation 1 **Action:** -- Create a cluster and wait for Ready: +- Create a cluster and wait for Reconciled: ```bash curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ @@ -272,7 +279,7 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ ``` **Expected Result:** -- Cluster reaches `Ready: True`, `Reconciled: True` at `generation: 1` +- Cluster reaches `Reconciled: True` at `generation: 1` #### Step 2: Send three PATCH requests in rapid succession @@ -321,7 +328,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} **Expected Result:** - Cluster `generation` equals 4 - Cluster `Reconciled` condition `status: "True"` with `observed_generation: 4` -- Cluster `Ready` condition `status: "True"` +- Cluster `Available` condition `status: "True"` - Cluster spec contains `{"update": "third"}` (the last applied value) #### Step 5: Cleanup resources @@ -334,25 +341,122 @@ curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} **Expected Result:** - Cluster and all associated resources are cleaned up + +## Test Title: Labels-only PATCH bumps generation and triggers reconciliation + +### Description + +This test validates that a PATCH request that only modifies `labels` (without changing `spec`) increments the cluster's `generation` and triggers adapter reconciliation. Generation is incremented when either `spec` or `labels` change. After a labels-only PATCH, `Reconciled` transitions to `False` (generation mismatch), adapters reconcile to the new generation, and `Reconciled` returns to `True`. + +--- + +| **Field** | **Value** | +|-----------|-----------| +| **Pos/Neg** | Positive | +| **Priority** | Tier1 | +| **Status** | Draft | +| **Automation** | Not Automated | +| **Version** | Post-MVP | +| **Created** | 2026-04-17 | +| **Updated** | 2026-04-20 | + +--- + +### Preconditions + +1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources +2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully +3. The adapters defined in testdata/adapter-configs are all deployed successfully + +--- + +### Test Steps + +#### Step 1: Create a cluster and wait for Reconciled state at generation 1 + +**Action:** +- Create a cluster and wait for full convergence: +```bash +curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ + -H "Content-Type: application/json" \ + -d @testdata/payloads/clusters/cluster-request.json +``` +- Wait for `Reconciled` condition `status: "True"` at `generation: 1` + +**Expected Result:** +- Cluster reaches `Reconciled: True` at `generation: 1` +- All adapters report `observed_generation: 1` + +#### Step 2: Send labels-only PATCH request + +**Action:** +```bash +curl -X PATCH ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} \ + -H "Content-Type: application/json" \ + -d '{"labels": {"env": "staging", "team": "fleet-management"}}' +``` + +**Expected Result:** +- Response returns HTTP 200 (OK) +- `generation` incremented from 1 to 2 +- Labels in the response include the new values (`env: staging`, `team: fleet-management`) +- `spec` is unchanged from Step 1 + +#### Step 3: Verify adapters reconcile to the new generation + +**Action:** +- Poll adapter statuses until all adapters report the new generation: +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses +``` + +**Expected Result:** +- All required adapters report `observed_generation: 2` +- Each adapter has `Applied: True`, `Available: True`, `Health: True` + +#### Step 4: Verify cluster reaches Reconciled=True and Available=True at new generation + +**Action:** +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} +``` + +**Expected Result:** +- `generation` equals 2 +- `Reconciled` condition `status: "True"` with `observed_generation: 2` +- `Available` condition `status: "True"` with `observed_generation: 2` +- Labels reflect the PATCH update +- `spec` is unchanged + +#### Step 5: Cleanup resources + +**Action:** +```bash +curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} +``` + +**Expected Result:** +- Cluster and all associated resources are cleaned up + --- -## Test Title: PATCH with invalid payload is rejected without changing cluster state +## Test Title: No-op PATCH does not increment generation ### Description -This test validates that the API rejects malformed or constraint-violating PATCH requests on a cluster with an HTTP 4xx response, does not increment `generation`, and does not trigger reconciliation. It complements the happy-path update tests by locking down the negative API contract for cluster spec mutations (malformed JSON, type violations, read-only field mutation). +This test validates PATCH behavior at the no-op boundary for cluster updates. It covers four deterministic cases: canonical replay of the current spec, semantically identical replay with different raw JSON formatting, explicit empty-object replacement, and repeated identical PATCHes after the replacement. The objective is to verify generation changes only when the effective spec state changes. --- | **Field** | **Value** | |-----------|-----------| -| **Pos/Neg** | Negative | +| **Pos/Neg** | Positive | | **Priority** | Tier1 | | **Status** | Draft | | **Automation** | Not Automated | | **Version** | Post-MVP | -| **Created** | 2026-04-16 | -| **Updated** | 2026-04-16 | +| **Created** | 2026-04-28 | +| **Updated** | 2026-04-28 | --- @@ -361,85 +465,125 @@ This test validates that the API rejects malformed or constraint-violating PATCH 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. PATCH endpoint is deployed and operational with request validation enabled --- ### Test Steps -#### Step 1: Create a cluster and wait for Ready at generation 1 +#### Step 1: Create a cluster and wait for Reconciled state at generation 1 **Action:** +- Create a cluster and wait for Reconciled: ```bash curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ -d @testdata/payloads/clusters/cluster-request.json ``` -- Wait for Ready and `Reconciled: True` with `observed_generation: 1` +- Capture the cluster's canonical spec into a shell variable for replay in Step 3: +```bash +CANONICAL_SPEC=$(curl -s ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} | jq -c '.spec') +``` +- Record `generation` as `{G1}` (expected: 1) **Expected Result:** -- Cluster reaches `Ready: True`, `Reconciled: True` at `generation: 1` +- Cluster reaches `Reconciled: True` at `generation: {G1}` +- All adapters report `observed_generation: {G1}` -#### Step 2: Capture baseline state and per-adapter `last_report_time` +#### Step 2: Capture baseline adapter `last_report_time` values **Action:** ```bash -curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} \ - | jq '{id, generation, created_at, deleted_time, conditions, labels, spec}' curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses \ | jq '[.items[] | {adapter, observed_generation, last_report_time}]' ``` **Expected Result:** -- Baseline captured for comparison in Step 4 +- Baseline captured for comparison after Case B and again after Case D -#### Step 3: Attempt PATCH with invalid payloads +#### Step 3: Exercise PATCH behavior at the no-op boundary -For each case, submit the PATCH and capture the response: +**Case A: Byte-identical replay of the canonical spec** -**Case A: Malformed JSON (truncated body, missing closing brace)** +Send the captured `CANONICAL_SPEC` back as the PATCH payload: ```bash curl -i -X PATCH ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} \ -H "Content-Type: application/json" \ - -d '{"spec": {"k": "v"' + -d "$(jq -n --argjson spec "$CANONICAL_SPEC" '{"spec": $spec}')" ``` -**Case B: Wrong type on a typed spec field** -Submit a PATCH that replaces a typed spec field with a value of the wrong type (e.g., provide a string where the schema expects an object or an array). Concrete field choice depends on the cluster OpenAPI schema -- pick any required typed subfield under `spec`. +**Expected Result:** +- Response returns HTTP 200 (OK) +- `generation` equals `{G1}` (unchanged) + +**Case B: Semantic replay with different raw JSON formatting or key order** + +Send the same key-value pairs as `CANONICAL_SPEC` but with reordered keys or pretty-printed formatting: ```bash +REORDERED_SPEC=$(echo "$CANONICAL_SPEC" | jq -S '.') curl -i -X PATCH ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} \ -H "Content-Type: application/json" \ - -d '{"spec": {"release": "not-an-object"}}' + -d "$(jq -n --argjson spec "$REORDERED_SPEC" '{"spec": $spec}')" ``` -**Case C: Attempt to mutate an immutable/server-controlled field (e.g., `id`, `generation`, `created_at`, `deleted_time`)** +**Expected Result:** +- Response returns HTTP 200 (OK) +- `generation` still equals `{G1}` because semantic equivalence alone does not change effective spec state +- No reconciliation is triggered; raw request formatting alone does not change effective state + +**Case C: Explicit empty-object replacement** + +Send a PATCH with `spec` set to an empty object: ```bash curl -i -X PATCH ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} \ -H "Content-Type: application/json" \ - -d '{"id": "different-id", "generation": 99, "created_at": "1970-01-01T00:00:00Z"}' + -d '{"spec": {}}' +``` + +**Expected Result:** +- Response returns HTTP 200 (OK) +- `generation` equals `{G1} + 1` +- Cluster `spec` is now `{}` +- This is the only case in the test that should trigger reconciliation + +**Case D: Repeat the Case C payload three more times (stability check)** + +Send the same empty-object PATCH from Case C three more times in succession: +```bash +for i in 1 2 3; do + curl -i -X PATCH ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} \ + -H "Content-Type: application/json" \ + -d '{"spec": {}}' +done ``` -**Expected Result (for every case):** -- Response is HTTP 400 (Bad Request) or 422 (Unprocessable Entity) -- no 5xx -- Response body contains a structured validation error message identifying the offending field -- For Case C, the server either rejects the request or silently ignores the read-only fields (both are acceptable, but read-only fields must remain unchanged in Step 4) +**Expected Result:** +- All three return HTTP 200 (OK) +- `generation` remains `{G1} + 1` for all three calls because no additional state change occurs after Case C -#### Step 4: Verify cluster state is unchanged +#### Step 4: Verify reconciliation only happened for the state-changing case **Action:** +- After Case B, capture the current cluster and adapter status timestamps: +```bash +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses \ + | jq '[.items[] | {adapter, observed_generation, last_report_time}]' +``` +- After Case D, capture them again: ```bash -curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} \ - | jq '{id, generation, created_at, deleted_time, conditions, labels, spec}' +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/statuses \ | jq '[.items[] | {adapter, observed_generation, last_report_time}]' ``` +- If Case C bumped `generation`, poll until all adapters report the current generation before comparing final state. **Expected Result:** -- `generation` is still 1 (no invalid PATCH incremented it) -- `id`, `created_at`, and `deleted_time` are unchanged from baseline -- Cluster `Reconciled` condition remains `status: "True"` with `observed_generation: 1` -- Cluster `Ready` condition remains `status: "True"` -- All adapter `last_report_time` values are unchanged vs baseline (no spurious reconciliation was triggered) +- After Case B, `generation` still equals `{G1}` and adapter `last_report_time` values still match the Step 2 baseline +- Case C causes the only additional generation bump in the test and may update adapter `last_report_time` values +- After Case D, there are no further generation increments or additional `last_report_time` changes beyond the post-Case-C state +- Final cluster `generation` equals `{G1} + 1` +- Final cluster `spec` equals `{}` +- `observed_generation` on all adapters matches the current cluster `generation` #### Step 5: Cleanup resources @@ -451,4 +595,3 @@ curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} **Expected Result:** - Cluster and all associated resources are cleaned up ---- diff --git a/test-design/testcases/update-delete-test-matrix.md b/test-design/testcases/update-delete-test-matrix.md index d9d0c82..f48149e 100644 --- a/test-design/testcases/update-delete-test-matrix.md +++ b/test-design/testcases/update-delete-test-matrix.md @@ -1,6 +1,17 @@ # Update and Delete Lifecycle -- Test Matrix -Consolidated test matrix for HYPERFLEET-859. Covers positive, negative, and edge case scenarios for the Update (PATCH) and Delete lifecycle across cluster and nodepool resources. +Consolidated test matrix covering positive, negative, and edge case scenarios for the Update (PATCH) and Delete lifecycle across cluster and nodepool resources. Out of scope: creation lifecycle, RBAC (no implementation exists), performance/load testing. + +**Key concepts:** [Cluster lifecycle](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/components/api-service/), [Sentinel](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/components/sentinel/), [Adapter framework](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/components/adapter/), [Hard-delete design](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/components/api-service/hard-delete-design.md). + +**Notation:** Shorthand `Reconciled=True` means `Reconciled` condition `status: "True"`. Same convention applies to other conditions (`Applied`, `Finalized`, `Available`, `Health`). + +**Design assumptions exercised by this matrix:** + +- **Sentinel event publishing:** Every update and delete test relies on Sentinel publishing events on `Reconciled` condition transitions. If Sentinel did not watch this condition, adapters would never reconcile to new generations (#1, #3, #6, #7, #15, #16 would fail). +- **Adapter deletion mode:** Every delete test exercises the adapter's deletion mode switch (triggered by the `lifecycle.delete.when` CEL expression detecting `deleted_time`). If adapters did not switch to deletion mode, they would apply spec instead of finalizing, and `Finalized=True` would never be reported. +- **Adapter delete ordering:** The `lifecycle.delete.when` CEL expression ordering is exercised by delete happy-path tests (#1, #3). Incorrect ordering would result in stuck or failed finalization. +- **Hard-delete mechanism:** Hard-delete executes inline within the `POST /adapter_statuses` request that computes `Reconciled=True` — no separate endpoint or background process. See [hard-delete-design.md](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/components/api-service/hard-delete-design.md). ## Test Matrix @@ -9,8 +20,8 @@ Consolidated test matrix for HYPERFLEET-859. Covers positive, negative, and edge | 1 | Cluster deletion happy path -- soft-delete through hard-delete | Cluster | Positive | Tier0 | [delete-cluster.md](delete-cluster.md#test-title-cluster-deletion-happy-path----soft-delete-through-hard-delete) | DELETE happy path | | 2 | Cluster deletion cascades to child nodepools | Cluster + Nodepool | Positive | Tier0 | [delete-cluster.md](delete-cluster.md#test-title-cluster-deletion-cascades-to-child-nodepools) | DELETE hierarchical | | 3 | Nodepool deletion happy path -- soft-delete through hard-delete | Nodepool | Positive | Tier0 | [delete-nodepool.md](delete-nodepool.md#test-title-nodepool-deletion-happy-path----soft-delete-through-hard-delete) | DELETE happy path | -| 4 | PATCH to soft-deleted cluster returns 409 Conflict | Cluster | Negative | Tier0 | [delete-cluster.md](delete-cluster.md#test-title-patch-to-soft-deleted-cluster-returns-409-conflict) | DELETE API behavior | -| 5 | PATCH to soft-deleted nodepool returns 409 Conflict | Nodepool | Negative | Tier0 | [delete-nodepool.md](delete-nodepool.md#test-title-patch-to-soft-deleted-nodepool-returns-409-conflict) | DELETE API behavior | +| 4 | PATCH to soft-deleted cluster returns 409 Conflict | Cluster | Negative | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-patch-to-soft-deleted-cluster-returns-409-conflict) | DELETE API behavior | +| 5 | PATCH to soft-deleted nodepool returns 409 Conflict | Nodepool | Negative | Tier1 | [delete-nodepool.md](delete-nodepool.md#test-title-patch-to-soft-deleted-nodepool-returns-409-conflict) | DELETE API behavior | | 6 | Cluster update via PATCH triggers reconciliation and reaches Reconciled | Cluster | Positive | Tier0 | [update-cluster.md](update-cluster.md#test-title-cluster-update-via-patch-triggers-reconciliation-and-reaches-reconciled) | UPDATE happy path | | 7 | Nodepool update via PATCH triggers reconciliation and reaches Reconciled | Nodepool | Positive | Tier0 | [update-nodepool.md](update-nodepool.md#test-title-nodepool-update-via-patch-triggers-reconciliation-and-reaches-reconciled) | UPDATE happy path | | 8 | Soft-deleted cluster remains visible via GET and LIST | Cluster | Positive | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-soft-deleted-cluster-remains-visible-via-get-and-list) | DELETE API behavior | @@ -22,44 +33,47 @@ Consolidated test matrix for HYPERFLEET-859. Covers positive, negative, and edge | 14 | DELETE non-existent nodepool returns 404 | Nodepool | Negative | Tier1 | [delete-nodepool.md](delete-nodepool.md#test-title-delete-non-existent-nodepool-returns-404) | DELETE edge cases | | 15 | Adapter statuses transition during update reconciliation | Cluster | Positive | Tier1 | [update-cluster.md](update-cluster.md#test-title-adapter-statuses-transition-during-update-reconciliation) | UPDATE happy path | | 16 | Multiple rapid updates coalesce to latest generation | Cluster | Positive | Tier1 | [update-cluster.md](update-cluster.md#test-title-multiple-rapid-updates-coalesce-to-latest-generation) | UPDATE edge cases | -| 17 | Stuck deletion -- adapter unable to finalize prevents hard-delete | Cluster | Negative | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-stuck-deletion----adapter-unable-to-finalize-prevents-hard-delete) | DELETE error cases | -| 18 | DELETE during initial creation before cluster reaches Ready | Cluster | Positive | Tier2 | [delete-cluster.md](delete-cluster.md#test-title-delete-during-initial-creation-before-cluster-reaches-ready) | DELETE edge cases | -| 19 | PATCH with invalid payload is rejected without changing cluster state | Cluster | Negative | Tier1 | [update-cluster.md](update-cluster.md#test-title-patch-with-invalid-payload-is-rejected-without-changing-cluster-state) | UPDATE negative | -| 20 | PATCH with invalid payload is rejected without changing nodepool state | Nodepool | Negative | Tier1 | [update-nodepool.md](update-nodepool.md#test-title-patch-with-invalid-payload-is-rejected-without-changing-nodepool-state) | UPDATE negative | -| 21 | Simultaneous DELETE requests produce a single tombstone | Cluster | Positive | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-simultaneous-delete-requests-produce-a-single-tombstone) | DELETE edge cases | -| 22 | Adapter treats externally-deleted K8s resources as finalized | Cluster | Positive | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-adapter-treats-externally-deleted-k8s-resources-as-finalized) | DELETE edge cases | -| 23 | DELETE during update reconciliation before adapters converge | Cluster | Positive | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-delete-during-update-reconciliation-before-adapters-converge) | DELETE edge cases | -| 24 | Recreate cluster with same name after hard-delete | Cluster | Positive | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-recreate-cluster-with-same-name-after-hard-delete) | DELETE edge cases | +| 17 | Stuck deletion -- adapter unable to finalize prevents hard-delete | Cluster | Negative | Tier2 | [delete-cluster.md](delete-cluster.md#test-title-stuck-deletion----adapter-unable-to-finalize-prevents-hard-delete) | DELETE error cases | +| 18 | DELETE during initial creation before cluster reaches Reconciled | Cluster | Positive | Tier2 | [delete-cluster.md](delete-cluster.md#test-title-delete-during-initial-creation-before-cluster-reaches-reconciled) | DELETE edge cases | +| 19 | Simultaneous DELETE requests produce a single soft-delete record | Cluster | Positive | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-simultaneous-delete-requests-produce-a-single-soft-delete-record) | DELETE edge cases | +| 20 | Adapter treats externally-deleted K8s resources as finalized | Cluster | Positive | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-adapter-treats-externally-deleted-k8s-resources-as-finalized) | DELETE edge cases | +| 21 | DELETE during update reconciliation before adapters converge | Cluster | Positive | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-delete-during-update-reconciliation-before-adapters-converge) | DELETE edge cases | +| 22 | Recreate cluster with same name after hard-delete | Cluster | Positive | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-recreate-cluster-with-same-name-after-hard-delete) | DELETE edge cases | +| 23 | Labels-only PATCH bumps generation and triggers reconciliation (cluster) | Cluster | Positive | Tier1 | [update-cluster.md](update-cluster.md#test-title-labels-only-patch-bumps-generation-and-triggers-reconciliation) | UPDATE edge cases | +| 24 | Labels-only PATCH bumps generation and triggers reconciliation (nodepool) | Nodepool | Positive | Tier1 | [update-nodepool.md](update-nodepool.md#test-title-labels-only-patch-bumps-generation-and-triggers-reconciliation) | UPDATE edge cases | +| 25 | LIST returns soft-deleted clusters alongside active clusters | Cluster | Positive | Tier1 | [delete-cluster.md](delete-cluster.md#test-title-list-returns-soft-deleted-clusters-alongside-active-clusters) | DELETE API behavior | +| 26 | Cascade DELETE on cluster while a child nodepool is already deleting | Cluster + Nodepool | Positive | Tier2 | [delete-cluster.md](delete-cluster.md#test-title-cascade-delete-on-cluster-while-a-child-nodepool-is-already-deleting) | DELETE hierarchical | +| 27 | Cascade DELETE on cluster while child nodepool is mid-update-reconciliation | Cluster + Nodepool | Positive | Tier2 | [delete-cluster.md](delete-cluster.md#test-title-cascade-delete-on-cluster-while-child-nodepool-is-mid-update-reconciliation) | DELETE hierarchical | +| 28 | Soft-deleted nodepool remains visible via GET and LIST | Nodepool | Positive | Tier1 | [delete-nodepool.md](delete-nodepool.md#test-title-soft-deleted-nodepool-remains-visible-via-get-and-list) | DELETE API behavior | +| 29 | No-op PATCH does not increment generation | Cluster | Positive | Tier1 | [update-cluster.md](update-cluster.md#test-title-no-op-patch-does-not-increment-generation) | UPDATE edge cases | ## Summary | Category | Tier0 | Tier1 | Tier2 | Total | |----------|-------|-------|-------|-------| -| Positive | 5 | 10 | 1 | 16 | -| Negative | 2 | 6 | 0 | 8 | -| **Total** | **7** | **16** | **1** | **24** | +| Positive | 5 | 15 | 3 | 23 | +| Negative | 0 | 5 | 1 | 6 | +| **Total** | **5** | **20** | **4** | **29** | ## Coverage by Ticket Area | Ticket Area | Test Cases | Status | |-------------|-----------|--------| -| DELETE happy path (tombstone -> Finalized -> Reconciled -> hard-delete) | #1, #3 | Covered | -| DELETE hierarchical (subresource cleanup before parent hard-delete) | #2, #12 | Covered | -| DELETE edge cases (idempotent re-DELETE, concurrent DELETEs, non-existent resource, stale pre-tombstone state, NotFound-as-success, DELETE during update, name reuse after hard-delete) | #9, #11, #13, #14, #18, #21, #22, #23, #24 | Covered | +| DELETE happy path (soft-delete -> Finalized -> Reconciled -> hard-delete) | #1, #3 | Covered | +| DELETE hierarchical (subresource cleanup before parent hard-delete) | #2, #12, #26, #27 | Covered | +| DELETE edge cases (idempotent re-DELETE, concurrent DELETEs, non-existent resource, stale pre-deletion state, NotFound-as-success, DELETE during update, name reuse after hard-delete) | #9, #11, #13, #14, #18, #19, #20, #21, #22 | Covered | | DELETE error cases (stuck adapter, unable to finalize) | #17 | Covered | -| DELETE API behavior (409 on mutations, GET/LIST still allowed) | #4, #5, #8, #10 | Covered | +| DELETE API behavior (409 on mutations, GET/LIST still allowed) | #4, #5, #8, #10, #25, #28 | Covered | | UPDATE happy path (PATCH -> generation -> reconciliation -> Reconciled) | #6, #7, #15 | Covered | -| UPDATE edge cases (rapid updates, coalescing) | #16 | Covered | -| UPDATE negative (invalid/malformed PATCH payloads) | #19, #20 | Covered | +| UPDATE edge cases (rapid updates, coalescing, labels-only PATCH, no-op PATCH) | #16, #23, #24, #29 | Covered | +| UPDATE negative (E2E-scoped) | — | Not applicable in E2E scope (API payload validation belongs in integration tests) | ## Deferred / Not Applicable -Items from HYPERFLEET-859 scope that are not covered as standalone test cases, with rationale: +Items considered for this matrix but deliberately not covered as standalone test cases: | Item | Status | Rationale | |------|--------|-----------| | RBAC denied on DELETE | N/A | No RBAC implementation exists in the API. Authentication is bearer-token only with no role/permission model. Revisit when RBAC is added. | -| Sentinel: events published using Reconciled (not Ready) | Implicitly covered | Every update and delete test case relies on Sentinel publishing events based on Reconciled condition. If Sentinel used Ready instead, adapters would not reconcile to new generations and tests #1, #3, #6, #7, #15, #16 would fail. No standalone test case needed. | -| Adapter: when_deleting mode switch | Implicitly covered | Every delete test case (#1, #2, #3, #17, #18) exercises the adapter's when_deleting mode. If adapters did not switch to deletion mode, they would apply spec instead of finalizing, and Finalized=True would never be reported. | -| Adapter: delete_options.when ordering | Implicitly covered | The delete happy path tests (#1, #3) validate that adapters process deletion in the correct order by confirming all adapters reach Finalized=True. Incorrect ordering would result in stuck or failed finalization. | -| Adapter: propagationPolicy passed to K8s API | Not covered | Internal adapter behavior. Could be validated by inspecting K8s resource state after deletion (e.g., verifying child resources are cleaned up according to the expected propagation policy). Deferred — consider adding when adapter delete_options configuration is finalized. | +| Concurrent PATCH + DELETE race condition | Deferred | Non-deterministic test — both outcomes (PATCH-first or DELETE-first) are acceptable. Hard to assert on reliably in E2E. Revisit if the team adds a deterministic ordering guarantee. | +| PATCH payload validation errors (malformed JSON, schema/type violations) | Out of E2E scope | API-boundary validation happens before lifecycle business logic; cover in API integration tests rather than cross-component E2E. | diff --git a/test-design/testcases/update-nodepool.md b/test-design/testcases/update-nodepool.md index c1d59c5..94c9fc3 100644 --- a/test-design/testcases/update-nodepool.md +++ b/test-design/testcases/update-nodepool.md @@ -3,7 +3,7 @@ ## Table of Contents 1. [Nodepool update via PATCH triggers reconciliation and reaches Reconciled](#test-title-nodepool-update-via-patch-triggers-reconciliation-and-reaches-reconciled) -2. [PATCH with invalid payload is rejected without changing nodepool state](#test-title-patch-with-invalid-payload-is-rejected-without-changing-nodepool-state) +2. [Labels-only PATCH bumps generation and triggers reconciliation](#test-title-labels-only-patch-bumps-generation-and-triggers-reconciliation) --- @@ -11,7 +11,7 @@ ### Description -This test validates the nodepool update lifecycle. It verifies that when a PATCH request modifies a nodepool's spec, the nodepool's `generation` is incremented independently of the parent cluster, nodepool adapters reconcile to the new generation, and the nodepool reaches `Reconciled=True`. The parent cluster must remain unaffected. +This test validates the nodepool update lifecycle. It verifies that when a PATCH request modifies a nodepool's spec, the nodepool's `generation` is incremented independently of the parent cluster, nodepool adapters reconcile to the new generation, and the nodepool reaches `Reconciled=True` and `Available=True`. The parent cluster must remain unaffected. --- @@ -23,7 +23,7 @@ This test validates the nodepool update lifecycle. It verifies that when a PATCH | **Automation** | Not Automated | | **Version** | Post-MVP | | **Created** | 2026-04-15 | -| **Updated** | 2026-04-15 | +| **Updated** | 2026-04-28 | --- @@ -32,14 +32,12 @@ This test validates the nodepool update lifecycle. It verifies that when a PATCH 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. PATCH endpoint is deployed and operational -5. Reconciled status aggregation is implemented --- ### Test Steps -#### Step 1: Create a cluster and nodepool, wait for Ready state +#### Step 1: Create a cluster and nodepool, wait for Reconciled and Available state **Action:** - Create a cluster and nodepool: @@ -53,11 +51,14 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools \ -H "Content-Type: application/json" \ -d @testdata/payloads/nodepools/nodepool-request.json ``` -- Wait for both to reach Ready state +- Wait for both to reach Reconciled state **Expected Result:** -- Cluster and nodepool reach `Ready` condition `status: "True"` -- Both at `generation: 1`, `Reconciled: True` +- Nodepool `generation` equals 1 +- Nodepool `Reconciled` condition `status: "True"` with `observed_generation: 1` +- Nodepool `Available` condition `status: "True"` with `observed_generation: 1` +- All required nodepool adapters report `observed_generation: 1` +- **Per-adapter conditions on nodepool status**: each required adapter condition on the nodepool resource has `status: "True"` #### Step 2: Send PATCH request to update the nodepool spec @@ -84,7 +85,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepo - All nodepool adapters report `observed_generation: 2` - Each adapter has `Applied: True`, `Available: True`, `Health: True` -#### Step 4: Verify nodepool reaches Reconciled=True at new generation +#### Step 4: Verify nodepool reaches Reconciled=True and Available=True at new generation **Action:** ```bash @@ -93,7 +94,11 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepo **Expected Result:** - Nodepool `Reconciled` condition `status: "True"` with `observed_generation: 2` -- Nodepool `Ready` condition `status: "True"` +- Nodepool `Available` condition `status: "True"` with `observed_generation: 2` +- Nodepool `id` is unchanged from Step 1 +- Nodepool `cluster_id` is unchanged from Step 1 +- `generation` equals 2 +- **Per-adapter conditions on nodepool status**: each required adapter condition on the nodepool resource has `status: "True"` #### Step 5: Verify parent cluster is unaffected @@ -105,7 +110,7 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} **Expected Result:** - Cluster `generation` remains at 1 (unchanged) - Cluster `Reconciled` condition `status: "True"` with `observed_generation: 1` -- Cluster `Ready` condition `status: "True"` +- Cluster `Available` condition `status: "True"` with `observed_generation: 1` #### Step 6: Cleanup resources @@ -117,25 +122,24 @@ curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} **Expected Result:** - Cluster, nodepool, and all associated resources are cleaned up ---- -## Test Title: PATCH with invalid payload is rejected without changing nodepool state +## Test Title: Labels-only PATCH bumps generation and triggers reconciliation ### Description -This test validates that the API rejects malformed or constraint-violating PATCH requests on a nodepool with an HTTP 4xx response, does not increment `generation`, and does not trigger reconciliation. It complements the happy-path update test by covering the negative API contract for nodepool spec mutations. +This test validates that a PATCH request that only modifies a nodepool's `labels` (without changing `spec`) increments the nodepool's `generation` and triggers adapter reconciliation. Generation is incremented when either `spec` or `labels` change. This mirrors the cluster-level labels PATCH behavior. The parent cluster must remain unaffected. --- | **Field** | **Value** | |-----------|-----------| -| **Pos/Neg** | Negative | +| **Pos/Neg** | Positive | | **Priority** | Tier1 | | **Status** | Draft | | **Automation** | Not Automated | | **Version** | Post-MVP | -| **Created** | 2026-04-16 | -| **Updated** | 2026-04-16 | +| **Created** | 2026-04-17 | +| **Updated** | 2026-04-20 | --- @@ -144,15 +148,15 @@ This test validates that the API rejects malformed or constraint-violating PATCH 1. Environment is prepared using [hyperfleet-infra](https://github.com/openshift-hyperfleet/hyperfleet-infra) with all required platform resources 2. HyperFleet API and HyperFleet Sentinel services are deployed and running successfully 3. The adapters defined in testdata/adapter-configs are all deployed successfully -4. PATCH endpoint is deployed and operational with request validation enabled --- ### Test Steps -#### Step 1: Create a cluster and nodepool, wait for Ready state +#### Step 1: Create a cluster and nodepool, wait for Reconciled state **Action:** +- Create a cluster and nodepool: ```bash curl -X POST ${API_URL}/api/hyperfleet/v1/clusters \ -H "Content-Type: application/json" \ @@ -163,69 +167,52 @@ curl -X POST ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools \ -H "Content-Type: application/json" \ -d @testdata/payloads/nodepools/nodepool-request.json ``` -- Wait for both to reach Ready state +- Wait for both to reach Reconciled state **Expected Result:** -- Nodepool at `generation: 1`, `Reconciled: True` with `observed_generation: 1` +- Cluster and nodepool reach `Reconciled` condition `status: "True"` +- Both at `generation: 1`, `Reconciled: True` -#### Step 2: Capture baseline state and `last_report_time` per adapter +#### Step 2: Send labels-only PATCH request to the nodepool **Action:** ```bash -curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} \ - | jq '{id, cluster_id, generation, conditions, labels, spec}' -curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id}/statuses \ - | jq '[.items[] | {adapter, observed_generation, last_report_time}]' +curl -X PATCH ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} \ + -H "Content-Type: application/json" \ + -d '{"labels": {"env": "staging", "pool-type": "gpu"}}' ``` **Expected Result:** -- Baseline captured for comparison in Step 4 - -#### Step 3: Attempt PATCH with invalid payloads - -For each of the following cases, submit a PATCH and verify it is rejected without any state change: - -**Case A: Malformed JSON** -```bash -curl -i -X PATCH ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} \ - -H "Content-Type: application/json" \ - -d '{"replicas": 3' -``` +- Response returns HTTP 200 (OK) +- `generation` incremented from 1 to 2 +- Labels in the response include the new values (`env: staging`, `pool-type: gpu`) +- `spec` is unchanged from Step 1 -**Case B: Wrong type on a typed spec field (e.g., `replicas` as a string)** -```bash -curl -i -X PATCH ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} \ - -H "Content-Type: application/json" \ - -d '{"replicas": "not-a-number"}' -``` +#### Step 3: Verify nodepool adapters reconcile to the new generation -**Case C: Attempt to mutate an immutable/server-controlled field (e.g., `id`, `generation`, `cluster_id`)** +**Action:** +- Poll nodepool adapter statuses until all report the new generation: ```bash -curl -i -X PATCH ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} \ - -H "Content-Type: application/json" \ - -d '{"id": "nodepool-other", "generation": 99, "cluster_id": "cluster-other"}' +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id}/statuses ``` -**Expected Result (for every case):** -- Response is HTTP 400 (Bad Request) or 422 (Unprocessable Entity) — no 5xx -- Response body contains a structured validation error message identifying the offending field -- For Case C, the server either rejects the request or silently ignores the read-only fields (both are acceptable, but read-only fields must remain unchanged) +**Expected Result:** +- All nodepool adapters report `observed_generation: 2` +- Each adapter has `Applied: True`, `Available: True`, `Health: True` -#### Step 4: Verify nodepool state is unchanged +#### Step 4: Verify nodepool reaches Reconciled=True and Available=True at new generation **Action:** ```bash -curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} \ - | jq '{id, cluster_id, generation, conditions, labels, spec}' -curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id}/statuses \ - | jq '[.items[] | {adapter, observed_generation, last_report_time}]' +curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id}/nodepools/{nodepool_id} ``` **Expected Result:** -- `generation` is still 1 (no invalid PATCH incremented it) -- `id` and `cluster_id` are unchanged from baseline -- Nodepool `Reconciled` condition remains `status: "True"` with `observed_generation: 1` -- All adapter `last_report_time` values are unchanged vs baseline (no spurious reconciliation was triggered) +- Nodepool `generation` equals 2 +- Nodepool `Reconciled` condition `status: "True"` with `observed_generation: 2` +- Nodepool `Available` condition `status: "True"` with `observed_generation: 2` +- Labels reflect the PATCH update +- `spec` is unchanged #### Step 5: Verify parent cluster is unaffected @@ -235,8 +222,8 @@ curl -X GET ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} ``` **Expected Result:** -- Cluster `generation` remains at 1 -- Cluster `Reconciled: True` with `observed_generation: 1` +- Cluster `generation` remains at 1 (unchanged) +- Cluster `Reconciled` condition `status: "True"` with `observed_generation: 1` #### Step 6: Cleanup resources @@ -248,4 +235,3 @@ curl -X DELETE ${API_URL}/api/hyperfleet/v1/clusters/{cluster_id} **Expected Result:** - Cluster, nodepool, and all associated resources are cleaned up ----