ontap-nas-economy: controller restart deletes all rules from the backend export policy, revoking NFS access for every node on the backend

[notsrch]

Describe the bug

During Trident controller startup (and backend re-initialization generally), export-policy
reconciliation for ontap-nas-economy backends with autoExportPolicy: true computed an
empty "desired nodes" set and deleted every rule from the backend-wide export policy
trident-<backendUUID>.

This policy is assigned to every FlexVol the driver manages (publishQtreeShare sets it on
the parent FlexVol on each publish). ONTAP evaluates the export policy of each junction on
the mount path, so an empty FlexVol-level policy denies new NFS mounts for every node, for
every volume on the backend, even though the per-qtree policies still contain the correct
rules. Established mounts survive; every new mount fails with
mount.nfs: access denied / exit status 32.

Because export-policy rules for a node are only ever re-added during that node's own
ControllerPublishVolume (see companion report on the missing repair path), and Kubernetes
does not re-issue ControllerPublish while a VolumeAttachment exists, access came back one
node at a time — only for nodes whose VolumeAttachments happened to be deleted and recreated.
In our incident this produced a multi-hour outage: 5 of 6 workers could not mount two shared
RWX volumes, and recovery required deleting pods/VolumeAttachments per node to force fresh
publishes.

Root cause in code (verified on tag v26.06.0; unchanged on master as of 2026-08-31):

1. core/orchestrator_core.go — reconcileNodeAccessOnBackend: because the economy driver
   implements PublishEnforceable, the node list passed to the driver is
   publishedNodesForBackend(b), not the registered-node list.
2. publishedNodesForBackend intersects three caches: the volume-publication cache, the
   backend object's own volume map (b.Volumes()), and the node cache. During controller
   startup and during backend replacement (addBackend → updateBackend builds a fresh
   backend object), this intersection is transiently empty.
3. The empty result flows into reconcileNASNodeAccess → reconcileExportPolicyRules
   (storage_drivers/ontap/ontap_common.go), which faithfully deletes every "undesired"
   rule (up to 10 per pass) from the live, in-use policy.

There is no guard anywhere on this path: no minimum-view check, no "never reconcile to an
empty rule set while publications exist for this backend", and no suppression of
reconciliation while a backend object is being replaced.

The same trust-the-cache pattern exists a second time in removeExportPolicyRules
(ontap_common.go), which is called on every unpublish of a qtree-level policy: if
publishInfo.Nodes arrives empty, it takes an explicit branch — "No active nodes remaining,
removing ALL export policy rules" — and empties the policy. We did not observe this branch
misfire in the incident, but it is reachable under the same transient-cache conditions on a
shared RWX volume, and it would revoke all remaining nodes at the qtree level. We suggest
fixing both call sites as one pattern. (#1086 was a previous incident of unpublish-side
collateral revocation in this same area.)

Environment

• Trident version: 26.06.0
• Trident installation flags used: operator-based install, defaults
• Container runtime: containerd (RKE2)
• Kubernetes version: v1.34.2
• Kubernetes orchestrator: RKE2 (Rancher-managed)
• Kubernetes enabled feature gates: defaults
• OS: Linux (RKE2 nodes)
• NetApp backend types: ONTAP, ontap-nas-economy driver, REST API, autoExportPolicy: true, autoExportCIDRs: ["10.63.0.0/21"]
• Other: 6 worker nodes; two shared RWX PVCs (tool caches for CI runners) mounted by all workers; ~30 short-lived RWO PVCs churning continuously

To Reproduce

Timing-dependent; observed in production during a controller pod rescheduling. Sequence:

1. ontap-nas-economy backend, autoExportPolicy: true, multiple nodes holding
   publications for RWX volumes, steady publish/unpublish churn from short-lived pods.
2. Drain the node running the Trident controller so the controller pod is rescheduled
   (in our case the drain also queued unpublish/publish work for the drained node).
3. On startup, the new controller re-initializes each backend from its
   TridentBackendConfig (crdControllerEvent=add reconciles) while the queued CSI calls
   drain.
4. Observe the backend export policy: all rules deleted.

Evidence from our incident (2026-08-31, times UTC). We captured an HTTP-level trace of
Trident's ONTAP REST traffic covering the whole window, plus the controller log:

• 19:05:41 — first ONTAP REST call from the restarted controller (rescheduled to a new node).
• 19:05:43.586–.762 — DELETE /api/protocols/nfs/export-policies/21474840540/rules/{2,3,4,5,6,1},
  six calls ~30 ms apart: every worker rule removed from the first backend's policy
  (trident-ce7d4ac1-…). Immediately followed by an attempted delete of the policy itself →
  400, ONTAP error 1703945 "Ruleset is in use by a volume" (see companion report on
  Terminate).
• 19:05:52.271–.448 — identical six-rule wipe + failed policy delete on the second backend's
  policy (trident-e0821f5f-…, id 21474855140).
• Controller log, same seconds: three warnings
  error deleting export policy: … [400] … "Ruleset is in use by a volume" tagged
  crdControllerEvent=add, workflow="cr=reconcile".
• First queued CSI unpublish was processed at 19:05:47 — after the first wipe, so the wipe
  was not unpublish-driven.
• tridentvolumepublications listed all 6 workers for the shared volumes throughout;
  tridentnodes IPs were correct. Trident's recorded intent was right; only the ONTAP policy
  was emptied.
• 19:11:29–:30 — the drained node's first fresh publish re-added its (and only its) rule;
  it became the only node able to mount. FailedMount exit status 32 on the other five
  workers from 19:12 onward.
• 20:02:38–:53 — after we deleted stuck pods (freeing VolumeAttachments), four fresh
  publishes re-added four more rules; those nodes recovered within seconds. One node whose
  replacement pods re-pinned the old VolumeAttachments stayed locked out until we intervened
  on the array.

Expected behavior

Reconciliation must never treat a transiently empty or partial cache view as authority to
delete rules from an in-use export policy. Concretely, some combination of:

• Skip (or defer) node-access reconciliation until the publication, volume, and node caches
  are fully populated after startup or backend replacement.
• Refuse to reconcile a backend policy down to zero rules while any publication exists for a
  volume on that backend; log and retry instead.
• Apply the same guard to removeExportPolicyRules' remove-all branch.

Additional context

• The blast radius is total because one policy is shared by every FlexVol on the backend;
  related discussion about FlexVol-level policies for this driver: With the ontap-nas-economy driver, use the same export policy for the trident_qtree_pool volume as with the qtrees #1149, Feature request: improve export policies at the FlexVol level in the context of ONTAP-NAS-ECO #709.
• Possibly related: failed to update export-policies with version 25.10.0 when concurrency enabled #1093 (export-policy update failures at controller scale-up with
  concurrency enabled).
• Companion reports filed together: driver Terminate destroys this same live policy on
  every backend update (ontap-nas-economy: driver Terminate attempts to delete the shared backend export policy while live FlexVols still use it — fires on every TridentBackendConfig update #1180); no repair path exists once a rule is missing
  (autoExportPolicy: a node with an existing publication is locked out permanently when its export rule is missing or stale — no code path ever repairs export policies #1181) — which is what turned this transient wipe into a multi-hour outage.

AI assistance

Assisted-by: Claude Fable 5 noreply@anthropic.com

[vivakeram]

@notsrch Did you hit this issue with concurrency turned off in trident ?

[notsrch]

Concurrency is enabled. We did not test with it disabled. This problem was seen on a cluster hosting Github ARC runners and it has a pretty high PV churn rate.

[notsrch]

A second production occurrence on a different cluster, with a different trigger, and with
dedicated (non-shared) volumes. Adding it because it widens the trigger list beyond "controller
restart" and shows the wipe is not tied to shared RWX volumes.

Environment. Trident 26.06.0 stock, operator install, enableConcurrency: true. Two
ontap-nas-economy backends in use (REST, autoExportPolicy: true), ten volumes, all
ReadWriteOnce and each used by exactly one StatefulSet pod. Kubernetes v1.34, five worker nodes.

Trigger: an update to the backend credentials secret. The Secret referenced by every
TridentBackendConfig was re-applied by our secrets tooling. Nothing else changed; every tbc is
still at generation: 1. Trident's secret handler reacts to any update of a referenced Secret
and rebuilds every backend that uses it:

02:00:11 level=info msg="Running update on backend due to secret update." backendConfig.Name=<backend-1> crdControllerEvent=update
02:00:11 level=info msg="Running update on backend due to secret update." backendConfig.Name=<backend-2> crdControllerEvent=update
... (one line per backend, ten in total)
02:00:12 level=info msg="Created new storage backend." backend="&{... <backend-1> ...}" crdControllerEvent=update
02:00:12 level=info msg="Created new storage backend." backend="&{... <backend-2> ...}" crdControllerEvent=update
...

That is the same UpdateBackendByBackendUUID -> upsertBackend path described above, so each
ontap-nas-economy backend was reconciled against an empty volume map and every rule was
removed from its trident-<backendUUID> policy. Six economy backends, all wiped in three
seconds. No symptom for the next 37 hours because established mounts are unaffected.

One difference from the earlier report: no 1703945 "Ruleset is in use by a volume" warning
was logged for the replaced instances' policy delete this time. It does not change the outcome,
but it means the absence of that warning is not a sign the wipe did not happen.

Symptom, 37 hours later: a rolling restart of a StatefulSet. Three replicas, one dedicated
RWO volume each, OrderedReady. Each replacement pod was scheduled back onto its previous node,
so Kubernetes reused the existing VolumeAttachment and issued no ControllerPublishVolume.
NodePublishVolume then mounted against a FlexVol policy with no rule for the node:

15:40:23 level=error msg="GRPC error: rpc error: code = Internal desc = error mounting NFS volume <data-lif>:/<flexvol>/<qtree> on mountpoint ...: exit status 32"

The controller log has no publish or unpublish for that volume between the wipe and the fix, and
tridentvolumepublications and tridentnodes were correct throughout. The first pod sat failing
for 2 h 45 min (109 kubelet retries); the next two failed the same way as soon as the rollout
reached them.

Recovery. Deleting the pod's VolumeAttachment (then the pod) forced an unpublish and a fresh
publish, which re-added the node's rule; the mount succeeded within seconds:

18:27:47 level=info msg="Unpublishing volume from node." node=<node-a> volume=<pv>
18:27:54 level=info msg="Publishing volume to node."     node=<node-a> volume=<pv>

Repeated per pod. As noted in #1181, that fresh publish is the only thing that writes the rule
back, so every other volume on the affected backends remains exposed until its node happens to
republish. On this cluster one of the two backends has had no publish since the wipe and is
presumably still empty.

Takeaways for this issue

• The trigger set is at least: controller restart, TridentBackendConfig spec change, and any
  update to the referenced credentials Secret. Credential rotation is routine and unattended,
  which makes it the most likely way clusters hit this in practice.
• Shared volumes are not required. The policy that denies the mount is per backend, so a
  dedicated RWO volume fails identically if its pod is rescheduled onto a node that reuses the
  VolumeAttachment.
• Because rules only return through a fresh publish, the blast radius after a wipe is "every
  volume on the backend whose pod restarts in place", and it persists until each one does.

AI assistance

Assisted-by: Claude Fable 5 noreply@anthropic.com