autoExportPolicy: a node with an existing publication is locked out permanently when its export rule is missing or stale — no code path ever repairs export policies

[notsrch]

Describe the bug

With autoExportPolicy: true, export-policy rules for a node are written in exactly one
place: that node's own ControllerPublishVolume (publishQtreeShare /
ensureNodeAccessForPolicyAndApply add the publishing node's IPs to the qtree policy and
the FlexVol's backend policy). Nothing else ever adds or corrects a rule:

• Kubernetes does not re-issue ControllerPublish while a VolumeAttachment exists, so a node
  with an intact publication never publishes again — sometimes for months.
• ReconcileVolumeNodeAccess is a no-op for the economy driver
  (storage_drivers/ontap/ontap_nas_qtree.go), so per-qtree policies are never reconciled
  against publication records.
• ReconcileNodeAccess reconciles only the backend policy, and node re-registration with a
  changed IP does not repair the per-qtree policies a node is published to.
• kubelet's mount retries (every ~2 minutes) never reach the Trident control plane.

Consequence: whenever a published node's rule goes missing or stale — node IP change,
out-of-band rule edit, or a Trident-side deletion such as the one in #1179 — that node
fails every new mount of the affected volume forever, while tridentvolumepublications
and tridentnodes remain perfectly correct. The unpublish path
(removeExportPolicyRules) even accelerates the decay: it prunes any rule that matches no
currently-registered node IP, so another node's routine unpublish removes the stale rule of
an unrelated re-IP'd node, with nothing to put the correct one back.

The only recoveries available today are manual: delete the node's VolumeAttachments (forcing
a fresh publish) or edit the export policy on the array.

We hit both variants of this in production on Trident 26.06.0:

1. Node IP change (2026-08-23). A worker re-IP'd. Its rule in two long-lived shared-RWX
   qtree policies kept the old IP (rule index 34, while the policies' active churn was at
   indexes 76–81 — the rule had not been touched since long before), then was pruned as
   stale by other nodes' unpublishes. All new mounts on that worker failed with
   exit status 32 for 8 days. Its publications for the shared volumes persisted the entire
   time, so no fresh publish ever added the new IP.
2. After a rule wipe (2026-08-31, ontap-nas-economy: controller restart deletes all rules from the backend export policy, revoking NFS access for every node on the backend #1179). Five workers with intact publications
   stayed locked out of every volume on the backend for hours. Each node recovered only at
   its next fresh publish, which required us to delete its stuck pods so the
   VolumeAttachments were recreated (rule re-adds observed at 19:11:30, then 20:02:38–:53
   for four nodes as we rolled pods). One node's replacement pods re-pinned the old
   VolumeAttachments within seconds, so no fresh publish could ever fire; it stayed locked
   out until we added its IP on the array.

Environment

• Trident version: 26.06.0 (IP-change variant first manifested while running this version; code unchanged on master as of 2026-08-31)
• Trident installation flags used: operator-based install, defaults
• Container runtime: containerd (RKE2)
• Kubernetes version: v1.34.2
• Kubernetes orchestrator: RKE2 (Rancher-managed)
• Kubernetes enabled feature gates: defaults
• OS: Linux (RKE2 nodes)
• NetApp backend types: ONTAP, ontap-nas-economy driver, REST API, autoExportPolicy: true, autoExportCIDRs: ["10.63.0.0/21"]
• Other: shared RWX volumes mounted by all 6 workers; short-lived pods churn continuously

To Reproduce

1. ontap-nas-economy backend, autoExportPolicy: true.
2. Publish an RWX volume to node A and keep at least one pod using it on node A (so the
   VolumeAttachment and publication persist).
3. Change node A's IP (reboot with a new DHCP lease reproduces our case), and let normal
   pod churn on other nodes trigger publish/unpublish cycles on the same volume.
4. Node A's registered IP in tridentnodes updates correctly, but the export policies keep
   (and eventually lose) only the old IP. Every new mount of the volume on node A now fails
   with mount.nfs: access denied / exit status 32.
5. Observe that no Trident activity ever repairs the rule; deleting node A's
   VolumeAttachment (forcing a fresh publish) fixes it immediately.

Step 3 can be replaced by deleting node A's rule on the array — the outcome is identical,
which is what makes this the severity multiplier for any rule-loss bug.

Expected behavior

Trident's recorded intent (publications + registered node IPs) should converge to ONTAP
state. Some combination of:

• Periodic or event-driven reconciliation of qtree-level and FlexVol-level policies against
  volume publications and current node IPs (e.g., extend the existing periodic node-access
  reconciliation, and implement ReconcileVolumeNodeAccess for this driver).
• Node update (IP change) triggering repair of policies for volumes published to that node.
• ControllerPublishVolume is already idempotent and re-adds rules when invoked — the gap is
  purely that nothing invokes it or an equivalent repair when state drifts.

Additional context

• Same symptom class reported before: NFS auto export policy didn't include all workers, making a restarted pod land on a new worker which couldn't access NFS export #1013 (closed as "resolved by upgrading Trident /
  inactivity" — the mechanism above is still present in 26.06.0 and master, so the class was
  not fixed) and NFS export policy is missing one k8s node's IP #965.
• Companion reports: ontap-nas-economy: controller restart deletes all rules from the backend export policy, revoking NFS access for every node on the backend #1179 (startup wipe of the backend policy — the rule-loss event
  this gap turned into an outage), ontap-nas-economy: driver Terminate attempts to delete the shared backend export policy while live FlexVols still use it — fires on every TridentBackendConfig update #1180 (backend updates attempt to delete the live
  policy).
• Static workaround we are deploying meanwhile: pre-created export policy covering the node
  subnet, exportPolicy set on the backend, autoExportPolicy: false.

AI assistance

Assisted-by: Claude Fable 5 noreply@anthropic.com

[enneitex]

Hi, same issue here. We reported it to NetApp in March but there's still no solution. We are stuck with trident v24. v25.02 introduced the issue.

[notsrch]

Reproduced under controlled conditions, and two things came out of it that the report above
does not say.

Setup. 3-node RKE2 cluster, Kubernetes v1.35.4, Trident 26.06.0, operator install,
enableConcurrency: true. One ontap-nas-economy backend (ONTAP 9.17.1P1, REST,
autoExportPolicy: true) with a single RWX PVC mounted by a DaemonSet, so every node held a
publication and a VolumeAttachment. Before: qtree policy and backend policy each had one rule
per node. Node IPs below are placeholders.

Trigger. The "delete node A's rule on the array" variant from To Reproduce:
DELETE /api/protocols/nfs/export-policies/{qtree-policy-id}/rules/{index} for node-b's rule
in the qtree policy. Backend policy left intact.

What Trident did about it

Attempt	Qtree rule restored?
Periodic node-access reconcile, more than 3 minutes (six 30 s ticks)	no
Fresh ControllerPublish of a different volume on node-b, same backend	no
Node re-registration (Trident node pod on node-b restarted; controller logged "Added a new node", which invalidates node access on all backends)	no (backend policy reconciled, qtree policy untouched)
Fresh ControllerPublish of this volume on node-b, after its VolumeAttachment was released	yes, immediately

Throughout, tridentvolumepublications listed all three nodes and tridentnodes had the
correct IPs. A new pod on node-b, whose VolumeAttachment already existed, sat in
ContainerCreating with error mounting NFS volume ... exit status 32.

So the claim holds as written: rules are written only by that node's own ControllerPublish
for that volume, and nothing else converges ONTAP state to Trident's records. Node
re-registration was the one path the report inferred from code rather than observed; it is
now observed.

Two things the report understates

1. The established mount breaks too, not only new mounts. NFSv3 evaluates the export
   policy per request. The DaemonSet pod that had the volume mounted since before the deletion
   got Permission denied on read and write immediately. (The ontap-nas-economy: controller restart deletes all rules from the backend export policy, revoking NFS access for every node on the backend #1179 incident did not show
   this because only the FlexVol-level policy was emptied there and the qtree policies kept
   their rules; ONTAP checks the FlexVol policy at mount time and the qtree policy on I/O.)
   The IP-change case in the report almost certainly had the same effect on that worker.

2. The lockout blocks the recovery the report recommends. "Delete the node's
   VolumeAttachments (forcing a fresh publish)" did not work here. NodeUnpublishVolume
   stats the target path before unmounting and returns codes.Internal on anything other than
   not-found (frontend/csi/node_server.go on v26.06.0; core/node/unmount.go,
   unmountGeneric, on master). With access denied, that stat fails with EACCES:

   GRPC error: rpc error: code = Internal desc = could not check if the target path
   (/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pv>/mount) is a directory;
   stat /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pv>/mount: permission denied
   

   kubelet retried it forever, the pod never finished terminating, the node kept the volume
   in volumesInUse, and the VolumeAttachment stayed. Force-deleting the pod removed the pod
   object but changed nothing else; the attach/detach controller did not force-detach within
   10 minutes on Kubernetes v1.35.4. The only way out was re-adding the rule on the array,
   after which the unpublish succeeded on its next retry and the VolumeAttachment was gone 31
   seconds later. ControllerUnpublish then pruned the manually added rule again, and the
   fresh publish on re-admission wrote it back.

   That is a separate defect (an unpublish that cannot stat its own mount point should still
   attempt the unmount), filed as NodeUnpublishVolume fails forever when the mount point returns EACCES #1184 . It matters here because it removes
   the last Kubernetes-side recovery: once a qtree rule is missing for a node that has the
   volume mounted, the only fix is on the array.

Suggested edit to Expected behavior: the recovery list should read "the only recovery
available today is to edit the export policy on the array; deleting the VolumeAttachment
only works while the node can still unmount, which it cannot once the qtree rule is gone."

AI assistance

Assisted-by: Claude Fable 5 noreply@anthropic.com