NodeUnpublishVolume fails forever when the mount point returns EACCES

[notsrch]

Describe the bug

NodeUnpublishVolume checks whether the target path is a directory before it unmounts:

• v26.06.0: frontend/csi/node_server.go, NodeUnpublishVolume, p.osutils.IsLikelyDir(targetPath).
• master: core/node/unmount.go, unmountGeneric, c.osutils.IsLikelyDir(targetPath).

IsLikelyDir is os.Stat. If the error is not-found the call returns success; any other error
is returned as codes.Internal:

could not check if the target path (%s) is a directory; %v

For an NFS mount whose export rule no longer admits the node, stat on the mount point itself
returns EACCES. So the unpublish fails before it ever tries umount, kubelet retries it
indefinitely with the same result, the pod never finishes terminating, the node keeps the
volume in status.volumesInUse, and the VolumeAttachment is never detached. Force-deleting the
pod removes the API object but kubelet keeps retrying the same failing unpublish.

Observed log line (Trident 26.06.0 node plugin, repeated on every kubelet retry):

level=error msg="GRPC error: rpc error: code = Internal desc = could not check if the target path
(/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pv>/mount) is a directory;
stat /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pv>/mount: permission denied"
logLayer=csi_frontend requestSource=CSI

Nothing about a revoked export should prevent umount(2); the server is reachable and the
mount is not stale. The call never gets that far: the failure is the pre-check alone. (Not
verified by running umount by hand while access was denied; verified in that the same
unpublish succeeded on its next retry the moment the rule was re-added.)

Why it matters

With autoExportPolicy: true, a missing per-qtree rule is never repaired by Trident (#1181);
the only Trident-side repair is a fresh ControllerPublish, which Kubernetes issues only after
the VolumeAttachment is gone. This bug makes that impossible for exactly the node that needs
it: the missing rule denies I/O, the denied I/O makes the unpublish fail, the failing unpublish
pins the VolumeAttachment, and the pinned VolumeAttachment prevents the publish that would
restore the rule. The only exit is editing the export policy on the array.

Environment

• Trident version: 26.06.0 (code path unchanged on master, now in core/node/unmount.go)
• Trident installation flags used: operator-based install, defaults, enableConcurrency: true
• Container runtime: containerd (RKE2)
• Kubernetes version: v1.35.4
• Kubernetes orchestrator: RKE2
• Kubernetes enabled feature gates: defaults
• OS: Linux
• NetApp backend types: ONTAP 9.17.1P1, ontap-nas-economy, REST, autoExportPolicy: true, NFSv3
• Other: the attach/detach controller did not force-detach within 10 minutes of the pod being
  force-deleted, so nothing on the Kubernetes side broke the loop either

To Reproduce

Deterministic:

1. ontap-nas-economy backend with autoExportPolicy: true; an RWX PVC mounted by a pod on
   node A.
2. On the array, delete node A's rule from the volume's qtree export policy
   (DELETE /api/protocols/nfs/export-policies/{id}/rules/{index}). The pod's reads and
   writes now fail with Permission denied.
3. Delete the pod. It stays in Terminating. The Trident node plugin logs the error above on
   every retry; kubectl get volumeattachments still shows the attachment for node A;
   kubectl get node A -o jsonpath='{.status.volumesInUse}' still lists the volume.
4. kubectl delete pod --force --grace-period=0: same result minus the pod object.
5. Re-add the rule on the array: the next retry succeeds, the VolumeAttachment is deleted
   within seconds.

Expected behavior

NodeUnpublishVolume should treat a target path it cannot stat as "possibly mounted" and
proceed to the unmount, or at minimum treat EACCES like a mount that exists: check the
mount table instead of stat, or try umount and only fail if that fails. Unmounting a path
should not require read access to the filesystem behind it.

Additional context

• Found while reproducing autoExportPolicy: a node with an existing publication is locked out permanently when its export rule is missing or stale — no code path ever repairs export policies #1181; that report's recovery advice ("delete the
  VolumeAttachment to force a fresh publish") does not work once this bug is in play.
• Related: ontap-nas-economy: controller restart deletes all rules from the backend export policy, revoking NFS access for every node on the backend #1179 (the rule-loss event that makes this reachable at scale).

AI assistance

Assisted-by: Claude Fable 5 noreply@anthropic.com