restrict-system-components blocks LINSTOR satellite cross-node traffic when Cilium is chained with kube-ovn

[2mind]

What happened

LINSTOR controller pod on node A cannot reach satellite on node B (port 3367 SSL). Only the satellite on the same node as the controller is Online. Cross-node pod-to-host TCP connections to port 3367 are silently dropped.

Root cause

restrict-system-components CCNP has:

ingressDeny:
- fromEntities:
  - world
  toPorts:
  - ports:
    - port: "3367"   # ← blocks LINSTOR satellite

In CNI chaining mode (generic-veth with kube-ovn), cross-node pod-to-host traffic arrives at the destination node with the pod overlay IP as source. Cilium ipcache does not have identity entries for pods managed by kube-ovn (they are not Cilium endpoints), so the traffic is classified as identity world→unknown and dropped.

Note: ingressDeny has priority over ingress in Cilium policy evaluation. Moving port 3367 to another CCNP (e.g. restrict-drbd-reactor) with ingress.fromEntities: [cluster] would NOT help — the ingressDeny.fromEntities: [world] rule is evaluated first, and since the traffic IS classified as world, it would still be denied.

Confirmed via cilium monitor:

xx drop (Policy denied by denylist) identity world->unknown:
  10.244.0.204:37124 -> 10.88.148.82:3367 tcp SYN

Why this does not affect all clusters

In standard kube-ovn distributed-gateway mode with natOutgoing: true, pod→remote-node traffic gets SNAT to the node IP. Cilium then classifies it as host (allowed by ingress.fromEntities). On affected clusters, SNAT is not applied.

Short-term fix

Remove port 3367 (and 3366) from restrict-system-components ingressDeny.

Long-term fix

Fix Cilium identity classification in generic-veth chaining mode with kube-ovn. Cross-node pod traffic should be classified as cluster, not world.

Steps to reproduce

1. Deploy cozystack with networking: kubeovn-cilium, Cilium in chaining mode
2. Enable LINSTOR on a multi-node cluster
3. kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor node list — satellites on remote nodes are OFFLINE
4. TCP connection from controller pod to remote node IP:3367 fails (world → denied)
5. HostNetwork pod on same node can reach remote satellite:3367 (host → allowed)
6. cilium monitor --type drop on remote node confirms: Policy denied by denylist, identity world->unknown

Environment

• Cozystack version: current main
• CNI: Cilium + kube-ovn (generic-veth chaining)
• Affected component: cilium-networkpolicy → restrict-system-components CCNP

[dosubot]

Hi Nikita Aboltin (@2mind)! I'm Dosu and I'm helping the Cozystack team.

Your analysis looks accurate. Port 3367 is indeed listed in the restrict-system-components CCNP's ingressDeny block [1], which denies world entity access. The key problem is that in Cilium chaining mode with kube-ovn, cross-node pod traffic gets classified as world→unknown because Cilium ipcache doesn't have identity entries for kube-ovn-managed pods.

The restrict-drbd-reactor policy in the LINSTOR chart [2] already follows the pattern you described — it allows fromEntities: [world, host, cluster] while only denying world on port 9942 (DRBD reactor). Moving ports 3366/3367 there (or at least adding a fromEntities: [cluster] exception in restrict-system-components) would let intra-cluster LINSTOR satellite traffic through while still blocking external access.

One thing to note: restrict-drbd-reactor is templated in the LINSTOR Helm chart at packages/system/linstor/templates/networkpolicy.yaml [3], while restrict-system-components lives in packages/system/cilium-networkpolicy/templates/networkpolicy.yaml [4]. If the fix moves ports to restrict-drbd-reactor, the cilium-networkpolicy chart's test suite will also need updating to remove the port 3367 assertion.

No prior PRs or issues were found tracking this, so this appears to be a new report.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[2mind]

# (@dosu) This comment addresses the original version of the issue. The body has since been updated with a correction: ingressDeny has priority over ingress in Cilium, so moving port 3367 from restrict-system-components to restrict-drbd-reactor does NOT help — that policy also has ingressDeny.fromEntities: [world], and since cross-node pod traffic IS classified as world (due to missing identity in chaining mode), it would still be denied regardless of ingress.fromEntities: [cluster].

Short-term fix: remove port 3367 from the ingressDeny list entirely.
Long-term fix: fix Cilium identity classification in generic-veth chaining mode so pod traffic is cluster, not world.

[dosubot]

Thanks for the correction, Nikita Aboltin (@2mind). You're right — since Cilium's ingressDeny takes priority over ingress allow rules, moving port 3367 into restrict-drbd-reactor wouldn't help because that policy also denies world on specific ports, and cross-node pod traffic is misclassified as world in chaining mode.

Looking at the current restrict-system-components policy [1], port 3367 sits alongside etcd, kubelet, and metallb ports in the ingressDeny block. The policy also has a broad ingress allow from [world, host, cluster] [2], so removing 3367 from the deny list would still leave the general ingress allow in place — LINSTOR satellite traffic would flow while the other system ports remain protected.

Your two-track approach makes sense:

• Short-term: Remove port 3367 from the ingressDeny list in restrict-system-components. This unblocks LINSTOR satellite connectivity without affecting the deny rules for the other 13 ports.
• Long-term: Fix identity classification in generic-veth chaining mode so kube-ovn-managed pod traffic gets a proper cluster identity instead of world.

Are you planning to submit a PR for the short-term fix?

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.