Skip to content

OVS br-ex drop rule silently discards OVN reply traffic for LoadBalancer connections #214

@brianredbeard

Description

@brianredbeard

OVS br-ex drop rule silently discards OVN reply traffic for LoadBalancer connections

Description

On a single-node MicroShift 4.21.0 (OKD SCOS) cluster using OVN-Kubernetes, reply traffic from OVN for externally-originated LoadBalancer connections is dropped by a low-priority OVS flow rule on br-ex. This is a second, independent bug that prevents external LoadBalancer access even after Bug 1 (ETP DNAT to unreachable 169.254.169.3) is bypassed.

Traffic that follows the OVN-KUBE-EXTERNALIP DNAT path to the ClusterIP enters OVN successfully and reaches the ingress router pod, but reply packets are dropped by priority=9, in_port=2, actions=drop on br-ex table 0.

Environment

Component Version / Detail
MicroShift 4.21.0_g29f429c21_4.21.0_okd_scos.ec.15
OS RHEL 9.7 (kernel 5.14)
OVS 3.5.2-51.el9fdp
CNI OVN-Kubernetes (cniPlugin: "ovnk")
Node Single-node, 4 CPU, 1 NIC (enp0s25)
Node IP 192.168.137.31

Affected Service

Service: router-default (openshift-ingress)
Type: LoadBalancer
ClusterIP: 10.43.52.69
ExternalTrafficPolicy: Local
NodePorts: 30703 (HTTP), 32298 (HTTPS)

Steps to Reproduce

  1. Install MicroShift 4.21.0 (OKD SCOS) on a single-node RHEL 9 system with OVN-K CNI
  2. Bypass Bug 1 by inserting RETURN rules in the OVN-KUBE-ETP chain:
    iptables -t nat -I OVN-KUBE-ETP 1 -d <node-ip> -p tcp --dport 443 -j RETURN
    iptables -t nat -I OVN-KUBE-ETP 1 -d <node-ip> -p tcp --dport 80 -j RETURN
  3. From an external machine, attempt to access a route:
    curl -kv https://<route-hostname>
    
  4. Observe: connection still times out

Expected Behavior

Reply traffic from OVN (entering br-ex via the OVN patch port, in_port=2) for externally-originated connections should be properly conntracked and forwarded back to the external client.

Actual Behavior

Reply packets are caught by the catch-all drop rule and never reach the external client.

Traffic Path

External client → 192.168.137.31:443
  → iptables PREROUTING: OVN-KUBE-ETP (bypassed via RETURN)
  → iptables PREROUTING: OVN-KUBE-EXTERNALIP DNAT to 10.43.52.69:443
  → iptables FORWARD: accepted
  → br-ex → OVN patch port (port 2) → OVN cluster network → ingress pod
  → Reply from pod enters br-ex via port 2
  → OVS table 0: priority=9, in_port=2, actions=drop   ← DROPPED HERE

Evidence

Kernel logs confirm br-ex bridge loop pattern

IN=br-ex OUT=br-ex SRC=<client> DST=10.43.52.69 ... SYN
(retransmitted 3x at 1-second intervals, no response)

Conntrack shows SYN_SENT with no reply

tcp  SYN_SENT src=169.254.169.2 dst=10.43.52.69 sport=42286 dport=443 [UNREPLIED]
  zone=9 mark=10
tcp  SYN_SENT src=169.254.169.2 dst=10.43.52.69 sport=42286 dport=443 [UNREPLIED]
  zone=64001 mark=0

OVS drop rule on br-ex table 0

$ ovs-ofctl dump-flows br-ex table=0
...
priority=9, in_port=2, actions=drop    ← catches all OVN reply traffic
...

OVS NAT tables show zero packet matches

table=4: ct(commit,table=3,zone=64002,nat(src=169.254.169.1))  n_packets=0
table=5: ct(commit,table=2,zone=64001,nat)                      n_packets=0

The NAT tables (4-5) that should handle return traffic conntrack and NAT processing have zero packet hits, confirming traffic is dropped before reaching them.

Proof that fixing the reply path changes behavior

Adding a high-priority OVS flow for the reply path changed behavior from timeout (exit 28) to connection refused (exit 7):

$ ovs-ofctl add-flow br-ex "table=0,priority=200,in_port=2,ip,nw_dst=169.254.169.2,actions=ct(table=3,zone=64001,nat)"

$ curl -k https://fuseki-packagegraph.apps.kafka.tel/
# exit code 7 (connection refused) instead of 28 (timeout)

However, OVN reconciliation removes custom OVS flows within seconds, making this impractical.

Analysis

The priority=9, in_port=2, actions=drop rule on br-ex table 0 acts as a catch-all for traffic arriving from the OVN patch port. Higher-priority rules in table 0 are expected to match legitimate return traffic and route it through the conntrack/NAT pipeline (tables 3-5). For externally-originated LoadBalancer traffic that was DNAT'd via iptables (the OVN-KUBE-EXTERNALIP path), no matching higher-priority rule exists — the return traffic falls through to the drop rule.

Additionally, net.ipv4.conf.br-ex.rp_filter = 1 (strict reverse path filtering) may contribute to packet drops for reply traffic where the source IP is a ClusterIP (10.43.52.69) that is not routable via br-ex. However, disabling rp_filter alone does not fix the issue because the OVS drop rule catches traffic first.

Additional Context

OVN load balancer configuration is correct

VIP 10.43.52.69:80  → 10.42.0.73:80   (ingress pod)
VIP 10.43.52.69:443 → 10.42.0.73:443  (ingress pod)
VIP 192.168.137.31:80  → 10.42.0.73:80
VIP 192.168.137.31:443 → 10.42.0.73:443

Load balancer rules exist on both the cluster router and the external gateway router. The control plane is correct — the bug is in the dataplane flow programming.

Direct pod access works

$ curl -k https://10.42.0.73 -H 'Host: fuseki-packagegraph.apps.kafka.tel'
# Returns expected response

Firewall is not the issue

Ports 80, 443, and NodePort range 30000-32767 are open. FORWARD policy is ACCEPT. Problem persists with all firewall rules bypassed.

Diagnostic Commands

# Verify OVS drop rule
ovs-ofctl dump-flows br-ex table=0 | grep "in_port=2.*drop"

# Verify conntrack shows unreplied connections
conntrack -L -p tcp --dport 443 2>/dev/null | grep UNREPLIED

# Check NAT table packet counts
ovs-ofctl dump-flows br-ex table=4
ovs-ofctl dump-flows br-ex table=5

Workaround

A host-level TCP proxy bypasses the entire OVN LoadBalancer dataplane:

# /usr/local/bin/ingress-proxy.py — forwards external ports to ClusterIP
# Systemd services: ingress-proxy-http.service, ingress-proxy-https.service
ExecStart=/usr/bin/python3 /usr/local/bin/ingress-proxy.py 443 10.43.52.69 443
ExecStart=/usr/bin/python3 /usr/local/bin/ingress-proxy.py 80  10.43.52.69 80

Combined with iptables ACCEPT rules in PREROUTING to prevent DNAT to the broken paths. Note: iptables rules may be reconciled by OVN-K on restart.

Prior Art

  • Red Hat Solution 6999225"Connections to NodePort and LoadBalancer services with externalTrafficPolicy: Local fail and timeout on OVNKubernetes" (may describe a related root cause; resolution behind paywall)

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions