USHIFT-7432: Fix OVN multinode SBDB connectivity regression by eslutsky · Pull Request #7344 · openshift/microshift

eslutsky · 2026-09-08T09:14:02Z

Summary

Fixes the OVN multinode networking regression introduced by PR #6912 (June 2026 rebase), which removed --nb-address/--sb-address from ovnkube --init-node without providing an equivalent in ovnkube.conf. Worker nodes lost their only pointer to the primary's NB/SB databases, causing each node to form an isolated 1-node OVN RAFT cluster — no geneve tunnels formed, all cross-node pod traffic dropped.

Root Cause

Commit 8a0f4f23eb (PR #6912) removed deprecated CLI flags without a config-file equivalent:

-  --nb-address "{{.OVN_NB_DB_LIST}}" \
-  --sb-address "{{.OVN_SB_DB_LIST}}" \

The ovnkube.conf had no [OvnNorth]/[OvnSouth] sections to replace them. Subsequently, PR #7052 cleaned up the render params and added the stale-socket wait guard — which works for single-node but blocks indefinitely on worker nodes that have no local SBDB.

Fix

assets/components/ovn/common/configmap.yaml — add [OvnNorth]/[OvnSouth] sections in multinode mode, pointing all nodes at the primary's TCP databases:

[OvnNorth]
address=tcp:{{.NodeIP}}:{{.OVN_NB_PORT}}

[OvnSouth]
address=tcp:{{.NodeIP}}:{{.OVN_SB_PORT}}

pkg/components/networking.go — two changes:

Pass MultiNodeEnabled render param so the configmap template can conditionally emit the stanzas.
Worker nodes (BootstrapKubeConfigExists() == true) only deploy the node/daemonset.yaml, not master/daemonset.yaml — prevents workers from starting an isolated SBDB/NBDB raft cluster.

assets/components/ovn/multi-node/node/daemonset.yaml — replace the hard local-socket wait with a combined condition that handles both primary (socket) and worker (TCP ovn-remote set by ovnkube-node). Also fixes a grep anchor bug (^tcp: → tcp:) so quoted OVS output "tcp:..." is correctly matched.

Validation

Tested on a 2-VM KVM cluster (CentOS Stream 9, MicroShift 5.0 rc.1):

Check	Result
`[OvnNorth]`/`[OvnSouth]` in configmap pointing to primary	✅
ovnkube-master only on primary node	✅
Geneve tunnels bidirectional, stable 77+ min	✅
Cross-node pod ping 5/5, 0% loss, ~1ms RTT	✅
`addLogicalPort` OVN errors ongoing	None
`[sig-network]` failures in CNCF conformance	Zero

440 CNCF conformance tests ran to completion; the 6 non-networking failures (StatefulSet, SchedulerPreemption, Aggregator) were due to missing oc adm policy add-scc-to-group grants in the test setup, unrelated to this fix.

Remaining known limitation

The primary VM must allow TCP 9641/9642/9643/9644 from worker nodes. configure-node.sh stops firewalld, but on some base images explicit iptables rules may be needed. This should be addressed in a follow-up to configure-node.sh.

Test plan

CI: periodic-ci-openshift-microshift-release-5.0-periodics-e2e-aws-tests-nightly passes el98-src@cncf-conformance scenario
Verify ovnkube-master pod runs only on primary node in multinode mode
Verify geneve tunnels form between all nodes
Verify cross-node pod communication works (ping between pods on different nodes)

🤖 Generated with Claude Code

Commit 8a0f4f2 (PR #6912, Jun 2026 rebase) removed the --nb-address/--sb-address flags from ovnkube --init-node, which were the only mechanism telling worker nodes where to find the primary's OVN NB/SB databases. After the removal, worker nodes fell back to a local unix socket that doesn't exist on workers, resulting in each node forming an isolated 1-node OVN RAFT cluster with no inter-node geneve tunnels. Root cause: after the upstream ovnkube binary dropped those CLI flags (deprecated in favour of config-file), the ovnkube.conf had no [OvnNorth] / [OvnSouth] section to replace them. Fix (3 files): 1. assets/components/ovn/common/configmap.yaml Add [OvnNorth] / [OvnSouth] sections in multinode mode, pointing all nodes at the primary's TCP NB/SB databases (tcp:PRIMARY_IP:9641/9642). The primary's NodeIP is already available in the render params. 2. pkg/components/networking.go - Pass MultiNodeEnabled as a render param so the configmap template can conditionally emit the [OvnNorth]/[OvnSouth] stanzas. - Worker nodes (identified by BootstrapKubeConfigExists()) only deploy the node DaemonSet, not the master DaemonSet. This prevents worker nodes from starting an isolated SBDB/NBDB raft cluster of their own. 3. assets/components/ovn/multi-node/node/daemonset.yaml Replace the hard "wait for local SBDB unix socket" guard with a combined condition: - Primary: exits when local socket appears (existing fast path). - Worker: exits when ovnkube-node sets ovn-remote=tcp:... in OVS external_ids (which happens once it connects to the primary SBDB via the new configmap stanzas). Also fixes the grep anchor (^tcp: → tcp:) so quoted OVS output "tcp:..." is correctly matched. Validated on a 2-VM KVM cluster (CentOS Stream 9, MicroShift 5.0 rc.1): - geneve tunnels form on both nodes - cross-node pod ping: 5/5 packets, 0% loss, ~1ms RTT - ovnkube-master runs only on the primary node - 440 CNCF conformance tests run; 0 sig-network failures Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

openshift-ci-robot · 2026-09-08T09:14:07Z

@eslutsky: This pull request explicitly references no jira issue.

Details

In response to this:

Summary

Fixes the OVN multinode networking regression introduced by PR #6912 (June 2026 rebase), which removed --nb-address/--sb-address from ovnkube --init-node without providing an equivalent in ovnkube.conf. Worker nodes lost their only pointer to the primary's NB/SB databases, causing each node to form an isolated 1-node OVN RAFT cluster — no geneve tunnels formed, all cross-node pod traffic dropped.

Root Cause

Commit 8a0f4f23eb (PR #6912) removed deprecated CLI flags without a config-file equivalent:
-  --nb-address "{{.OVN_NB_DB_LIST}}" \
-  --sb-address "{{.OVN_SB_DB_LIST}}" \
The ovnkube.conf had no [OvnNorth]/[OvnSouth] sections to replace them. Subsequently, PR #7052 cleaned up the render params and added the stale-socket wait guard — which works for single-node but blocks indefinitely on worker nodes that have no local SBDB.

Fix

assets/components/ovn/common/configmap.yaml — add [OvnNorth]/[OvnSouth] sections in multinode mode, pointing all nodes at the primary's TCP databases:
[OvnNorth]
address=tcp:{{.NodeIP}}:{{.OVN_NB_PORT}}

[OvnSouth]
address=tcp:{{.NodeIP}}:{{.OVN_SB_PORT}}
pkg/components/networking.go — two changes:

Pass MultiNodeEnabled render param so the configmap template can conditionally emit the stanzas.

Worker nodes (BootstrapKubeConfigExists() == true) only deploy the node/daemonset.yaml, not master/daemonset.yaml — prevents workers from starting an isolated SBDB/NBDB raft cluster.

assets/components/ovn/multi-node/node/daemonset.yaml — replace the hard local-socket wait with a combined condition that handles both primary (socket) and worker (TCP ovn-remote set by ovnkube-node). Also fixes a grep anchor bug (^tcp: → tcp:) so quoted OVS output "tcp:..." is correctly matched.

Validation

Tested on a 2-VM KVM cluster (CentOS Stream 9, MicroShift 5.0 rc.1):

Check Result

[OvnNorth]/[OvnSouth] in configmap pointing to primary ✅

ovnkube-master only on primary node ✅

Geneve tunnels bidirectional, stable 77+ min ✅

Cross-node pod ping 5/5, 0% loss, ~1ms RTT ✅

addLogicalPort OVN errors ongoing None

[sig-network] failures in CNCF conformance Zero

440 CNCF conformance tests ran to completion; the 6 non-networking failures (StatefulSet, SchedulerPreemption, Aggregator) were due to missing oc adm policy add-scc-to-group grants in the test setup, unrelated to this fix.

Remaining known limitation

The primary VM must allow TCP 9641/9642/9643/9644 from worker nodes. configure-node.sh stops firewalld, but on some base images explicit iptables rules may be needed. This should be addressed in a follow-up to configure-node.sh.

Test plan

CI: periodic-ci-openshift-microshift-release-5.0-periodics-e2e-aws-tests-nightly passes el98-src@cncf-conformance scenario

Verify ovnkube-master pod runs only on primary node in multinode mode

Verify geneve tunnels form between all nodes

Verify cross-node pod communication works (ping between pods on different nodes)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-bot · 2026-09-08T09:14:10Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

coderabbitai · 2026-09-08T09:14:11Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 62b4ed3b-b6a7-48e9-90f2-608e337893b6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

openshift-ci · 2026-09-08T09:14:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: eslutsky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [eslutsky]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pmtk · 2026-09-08T09:46:53Z

/test ?

pmtk · 2026-09-08T09:48:00Z

/test e2e-aws-tests-bootc-periodic-el10 e2e-aws-tests-bootc-periodic-arm-el9

Testing periodic jobs because they run the multi-node CNCF scenarios

openshift-ci-robot · 2026-09-08T09:57:30Z

@eslutsky: This pull request references USHIFT-7432 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Fixes the OVN multinode networking regression introduced by PR #6912 (June 2026 rebase), which removed --nb-address/--sb-address from ovnkube --init-node without providing an equivalent in ovnkube.conf. Worker nodes lost their only pointer to the primary's NB/SB databases, causing each node to form an isolated 1-node OVN RAFT cluster — no geneve tunnels formed, all cross-node pod traffic dropped.

Root Cause

Commit 8a0f4f23eb (PR #6912) removed deprecated CLI flags without a config-file equivalent:
-  --nb-address "{{.OVN_NB_DB_LIST}}" \
-  --sb-address "{{.OVN_SB_DB_LIST}}" \
The ovnkube.conf had no [OvnNorth]/[OvnSouth] sections to replace them. Subsequently, PR #7052 cleaned up the render params and added the stale-socket wait guard — which works for single-node but blocks indefinitely on worker nodes that have no local SBDB.

Fix

assets/components/ovn/common/configmap.yaml — add [OvnNorth]/[OvnSouth] sections in multinode mode, pointing all nodes at the primary's TCP databases:
[OvnNorth]
address=tcp:{{.NodeIP}}:{{.OVN_NB_PORT}}

[OvnSouth]
address=tcp:{{.NodeIP}}:{{.OVN_SB_PORT}}
pkg/components/networking.go — two changes:

Pass MultiNodeEnabled render param so the configmap template can conditionally emit the stanzas.

Worker nodes (BootstrapKubeConfigExists() == true) only deploy the node/daemonset.yaml, not master/daemonset.yaml — prevents workers from starting an isolated SBDB/NBDB raft cluster.

assets/components/ovn/multi-node/node/daemonset.yaml — replace the hard local-socket wait with a combined condition that handles both primary (socket) and worker (TCP ovn-remote set by ovnkube-node). Also fixes a grep anchor bug (^tcp: → tcp:) so quoted OVS output "tcp:..." is correctly matched.

Validation

Tested on a 2-VM KVM cluster (CentOS Stream 9, MicroShift 5.0 rc.1):

Check Result

[OvnNorth]/[OvnSouth] in configmap pointing to primary ✅

ovnkube-master only on primary node ✅

Geneve tunnels bidirectional, stable 77+ min ✅

Cross-node pod ping 5/5, 0% loss, ~1ms RTT ✅

addLogicalPort OVN errors ongoing None

[sig-network] failures in CNCF conformance Zero

440 CNCF conformance tests ran to completion; the 6 non-networking failures (StatefulSet, SchedulerPreemption, Aggregator) were due to missing oc adm policy add-scc-to-group grants in the test setup, unrelated to this fix.

Remaining known limitation

The primary VM must allow TCP 9641/9642/9643/9644 from worker nodes. configure-node.sh stops firewalld, but on some base images explicit iptables rules may be needed. This should be addressed in a follow-up to configure-node.sh.

Test plan

CI: periodic-ci-openshift-microshift-release-5.0-periodics-e2e-aws-tests-nightly passes el98-src@cncf-conformance scenario

Verify ovnkube-master pod runs only on primary node in multinode mode

Verify geneve tunnels form between all nodes

Verify cross-node pod communication works (ping between pods on different nodes)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Make it explicit that node/daemonset.yaml is shared across all nodes while master/daemonset.yaml is primary-only, instead of repeating the node entry in both branches of an if/else. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

eslutsky · 2026-09-08T12:57:53Z

/test all

…ation Disable the EXCLUDE_CNCF_CONFORMANCE gate so the CNCF conformance scenario runs unconditionally while the multinode OVN SBDB fix (PR #7344) is validated in CI. Revert once the job is confirmed green. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

eslutsky · 2026-09-08T13:52:02Z

/test e2e-aws-tests-bootc-periodic-el10 e2e-aws-tests-bootc-periodic-arm-el9

… label In MicroShift multinode mode both nodes carry node-role.kubernetes.io/master, so the ovnkube-master DaemonSet (which uses that nodeSelector) was being scheduled on all nodes regardless of the networking.go primary/worker split. Each node ended up running its own isolated OVN SBDB, preventing geneve tunnel formation and breaking all cross-node pod networking. Fix: at kubelet startup the primary node (identified by the absence of a bootstrap kubeconfig) receives the label node.microshift.io/role=primary. The ovnkube-master DaemonSet nodeSelector is updated to match that label, so it is scheduled exclusively on the primary node. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

eslutsky · 2026-09-08T18:41:43Z

/test e2e-aws-tests-bootc-periodic-el10 e2e-aws-tests-bootc-periodic-arm-el9

… mode In multinode mode the ovnkube-config ConfigMap contains [OvnNorth] and [OvnSouth] stanzas with the primary node's IP address. Both the primary and worker previously called ApplyConfigMaps unconditionally, so whichever node wrote last won: the worker overwrote the correct primary IP with its own IP, causing ovn-controller on all nodes to attempt SBDB connections to the worker's address (which has no SBDB), breaking cluster networking. Fix: skip ApplyConfigMaps on worker nodes in multinode mode. The primary writes the configmap once at startup with the correct IP; workers only read it via the DaemonSet volume mount. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

microshift-cleanup-data removed the br-int bridge and killed OVN processes, but left OVN-related entries (ovn-remote, ovn-encap-ip, etc.) in the OVS Open_vSwitch external_ids table. On a subsequent MicroShift start in multinode mode the stale ovn-remote value (e.g. a unix socket path or a worker's own TCP address from a previous run) prevented ovn-controller from connecting to the correct SBDB, breaking cluster networking. Clear all OVN-managed external_ids keys during full/OVN cleanup so the next startup always begins from a clean OVS state. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

The previous commit placed the ovs-vsctl external_ids removal loop after 'systemctl stop ovsdb-server', making ovs-vsctl fail silently because the OVS socket was already gone. The stale ovn-remote value was never cleared. Move the loop before the ovsdb-server stop so ovs-vsctl can still reach the running daemon. ovsdb-server and OVN processes are killed afterwards as before. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

…ode DaemonSet The ovnkube binary may not parse the [OvnSouth] address= stanza from ovnkube.conf, causing it to default ovn-remote to the local unix socket path instead of the primary's TCP endpoint. On worker nodes this breaks geneve tunnel formation because ovn-controller connects to a non-existent local socket rather than the primary SBDB. Before starting ovnkube --init-node, read the [OvnSouth] address= value directly from the mounted configmap file using awk and set it explicitly in OVS external_ids. This ensures ovn-controller on worker nodes connects to the primary's SBDB over TCP regardless of how the ini file is parsed. The snippet is a no-op on the primary where [OvnSouth].address is absent. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

…startup The previous commit set ovn-remote so ovn-controller could reach the primary SBDB, but ovn-controller also needs ovn-encap-type=geneve and ovn-encap-ip=<node-ip> in OVS external_ids to register the local chassis and form geneve tunnels. These are normally set by ovnkube --init-node, but ovnkube is blocked waiting for a NBDB connection it cannot complete (same ini-parse issue). Set them explicitly before starting ovnkube so ovn-controller can proceed independently. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

The ovnkube binary cannot parse [OvnNorth]/[OvnSouth] address= fields from the ini config (upstream format mismatch in this binary version). Without the primary's NB/SB TCP addresses, ovnkube-node on workers falls back to the local unix socket paths, which do not exist on workers, and stalls before it can set the encap external_ids needed by ovn-controller. Fix the ovnkube-node startup script to handle workers explicitly: 1. Extract NB/SB addresses from the mounted configmap using awk. 2. Set ovn-encap-type=geneve and ovn-encap-ip=<node-ip> in OVS so ovn-controller can register the chassis and form geneve tunnels. 3. On worker nodes (SB address points to a different host): a. Remove stale socket files from any previous run. b. Start socat relays that serve the local unix sockets and forward connections to the primary's TCP NB (9641) and SB (9642) ports. This lets the ovnkube binary connect via its expected unix path while actually reaching the remote primary databases. c. Set ovn-remote=tcp:PRIMARY:9642 in OVS so ovn-controller connects directly to the primary SBDB over TCP (faster than the relay). d. Export OVN_SB_DB/OVN_NB_DB so ovn-sbctl/ovn-nbctl subprocess calls inside ovnkube also reach the primary. 4. Improve ovn-controller wait logic: for unix: remotes verify the socket actually accepts connections (not just the file exists). 5. Handle stale ovn-controller processes and .ctl sockets gracefully. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

eslutsky · 2026-09-09T10:21:10Z

/test e2e-aws-tests-bootc-periodic-el10 e2e-aws-tests-bootc-periodic-arm-el9

The val=$(ovs-vsctl ...) assignment in the OVN external_ids cleanup loop ran under set -euo pipefail without || true. If ovsdb-server was already stopped (e.g., by a prior --ovn cleanup or by the test body), ovs-vsctl would exit non-zero and kill the script immediately, leaving cleanup incomplete and causing the test teardown to fail. Add || true to the assignment so a failed ovs-vsctl call is treated as "key not present" rather than a fatal error, making the loop idempotent when ovsdb-server is unavailable. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

The 'A && B || C' pattern triggers SC2015 because if B fails, C runs unexpectedly. Replace with an explicit if/fi block to make the intent clear and silence the shellcheck warning. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

eslutsky · 2026-09-09T15:28:21Z

/test e2e-aws-tests-bootc-periodic-el10 e2e-aws-tests-bootc-periodic-arm-el9

eslutsky · 2026-09-09T19:25:57Z

/test all

openshift-ci · 2026-09-09T21:14:38Z

@eslutsky: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-tests-bootc-periodic-el10	`8bf5a8c`	link	true	`/test e2e-aws-tests-bootc-periodic-el10`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 8, 2026

openshift-ci Bot requested review from kasturinarra and pacevedom September 8, 2026 09:14

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 8, 2026

pmtk reviewed Sep 8, 2026

View reviewed changes

Comment thread pkg/components/networking.go Outdated

eslutsky changed the title ~~NO-ISSUE: Fix OVN multinode SBDB connectivity regression~~ USHIFT-7432: Fix OVN multinode SBDB connectivity regression Sep 8, 2026

eslutsky force-pushed the USHIFT-cncf-ovn-multinode-fix branch 2 times, most recently from 4d53726 to 358e6c8 Compare September 8, 2026 13:23

eslutsky and others added 6 commits September 9, 2026 07:20

Uh oh!

Conversation

eslutsky commented Sep 8, 2026

Summary

Root Cause

Fix

Validation

Remaining known limitation

Test plan

Uh oh!

openshift-ci-robot commented Sep 8, 2026

Summary

Root Cause

Fix

Validation

Remaining known limitation

Test plan

Uh oh!

openshift-merge-bot Bot commented Sep 8, 2026

Uh oh!

coderabbitai Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

openshift-ci Bot commented Sep 8, 2026

Uh oh!

Uh oh!

pmtk commented Sep 8, 2026

Uh oh!

pmtk commented Sep 8, 2026

Uh oh!

openshift-ci-robot commented Sep 8, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root Cause

Fix

Validation

Remaining known limitation

Test plan

Uh oh!

eslutsky commented Sep 8, 2026

Uh oh!

eslutsky commented Sep 8, 2026

Uh oh!

eslutsky commented Sep 8, 2026

Uh oh!

eslutsky commented Sep 9, 2026

Uh oh!

eslutsky commented Sep 9, 2026

Uh oh!

eslutsky commented Sep 9, 2026

Uh oh!

openshift-ci Bot commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderabbitai Bot commented Sep 8, 2026 •

edited

Loading

openshift-ci-robot commented Sep 8, 2026 •

edited by openshift-ci Bot

Loading