Repository navigation
bug(sandbox): mediated nonblocking TCP connect returns 0 instead of EINPROGRESS, breaking Microsoft.Data.SqlClient #4980
Description
User Story
I use OpenShell 0.1.5 as the sandbox for the development workspaces of our internal developer platform: each workspace is an OpenShell sandbox (Kubernetes driver) where developers and coding agents build and run .NET services, with network policies that allow the private services each workspace needs.
I directly encountered a .NET 10 service inside a sandbox that could not open its SQL Server database through a TCP endpoint allowed by the policy, while the same service connects fine outside the sandbox.
This blocks running any .NET service that talks to SQL Server inside a sandbox.
Problem Statement
When a workload calls connect() on a nonblocking TCP socket (O_NONBLOCK set with fcntl after socket(2)) to an external destination that the policy allows, and the sandbox mediates the connect through its loopback relay, the call returns 0 (connected immediately). The kernel returns -1 / EINPROGRESS for a nonblocking TCP connect, and the sandbox's own loopback path (connect_local_tcp) already preserves EINPROGRESS. Blocking connects behave correctly.
Impact / Why This Matters
Clients that follow the nonblocking-connect contract strictly treat an immediate success as an error. Microsoft.Data.SqlClient (managed SNI, which every .NET application uses on Linux) opens its socket nonblocking whenever the connect timeout is finite (the default is 15 s), expects WouldBlock, and otherwise aborts:
- inside the sandbox the connection fails only after the whole connect timeout (14.6 s in our runs) with
error: 35 - An internal exception was caught/Socket did not throw expected 'SocketException' with error code 'WouldBlock'; - the only workaround is
Connect Timeout=0in the connection string, which removes the connect timeout entirely (an unreachable server then hangs the caller forever) and forces every application to change its configuration just to run inside OpenShell.
Acceptance Criteria
- A nonblocking
connect()to an allowed external TCP destination returns-1witherrno == EINPROGRESS. - The socket then polls writable with
SO_ERROR == 0, data flows through the relay, and a repeatedconnect()returnsEISCONN. - A blocking
connect()still returns0. - Microsoft.Data.SqlClient with the default
Connect Timeoutconnects to a SQL Server reached through a TCP endpoint allowed by the policy.
Reproduction Steps
- Create a sandbox:
openshell sandbox create --name nb-connect --no-auto-providers
- Allow a plain TCP endpoint for Python:
openshell policy update nb-connect \ --rule-name tcp_probe \ --binary '/usr/bin/python3*' \ --add-endpoint example.com:443::tcp \ --wait - Run a nonblocking connect (python3 from the sandbox image; any language shows the same):
Expected:
openshell sandbox exec -n nb-connect --no-login-shell -- python3 -c ' import errno, socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.setblocking(False) r = s.connect_ex(("example.com", 443)) print(r, errno.errorcode.get(r, "OK"))'
115 EINPROGRESS(what the same code prints outside the sandbox). Actual:0 OK.
We hit it with a policy created through the gateway API: an L4 endpoint (tls: skip, enforcement: enforce) for a SQL Server host on a private address listed in allowed_ips, binaries /**. The steps above are the equivalent with the CLI and a public host.
Environment
- OpenShell: v0.1.5 (gateway, supervisor and sandbox runtime
0.1.5). The same reply is inv0.1.6and inmain(2c31ac7). - Deployment: Kubernetes driver on k3s (k3d) on a Linux 6.14 host; also reproduced on a Docker-driver gateway 0.1.5 on the same host.
- Workload: .NET 10, Microsoft.Data.SqlClient 7.0.2, SQL Server 2022 reached through a TCP endpoint.
Logs
# official 0.1.5 sandbox runtime: nonblocking probe, then SqlClient with Connect Timeout=15
connect_ex: 0 OK
repeated connect: EISCONN
SqlException after 14606 ms: A network-related or instance-specific error occurred while establishing a connection to SQL Server. [...] (provider: TCP Provider, error: 35 - An internal exception was caught)
-> Exception: Socket did not throw expected 'SocketException' with error code 'WouldBlock'.
# the same sandbox runtime with the patch below
connect_ex: 115 EINPROGRESS
writable: True, SO_ERROR: 0
repeated connect: EISCONN
SqlClient: connected in 152 ms (SQL Server 16.0)Root cause and a tested fix
In crates/openshell-sandbox/src/network_broker.rs, connect_socket answers TcpOpenDecision::RelayReady with respond_value(notification.id, 0) whatever the socket's O_NONBLOCK flag is (line 1060 on main). By then establish_relay has already connected the workload's shared open file to the loopback relay (connect_exact sets O_NONBLOCK for its bounded attempt and restores the workload's value afterwards), so the kernel-level connect is complete when the notification is answered.
The fix we tested reads F_GETFL on the retained socket in establish_relay, before connect_exact, returns the flag together with the stream, and answers EINPROGRESS when the socket is nonblocking (0 otherwise). Since the relay connection is already established, the socket is immediately writable with SO_ERROR == 0 and a repeated connect returns EISCONN, which is exactly what the kernel reports after a nonblocking handshake completes. The flag has to be read at connect time rather than from the metadata recorded at socket(2), because .NET (and many other runtimes) set O_NONBLOCK with fcntl after creating the socket.
The patch adds a unit test next to denied_external_connect_keeps_socket_unconnected: a socket made nonblocking after creation gets EINPROGRESS, then POLLOUT, SO_ERROR == 0, EISCONN on a repeated connect, and a request/response through the relay. The broker tests pass, and the patch applies cleanly to main (offset 39 lines). We are running a 0.1.5 sandbox runtime built with it in our deployment, where SqlClient now connects with its default timeout. Happy to open a PR if the issue is accepted.
Patch (against v0.1.5; applies to main)
diff --git a/crates/openshell-sandbox/src/network_broker.rs b/crates/openshell-sandbox/src/network_broker.rs
index faaff92..95753ba 100644
--- a/crates/openshell-sandbox/src/network_broker.rs
+++ b/crates/openshell-sandbox/src/network_broker.rs
@@ -1016,10 +1016,19 @@ fn connect_socket(
destination,
)
}) {
- Ok(stream) => {
- let result = worker_listener
- .respond_value(notification.id, 0)
- .map(|()| stream);
+ Ok((stream, nonblocking)) => {
+ // A nonblocking connect reports what the kernel
+ // reports for a TCP handshake: EINPROGRESS, with
+ // the socket already connected and writable.
+ // Clients such as Microsoft.Data.SqlClient treat
+ // an immediate success on a nonblocking socket as
+ // an error.
+ let reply = if nonblocking {
+ worker_listener.respond_errno(notification.id, libc::EINPROGRESS)
+ } else {
+ worker_listener.respond_value(notification.id, 0)
+ };
+ let result = reply.map(|()| stream);
let _ = relay_tx.send(result);
}
Err(error) => {
@@ -1200,32 +1209,44 @@ fn ensure_dns_source_bound(fd: RawFd, family: InetFamily) -> io::Result<SocketAd
Ok(address)
}
+/// Connect the workload's socket to a fresh loopback relay and accept it.
+/// Returns the relay stream and whether the workload's open file was
+/// nonblocking when its connect was mediated: O_NONBLOCK is an OFD flag the
+/// workload may set after socket(2) (fcntl), so the socket metadata recorded at
+/// creation cannot tell.
fn establish_relay(
registry: &Mutex<SocketRegistry>,
tid: u32,
fd: RawFd,
expected_socket: SocketIdentity,
destination: SocketAddr,
-) -> io::Result<TcpStream> {
+) -> io::Result<(TcpStream, bool)> {
let relay = TcpListener::bind(match destination {
SocketAddr::V4(_) => SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0),
SocketAddr::V6(_) => SocketAddr::new(IpAddr::V6(Ipv6Addr::LOCALHOST), 0),
})?;
relay.set_nonblocking(false)?;
let relay_address = relay.local_addr()?;
- let expected_peer = {
+ let (expected_peer, nonblocking) = {
let mut registry = lock(registry);
let entry = registry.resolve_mut(tid, fd)?;
if entry.identity() != expected_socket {
return Err(io::Error::from_raw_os_error(libc::EBADF));
}
+ // Read before connect_exact, which sets O_NONBLOCK on the shared open
+ // file for its bounded attempt and then restores the workload's value.
+ // SAFETY: F_GETFL reads the flags of the live retained socket OFD.
+ let flags = unsafe { libc::fcntl(entry.retained_preconnect()?.as_raw_fd(), libc::F_GETFL) };
+ if flags < 0 {
+ return Err(io::Error::last_os_error());
+ }
connect_exact(entry.retained_preconnect()?.as_raw_fd(), relay_address)?;
let expected_peer = socket_local_addr(entry.retained_preconnect()?.as_raw_fd())?;
entry.set_state(SocketState::Connected {
original_peer: destination,
});
entry.release_preconnect();
- expected_peer
+ (expected_peer, flags & libc::O_NONBLOCK != 0)
};
relay.set_nonblocking(true)?;
let deadline = Instant::now() + RELAY_CONNECT_TIMEOUT;
@@ -1255,7 +1276,7 @@ fn establish_relay(
}
};
stream.set_nodelay(true)?;
- Ok(stream)
+ Ok((stream, nonblocking))
}
fn bind_socket(
@@ -2982,6 +3003,82 @@ mod tests {
client.join().expect("join client").expect("client relay");
}
+ #[test]
+ fn nonblocking_external_connect_reports_in_progress_like_the_kernel() {
+ let (launcher, listener) =
+ crate::linux::workload_launcher::start().expect("start workload launcher");
+ let broker = NetworkBroker::start_for_test(listener).expect("start network broker");
+ let client = std::thread::spawn(move || {
+ launcher
+ .execute(|| -> io::Result<()> {
+ // Blocking at socket(2), nonblocking by fcntl afterwards,
+ // as .NET's Socket.Blocking = false does.
+ let socket = socket2::Socket::new(
+ socket2::Domain::IPV4,
+ socket2::Type::STREAM,
+ None,
+ )?;
+ socket.set_nonblocking(true)?;
+ let address: SocketAddr = "203.0.113.7:443".parse().unwrap();
+ let started = socket.connect(&address.into());
+ if started.as_ref().err().and_then(io::Error::raw_os_error)
+ != Some(libc::EINPROGRESS)
+ {
+ return Err(io::Error::other(format!(
+ "nonblocking connect returned {started:?}, not EINPROGRESS"
+ )));
+ }
+ let mut poll = libc::pollfd {
+ fd: socket.as_raw_fd(),
+ events: libc::POLLOUT,
+ revents: 0,
+ };
+ // SAFETY: poll points to one live pollfd.
+ if unsafe { libc::poll(&raw mut poll, 1, 5_000) } != 1
+ || poll.revents & libc::POLLOUT == 0
+ {
+ return Err(io::Error::other("socket never became writable"));
+ }
+ if let Some(error) = socket.take_error()? {
+ return Err(error);
+ }
+ // A repeated connect reports the completed handshake.
+ if socket.connect(&address.into()).err().and_then(|e| e.raw_os_error())
+ != Some(libc::EISCONN)
+ {
+ return Err(io::Error::other("repeated connect did not report EISCONN"));
+ }
+ socket.set_nonblocking(false)?;
+ let mut stream: TcpStream = socket.into();
+ stream.write_all(b"request")?;
+ let mut response = [0_u8; 8];
+ stream.read_exact(&mut response)?;
+ if &response != b"response" {
+ return Err(io::Error::other("relay returned wrong response"));
+ }
+ Ok(())
+ })
+ .expect("launcher result")
+ });
+
+ let runtime = tokio::runtime::Builder::new_current_thread()
+ .enable_all()
+ .build()
+ .expect("test runtime");
+ let pending = runtime.block_on(broker.accept()).expect("pending TCP open");
+ let mut relay = runtime
+ .block_on(pending.complete(TcpOpenDecision::RelayReady))
+ .expect("complete relay")
+ .expect("authorized relay stream");
+ let mut request = [0_u8; 7];
+ relay
+ .read_exact(&mut request)
+ .expect("read relayed request");
+ assert_eq!(&request, b"request");
+ relay.write_all(b"response").expect("write relay response");
+ client.join().expect("join client").expect("client relay");
+ }
+
#[test]
fn denied_external_connect_keeps_socket_unconnected() {
let (launcher, listener) =
Activity