Skip to content

Android: Fix stale connection and panic - #886

Open
gonzaarcr wants to merge 1 commit into
tailscale:mainfrom
gonzaarcr:main
Open

gonzaarcr wants to merge 1 commit into
tailscale:mainfrom
gonzaarcr:main

Conversation

@gonzaarcr

@gonzaarcr gonzaarcr commented Sep 26, 2026 •

Copy link
Copy Markdown

Hi, I'm Gonzalo, I’ll try to be brief since I’ve been in contact with the support team for months nowchasin this bug. The tldr is: I work with Android scanners in a warehouse and we are using Tailscale. Ever since we installed Tailscale we been having intermittent connection bugs. The stack trace from the app say DNS error but I know is not that. The symptoms: it happens when the device is left idle for some time, but it also happens mid-use. I wasn’t able to turn off and on Tailscale because is forced always-on on the devices, but when I was able to replicate the bug on a dev device and tried to turn off the Tailscale this happened:

09-21 13:35:56.361 D gojni: panic in multiTUN.run close of closed channel: goroutine 2 [running]:
      runtime/debug.Stack()
          /build/.cache/tailscale-go/src/runtime/debug/stack.go:26 +0x64
      github.com/tailscale/tailscale-android/libtailscale.(*multiTUN).run.func1()
          /build/tailscale-android/libtailscale/multitun.go:106 +0x3c
      panic({0x7041e7e660?, 0x7042041950?})
          /build/.cache/tailscale-go/src/runtime/panic.go:860 +0x12c
      created by github.com/tailscale/tailscale-android/libtailscale.newTUNDevices in goroutine 40
          /build/tailscale-android/libtailscale/multitun.go:99 +0x20c

  09-21 13:35:56.379 E Go  : panic: close of closed channel [recovered, repanicked]
  09-21 13:35:56.384 F libc: Fatal signal 6 (SIGABRT), code -6 (SI_TKILL)
                             in tid 31023 (Thread-21), pid 30757 (m.tailscale.ipn)
  09-21 13:35:58.345 I Zygote: Process 30757 exited due to signal 6 (Aborted)

It seems there’s some stale tunnel that was supposed to be closed but it wasn’t, from there I just put AI to work and came up with this PR. Don’t disregard the PR just because the code is made by AI, there’s quite a lot of work put in it (the bug and research, not the code).

Here are some insights from AI that may help to understand the bug:

The tunnel is dead for everything, not just DNS, and it is still dead as I write this:

  • Name lookups through Tailscale fail after 80 s (the full Android resolver budget).
  • Raw-IP pings to the prod and staging VIPs through the tunnel: 100% loss.
  • Even a ping to 100.100.100.100, which the client answers itself without touching the network: 100% loss.
  • Meanwhile my Mac can ping this scanner over the tailnet, via the relay and directly over the office LAN. Its transport is fine.

So tailscaled is simply not reading packets off the live tunnel interface. That is the second, nastier variant I described on Sunday, and your patch does not cover it. The evidence for
the mechanism is on the device: tun1 has been sitting DOWN but present since 12:28:28, which means its file descriptor is still open inside tailscaled. It was created in a burst of three
interface rebuilds at 12:28:18, 12:28:28.684 and 12:28:28.750, the last two 66 ms apart. That is the race in the client's multi-interface code: when a second rebuild lands before the
first teardown finishes, the middle interface is never closed, and the packet reader can end up parked on it forever while Android routes the app's traffic to the newer one. Only a
process restart clears it.

The thing we had not seen before: what causes the rebuilds

On this unit the Wi-Fi network ID never changed all day, yet the VPN interface was rebuilt 53 times today and 74 times yesterday, six to twelve times an hour. Every rebuild lines up with
a change in the tailnet's node list: the route count steps between 125 and 129 as nodes appear and disappear. The nodes appearing are ECS tasks. Today alone: staging intranet and
privateapi at 11:24, prod intranet at 11:35, prod privateapi at 11:36, staging odoo-client at 11:43, smoke privateapi at 13:25, each with its egress twin, followed 10 to 30 minutes later
by the deletion of the old ephemeral nodes. At 12:38 five rebuilds happened within 600 ms as a batch of old nodes was deleted.

None of those services has anything to do with the scanners. They appear in every scanner's netmap because of one ACL grant in nettex-iac: network/index.ts:290-292 is a Taildrive
capability grant with src: * and dst: *, which makes every node a peer of every other node. So a deploy of any service, in any environment, rebuilds the VPN interface on every scanner in
the warehouse, dropping their TCP connections and rolling the dice on this wedge each time. That is what makes the failures look random and unrelated to anything the scanner is doing.

About the fix

What the new loop does:

  • Tracks the reader and the writer separately, and always attaches both to the newest device.
  • When a device is superseded, the loop itself closes it unless a writer is attached (the writer closes it on exit). Closing is what unblocks a reader parked on a dead interface, so
    nothing can strand and no DOWN tunN is left behind.
  • Reports tun write errors as a dropped packet, logged once per device, instead of failing the call. Android resets the old interface before Go is handed the new one, so a write in that
    window fails with EIO. The vendored core (b5e07cbf538e) still has injectToHost return permanently on the first write error, which is the DNS-only wedge in upstream #21155. Absorbing
    the error here neutralises that variant without patching core.
  • Shutdown and Close no longer double-close channels or block on goroutines that never started.

09-21 13:35:56.361 D gojni: panic in multiTUN.run close of closed channel: goroutine 2 [running]:
      runtime/debug.Stack()
          /build/.cache/tailscale-go/src/runtime/debug/stack.go:26 +0x64
      github.com/tailscale/tailscale-android/libtailscale.(*multiTUN).run.func1()
          /build/tailscale-android/libtailscale/multitun.go:106 +0x3c
      panic({0x7041e7e660?, 0x7042041950?})
          /build/.cache/tailscale-go/src/runtime/panic.go:860 +0x12c
      created by github.com/tailscale/tailscale-android/libtailscale.newTUNDevices in goroutine 40
          /build/tailscale-android/libtailscale/multitun.go:99 +0x20c

  09-21 13:35:56.379 E Go  : panic: close of closed channel [recovered, repanicked]
  09-21 13:35:56.384 F libc: Fatal signal 6 (SIGABRT), code -6 (SI_TKILL)
                             in tid 31023 (Thread-21), pid 30757 (m.tailscale.ipn)
  09-21 13:35:58.345 I Zygote: Process 30757 exited due to signal 6 (Aborted)

Signed-off-by: Gonzalo <gonza.ar.cr@gmail.com>
@gonzaarcr gonzaarcr changed the title Fix stale connection and panic Android: Fix stale connection and panic Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant