Artery ActorSystem termination slow for Aeron-UDP transport due to stacked liveness timeouts

[He-Pin]

Description

When shutting down an ActorSystem using Artery with Aeron-UDP transport, termination can take excessively long due to stacked Aeron liveness timeouts and the embedded media driver drain sequence. This is most pronounced in virtualized environments.

Evidence from Pekko CI

The issue has been acknowledged through test timeout widening commits:

• Commit 4d17e08c5b: widened shutdownAll await for aeron-udp drain on JDK 25 nightly. The commit message states: "aeron-udp transport is the slowest to drain (embedded media driver + stacked Aeron liveness timeouts)"
• Commit 3fcbea76ae: widened TCK actor-system-terminate phase timeout because "the inner CoordinatedShutdown actor-system-terminate phase keeps its default 10s, which is occasionally too tight"

Both commits show CoordinatedShutdown phase actor-system-terminate timing out with ActorGraphInterpreter and stream actors still mid-shutdown.

Root cause

The shutdown flow involves multiple sequential waits that accumulate:

1. FlushOnShutdown — waits for all associations to acknowledge termination (up to shutdown-flush-timeout = 1s)
2. ArteryTransport.shutdown() — internalShutdown() waits for streamsCompleted (all outbound/inbound stream completions)
3. shutdownHook — blocking Await.result with up to DriverTimeout + 3.seconds (23s total)
4. Aeron liveness timeouts stack up:
   • client-liveness-timeout = 20s
   • publication-unblock-timeout = 40s
   • image-liveness-timeout = 10s
   • driver-timeout = 20s
5. give-up-message-after = 60s — messages can be retried for up to 60 seconds

Impact

• TCP transport is not affected
• Healthy shutdowns complete in under a second
• The issue manifests with Aeron-UDP in virtualized environments or with embedded media drivers
• Test CI intermittently times out, requiring repeated timeout widening

Current mitigation

All timeouts are configurable via reference.conf, but the default stacking of Aeron liveness timeouts makes the worst-case shutdown duration unnecessarily long.

Suggested approach

• Decouple the actor-system-terminate phase from Aeron liveness timeouts (force-drain with a shorter deadline during shutdown)
• Add a dedicated artery-shutdown-timeout that bounds the total Artery drain time independently of individual Aeron liveness settings