Problem
grpc-proxy closes worker tunnel connections for several ordinary reasons: the worker connection cache TTL expiring, the connection going inactive, capacity eviction, and shutdown drain.
It already computes which reason applied, via mapEvictionReason, on every single close. That value is then written to a debug-level log and discarded:
cache.OnEviction(func(ctx, reason, i) {
reasonStr := mapEvictionReason(reason) // ttl_expired|deleted|capacity_reached|unknown
zap.L().Debug("worker connection cache eviction triggered",
zap.String("eviction_reason", reasonStr), ...)
_ = wc.Close()
})No metric. No span. Debug level, so effectively unavailable in production.
Impact
"Why did the tunnel drop?" cannot be answered from production data. It can only be inferred by reading source, or by enabling debug logging on a specific pod during a live reproduction.
There is also no telemetry of any kind on the proxy-to-worker leg. The existing metrics cover client connections and NATS, so an entire hop is unobserved, and it is the hop where connection-lifecycle failures occur.
This has been a concrete blocker on at least one long-running streaming-disconnect investigation, where the question "is the proxy dropping these connections, and if so why" stayed open for weeks despite the service knowing the answer every time it happened.
Proposal
Emit the reason that is already computed:
- counter labelled by reason, bounded to four values, pre-initialised to zero
- gauge for currently-open worker tunnels
- histogram of tunnel duration, with buckets straddling the timers that can close one, so a timer kill shows as a cliff
- promote the existing log line from debug to info, keeping the message text unchanged so existing searches keep working
- span carrying request id, reason and duration, for correlation in tracing
No behaviour change; this is observability only.
Problem
grpc-proxy closes worker tunnel connections for several ordinary reasons: the worker connection cache TTL expiring, the connection going inactive, capacity eviction, and shutdown drain.
It already computes which reason applied, via
mapEvictionReason, on every single close. That value is then written to a debug-level log and discarded:No metric. No span. Debug level, so effectively unavailable in production.
Impact
"Why did the tunnel drop?" cannot be answered from production data. It can only be inferred by reading source, or by enabling debug logging on a specific pod during a live reproduction.
There is also no telemetry of any kind on the proxy-to-worker leg. The existing metrics cover client connections and NATS, so an entire hop is unobserved, and it is the hop where connection-lifecycle failures occur.
This has been a concrete blocker on at least one long-running streaming-disconnect investigation, where the question "is the proxy dropping these connections, and if so why" stayed open for weeks despite the service knowing the answer every time it happened.
Proposal
Emit the reason that is already computed:
No behaviour change; this is observability only.