sdk/js: the probe's JS programs hang silently in-cluster for 120s, intermittently, on both testnets #732

The JS SDK probe (agentmesh-sdk-probe-* cronjob, every 15 min) fails intermittently on both testnets since the night of 2026-10-15/16, always the same way: the JS program prints nothing for 120 s and is killed by the probe's timeout. The Python SDK in the same pod, the same minute, against the same control plane and routers, passes. The Go cold-path probe (agentmesh-probe-*) passed every run in the same windows.

Observed

Two shapes, both a silent hang of the JS process:

• {"probe":"agentmesh-sdk-js","ok":false,"stage":"node-mcp","error":"no answer with 'Echo: agentmesh-sdk-probe' in 120s (1 attempts; last: (hung for 120s))"} — node build/examples/call.js mcp://everything echo … produced no output at all, so it never reached the point where it prints mcp://everything is served by <peer> (enroll → join → directory lookup all happen before that line). In the 04:37 and 05:07 bananas runs the js-agent sidecar in the same pod did join (accepting a2a://agent as …), so joining itself worked for one JS process while the other hung.
• {"probe":"agentmesh-sdk-python","ok":false,"stage":"agent","error":"no agent announced itself in /var/run/agentmesh/js-agent.log within 120s"} — the a2a-agent.js sidecar printed nothing in 120 s (its first line is after enroll and join), so the Python half of the probe had no agent to call; the Python SDK's own enrollment and join were fine (python-agent announced).

Ruled out

• Keys: both testnets rotated their signing key hours before (bananas 21:05Z by the new scheduler, hub 22:09Z by the 24 h ticker). No control plane 401 and no router no valid key for any peer other than one stale developer node whose credential predates both keys; the Python SDK verifies with the same /keys. Hub has rotated daily for a week with this probe passing.
• Routers and the provider: no WARN/ERROR on any router or on the everything nodes in the failure windows; connected ≈ authenticated; no refusals.
• The SDK build: call.js built from 03a374c, run from a workstation against https://bananas.sam-mesh.dev with a bootstrap token: 3/3 passes in ~10 s each. In-cluster debug runs of the same cronjob (see below): 3/3 passes so far.
• The GKE node: every probe pod, pass or fail, on both testnets, ran on the same node (gke-cluster-sam-mesh-default-pool-21713d03-130d).
• Control plane load: p99 ≤ 25 ms, no scrape failures, two replicas healthy.

What is different in the failing path

In-cluster only (workstation runs pass), JS only (Python passes in the same pod), intermittent (~40–60 % in the windows), and silent: the hung process never prints, so neither the probe nor the SDK says which await it is in. Candidates, none confirmed: a dial from inside the cluster to a router's public address that neither completes nor fails and a step in the JS enroll/join/lookup path without a timeout (the control plane client has a 30 s per-request timeout; the libp2p dial has its default; a directory lookup stream may not); the enrollment retry budget (3 min) on an unexpected 429; something in the k8s-SA-JWT enrollment path that the bootstrap-token path from the workstation does not exercise.

To continue

• /tmp/mj/sdk-debug-loop.sh on the operator workstation runs the bananas cronjob's pod spec as a Job with DEBUG=libp2p:*,-libp2p:*:trace,agentmesh* and NODE_DEBUG=http,net,tls on the js and js-agent containers, keeps every container's log under /tmp/mj/sdkdbg-<run>-<container>.log, and reports when a run fails; the manifest is /tmp/mj/sdk-probe-debug.json. A caught failure says which await hangs.
• Instrumentation to add regardless of the cause: the examples the probe runs should print each stage to stderr (enrolled <peer>, joined via <router>, provider <peer>, dialing), so a 120 s hang is readable from the probe's own output; and the SDK's join and directory lookup should have a bounded timeout with an error naming the router.
• The two Testnet Health runs during the windows reported success, because the health workflow reads the newest Go probe and not the SDK probe's job history; the SDK probe's failures should fail the health check too.

Not a release blocker in itself: the Go node and the Python SDK are unaffected, the failures are confined to the JS SDK's in-cluster path and each one is a transient of one run. Filed so the investigation can be resumed from here.