Broker spawn treats Relaycast database_overloaded preregistration as fatal after one attempt

[khaliqgant]

Problem

Agent Relay v11.10.4 cannot start a new broker worker when Relaycast agent registration returns database_overloaded. The local broker is healthy and five existing agents remain deliverable, but POST /api/spawn fails before process creation.

Clean reproduction

Environment: running chief-broker, Agent Relay v11.10.4, headless task-exit OpenCode worker, no Daytona sandbox.

Observed across independent paths on 2026-09-09:

• Direct agent-relay node agent spawn timed out; broker roster confirmed no process.
• Direct broker POST /api/spawn returned failure: failed to pre-register worker; HTTP 503; code database_overloaded; attempts: 1.
• Three further bounded broker retries returned the same typed 503; a fourth returned 500 internal_error.
• Eight explicit agent-relay agent register attempts also failed; no worker token or process was created.
• Relaycast incident evidence is tracked at GET /v1/agents latency varies 5x on an identical payload (5.3s-28.7s), and intermittently fails relaycast#389.

Contract mismatch

The v11.10.4 source documents retry_agent_registration as up to three retries for transient errors. The HTTP spawn path also has a RetryableExhausted branch intended to continue spawning without preregistration while emitting a warning. Actual database_overloaded responses report attempts: 1 and take the fatal branch, so a transient control-plane write outage prevents even a headless local task from starting.

Expected

• Classify database_overloaded and equivalent Retry-After 503 registration failures as retryable.
• Honor Retry-After within a bounded total deadline and perform the documented retry count.
• When retries exhaust and the requested task can safely run without Relay messaging, continue the local spawn with an explicit preregistration warning, as the source path already intends.
• Never report success unless the worker process is actually present; preserve clean task-exit cleanup.
• Add a regression covering node agent.register failure followed by HTTP registration database_overloaded and proving the fallback branch is reached rather than fatal after one attempt.

This blocks using cheap OpenCode workers to supervise the current cleanroom/Fleet qualification campaign.

[khaliqgant]

Additional live targeted-Fleet reproduction (2026-09-09 23:36–23:41 UTC, Agent Relay 11.10.4):

• finn-mini-node was online/live, handlersLive=true, advertising spawn:claude, with continuous heartbeats.
• agent-relay fleet spawn claude --node finn-mini-node --name fleet-proof-audit-sonnet-0910 --model claude-sonnet-4-6 ... produced no launch confirmation or terminal error for more than three minutes (past the documented 120000ms confirmation window).
• The exact name never appeared in agent-relay agent list and never appeared in the node's relay:live-agents:v1 metadata.
• The client was interrupted after the bounded observation. Exact cleanup via agent-relay fleet release fleet-proof-audit-sonnet-0910 returned not found.

This is a second direct proof that an online target node can accept no observable worker and the Fleet spawn client can outlive its advertised confirmation timeout. It may share the preregistration overload path described here, but the absence of a surfaced typed error is itself part of the defect. No remote agent/process remained to release.

[khaliqgant]

Fresh production Fleet evidence on 2026-09-10 using Agent Relay 11.10.4 from the workspace-pinned ../chief broker:

• fleet nodes --all reported chief-broker, sf-mini, and finn-mini-node online/live with spawn capacity.
• Targeted OpenCode spawn to chief-broker failed: target node chief-broker is not placement-ready; placement failed fast.
• Targeted spawn to listed-live sf-mini failed: Node not found.
• Targeted spawn to finn-mini-node was accepted as invocation inv_223604578685480960 but never confirmed within 30s; launcher cleanup then failed with The database is temporarily overloaded.
• The exact worker relay-surface-audit-mini-0910b appeared offline in the workspace roster. Exact release invocation inv_223605433137307648 was dispatched to finn-mini-node.

This is direct pre-fix evidence for the placement/confirmation/cleanup failure family. PR #1730 head 82a7cd5e68eaecdb8f5fb2c038e35b98a9159642 now includes the trusted 1715-spawn-overload proof case and the bounded retry/safe fallback changes; production behavior remains unproven until that candidate is installed and this exact matrix is rerun.

[khaliqgant]

Fresh exact reproduction from Relay PR #1739 proof run 34509743945 (2026-09-10):

• Cloud RelayFlow run: 3f2b7a4f-9860-4332-8bd9-b8405f1ee314
• step sandbox: b856de72-f591-4aaa-9aaf-58e664942f59
• registration attempt 1 failed with database_overloaded and retried after 8011 ms
• attempt 2 failed and retried after 16223 ms
• attempt 3 remained terminal, so prove-base failed before any task work
• surfaced error says mcp-args --register failed after 3 attempts, while the nested Relaycast detail still reports attempts: 1

This shows bounded retries are present on the current Cloud/RelayFlow path but are insufficient during a sustained overload window, and the inner attempt count remains misleading. The proof run is correctly red; do not retry-until-green and call it product proof.