feat: request throttling — bounded concurrency, jitter, and per-provider pacing #48

Summary

Add per-provider request pacing and bounded concurrency. Retry-on-429 (#37) handles a rate limit after it happens; this is about not tripping it in the first place, and about keeping one agent's failure from taking down a tick.

Research/context only; design still to be done.

Problem

llm:chat-async fires one Future per call onto ExecutionContext.global with no cap (LLMExtension.scala:64, :479-509). ask turtles [ ... llm:chat-async ... ] with N turtles issues N simultaneous HTTP requests. There is no semaphore, no queue, and no per-provider connection bound.

Current retry (BaseHttpProvider.executeWithRetry, added in 2d083b6) covers 429 only — 5xx, timeouts, and connection errors fail immediately. Backoff is deterministic (1s/2s/4s) with no jitter, so a burst of rate-limited agents retries in lockstep.

Prior art: mesa-llm hit exactly this wall

mesa-llm has the same unbounded fan-out shape, and their own issue #200 reports the outcome:

"Mesa-LLM suffers from critical performance bottlenecks that make it unsuitable for large-scale agent simulations... resulting in exponential performance degradation beyond ~10 agents."

Reported: 20 agents ≈ 3 min/step, 50 agents ≈ 15+ min/step.

Verified from their source (local clone, HEAD 57a51ff):

• No concurrency limiting anywhere — grep for Semaphore|throttle|rate_limit_delay across their package returns zero hits.
• Bare asyncio.gather at mesa_llm/parallel_stepping.py:31 and :124, and mesa_llm/tools/tool_manager.py:388 — none pass return_exceptions=True. One agent raising kills the entire tick. That is their separate issue #220.
• No response caching — grep for lru_cache|functools.cache|redis|memoize returns zero hits.

Their contributing factor we don't share is 2 LLM calls per agent per step. But the unbounded fan-out is identical, and it is the part that turns a slow tick into a failing one.

Design requirements this suggests

1. Bounded concurrency — a configurable cap on in-flight requests, ideally per provider (limits differ a lot between e.g. a Gemini free tier and a paid OpenAI key).
2. Jitter on backoff — deterministic backoff across many agents produces a thundering herd. Randomized backoff is a small change to executeWithRetry.
3. Per-agent failure isolation — one agent's failed call should not abort the tick or poison other agents' futures.
4. Broaden retry — 5xx and timeouts, not just 429.
5. Optional pacing — a minimum interval between requests to a given provider, for free tiers with requests-per-minute limits.

Open questions

• Config surface: a max_concurrent_requests key, per-provider descriptor defaults, or both?
• Should pacing be automatic from a known per-provider RPM limit, or purely modeler-set?
• What does the modeler see when requests are queued — nothing, or a monitor-able count?
• Interaction with timeout_seconds: does queue wait count against the timeout budget? (It should not.)

Related

• Add retry with backoff for rate limit errors (HTTP 429) #37 — retry with backoff (merged; this is the proactive counterpart)
• feat: Add structured output / JSON mode support across all LLM providers #22 — structured output