Adaptive per-provider concurrency limiter (HAL-1372)

[hallelx2]

Closes HAL-1372 (llmgate half).

middleware/limit: an AIMD semaphore per provider that wraps both Client and Judge. Halves on a 429 or transport failure and honours Retry-After as a pause; widens by one after a window of successes; auth and bad-request errors leave it alone; every change is an OnChange event. Sits inside retry so each attempt takes a slot.

Test: a mock provider that admits 4 concurrent calls and 429s the rest — 200 requests from an initial limit of 16 all complete under retry, and the limit settles near 4. Race detector clean.

Engine wiring (both binaries, tocdump and navbench in parallel through it, judgewalk page batches concurrent) follows in vectorless-engine.

Summary by Sourcery

Introduce adaptive per-provider concurrency control for client and judge requests, with retry-aware slot acquisition and operational visibility into capacity changes.

New Features:

• Add an adaptive per-provider concurrency limiter that adjusts capacity based on rate-limit and transient failures or sustained success.
• Support concurrency limiting for both chat completions and judge batches, including provider-requested pauses via Retry-After and observable change events.

Enhancements:

• Preserve client capabilities and local token counting through the concurrency wrapper.
• Document middleware ordering so the limiter operates per retry attempt.

Documentation:

• Document the adaptive limiter and its recommended placement relative to retry middleware.

Tests:

• Add coverage for convergence on provider capacity, Retry-After pauses, bounds, non-capacity errors, context cancellation, and judge concurrency.

Summary by CodeRabbit

• New Features
  • Added adaptive concurrency limiting for client and judge requests.
  • Automatically adjusts concurrency based on request success, rate limits, and transient failures.
  • Honors provider-requested retry delays and supports cancellation while waiting.
  • Added runtime limit and activity statistics, with notifications when limits change.
• Bug Fixes
  • Prevents excess concurrent requests from overwhelming providers while allowing capacity to recover after successful requests.

[sourcery-ai]

Sorry @hallelx2, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 6 days by commenting @sourcery-ai review. Upgrade to get a review now.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: e9a694f5-b052-46b9-9de1-607040f6243d

📥 Commits

Reviewing files that changed from the base of the PR and between d9d713a and b2ec964.

📒 Files selected for processing (2)

• middleware/limit/limit.go
• middleware/limit/limit_test.go

 _____________________________________________________________
< Please stop using global state like it's a communal fridge. >
 -------------------------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

Introduces a shared adaptive AIMD semaphore for provider chat and judge operations, positioned inside retry so each attempt is capacity-controlled; it reacts to rate limits and transient failures, honors Retry-After pauses, reports changes and stats, and is covered by concurrency and behavior tests.

Sequence diagram for adaptive per-provider request limiting

sequenceDiagram
    participant Caller
    participant Retry
    participant Limiter
    participant Provider
    participant OnChange

    Caller->>Retry: Complete or Judge
    Retry->>Limiter: Acquire(ctx)
    Limiter-->>Retry: release
    Retry->>Provider: Complete or Judge
    Provider-->>Retry: response or error
    Retry->>Limiter: release(err)
    alt rate limit or transient failure
        Limiter->>OnChange: Event(rate_limited or transient)
        Limiter-->>Retry: narrow limit and honor Retry-After
        Retry->>Retry: retry attempt
    else success window reached
        Limiter->>OnChange: Event(success)
        Limiter-->>Retry: widen limit by one
    end

Loading

File-Level Changes

Change	Details	Files
Add an adaptive per-provider AIMD concurrency limiter with pause handling and observability.	Normalize limiter configuration with initial, floor, ceiling, success-window, decrease-factor, and injectable clock defaults. Implement context-aware slot acquisition, in-flight/waiter tracking, provider-requested pauses, and idempotent release handling. Halve capacity for rate-limit or transient transport failures, honor Retry-After, increase capacity after consecutive successes, and emit OnChange events. Expose current limit and runtime statistics.	middleware/limit/limit.go
Integrate concurrency control into both chat and judge provider calls.	Wrap Complete calls so every retry attempt acquires and releases a slot. Leave local CountTokens unthrottled and preserve delegated capabilities. Wrap each Judge batch with the shared limiter.	middleware/limit/limit.go
Validate convergence, failure classification, pause behavior, wrappers, and synchronization under representative load.	Exercise 200 concurrent requests against a provider capacity of four through retry and verify no losses and convergence near capacity. Test multiplicative decrease, Retry-After pauses, success-window growth, floor/ceiling bounds, and non-capacity errors. Verify context cancellation while waiting, judge serialization, and race-safe shared state.	middleware/limit/limit_test.go

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.