Skip to content

Rate limit the job submission request - #866

Merged
hellolittlej merged 3 commits into
masterfrom
rate-limit-job-submission
Aug 13, 2026
Merged

Rate limit the job submission request#866
hellolittlej merged 3 commits into
masterfrom
rate-limit-job-submission

Conversation

@hellolittlej

@hellolittlej hellolittlej commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

What

Adds an opt-in, per route based bucket rate limit.For jobsRoute, add the rate limit in front of the three v1 job-creation endpoints:

  • POST /api/v1/jobs
  • POST /api/v1/jobClusters/{cluster}/jobs
  • POST /api/v1/jobs/actions/quickSubmit

The rate

Configuration

Property Default Notes
mantis.master.api.v1.submitJob.throttle.enabled false Off by default; no limiter is constructed when off
mantis.master.api.v1.submitJob.permitsPerSecond 1000 Re-readable at runtime, no deploy needed
  • Guava RateLimiter, non-blocking tryAcquire(); token-bucket with the usual ~1s of burst accumulation, so a 1000/s limit admits up to a ~1000-request burst after an idle period.
  • Shed → HTTP 429, checked in BaseRoute.withThrottle:141 before the entity is unmarshalled — a storm costs a permit check, not a parse plus an actor ask.

Off by default because the right ceiling depends on cluster size and traffic —
operators opt in once they've measured their own submit rate. The rate is backed by a
LongDynamicProperty, so it re-reads on the dynamic-property refresh interval
(mantis.config.dynamic.refreshSecs, default 30s).

Metrics

MasterApiMetrics/throttledRequestCount gains a source tag (submitJob,
resourceCluster), so an operator can tell which limit is shedding. Previously the
route-level throttle and the resource-cluster gateway's RequestThrottledException
both incremented the same untagged counter.

Checklist

  • ./gradlew build compiles code correctly
  • Added new tests where applicable
  • ./gradlew test passes all tests
  • Extended README or added javadocs where applicable

@hellolittlej hellolittlej changed the title Rate limit the job submission request [TESTING] Rate limit the job submission request Jul 30, 2026
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Test Results

823 tests  +26   812 ✅ +26   10m 17s ⏱️ -1s
167 suites + 3    11 💤 ± 0 
167 files   + 3     0 ❌ ± 0 

Results for commit 29a251e. ± Comparison against base commit 25ff35c.

♻️ This comment has been updated with latest results.

@hellolittlej
hellolittlej force-pushed the rate-limit-job-submission branch 2 times, most recently from 9cfd8a5 to a8c0223 Compare August 4, 2026 20:30
@hellolittlej hellolittlej changed the title [TESTING] Rate limit the job submission request Rate limit the job submission request Aug 4, 2026
@hellolittlej
hellolittlej force-pushed the rate-limit-job-submission branch 3 times, most recently from ca6bbe8 to 8d0199a Compare August 5, 2026 06:43
@hellolittlej
hellolittlej force-pushed the rate-limit-job-submission branch from 8d0199a to 168d7d6 Compare August 11, 2026 18:06
@hellolittlej
hellolittlej force-pushed the rate-limit-job-submission branch from c4eeb49 to 377e8db Compare August 12, 2026 21:40
@hellolittlej
hellolittlej force-pushed the rate-limit-job-submission branch from 377e8db to c724401 Compare August 12, 2026 22:15
@hellolittlej
hellolittlej merged commit 9ff3102 into master Aug 13, 2026
7 checks passed
@hellolittlej
hellolittlej deleted the rate-limit-job-submission branch August 13, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants