Skip to content

feat(HNT-2086): add the distributed Zyte rate limiter - #33

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-discovery-consumerfrom
claude/hnt-2086-zyte-rate-limit
Draft

feat(HNT-2086): add the distributed Zyte rate limiter#33
mmiermans wants to merge 1 commit into
claude/hnt-2086-discovery-consumerfrom
claude/hnt-2086-zyte-rate-limit

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2086

Add the distributed Zyte rate limiter that gates every Zyte call against a shared per-role token bucket in Redis, so both worker roles stay within the Zyte account limit and shed load instead of pinning a worker when the account is saturated. This builds on the redis-state slice by using its generic token-bucket primitive acquireRateLimitToken, and on the worker-config slice by using its split of the per-account rate between the two roles. See docs/ARCHITECTURE.md in #14 for where rate limiting sits in the worker Zyte flow.

Implementation decisions

Decision Approach Why
Keep the token bucket in redis-state and add only the crawler-specific gate here awaitZyteToken loops over acquireRateLimitToken, which owns the atomic refill-and-take Lua script in the generic redis-state package The token bucket is a generic Redis operation with no crawl knowledge, so it belongs in the shared package while this slice holds only the Zyte wiring, following the split established in the redis-state slice.
Give each worker role its own bucket, with the two per-role rates summing to the account limit The key is zyte:rate-limit:${workerRole}, and the worker-config slice sets the two default rates so they add up to the Zyte account limit The article worker fetches one article per job while the discovery worker lists many per page, so a single shared bucket would let one role starve the other. A distributed Redis rate limiter (HNT-2441) shares one token bucket across replicas. Redis is per-environment, so the key needs no env prefix.
Throw once the max wait elapses rather than block indefinitely The loop waits as the bucket refills and throws when the deadline passes The throw lets the subscriber nack so Pub/Sub redelivers the job later, shedding load when Zyte is saturated instead of holding a worker. Redelivery is safe because the handlers are idempotent, which at-least-once delivery requires.
Default the burst to one minute of tokens When ZYTE_RATE_LIMIT_BURST is 0 the burst falls back to the per-minute rate Absent an explicit burst, allowing one minute of accumulated tokens smooths short spikes without letting the bucket build up without bound.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-discovery-consumer branch from 608d3d1 to a76ed67 Compare July 6, 2026 17:08
@mmiermans
mmiermans force-pushed the claude/hnt-2086-zyte-rate-limit branch from 8de019d to e5140b5 Compare July 6, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant