Skip to content

feat(HNT-2115): add redis-state client package - #18

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-extract-pubsub-packagefrom
claude/hnt-2086-redis-state-package
Draft

feat(HNT-2115): add redis-state client package#18
mmiermans wants to merge 1 commit into
claude/hnt-2086-extract-pubsub-packagefrom
claude/hnt-2086-redis-state-package

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Goal

HNT-2115

Add a generic redis-state package: a module-level ioredis client that exposes fetch and enqueue timestamps, string values, ownership-checked locks, and an atomic token-bucket primitive. This is the state the crawler needs for deduplication and for a shared rate limit, and it lives in its own package because none of it carries any crawl domain.

See docs/ARCHITECTURE.md in #14 for where the state store sits in the system.

Implementation decisions

Decision Approach Why
Use ioredis as the Redis client A module-level singleton wraps ioredis, mirroring the shape of the Zyte and Pub/Sub clients The state store needs atomic Lua scripting through eval, lazy connect, and automatic reconnection, all of which ioredis provides directly. It is also the Redis client already used elsewhere in the Mozilla org, in fxa and blurts-server.
Release a lock only when the caller still owns it A Lua script compares the stored token to the caller's token before deleting, running atomically inside Redis A worker must never free a lock that already expired and was re-acquired by another worker, so the get and the delete run as one script and cannot interleave. This matches the idempotent, lock-guarded worker design in the Article Crawler tech spec, where the lock TTL is the Pub/Sub ack deadline minus a margin.
Measure the token bucket against the Redis server clock The token-bucket Lua script reads Redis TIME for elapsed time and runs atomically Worker replicas then agree on refill without depending on synchronized client clocks, and concurrent callers cannot over-draw the bucket. This is the atomic primitive behind the distributed Zyte rate limiter the tech spec plans.
Ship the token-bucket primitive in this generic package rather than the rate-limiter feature The package exposes acquireRateLimitToken as a plain Redis operation, and the crawler wiring lands later in the distributed Zyte rate limiter slice It is a general Redis operation with no crawl domain, so it belongs with the client. Keeping the package free of crawler specifics lets the worker gate wire it later without the package knowing the crawler.
Fail a stalled Redis command fast rather than let it hang Set a short commandTimeout and cap retries per request A hung command blocks the handler, holds the Pub/Sub lease, and starves throughput. A rejected command throws, the handler nacks, and Pub/Sub redelivers, which is the safe failure the tech spec expects when Redis is in trouble.

@mmiermans mmiermans changed the title feat(HNT-2086): add redis-state client package feat(HNT-2115): add redis-state client package Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant