Skip to content

Bound the rate limiter's Redis wait with a short timer #211

Description

@HMarzban

Problem

A slow Redis holds every rate-limited REST route. The limiter is mounted on *, and only OPTIONS and /health are skipped (apps/hocuspocus.server/src/middleware/index.ts:206, built once at apps/hocuspocus.server/src/middleware/index.ts:201). Inside it, the limiter awaits Redis with no bound of its own (apps/hocuspocus.server/src/middleware/index.ts:56).

A broken Redis is already handled. A store fault rejects, and the catch arm logs and lets the request through (apps/hocuspocus.server/src/middleware/index.ts:72-73). There is no insurance limiter behind it either (apps/hocuspocus.server/src/middleware/index.ts:34). Slow is the case nothing covers.

The only bound left is the Redis client command timeout. The limiter uses the shared client, so the value flows through three hops:

  • REDIS_COMMAND_TIMEOUT: numericString('60000'), (apps/hocuspocus.server/src/config/env.schema.ts:67)
  • commandTimeout: env.REDIS_COMMAND_TIMEOUT, (apps/hocuspocus.server/src/config/env.ts:49)
  • commandTimeout: config.redis.commandTimeout, (apps/hocuspocus.server/src/lib/redis.ts:77)

So a slow Redis can hold one request for up to REDIS_COMMAND_TIMEOUT, which defaults to 60000 ms.

What to do

Race limiter.consume(ip, 1) against a short timer inside the middleware. When the timer wins, log it and call next(), the same way the store-fault arm does.

Do not lower REDIS_COMMAND_TIMEOUT. The queue module already states the reason in code: "Never lower the shared 60s REDIS_COMMAND_TIMEOUT: it would race the workers' blocking bzpopmin" (apps/hocuspocus.server/src/lib/queue.ts:61-62, one comment wrapped over two lines).

Acceptance

  • With redis-cli DEBUG SLEEP 20 running, a rate-limited route answers in under a second.
  • With Redis healthy, the limit still returns 429 after the budget is spent.

Notes

grep -nic "timeout" apps/hocuspocus.server/src/middleware/index.ts returns 0. The middleware holds no timer today.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions