choiyounggi · choiyounggi · Aug 5, 2026
@@ -12,9 +12,9 @@ follow the cross-pointers in their index or take the next matching seeded domain
 | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior |
 | [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
 | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility |
-| [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) |
+| [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting), orchestrating parallel agent worker sessions (pane delivery, completion gates, worktree-isolated briefs) |
 | [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) |
-| [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing, automated verification of document deliverables (spec/RFC gates) (writing automated test code → testing) |
+| [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, call-site/consumer enumeration for a signature or contract change, bug reports, severity/priority triage, exploratory testing, automated verification of document deliverables (spec/RFC gates) (writing automated test code → testing) |
 | [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) |
 | [security](wiki/security/index.md) | **seeded** | Trust-boundary decisions: input validation, session-vs-token auth choice, per-resource authorization (IDOR), secrets hygiene, dependency trust, PII handling (XSS rendering → frontend; CI secrets → infrastructure; JWT implementation → backend/frontend auth) |
 | [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, Unicode normalization in text/file-name matching, commands inspected before execution, background services/cron, invoking prompt-capable CLIs non-interactively, toolchain version pinning |

@@ -37,3 +37,6 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-08-03] ingest | Consolidated review of knowledge PRs #6–#13 (8 fork PRs) into 12 pages. New: backend/common/llm (completion-response-validation, context-window-budget), backend/common/integrations (externally-owned-defaults), backend/common/storage (object-key-persistence), infrastructure/containers/host-cgroup-visibility, infrastructure/observability/missing-container-metrics, platforms/environment/unicode-text-matching, platforms/shells/command-text-inspected-before-execution, platforms/processes/non-interactive-cli-invocation, qa/document-verification (spec-document-gates, editing-a-gated-document), testing/quality (checks-that-cannot-pass, spec-artifact-checks, harness-reverse-controls). All cited URLs are per-PR live-verified; three pages were reconciled from two overlapping PR versions each (see revise/dedup entries below).
 ## [2026-08-03] revise | Reconciled 3 pages from overlapping PR pairs, taking the more complete/better-sourced body and folding in the other's unique cases: backend/common/llm/completion-response-validation (#12 body — tool_calls/function_call carve-out, streaming, Responses API status==incomplete, "reasoning is scratch, not deliverable" — kept in llm/ per #6/#13 category, folded in #6's DeepSeek-first-party edge + the 8,173-char reasoning_content field incident); backend/common/integrations/externally-owned-defaults (#12 generalized body — any repo-external resource — folded in #6's LiteLLM-alias-removed field incident + gateway-config-vs-live-upstream nuance); platforms/processes/non-interactive-cli-invocation (#12 body — GNU-nohup extension precision, ssh -n stdin-detach vs BatchMode, pre-log DNS/TLS/proxy + curl -v — folded in #11's DEBIAN_FRONTEND, pager/color TTY case, wrapper-CLI case, and the no-request-in-gateway-log field incident).
 ## [2026-08-03] dedup | Dropped 3 candidate pages as duplicates/superseded during the #6–#13 consolidation: testing/docs-as-spec/document-conformance-checks (#9 — same case as testing/quality/spec-artifact-checks from #8: coverage-vs-validity split, per-check negative controls, GFM pipe parsing; #8 kept as canonical, docs-as-spec category not created); backend/common/llm/gateway-model-alias-defaults (#6 — subsumed by the generalized integrations/externally-owned-defaults; the model-alias case is one instance); backend/common/integrations/llm-response-completeness (#12 — folded into llm/completion-response-validation, kept in llm/ for category coherence with context-window-budget).
+## [2026-08-05] ingest | knowledge-flush of 9 queued insights → 9 pages across 5 domains. NEW category infrastructure/agent-orchestration (+3): pane-delivery-confirmation (tty ECHO makes a capture-pane diff worthless as delivery evidence — check the target's busy indicator first; reproduced on tmux 3.7b), session-completion-gates (a Stop-gate's terminal set must include instructed mid-workflow pauses, self-limited via stop_hook_active; reproduced against loop-gate.sh:55 vs session-prompt.md:20 in this repo), worktree-isolated-workers (worktree-relative output paths; the escape guardrail is write-only — reads pass). platforms/shells +3: exit-status-vs-diagnostics (warnings are exit 0 — capture stderr with `2>&1 >/dev/null`, order matters; clang repro), option-like-argument-values (`--` before any interpolated operand; POSIX Guideline 10 + tmux repro), env-var-off-switches (`${VAR:-d}` swallows an empty value — use `${VAR-d}` or a sentinel; POSIX 2.6.2). backend/common/reliability +1: client-side-rate-limiting (auth refresh bypasses a per-method throttle; kis_client.py 82a077e). testing/quality +1: write-path-assertions (assert the persisted row, not the status; httpx form-encodes only a Mapping — list-of-tuples is sent as raw content, verified in httpx/_content.py). qa/process +1: call-site-enumeration (search by callee, not by parameter name — positional callers and test helpers hide from a keyword grep).
+## [2026-08-05] revise | Reciprocal related-links + one merge row for the 2026-08-05 ingest: portable-shell-scripts → the 3 new shells pages; checks-that-cannot-pass → exit-status-vs-diagnostics; tests-that-cannot-fail → write-path-assertions (plus a new never-fails row: "HTTP test of a write endpoint asserting only the response status"); regression-scope → call-site-enumeration; timeouts-and-retries → client-side-rate-limiting; environment-config → env-var-off-switches; non-interactive-cli-invocation → pane-delivery-confirmation.
+## [2026-08-05] contradiction | A queued insight asserted the groundwork `worktree_escape` guardrail blocks READS (`ls`, `cat`) out of the main checkout as well as writes. Refuted by direct measurement against guardrails 1.0.0 `hooks/bash-guard.sh`: the rule fires only on a write verb (rm|mv|cp|tee|mkdir|touch|install|dd) or a redirect to an absolute path — `cat`, `ls`, and `grep` on main-checkout paths from a linked worktree all pass. infrastructure/agent-orchestration/worktree-isolated-workers.md documents the verified write-only behavior; the directive (worktree-relative output paths) survives, its stated mechanism did not.
@@ -0,0 +1,85 @@
+---
+id: backend-common-reliability-client-side-rate-limiting
+domain: backend
+category: reliability
+applies_to: [general]
+confidence: field-tested
+sources:
+  - https://developer.okta.com/docs/reference/rl2-token-oauth/
+  - https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api
+last_verified: 2026-08-05
+related: [backend-common-reliability-timeouts-and-retries, backend-common-auth-jwt-server-side, debugging-concurrency-intermittent-failures]
+---
+
+# A Client-Side Throttle That the Auth Refresh Slips Past
+
+## When this applies
+
+Your API client wrapper enforces a provider's requests-per-second cap itself
+(a minimum interval or token bucket around each call), and a rate-limit error
+still comes back — characteristically on the **first** call of a process, or on
+some days and not others.
+
+## Do this
+
+1. **Put the throttle where every outbound request passes, including the ones
+   the client issues on its own behalf.** Token/credential refresh is issued from
+   inside a header builder or an interceptor *below* the wrapper method, so a
+   throttle applied per public method never sees it. Either wrap the transport
+   (session/adapter/middleware) or call the throttle explicitly from the refresh
+   path as well.
+2. **Stamp the throttle's timestamp immediately before the request goes out**,
+   inside the same function that issues it. A timestamp written by the caller
+   before an inner request happens leaves that inner request unaccounted for and
+   lets the next one land in the same second.
+3. **Make the refresh and the call it enables two separate slots.** After a
+   refresh has consumed a slot, the request that needed the token waits its own
+   interval:
+
+   ```python
+   def _headers(self, tr_id):
+       token = self._get_token()   # throttles internally when it must refresh
+       self._throttle()            # separates this call from that refresh
+       return {...}
+   ```
+
+4. **Read the provider's docs for which bucket the token endpoint is in** — this
+   differs by provider and decides step 1's shape:
+
+| Provider's rule | Client design |
+|-----------------|---------------|
+| Token endpoint shares the general request budget | One throttle covering every request, refresh included |
+| Token endpoint has its own separate budget | Two counters; the refresh must not consume the API budget's slot, and must still respect its own |
+| Undocumented | Route the refresh through the shared throttle — one extra interval per refresh costs a fraction of a second and a shared bucket costs a failed call |
+
+5. **Check the process's initial state.** A last-request timestamp initialized to
+   zero must produce "no wait" against a monotonic clock and "no wait" only for
+   the genuinely first request — assert it in a test that issues two calls back
+   to back from a fresh client and requires the measured gap.
+6. **Keep the server's own 429 handling as well** — the client throttle prevents
+   the common case, the retry path ([backend-common-reliability-timeouts-and-retries])
+   handles the rest.
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| The failure reproduces only on days the cached token expired | That is the signature of this bug, not intermittency — compare a failing day's log timestamps against a day the cache was warm |
+| Several client instances run in one process | The throttle state must be shared across them (class/module-level or an injected limiter); per-instance state multiplies the effective rate by the instance count |
+| Several processes or hosts call the same account | Per-process throttling cannot hold an account-wide cap — move the limiter to a shared store ([backend-common-concurrency-distributed-locks] for the coordination primitive) or divide the budget explicitly per process |
+| The refresh is triggered lazily by a 401 retry rather than by expiry | The retry path issues a token request too — route it through the same throttle, or the retry storms the limit it was recovering from |
+| The provider counts by endpoint class, not per account | Model the buckets the provider documents; one global interval under-uses the fast bucket and still overruns the slow one |
+
+## Instead of
+
+| If you are about to | Do this instead | Why |
+|---------------------|-----------------|-----|
+| Apply the throttle in each public wrapper method | Apply it at the transport, or call it from the refresh path too | Auth refresh is issued from below the wrapper and is invisible to a per-method throttle |
+| Raise the minimum interval until the errors stop | Find which request was unaccounted for and route it through the throttle | A larger interval slows every call to hide one uncounted request, and still fails when two uncounted ones coincide |
+| File the first-call failure as intermittent and add a retry | Compare a fresh-token run against a cached-token run | The trigger is the token cache state, which is deterministic — a retry hides a reproducible ordering bug |
+
+## Sources
+
+- https://developer.okta.com/docs/reference/rl2-token-oauth/ — OAuth token endpoints carry their own documented rate limits, separate from general API limits; which bucket applies is provider-specific
+- https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api — access-token requests are budgeted separately from REST API requests
+- Field reproduction, `auto-trading-bot` commit `82a077e` (`src/broker/kis_client.py`): the wrapper's `_throttle()` ran in `_headers()`, which then called `_get_token()`; on days the token was newly issued the token POST and the following API GET landed in the same second and the provider rejected the call for exceeding its per-second cap. Logs from two such days show `POST …:00.354` → token issued `…:00.495` → balance call rejected `…:00.543`; on cached-token days the same code succeeded. The fix throttles inside `_get_token()` and throttles again after it in `_headers()`
@@ -9,7 +9,7 @@ sources:
   - https://sre.google/sre-book/addressing-cascading-failures/
   - https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
 last_verified: 2026-07-10
-related: [backend-common-api-design-idempotency, backend-common-llm-completion-response-validation]
+related: [backend-common-api-design-idempotency, backend-common-llm-completion-response-validation, backend-common-reliability-client-side-rate-limiting]
 ---
 
 # Calling Another Service over the Network: Timeouts, Retries, Backoff

@@ -31,6 +31,7 @@ Match your situation to a "load when" line; load only matching pages.
 | Page | Load when |
 |------|-----------|
 | [timeouts-and-retries](common/reliability/timeouts-and-retries.md) | Your service calls another service/external API/DB over the network — setting timeouts and deadlines, deciding what to retry per failure type, backoff/jitter, capping concurrency against a slow dependency; debugging pool exhaustion or retry storms |
+| [client-side-rate-limiting](common/reliability/client-side-rate-limiting.md) | Your API client enforces a provider's requests-per-second cap itself and still hits the limit — on the first call of a process, or on some days only; deciding where the throttle lives so token/credential refresh cannot bypass it, whether the token endpoint shares the request budget, and how the limit holds across instances/processes |
 
 ### caching