choiyounggi · choiyounggi · Aug 5, 2026
@@ -12,12 +12,12 @@ follow the cross-pointers in their index or take the next matching seeded domain
 | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior |
 | [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
 | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility |
-| [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) |
+| [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting), orchestrating parallel worker sessions over one repository (status signals, liveness verdicts, run-scoped shared state) |
 | [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) |
 | [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing, automated verification of document deliverables (spec/RFC gates) (writing automated test code → testing) |
 | [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) |
 | [security](wiki/security/index.md) | **seeded** | Trust-boundary decisions: input validation, session-vs-token auth choice, per-resource authorization (IDOR), secrets hygiene, dependency trust, PII handling (XSS rendering → frontend; CI secrets → infrastructure; JWT implementation → backend/frontend auth) |
-| [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, Unicode normalization in text/file-name matching, commands inspected before execution, background services/cron, invoking prompt-capable CLIs non-interactively, toolchain version pinning |
+| [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, Unicode normalization in text/file-name matching, commands inspected before execution, agent-harness hooks that rewrite tool results, background services/cron, invoking prompt-capable CLIs non-interactively, toolchain version pinning |
 | [mobile](wiki/mobile/index.md) | **seeded** | App-side iOS/Android/cross-platform: process death/state survival, offline-first sync, mobile-network calls, store rollout/hotfix strategy, startup time |
 
 All ten domains are seeded. New categories grow via `skills/wiki-ingest/SKILL.md`.
@@ -37,3 +37,6 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-08-03] ingest | Consolidated review of knowledge PRs #6–#13 (8 fork PRs) into 12 pages. New: backend/common/llm (completion-response-validation, context-window-budget), backend/common/integrations (externally-owned-defaults), backend/common/storage (object-key-persistence), infrastructure/containers/host-cgroup-visibility, infrastructure/observability/missing-container-metrics, platforms/environment/unicode-text-matching, platforms/shells/command-text-inspected-before-execution, platforms/processes/non-interactive-cli-invocation, qa/document-verification (spec-document-gates, editing-a-gated-document), testing/quality (checks-that-cannot-pass, spec-artifact-checks, harness-reverse-controls). All cited URLs are per-PR live-verified; three pages were reconciled from two overlapping PR versions each (see revise/dedup entries below).
 ## [2026-08-03] revise | Reconciled 3 pages from overlapping PR pairs, taking the more complete/better-sourced body and folding in the other's unique cases: backend/common/llm/completion-response-validation (#12 body — tool_calls/function_call carve-out, streaming, Responses API status==incomplete, "reasoning is scratch, not deliverable" — kept in llm/ per #6/#13 category, folded in #6's DeepSeek-first-party edge + the 8,173-char reasoning_content field incident); backend/common/integrations/externally-owned-defaults (#12 generalized body — any repo-external resource — folded in #6's LiteLLM-alias-removed field incident + gateway-config-vs-live-upstream nuance); platforms/processes/non-interactive-cli-invocation (#12 body — GNU-nohup extension precision, ssh -n stdin-detach vs BatchMode, pre-log DNS/TLS/proxy + curl -v — folded in #11's DEBIAN_FRONTEND, pager/color TTY case, wrapper-CLI case, and the no-request-in-gateway-log field incident).
 ## [2026-08-03] dedup | Dropped 3 candidate pages as duplicates/superseded during the #6–#13 consolidation: testing/docs-as-spec/document-conformance-checks (#9 — same case as testing/quality/spec-artifact-checks from #8: coverage-vs-validity split, per-check negative controls, GFM pipe parsing; #8 kept as canonical, docs-as-spec category not created); backend/common/llm/gateway-model-alias-defaults (#6 — subsumed by the generalized integrations/externally-owned-defaults; the model-alias case is one instance); backend/common/integrations/llm-response-completeness (#12 — folded into llm/completion-response-validation, kept in llm/ for category coherence with context-window-budget).
+## [2026-08-05] ingest | Knowledge flush of 7 queued insights → 5 new pages, 2 revised. New category infrastructure/orchestration (control-signals-vs-primary-artifacts: verify a status/watcher verdict against git log + substrate liveness before restarting or discarding a worker, report a worker's own assigned task id rather than a discovered session name, stat the status file after writing it; shared-run-state: namespace the orchestration state directory per run id, survey it for foreign task ids before writing, confirm a foreign run from worktrees/branches/default-branch HEAD rather than from the status files). New: platforms/tools/harness-mediated-tool-results (a PostToolUse hook's updatedToolOutput replaces a tool's result — test the substituted text's own remediation once, then switch to grep+awk and name the fallback in every spawned worker's brief); backend/common/reliability/client-side-rate-limiting (throttle at the transport layer every request passes through, stamp the timestamp immediately before the send, and branch on whether the provider meters the token endpoint in the same bucket — GitHub/Okta/Auth0 separate, KIS shared); testing/quality/migration-call-site-survey (enumerate call sites by callee symbol, not parameter name — positional callers carry no parameter text — and audit test helper definitions that reproduce the old shape).
+## [2026-08-05] revise | platforms/shells/portable-shell-scripts +1 directive/+2 rows: `${VAR:-default}` substitutes for empty as well as unset, so passing `VAR=` to disable a feature is silently ignored; use `${VAR-default}` when empty must mean "deliberately off", or pass a value the script's own validation rejects. POSIX 2.6.2 quoted; reproduced under sh/bash/zsh/dash. last_verified 2026-07-10 → 2026-08-05.
+## [2026-08-05] revise | platforms/shells/command-text-inspected-before-execution +3 edge cases: a blocked command emits no side effect, so a consumer polling for the signal it would have written waits forever (stat the artifact, report the signal as un-emitted); a path-scoped rule matches command text not purpose, so the harness's own helper script is refused like any other write; a Write-tool write succeeding on the exact path Bash was refused for is gate scope, not permission to route around the rule. Cross-linked to the new orchestration pages. last_verified 2026-07-30 → 2026-08-05.
@@ -0,0 +1,98 @@
+---
+id: backend-common-reliability-client-side-rate-limiting
+domain: backend
+category: reliability
+applies_to: [general]
+confidence: verified
+sources:
+  - https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api
+  - https://developer.okta.com/docs/reference/rl2-token-oauth/
+  - https://auth0.com/docs/troubleshoot/customer-support/operational-policies/rate-limit-policy
+  - https://github.com/koreainvestment/open-trading-api
+  - https://datatracker.ietf.org/doc/html/rfc6749#section-4.4
+last_verified: 2026-08-05
+related: [backend-common-reliability-timeouts-and-retries, backend-common-auth-jwt-server-side, debugging-concurrency-intermittent-failures]
+---
+
+# A Client-Side Throttle That the Auth/Token Request Slips Past
+
+## When this applies
+
+You wrapped an external API client with a per-second (or per-minute) throttle to
+respect the provider's published limit, and a rate-limit error still comes back —
+characteristically on the *first* call after process start, or only on some days.
+Also when adding a throttle to a client that refreshes its own credentials.
+
+## Do this
+
+1. **Place the throttle at the single lowest point every outbound request passes
+   through** — the transport/session layer (`Session.send`, an httpx event hook, an
+   interceptor), not the public methods. Credential refresh is issued from inside
+   `_headers()`, an interceptor, or a retry path; a throttle wrapped around the
+   public methods never sees it.
+
+2. **Stamp the throttle's timestamp immediately before the request is written, and
+   only there.** Stamping in the caller lets any request issued between the stamp
+   and the send share one slot.
+
+3. **Establish which bucket the token endpoint consumes, and encode the answer.**
+   This is provider-specific and decides whether the token request needs a slot:
+
+| Provider policy | Do |
+|-----------------|----|
+| One limit covers the API host, token endpoint included (KIS `EGW00201` 초당 거래건수) | Take a throttle slot for the token request too |
+| Token endpoint has its own bucket (GitHub: 2,000 OAuth token requests/hour as a secondary limit, separate from the primary REST limit; Okta and Auth0 meter `/oauth/token` separately) | Give the token request its own limiter at that endpoint's rate, and keep it out of the API limiter |
+| Policy not documented | Take a slot for it and cache the token; the cost is one slot per refresh, the alternative is a hard failure on refresh day |
+
+4. **Cache the credential and refresh it before expiry**, so the token request is
+   rare rather than per-call. Providers additionally cap issuance itself — KIS
+   reissues an access token once per minute — so a client that re-issues per
+   request fails on the token endpoint even when the API limiter is correct.
+
+5. **Verify the cold-start path, not just the steady state.** With
+   `_last_request_at = 0`, the first call computes a huge elapsed time and sleeps
+   zero — correct for a single request, and exactly the state in which a token POST
+   plus the first API GET both proceed with no gap. Assert the first *two* requests
+   are spaced, not the first one.
+
+6. **Log the wall-clock time of every outbound request with its endpoint.** The
+   ordering of token POST versus API GET within one second is what identifies this
+   fault; without it, the failure reads as a provider-side flake.
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| The failure reproduces only on some days | Correlate with token-cache expiry, not with load. A valid cached token removes the extra request, so the same code passes every day the cache is warm |
+| Multiple processes or workers share one credential and one limit | The limit is per account/key, not per process. Move the counter to a shared store or give each process a documented fraction of the limit — an in-process throttle cannot see its siblings |
+| The client retries internally on 429 | Retries are outbound requests and must take slots too, or the retry storm sustains the violation ([backend-common-reliability-timeouts-and-retries]) |
+| Provider returns a rate-limit error as HTTP 200 with an error code in the body | Parse the body for the code (KIS returns `EGW00201`); a status-only check counts the response as success and the throttle is never seen to be wrong |
+| Sandbox/paper credentials fail where production credentials pass | Paper accounts carry lower per-second limits than live ones; size the throttle from the environment's own documented limit, not from production's |
+| A token refresh is triggered concurrently by several in-flight calls | Guard the refresh with a single-flight lock so N callers produce one token request, not N |
+
+## Instead of
+
+| If you are about to | Do this instead | Why |
+|---------------------|-----------------|-----|
+| Wrap each public API method in the throttle | Throttle in the transport layer every request passes through | Token/refresh requests originate below the public methods and bypass a method-level throttle entirely |
+| Stamp the throttle timestamp when the wrapper is entered | Stamp immediately before the request is sent | Any request issued between the stamp and the send shares that slot, so two requests land in one window |
+| Call the failure intermittent and add a retry | Log request times and endpoints, then compare a token-issuing day against a cached-token day | The trigger is cache expiry, which is deterministic; a retry hides it until the limit tightens |
+| Assume the token endpoint is free | Read the provider's limit for it and encode the branch in step 3 | Providers split about evenly between shared and separate buckets, and the wrong assumption fails only on refresh |
+
+## Sources
+
+- https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api — "No more than 2,000 OAuth access token requests per hour are allowed for GitHub Apps and OAuth apps", documented as a secondary rate limit distinct from the primary REST API limit
+- https://developer.okta.com/docs/reference/rl2-token-oauth/ — OAuth 2.0 token endpoint limits are metered per authorization server, separately from other API endpoints
+- https://auth0.com/docs/troubleshoot/customer-support/operational-policies/rate-limit-policy — Auth0 meters "the number of requests made to API endpoints and, in some cases, the number of endpoint operations", with per-endpoint policies including the Authentication API
+- https://github.com/koreainvestment/open-trading-api — KIS Open API: `EGW00201` 초당 거래건수 초과; "토큰 재발급 - 1분당 1회 발급됩니다"; 모의투자 계좌는 REST API 호출 제한이 낮음
+- https://datatracker.ietf.org/doc/html/rfc6749#section-4.4 — the client credentials grant obtains the token by an HTTP POST to the token endpoint; it is an ordinary request subject to the server's limits
+
+## Field context
+
+Diagnosed 2026-08-04 in a trading client whose `_headers()` called `_throttle()`
+and *then* `_get_token()`, so the token POST and the following API GET were issued
+inside one throttle window against a 2-requests-per-second paper-account limit.
+Logs from the two token-issuance days (07-23, 08-04) show POST at `…00.354`, token
+returned at `…00.495`, balance call failing at `…00.543`; on every day the cached
+token was still valid the identical code succeeded, which is why it had been
+recorded as an intermittent outage.
@@ -9,7 +9,7 @@ sources:
   - https://sre.google/sre-book/addressing-cascading-failures/
   - https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
 last_verified: 2026-07-10
-related: [backend-common-api-design-idempotency, backend-common-llm-completion-response-validation]
+related: [backend-common-api-design-idempotency, backend-common-llm-completion-response-validation, backend-common-reliability-client-side-rate-limiting]
 ---
 
 # Calling Another Service over the Network: Timeouts, Retries, Backoff

@@ -31,6 +31,7 @@ Match your situation to a "load when" line; load only matching pages.
 | Page | Load when |
 |------|-----------|
 | [timeouts-and-retries](common/reliability/timeouts-and-retries.md) | Your service calls another service/external API/DB over the network — setting timeouts and deadlines, deciding what to retry per failure type, backoff/jitter, capping concurrency against a slow dependency; debugging pool exhaustion or retry storms |
+| [client-side-rate-limiting](common/reliability/client-side-rate-limiting.md) | Adding or reviewing a throttle that keeps an API client inside a provider's published rate limit; a rate-limit error still fires despite the throttle, on the first call after start or only on some days; deciding where the throttle layer belongs and whether token/credential-refresh requests take a slot; sizing a limit shared across processes or across sandbox vs production credentials |
 
 ### caching
 

@@ -4,7 +4,8 @@ Route here for: CI/CD pipeline design, secrets in build/deploy flows, container
 image builds, container resource limits and health probes, per-environment
 configuration (env vars, config drift, startup validation), rollout/rollback
 strategy, observability (logging, metrics, alerting), datastore backup/restore
-and data-loss planning.
+and data-loss planning, and orchestrating parallel worker sessions over one
+repository (status signals, liveness verdicts, run-scoped shared state).
 
 Match your situation to a "load when" line; load only matching pages.
 
@@ -41,6 +42,13 @@ Match your situation to a "load when" line; load only matching pages.
 |------|-----------|
 | [rollout-and-rollback](deploy/rollout-and-rollback.md) | Designing how a service reaches production (rollout strategy, health gating); preparing a risky release; a deploy involves a schema change, data migration, or feature flag and you need rollback mechanics |
 
+## orchestration
+
+| Page | Load when |
+|------|-----------|
+| [control-signals-vs-primary-artifacts](orchestration/control-signals-vs-primary-artifacts.md) | An orchestrator is about to restart, discard, merge, or keep waiting on a worker based on a status file, a watcher's exit code, or a heartbeat; a monitor reports a worker dead while it is committing; a worker's status write produced no output and you must decide whether it landed; deciding what identifier a worker should report itself by; distinguishing alive-and-progressing from stalled from dead |
+| [shared-run-state](orchestration/shared-run-state.md) | Several agent/worker sessions coordinate through files in one repository (status directory, briefs, escalations, claim files); choosing the path layout for that state; starting an orchestration in a repo that may already have one running; a watcher woke on a task id it did not create; the default branch moved during a run |
+
 ## observability
 
 | Page | Load when |