From 9da707ca7a1e7a61e1053ebbbab01ccdbeacc204 Mon Sep 17 00:00:00 2001 From: dch0202 Date: Wed, 5 Aug 2026 18:40:18 +0900 Subject: [PATCH] knowledge: ingest 6 verified insight(s), revise 1, drop 2 duplicates 10 candidates (4 queued + 6 session-derived). New pages: platforms/toolchains/compiler-sysroot-on-macos, platforms/processes/tool-diagnostics-without-a-failing-exit-code, testing/strategy/differential-testing, testing/data/artifact-leakage-from-a-suite, backend/common/api-design/unenforced-declarations, backend/common/integrations/client-side-rate-limiting. Revised testing/quality/spec-artifact-checks (code constant as canonical for a doc table). Dropped 2 candidates already covered by spec-artifact-checks and harness-reverse-controls. --- .dev-loop/INGEST_REPORT.md | 289 ++++++++++++------ INDEX.md | 6 +- log.md | 3 + .../api-design/unenforced-declarations.md | 80 +++++ .../integrations/client-side-rate-limiting.md | 78 +++++ wiki/backend/index.md | 4 +- wiki/platforms/index.md | 2 + ...diagnostics-without-a-failing-exit-code.md | 83 +++++ .../toolchains/compiler-sysroot-on-macos.md | 79 +++++ .../data/artifact-leakage-from-a-suite.md | 85 ++++++ wiki/testing/index.md | 2 + wiki/testing/quality/spec-artifact-checks.md | 7 +- wiki/testing/strategy/differential-testing.md | 84 +++++ 13 files changed, 711 insertions(+), 91 deletions(-) create mode 100644 wiki/backend/common/api-design/unenforced-declarations.md create mode 100644 wiki/backend/common/integrations/client-side-rate-limiting.md create mode 100644 wiki/platforms/processes/tool-diagnostics-without-a-failing-exit-code.md create mode 100644 wiki/platforms/toolchains/compiler-sysroot-on-macos.md create mode 100644 wiki/testing/data/artifact-leakage-from-a-suite.md create mode 100644 wiki/testing/strategy/differential-testing.md diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index c375750..f3d9973 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,95 +1,214 @@ -# Consolidated review — knowledge PRs #6–#13 +# Knowledge flush — 10 candidate(s) → 6 new pages, 1 revise, 2 dropped -Eight fork PRs (`dch0202-rsquare`, 2026-07-28 → 2026-08-02) were reviewed together -against `AGENTS.md`. Each PR was audited by an independent reviewer (format rules, -sources, vague-qualifier ban, ≤120 body lines, index/log invariants), then -cross-compared to catch duplication the per-PR flushes could not see — they branched -independently off the same main and rewrote the same shared index/log files. Fork -branches can't be edited from here and several PRs needed content changes (drop a -duplicate, merge a colliding page), so this branch carries the reconciled end-state -rather than merging each PR as-is (which would import the duplicates). +10 candidates entered the pipeline: 4 drained from `~/.dev-loop/queue/*.jsonl`, +and 6 derived in-session from work on an LLM-native compiler/runtime platform. +Two of them covered the same case and were merged into one page. Project-specific +names were generalized out of every page body. ## Verified best-practice -Sources are per-page and were live-verified in each originating PR's flush; the -independent re-reviews re-checked them. Landed pages and their evidence base: - -| Page | Confidence | Source basis | -|------|-----------|--------------| -| backend/common/llm/completion-response-validation | verified | OpenAI reasoning guide + chat `object` spec (5 `finish_reason` values), vLLM/LiteLLM reasoning fields; field incident (200/`length`/empty content/8,173-char reasoning) | -| backend/common/llm/context-window-budget | verified | Claude context-window docs, LiteLLM exception mapping, vLLM/Claude Code env-var docs | -| backend/common/integrations/externally-owned-defaults | verified | OpenAI deprecations (notice windows) + models `list`, LiteLLM model_discovery; field incident (alias removed between PR verify and review → 400) | -| backend/common/storage/object-key-persistence | verified | AWS S3 CompleteMultipartUpload + managed-upload API/source, aws-sdk-js issues #1158/#5656 | -| infrastructure/containers/host-cgroup-visibility | field-tested | cgroup_namespaces(7), Docker `--cgroupns=host`, nsenter, k8s #103363; OrbStack repro | -| infrastructure/observability/missing-container-metrics | verified/field-tested | k8s resource-metrics-pipeline docs, kube-prometheus-stack values, kubernetes-mixin; OrbStack #2217 repro | -| platforms/environment/unicode-text-matching | verified | UAX #15, Unicode core §3.12, APFS FAQ, POSIX grep; local repro (macOS 15/APFS, grep 2.6.0-FreeBSD, Python 3.13) | -| platforms/shells/command-text-inspected-before-execution | verified | Claude Code hooks docs, POSIX shell §2.6; local reproduction | -| platforms/processes/non-interactive-cli-invocation | verified | GNU nohup, OpenBSD ssh/ssh_config, git, timeout man pages; no-request-in-gateway-log field incident | -| qa/document-verification/spec-document-gates | field-tested | ESLint, Google mutation testing, RFC 2119, Vale, markdownlint; 32/32 mutant / 62/62 intact RFC sessions | -| qa/document-verification/editing-a-gated-document | field-tested | pgrep, Vale, markdownlint; in-house editing methodology | -| testing/quality/checks-that-cannot-pass | verified | James Shore AoAD2, POSIX grep exit status, Semgrep rule-testing, pytest exit codes; BSD/ugrep measurement | -| testing/quality/spec-artifact-checks | verified | JSON Schema, ESLint RuleTester, pitest, GFM table spec; local cell-count repro + GitHub renderer cross-check | -| testing/quality/harness-reverse-controls | verified | mutation-testing + CI-control sources; field repro (re-fetched all cited URLs, PASS) | - -Three pages were reconciled from two overlapping PR versions each, keeping the more -complete/better-sourced body and folding in the other's unique cases: -- **completion-response-validation** — #12 body (all five `finish_reason` values, - `tool_calls`/`function_call` carve-out, streaming, Responses API, "reasoning is - scratch, not deliverable") kept in `llm/` (coherent with #6/#13); folded in #6's - DeepSeek first-party edge + the field incident. -- **externally-owned-defaults** — #12 generalized body (any repo-external resource) - in `integrations/`; folded in #6's alias-removed field incident + the - gateway-config-vs-live-upstream nuance. -- **non-interactive-cli-invocation** — #12 body (GNU-nohup extension precision, - ssh -n stdin-detach vs BatchMode, pre-log DNS/TLS/proxy + `curl -v`) kept; folded - in #11's DEBIAN_FRONTEND, pager/color TTY case, wrapper-CLI case, field incident. +### 1. A non-Xcode compiler resolving the macOS SDK → `confidence: verified` + +**Claim.** On macOS a compiler installed outside Xcode selects no default sysroot; +supply it with `-isysroot "$(xcrun --show-sdk-path)"` when you own the command +line, `SDKROOT` when you only own the environment, and `CPATH` / `LIBRARY_PATH` +when the driver still resolves the wrong root. + +**Checked.** +- https://github.com/llvm/llvm-project/issues/137352 — confirms the driver selects + no default sysroot on macOS as of LLVM/Clang 20.1.2, that the symptoms are + `'stdio.h' file not found` and `ld: library 'System' not found`, and that the + documented workaround is `export SDKROOT="$(xcrun --sdk macosx --show-sdk-path)"` + or passing `-isysroot`. +- https://github.com/Homebrew/homebrew-core/issues/197277 — **corrected the + candidate.** The queued insight named `-isysroot` as the single fix. This + upstream issue states Homebrew clang "always passes the same value to `ld`'s + `-syslibroot`" — the SDK it was built with — and ignores `SDKROOT`/`-isysroot` + for the link step. The page therefore splits compile-time from link-time and + routes the link case to `LIBRARY_PATH`/`-L` instead of presenting one flag as + the answer. +- https://clang.llvm.org/docs/UsersManual.html — `SDKROOT` is honored as the + default `isysroot`, with an explicit `-isysroot` taking precedence. +- Local reproduction 2026-08-05 (Homebrew LLVM, macOS): bare `clang probe.c` + produced `warning: no such sysroot directory: '…/CommandLineTools/SDKs/MacOSX26.sdk' + [-Wmissing-sysroot]` then `fatal error: 'stdio.h' file not found`; the same + command with `-isysroot "$(xcrun --show-sdk-path)"` compiled and linked. + +### 2. Feeding a tool's warnings back when it exits 0 → `confidence: verified` + +**Claim.** Warnings are non-failures, so a wrapper branching only on the exit code +loses them; capture the diagnostic stream with `2>&1 >/dev/null` — in that order — +or promote warnings with the tool's own switch. + +**Checked.** +- https://clang.llvm.org/docs/UsersManual.html — diagnostics carry levels + (ignored / warning / error / fatal); warnings do not produce a non-zero exit + status unless promoted with `-Werror`. +- https://www.gnu.org/software/bash/manual/bash.html#Redirections — "Redirections + are processed in the order they appear, from left to right", which is exactly + why the two spellings capture different streams. The candidate's claim about + redirection order is correct as stated. +- https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html — a + pipeline's exit status is that of its last command, supporting the "read the + code before piping" directive. +- Local reproduction 2026-08-05: the same CLI produced rc=0 with three `warning:` + lines, rc=0 with empty stderr, and rc=2 with an error — three outcomes + distinguishable only by reading both the code and the stream. + +### 3. Differential testing of two implementations → `confidence: verified` + +**Claim.** When two paths implement one spec, fix which observable classes must +agree and which the contract permits to differ *before* writing the comparison, +and compare only the former. + +**Checked.** +- https://arxiv.org/pdf/2102.07498 — differential testing "has been widely used + for checking the consistency of two (or more) alternative implementations of a + common specification". +- https://handwiki.org/wiki/Differential_testing — also called back-to-back + testing; a second implementation is a stronger oracle than a crash check. +- https://arxiv.org/pdf/2212.01748 — experience report applying it where a + reference implementation supplies the oracle a single implementation lacks. +- The *declare-permitted-differences-first* half is not in the literature under + that name; it is carried as a dated field observation in Sources (four classes + required to agree, five permitted to differ, timing deliberately excluded + because comparing it would fail for contract-permitted reasons). + +### 4. Artifact leakage from a suite → `confidence: verified` + +**Claim.** Diagnose by counting leftovers per name prefix and matching the +distribution to call sites; fix with the runner's owned-temp API; then enforce the +convention with a static rule, watching it fail first. + +**Checked.** +- https://pkg.go.dev/testing#T.TempDir — the directory "is automatically removed + when the test and all its subtests complete", tying removal to test lifetime + including failures. +- https://docs.pytest.org/en/stable/how-to/tmp_path.html — `tmp_path` plus + `tmp_path_retention_count` / `tmp_path_retention_policy`. This **refined the + candidate**: "keep nothing" is wrong when failures need inspecting, so the page + routes that case to a retention policy the runner owns. +- https://eslint.org/docs/latest/extend/custom-rule-tutorial — AST-rule authoring, + the mechanism behind "encode the convention as a static check". +- Field measurement carried in Sources: 998 leftovers under two prefixes + (686 + 306), matching exactly the two of six `mkdtemp` call sites with no + cleanup; post-fix full-suite delta 0, scratch area 72M → 3.3M. + +### 5. Accepting a declaration the system does not enforce → `confidence: verified` + +**Claim.** Split "unrecognized" from "recognized but unenforced" and diagnose each; +resolve declarations by closed-table lookup rather than inference; expose +strictness as a caller-selected level. + +**Checked.** +- https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md + — KEP-2885 defines exactly the three levels the page prescribes, selected per + request via `?fieldValidation=`: `Strict` ("erroring on unknown fields"), `Warn` + (returned as warnings in response headers), `Ignore`. +- https://kubernetes.io/blog/2023/04/24/openapi-v3-field-validation-ga/ — Server + Side Field Validation reached GA in Kubernetes 1.27 for create/update/patch. +- https://json-schema.org/draft/2020-12/json-schema-validation — + `additionalProperties` is the schema-level strictness control, an explicit + decision rather than a parser default. +- The rolling-upgrade edge case (warn on the server, be strict in the client) is + reasoned from the same KEP's client/server split and is stated as guidance, not + attributed to a quotation. + +### 6. Client-side rate limiting → `confidence: field-tested` + +**Claim.** Throttle at the transport layer every request passes; count auth/token +issuance; stamp the clock immediately before the send; give the process's first +call a defined starting state; prefer the provider's headers to a modelled window. + +**Checked.** +- https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api + — exemptions are enumerated individually (`GET /rate_limit` "does not count + against your primary rate limit"), so exemption is a named property rather than + a default for meta requests. +- https://developer.okta.com/docs/reference/rate-limits/ — only specific public + metadata endpoints (`/oauth2/v1/keys`, `/.well-known/*`) are listed as exempt, + while `/oauth2/v1/authorize` is given as rate limited. This supports "a token + endpoint is normally not exempt" **by absence from an exemption list**, which is + weaker than a positive statement — hence `field-tested`, not `verified`. +- https://developers.openai.com/api/docs/guides/rate-limits — remaining/reset are + returned as response headers for the client to consume. +- https://www.rfc-editor.org/rfc/rfc6585 — 429 and `Retry-After`. +- The originating incident is carried as a dated field observation with its log + timeline (token POST 00.354 → issued 00.495 → rejected call 00.543 against a + 2/second limit, reproducing only on cache-cold days). + +### 7. Doc table that is a projection of a code constant → merged, `verified` + +**Claim.** When a document restates a code constant, declare the code canonical +*in the document* and assert the document against the imported constant. + +**Checked.** No external source asserts this as a named practice. It is supported +by the host page's already-cited premise (JSON Schema `required` vs `enum`; ESLint +`RuleTester` must-fail inputs) plus a dated field observation. It was merged into +an already-`verified` page rather than published as a new page on a thin source +list. + +No candidate was upgraded to `verified` without a citation or a reproduction. The +two dropped candidates were dropped as duplicates, not for lack of evidence. ## Existing-layer check -Cross-PR and against-main duplication was the focus. Findings and resolutions: - -- **spec-artifact-checks (#8) ≡ document-conformance-checks (#9)** — same case - (coverage-vs-validity split, per-check negative controls, GFM pipe parsing, - ESLint/Semgrep/mutation examples). #9's report predated awareness of #8. → - **#8 kept canonical; #9's page dropped, `testing/docs-as-spec` category not created.** -- **completion-response-validation (#6) ≈ llm-response-completeness (#12)** — ~95% - same case (HTTP 200 ≠ usable output; `length`/blank/reasoning-budget). → - **merged into one `llm/` page; #12's `integrations/` copy dropped.** -- **gateway-model-alias-defaults (#6) ≈ externally-owned-defaults (#12)** — ~80%; - #12 generalizes the model-alias case to any external resource. → - **kept the general `integrations/` page; #6's LLM-only page dropped.** -- **non-interactive-cli-invocation** — created by BOTH #11 and #12 (file collision). - → **single reconciled page.** -- Distinct (no overlap, all landed): checks-that-cannot-pass, harness-reverse-controls, - spec-document-gates, editing-a-gated-document, unicode-text-matching, - command-text-inspected-before-execution, object-key-persistence, context-window-budget, - host-cgroup-visibility, missing-container-metrics. -- Reciprocal `related:` links added on existing pages (tests-that-cannot-fail, - timeouts-and-retries, environment-config, release-gates, background-services, - portable-shell-scripts, timezone-and-locale, paths-case-and-line-endings, - acceptance-criteria, resource-limits-and-probes, logs-metrics-signals, - minimum-case-set). A dropped-page backlink (#6 → gateway-model-alias-defaults on - environment-config and release-gates) was retargeted to externally-owned-defaults. -- Invariants verified programmatically: all `related:`/inline `[id]` references - resolve, every page listed in its domain index, no duplicate ids, no page >120 - body lines. +Routed via `INDEX.md`, then read each domain index and every page whose "load +when" line overlapped. + +**Pages read:** `testing/index.md`, `testing/quality/harness-reverse-controls.md`, +`testing/quality/spec-artifact-checks.md`, +`testing/quality/behavior-not-implementation.md`, +`testing/data/test-data-and-isolation.md`, `platforms/index.md`, +`platforms/processes/non-interactive-cli-invocation.md`, +`platforms/toolchains/version-management.md`, `backend/index.md`, +`backend/common/reliability/timeouts-and-retries.md`, +`backend/common/integrations/externally-owned-defaults.md`, +`infrastructure/config/environment-config.md`, +`security/input/validation-at-trust-boundaries.md`. + +| Candidate | Overlapping page | Resolution | +|---|---|---| +| Golden fixtures should be generated, not hand-maintained | `testing/quality/spec-artifact-checks` edge case "The artifact is generated rather than hand-written"; `behavior-not-implementation` #4 (when a snapshot is appropriate at all) | **Dropped** — both halves already present; no new directive | +| Harness needs a surviving no-op control; a partial working tree kills every case before the rule runs | `testing/quality/harness-reverse-controls` directives 1 and 4 — directive 4 already prescribes copying "the whole repository rather than the directory under test", the candidate's exact failure | **Dropped** — fully covered, including the mechanism | +| Doc table restating a code constant | `testing/quality/spec-artifact-checks` — same frame, and an adjacent edge case covers a canonical set living in *another document* | **Merged** as one edge-case row, one Instead-of row, and a Sources line; `last_verified` bumped, `related` extended | +| Temp/build artifacts left behind (2 candidates, same case) | `testing/data/test-data-and-isolation` row "Filesystem / temp files → create a fresh per-test temp directory and remove it in teardown" | **New page** — the existing row is the per-test instruction; the new page owns cross-suite diagnosis, the runner-API table, static enforcement, and the zero-delta assertion. Cross-linked | +| Tool warnings with exit 0 | `platforms/processes/non-interactive-cli-invocation` (stdin/prompt/timeout) | **New page** — no overlap on stream-vs-exit-code semantics; sibling in the same category, cross-linked | +| macOS SDK/sysroot | `platforms/toolchains/version-management` (pinning versions) | **New page** — pinning does not cover SDK resolution; the new page defers the pinning half to it | +| Unenforced declarations | `security/input/validation-at-trust-boundaries`; `infrastructure/config/environment-config` #4 | **New page** — both existing pages handle input that is *wrong*; this one handles input that is well-formed, accepted, then not acted on. Cross-linked to both | +| Client-side rate limiting | `backend/common/reliability/timeouts-and-retries`; `integrations/externally-owned-defaults` | **New page** — neither covers throttling one's own outbound rate or which of a client's requests count. Cross-linked to both | + +**Conflicts flagged:** one. The queued LLVM candidate's directive named `-isysroot` +as *the* fix; upstream evidence shows it does not apply to the link step under +Homebrew LLVM. The page states both channels with the condition selecting each, +rather than propagating the candidate as written. + +**Related-links added:** the new pages link to `version-management`, +`path-resolution`, `reading-error-messages`, `non-interactive-cli-invocation`, +`command-text-inspected-before-execution`, `test-data-and-isolation`, +`checks-that-cannot-pass`, `isolate-by-bisection`, `test-level-choice`, +`behavior-not-implementation`, `minimum-case-set`, `regression-scope`, +`validation-at-trust-boundaries`, `environment-config`, `error-responses`, +`acceptance-criteria`, `timeouts-and-retries`, `externally-owned-defaults`, +`intermittent-failures`, and `jwt-server-side`. `spec-artifact-checks` gained a +link to the new `unenforced-declarations`. ## Routing decision -- `backend/common/llm/` (new) — LLM-specific server concerns: completion-response-validation, - context-window-budget. Coherent home shared by #6 and #13. -- `backend/common/integrations/` (new) — general repo-external-dependency concern: - externally-owned-defaults. Kept separate from `llm/` because its scope is any - external resource (bucket/queue/index), not LLM-only. -- `backend/common/storage/` (new) — object-key-persistence. -- `qa/document-verification/` (new) — spec-document-gates, editing-a-gated-document. - Introduced by both #10 and #11; unified into one index section. -- `testing/quality/` (existing) — checks-that-cannot-pass, spec-artifact-checks, - harness-reverse-controls (test/check-authoring discipline, distinct from - qa/document-verification which is release-process gate design). -- `platforms/{environment,shells,processes}/` (existing) — unicode-text-matching, - command-text-inspected-before-execution, non-interactive-cli-invocation. -- `infrastructure/{containers,observability}/` (existing) — host-cgroup-visibility, - missing-container-metrics. - -Source PRs #6–#13 are closed with a disposition comment crediting the author. +| # | Target | New category? | +|---|--------|---------------| +| 1 | `platforms/toolchains/compiler-sysroot-on-macos.md` | No — `toolchains` exists | +| 2 | `platforms/processes/tool-diagnostics-without-a-failing-exit-code.md` | No — `processes` exists; `non-interactive-cli-invocation` owns invoking such a tool, this owns reading what it emitted | +| 3 | `testing/strategy/differential-testing.md` | No — `strategy` owns how to verify a change; this is a verification strategy for a migration/reimplementation rather than a level choice | +| 4 | `testing/data/artifact-leakage-from-a-suite.md` | No — `data` owns what a suite creates and must clean up; placed beside `test-data-and-isolation` | +| 5 | `backend/common/api-design/unenforced-declarations.md` | No — `api-design` owns the contract between a caller and the system; a config/DSL/manifest surface is such a contract even when it is not HTTP | +| 6 | `backend/common/integrations/client-side-rate-limiting.md` | No — `integrations` owns behavior toward a provider the repo does not control | +| 7 | `testing/quality/spec-artifact-checks.md` (revise) | No — merged into the existing page | + +No new category was created. `api-design` was the closest fit for #5 rather than +opening a `declarative-input` category for a single page; if a second page on +declarative-surface design lands, revisit that decision. + +Domain indexes updated: `wiki/platforms/index.md`, `wiki/testing/index.md`, +`wiki/backend/index.md`. Root `INDEX.md` route lines extended for backend, testing, +and platforms. `log.md` carries one `ingest`, one `revise`, and one `dedup` entry. diff --git a/INDEX.md b/INDEX.md index d6d4238..07ee2d1 100644 --- a/INDEX.md +++ b/INDEX.md @@ -10,14 +10,14 @@ follow the cross-pointers in their index or take the next matching seeded domain | Domain | Status | Route here when | |--------|--------|-----------------| | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior | -| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | +| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, declarative input the system does not enforce, JWT, timeouts/retries, client-side rate limiting, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility | | [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) | -| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) | +| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data and leftover artifacts, mock decisions, flaky tests, verifying two implementations of one spec against each other (release-process quality → qa) | | [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing, automated verification of document deliverables (spec/RFC gates) (writing automated test code → testing) | | [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) | | [security](wiki/security/index.md) | **seeded** | Trust-boundary decisions: input validation, session-vs-token auth choice, per-resource authorization (IDOR), secrets hygiene, dependency trust, PII handling (XSS rendering → frontend; CI secrets → infrastructure; JWT implementation → backend/frontend auth) | -| [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, Unicode normalization in text/file-name matching, commands inspected before execution, background services/cron, invoking prompt-capable CLIs non-interactively, toolchain version pinning | +| [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, Unicode normalization in text/file-name matching, commands inspected before execution, background services/cron, invoking prompt-capable CLIs non-interactively, consuming a tool's diagnostics when it exits 0, toolchain version pinning and SDK/sysroot resolution | | [mobile](wiki/mobile/index.md) | **seeded** | App-side iOS/Android/cross-platform: process death/state survival, offline-first sync, mobile-network calls, store rollout/hotfix strategy, startup time | All ten domains are seeded. New categories grow via `skills/wiki-ingest/SKILL.md`. diff --git a/log.md b/log.md index 1c6293b..6dd7a4b 100644 --- a/log.md +++ b/log.md @@ -37,3 +37,6 @@ Append-only. Format: `## [YYYY-MM-DD] &1 >/dev/null` in that order, three-outcome table clean/warned/failed, -Werror as the alternative when you own the invocation), testing/strategy/differential-testing (declare which observable classes must agree and which the contract permits to differ BEFORE writing the comparison; seed a divergence to prove the check can fail; keep the pair running through the migration window), testing/data/artifact-leakage-from-a-suite (count leftovers by name prefix and match the distribution against call sites instead of sweeping code; adopt the runner's owned-temp API; encode the convention as an AST rule and watch it fail first; assert a zero delta), backend/common/api-design/unenforced-declarations (split unrecognized from recognized-but-unenforced and diagnose each; closed-table lookup has a defined miss where inference has none; Ignore/Warn/Strict as a caller-selected level per KEP-2885), backend/common/integrations/client-side-rate-limiting (throttle at the transport layer every request passes, count auth/token issuance, stamp the clock immediately before the send, give the process's first call a defined starting state, prefer the provider's headers over a modelled window). +## [2026-08-05] revise | testing/quality/spec-artifact-checks +1 edge case +1 Instead-of row: the artifact is a hand-written projection of a code constant (a doc table restating an enum/lexicon/rule constant) — name the constant canonical inside the document and assert the document against the imported constant, so the checker cannot become a third copy and a reader who finds the table wrong knows the edit belongs in the code. last_verified bumped to 2026-08-05; related now links backend-common-api-design-unenforced-declarations. +## [2026-08-05] dedup | Dropped 2 session-derived candidates as already covered: "golden/snapshot fixtures should be generated rather than hand-maintained" (testing/quality/spec-artifact-checks already carries "The artifact is generated rather than hand-written → point the must-pass input at a committed golden output of the generator", and behavior-not-implementation owns when a snapshot is appropriate at all); "a detection harness needs a no-op control that must survive, and a partial working tree makes every case die before the rule runs" (testing/quality/harness-reverse-controls owns both — directive 1 requires the equivalent-mutation control to survive and stop the run when it is caught, directive 4 requires a working tree equivalent to the real runner's rather than a partial copy). Neither candidate added a directive the existing page lacked. diff --git a/wiki/backend/common/api-design/unenforced-declarations.md b/wiki/backend/common/api-design/unenforced-declarations.md new file mode 100644 index 0000000..d0e56cd --- /dev/null +++ b/wiki/backend/common/api-design/unenforced-declarations.md @@ -0,0 +1,80 @@ +--- +id: backend-common-api-design-unenforced-declarations +domain: backend +category: api-design +applies_to: [general] +confidence: verified +sources: + - https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md + - https://kubernetes.io/blog/2023/04/24/openapi-v3-field-validation-ga/ + - https://json-schema.org/draft/2020-12/json-schema-validation +last_verified: 2026-08-05 +related: [security-input-validation-at-trust-boundaries, infrastructure-config-environment-config, backend-common-api-design-error-responses, qa-process-acceptance-criteria] +--- + +# Accepting a Declaration the System Does Not Enforce + +## When this applies + +Your system takes declarative input — a config file, a DSL or manifest, a policy +block, a schema annotation — and some of what a caller can write is not +implemented: an unknown key, a verb outside your vocabulary, or a knob that is +recorded but never acted on. Also when a user reports "I declared X and nothing +happened", or a feature was "configured" in an environment where it never ran. + +## Do this + +1. **Separate the two failure shapes and give each its own diagnostic**, because + they read identically to the author — nothing happened: + +| Shape | Meaning | Emit | +|-------|---------|------| +| Unrecognized | The name is not in your vocabulary at all | Reject, naming the unknown token and the accepted set | +| Recognized but unenforced | You parse and store it, but no code path acts on it | Accept and warn, naming the declaration and what it does *not* do | + +2. **Resolve declarations by lookup in a closed, enumerable table, not by + inference.** A lookup has a defined miss (the name is absent → diagnostic); an + inference silently produces a plausible no-op for anything that looks close. + Keep the table one artifact, so "what does this system accept" has a single + answer. +3. **Offer the strictness as a caller-selected level rather than one global + choice** — `Ignore` / `Warn` / `Strict` — so a caller can demand rejection in + CI while a compatibility path keeps warning. Kubernetes made exactly this + split a first-class request parameter. +4. **Default new surfaces to rejection.** Accept-and-ignore is a compatibility + affordance for a surface that already shipped with it, not a starting point; + once callers depend on silent acceptance, tightening it is a breaking change. +5. **Publish the enforcement status next to the vocabulary.** Every declaration a + caller can write gets a row saying what the system actually does with it, and + that table is derived from the code that implements it, not maintained beside + it ([testing-quality-spec-artifact-checks]). +6. **Make deliberate use possible.** Declaring something you know is recorded but + unenforced is legitimate — documenting intent, staging a rollout. The defect + is doing it unknowingly, so the warning must be suppressible per declaration + with an explicit acknowledgement, not by lowering the level globally. + +## Edge cases + +| Case | Then | +|------|------| +| A newer client sends a field this older server has not learned yet | Warn rather than reject on the server, and let the *client's* strict mode catch it at author time; rejecting forward-compatible traffic breaks rolling upgrades | +| The declaration is enforced on one execution path but not another | Report it as unenforced on the path that ignores it, keyed by path — a single global status makes one of the two paths lie | +| The vocabulary is generated (parsed from a schema or enum) | Assert the parsed table is non-empty before using it to validate; an empty table accepts everything and turns strict mode into a no-op | +| Enforcement is measured but not applied (a budget reported, never imposed) | Say so in the diagnostic's wording — "measured, not enforced" — so a reader does not infer a guarantee from the value appearing in output | +| Rejecting would break an existing deployment | Ship the warning first with the version that starts rejecting named in the message, then flip the default | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Ignore an unrecognized key so the parse "just works" | Reject it, naming the unknown token and the accepted set | Silent acceptance turns a typo into a feature that never ran, discovered in production instead of at parse time | +| Infer an action for a verb outside your vocabulary because it looks close | Look it up in the closed table and fail the miss | Inference has no defined miss, so an unsupported name produces a plausible no-op instead of a diagnostic | +| Record a policy/security declaration and treat its presence as the guarantee | Emit what is recorded-only, and keep the enforcement table alongside the vocabulary | A declaration that only annotates reads as protection to everyone downstream who sees it in the source | +| Add a global "strict mode" flag nobody turns on | Make the level a per-request/per-invocation parameter with rejection as the default for new surfaces | A global opt-in stays off, so the diagnostics exist without reaching anyone | + +## Sources + +- https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md — KEP-2885 defines three server-side validation levels selected per request via `?fieldValidation=`: `Strict` ("erroring on unknown fields"), `Warn` (errors returned as warnings in response headers), and `Ignore` (no validation); it moves the decision from each client to the server so unrecognized fields are not simply accepted +- https://kubernetes.io/blog/2023/04/24/openapi-v3-field-validation-ga/ — Server Side Field Validation reached GA in Kubernetes 1.27, validating create, update and patch requests at the apiserver +- https://json-schema.org/draft/2020-12/json-schema-validation — `additionalProperties` is the schema-level control over whether properties outside the declared set are permitted; strictness is an explicit schema decision rather than a parser default +- Field observation 2026-08-05: a declarative platform carried two issues with the same root — step verbs outside its closed lexicon compiled to silent no-ops, and `security`/`policy`/`performance` declarations were recorded without being enforced. Both were resolved by one enforcement matrix generated from the implementing constants plus a diagnostic channel that names the unenforced declaration diff --git a/wiki/backend/common/integrations/client-side-rate-limiting.md b/wiki/backend/common/integrations/client-side-rate-limiting.md new file mode 100644 index 0000000..7dee261 --- /dev/null +++ b/wiki/backend/common/integrations/client-side-rate-limiting.md @@ -0,0 +1,78 @@ +--- +id: backend-common-integrations-client-side-rate-limiting +domain: backend +category: integrations +applies_to: [general] +confidence: field-tested +sources: + - https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api + - https://developer.okta.com/docs/reference/rate-limits/ + - https://developers.openai.com/api/docs/guides/rate-limits + - https://www.rfc-editor.org/rfc/rfc6585 +last_verified: 2026-08-05 +related: [backend-common-reliability-timeouts-and-retries, backend-common-integrations-externally-owned-defaults, debugging-concurrency-intermittent-failures, backend-common-auth-jwt-server-side] +--- + +# A Client-Side Throttle That the Client's Own Requests Bypass + +## When this applies + +You added a rate limiter to an API client wrapper and the provider still returns +429 / "too many requests" — especially on the first call after process start, or +only on some days. Also when a rate-limit failure is filed as "intermittent" +because a rerun minutes later succeeds. + +## Do this + +1. **Place the throttle where every outbound request passes**, at the transport + layer — the interceptor, session hook, or `send()` override — not on the + business-facing methods. Token refresh, discovery/metadata lookups, retries, + and pagination prefetch are requests the provider counts and the wrapper's + public methods do not name. +2. **Count auth and token issuance as requests.** Providers exempt only the + endpoints they explicitly list, and the lists cited below name only public + metadata endpoints (JWKS, discovery, a limit-status endpoint) — not token + issuance. Treat "exempt" as a documented property: count every request whose + endpoint you cannot find on the provider's exemption list. +3. **Stamp the throttle clock immediately before the request leaves**, not when + the wrapper method is entered. Work between the two — a token fetch, a + signature computation, a DNS resolution — moves the real send into the next + window's neighbour. +4. **Give the first call of the process a defined starting state.** A "last + request at" initialized to zero or epoch makes the first N calls skip the wait + entirely, so a burst at startup exceeds a small per-second limit + deterministically while a warm process never reproduces it. +5. **Read the provider's own accounting headers rather than modelling its + limit.** Consume the remaining/reset/retry-after headers and back off from + them; a client-side counter is a courtesy layer, and the server's numbers are + the only authority on what has been spent. +6. **Honour `Retry-After` on 429 before any backoff of your own.** The status is + defined for this case and the header carries the wait the server wants. + +## Edge cases + +| Case | Then | +|------|------| +| A cached credential hides the extra request on most runs | Reproduce with the cache cleared — the token-issuance day is the failing case, and the code path that only runs then is the one to inspect | +| The limit is per-second and the wrapper throttles per-minute averages | Enforce the shortest published window; an average that satisfies 120/min still sends 10 in one second | +| Several processes or workers share one credential and one quota | The quota is per credential, not per process — move the throttle to a shared store, or partition the budget explicitly per worker | +| The provider counts by cost/points rather than by request | Track the published cost unit; a request counter under-counts expensive calls and over-throttles cheap ones | +| Retries are added by an HTTP library beneath your throttle | Configure the library's retry policy through the same layer that throttles, or disable it and retry above the throttle; otherwise each logical call can emit several counted requests | +| A metadata/discovery endpoint is documented as exempt | Keep it outside the counter, and cite the doc line in a comment — the exemption is the provider's to change | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Wrap only the public API methods with the throttle | Throttle in the transport layer every request passes through | Auth refresh and retries happen inside lower layers, so they never reach a method-level limiter | +| Classify a rate-limit failure as intermittent because a rerun passed | Check whether the failing runs share a state the passing runs lack (cold cache, expired token, process start) | A failure conditional on cache state is deterministic given that state, and reruns silently supply the state that hides it | +| Model the provider's window in the client and trust the model | Consume the provider's remaining/reset headers and honour `Retry-After` | The server's accounting includes requests your model never saw, from other processes sharing the credential | +| Initialize "last request at" to zero so the first call is never delayed | Initialize so the first call is subject to the same spacing, or seed from the first send | The unthrottled prefix is exactly the startup burst that trips a per-second limit | + +## Sources + +- https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api — requests count against the authenticated user's limit, and exemptions are named individually (`GET /rate_limit` "does not count against your primary rate limit"): exemption is an enumerated property, not a default for meta requests +- https://developer.okta.com/docs/reference/rate-limits/ — only specific public metadata endpoints (`/oauth2/v1/keys`, `/.well-known/openid-configuration`, `/.well-known/oauth-authorization-server`) are listed as not subject to rate limits, while OAuth endpoints such as `/oauth2/v1/authorize` are rate limited +- https://developers.openai.com/api/docs/guides/rate-limits — limits are enforced per window with the remaining/reset values returned in response headers for the client to consume +- https://www.rfc-editor.org/rfc/rfc6585 — 429 Too Many Requests, and `Retry-After` as the server-supplied wait before a further request +- Field observation 2026-08-05: an API client throttled inside its public methods issued the token POST from within its header-building path, after the throttle had already waited; on days the token cache was cold, the token POST and the following GET landed in the same second and exceeded a 2/second limit, while cache-warm days ran the identical code without failing — logged at 00.354 (token POST), 00.495 (token issued), 00.543 (rejected call) diff --git a/wiki/backend/index.md b/wiki/backend/index.md index 9e683bd..57f4b1f 100644 --- a/wiki/backend/index.md +++ b/wiki/backend/index.md @@ -5,7 +5,7 @@ three stack subtrees — route by concern first, stack second: | Subtree | Route there when | |---------|------------------| -| [common](#common-language-agnostic) (below) | The concern is language-agnostic: API contracts, idempotency, JWT issuance, outbound calls, caching, jobs, transactions in app code, shared state/pools, exception structure, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references | +| [common](#common-language-agnostic) (below) | The concern is language-agnostic: API contracts, idempotency, declarative input the system does not enforce, JWT issuance, outbound calls, client-side rate limiting, caching, jobs, transactions in app code, shared state/pools, exception structure, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references | | [java](java/index.md) | You are writing/reviewing JVM backend code (Java/Kotlin, Spring, JPA/Hibernate) and the concern is stack-specific: entity mapping, persistence context, proxy pitfalls, JVM threads/memory | | [node](node/index.md) | You are writing/reviewing Node.js/TypeScript backend code: event-loop blocking, promise error handling, runtime validation at boundaries, graceful shutdown | | [python](python/index.md) | You are writing/reviewing Python backend code: GIL/concurrency model, pydantic validation, WSGI/ASGI workers, language traps | @@ -25,6 +25,7 @@ Match your situation to a "load when" line; load only matching pages. | [error-responses](common/api-design/error-responses.md) | Designing or reviewing API error handling — choosing status codes (400/401/403/404/409/422/500), defining the error body shape, deciding what a 500 may reveal; clients report inconsistent/unparseable errors | | [idempotency](common/api-design/idempotency.md) | An endpoint with side effects (create, charge, send) can receive the same request twice — client retry after timeout, user double-submit, gateway retry; designing idempotency-key storage; deciding which operations are safe to retry | | [pagination-contract](common/api-design/pagination-contract.md) | Designing a list endpoint's request/response contract — cursor vs page-number, limit caps, total counts, expired-cursor behavior (the backing SQL/index → databases/query-optimization/keyset-pagination) | +| [unenforced-declarations](common/api-design/unenforced-declarations.md) | Your system accepts declarative input (config file, DSL/manifest, policy block, schema annotation) and part of what a caller may write is unimplemented — an unknown key, a verb outside your vocabulary, or a knob recorded but never acted on; a user reports "I declared X and nothing happened"; choosing between reject/warn/ignore and where that strictness is selected | ### reliability @@ -82,6 +83,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| +| [client-side-rate-limiting](common/integrations/client-side-rate-limiting.md) | You added a throttle to an API client wrapper and the provider still returns 429 — especially on the first call after process start, or only on some days; a rate-limit failure was filed as intermittent because a rerun passed; deciding which layer the throttle belongs in and whether token/auth requests count against the quota | | [externally-owned-defaults](common/integrations/externally-owned-defaults.md) | A code/config default names a resource the repo does not own (model alias, endpoint, bucket, queue, index) — reviewing or merging a PR that claims that default works, adding a startup check that the name still resolves, or diagnosing a default path that broke with no code change | ### storage diff --git a/wiki/platforms/index.md b/wiki/platforms/index.md index b97fa71..a338ace 100644 --- a/wiki/platforms/index.md +++ b/wiki/platforms/index.md @@ -44,12 +44,14 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [background-services](processes/background-services.md) | Something must run persistently or on a schedule on a dev machine or server (daemon, watcher, cron-style job); a "started" process dies when the terminal/SSH/agent session ends; choosing nohup vs LaunchAgent vs systemd unit vs cron/timer; a job works in the terminal but fails under cron/launchd (minimal environment); wiring service logs and restart policy | +| [tool-diagnostics-without-a-failing-exit-code](processes/tool-diagnostics-without-a-failing-exit-code.md) | Wiring a compiler/linter/type-checker/validator into a hook, CI step, or agent loop so its complaints reach the author; the tool prints warnings but exits 0 so an exit-code-only wrapper reports success; choosing between capturing the diagnostic stream and promoting warnings with `-Werror`/`--max-warnings 0`; a redirection captured the tool's artifact instead of its diagnostics | | [non-interactive-cli-invocation](processes/non-interactive-cli-invocation.md) | Calling a tool that can prompt (agent CLI, ssh, git, package manager) from a script, CI step, hook, or agent session, including with its own `-p`/`--print`/`--yes` flag; such a call produced no output and never returned; deciding whether a hang belongs to the client, the network, or the far-side service; choosing the stdin/timeout/fail-fast switches for an unattended call; a TTY-detecting tool changes its output format under automation | ## toolchains | Page | Load when | |------|-----------| +| [compiler-sysroot-on-macos](toolchains/compiler-sysroot-on-macos.md) | On macOS a non-Xcode compiler (Homebrew/MacPorts LLVM) fails with `'stdio.h' file not found`, `ld: library 'System' not found`, or a `-Wmissing-sysroot` warning naming an SDK directory that does not exist; a build works under `/usr/bin/clang` but not under the toolchain the project requires; choosing between `-isysroot`, `SDKROOT`, `CPATH`, and `LIBRARY_PATH`; separating a toolchain precondition from a code regression when only the compiled tests fail | | [version-management](toolchains/version-management.md) | "Works on my machine" from tool-version drift; a project needs a pinned language/tool version (.nvmrc, .python-version, .tool-versions); making CI use the same versions as local; onboarding a machine reproducibly; a script/cron/CI step can't find a version-managed binary (shims absent in non-interactive shells); deciding where lockfiles fit in reproducibility | ## Planned (unseeded categories) diff --git a/wiki/platforms/processes/tool-diagnostics-without-a-failing-exit-code.md b/wiki/platforms/processes/tool-diagnostics-without-a-failing-exit-code.md new file mode 100644 index 0000000..5309d54 --- /dev/null +++ b/wiki/platforms/processes/tool-diagnostics-without-a-failing-exit-code.md @@ -0,0 +1,83 @@ +--- +id: platforms-processes-tool-diagnostics-without-a-failing-exit-code +domain: platforms +category: processes +applies_to: [general] +confidence: verified +sources: + - https://clang.llvm.org/docs/UsersManual.html + - https://www.gnu.org/software/bash/manual/bash.html#Redirections + - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html +last_verified: 2026-08-05 +related: [platforms-processes-non-interactive-cli-invocation, platforms-shells-command-text-inspected-before-execution, debugging-signals-reading-error-messages] +--- + +# Feeding a Tool's Warnings Back When It Exits 0 + +## When this applies + +You are wiring a compiler, linter, type checker, or validator into a hook, CI +step, or agent loop so its complaints reach the author. The tool prints +diagnostics but exits 0 for anything it classifies below "error", so a wrapper +that branches on the exit code reports success on a file that was warned about. + +## Do this + +1. **Decide what the wrapper treats as failure, then read the channel that + carries it.** An exit code answers "did the tool refuse to finish"; it does + not answer "did the tool have something to say". Warnings are by definition + not failures, so a tool that emits them still exits 0. +2. **Capture the diagnostic stream without swallowing the artifact:** + +```sh +OUT=$(tool "$FILE" 2>&1 >/dev/null) # stderr -> current stdout, then stdout -> /dev/null +[ -n "$OUT" ] && printf '%s\n' "$OUT" >&2 && exit 2 +``` + + Redirections are applied left to right, so `2>&1 >/dev/null` duplicates + stderr onto the terminal/pipe stdout currently points at and *then* discards + stdout. The reversed spelling `>/dev/null 2>&1` sends both to `/dev/null` and + captures nothing. + +3. **Prefer the tool's own promotion switch when you own the invocation and want + a hard gate**: `-Werror` (clang/gcc), `--max-warnings 0` (ESLint), + `--strict` equivalents. Then the exit code carries the decision again and the + wrapper stays a one-liner. Use stream capture when you must keep warnings + non-fatal for humans while still surfacing them to the loop. +4. **Distinguish the three outcomes explicitly** — clean, warned, failed — and + emit a different signal for each. A wrapper with two branches cannot tell + "nothing to say" from "said something the exit code did not encode". + +| Observed | Read it as | Wrapper does | +|----------|------------|--------------| +| exit 0, diagnostic stream empty | Clean | Pass silently | +| exit 0, diagnostic stream non-empty | Warned — the case an exit-code-only wrapper loses | Surface the text to the author; choose block or advise per your gate policy | +| exit non-zero | Failed | Surface the text and block | + +5. **Keep the exit code readable end to end.** When the wrapper's own exit code + is a decision, do not pipe it into `head`/`tail`/`grep` — the pipeline's status + is the last command's. Capture to a variable or read `${PIPESTATUS[0]}`. + +## Edge cases + +| Case | Then | +|------|------| +| The tool writes diagnostics to stdout, mixed with its real output | Prefer a machine-readable flag (`--format json`, `--output-file`) and parse the diagnostics field; when none exists, send the artifact to a file (`-o out.bin`) so the remaining stream is diagnostics only | +| Diagnostics include absolute paths, timings, or ids that change per run | Strip the volatile fields before comparing runs; an unstable string makes "did anything change" untestable | +| The tool prints a banner or progress line on stderr even when clean | Match on the diagnostic form (a `warning:`/`error:` prefix, a non-empty JSON array), not on stream non-emptiness | +| The wrapper runs on every save and the tool is slow | Gate on the changed file only, and bound the call with a timeout so a hung tool fails the step rather than the session ([platforms-processes-non-interactive-cli-invocation]) | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Branch a hook solely on `$?` from a compiler or linter | Read the diagnostic stream, and branch on the exit code only for hard failures | Warnings are defined as non-failures, so the exit code is 0 for every one of them | +| Write `tool "$F" 2>/dev/null >&1` to keep stderr | Write `tool "$F" 2>&1 >/dev/null` | The reversed order points stderr at the old stdout and then keeps stdout, so the captured text is the tool's artifact, not its diagnostics | +| Add `-Werror` to a shared build so a hook can use the exit code | Capture the stream in the hook, and keep `-Werror` a deliberate build policy decision | Promoting every warning changes what the build rejects for everyone, to fix a wrapper's read of one stream | + +## Sources + +- https://clang.llvm.org/docs/UsersManual.html — diagnostics are emitted at levels (ignored / warning / error / fatal); warnings do not produce a non-zero exit status unless promoted with `-Werror`, and clang warns rather than errors on flags it has not implemented +- https://www.gnu.org/software/bash/manual/bash.html#Redirections — "Redirections are processed in the order they appear, from left to right", which is why `2>&1 >/dev/null` and `>/dev/null 2>&1` capture different streams +- https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html — a pipeline's exit status is that of its last command, so a decision-bearing status must be read before piping +- Local reproduction 2026-08-05: the same CLI returned rc=0 with three `warning:` lines on stderr for one input, rc=0 with an empty stderr for a clean input, and rc=2 with an error for a rejected input — the three cases are distinguishable only by reading both the code and the stream diff --git a/wiki/platforms/toolchains/compiler-sysroot-on-macos.md b/wiki/platforms/toolchains/compiler-sysroot-on-macos.md new file mode 100644 index 0000000..b9970ca --- /dev/null +++ b/wiki/platforms/toolchains/compiler-sysroot-on-macos.md @@ -0,0 +1,79 @@ +--- +id: platforms-toolchains-compiler-sysroot-on-macos +domain: platforms +category: toolchains +applies_to: [macos, clang, llvm] +confidence: verified +sources: + - https://github.com/llvm/llvm-project/issues/137352 + - https://github.com/Homebrew/homebrew-core/issues/197277 + - https://clang.llvm.org/docs/UsersManual.html +last_verified: 2026-08-05 +related: [platforms-toolchains-version-management, platforms-environment-path-resolution, debugging-signals-reading-error-messages] +--- + +# A Non-Apple Compiler Resolving the macOS SDK + +## When this applies + +On macOS, a compiler installed outside Xcode (Homebrew LLVM, MacPorts, a +downloaded toolchain) fails with `'stdio.h' file not found`, `ld: library +'System' not found`, or a `-Wmissing-sysroot` warning naming an SDK directory +that does not exist. Also when a build works under `/usr/bin/clang` but not +under the newer toolchain a project requires. + +## Do this + +1. **Resolve the SDK at build time, never hardcode it**: `xcrun --show-sdk-path` + (or `xcrun --sdk macosx --show-sdk-path`) returns the SDK the active developer + directory currently provides. A path written into a script pins an SDK version + that a Command Line Tools or Xcode update removes. +2. **Feed it in by the channel that matches who owns the command line:** + +| You control | Do | +|-------------|-----| +| The compile command | Pass `-isysroot "$(xcrun --show-sdk-path)"` on the command line | +| Only the environment (a build script, generator, or test harness invokes the compiler internally) | Export `SDKROOT="$(xcrun --show-sdk-path)"`; clang reads it as the default sysroot | +| Neither, and headers still are not found | Export `CPATH="$SDK/usr/include"`, which the driver adds to the include search path regardless of the sysroot the driver picked | +| Neither, and the **link** step fails | Export `LIBRARY_PATH="$SDK/usr/lib"` (or pass `-L`) — see the edge case below | + +3. **Read `-Wmissing-sysroot` as the cause, not as noise.** The driver emits it + and then continues without system headers, so the build dies later at the + first `#include` with a message that names a header instead of the SDK. Treat + the first warning line as the diagnosis and stop reading the header error. +4. **Verify the fix with a two-line probe before re-running the real build**, so + a toolchain problem is never re-diagnosed as a code regression: + +```sh +printf '#include \nint main(){return 0;}\n' > probe.c +"$CC" -isysroot "$(xcrun --show-sdk-path)" probe.c -o probe && echo "sysroot ok" +``` + +5. **Pin the toolchain and its environment together.** The `SDKROOT`/`CPATH` + exports belong in the same committed setup script as the compiler version + ([platforms-toolchains-version-management]); an environment assembled by hand + per machine reproduces this failure on the next clone. + +## Edge cases + +| Case | Then | +|------|------| +| Headers resolve but linking still fails on a Homebrew LLVM | That toolchain passes the SDK it was **built** with to `ld -syslibroot` and ignores `-isysroot`/`SDKROOT` for the link step (an open upstream issue). Supply the library path separately: `LIBRARY_PATH="$SDK/usr/lib"` or explicit `-L` | +| The compiler lives in a keg-only/unlinked prefix and `command -v` finds nothing | Call it by absolute path, or prepend its `bin` to `PATH` in the setup script; a tool absent from `PATH` makes a harness take its "tool not installed" branch and report an unrelated failure | +| `xcrun` itself errors | The active developer directory is unset or points at a removed install; fix with `xcode-select` before touching compiler flags | +| A subset of a test suite fails only on the compiled path | Reproduce the same failure on an unmodified checkout of the base branch first; when it reproduces there, it is an environment precondition, not a regression in the change under review | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Hardcode `/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk` (or a versioned SDK path) into a build script | Resolve it with `xcrun --show-sdk-path` at build time | The versioned directory disappears on the next toolchain update, and the compiler degrades to a warning rather than failing at the point of the missing SDK | +| Report "N tests fail on this branch" from a run whose compile step warned about a missing sysroot | Fix the sysroot, or reproduce on the base branch and report it as an environment precondition | The failures are one step removed from their cause and read as a code regression | +| Set only `CPATH` and treat the toolchain as configured | Set the sysroot too, and check the link step separately | `CPATH` adds include directories; it does not give the linker a library search root | + +## Sources + +- https://github.com/llvm/llvm-project/issues/137352 — as of LLVM/Clang 20.1.2 `clang hello.c` on macOS fails with `ld: library 'System' not found` and `'stdio.h' file not found`; "one must manually populate the `SDKROOT` environment variable (or pass `-isysroot`): `export SDKROOT="$(xcrun --sdk macosx --show-sdk-path)"`" — the driver selects no default sysroot on macOS +- https://github.com/Homebrew/homebrew-core/issues/197277 — Homebrew clang "always passes the same value to `ld`'s `-syslibroot`": "the SDK that `llvm` was built with is always selected when linking instead of the value set by `xcrun` / in `SDKROOT` / with `-isysroot`"; compilation respects `-isysroot` while the link step does not (labelled an upstream issue) +- https://clang.llvm.org/docs/UsersManual.html — clang honors `SDKROOT` as the default `isysroot` when set, and an explicit `-isysroot` on the command line takes precedence +- Local reproduction 2026-08-05 (macOS, Homebrew LLVM at `/opt/homebrew/opt/llvm`): `clang probe.c` emitted `warning: no such sysroot directory: '/Library/Developer/CommandLineTools/SDKs/MacOSX26.sdk' [-Wmissing-sysroot]` followed by `fatal error: 'stdio.h' file not found`; the same command with `-isysroot "$(xcrun --show-sdk-path)"` (an Xcode SDK path) compiled and linked successfully diff --git a/wiki/testing/data/artifact-leakage-from-a-suite.md b/wiki/testing/data/artifact-leakage-from-a-suite.md new file mode 100644 index 0000000..415c028 --- /dev/null +++ b/wiki/testing/data/artifact-leakage-from-a-suite.md @@ -0,0 +1,85 @@ +--- +id: testing-data-artifact-leakage-from-a-suite +domain: testing +category: data +applies_to: [general] +confidence: verified +sources: + - https://pkg.go.dev/testing#T.TempDir + - https://docs.pytest.org/en/stable/how-to/tmp_path.html + - https://eslint.org/docs/latest/extend/custom-rule-tutorial +last_verified: 2026-08-05 +related: [testing-data-test-data-and-isolation, testing-quality-checks-that-cannot-pass, debugging-methodology-isolate-by-bisection] +--- + +# A Suite That Leaves Working Directories Behind + +## When this applies + +Temp directories, build outputs, or scratch files accumulate in the repo or the +system temp area after a suite runs; a clone grows without an obvious owner; +`git status` shows untracked artifacts nobody created by hand. You suspect +"they leak from everywhere" and are deciding where to start. + +## Do this + +1. **Count the leftovers by name prefix before reading any code**, and treat the + distribution as the diagnosis: + +```sh +ls "$SCRATCH_DIR" | sed 's/-[a-z0-9]*$//' | sort | uniq -c | sort -rn +``` + + Concentration is the normal shape — a few call sites produce nearly all of + them. Match the top prefixes against the call sites that create scratch + directories; when the counts line up with the sites that have no cleanup, the + cause is identified without guessing. + +2. **Fix by adopting the runner's owned-temp API at those sites**, not by adding + a `rm` at the end of each test. The API ties removal to the test's lifetime, + so it also runs when the test fails or panics: + +| Runner | Use | +|--------|-----| +| Go | `t.TempDir()` — the directory "is automatically removed when the test and all its subtests complete" | +| pytest | the `tmp_path` fixture; set `tmp_path_retention_policy`/`tmp_path_retention_count` when you want failed runs kept for inspection | +| Anything without such an API | Create in setup and remove in a teardown that runs on failure too (`finally`, fixture teardown, `t.Cleanup`) | + +3. **Encode the convention as a static check once most sites already follow it.** + A convention that N−2 of N call sites obey is machine-checkable: write an + AST/lint rule that flags a raw `mkdtemp`/`TempDir`-equivalent call outside the + approved helper. Fixing only the instances leaves the next author free to + repeat it. +4. **Watch the new check fail before you fix the code.** Run it against the + unfixed tree and require it to report exactly the known-bad sites; a rule + authored after the fix has only ever been observed green + ([testing-quality-checks-that-cannot-pass]). +5. **Assert the invariant, not the cleanup call.** Add one test that records the + scratch area's entry count before and after the suite and requires a delta of + zero. That survives refactors of how directories are created. + +## Edge cases + +| Case | Then | +|------|------| +| The artifacts are wanted on failure for debugging | Keep them under a retention policy the runner owns (pytest's retention settings) rather than by skipping cleanup; an unconditional leak is not a debugging feature | +| A crashed or killed run leaves directories no teardown could remove | Give the suite a session-scoped root it creates and removes wholesale, so one removal reclaims every orphan from prior aborted runs | +| Cleanup code exists but the directory survives | The path being removed is not the path being created — log both at one failing site before editing; a `cd` or a relative path resolved from a different working directory is the usual gap | +| The leak is outside the repo (system temp), so `git status` is clean | Measure the scratch area's size and entry count as the signal; repo-only checks report a suite that fills the disk as healthy | +| Artifacts are produced by a subprocess (a compiler, a bundler) the test invokes | Point the subprocess at the test-owned directory via its output flag or `cwd`; a subprocess inherits neither the fixture nor its teardown | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Read every call site because "it leaks from everywhere" | Count leftovers by prefix and compare the distribution with the call-site list | Leftovers concentrate: a few sites produce the bulk, so the counts close the question without a code sweep | +| Add a `rm -rf` at the end of each affected test | Adopt the runner's owned-temp API, or a teardown that runs on failure | A trailing removal is skipped exactly when the test fails, which is when artifacts pile up fastest | +| Fix the offending sites and move on | Add the static rule and a zero-delta assertion as well | The instances are the symptom; the convention is what the next author will or will not follow | +| Add `*.tmp`-style entries to `.gitignore` | Remove the artifacts at their source | Ignoring hides the growth from `git status` while the disk keeps filling | + +## Sources + +- https://pkg.go.dev/testing#T.TempDir — "TempDir returns a temporary directory for the test to use. The directory is automatically removed when the test and all its subtests complete." +- https://docs.pytest.org/en/stable/how-to/tmp_path.html — the `tmp_path` fixture provides a per-test temporary directory; by default recent directories are retained, configurable via `tmp_path_retention_count` and `tmp_path_retention_policy`, so retention is an explicit policy rather than a leak +- https://eslint.org/docs/latest/extend/custom-rule-tutorial — authoring an AST-based rule that reports a disallowed call pattern, the mechanism for enforcing a call-site convention once it is established +- Field observation 2026-08-05: in one repository, 998 leftover directories carried two name prefixes (686 + 306); of the six files calling `mkdtemp`, the two with no cleanup were exactly those two prefixes' owners. After adopting an owned-temp helper at those sites, a full-suite run measured a scratch-directory delta of 0 and the scratch area shrank from 72M to 3.3M diff --git a/wiki/testing/index.md b/wiki/testing/index.md index 2cbad76..83599cc 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -15,6 +15,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [test-level-choice](strategy/test-level-choice.md) | Deciding at which level (unit/integration/e2e) to test new or changed behavior; reviewing a test plan's level distribution; logic buried in a controller or framework wiring needs coverage | +| [differential-testing](strategy/differential-testing.md) | Two code paths must satisfy one specification and you want each to act as the other's oracle — interpreter vs compiled backend, a rewrite beside the service it replaces, v1 vs v2, a fast path beside a reference path, a cache beside its source; deciding which observable classes must agree and which the contract permits to differ; needing migration evidence beyond "the new one's own tests pass" | ## quality @@ -32,6 +33,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [test-data-and-isolation](data/test-data-and-isolation.md) | Tests need fixture data and you are choosing how to create it; tests pass alone but fail together (or vice versa); DB cleanup, shared fixtures, time-dependent logic, or unique-value collisions | +| [artifact-leakage-from-a-suite](data/artifact-leakage-from-a-suite.md) | Temp directories, build outputs, or scratch files pile up in the repo or system temp after a suite runs; a clone grows with no obvious owner; you suspect the leak comes from everywhere and need a way to locate it; deciding between per-site cleanup, the runner's owned-temp API, and a static rule that enforces the convention | ## mocking diff --git a/wiki/testing/quality/spec-artifact-checks.md b/wiki/testing/quality/spec-artifact-checks.md index d754f6e..318f300 100644 --- a/wiki/testing/quality/spec-artifact-checks.md +++ b/wiki/testing/quality/spec-artifact-checks.md @@ -9,8 +9,8 @@ sources: - https://eslint.org/docs/latest/integrate/nodejs-api - https://pitest.org/ - https://github.github.com/gfm/ -last_verified: 2026-07-29 -related: [testing-quality-tests-that-cannot-fail] +last_verified: 2026-08-05 +related: [testing-quality-tests-that-cannot-fail, backend-common-api-design-unenforced-declarations] --- # Checks That Verify a Spec or Mapping Artifact @@ -84,6 +84,7 @@ cells = [c.replace(r"\|", "|").strip() | The source list the mapping is checked against is itself derived (a parsed enum, a globbed file set) | Assert the parsed list's length against a known count before comparing; a source list that parses to zero items makes coverage pass vacuously | | Asserting a fixed cell count on body rows | Enforce it as your repo's convention and state it as one: GFM requires only the header row to match the delimiter row in cell count, while body rows with fewer cells get empty cells inserted and excess cells are ignored | | The artifact is generated rather than hand-written | Point the must-pass input at a committed golden output of the generator, so a generator change reddens the check instead of silently redefining the spec | +| The artifact is a hand-written projection of a value that lives in code (a doc table restating an enum, lexicon, or rule constant) | Name the code constant as canonical **inside the document**, and assert the document against the imported constant rather than re-listing its members in the checker. A reader who then finds the table wrong knows the edit belongs in the code, and the checker cannot drift into a third copy | | A cell legitimately holds no value (a rule with no mapped target) | Give the validity check an explicit sentinel to accept (`—`, `n/a`) and assert that sentinel's spelling, so an empty cell stays distinguishable from an intended blank | ## Instead of @@ -94,6 +95,7 @@ cells = [c.replace(r"\|", "|").strip() | Add every check, mutate one thing, and confirm the harness goes red | Mutate once per check and require the owning check to be the one that reddens | One red proves some check fired, not that each check detects the defect it claims to own | | Split Markdown rows with `row.split("\|")` and assert the cell count | Split on unescaped pipes and unescape each cell first | An escaped `\|` inside a cell inflates the naive count, so the checker reports a false "broken table" on a document that renders correctly | | Treat a green coverage run over a parsed source list as proof of completeness | Assert the parsed list's item count first, then compare sets | An empty source list satisfies set equality against anything, so coverage passes while nothing was checked | +| Keep a documentation table in sync with a code constant by review discipline | Declare the constant canonical in the document and add a check that imports it and fails on divergence | Two hand-maintained copies drift, and the document is the copy readers act on — so the drift is discovered by someone following instructions that are already wrong | ## Sources @@ -102,4 +104,5 @@ cells = [c.replace(r"\|", "|").strip() - https://pitest.org/ — "Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived"; line coverage "measures only which code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code" - https://github.github.com/gfm/ — tables extension: "Include a pipe in a cell's content by escaping it, including inside other inline spans"; "The header row must match the delimiter row in the number of cells. If not, a table will not be recognized"; for body rows, "If there are a number of cells fewer than the number of cells in the header row, empty cells are inserted. If there are greater, the excess is ignored" - Local reproduction 2026-07-29 (Python 3.9.6, macOS): a 5-column row whose third cell holds `create\|update\|delete` splits to 7 cells with `split("|")` and to 5 cells with `re.split(r"(?