diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 7172a68..0442afa 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,192 +1,58 @@ -# Knowledge flush — 3 insight(s) - -Queue drained: `~/.dev-loop/queue/0c6a5439-….jsonl` (1 row), -`~/.dev-loop/queue/fb7e7221-….jsonl` (2 rows). All three were harvested -2026-08-04 from the `linkly-t1-spec-notation` and `linkly-t1-repo-policy` repos. - -Result: **2 new pages, 1 merge into an existing page, 1 new category.** - -| # | Insight (trigger → directive) | Outcome | Confidence | -|---|-------------------------------|---------|------------| -| 1 | Adding a node kind to a format whose only gate mutates a golden example → commit a fixture holding the new kind, one negative per keyword, verify each reddens | **New page** `testing/quality/schema-additions-under-a-golden-gate` | verified | -| 2 | Enumerating call sites before a contract change → enumerate by callee name; a parameter-name search is a partial index | **New page + new category** `backend/common/change-impact/call-site-enumeration` | verified | -| 3 | A fixture helper whose shape depends on a value the test also passes to the SUT → put that value in the helper's signature | **Merged** into `testing/data/test-data-and-isolation` | verified | - ---- +# Knowledge flush — 7 insight(s) ## Verified best-practice -Every URL below was fetched during this flush; the quotes are from those -fetches, not from memory. Nothing was cited that I did not open. - -### Insight 1 — a golden-derived negative corpus cannot reach a newly added schema branch - -*Claim under test:* when a format's only gate builds negatives by mutating one -committed golden example, adding a new node kind to the schema produces a green -run that proves nothing about the addition. - -| Source checked | What it establishes | -|----------------|---------------------| -| https://json-schema.org/understanding-json-schema/reference/conditionals | The mechanism, verbatim: *"If `if` is invalid, `else` must also be valid (and `then` is ignored)"* — a branch keyed on the new kind is simply **not applied** to an instance that lacks it. So a mutant of a golden without the new kind cannot exercise the new branch. | -| https://json-schema.org/understanding-json-schema/reference/object | Why the new branch also needs constraining before a negative can even exist: *"By default any additional properties are allowed"* and *"By default, the properties defined by the `properties` keyword are not required"*. | -| https://json-schema.org/draft/2020-12/json-schema-core | Sibling applicators *"MUST NOT impact the results of sibling subschemas"*; and *"Unknown keywords SHOULD be treated as annotations"* — a misspelled keyword in a new branch is ignored rather than rejected, a second silent-pass mode. | -| https://pitest.org/quickstart/basic_concepts/ | The same gap has a name in mutation tooling: *"**No coverage**: the same as **Survived** except there were no tests that exercised the line of code where the mutation was created"* — cited so the page tells a reader using PIT/Stryker what the symptom looks like there. | - -*Session evidence (kept as a field observation, not as the basis of the -directive):* in `linkly-t1-spec-notation`, `grep -rln "lir.schema\|jsonschema" -impl/tests/` returned 0 of 447 tests, and the only schema gate over `*.lir.json` -was `scripts/validate_ir.py --self-test`, whose three negatives are all -`copy.deepcopy` mutations of `examples/login.lir.json`. - -**Confidence: `verified`** — the directive's mechanism is stated in the official -JSON Schema documentation; the incident is corroborating, not load-bearing. +**1. Gate a warning-emitting tool on captured stderr, not the exit code** (`lnpl` compiler hook, linkly session) +- Claim: tools emit warnings on stderr while exiting 0, so a hook keyed on exit codes misses all warnings; capture with `OUT=$(tool "$F" 2>&1 >/dev/null)` (order matters) and feed back via exit 2. +- Verified: local reproduction this session — `2>&1 >/dev/null` inside `$(...)` captured exactly `warning: W1` while the reversed order captured nothing. Sources fetched live: POSIX 2.7 Redirection ("the order of evaluation is from beginning to end"), GNU bash manual Redirections ("processed in the order they appear, from left to right", with the `ls > dirlist 2>&1` example), and code.claude.com/docs/en/hooks (exit 2: "stderr text is fed back to Claude as an error message"; PostToolUse "Shows stderr to Claude; the tool already ran"). +- Confidence: **verified**. -### Insight 2 — enumerate by callee name, not by parameter name +**2. Attribute leaked test artifacts by prefix counts, then enforce the convention statically** (linkly, 998 leftover temp entries) +- Claim: leak volume concentrates in a few producers; count leftovers by name prefix, match against creating call sites, fix those, then add a static check proved red first. +- Verified: measured field incident only (686+306 of 998 entries = exactly the two `mkdtemp` sites without cleanup; post-fix tmp delta 0, 72M → 3.3M). No external doc claims to check. +- Confidence: **field-tested**. -*Claim under test:* a keyword-argument search (`repo_rows=`) is structurally -incapable of finding call sites that pass the same argument positionally. +**3. Restoring after a red-run mutation when the fix is uncommitted: copy+hash, not `git checkout --`** +- Claim: `git checkout -- ` discards the unstaged fix along with the mutation. +- Verified: local git reproduction this session — uncommitted fix + mutation + `git checkout --` returned the file to the last commit (fix lost); a *staged* fix survived, confirming the restore source is the **index** (the harvested candidate said HEAD — corrected in the page); copy+hash restore round-tripped identically. Source fetched live: git-scm.com/docs/git-checkout ("Replace the specified files ... with the version from the index"). +- Confidence: **verified**. -| Source checked | What it establishes | -|----------------|---------------------| -| https://docs.python.org/3/glossary.html | *positional-or-keyword* — *"specifies an argument that can be passed either positionally or as a keyword argument. **This is the default kind of parameter**"*. The blindness is therefore the default case, not an edge case. | -| https://docs.python.org/3/library/ast.html | `ast.Call`: *"`args` holds a list of the arguments passed by position"*, *"`keywords` holds a list of `keyword` objects representing arguments passed by keyword"* — the two forms live in **separate fields**, so a keyword-name text search reads only one of them. | -| https://peps.python.org/pep-0570/ | The `/` and `*` markers, which is what makes the page's "declare it keyword-only so a stale positional call errors instead of rebinding" edge case actionable. | +**4. Route token/auth requests through the client-side throttle** (`stock-trader` `kis_client.py`) +- Claim: token refresh inside the header-builder bypasses a wrapper-level throttle; token POST + first API call land in the same second, so the failure only fires on cold-token days and reads as intermittent. +- Verified: field incident with log timestamps (POST 00.354 → issue 00.495 → rejected call 00.543). The draft page's two source URLs did **not** state the claimed facts (Okta rl2-token-oauth is about per-token limit allocations), so they were replaced with pages that do: Okta rate-limits overview (OAuth2 endpoints sit in rate-limit buckets; only public metadata endpoints are exempt) and Auth0's Authentication API endpoint rate-limit policy. Both fetched live; supporting, not primary, evidence. +- Confidence: **field-tested**. -*Reproduction run this session* (Python 3.14.6, macOS, no files written — piped -to `python3` on stdin): over four call sites of `verify(...)` of which one passes -`repo_rows=` by keyword, a regex search for `repo_rows\s*=` matches **1** while -an AST pass over `Call` nodes named `verify` finds **4** — 3 sites invisible to -the keyword search. This is the minimal version of the reported incident. +**5. Homebrew clang on macOS needs `-isysroot "$(xcrun --show-sdk-path)"`** +- Claim: Homebrew clang defaults to a baked-in CommandLineTools SDK path; when it vanishes, clang warns (`-Wmissing-sysroot`) and proceeds without system headers, failing one step downstream. +- Verified: clang DiagnosticsReference fetched live (`-Wmissing-sysroot` exists, enabled by default); LLVM Discourse #77604 (recommends `-isysroot $(xcrun -show-sdk-path)`); Homebrew/homebrew-core#45061 (Homebrew clang does not find the system headers Apple's driver finds); local `xcrun --show-sdk-path` resolves. Session evidence: 69 failing tests reduced to a one-file probe, fixed by the flag. +- Confidence: **verified**. -*Session evidence:* recon reported "13 call sites, 7 need editing"; 8 further -seeds passed the value as `verify()`'s 4th positional argument, and the suite the -session had reported green then ran `472 tests / FAILED (failures=11)`. +**6. Enumerate call sites by callee, not parameter name** — **dropped as a duplicate.** The page `backend/common/change-impact/call-site-enumeration.md` (merged to main 2026-08-04, PR #20) already carries this directive, the same linkly field incident, and the Python positional-or-keyword mechanism. The one novel fragment in the re-harvest — a test helper appearing once in the enumeration while feeding the old contract to N callers — was added as one edge-case row + one source line. -**Confidence: `verified`** — official language reference plus a reproduction. - -### Insight 3 — a fixture factory takes the value the test also passes to the SUT - -*Claim under test:* when a fixture helper's shape depends on a value the test -also feeds the system under test, that value belongs in the helper's signature -rather than in a module-level default. - -| Source checked | What it establishes | -|----------------|---------------------| -| https://abseil.io/resources/swe-book/html/ch12.html | A test is *complete* when *"its body contains all of the information a reader needs in order to understand how it arrives at its result"*; DAMP over DRY; and, directly on point, that engineers should *"use helper methods with descriptive parameters that make dependencies explicit"* rather than reusing shared constants. | -| https://testing.googleblog.com/2017/01/testing-on-toilet-keep-cause-and-effect.html | Keep the inputs a result depends on visible in the test method rather than in shared setup, so cause and effect is readable without jumping elsewhere. | - -*Session evidence:* `rows_for(doc)` seeded from the module constant `PAYLOAD` -while its tests ran payload `{}`; a shape-only migration fixed 1 of 11 failures, -moving the payload into the signature fixed 11 of 11. - -**Confidence: `verified`** — both sources are prescriptive on the exact point. - -Note: the canonical name for this smell is xUnit Patterns' *Mystery Guest*. -`xunitpatterns.com` is HTTP-only and could not be fetched over HTTPS this -session, so **it is deliberately not cited** — an unfetchable URL is worse than -none, per the ingest rules. - ---- +**7. `${VAR:-default}` treats empty as unset, defeating `VAR=` off-switches** +- Claim: to disable via env against a `:-` read, pass a value the script's own validation rejects (e.g. `WATCH_TMUX=/nonexistent`), or change the read to `${VAR-default}`. +- Verified: local reproduction in bash 3.2 **and** zsh 5.9 (colon form substituted on empty; colon-less form respected empty). GNU bash manual fetched live: "Omitting the colon results in a test only for a parameter that is unset." +- Confidence: **verified**. ## Existing-layer check -Routed via `INDEX.md` → domain `index.md` → every page whose "load when" line -overlapped. Pages read in full before deciding: all six of `testing/quality/`, -`testing/data/test-data-and-isolation`, `testing/index`, `qa/process/regression-scope`, -`qa/index`, `debugging/index`, `backend/index`, `backend/python/index`, plus a -repo-wide search for prior coverage (`grep -rliE "call site|callee|positional -argument|keyword argument"` and a `\bgrep\b` sweep over `wiki/`). - -### Insight 1 — nearest neighbours, and why it is not a duplicate +Read before writing: `INDEX.md`; domain indexes for testing, backend, platforms; pages `tests-that-cannot-fail`, `test-data-and-isolation`, `call-site-enumeration`, `portable-shell-scripts`; the three draft pages left untracked by an interrupted earlier flush run (adopted after independent re-verification, one with corrected sources). -| Page read | Overlap | Decision | -|-----------|---------|----------| -| `testing/quality/spec-artifact-checks` | Closest. Already owns *"one negative control per check, mutating only what that check owns"* and the coverage-vs-validity split. | **Not a merge.** Its trigger is *authoring a check*; this insight's trigger is *evolving the artifact the check validates* — the negatives already exist and are individually sound, yet the corpus as a whole cannot reach the new shape. Per the wiki's one-case-per-page rule this is a new trigger. Linked both ways. | -| `testing/quality/tests-that-cannot-fail` | Owns the break-the-code red-run rule the new page's step 4 depends on. | Referenced inline; `related:` added both ways. | -| `testing/quality/harness-reverse-controls` | Owns the *harness-level* control (can this harness go green). | Complementary, not overlapping: that page asks whether the harness discriminates at all, this one asks whether the fixture corpus reaches a newly added shape. `related:` added both ways. | -| `testing/quality/checks-that-cannot-pass` | Trigger is a check whose **target does not exist yet**. | Distinct — here the target exists and the gate is green. No edit. | -| `qa/document-verification/spec-document-gates` | Release-gate altitude for document deliverables. | Kept as a one-way `related:` from the new page. | - -**No conflicts found.** Nothing in the wiki contradicts the new directive. - -### Insight 2 — no existing coverage anywhere - -The repo-wide search found **zero** pages discussing call-site enumeration, -callee-name search, or positional-vs-keyword arguments as a recon concern. The -single adjacent line is `qa/process/regression-scope`'s edge-case row *"The -change is in code with no test coverage and unclear callers | Trace callers -before scoping"* — which names the need and does not say how. That row now -points at the new page, and `regression-scope` gained the new page in -`related:` (both directions). - -Checked and rejected as homes: `debugging/*` (diagnosis of a failure, not -pre-change recon), `backend/python/language/mutable-state-traps` (mutable -defaults and shared state — a different mechanism), `platforms/tools/bsd-vs-gnu-cli` -(grep *flag* portability, not search strategy). - -### Insight 3 — merged, not created - -`testing/data/test-data-and-isolation` already owns fixture construction and -carries the adjacent rule *"pass explicitly only the fields the test's behavior -depends on"* plus a row for shared **mutable** fixture objects. This insight is -the same trigger with a directive that extends it (a *defaulted* constant, which -is not mutated and so is not covered by the existing row). Merge-before-create -applied — no new page. Added: 1 `Do` row, 1 edge-case row (the symptom is a -lookup miss far from the helper), 1 `Instead of` row, 2 sources, and -`last_verified` bumped to 2026-08-04. - ---- +- **Merged, not created** (same trigger, compatible directive): #3 → `testing/quality/tests-that-cannot-fail` (edge-case row + Instead-of row + 3 source lines); #2 → `testing/data/test-data-and-isolation` (edge-case row + Instead-of row + field-incident source); #7 → `platforms/shells/portable-shell-scripts` (edge-case row extending the existing `"${OPT:-}"` row, Instead-of row, GNU-manual source); #6 remainder → `backend/common/change-impact/call-site-enumeration` (one edge row). +- **No conflicts found**: no existing directive contradicts any candidate; #3 sharpens the harvested claim (index, not HEAD) rather than conflicting with a page. +- **Related links added both ways**: warnings page ↔ `portable-shell-scripts`, ↔ `command-text-inspected-before-execution`; `macos-sdk-sysroot` ↔ `path-resolution`, ↔ `version-management`; `client-side-rate-limiting` ↔ `timeouts-and-retries`; `call-site-enumeration` ↔ `test-data-and-isolation`. +- **Overlap with open PRs**: unmerged PRs #32 and #34 (earlier flushes of parallel sessions' re-harvests of the same incidents) cover much of the same ground. Dedup here is against merged `main` per the skill; reviewer should merge one flush and close the overlapping ones — flagged in `log.md` too. ## Routing decision -| Insight | Target | Rationale | -|---------|--------|-----------| -| 1 | `testing` / `quality` / **new page** `schema-additions-under-a-golden-gate` | `INDEX.md` routes "writing or structuring automated tests … verifying tests can actually fail" to `testing`; within it, `quality` already holds the five pages about whether a check proves anything. Existing category, no structural change. | -| 3 | `testing` / `data` / **merge** into `test-data-and-isolation` | Same trigger as the page's own ("tests need fixture data and you are choosing how to create it"); directive extends step 1. | -| 2 | `backend` / `common` / **NEW category `change-impact`** / `call-site-enumeration` | See below. | - -### New category: `backend/common/change-impact/` - -**Why `backend`:** `AGENTS.md`'s routing protocol resolves a multi-domain match -by "the domain that owns **the artifact you will change**" — here, application -code. **Why `common`:** the directive is language-agnostic (any language with -optional or positional arguments; in a language with no keyword arguments at all -the parameter-name search returns nothing whatsoever). The Python mechanics are -cited as the mechanism, not as the scope. - -**Why not an existing category** — all eleven `backend/common` categories were -re-checked by name before creating a new one: - -- `api-design` is the only near miss, and all three of its pages are HTTP-shaped - (status codes, endpoint idempotency, list-endpoint pagination). Its "load when" - lines are written about endpoints; filing an in-process function-signature - concern there would make the category's routing lines contradict its contents, - which invariant 1 forbids. -- `reliability`, `caching`, `jobs`, `errors`, `auth`, `orm`, `concurrency`, - `llm`, `integrations`, `storage` — all runtime-behaviour categories; none - covers a design-time change-impact question under any other name. - -`change-impact` is the noun for the concern, leaves room for sibling pages -(schema/event-contract consumers, deprecation windows), and is registered in -`wiki/backend/index.md` plus the root `INDEX.md` backend row. - -### Plumbing - -- `wiki/testing/index.md` — new `quality` row with a use-case-enumerating "load when" line. -- `wiki/backend/index.md` — new `change-impact` section + subtree summary row updated. -- `INDEX.md` — backend row's `common/` concern list updated. -- `log.md` — `## [2026-08-04] ingest | …` entry appended. -- `related:` added both ways for all four adjacent pages. - -### Invariants verified mechanically (not by eye) - -Ran over all **141** pages after the edits: - -- every page is listed in an ancestor `index.md` → **0 unlisted** -- every `related:` id and inline `[page-id]` reference resolves → **0 broken** -- no page exceeds 120 body lines → **0 over** -- banned vague qualifiers (`usually`, `consider`, `might want to`, `generally`, - `as appropriate`) in the three touched pages → **0 hits** -- every "don't"-shaped statement in the new pages is descriptive prose, not a - bare prohibition; anti-patterns appear only in `Instead of` tables, each paired - with its replacement. +| # | Insight | Target | New/merge | Note | +|---|---------|--------|-----------|------| +| 1 | stderr-warnings gate | `platforms/shells/warnings-on-stderr-with-exit-zero` | new page | Harvest hinted `testing`, but the mechanics are shell redirection + hook contract → platforms/shells; testing owns none of it | +| 2 | artifact-leak attribution | `testing/data/test-data-and-isolation` | merge | Existing page already owns temp-file hygiene rows | +| 3 | mutation restore | `testing/quality/tests-that-cannot-fail` | merge | The red-run procedure this trap occurs in lives on this page | +| 4 | throttle bypassed by auth | `backend/common/reliability/client-side-rate-limiting` | new page | `timeouts-and-retries` covers outbound-call policy, not client-side throttle design; same category, new page | +| 5 | macOS SDK sysroot | `platforms/toolchains/macos-sdk-sysroot` | new page | `version-management` is about version drift, not SDK resolution; same category, new page | +| 6 | call-site enumeration | `backend/common/change-impact/call-site-enumeration` | drop (dup) + 1 edge row | Already merged to main 2026-08-04 | +| 7 | `${VAR:-}` off-switch | `platforms/shells/portable-shell-scripts` | merge | Page already documents `${OPT:-}` under `set -u`; this is its inverse trap | + +No new categories were needed; `reliability`, `toolchains`, and `shells` all pre-exist. Index "load when" lines added/extended for every touched page; `log.md` ingest entry appended. diff --git a/log.md b/log.md index c519858..205aaf5 100644 --- a/log.md +++ b/log.md @@ -39,3 +39,4 @@ Append-only. Format: `## [YYYY-MM-DD] /bin` (verified: `which mlir-opt` not found vs `/opt/homebrew/opt/llvm/bin/mlir-opt` → LLVM 22.1.8; source docs.brew.sh/FAQ). platforms/processes/non-interactive-cli-invocation +bracketed-paste edge: injecting a long/multiline prompt into a REPL (tmux `send-keys -l`) stalls at `❯ [Pasted text #1]` because the input is one bracketed-paste block (ESC[200~…201~) whose embedded newline is not submit — send Enter as a separate keystroke a beat later (source en.wikipedia.org/wiki/Bracketed-paste + claude-code#43169). testing/strategy/test-level-choice +import-side-effect edge/instead-of: a "pure" function's test is not dependency-free if its module runs I/O at import; `@pytest.mark.skipif` evaluates after the module import so it can't gate it — use `importorskip`/`skip(allow_module_level=True)` or move the function to a side-effect-free module (source docs.pytest.org skipping). Harvested "infrastructure" hint for the keg-only insight re-routed to platforms/environment (dedicated PATH page). Confidence: keg-only & pytest verified vs official docs; bracketed-paste mechanism doc-verified, the claude-CLI submit specifics field-tested. +## [2026-08-05] ingest | knowledge-flush of 7 queued insights. New: platforms/shells/warnings-on-stderr-with-exit-zero (gate a warning-emitting tool on captured stderr, not the exit code; `2>&1 >/dev/null` order; Claude Code hook exit-2 feedback), platforms/toolchains/macos-sdk-sysroot (Homebrew clang needs `-isysroot "$(xcrun --show-sdk-path)"`; `-Wmissing-sysroot` proceeds without headers so the failure lands one step downstream), backend/common/reliability/client-side-rate-limiting (route token/auth requests through the throttle; stamp immediately before send; check `last_request_at=0` initial state). Merged: testing/quality/tests-that-cannot-fail (+uncommitted-mutation restore: copy+hash, `git checkout --` restores from the index and eats the unstaged fix — local git repro), testing/data/test-data-and-isolation (+artifact-leak attribution by prefix counts, static-check enforcement proved red first), platforms/shells/portable-shell-scripts (+`${VAR:-}` empty-vs-unset off-switch trap, bash 3.2/zsh 5.9 repro + GNU manual quote), backend/common/change-impact/call-site-enumeration (+test-helper fan-out edge row). Dropped 1 duplicate: a re-harvest of the call-site-enumeration insight already ingested 2026-08-04. Verification: 4 local reproductions (redirection order, param expansion, git checkout index source, xcrun) + 9 URLs fetched live; one draft source swapped for pages that actually state the claim (Okta rl2-token-oauth → rate-limits overview). Note: heavy overlap with open PRs #32/#34 from earlier flushes of parallel sessions — reviewer should pick one and close the others. diff --git a/wiki/backend/common/change-impact/call-site-enumeration.md b/wiki/backend/common/change-impact/call-site-enumeration.md index 649c9df..7f7081c 100644 --- a/wiki/backend/common/change-impact/call-site-enumeration.md +++ b/wiki/backend/common/change-impact/call-site-enumeration.md @@ -8,8 +8,8 @@ sources: - https://docs.python.org/3/glossary.html - https://docs.python.org/3/library/ast.html - https://peps.python.org/pep-0570/ -last_verified: 2026-08-04 -related: [qa-process-regression-scope, backend-python-language-mutable-state-traps] +last_verified: 2026-08-05 +related: [qa-process-regression-scope, backend-python-language-mutable-state-traps, testing-data-test-data-and-isolation] --- # Enumerating Call Sites Before Changing a Callee's Contract @@ -64,6 +64,7 @@ the search never listed. | Call sites live in another repository or a published package | The change is a versioned deprecation, not an in-place edit: keep the old contract accepting its old shape for a release, and enumerate what you own now | | The language has no keyword arguments at all (JavaScript, Go) | Every site is positional, so a parameter-name search returns nothing at all — enumerate by callee name from the start | | The repo has no working language server for the language | Enumerate by callee name and say so; an AST pass over `Call` nodes is the fallback that survives aliasing | +| A test helper wraps the callee or rebuilds its data shape (a fixture builder feeding it) | Read every helper definition the enumeration surfaces and enumerate the helper's own call sites too — the helper appears once in the callee enumeration while supplying the old contract to every one of its callers ([testing-data-test-data-and-isolation]) | ## Instead of @@ -81,3 +82,4 @@ the search never listed. - https://peps.python.org/pep-0570/ — the `/` marker for positional-only parameters, alongside the existing `*` marker for keyword-only, as the way a signature fixes how an argument may be passed - Local reproduction 2026-08-04 (Python 3.14.6, macOS): over four call sites of `verify(...)` where one passes `repo_rows=` by keyword, a regex search for `repo_rows\s*=` matches 1 while an AST pass over `Call` nodes named `verify` finds 4 — 3 sites invisible to the keyword search - Field incident 2026-08-04 (`linkly-t1-repo-policy`, Python): recon by keyword search reported "13 call sites, 7 need editing"; 8 further seeds passed the same value as `verify()`'s fourth positional argument, and the suite the session had reported green then ran `472 tests / FAILED (failures=11)` +- Field incident 2026-08-05 (`linkly`, Python): a `rows_for()` test helper kept reproducing a removed rule for five call sites while appearing as a single hit in the callee enumeration — helper definitions are fan-out points, not one site diff --git a/wiki/backend/common/reliability/client-side-rate-limiting.md b/wiki/backend/common/reliability/client-side-rate-limiting.md new file mode 100644 index 0000000..fa4923b --- /dev/null +++ b/wiki/backend/common/reliability/client-side-rate-limiting.md @@ -0,0 +1,60 @@ +--- +id: backend-common-reliability-client-side-rate-limiting +domain: backend +category: reliability +applies_to: [general] +confidence: field-tested +sources: + - https://developer.okta.com/docs/reference/rate-limits/ + - https://auth0.com/docs/troubleshoot/customer-support/operational-policies/rate-limit-policy/authentication-api-endpoint-rate-limits +last_verified: 2026-08-05 +related: [backend-common-reliability-timeouts-and-retries] +--- + +# Client-Side Throttles That Miss Auth Requests + +## When this applies + +You added a requests-per-second throttle to an API client wrapper, yet the +provider still returns rate-limit errors — especially on the **first call of a +process**, or only on some days. Also when designing the throttle layer of any +client whose requests carry a token the client itself refreshes. + +## Do this + +1. **Route every HTTP request through the throttle, including token/auth + acquisition.** Identity providers rate-limit their auth endpoints like any + other endpoint (Auth0 limits its Authentication API endpoints; Okta's + org-wide rate-limit buckets cover the OAuth2 endpoints), and a token POST + plus the first real API call land in the same second — deterministically + exceeding a low per-second cap. +2. **Audit the interceptor path.** Token refresh usually happens inside a + header-builder or request interceptor, which sits *below* a wrapper-level + throttle and silently bypasses it. The throttle must wrap the layer that + actually performs HTTP, not the layer that composes calls. +3. **Stamp the throttle timestamp immediately before the request goes out**, + not at wrapper entry — work done between the stamp and the send (like a + nested token fetch) otherwise consumes the gap the stamp claimed. +4. **Check the initial state.** A `last_request_at = 0` default makes the + first gap check pass trivially; the first two physical requests of the + process then go out unthrottled. + +## Edge cases + +| Case | Then | +|------|------| +| The failure reproduces only on some days and looks like provider flakiness | Correlate failure timestamps with token issuance in logs: a cached token skips the extra request, so the bug only fires when the cache is cold/expired — that schedule-shaped intermittency is the signature | +| The provider counts limits per endpoint, not globally | The token POST may have its own (often stricter) limit; throttling it with the data calls is still safe, but its own 429 handling needs backoff too | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Dismiss first-call rate-limit errors as intermittent provider issues | Diff a failing day's log against a working day's around the first call | The extra token request is visible as one added line; the "intermittency" is the token cache's TTL | +| Throttle at the public-method layer of the client | Throttle at the transport layer every request traverses | Auth refresh, retries, and pagination helpers all issue requests the public layer never sees | + +## Sources + +- https://developer.okta.com/docs/reference/rate-limits/ — Okta's org-wide rate-limit buckets cover the OAuth2 endpoints; only the public metadata endpoints (`/oauth2/v1/keys`, the `.well-known` documents) are exempt +- https://auth0.com/docs/troubleshoot/customer-support/operational-policies/rate-limit-policy/authentication-api-endpoint-rate-limits — Auth0 limits the number of requests made to Authentication API endpoints, which include the token endpoint +- Field incident 2026-08-05 (`stock-trader` `kis_client.py`, 2 req/s provider cap): `_headers()` called `_get_token()` *after* `_throttle()`, so on token-issue days the log shows token POST at 00.354 → token issued 00.495 → balance call rejected 00.543; on cached-token days the identical code passed, which had the failure filed as intermittent diff --git a/wiki/backend/common/reliability/timeouts-and-retries.md b/wiki/backend/common/reliability/timeouts-and-retries.md index df130d6..6a29ee1 100644 --- a/wiki/backend/common/reliability/timeouts-and-retries.md +++ b/wiki/backend/common/reliability/timeouts-and-retries.md @@ -9,7 +9,7 @@ sources: - https://sre.google/sre-book/addressing-cascading-failures/ - https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ last_verified: 2026-07-10 -related: [backend-common-api-design-idempotency, backend-common-llm-completion-response-validation] +related: [backend-common-api-design-idempotency, backend-common-llm-completion-response-validation, backend-common-reliability-client-side-rate-limiting] --- # Calling Another Service over the Network: Timeouts, Retries, Backoff diff --git a/wiki/backend/index.md b/wiki/backend/index.md index d7fc90b..f83de50 100644 --- a/wiki/backend/index.md +++ b/wiki/backend/index.md @@ -37,6 +37,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [timeouts-and-retries](common/reliability/timeouts-and-retries.md) | Your service calls another service/external API/DB over the network — setting timeouts and deadlines, deciding what to retry per failure type, backoff/jitter, capping concurrency against a slow dependency; debugging pool exhaustion or retry storms | +| [client-side-rate-limiting](common/reliability/client-side-rate-limiting.md) | A client-side request throttle exists yet the provider still returns rate-limit errors — especially on a process's first call, or only on some days; designing the throttle layer of a client that refreshes its own auth token | ### caching diff --git a/wiki/platforms/environment/path-resolution.md b/wiki/platforms/environment/path-resolution.md index 7a54340..d43d3bf 100644 --- a/wiki/platforms/environment/path-resolution.md +++ b/wiki/platforms/environment/path-resolution.md @@ -12,7 +12,7 @@ sources: - https://www.sudo.ws/docs/man/sudoers.man/ - https://docs.brew.sh/FAQ last_verified: 2026-08-04 -related: [platforms-toolchains-version-management, platforms-processes-background-services] +related: [platforms-toolchains-version-management, platforms-processes-background-services, platforms-toolchains-macos-sdk-sysroot] --- # The Wrong Binary (or None) Resolving From PATH diff --git a/wiki/platforms/index.md b/wiki/platforms/index.md index 844b436..cfae82d 100644 --- a/wiki/platforms/index.md +++ b/wiki/platforms/index.md @@ -15,7 +15,8 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| -| [portable-shell-scripts](shells/portable-shell-scripts.md) | Writing a shell script that must run on more than one machine/OS/shell or in CI; a script that works locally fails elsewhere; choosing a shebang (bash vs sh); a bash script misbehaves in zsh or vice versa (unquoted vars, `=word`, array indexing); deciding how `set -euo pipefail` protects (and doesn't); building argument lists safely | +| [portable-shell-scripts](shells/portable-shell-scripts.md) | Writing a shell script that must run on more than one machine/OS/shell or in CI; a script that works locally fails elsewhere; choosing a shebang (bash vs sh); a bash script misbehaves in zsh or vice versa (unquoted vars, `=word`, array indexing); deciding how `set -euo pipefail` protects (and doesn't); building argument lists safely; an empty env override (`VAR=`) fails to disable a feature | +| [warnings-on-stderr-with-exit-zero](shells/warnings-on-stderr-with-exit-zero.md) | Wiring a compiler/linter/build tool into a CI step, pre-commit check, or agent hook and the tool emits warnings on stderr while exiting 0; a gate keyed on the exit code lets warnings through silently; capturing stderr for feedback without mixing in the tool's stdout | | [command-text-inspected-before-execution](shells/command-text-inspected-before-execution.md) | A hook, policy gate, allow-list, or audit rule blocked a command that is correct as written; composing a command that must satisfy such a gate first try; deciding whether to write a path literally or as `"$VAR"` in an inspected argument; a gate reports an argument missing or a file nonexistent though both are right; a gate must read a file your command creates; prose containing a dangerous-looking command (release notes, docs, fixtures) trips a text scanner | ## tools @@ -51,6 +52,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [version-management](toolchains/version-management.md) | "Works on my machine" from tool-version drift; a project needs a pinned language/tool version (.nvmrc, .python-version, .tool-versions); making CI use the same versions as local; onboarding a machine reproducibly; a script/cron/CI step can't find a version-managed binary (shims absent in non-interactive shells); deciding where lockfiles fit in reproducibility | +| [macos-sdk-sysroot](toolchains/macos-sdk-sysroot.md) | Homebrew keg-only clang on macOS cannot find system headers (`stdio.h`) while Apple's `/usr/bin/clang` works; mass test failures through a toolchain right after a CommandLineTools/Xcode change; deciding how a build script should locate the macOS SDK | ## Planned (unseeded categories) diff --git a/wiki/platforms/shells/command-text-inspected-before-execution.md b/wiki/platforms/shells/command-text-inspected-before-execution.md index e69daff..ac133d6 100644 --- a/wiki/platforms/shells/command-text-inspected-before-execution.md +++ b/wiki/platforms/shells/command-text-inspected-before-execution.md @@ -8,7 +8,7 @@ sources: - https://code.claude.com/docs/en/hooks - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html last_verified: 2026-07-30 -related: [platforms-shells-portable-shell-scripts, platforms-environment-path-resolution] +related: [platforms-shells-portable-shell-scripts, platforms-environment-path-resolution, platforms-shells-warnings-on-stderr-with-exit-zero] --- # Commands Read as Text by a Gate Before the Shell Runs Them diff --git a/wiki/platforms/shells/portable-shell-scripts.md b/wiki/platforms/shells/portable-shell-scripts.md index 23eed9b..bf3839f 100644 --- a/wiki/platforms/shells/portable-shell-scripts.md +++ b/wiki/platforms/shells/portable-shell-scripts.md @@ -6,12 +6,13 @@ applies_to: [bash, zsh, posix-sh] confidence: verified sources: - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html + - https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html - https://zsh.sourceforge.io/Doc/Release/Expansion.html - https://zsh.sourceforge.io/Doc/Release/Parameters.html - https://google.github.io/styleguide/shellguide.html - https://www.shellcheck.net/ -last_verified: 2026-07-10 -related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management, platforms-shells-command-text-inspected-before-execution] +last_verified: 2026-08-05 +related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management, platforms-shells-command-text-inspected-before-execution, platforms-shells-warnings-on-stderr-with-exit-zero] --- # Shell Scripts That Must Run on More Than One Machine or Shell @@ -62,6 +63,7 @@ non-interactive environment). | Critical command is in a pipeline but the interpreter is POSIX sh (no `pipefail`) | Run the critical command outside the pipeline (temp file between stages) and test `$?` directly | | Script runs via cron/CI/hooks and commands are "not found" | Non-interactive shells load no rc files — no user PATH, no version-manager shims. Call binaries by absolute path (see platforms-toolchains-version-management) | | `set -u` breaks on optional variables | Expand with an explicit default: `"${OPT:-}"` | +| Passing `VAR=` (empty) to disable a feature has no effect | `${VAR:-default}` substitutes for unset **and** empty — omitting the colon (`${VAR-default}`) tests only for unset. Make the script read `${VAR-default}` when empty must mean "off"; when you cannot edit the script, pass a value its own validation rejects (e.g. `WATCH_TMUX=/nonexistent` so a `command -v` probe fails) | ## Instead of @@ -70,10 +72,12 @@ non-interactive environment). | Build a command string and `eval` it | Build an array and expand it: `cmd "${args[@]}"` | `eval` re-parses quotes and globs; arrays pass arguments through exactly | | Put a command plus its flags in one variable and run `$cmd` | Variable holds the binary path only; flags are separate words | zsh runs the whole value as one command name; bash re-splits and re-globs it | | Trust a trailing `echo "done"` as proof a step ran | Verify the produced state with an independent command | Inside `&&` chains and subshells, `set -e` misses failures and the echo still prints | +| Disable a script feature by exporting an empty value against a `${VAR:-default}` read | Pass a deliberately invalid value that fails the script's own probe, or change the read to `${VAR-default}` | The colon form treats empty as unset, silently re-enabling the default you meant to turn off | ## Sources - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html — POSIX shell quoting and field splitting (sections 2.2, 2.6.5) +- https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html — "Omitting the colon results in a test only for a parameter that is unset"; behavior reproduced identically in bash 3.2 and zsh 5.9 (2026-08-05) - https://zsh.sourceforge.io/Doc/Release/Expansion.html — zsh: no word splitting of unquoted parameters (14.3); `=word` expansion (14.7.3) - https://zsh.sourceforge.io/Doc/Release/Parameters.html — zsh arrays numbered from 1 (KSH_ARRAYS excepted) - https://google.github.io/styleguide/shellguide.html — quote variables, prefer bash for scripts, arrays over eval diff --git a/wiki/platforms/shells/warnings-on-stderr-with-exit-zero.md b/wiki/platforms/shells/warnings-on-stderr-with-exit-zero.md new file mode 100644 index 0000000..5607bd7 --- /dev/null +++ b/wiki/platforms/shells/warnings-on-stderr-with-exit-zero.md @@ -0,0 +1,71 @@ +--- +id: platforms-shells-warnings-on-stderr-with-exit-zero +domain: platforms +category: shells +applies_to: [bash, zsh, posix-sh] +confidence: verified +sources: + - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html + - https://www.gnu.org/software/bash/manual/html_node/Redirections.html + - https://code.claude.com/docs/en/hooks +last_verified: 2026-08-05 +related: [platforms-shells-portable-shell-scripts, platforms-shells-command-text-inspected-before-execution] +--- + +# Feeding a Tool's Warnings Into a Gate When It Exits Zero + +## When this applies + +Wiring a compiler, linter, or build tool into an automated gate — a CI step, a +pre-commit check, or an agent hook (e.g. Claude Code `PostToolUse`) — and the +tool emits its warnings on stderr while still exiting 0. Also when such a gate +exists but warnings sail through it silently. + +## Do this + +1. **Branch on captured stderr, not on the exit code.** Warnings are by + definition not failures, so most tools exit 0 when they emit them; a gate + keyed on the exit code misses 100% of warnings and cannot tell "clean" from + "warned". + +2. **Capture stderr alone with `OUT=$(tool "$F" 2>&1 >/dev/null)`.** POSIX + evaluates redirections "from beginning to end": `2>&1` first duplicates + stderr onto the current stdout — which inside `$(...)` is the capture pipe — + then `>/dev/null` discards the tool's own stdout. Build artifacts and IR + printed to stdout stay out of the captured text. + +3. **Distinguish three outcomes, not two:** + +| Observed | Meaning | Gate action | +|----------|---------|-------------| +| exit ≠ 0 | Error | Fail the gate; forward stderr | +| exit 0, captured stderr non-empty | Warnings | Forward the warning text to the loop (below) | +| exit 0, captured stderr empty | Clean | Pass silently | + +4. **In a Claude Code hook, return the text via exit 2 with the warnings on + stderr.** The hooks contract: on exit 2 "stderr text is fed back to Claude as + an error message". For `PostToolUse` the tool has already run, so exit 2 does + not block — it shows the stderr to the model, which is exactly the feedback + loop a warning needs. + +## Edge cases + +| Case | Then | +|------|------| +| The tool prints diagnostics to **stdout** instead | Verify where diagnostics go before wiring the gate: run one known-warning input and inspect both streams separately | +| The tool offers a `--werror`-style flag promoting warnings to failures | Prefer the flag when the policy is "warnings fail the build"; keep stderr capture when warnings should inform but not block | +| The gate must also preserve the tool's stdout for a later stage | Redirect stdout to a file instead of `/dev/null`: `OUT=$(tool "$F" 2>&1 >"$ART")` | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Gate on `if tool "$F"; then ... fi` (exit code only) | Capture stderr and test it for content as well | Warnings exit 0; the exit-code gate passes them indistinguishably from clean runs | +| Write the capture as `2>/dev/null >&1` or `>/dev/null 2>&1` inside `$(...)` | `2>&1 >/dev/null` — stderr onto the capture pipe first, then discard stdout | Redirections apply left to right; the reversed order discards stderr or captures stdout, mixing build output into the feedback | + +## Sources + +- https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html — 2.7 Redirection: "If more than one redirection operator is specified with a command, the order of evaluation is from beginning to end" +- https://www.gnu.org/software/bash/manual/html_node/Redirections.html — "Redirections are processed in the order they appear, from left to right", with the `ls > dirlist 2>&1` vs `ls 2>&1 > dirlist` example showing why the order matters +- https://code.claude.com/docs/en/hooks — exit 2: "stderr text is fed back to Claude as an error message"; PostToolUse cannot block ("the tool already ran") but shows stderr to Claude +- Local reproduction 2026-08-05 (`lnpl` compiler, macOS): warning input → rc=0 with 3 warnings on stderr; minimal clean input → rc=0, empty stderr; reserved-word error → rc=2 with errors on stderr — the three-way split above observed directly diff --git a/wiki/platforms/toolchains/macos-sdk-sysroot.md b/wiki/platforms/toolchains/macos-sdk-sysroot.md new file mode 100644 index 0000000..0d52325 --- /dev/null +++ b/wiki/platforms/toolchains/macos-sdk-sysroot.md @@ -0,0 +1,63 @@ +--- +id: platforms-toolchains-macos-sdk-sysroot +domain: platforms +category: toolchains +applies_to: [macos] +confidence: verified +sources: + - https://clang.llvm.org/docs/DiagnosticsReference.html + - https://discourse.llvm.org/t/stdio-h-not-found-on-mac-how-to-add-system-headers-includes-into-clang/77604 + - https://github.com/Homebrew/homebrew-core/issues/45061 +last_verified: 2026-08-05 +related: [platforms-environment-path-resolution, platforms-toolchains-version-management] +--- + +# System Headers for Homebrew clang on macOS (sysroot) + +## When this applies + +Compiling C/C++ on macOS with a Homebrew keg-only LLVM +(`/opt/homebrew/opt/llvm/bin/clang`) and system headers (`stdio.h`, +`stdlib.h`) come back "file not found"; or a build that works with Apple's +`/usr/bin/clang` fails under Homebrew clang; or a test suite driving that +toolchain fails en masse with header errors. + +## Do this + +1. **Pass the SDK explicitly:** add `-isysroot "$(xcrun --show-sdk-path)"` to + the compile command, and make the build script that invokes the toolchain + inject it rather than relying on each caller to remember. +2. **Know the failure shape so you don't misroute the diagnosis.** Homebrew + clang defaults to a CommandLineTools SDK path baked in at build time. When + CommandLineTools is absent or its versioned SDK directory has moved, clang + does **not** stop: it emits only the `-Wmissing-sysroot` warning (on by + default) and proceeds without system headers, so the run dies one step later + on "file not found" — a symptom that reads like a code defect in whatever + you were compiling. +3. **Read the warning text** — it prints the exact sysroot path clang tried, so + the missing/renamed SDK directory is named in the output. +4. `xcrun --show-sdk-path` resolves the currently selected SDK dynamically, so + the injected value survives Xcode/CommandLineTools upgrades that break a + hardcoded path. + +## Edge cases + +| Case | Then | +|------|------| +| Many tests fail at once through the toolchain | Reduce to a one-file probe (`clang probe.c` with `#include `) before touching the code under test — same error confirms toolchain, not code | +| The build system reads an environment variable instead of flags | `export SDKROOT="$(xcrun --show-sdk-path)"` is the environment-variable equivalent clang honors | +| Apple's `/usr/bin/clang` works but Homebrew's does not | That differential is the signature of this problem: the Apple driver resolves the SDK via xcrun automatically; the Homebrew build trusts its baked-in path | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Debug the source files after mass "header not found" failures | Probe-compile a trivial file with the same toolchain first | The failure point is one step downstream of the cause; the probe separates toolchain from code in seconds | +| Hardcode `/Library/Developer/CommandLineTools/SDKs/MacOSX*.sdk` | Inject `$(xcrun --show-sdk-path)` | The versioned path disappears on CLT upgrade/removal; xcrun re-resolves it | + +## Sources + +- https://clang.llvm.org/docs/DiagnosticsReference.html — `-Wmissing-sysroot` exists and is enabled by default (a warning, not an error) +- https://discourse.llvm.org/t/stdio-h-not-found-on-mac-how-to-add-system-headers-includes-into-clang/77604 — non-Apple clang on macOS needs the SDK pointed at explicitly (`-isysroot`/SDKROOT) +- https://github.com/Homebrew/homebrew-core/issues/45061 — Homebrew clang does not search the macOS system include directories Apple's driver finds +- Local reproduction 2026-08-05 (macOS, Homebrew LLVM): 69 toolchain-driven tests failing → `clang probe.c` reproduced the identical header error → `clang -isysroot "$(xcrun --show-sdk-path)" probe.c` succeeded; the `-Wmissing-sysroot` output named the vanished `.../SDKs/MacOSX26.sdk` path diff --git a/wiki/platforms/toolchains/version-management.md b/wiki/platforms/toolchains/version-management.md index df433b4..513d40e 100644 --- a/wiki/platforms/toolchains/version-management.md +++ b/wiki/platforms/toolchains/version-management.md @@ -10,7 +10,7 @@ sources: - https://mise.jdx.dev/configuration.html - https://docs.npmjs.com/cli/v11/configuring-npm/package-json last_verified: 2026-07-10 -related: [platforms-processes-background-services, platforms-shells-portable-shell-scripts] +related: [platforms-processes-background-services, platforms-shells-portable-shell-scripts, platforms-toolchains-macos-sdk-sysroot] --- # Pinning Tool Versions So Every Machine Runs the Same Toolchain diff --git a/wiki/testing/data/test-data-and-isolation.md b/wiki/testing/data/test-data-and-isolation.md index c883909..b2002d7 100644 --- a/wiki/testing/data/test-data-and-isolation.md +++ b/wiki/testing/data/test-data-and-isolation.md @@ -8,8 +8,8 @@ sources: - https://martinfowler.com/articles/nonDeterminism.html - https://abseil.io/resources/swe-book/html/ch12.html - https://testing.googleblog.com/2017/01/testing-on-toilet-keep-cause-and-effect.html -last_verified: 2026-08-04 -related: [testing-flaky-diagnosing-flaky-tests, testing-strategy-test-level-choice] +last_verified: 2026-08-05 +related: [testing-flaky-diagnosing-flaky-tests, testing-strategy-test-level-choice, backend-common-change-impact-call-site-enumeration] --- # Owning Test Data and Isolating Test State @@ -56,6 +56,7 @@ state-leak symptom. | Failure appears only in the full suite, never alone | Run the suite in random order to expose the order dependency, then bisect to the polluting test; fix the polluter's ownership, not the victim ([testing-flaky-diagnosing-flaky-tests]) | | Test needs "now"-relative data but the code reads the system clock directly | Refactor the code to accept an injected clock; that seam is the fix — assertions with tolerance windows around real time stay flaky | | A group of tests fails as a lookup miss, an empty result, or a "not found" far from any fixture code | Compare each factory's defaulted values against the input the test actually runs before migrating fixture shape; when the two disagree, the fixture was built for a different input and only the signature change fixes the group | +| Leftover test artifacts (temp dirs/files) accumulate in the repo and the producers look diffuse | Count leftovers by name prefix (`ls \| sed 's/-[a-z0-9]*$//' \| sort \| uniq -c`) and match the distribution against the sites that create such files — a match closes the attribution; fix those sites, then enforce the cleanup convention with a static check over the test tree, proving the check red against the unfixed code first | ## Instead of @@ -66,6 +67,7 @@ state-leak symptom. | `sleep()` until background work lands the data | Wait explicitly on the condition/event with a timeout | Sleeps are both too slow and too short; the race remains | | Copy a full production-like JSON blob as fixture for one field's behavior | Build the minimal object via a factory, explicit only in that field | Giant fixtures hide the relevant value and break on unrelated schema changes | | Let a factory seed itself from a module-level constant while the test passes a different value to the code under test | Take that value as a factory parameter and name it at every call site | The fixture is then built for the input the test actually runs; a defaulted constant makes the two drift apart with nothing in the test body showing it | +| Add cleanup at every temp-file call site a grep finds | Attribute first by prefix counts, fix the dominant producers, then add a static check for the convention the compliant files already follow | Leak volume concentrates in a few sites; matching counts to call sites confirms the cause without guessing, and the static check stops the recurrence that an instance-only fix invites | ## Sources @@ -73,3 +75,4 @@ state-leak symptom. - https://abseil.io/resources/swe-book/html/ch12.html — a test is complete when "its body contains all of the information a reader needs in order to understand how it arrives at its result"; prefer DAMP over DRY, and where a helper is used, give it "descriptive parameters that make dependencies explicit" rather than reusing shared constants - https://testing.googleblog.com/2017/01/testing-on-toilet-keep-cause-and-effect.html — keep the inputs a test's result depends on visible in the test method instead of in shared setup, so the cause-and-effect relationship is readable without jumping elsewhere - Field incident 2026-08-04 (`linkly-t1-repo-policy`, Python): `rows_for(doc)` seeded its rows from the module constant `PAYLOAD` while its tests ran payload `{}`; a shape-only migration of the helper fixed 1 of 11 failures, and moving the payload into the helper's signature fixed 11 of 11 +- Field incident 2026-08-05 (`linkly`, Python): 998 leftover temp entries resolved to two name prefixes (686 + 306); of six `mkdtemp` call sites, exactly the two without cleanup matched those prefixes — after fixing them and adding an AST-based cleanup check (proved red first), a full-suite run left a measured tmp delta of 0 (72M → 3.3M) diff --git a/wiki/testing/index.md b/wiki/testing/index.md index efb8ac8..720f4b9 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -22,7 +22,7 @@ Match your situation to a "load when" line; load only matching pages. |------|-----------| | [minimum-case-set](quality/minimum-case-set.md) | Writing tests for a function/endpoint/change and choosing which cases to cover; reviewing whether coverage suffices; picking boundary values by input type; adding a regression test for a bug fix | | [behavior-not-implementation](quality/behavior-not-implementation.md) | Deciding what a test should assert; a behavior-preserving refactor broke tests; tempted to expose privates for testing; deciding whether a snapshot test is appropriate | -| [tests-that-cannot-fail](quality/tests-that-cannot-fail.md) | Reviewing tests that always pass; a bug shipped through an area the suite reported as covered; auditing a suspiciously green suite; judging whether an assertion, error-path test, or mock-based test can actually detect a defect | +| [tests-that-cannot-fail](quality/tests-that-cannot-fail.md) | Reviewing tests that always pass; a bug shipped through an area the suite reported as covered; auditing a suspiciously green suite; judging whether an assertion, error-path test, or mock-based test can actually detect a defect; restoring a file after a deliberate red-run mutation while the fix under test is uncommitted | | [checks-that-cannot-pass](quality/checks-that-cannot-pass.md) | Authoring a check whose target does not exist yet (grep/regex gate on an unwritten file or doc section, lint/scan rule, schema assertion on an unbuilt endpoint, a plan's verification command) and it has only ever been observed failing; reviewing a plan's gates before adopting them; separating "target missing" from "content missing" in a gate's exit status | | [spec-artifact-checks](quality/spec-artifact-checks.md) | Writing or reviewing an automated check that a mapping table covers every rule/field/enum case, or that ids resolve across documents; deciding whether a green check earned "verified" or only "present"; designing one negative control per check in a multi-check harness; parsing Markdown table rows programmatically in a doc-as-spec repo | | [schema-additions-under-a-golden-gate](quality/schema-additions-under-a-golden-gate.md) | Adding a node kind, variant, discriminator value, or field to a document format (IR, JSON Schema, spec artifact) whose only automated gate builds its negatives by mutating one committed golden example; the gate or the whole suite comes back green right after a schema change; deciding which negative each new schema keyword needs, and whether a green suite that never loads the schema is evidence at all | @@ -32,7 +32,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| -| [test-data-and-isolation](data/test-data-and-isolation.md) | Tests need fixture data and you are choosing how to create it; tests pass alone but fail together (or vice versa); DB cleanup, shared fixtures, time-dependent logic, or unique-value collisions | +| [test-data-and-isolation](data/test-data-and-isolation.md) | Tests need fixture data and you are choosing how to create it; tests pass alone but fail together (or vice versa); DB cleanup, shared fixtures, time-dependent logic, or unique-value collisions; leftover test artifacts (temp dirs/files) accumulating in the repo | ## mocking diff --git a/wiki/testing/quality/tests-that-cannot-fail.md b/wiki/testing/quality/tests-that-cannot-fail.md index dcb33f7..009c3d3 100644 --- a/wiki/testing/quality/tests-that-cannot-fail.md +++ b/wiki/testing/quality/tests-that-cannot-fail.md @@ -10,7 +10,7 @@ sources: - https://testing.googleblog.com/2021/04/mutation-testing.html - https://martinfowler.com/bliki/TestCoverage.html - https://testing.googleblog.com/2013/05/testing-on-toilet-dont-overuse-mocks.html -last_verified: 2026-07-10 +last_verified: 2026-08-05 related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-checks-that-cannot-pass, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, testing-quality-schema-additions-under-a-golden-gate, qa-document-verification-spec-document-gates] --- @@ -49,6 +49,7 @@ suite reported as covered, or you are auditing a suspiciously green suite. | Case | Then | |------|------| | Mutating the code under test is impractical right now (slow build, shared branch) | Invert the expected value in the assertion instead and require red — this proves the assertion executes and compares, though not which code defects it catches | +| The file you are about to mutate contains uncommitted work (the fix under test is not committed) | Save a copy and its hash before mutating (`cp f f.bak; shasum f`), restore from the copy, and verify the hash — `git checkout -- ` replaces the file with the index version, which still holds the pre-fix content | | Auditing a whole suite, not one test | Run an automated mutation-testing tool (PIT, Stryker) and treat surviving mutants in changed code as missing or defective tests | | The mutation run is your own script rather than PIT/Stryker | Prove the harness discriminates before citing its score — a semantics-preserving no-op must survive ([testing-quality-harness-reverse-controls]) | | A test intentionally has no outcome assertion (smoke test: module loads, page renders) | Keep it only when the regression it guards manifests as a throw; name it as a smoke test so reviewers do not count it as behavior coverage | @@ -62,6 +63,7 @@ suite reported as covered, or you are auditing a suspiciously green suite. | Prove an error path with `try { await f() } catch (e) { expect(e.message)... }` alone | Use `rejects`/`assertThrows`-style assertion, or add `expect.assertions(1)` above the try | When `f()` succeeds, the catch never runs and the test passes with zero assertions | | Trust "green suite + high coverage" as proof an area is tested | Break the behavior once and require a red run | Coverage counts execution, not detection; high numbers are reachable with assertion-free tests | | Delete a suspicious always-green test to clean up | Fix it via the table above, then re-verify it can fail | The test names a behavior someone meant to guard; deletion drops the intent along with the defect | +| Undo a red-run mutation with `git checkout -- ` while the fix under test is uncommitted | Restore from a copy saved before mutating and compare hashes | Undoing the mutation and undoing the unstaged fix are the same checkout; the loss surfaces later as a smaller collected-test count, not as a failure | ## Sources @@ -70,3 +72,6 @@ suite reported as covered, or you are auditing a suspiciously green suite. - https://testing.googleblog.com/2021/04/mutation-testing.html — inserting faults and requiring test failure measures whether tests detect bugs; coverage alone does not - https://martinfowler.com/bliki/TestCoverage.html — coverage finds untested code; it is not a measure of test quality - https://testing.googleblog.com/2013/05/testing-on-toilet-dont-overuse-mocks.html — mock-heavy tests can pass while the real code is broken +- https://git-scm.com/docs/git-checkout — `git checkout [--] `: "Replace the specified files ... with the version from the index"; unstaged changes are discarded +- Local reproduction 2026-08-05 (git, macOS): baseline commit + uncommitted fix + mutation → `git checkout -- f.py` returned the file to the baseline (fix lost); a staged fix survived the same checkout (the index, not HEAD, is the restore source); a copy+hash restore round-tripped identically +- Field incident 2026-08-05 (`linkly-t1-retry-ceiling`, Python): a `git checkout --` after a mutation check silently deleted the uncommitted implementation; the lost import failed test collection, so the suite reported `Ran 1042 tests … errors=1` instead of 1098 — a green-looking run with 56 tests missing