diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 439eb5e..7172a68 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,102 +1,192 @@ # Knowledge flush — 3 insight(s) -Three `★ Insight` candidates drained from `~/.dev-loop/queue`. Each was -researched, verified, deduped against existing pages, and **merged into an -existing page** (merge-before-create — no new pages, no new categories). +Queue drained: `~/.dev-loop/queue/0c6a5439-….jsonl` (1 row), +`~/.dev-loop/queue/fb7e7221-….jsonl` (2 rows). All three were harvested +2026-08-04 from the `linkly-t1-spec-notation` and `linkly-t1-repo-policy` repos. + +Result: **2 new pages, 1 merge into an existing page, 1 new category.** + +| # | Insight (trigger → directive) | Outcome | Confidence | +|---|-------------------------------|---------|------------| +| 1 | Adding a node kind to a format whose only gate mutates a golden example → commit a fixture holding the new kind, one negative per keyword, verify each reddens | **New page** `testing/quality/schema-additions-under-a-golden-gate` | verified | +| 2 | Enumerating call sites before a contract change → enumerate by callee name; a parameter-name search is a partial index | **New page + new category** `backend/common/change-impact/call-site-enumeration` | verified | +| 3 | A fixture helper whose shape depends on a value the test also passes to the SUT → put that value in the helper's signature | **Merged** into `testing/data/test-data-and-isolation` | verified | + +--- ## Verified best-practice -### 1. Homebrew keg-only formulae are installed but deliberately off PATH -- **Claim:** `which ` / `command -v ` returning "not found" on macOS - does **not** mean the toolchain is absent — Homebrew keg-only formulae (llvm, - openssl, curl, node@N, libpq) are built into the Cellar but not symlinked into - the prefix, so they are off PATH by design. Check the keg directly - (`/opt/homebrew/opt//bin/`, `brew info `) before deferring. -- **Sources checked:** — verbatim: "the formula is - installed only into the Cellar and is not linked into the default prefix … - most tools will not find it"; "You can see why a formula was installed as - keg-only, and instructions for including it in your `PATH`, by running - `brew info `." The active version stays reachable at - `$(brew --prefix)/opt//bin` regardless of version. -- **How verified:** official Homebrew docs + the field reproduction on record - (`which mlir-opt` → not found, while `/opt/homebrew/opt/llvm/bin/mlir-opt - --version` → LLVM 22.1.8). -- **Confidence: verified.** - -### 2. tmux/REPL prompt injection stalls on bracketed paste; submit is a separate keystroke -- **Claim:** injecting a long/multiline prompt into an interactive REPL (tmux - `send-keys -l`, `expect`, a PTY) and seeing it stall as a collapsed paste - (`❯ [Pasted text #1]`) means the REPL captured it as one **bracketed-paste** - block; the embedded newline is not treated as submit, so a **separate** Enter - (a couple seconds later) is required — or clear with `C-u` and re-inject. -- **Sources checked:** — the - terminal wraps a paste in `ESC[200~`/`ESC[201~` so the application treats it as - one block and does not act on embedded control characters (newlines) as - keypresses. Corroborated by - (tmux multi-line paste under `extended-keys-format csi-u`). -- **How verified:** the bracketed-paste mechanism is doc-verified; the specific - Claude-CLI "first Enter confirms the paste, second Enter submits" behavior is - the session's own reproduction (three orchestration sessions all stalled at - `[Pasted text #1]` until a second Enter flipped them to `esc to interrupt`). -- **Confidence: field-tested** (mechanism doc-verified; the CLI submit specifics - are production observation, not an official CLI doc). - -### 3. A "pure" function's test is not dependency-free if its module runs I/O at import -- **Claim:** importing a module to unit-test a pure function within it runs the - module's import-time side effects (module-scope `init_db()`, a top-level DB/HTTP - client), so the test is really integration. A function-level - `@pytest.mark.skipif` cannot prevent it, because skipif is evaluated at - collection **after** the test module (and its top-level imports) has been - imported. Gate at module load (`pytest.importorskip`, `pytest.skip(..., - allow_module_level=True)`) or move the function to a side-effect-free module. -- **Sources checked:** — - skipif's condition "is evaluated at collection time"; skip an entire module at - import with `pytest.skip(reason, allow_module_level=True)`; use - `pytest.importorskip` at module level for a missing import. -- **How verified:** official pytest docs for the timing/mechanism + the field - case (`src/web/app.py` calling `init_db()` at import, forcing Postgres onto a - `from app import _polygon_centroid` unit test). -- **Confidence: verified.** +Every URL below was fetched during this flush; the quotes are from those +fetches, not from memory. Nothing was cited that I did not open. + +### Insight 1 — a golden-derived negative corpus cannot reach a newly added schema branch + +*Claim under test:* when a format's only gate builds negatives by mutating one +committed golden example, adding a new node kind to the schema produces a green +run that proves nothing about the addition. + +| Source checked | What it establishes | +|----------------|---------------------| +| https://json-schema.org/understanding-json-schema/reference/conditionals | The mechanism, verbatim: *"If `if` is invalid, `else` must also be valid (and `then` is ignored)"* — a branch keyed on the new kind is simply **not applied** to an instance that lacks it. So a mutant of a golden without the new kind cannot exercise the new branch. | +| https://json-schema.org/understanding-json-schema/reference/object | Why the new branch also needs constraining before a negative can even exist: *"By default any additional properties are allowed"* and *"By default, the properties defined by the `properties` keyword are not required"*. | +| https://json-schema.org/draft/2020-12/json-schema-core | Sibling applicators *"MUST NOT impact the results of sibling subschemas"*; and *"Unknown keywords SHOULD be treated as annotations"* — a misspelled keyword in a new branch is ignored rather than rejected, a second silent-pass mode. | +| https://pitest.org/quickstart/basic_concepts/ | The same gap has a name in mutation tooling: *"**No coverage**: the same as **Survived** except there were no tests that exercised the line of code where the mutation was created"* — cited so the page tells a reader using PIT/Stryker what the symptom looks like there. | + +*Session evidence (kept as a field observation, not as the basis of the +directive):* in `linkly-t1-spec-notation`, `grep -rln "lir.schema\|jsonschema" +impl/tests/` returned 0 of 447 tests, and the only schema gate over `*.lir.json` +was `scripts/validate_ir.py --self-test`, whose three negatives are all +`copy.deepcopy` mutations of `examples/login.lir.json`. + +**Confidence: `verified`** — the directive's mechanism is stated in the official +JSON Schema documentation; the incident is corroborating, not load-bearing. + +### Insight 2 — enumerate by callee name, not by parameter name + +*Claim under test:* a keyword-argument search (`repo_rows=`) is structurally +incapable of finding call sites that pass the same argument positionally. + +| Source checked | What it establishes | +|----------------|---------------------| +| https://docs.python.org/3/glossary.html | *positional-or-keyword* — *"specifies an argument that can be passed either positionally or as a keyword argument. **This is the default kind of parameter**"*. The blindness is therefore the default case, not an edge case. | +| https://docs.python.org/3/library/ast.html | `ast.Call`: *"`args` holds a list of the arguments passed by position"*, *"`keywords` holds a list of `keyword` objects representing arguments passed by keyword"* — the two forms live in **separate fields**, so a keyword-name text search reads only one of them. | +| https://peps.python.org/pep-0570/ | The `/` and `*` markers, which is what makes the page's "declare it keyword-only so a stale positional call errors instead of rebinding" edge case actionable. | + +*Reproduction run this session* (Python 3.14.6, macOS, no files written — piped +to `python3` on stdin): over four call sites of `verify(...)` of which one passes +`repo_rows=` by keyword, a regex search for `repo_rows\s*=` matches **1** while +an AST pass over `Call` nodes named `verify` finds **4** — 3 sites invisible to +the keyword search. This is the minimal version of the reported incident. + +*Session evidence:* recon reported "13 call sites, 7 need editing"; 8 further +seeds passed the value as `verify()`'s 4th positional argument, and the suite the +session had reported green then ran `472 tests / FAILED (failures=11)`. + +**Confidence: `verified`** — official language reference plus a reproduction. + +### Insight 3 — a fixture factory takes the value the test also passes to the SUT + +*Claim under test:* when a fixture helper's shape depends on a value the test +also feeds the system under test, that value belongs in the helper's signature +rather than in a module-level default. + +| Source checked | What it establishes | +|----------------|---------------------| +| https://abseil.io/resources/swe-book/html/ch12.html | A test is *complete* when *"its body contains all of the information a reader needs in order to understand how it arrives at its result"*; DAMP over DRY; and, directly on point, that engineers should *"use helper methods with descriptive parameters that make dependencies explicit"* rather than reusing shared constants. | +| https://testing.googleblog.com/2017/01/testing-on-toilet-keep-cause-and-effect.html | Keep the inputs a result depends on visible in the test method rather than in shared setup, so cause and effect is readable without jumping elsewhere. | + +*Session evidence:* `rows_for(doc)` seeded from the module constant `PAYLOAD` +while its tests ran payload `{}`; a shape-only migration fixed 1 of 11 failures, +moving the payload into the signature fixed 11 of 11. + +**Confidence: `verified`** — both sources are prescriptive on the exact point. + +Note: the canonical name for this smell is xUnit Patterns' *Mystery Guest*. +`xunitpatterns.com` is HTTP-only and could not be fetched over HTTPS this +session, so **it is deliberately not cited** — an unfetchable URL is worse than +none, per the ingest rules. + +--- ## Existing-layer check -Pages read for overlap in each target domain/category before editing: - -- **path-resolution** (`platforms/environment`): its "When this applies" already - owns "command not found though the tool is installed" and its "Instead of" - already steers `which` → `command -v`/`type`. The keg-only case **extends** it - with a mechanism neither the page nor `command -v` covers (the binary is off - PATH by design, so `command -v` also misses it). **Merged** a "Do this" row, an - "Instead of" row, a source, and a "When this applies" clause — no new page. - Also read `toolchains/version-management` (owns version-manager shims, a - different off-PATH mechanism — left untouched, no conflict). -- **non-interactive-cli-invocation** (`platforms/processes`): the page owns - "automating a prompt-capable CLI (agent CLI) from a harness." The bracketed-paste - stall is a distinct failure mode within that theme (driving the *interactive* - REPL rather than `-p`). **Merged** an Edge-cases row, a source+field-context - bullet, and a "When this applies" clause. No conflict with its stdin-detach - guidance (orthogonal: that is fd 0 for `-p` calls; this is paste framing for - keystroke injection). -- **test-level-choice** (`testing/strategy`): its Edge-cases already has "logic - worth unit-testing is buried inside a controller/handler that needs the - framework to run → extract it." The import-side-effect case is the same family - (buried-in-a-module-that-needs-I/O) with a pytest-specific skipif twist. - **Merged** two Edge-cases rows + one "Instead of" row + a source. Also read - `testing/data/test-data-and-isolation` (owns runtime state leak, not import - time) — no duplication; added a reciprocal `related:` link (it already linked - back to test-level-choice). - -Conflicts flagged: none. Related links added: test-level-choice ↔ -test-data-and-isolation (reciprocal now complete). +Routed via `INDEX.md` → domain `index.md` → every page whose "load when" line +overlapped. Pages read in full before deciding: all six of `testing/quality/`, +`testing/data/test-data-and-isolation`, `testing/index`, `qa/process/regression-scope`, +`qa/index`, `debugging/index`, `backend/index`, `backend/python/index`, plus a +repo-wide search for prior coverage (`grep -rliE "call site|callee|positional +argument|keyword argument"` and a `\bgrep\b` sweep over `wiki/`). + +### Insight 1 — nearest neighbours, and why it is not a duplicate + +| Page read | Overlap | Decision | +|-----------|---------|----------| +| `testing/quality/spec-artifact-checks` | Closest. Already owns *"one negative control per check, mutating only what that check owns"* and the coverage-vs-validity split. | **Not a merge.** Its trigger is *authoring a check*; this insight's trigger is *evolving the artifact the check validates* — the negatives already exist and are individually sound, yet the corpus as a whole cannot reach the new shape. Per the wiki's one-case-per-page rule this is a new trigger. Linked both ways. | +| `testing/quality/tests-that-cannot-fail` | Owns the break-the-code red-run rule the new page's step 4 depends on. | Referenced inline; `related:` added both ways. | +| `testing/quality/harness-reverse-controls` | Owns the *harness-level* control (can this harness go green). | Complementary, not overlapping: that page asks whether the harness discriminates at all, this one asks whether the fixture corpus reaches a newly added shape. `related:` added both ways. | +| `testing/quality/checks-that-cannot-pass` | Trigger is a check whose **target does not exist yet**. | Distinct — here the target exists and the gate is green. No edit. | +| `qa/document-verification/spec-document-gates` | Release-gate altitude for document deliverables. | Kept as a one-way `related:` from the new page. | + +**No conflicts found.** Nothing in the wiki contradicts the new directive. + +### Insight 2 — no existing coverage anywhere + +The repo-wide search found **zero** pages discussing call-site enumeration, +callee-name search, or positional-vs-keyword arguments as a recon concern. The +single adjacent line is `qa/process/regression-scope`'s edge-case row *"The +change is in code with no test coverage and unclear callers | Trace callers +before scoping"* — which names the need and does not say how. That row now +points at the new page, and `regression-scope` gained the new page in +`related:` (both directions). + +Checked and rejected as homes: `debugging/*` (diagnosis of a failure, not +pre-change recon), `backend/python/language/mutable-state-traps` (mutable +defaults and shared state — a different mechanism), `platforms/tools/bsd-vs-gnu-cli` +(grep *flag* portability, not search strategy). + +### Insight 3 — merged, not created + +`testing/data/test-data-and-isolation` already owns fixture construction and +carries the adjacent rule *"pass explicitly only the fields the test's behavior +depends on"* plus a row for shared **mutable** fixture objects. This insight is +the same trigger with a directive that extends it (a *defaulted* constant, which +is not mutated and so is not covered by the existing row). Merge-before-create +applied — no new page. Added: 1 `Do` row, 1 edge-case row (the symptom is a +lookup miss far from the helper), 1 `Instead of` row, 2 sources, and +`last_verified` bumped to 2026-08-04. + +--- ## Routing decision -| Insight | Target domain/category/page | Notes | -|---------|-----------------------------|-------| -| 1. keg-only off PATH | `platforms/environment/path-resolution.md` | Harvested hint said `infrastructure`; **re-routed** — the wiki has a dedicated PATH page and the insight is a PATH-resolution fact, not a CI/CD or deploy concern | -| 2. bracketed-paste REPL injection | `platforms/processes/non-interactive-cli-invocation.md` | Matches harvested `platforms`; merged as an edge case of automating a prompt-capable CLI | -| 3. import-time side effects vs unit test | `testing/strategy/test-level-choice.md` | Matches harvested `testing`; chosen over `testing/data/test-data-and-isolation` because the resolution is a *level/structure* decision (extract to a side-effect-free module / recognize it as integration), extending the page's existing "extract buried logic" edge case | +| Insight | Target | Rationale | +|---------|--------|-----------| +| 1 | `testing` / `quality` / **new page** `schema-additions-under-a-golden-gate` | `INDEX.md` routes "writing or structuring automated tests … verifying tests can actually fail" to `testing`; within it, `quality` already holds the five pages about whether a check proves anything. Existing category, no structural change. | +| 3 | `testing` / `data` / **merge** into `test-data-and-isolation` | Same trigger as the page's own ("tests need fixture data and you are choosing how to create it"); directive extends step 1. | +| 2 | `backend` / `common` / **NEW category `change-impact`** / `call-site-enumeration` | See below. | + +### New category: `backend/common/change-impact/` + +**Why `backend`:** `AGENTS.md`'s routing protocol resolves a multi-domain match +by "the domain that owns **the artifact you will change**" — here, application +code. **Why `common`:** the directive is language-agnostic (any language with +optional or positional arguments; in a language with no keyword arguments at all +the parameter-name search returns nothing whatsoever). The Python mechanics are +cited as the mechanism, not as the scope. + +**Why not an existing category** — all eleven `backend/common` categories were +re-checked by name before creating a new one: + +- `api-design` is the only near miss, and all three of its pages are HTTP-shaped + (status codes, endpoint idempotency, list-endpoint pagination). Its "load when" + lines are written about endpoints; filing an in-process function-signature + concern there would make the category's routing lines contradict its contents, + which invariant 1 forbids. +- `reliability`, `caching`, `jobs`, `errors`, `auth`, `orm`, `concurrency`, + `llm`, `integrations`, `storage` — all runtime-behaviour categories; none + covers a design-time change-impact question under any other name. + +`change-impact` is the noun for the concern, leaves room for sibling pages +(schema/event-contract consumers, deprecation windows), and is registered in +`wiki/backend/index.md` plus the root `INDEX.md` backend row. + +### Plumbing + +- `wiki/testing/index.md` — new `quality` row with a use-case-enumerating "load when" line. +- `wiki/backend/index.md` — new `change-impact` section + subtree summary row updated. +- `INDEX.md` — backend row's `common/` concern list updated. +- `log.md` — `## [2026-08-04] ingest | …` entry appended. +- `related:` added both ways for all four adjacent pages. + +### Invariants verified mechanically (not by eye) + +Ran over all **141** pages after the edits: -No new categories created — every insight merged into an existing page under an -existing category. Domain `index.md` load-when lines for all three pages were -extended so the new cases are routable, and `log.md` records the ingest. +- every page is listed in an ancestor `index.md` → **0 unlisted** +- every `related:` id and inline `[page-id]` reference resolves → **0 broken** +- no page exceeds 120 body lines → **0 over** +- banned vague qualifiers (`usually`, `consider`, `might want to`, `generally`, + `as appropriate`) in the three touched pages → **0 hits** +- every "don't"-shaped statement in the new pages is descriptive prose, not a + bare prohibition; anti-patterns appear only in `Instead of` tables, each paired + with its replacement. diff --git a/INDEX.md b/INDEX.md index d6d4238..6c536e3 100644 --- a/INDEX.md +++ b/INDEX.md @@ -10,7 +10,7 @@ follow the cross-pointers in their index or take the next matching seeded domain | Domain | Status | Route here when | |--------|--------|-----------------| | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior | -| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | +| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, call-site enumeration before a contract change, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility | | [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) | | [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) | diff --git a/log.md b/log.md index 389f06b..c519858 100644 --- a/log.md +++ b/log.md @@ -37,4 +37,5 @@ Append-only. Format: `## [YYYY-MM-DD] /bin` (verified: `which mlir-opt` not found vs `/opt/homebrew/opt/llvm/bin/mlir-opt` → LLVM 22.1.8; source docs.brew.sh/FAQ). platforms/processes/non-interactive-cli-invocation +bracketed-paste edge: injecting a long/multiline prompt into a REPL (tmux `send-keys -l`) stalls at `❯ [Pasted text #1]` because the input is one bracketed-paste block (ESC[200~…201~) whose embedded newline is not submit — send Enter as a separate keystroke a beat later (source en.wikipedia.org/wiki/Bracketed-paste + claude-code#43169). testing/strategy/test-level-choice +import-side-effect edge/instead-of: a "pure" function's test is not dependency-free if its module runs I/O at import; `@pytest.mark.skipif` evaluates after the module import so it can't gate it — use `importorskip`/`skip(allow_module_level=True)` or move the function to a side-effect-free module (source docs.pytest.org skipping). Harvested "infrastructure" hint for the keg-only insight re-routed to platforms/environment (dedicated PATH page). Confidence: keg-only & pytest verified vs official docs; bracketed-paste mechanism doc-verified, the claude-CLI submit specifics field-tested. diff --git a/wiki/backend/common/change-impact/call-site-enumeration.md b/wiki/backend/common/change-impact/call-site-enumeration.md new file mode 100644 index 0000000..649c9df --- /dev/null +++ b/wiki/backend/common/change-impact/call-site-enumeration.md @@ -0,0 +1,83 @@ +--- +id: backend-common-change-impact-call-site-enumeration +domain: backend +category: change-impact +applies_to: [general] +confidence: verified +sources: + - https://docs.python.org/3/glossary.html + - https://docs.python.org/3/library/ast.html + - https://peps.python.org/pep-0570/ +last_verified: 2026-08-04 +related: [qa-process-regression-scope, backend-python-language-mutable-state-traps] +--- + +# Enumerating Call Sites Before Changing a Callee's Contract + +## When this applies + +You are about to change the contract of a function, method, or constructor that +other code calls — adding, removing, reordering, or redefining a parameter — and +the plan depends on having the complete list of call sites. Also when a +migration you scoped from a search came back green and then failed on call sites +the search never listed. + +## Do this + +1. **Enumerate by the callee's name (`verify(`, `Interpreter(`) and read every + hit.** Treat a search for a parameter name (`repo_rows=`) as a partial index: + it lists only the sites that happen to pass that argument by keyword. + +2. **Read the partiality as a language property, not a search-quality problem.** + Python's default parameter kind is positional-or-keyword — it "specifies an + argument that can be passed either positionally or as a keyword argument. + This is the default kind of parameter". The parse tree keeps the two forms in + separate fields: in `ast.Call`, "`args` holds a list of the arguments passed + by position" while "`keywords` holds a list of `keyword` objects representing + arguments passed by keyword". A keyword-name search reads `keywords` only. + +3. **Pick the enumeration handle from the callee's shape:** + +| Callee | Enumerate by | +|--------|--------------| +| A distinctively named function | The name plus `(`, across the whole repo including tests, fixtures, and scripts | +| A name common enough to collide (`run`, `get`, `verify`) | The language server's find-references, or an AST pass collecting `Call` nodes whose `func` resolves to it — text search cannot separate the homonyms | +| A constructor | The class name plus `(`, plus each import form that renames it (`from m import C as D` → `D(`) | +| Something passed as a value (callback, decorator, registry entry, `functools.partial`) | The bare name without `(` as well — the argument list at those sites lives where the value is invoked, not where it is referenced | +| Dispatched dynamically (`getattr`, a name in config/YAML) | The string form too, and record in the plan that this class of site is not statically enumerable | + +4. **State the method next to the count.** "13 call sites (grep `verify(`, + including tests)" is checkable; "13 call sites" is not, and a plan built on + an unstated method cannot be reviewed for this gap. + +5. **Re-run the same enumeration after the edit and require zero old-contract + sites**, then run the tests. The re-run is what converts the enumeration from + a plan input into a completion check. + +## Edge cases + +| Case | Then | +|------|------| +| The new parameter can be keyword-only | Declare it after a bare `*`; a stale positional call then fails at the call site instead of silently binding to the wrong parameter (PEP 570 defines the `/` and `*` markers that fix a parameter's passing form) | +| A parameter is inserted before existing ones rather than appended | Every positional site rebinds silently and none of them changes text — a keyword search cannot bound the risk, so a full callee-name enumeration is the only scope; prefer appending | +| The callee is re-exported through a package `__init__` or a facade | Enumerate the re-exported name as well; sites importing through the facade never mention the defining module | +| Call sites live in another repository or a published package | The change is a versioned deprecation, not an in-place edit: keep the old contract accepting its old shape for a release, and enumerate what you own now | +| The language has no keyword arguments at all (JavaScript, Go) | Every site is positional, so a parameter-name search returns nothing at all — enumerate by callee name from the start | +| The repo has no working language server for the language | Enumerate by callee name and say so; an AST pass over `Call` nodes is the fallback that survives aliasing | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Report "N call sites, M need editing" from a search for the parameter name | Search the callee name, read each hit, and report the method with the count | Positional-or-keyword is the default parameter kind, so a keyword search is blind to every site that passes the argument by position | +| Treat a green suite after a partial migration as proof the migration is complete | Re-run the callee-name enumeration and require zero old-contract sites | The suite exercises the sites it reaches; the ones the recon missed are the ones that break later | +| Scope a contract change from the plan's original recon | Re-enumerate at edit time | Call sites are added between planning and editing, and the plan's count is what makes the omission invisible | +| Add the parameter in the middle of the signature and update the keyword sites | Append it, or make it keyword-only, then migrate every enumerated site | A middle insertion rebinds existing positional arguments without changing a character at those sites | + +## Sources + +- https://docs.python.org/3/glossary.html — *positional-or-keyword*: "specifies an argument that can be passed either positionally or as a keyword argument. This is the default kind of parameter"; *keyword-only* requires a bare `*`, *positional-only* a `/` +- https://docs.python.org/3/library/ast.html — `ast.Call`: "`args` holds a list of the arguments passed by position", "`keywords` holds a list of `keyword` objects representing arguments passed by keyword" — the two forms are distinct fields, so a keyword-text search cannot reach positional arguments +- https://peps.python.org/pep-0570/ — the `/` marker for positional-only parameters, alongside the existing `*` marker for keyword-only, as the way a signature fixes how an argument may be passed +- Local reproduction 2026-08-04 (Python 3.14.6, macOS): over four call sites of `verify(...)` where one passes `repo_rows=` by keyword, a regex search for `repo_rows\s*=` matches 1 while an AST pass over `Call` nodes named `verify` finds 4 — 3 sites invisible to the keyword search +- Field incident 2026-08-04 (`linkly-t1-repo-policy`, Python): recon by keyword search reported "13 call sites, 7 need editing"; 8 further seeds passed the same value as `verify()`'s fourth positional argument, and the suite the session had reported green then ran `472 tests / FAILED (failures=11)` diff --git a/wiki/backend/index.md b/wiki/backend/index.md index 9e683bd..d7fc90b 100644 --- a/wiki/backend/index.md +++ b/wiki/backend/index.md @@ -5,7 +5,7 @@ three stack subtrees — route by concern first, stack second: | Subtree | Route there when | |---------|------------------| -| [common](#common-language-agnostic) (below) | The concern is language-agnostic: API contracts, idempotency, JWT issuance, outbound calls, caching, jobs, transactions in app code, shared state/pools, exception structure, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references | +| [common](#common-language-agnostic) (below) | The concern is language-agnostic: API contracts, enumerating call sites before a contract change, idempotency, JWT issuance, outbound calls, caching, jobs, transactions in app code, shared state/pools, exception structure, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references | | [java](java/index.md) | You are writing/reviewing JVM backend code (Java/Kotlin, Spring, JPA/Hibernate) and the concern is stack-specific: entity mapping, persistence context, proxy pitfalls, JVM threads/memory | | [node](node/index.md) | You are writing/reviewing Node.js/TypeScript backend code: event-loop blocking, promise error handling, runtime validation at boundaries, graceful shutdown | | [python](python/index.md) | You are writing/reviewing Python backend code: GIL/concurrency model, pydantic validation, WSGI/ASGI workers, language traps | @@ -26,6 +26,12 @@ Match your situation to a "load when" line; load only matching pages. | [idempotency](common/api-design/idempotency.md) | An endpoint with side effects (create, charge, send) can receive the same request twice — client retry after timeout, user double-submit, gateway retry; designing idempotency-key storage; deciding which operations are safe to retry | | [pagination-contract](common/api-design/pagination-contract.md) | Designing a list endpoint's request/response contract — cursor vs page-number, limit caps, total counts, expired-cursor behavior (the backing SQL/index → databases/query-optimization/keyset-pagination) | +### change-impact + +| Page | Load when | +|------|-----------| +| [call-site-enumeration](common/change-impact/call-site-enumeration.md) | Changing the contract of a function/method/constructor other code calls — adding, removing, reordering or redefining a parameter — and you need the complete call-site list; scoping such a migration from a search; a migration scoped from recon came back green and then failed on call sites the search never listed; deciding whether to append a parameter or make it keyword-only (release-level re-test scope → qa/process/regression-scope) | + ### reliability | Page | Load when | diff --git a/wiki/qa/process/regression-scope.md b/wiki/qa/process/regression-scope.md index 6c473cb..6127cb6 100644 --- a/wiki/qa/process/regression-scope.md +++ b/wiki/qa/process/regression-scope.md @@ -7,7 +7,7 @@ confidence: field-tested sources: - https://martinfowler.com/articles/rise-test-impact-analysis.html last_verified: 2026-07-10 -related: [qa-process-release-gates, qa-bug-reports-reproducible-reports] +related: [qa-process-release-gates, qa-bug-reports-reproducible-reports, backend-common-change-impact-call-site-enumeration] --- # Choosing What to Re-Test for a Change @@ -44,7 +44,7 @@ Two standing practices that make the rings work: | Case | Then | |------|------| -| The change is in code with no test coverage and unclear callers | Trace callers before scoping (text pointer wiki/testing/ for coverage tooling); an adjacency you cannot enumerate defaults into scope | +| The change is in code with no test coverage and unclear callers | Trace callers before scoping ([backend-common-change-impact-call-site-enumeration] for enumerating them; text pointer wiki/testing/ for coverage tooling); an adjacency you cannot enumerate defaults into scope | | Data migration or backfill ships with the release | Add the migrated data's read paths to the direct ring — the "change" is the data, and its consumers are the changed feature | | Two changes in one release touch the same adjacency | Test that adjacency once, after both changes are in the release build — testing between them validates a build that will never ship | | Config-only or copy-only change | Direct ring only: verify the changed value/text where it surfaces, plus the critical-flows floor if the config gates a critical flow | diff --git a/wiki/testing/data/test-data-and-isolation.md b/wiki/testing/data/test-data-and-isolation.md index 994a602..c883909 100644 --- a/wiki/testing/data/test-data-and-isolation.md +++ b/wiki/testing/data/test-data-and-isolation.md @@ -7,7 +7,8 @@ confidence: verified sources: - https://martinfowler.com/articles/nonDeterminism.html - https://abseil.io/resources/swe-book/html/ch12.html -last_verified: 2026-07-10 + - https://testing.googleblog.com/2017/01/testing-on-toilet-keep-cause-and-effect.html +last_verified: 2026-08-04 related: [testing-flaky-diagnosing-flaky-tests, testing-strategy-test-level-choice] --- @@ -36,6 +37,7 @@ state-leak symptom. | DB-backed tests | Wrap each test in a transaction rolled back at the end, or truncate the touched tables between tests — pick one mechanism per suite and apply it uniformly | | Rollback impossible (code under test commits, or asserts across connections) | Truncate/reset between tests, or give each test uniquely-keyed rows it queries back by its own keys | | Shared mutable fixture object (module-level constant a test mutates) | Give each test its own copy from the factory; reserve shared fixtures for immutable data | +| The fixture's shape depends on a value the test also passes to the code under test (a key, a tenant id, a payload, a timestamp) | Put that value in the factory's signature so every call site names it — a factory that defaults it to a module-level constant lets fixture and run diverge silently | | Time-dependent logic (expiry, scheduling, "created today") | Inject a clock/time source and freeze it in the test; assert against the frozen instant | | Unique-constrained values (emails, usernames, external ids) | Generate per test (counter, UUID suffix) inside the factory — hardcoded constants collide across tests and across parallel runs | | Filesystem / temp files | Create a fresh per-test temp directory and remove it in teardown | @@ -53,6 +55,7 @@ state-leak symptom. | Suite is too slow because every test builds a deep object graph | Move the invariant graph into a per-suite setup that tests never mutate; keep mutated entities per-test | | Failure appears only in the full suite, never alone | Run the suite in random order to expose the order dependency, then bisect to the polluting test; fix the polluter's ownership, not the victim ([testing-flaky-diagnosing-flaky-tests]) | | Test needs "now"-relative data but the code reads the system clock directly | Refactor the code to accept an injected clock; that seam is the fix — assertions with tolerance windows around real time stay flaky | +| A group of tests fails as a lookup miss, an empty result, or a "not found" far from any fixture code | Compare each factory's defaulted values against the input the test actually runs before migrating fixture shape; when the two disagree, the fixture was built for a different input and only the signature change fixes the group | ## Instead of @@ -62,8 +65,11 @@ state-leak symptom. | Hardcode `test@example.com` / `user1` in many tests | Generate unique values in the factory per test | Unique-constraint collisions fail tests that are individually correct | | `sleep()` until background work lands the data | Wait explicitly on the condition/event with a timeout | Sleeps are both too slow and too short; the race remains | | Copy a full production-like JSON blob as fixture for one field's behavior | Build the minimal object via a factory, explicit only in that field | Giant fixtures hide the relevant value and break on unrelated schema changes | +| Let a factory seed itself from a module-level constant while the test passes a different value to the code under test | Take that value as a factory parameter and name it at every call site | The fixture is then built for the input the test actually runs; a defaulted constant makes the two drift apart with nothing in the test body showing it | ## Sources - https://martinfowler.com/articles/nonDeterminism.html — isolation between tests, wrapping the system clock, callbacks/polling over bare sleeps -- https://abseil.io/resources/swe-book/html/ch12.html — tests should contain the values they depend on; clarity over shared magic setup +- https://abseil.io/resources/swe-book/html/ch12.html — a test is complete when "its body contains all of the information a reader needs in order to understand how it arrives at its result"; prefer DAMP over DRY, and where a helper is used, give it "descriptive parameters that make dependencies explicit" rather than reusing shared constants +- https://testing.googleblog.com/2017/01/testing-on-toilet-keep-cause-and-effect.html — keep the inputs a test's result depends on visible in the test method instead of in shared setup, so the cause-and-effect relationship is readable without jumping elsewhere +- Field incident 2026-08-04 (`linkly-t1-repo-policy`, Python): `rows_for(doc)` seeded its rows from the module constant `PAYLOAD` while its tests ran payload `{}`; a shape-only migration of the helper fixed 1 of 11 failures, and moving the payload into the helper's signature fixed 11 of 11 diff --git a/wiki/testing/index.md b/wiki/testing/index.md index 00c9503..efb8ac8 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -25,6 +25,7 @@ Match your situation to a "load when" line; load only matching pages. | [tests-that-cannot-fail](quality/tests-that-cannot-fail.md) | Reviewing tests that always pass; a bug shipped through an area the suite reported as covered; auditing a suspiciously green suite; judging whether an assertion, error-path test, or mock-based test can actually detect a defect | | [checks-that-cannot-pass](quality/checks-that-cannot-pass.md) | Authoring a check whose target does not exist yet (grep/regex gate on an unwritten file or doc section, lint/scan rule, schema assertion on an unbuilt endpoint, a plan's verification command) and it has only ever been observed failing; reviewing a plan's gates before adopting them; separating "target missing" from "content missing" in a gate's exit status | | [spec-artifact-checks](quality/spec-artifact-checks.md) | Writing or reviewing an automated check that a mapping table covers every rule/field/enum case, or that ids resolve across documents; deciding whether a green check earned "verified" or only "present"; designing one negative control per check in a multi-check harness; parsing Markdown table rows programmatically in a doc-as-spec repo | +| [schema-additions-under-a-golden-gate](quality/schema-additions-under-a-golden-gate.md) | Adding a node kind, variant, discriminator value, or field to a document format (IR, JSON Schema, spec artifact) whose only automated gate builds its negatives by mutating one committed golden example; the gate or the whole suite comes back green right after a schema change; deciding which negative each new schema keyword needs, and whether a green suite that never loads the schema is evidence at all | | [harness-reverse-controls](quality/harness-reverse-controls.md) | You built a harness that scores how well something is verified (mutation run, doc/spec gate suite, CI check matrix) and are about to cite its score in a commit, PR, README, or report; its verdicts come out uniform (every case caught, or every case green); deciding what control run proves the harness discriminates, how to score errored/never-ran cases, and what the harness's isolated working tree must contain | ## data diff --git a/wiki/testing/quality/harness-reverse-controls.md b/wiki/testing/quality/harness-reverse-controls.md index 088268a..2fcf98c 100644 --- a/wiki/testing/quality/harness-reverse-controls.md +++ b/wiki/testing/quality/harness-reverse-controls.md @@ -12,7 +12,7 @@ sources: - https://stryker-mutator.io/docs/mutation-testing-elements/equivalent-mutants/ - https://testing.googleblog.com/2021/04/mutation-testing.html last_verified: 2026-08-02 -related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set] +related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, testing-quality-schema-additions-under-a-golden-gate] --- # Citing a Verification Harness's Own Score diff --git a/wiki/testing/quality/schema-additions-under-a-golden-gate.md b/wiki/testing/quality/schema-additions-under-a-golden-gate.md new file mode 100644 index 0000000..3f4a4b6 --- /dev/null +++ b/wiki/testing/quality/schema-additions-under-a-golden-gate.md @@ -0,0 +1,94 @@ +--- +id: testing-quality-schema-additions-under-a-golden-gate +domain: testing +category: quality +applies_to: [general] +confidence: verified +sources: + - https://json-schema.org/understanding-json-schema/reference/conditionals + - https://json-schema.org/understanding-json-schema/reference/object + - https://json-schema.org/draft/2020-12/json-schema-core + - https://pitest.org/quickstart/basic_concepts/ +last_verified: 2026-08-04 +related: [testing-quality-spec-artifact-checks, testing-quality-tests-that-cannot-fail, testing-quality-harness-reverse-controls, qa-document-verification-spec-document-gates] +--- + +# Adding a Case to a Format Whose Only Gate Mutates a Golden Example + +## When this applies + +You are adding a node kind, variant, discriminator value, or field to a document +format (an IR, a JSON Schema, a spec artifact), and the format's only automated +gate builds its negative cases by mutating one committed golden example. Also +when a schema change lands and the gate — or the whole suite — comes back green. + +## Do this + +1. **Check whether the golden example contains an instance of the new kind + before reading the gate's verdict.** When it does not, every mutant derived + from that golden leaves the new branch untouched, so the green run reports on + the old shapes only. JSON Schema states the mechanism for conditional + branches: "If `if` is invalid, `else` must also be valid (and `then` is + ignored)" — a branch keyed on the new kind is never applied to an instance + that lacks it. The same gap has a name in mutation tooling: PIT's **No + coverage** is "the same as **Survived** except there were no tests that + exercised the line of code where the mutation was created". + +2. **Commit a minimal conforming fixture that contains the new kind, next to the + gate, and register it as a must-pass input.** Build it to exercise the new + branch only — the smallest document the schema accepts that reaches it. + +3. **Add one negative per keyword the new branch introduces**, mutating only + what that keyword owns: + +| Keyword the new branch adds | Negative that must redden the gate | +|-----------------------------|------------------------------------| +| `required` (a new mandatory field) | Delete that field from the new fixture | +| `type` | Replace the field's value with another JSON type | +| `enum` / `const` (the discriminator) | Replace the value with one absent from the set | +| `additionalProperties: false` | Add one property the branch does not declare | +| A cross-document id that must resolve | Repoint the id at a target that does not exist | + +4. **Restore each negative one at a time and observe the gate go red.** A + negative that has never been seen failing is an assertion about the gate, not + a control ([testing-quality-tests-that-cannot-fail]). + +5. **Constrain the new branch before writing its negatives.** With + `additionalProperties` omitted, "By default any additional properties are + allowed", and `properties` alone mandates nothing — "the properties defined + by the `properties` keyword are not required". A branch carrying neither + `required` nor `additionalProperties: false` has no negative to write, + because no instance of the new kind can fail it. + +6. **Judge the rest of the suite by whether it reads the schema at all.** + Enumerate the tests that load the schema file by name; when that count is + zero, a full green suite is unrelated to the change and is not evidence for + it. Report the two verdicts separately. + +## Edge cases + +| Case | Then | +|------|------| +| The gate is a real mutation tool (PIT, Stryker) rather than a hand-rolled script | Same gap under its own label: read **No coverage** / **Survived** on the new branch as the missing fixture, not as a missing rule | +| The golden example is generated rather than hand-written | Regenerate it from a generator input that includes the new kind and commit the regenerated output as the fixture, so a generator change reddens the gate | +| The new kind is valid only in a nested position (inside a specific parent node) | Place the fixture's instance at a position the schema actually admits; a top-level instance of a nested-only kind tests the parent's rejection path instead | +| You add the new kind to the existing golden instead of a separate fixture | Re-run every pre-existing negative afterwards and require each one's prior verdict; enlarging the golden changes the input all of them mutate | +| The gate only checks that the document parses, never validating it against the schema | The addition has no gate — say so, and add schema validation before adding negatives | +| The new branch is `unevaluatedProperties`- or `$ref`-based rather than inline | Mutate through the reference: change the target subschema's own keyword, and require the referring branch to redden | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Add the schema branch, run the gate, and read green as "the new kind is validated" | Commit a fixture containing the new kind, then one negative per keyword the branch adds | Mutants of a golden that lacks the new kind never reach the new branch, so the verdict predates the change | +| Cite "the full test suite passes" as coverage for a schema change | Enumerate the tests that read the schema file and report that count first | A suite in which nothing loads the schema is unaffected by any edit to it | +| Write one negative for the whole new branch | Write one per keyword and require the branch to redden for each | One red proves some keyword fires, not that each keyword the branch added is enforced | +| Trust `properties` on the new branch to reject a malformed instance | Add `required` for the mandatory fields and `additionalProperties: false` | Both omitted defaults are permissive: extra properties are allowed and declared properties are optional | + +## Sources + +- https://json-schema.org/understanding-json-schema/reference/conditionals — "If `if` is invalid, `else` must also be valid (and `then` is ignored)"; a conditional branch is not applied to an instance that fails its `if` +- https://json-schema.org/understanding-json-schema/reference/object — "By default any additional properties are allowed"; "By default, the properties defined by the `properties` keyword are not required" +- https://json-schema.org/draft/2020-12/json-schema-core — subschema applicators "MUST NOT impact the results of sibling subschemas"; "A JSON Schema MAY contain properties which are not schema keywords. Unknown keywords SHOULD be treated as annotations" — a misspelled keyword in a new branch is ignored rather than rejected +- https://pitest.org/quickstart/basic_concepts/ — "**Survived**: the mutation was not detected by the covering test"; "**No coverage**: the same as **Survived** except there were no tests that exercised the line of code where the mutation was created" +- Field observation 2026-08-04 (`linkly-t1-spec-notation`, Python): the only schema gate over `*.lir.json` was `scripts/validate_ir.py --self-test`, whose three negatives were all `copy.deepcopy` mutations of `examples/login.lir.json`; `grep -rln "lir.schema\|jsonschema" impl/tests/` returned zero of 447 tests, so neither the gate's green run nor the suite's constrained any node kind absent from that one example diff --git a/wiki/testing/quality/spec-artifact-checks.md b/wiki/testing/quality/spec-artifact-checks.md index d754f6e..73f9208 100644 --- a/wiki/testing/quality/spec-artifact-checks.md +++ b/wiki/testing/quality/spec-artifact-checks.md @@ -10,7 +10,7 @@ sources: - https://pitest.org/ - https://github.github.com/gfm/ last_verified: 2026-07-29 -related: [testing-quality-tests-that-cannot-fail] +related: [testing-quality-tests-that-cannot-fail, testing-quality-schema-additions-under-a-golden-gate] --- # Checks That Verify a Spec or Mapping Artifact diff --git a/wiki/testing/quality/tests-that-cannot-fail.md b/wiki/testing/quality/tests-that-cannot-fail.md index 884ff61..dcb33f7 100644 --- a/wiki/testing/quality/tests-that-cannot-fail.md +++ b/wiki/testing/quality/tests-that-cannot-fail.md @@ -11,7 +11,7 @@ sources: - https://martinfowler.com/bliki/TestCoverage.html - https://testing.googleblog.com/2013/05/testing-on-toilet-dont-overuse-mocks.html last_verified: 2026-07-10 -related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-checks-that-cannot-pass, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, qa-document-verification-spec-document-gates] +related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-checks-that-cannot-pass, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, testing-quality-schema-additions-under-a-golden-gate, qa-document-verification-spec-document-gates] --- # Proving a Test Can Fail