From 0d8f47967d74cae9b21381bc701bc23d8548203f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=B5=9C=EC=98=81=EA=B8=B0?= Date: Sun, 2 Aug 2026 16:01:34 +0900 Subject: [PATCH 1/2] knowledge: ingest 2 verified insight(s) - backend/common/llm/context-window-budget (NEW category common/llm): size an LLM client's output cap to the model actually serving it; the window holds request and generation together, so max_tokens is a reservation inside it. - testing/quality/harness-reverse-controls: prove a verification harness discriminates before citing its score; a semantics-preserving no-op must survive, and errored cases leave the denominator rather than counting as catches. Third candidate (non-interactive CLI stdin) retired as already in flight in PRs #11/#12 rather than opened as a third duplicate. See .dev-loop/INGEST_REPORT.md. --- .dev-loop/INGEST_REPORT.md | 165 +++++++++++++++--- log.md | 3 + .../common/llm/context-window-budget.md | 88 ++++++++++ wiki/backend/index.md | 6 + wiki/testing/index.md | 1 + .../quality/harness-reverse-controls.md | 96 ++++++++++ .../testing/quality/tests-that-cannot-fail.md | 3 +- 7 files changed, 339 insertions(+), 23 deletions(-) create mode 100644 wiki/backend/common/llm/context-window-budget.md create mode 100644 wiki/testing/quality/harness-reverse-controls.md diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 2ff3871..819f42c 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,35 +1,156 @@ -# Knowledge flush — 1 insight +# Knowledge flush — 3 insight(s) -Source: RNR-3440 (사내 잠재매물 주간 추출 스크립트 메모리 피크 저감). Candidate: -"QueryPie 프록시 경유로 대용량 결과를 스트리밍할 때 server-side named cursor 대신 -일반 커서 + `fetchmany` + openpyxl `write_only`." +Queue drained: 3 pending candidates (2 × `platforms`/`backend`, 1 × `testing`). +Result: **2 pages ingested, 1 candidate retired as already in flight.** + +> **Read this first.** My initial existing-layer check was contaminated and I reversed +> two of its three conclusions. The checkout carried **untracked leftovers from an +> earlier flush that died before committing** — `git reset --hard origin/main` does not +> remove untracked files. Two pages were sitting in the working tree looking exactly +> like merged wiki content while belonging to no branch, no PR, and not to `main`. +> I only caught it when `git status` staged them as **new** rather than modified. +> Everything below is re-derived against clean `origin/main` **plus the 7 open PRs**, +> which the skill's dedup step does not currently mention and which turned out to +> matter. See "What I got wrong" at the end. ## Verified best-practice -**Claim 1 — psycopg2 server-side (named) cursor requires a transaction; fails under autocommit.** -- Source: psycopg2 usage docs — `https://github.com/psycopg/psycopg2/blob/master/doc/src/usage.rst` (via Context7). Quote: "Named cursors are typically created 'WITHOUT HOLD', meaning they exist only within the current transaction. Attempting to fetch from them after a commit or in autocommit mode raises an exception." -- Matches my live repro (`can't use a named cursor outside of transactions`). → **verified** +### 1. Non-interactive CLI blocking on stdin → **already in flight, not ingested** + +The knowledge is real and I verified it independently, but it is **already an open +PR**: `wiki/platforms/processes/non-interactive-cli-invocation.md` is added by both +**PR #11** and **PR #12**, same page id, same directive. Opening a third copy would +have been the third duplicate of one insight. Retired against PR #12. + +What I verified before finding the overlap (offered as review material for whichever +PR survives, since **neither** currently cites any of it): + +| Source | What it establishes | +|--------|---------------------| +| | Why bare `&` is not a fix: SIGTTIN on "a member of a background process group attempting to read from its controlling terminal" | +| + local `man nohup` (macOS 15) | BSD `nohup` DESCRIPTION covers SIGHUP, stdout and stderr only — stdin is never mentioned. PR #12 asserts the stdin redirect "is a GNU extension" without citing the BSD page | +| | GNU side of the same divergence: "If standard input is a terminal, redirect it from an unreadable file" | +| | "Non-interactive mode reads stdin, so you can pipe data in"; `-p` background shells end "about five seconds after Claude has returned its final result **and stdin has closed**" — an open descriptor gates *exit*, not just input | + +**Correction to the harvested candidate**, worth carrying into whichever PR merges: it +credited the fix to `nohup (stdin=/dev/null)`. That is wrong on macOS — BSD `nohup` +leaves stdin attached. The portable directive is an explicit `; matches the page exactly | +| The window holds request *and* generation; `max_tokens` is a reservation inside it | | +| LiteLLM raises `ContextWindowExceededError` as a 400 | | +| Base URL is the root, not `/v1` | sets `ANTHROPIC_BASE_URL=http://localhost:8000` | +| `CLAUDE_CODE_MAX_OUTPUT_TOKENS` / `CLAUDE_CODE_MAX_CONTEXT_TOKENS` exist despite being absent from the published env-vars page | Re-confirmed in the shipped v2.1.220 binary: **10 and 6 occurrences**, against an `ANTHROPIC_BASE_URL` **positive control at 43** | + +That last check is the one claim resting on binary inspection rather than docs, so it +is the one to challenge if you disagree. My first probe returned **0 for everything** — +`claude` resolves to a shell function, so `command -v` yielded a bare name and `strings` +read nothing. The uniform zero was the tell; the positive control is what turned it into +a real measurement. (Which is, with some irony, insight 3.) + +**Confidence: `verified`** (as authored — every directive carries an official doc). -**Claim 2 — a client-side (default) cursor pulls the whole result set to the client on execute; `fetchmany` only caps the Python-list explosion.** -- Source: psycopg2 cursor/usage docs + FAQ (named-cursor advantage = "data is fetched in chunks … minimal client memory"). By contrast the default cursor buffers the full result in libpq. → **verified** +### 3. Verification-harness reverse controls → `testing` (ingested) -**Claim 3 — openpyxl `write_only` gives near-constant memory (<10 MB); one save only; lxml is for serialization speed, not the memory saving.** -- Source: openpyxl Optimised Modes — `https://openpyxl.readthedocs.io/en/stable/optimized.html` (via WebSearch). "keeping memory usage under 10Mb"; "A write-only workbook can only be saved once"; "make sure you have lxml installed" for large dumps (speed). -- This **corrects** the raw candidate's "lxml unnecessary" → precise form: unnecessary *for the memory win*, recommended *for large-dump speed*. Confirmed by my server test (write-only worked with lxml absent). → **verified** +Also an orphan (`wiki/testing/quality/harness-reverse-controls.md`) — authored by the +dead flush, in no branch and no PR. **Not** covered by `main` or by the open PRs: #8 +(`spec-artifact-checks`) and #9 (`document-conformance-checks`) both add *per-check +negative controls*, which is the adjacent layer this page explicitly distinguishes +itself from — "a per-check control asks *can this check go red*, the harness control +asks *can this harness go green*". Ingested, with my own verification folded in. -**Claim 4 — QueryPie blocks `BEGIN`, so server-side cursor is impossible there.** -- Environment-specific, no external source. Live repro in gui context: `autocommit=False` + named cursor → `[ENGINE] No permission to execute BEGIN statement`. → **field-tested**. Generalized in the page to "a read-only access-control proxy that blocks transaction control", with QueryPie as the concrete example (not a product-specific page). -- Memory figure 838 MB → 38 MB (300k synthetic rows) is my RNR-3440 measurement (`ru_maxrss`, separate processes). +Sources I checked independently, and what each settles: + +| Source | What it establishes | +|--------|---------------------| +| | The control is sound in principle: an equivalent mutant has no observable difference to detect and "There is no definitive way for Stryker to find and ignore them" — so a no-op returning *caught* means something other than the assertions is failing | +| | **My addition.** `Killed`/`Survived` are modelled apart from `Timeout`, `Runtime error`, `Compile error`, and the score is "detected / valid * 100" — errored cases leave the denominator. That is precisely the accounting error in the field case | +| | Baseline requirement, and "No tests were executed. Stryker will exit prematurely." treated as failure rather than a clean sweep | +| | Second toolchain agreeing: "non viable", "memory error", "run error" are outcomes distinct from killed | + +**Changes I made to the orphan page:** added `Do this` step 6 (score `detected / valid`, +errored cases out of the denominator), added the two Stryker sources above, renumbered +the closing step, bumped `last_verified` to 2026-08-02. + +**Confidence: `verified`** (mechanism and scoring rule from two independent framework +specifications; the incident is cited as a dated field reproduction, not as the basis +for the directive). ## Existing-layer check -- Pages read: `databases/index.md`, `databases/query-optimization/keyset-pagination.md`, `backend/python/index.md`. -- Overlap: keyset-pagination is the nearest neighbor (both handle large result sets) but a **distinct** topic — pagination splits the read into many bounded queries; this page streams a *single* query's result in chunks. Not a duplicate → new page + **bidirectional `related` link** added to both. -- backend/python has no DB-cursor page; the psycopg2/openpyxl specifics live as concrete examples inside the databases page rather than a separate python page (no duplication). -- Conflicts: none found. +Routed via `INDEX.md` → domain `index.md` → every page whose "load when" overlapped, +**then** re-run against `origin/main` and `gh pr list` after the contamination surfaced. + +**Read in full:** `platforms/index.md`, `platforms/processes/background-services.md`, +`platforms/shells/portable-shell-scripts.md`, `platforms/tools/bsd-vs-gnu-cli.md`, +`debugging/index.md`, `testing/index.md`, `testing/quality/tests-that-cannot-fail.md`, +`backend/index.md`, plus the diffs of PRs #8, #9, #11, #12. Repo-wide greps for +`litellm` / `ANTHROPIC_BASE_URL` / `claude code` and `stdin` / `nohup` / `tty` / +`/dev/null` located every candidate overlap. + +| Overlap examined | Finding | Action | +|------------------|---------|--------| +| Candidate 1 vs **PR #11 and PR #12** | Both add the identical page id with the same directive, better sourced than my draft | **Retired.** My draft page deleted, and the supporting edits I had made to `platforms/index.md`, `background-services.md` and `bsd-vs-gnu-cli.md` reverted — they referenced a page id that would not exist in `main`, i.e. a broken-link lint failure | +| Candidate 2 vs `main` | No LLM page exists anywhere in `main`; the file I first read was an orphan | **Reversed to ingest** | +| Candidate 3 vs `testing/quality/tests-that-cannot-fail` | Different trigger: that page audits *one test* that always passes, this audits *the harness* grading them. Its "auditing a whole suite" edge case assumes PIT/Stryker, which supply the baseline gate a hand-rolled script lacks | New page + edge-case row added there + `related:` **both ways** | +| Candidate 3 vs PR #8 / PR #9 | Per-check negative controls — the adjacent layer, not this one | No conflict; the distinction is stated in the page's own edge-case table | +| Candidate 3: my draft vs the orphan | The orphan is the better page (uniform-verdict table, `dryRunOnly`, PIT's equivalent-mutation quote, "all mutants survive unexpectedly"); mine had the scoring rule it lacked | **Merged into the orphan**, my draft deleted | + +**Conflicts found: none.** Nothing overwritten, nothing to flag as `contradiction`. ## Routing decision -- Target: **`databases/query-optimization/streaming-large-result-sets.md`** (new page). -- Category `query-optimization` fits (memory-bounding how a query's result is pulled into the app is query-execution optimization); no new category needed. -- Registered in `databases/index.md` (query-optimization section) and appended to `log.md`. +| # | Insight | Target | New category? | +|---|---------|--------|---------------| +| 1 | Non-interactive CLI blocking on stdin | *(none — already open as PR #12; `platforms`/`processes`)* | n/a | +| 2 | LLM output cap vs serving model's window | `backend` / **`common/llm`** / `context-window-budget.md` (`backend-common-llm-context-window-budget`) | **Yes — new `common/llm` category.** Existing `common/*` categories are api-design, reliability, caching, jobs, errors, auth, orm, concurrency. The nearest fit, `reliability`, owns timeouts/retries between services and would misfile the directive as a transient-failure concern when the 400 is arithmetic. Per `AGENTS.md`, `common/` holds language-agnostic server-side concerns, which an LLM client's token budget is | +| 3 | Verification-harness reverse controls | `testing` / `quality` / `harness-reverse-controls.md` (`testing-quality-harness-reverse-controls`) | No — `quality` already holds the "can this detect a defect" family; this extends it one level up to the measuring apparatus | + +**Plumbing:** `backend/index.md` gains the `llm` category and page; `testing/index.md` +gains the page; `log.md` has three entries (2 `ingest`, 1 `dedup`); every `related:` id +and inline `[page-id]` reference resolves. `INDEX.md` unchanged — both domains already +listed and both remain accurately described. + +**Format checks:** bodies 73 and 80 lines (limit 120); five required sections each; no +banned vague qualifiers; anti-patterns confined to `Instead of` rows, each paired with a +replacement; branches as tables. + +## What I got wrong, and what would prevent it + +1. **Untracked leftovers read as merged wiki content.** `reset --hard origin/main` + leaves untracked files in place, so a flush that dies after writing pages but before + committing poisons the *next* flush's dedup — in the most misleading direction, by + making new knowledge look already-covered. I nearly dropped candidate 2 on that + basis. A `git clean -fd` (or a `git status` check) before the existing-layer step + would close this; the skill currently prescribes neither. +2. **Open PRs are not part of the dedup step.** The skill's step 2b checks "existing + wiki layers" only. With 7 open knowledge PRs, candidate 1 had already been flushed + **twice** (#11 and #12, same page path) and would have become a third. +3. **The queue is not being retired.** These rows were harvested 2026-07-31 and were + already processed into PRs #11/#12; they came back because the dead flush never ran + step 5. Retiring them is handled below. + +## Reviewer notes + +- **PRs #11 and #12 add the same page path** (`non-interactive-cli-invocation.md`) and + need dedup against each other regardless of this PR. +- Candidate 1's `nohup` correction and the four sources in §1 are unclaimed by either + PR; fold them into whichever you keep. +- The `common/llm` category is the only structural addition here — the one decision in + this PR worth pushing back on. diff --git a/log.md b/log.md index 587a5dc..c071980 100644 --- a/log.md +++ b/log.md @@ -34,3 +34,6 @@ Append-only. Format: `## [YYYY-MM-DD] window (Claude 4.5 and newer) | The request succeeds and generation stops with `stop_reason: "model_context_window_exceeded"` — branch on the stop reason, because a truncated answer is not an error and arrives as a normal 200 | +| Extended thinking is on | Thinking tokens are a subset of `max_tokens` and billed as output — raise the reservation for thinking instead of assuming it is free, then re-derive step 2 | +| The gateway advertises a window that differs from the server's | Trust the serving engine's configured length (the value that rejects the request), and correct the gateway's model config so its fallbacks compute against the same number | +| Prompt caching is enabled | Cached prefixes still occupy the window — caching changes billing, not occupancy, so the input side of step 2 is unchanged | +| The client is Claude Code and the base URL is not `api.anthropic.com` | MCP tool search is disabled by default (`ENABLE_TOOL_SEARCH=true` when the proxy forwards `tool_reference` blocks) and Remote Control is off as of v2.1.196 — budget the tool definitions as always-present input | +| The window is large enough but responses truncate mid-structure | The cap, not the window, is the limit — raise `max_output` toward the step-2 ceiling ([backend-common-api-design-error-responses] for surfacing the truncation to callers) | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Retry or raise the timeout after a context-window 400 | Recompute the cap from the error's own three numbers and resend once | The rejection is arithmetic, not transient; every retry fails identically ([backend-common-reliability-timeouts-and-retries] owns what is retryable) | +| Keep the client's default output cap when swapping in a smaller model | Set the cap from the new model's window before the first request | The default is sized for the vendor's largest window, so the smaller model fails on the first full-context turn rather than degrading later | +| Set the base URL to `http://host:4000/v1` | Set it to `http://host:4000` | The client appends `/v1/messages` itself, so the duplicated prefix 404s while looking like an auth or routing fault | +| Lower the cap until requests stop failing | Compute `window − worst_case_input` once and set that | Trial-and-error lands on a number that holds for today's history length and breaks as the conversation grows | + +## Sources + +- https://platform.claude.com/docs/en/build-with-claude/context-windows — the window "holds the conversation history plus the new output"; "Everything in the request counts toward the context window: the system prompt, every message in `messages` (including tool results, images, and documents), and your tool definitions"; overflow behavior: input alone over the window → 400 `invalid_request_error`, while on Claude 4.5+ input + `max_tokens` over the window is accepted and stops with `stop_reason: "model_context_window_exceeded"` (earlier models return a validation error); thinking tokens "are a subset of your `max_tokens` parameter"; "Cached prompt prefixes still occupy the context window" +- https://docs.litellm.ai/docs/exception_mapping — "400 | ContextWindowExceededError | litellm.BadRequestError | Special error type for context window exceeded error messages - enables context window fallbacks" +- https://docs.litellm.ai/docs/anthropic_unified — LiteLLM serves the Anthropic-format `/v1/messages` endpoint for "All LiteLLM supported providers" (openai, bedrock, vertex_ai, gemini, azure …), which is what lets an Anthropic-protocol client sit in front of an OpenAI-compatible model +- https://code.claude.com/docs/en/env-vars — `ANTHROPIC_BASE_URL`: "Override the API endpoint to route requests through a proxy or gateway. When set to a non-first-party host, MCP tool search is disabled by default. Set `ENABLE_TOOL_SEARCH=true` if your proxy forwards `tool_reference` blocks"; Remote Control disabled for non-`api.anthropic.com` hosts as of v2.1.196 +- https://docs.vllm.ai/en/stable/serving/integrations/claude_code/ — `ANTHROPIC_BASE_URL=http://localhost:8000` "Points to your vLLM server (default port is 8000)" — the root, with no `/v1` suffix +- Field reproduction 2026-07-31 (Claude Code 2.1.220 → LiteLLM → 128k-window OpenAI-compatible model): default output cap + 99,073 input tokens exceeded the 131,072-token window and returned 400 on the first request; an 8,192-token cap ran the same session through tool calls. `CLAUDE_CODE_MAX_OUTPUT_TOKENS` and `CLAUDE_CODE_MAX_CONTEXT_TOKENS` confirmed present in the shipped binary, and absent from the published env-vars reference diff --git a/wiki/backend/index.md b/wiki/backend/index.md index e991b04..d813d86 100644 --- a/wiki/backend/index.md +++ b/wiki/backend/index.md @@ -70,3 +70,9 @@ Match your situation to a "load when" line; load only matching pages. |------|-----------| | [shared-state-and-pools](common/concurrency/shared-state-and-pools.md) | Request handlers share in-process mutable state — concurrency-safe structures/confinement vs shared store in multi-instance deployments; sizing thread/connection pools; same-pool nested-acquisition deadlock; bounding queues; debugging deadlock or starvation under load | | [distributed-locks](common/concurrency/distributed-locks.md) | Only one instance may perform an action at a time — Redis-style lock with owner token and TTL/watchdog, safe release, when a DB constraint/advisory lock suffices instead; debugging locks released by the wrong holder or work done twice despite a lock | + +### llm + +| Page | Load when | +|------|-----------| +| [context-window-budget](common/llm/context-window-budget.md) | Repointing an LLM client or agent CLI at a different model, a self-hosted server (vLLM/Ollama), or a gateway (LiteLLM); setting `max_tokens` for a client whose default was sized for a larger model; the first request after such a switch returns 400 with a context-window error; deciding where to set the cap (request body vs client env var vs gateway config) and how to point the base URL at a proxy; handling truncation that arrives as a normal 200 | diff --git a/wiki/testing/index.md b/wiki/testing/index.md index 78418d7..5417696 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -22,6 +22,7 @@ Match your situation to a "load when" line; load only matching pages. | [minimum-case-set](quality/minimum-case-set.md) | Writing tests for a function/endpoint/change and choosing which cases to cover; reviewing whether coverage suffices; picking boundary values by input type; adding a regression test for a bug fix | | [behavior-not-implementation](quality/behavior-not-implementation.md) | Deciding what a test should assert; a behavior-preserving refactor broke tests; tempted to expose privates for testing; deciding whether a snapshot test is appropriate | | [tests-that-cannot-fail](quality/tests-that-cannot-fail.md) | Reviewing tests that always pass; a bug shipped through an area the suite reported as covered; auditing a suspiciously green suite; judging whether an assertion, error-path test, or mock-based test can actually detect a defect | +| [harness-reverse-controls](quality/harness-reverse-controls.md) | You built a harness that scores how well something is verified (mutation run, doc/spec gate suite, CI check matrix) and are about to cite its score in a commit, PR, README, or report; its verdicts come out uniform (every case caught, or every case green); deciding what control run proves the harness discriminates, how to score errored/never-ran cases, and what the harness's isolated working tree must contain | ## data diff --git a/wiki/testing/quality/harness-reverse-controls.md b/wiki/testing/quality/harness-reverse-controls.md new file mode 100644 index 0000000..d91cf27 --- /dev/null +++ b/wiki/testing/quality/harness-reverse-controls.md @@ -0,0 +1,96 @@ +--- +id: testing-quality-harness-reverse-controls +domain: testing +category: quality +applies_to: [general] +confidence: verified +sources: + - https://pitest.org/quickstart/basic_concepts/ + - https://stryker-mutator.io/docs/stryker-js/configuration/ + - https://stryker-mutator.io/docs/stryker-js/troubleshooting/ + - https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/ + - https://stryker-mutator.io/docs/mutation-testing-elements/equivalent-mutants/ + - https://testing.googleblog.com/2021/04/mutation-testing.html +last_verified: 2026-08-02 +related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set] +--- + +# Citing a Verification Harness's Own Score + +## When this applies + +You built a harness that reports how well something is verified — a mutation run, a +doc/spec gate suite, a matrix of CI checks — and you are about to cite its score as +evidence in a commit message, PR body, README, or report. Also when its verdicts +come out uniform across every case: every mutant caught, or every one surviving. + +## Do this + +1. **Run a case whose correct verdict is the opposite of the failure you are + hunting, and require that verdict, before citing any score.** For a mutation + harness the control is a semantics-preserving change (reformat, rename a local, + edit a comment or docstring). That is an *equivalent mutation* — PIT's term for a + mutant that "behaves in exactly the same way as the original" — so no correct + test can kill it. Require **survived**. When the harness reports it caught, stop + and report that nothing was measured: the cases are failing before the rule under + test ever runs. +2. **Read a uniform verdict as a property of the harness, not of the code:** + +| Observed | Read it as | Do | +|----------|------------|-----| +| Mixed verdicts, and the no-op control survived | The harness discriminates | Cite the score together with the control's result | +| Every case caught / red, including the no-op control | Cases die before the rule executes — a broken isolated environment (missing input files, absent dependency, wrong working directory) | Fix the environment, then re-run the control; a 100% catch rate here is a 0% detection rate | +| Every case survives / green | The harness never applied the mutation or never reached the rule — Stryker documents "all mutants survive unexpectedly" as a configuration fault (hidden directory names, imports missing from test files) | Verify one mutation reaches the artifact by hand before adjusting the rules | + +3. **Observe the harness produce a verdict on the unmutated artifact first.** Both + mutation frameworks build this in: Stryker's initial test run has its own options + (`dryRunOnly` — "Execute the initial test run only without doing actual mutation + testing") and is what establishes baseline coverage and timing. A run that has + never been seen to report the unmutated state has no reference point. +4. **Give the harness a working tree equivalent to the real runner's.** When + isolating into a temp directory, copy the whole repository rather than the + directory under test — a partial copy silently removes fixtures, data files, and + path anchors that tests resolve relative to the repo root. +5. **Make "the case never ran" a distinct outcome from "the case ran and passed."** + Count executed cases and fail the harness when the count is zero, the way Stryker + exits on "No tests were executed" and PIT separates **no coverage** from + **survived** ("the same as Survived except there were no tests that exercised the + line of code where the mutation was created"). +6. **Score `detected / valid`, keeping errored cases out of the numerator *and* the + denominator.** Stryker models `Killed` and `Survived` alongside `Timeout`, + `Runtime error` and `Compile error`, and computes the score as + "detected / valid * 100". A case that blew up before reaching the rule is invalid, + not a detection — folding the two together is the arithmetic that turns a broken + environment into a perfect score. +7. **Publish the score with the control alongside it** — "34/36 caught; no-op + control survived" — so the number carries its own proof of discrimination. + +## Edge cases + +| Case | Then | +|------|------| +| A no-op change is impossible to construct (fully generated artifact) | Use a whitespace- or comment-only edit of the generator's input, and require the same survived verdict | +| The no-op control is legitimately caught | An assertion is pinned to formatting rather than behavior — narrow that assertion, then re-run; the control is measuring the right thing and finding a real over-specification ([testing-quality-tests-that-cannot-fail]) | +| The harness genuinely catches every real mutant (small rule set, exhaustive cases) | The no-op control is then the only evidence separating that from a broken harness — report it explicitly rather than the bare percentage | +| A mutation changes behavior in a way outside what the suite is meant to cover | PIT's second undetectable class (it excludes logging code for this reason) — exclude that region from the mutation set instead of adding a test to chase it | +| The score is already published and cited | Re-run with the control before defending the number, and correct the citation when the control fails | +| Individual checks each have a negative control already | Add the harness-level control too — a per-check control asks "can this check go red", the harness control asks "can this harness go green"; the second failure mode survives the first | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Cite "N/N caught" as proof the rules are enforced | Run the no-op control first and cite the score with its result | A harness whose cases all die before the rule runs reports every mutant as caught while detecting nothing | +| Read a uniform 100% detection rate as strength | Treat uniformity as the fault signal and run the control | Discriminating measurement produces mixed results; a single verdict for every input is what a constant function looks like | +| Tighten the rules when every case comes back red | Verify one case reaches the rule, then re-run the control | Rules are not what fails when the environment is missing the inputs the cases need | +| Copy only the directory under test into the harness's temp tree | Copy the repository, or fail the case on a missing input | Tests that resolve paths from the repo root read as detections when they die on a missing file | + +## Sources + +- https://pitest.org/quickstart/basic_concepts/ — "not all mutations will behave differently than the unmutated class. These mutants are referred to as **equivalent mutations**"; "The resulting mutant behaves in exactly the same way as the original"; a second undetectable class "behaves differently but in a way that is outside the scope of testing" (PIT excludes logging code); "**No coverage** is the same as **Survived** except there were no tests that exercised the line of code where the mutation was created" +- https://stryker-mutator.io/docs/stryker-js/configuration/ — the initial test run is a distinct phase with its own options: `dryRunOnly` "Execute the initial test run only without doing actual mutation testing", `dryRunTimeoutMinutes`, and coverage/timing analysis derived from that run +- https://stryker-mutator.io/docs/stryker-js/troubleshooting/ — "all mutants survive unexpectedly" is documented as a configuration/environment fault (hidden directory names, imports missing from test files); "No tests were executed. Stryker will exit prematurely. Please check your configuration." +- https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/ — the state set (`Killed`, `Survived`, `No coverage`, `Timeout`, `Runtime error`, `Compile error`, `Ignored`) and the score as "detected / valid * 100", so errored cases leave the denominator rather than counting as catches +- https://stryker-mutator.io/docs/mutation-testing-elements/equivalent-mutants/ — an equivalent mutant cannot be killed and "There is no definitive way for Stryker to find and ignore them", which is why a surviving no-op is the correct control verdict +- https://testing.googleblog.com/2021/04/mutation-testing.html — inserting faults and requiring test failure is what measures detection, as opposed to coverage +- Field reproduction 2026-07-31 (Python rule-conformance harness): the harness copied only the implementation directory into its temp tree while the tests resolved `examples/*.json` from the repo root, so all 105 tests died on `FileNotFoundError` and every mutation reported as caught — "36/36" was cited in five commits and a README. A docstring-capitalization no-op reproduced the red verdict and exposed it; copying the full repository plus adding the no-op control moved the score to 34/36 and surfaced two rules no test asserted diff --git a/wiki/testing/quality/tests-that-cannot-fail.md b/wiki/testing/quality/tests-that-cannot-fail.md index f55c155..e4e791e 100644 --- a/wiki/testing/quality/tests-that-cannot-fail.md +++ b/wiki/testing/quality/tests-that-cannot-fail.md @@ -11,7 +11,7 @@ sources: - https://martinfowler.com/bliki/TestCoverage.html - https://testing.googleblog.com/2013/05/testing-on-toilet-dont-overuse-mocks.html last_verified: 2026-07-10 -related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing] +related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-harness-reverse-controls] --- # Proving a Test Can Fail @@ -50,6 +50,7 @@ suite reported as covered, or you are auditing a suspiciously green suite. |------|------| | Mutating the code under test is impractical right now (slow build, shared branch) | Invert the expected value in the assertion instead and require red — this proves the assertion executes and compares, though not which code defects it catches | | Auditing a whole suite, not one test | Run an automated mutation-testing tool (PIT, Stryker) and treat surviving mutants in changed code as missing or defective tests | +| The mutation run is your own script rather than PIT/Stryker | Prove the harness discriminates before citing its score — a semantics-preserving no-op must survive ([testing-quality-harness-reverse-controls]) | | A test intentionally has no outcome assertion (smoke test: module loads, page renders) | Keep it only when the regression it guards manifests as a throw; name it as a smoke test so reviewers do not count it as behavior coverage | | The always-green test is a snapshot approved without reading | Snapshot rules → [testing-quality-behavior-not-implementation] | From 625cf1dfb7521b07e52451cc7c5751f6aa7dcd03 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=B5=9C=EC=98=81=EA=B8=B0?= Date: Sun, 2 Aug 2026 16:08:39 +0900 Subject: [PATCH 2/2] knowledge: fix 5 sourcing defects found by independent cross-check Removes a fabricated Stryker quotation ("all mutants survive unexpectedly" is not its wording; the real headings are runner-specific and the causes are sandbox mechanics), scopes the baseline-run claim to Stryker alone, corrects the initial-run derivation to timing only, labels the Vitest-runner message as such, and attributes three field-derived claims in the LLM page to the reproduction instead of leaving them adjacent to doc citations. --- .dev-loop/INGEST_REPORT.md | 19 ++++++++++++++++ .../common/llm/context-window-budget.md | 21 ++++++++++-------- .../quality/harness-reverse-controls.md | 22 ++++++++++--------- 3 files changed, 43 insertions(+), 19 deletions(-) diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 819f42c..b1d971c 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -146,6 +146,25 @@ replacement; branches as tables. already processed into PRs #11/#12; they came back because the dead flush never ran step 5. Retiring them is handled below. +## Cross-check + +Cross-Check: independent headless `claude` run (`--permission-mode plan`) re-fetched all +11 cited URLs and audited every quoted string to refute it — **PASS on both pages**, with +5 sourcing defects found and all 5 fixed in the follow-up commit. + +| Finding | Status | +|---------|--------| +| **MISQUOTED** — `"all mutants survive unexpectedly"` is not Stryker's wording | **Fixed.** I re-fetched the page and confirmed: the real headings are "All mutants survive - Jest runner" and "All mutants survive - module-alias", and both causes are sandbox mechanics (hidden temp dir; `module-alias/register` unsupported), not the "configuration fault" gloss. Quotes removed, causes attributed to the two sections. This was a fabricated quotation and the single most important catch | +| **OVERREACH** — "Both mutation frameworks build this in" (baseline run) | **Fixed.** Only Stryker is sourced for it; now attributed to Stryker's "Initial test run fails" section and `dryRunOnly` alone | +| **OVERREACH** — "coverage/timing analysis derived from that run" | **Fixed.** The config page supports timing (`netTimeMs`/`overheadMs`) only; coverage claim dropped | +| Minor — "No tests were executed…" generalized to framework policy | **Fixed.** Now labelled as the Vitest runner's message, which is how the doc presents it | +| **UNSUPPORTED ×3** in the LLM page — the 400 message stating the arithmetic; a gateway `max_output_tokens` key; the `/v1` path-append rule | **Fixed.** All three are field-derived and were sitting adjacent to doc citations as if doc-backed; each is now explicitly attributed to the reproduction, and the gateway row no longer invents a config key | + +One citation could not be checked: the Google Testing Blog URL renders its body +client-side, so a fetch returns only page chrome. The claim attributed to it is generic +("inserting faults and requiring failure measures detection") and is independently +carried by the PIT and Stryker citations, but flagging it rather than calling it verified. + ## Reviewer notes - **PRs #11 and #12 add the same page path** (`non-interactive-cli-invocation.md`) and diff --git a/wiki/backend/common/llm/context-window-budget.md b/wiki/backend/common/llm/context-window-budget.md index a317e01..6053b89 100644 --- a/wiki/backend/common/llm/context-window-budget.md +++ b/wiki/backend/common/llm/context-window-budget.md @@ -45,18 +45,21 @@ error while the same client worked against the original endpoint. |--------|------| | Direct SDK/HTTP call | `max_tokens` in the request body | | Claude Code | `CLAUDE_CODE_MAX_OUTPUT_TOKENS` (output reservation) and `CLAUDE_CODE_MAX_CONTEXT_TOKENS` (input budget) — both present in the shipped v2.1.220 binary; the CLI's own message names the first as the fix for "exceeded the … output token maximum" | -| Gateway in front of many clients | The gateway's per-model `max_output_tokens` config, so a client that omits the cap still gets a valid one | +| Gateway in front of many clients | The gateway's own per-model output-cap setting (key name varies by gateway — read its model-config reference), so a client that omits the cap still gets a valid one | -4. **Point the base URL at the endpoint root, not `/v1`.** The client appends its - own path (`/v1/messages`), so a base URL that already ends in `/v1` produces - `/v1/v1/messages`. Claude Code's `ANTHROPIC_BASE_URL` "override[s] the API - endpoint to route requests through a proxy or gateway"; vLLM's own integration - sets it to `http://localhost:8000`. +4. **Point the base URL at the endpoint root, not `/v1`.** Claude Code's + `ANTHROPIC_BASE_URL` "override[s] the API endpoint to route requests through a + proxy or gateway", and vLLM's own integration sets it to `http://localhost:8000` — + the root, no `/v1`. The client appends `/v1/messages` itself, so a base URL already + ending in `/v1` yields `/v1/v1/messages` and 404s while looking like an auth or + routing fault (observed in the reproduction below; the doc gives the root form as + the example, not the append rule). 5. **Read a 400 on the first request as a budget error before touching the network.** LiteLLM raises `ContextWindowExceededError` (400) as a "special error - type for context window exceeded error messages"; the message states the - arithmetic (input + requested output vs the window). Recompute step 2 from those - three numbers. + type for context window exceeded error messages". In the reproduction below the + message carried the arithmetic itself (input + requested output vs the window); + when it does, recompute step 2 from those three numbers rather than measuring + anything. ## Edge cases diff --git a/wiki/testing/quality/harness-reverse-controls.md b/wiki/testing/quality/harness-reverse-controls.md index d91cf27..088268a 100644 --- a/wiki/testing/quality/harness-reverse-controls.md +++ b/wiki/testing/quality/harness-reverse-controls.md @@ -40,20 +40,22 @@ come out uniform across every case: every mutant caught, or every one surviving. |----------|------------|-----| | Mixed verdicts, and the no-op control survived | The harness discriminates | Cite the score together with the control's result | | Every case caught / red, including the no-op control | Cases die before the rule executes — a broken isolated environment (missing input files, absent dependency, wrong working directory) | Fix the environment, then re-run the control; a 100% catch rate here is a 0% detection rate | -| Every case survives / green | The harness never applied the mutation or never reached the rule — Stryker documents "all mutants survive unexpectedly" as a configuration fault (hidden directory names, imports missing from test files) | Verify one mutation reaches the artifact by hand before adjusting the rules | +| Every case survives / green | The harness never applied the mutation or never reached the rule — Stryker's troubleshooting carries two distinct "All mutants survive" sections whose documented causes are both sandbox mechanics, not weak tests (the Jest runner cannot run in a hidden temp directory; sandboxing does not support `module-alias/register`) | Verify one mutation reaches the artifact by hand before adjusting the rules | -3. **Observe the harness produce a verdict on the unmutated artifact first.** Both - mutation frameworks build this in: Stryker's initial test run has its own options - (`dryRunOnly` — "Execute the initial test run only without doing actual mutation - testing") and is what establishes baseline coverage and timing. A run that has - never been seen to report the unmutated state has no reference point. +3. **Observe the harness produce a verdict on the unmutated artifact first.** Stryker + makes this a named phase — "Initial test run fails" is its own documented failure + mode, and `dryRunOnly` ("Execute the initial test run only without doing actual + mutation testing") runs the phase alone; the run's timing (`netTimeMs`, + `overheadMs`) is derived from it. A harness never seen reporting the unmutated + state has no reference point. 4. **Give the harness a working tree equivalent to the real runner's.** When isolating into a temp directory, copy the whole repository rather than the directory under test — a partial copy silently removes fixtures, data files, and path anchors that tests resolve relative to the repo root. 5. **Make "the case never ran" a distinct outcome from "the case ran and passed."** - Count executed cases and fail the harness when the count is zero, the way Stryker - exits on "No tests were executed" and PIT separates **no coverage** from + Count executed cases and fail the harness when the count is zero. Stryker's Vitest + runner does exactly this — "No tests were executed. Stryker will exit prematurely. + Please check your configuration." — and PIT separates **no coverage** from **survived** ("the same as Survived except there were no tests that exercised the line of code where the mutation was created"). 6. **Score `detected / valid`, keeping errored cases out of the numerator *and* the @@ -88,8 +90,8 @@ come out uniform across every case: every mutant caught, or every one surviving. ## Sources - https://pitest.org/quickstart/basic_concepts/ — "not all mutations will behave differently than the unmutated class. These mutants are referred to as **equivalent mutations**"; "The resulting mutant behaves in exactly the same way as the original"; a second undetectable class "behaves differently but in a way that is outside the scope of testing" (PIT excludes logging code); "**No coverage** is the same as **Survived** except there were no tests that exercised the line of code where the mutation was created" -- https://stryker-mutator.io/docs/stryker-js/configuration/ — the initial test run is a distinct phase with its own options: `dryRunOnly` "Execute the initial test run only without doing actual mutation testing", `dryRunTimeoutMinutes`, and coverage/timing analysis derived from that run -- https://stryker-mutator.io/docs/stryker-js/troubleshooting/ — "all mutants survive unexpectedly" is documented as a configuration/environment fault (hidden directory names, imports missing from test files); "No tests were executed. Stryker will exit prematurely. Please check your configuration." +- https://stryker-mutator.io/docs/stryker-js/configuration/ — the initial test run is a distinct phase with its own options: `dryRunOnly` "Execute the initial test run only without doing actual mutation testing", `dryRunTimeoutMinutes`; run timing (`netTimeMs`/`overheadMs`) is calculated during it +- https://stryker-mutator.io/docs/stryker-js/troubleshooting/ — section headings "Initial test run fails", "All mutants survive - Jest runner" (cause: Jest "doesn't support running in a hidden directory on windows") and "All mutants survive - module-alias" (cause: "StrykerJS's sandboxing does not support alias imports like `module-alias/register`") — both sandbox mechanics rather than test weakness; the Vitest-runner example "No tests were executed. Stryker will exit prematurely. Please check your configuration." - https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/ — the state set (`Killed`, `Survived`, `No coverage`, `Timeout`, `Runtime error`, `Compile error`, `Ignored`) and the score as "detected / valid * 100", so errored cases leave the denominator rather than counting as catches - https://stryker-mutator.io/docs/mutation-testing-elements/equivalent-mutants/ — an equivalent mutant cannot be killed and "There is no definitive way for Stryker to find and ignore them", which is why a surviving no-op is the correct control verdict - https://testing.googleblog.com/2021/04/mutation-testing.html — inserting faults and requiring test failure is what measures detection, as opposed to coverage