From d43ab2fb70320e7aefbbfe3b16aa2cd799b2b62e Mon Sep 17 00:00:00 2001 From: dch0202 Date: Tue, 4 Aug 2026 15:34:43 +0900 Subject: [PATCH 1/2] knowledge: ingest 7 verified insight(s) --- .dev-loop/INGEST_REPORT.md | 329 +++++++++++++----- INDEX.md | 4 +- log.md | 2 + .../robots-txt-and-source-selection.md | 90 +++++ .../signature-change-call-sites.md | 86 +++++ .../reliability/client-side-rate-limiting.md | 87 +++++ wiki/backend/index.md | 10 +- wiki/testing/index.md | 7 +- wiki/testing/quality/completion-predicates.md | 86 +++++ .../quality/differential-run-agreement.md | 88 +++++ .../quality/harness-reverse-controls.md | 22 +- .../testing/quality/tests-that-cannot-fail.md | 29 +- 12 files changed, 743 insertions(+), 97 deletions(-) create mode 100644 wiki/backend/common/integrations/robots-txt-and-source-selection.md create mode 100644 wiki/backend/common/refactoring/signature-change-call-sites.md create mode 100644 wiki/backend/common/reliability/client-side-rate-limiting.md create mode 100644 wiki/testing/quality/completion-predicates.md create mode 100644 wiki/testing/quality/differential-run-agreement.md diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index c375750..9174533 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,95 +1,254 @@ -# Consolidated review — knowledge PRs #6–#13 +# Knowledge flush — 7 insight(s) -Eight fork PRs (`dch0202-rsquare`, 2026-07-28 → 2026-08-02) were reviewed together -against `AGENTS.md`. Each PR was audited by an independent reviewer (format rules, -sources, vague-qualifier ban, ≤120 body lines, index/log invariants), then -cross-compared to catch duplication the per-PR flushes could not see — they branched -independently off the same main and rewrote the same shared index/log files. Fork -branches can't be edited from here and several PRs needed content changes (drop a -duplicate, merge a colliding page), so this branch carries the reconciled end-state -rather than merging each PR as-is (which would import the duplicates). +Drained the whole pending queue (`~/.dev-loop/queue/`, 7 rows across 5 session +files). Result: **5 new pages, 2 amendments, 1 new category**. Two candidate +claims were **disproved during verification** and the resulting directives were +rewritten before ingest — details under *Verified best-practice* (B1). + +| # | Candidate (trigger, abbreviated) | Outcome | Confidence | +|---|----------------------------------|---------|------------| +| T1 | Differential harness reports EQUIVALENT while one side models less state | NEW `testing/quality/differential-run-agreement` | verified | +| T2 | Proving a test file "can fail" after assertions moved into a shared contract | MERGE → `testing/quality/tests-that-cannot-fail` | verified | +| T3 | Monkeypatched deliberate-fault control that never reaches the seam | MERGE → `testing/quality/harness-reverse-controls` | verified | +| T4 | Signature change; enumerating call sites for migration | NEW `backend/common/refactoring/signature-change-call-sites` (new category) | verified | +| D1 | Monitor whose grep completion predicate reports done immediately | NEW `testing/quality/completion-predicates` | verified | +| B1 | Picking a crawl source for Korean public-institution notices | NEW `backend/common/integrations/robots-txt-and-source-selection` (**claim corrected**) | verified | +| B2 | Client throttle in place, still rate-limited on the first call | NEW `backend/common/reliability/client-side-rate-limiting` | field-tested | + +--- ## Verified best-practice -Sources are per-page and were live-verified in each originating PR's flush; the -independent re-reviews re-checked them. Landed pages and their evidence base: - -| Page | Confidence | Source basis | -|------|-----------|--------------| -| backend/common/llm/completion-response-validation | verified | OpenAI reasoning guide + chat `object` spec (5 `finish_reason` values), vLLM/LiteLLM reasoning fields; field incident (200/`length`/empty content/8,173-char reasoning) | -| backend/common/llm/context-window-budget | verified | Claude context-window docs, LiteLLM exception mapping, vLLM/Claude Code env-var docs | -| backend/common/integrations/externally-owned-defaults | verified | OpenAI deprecations (notice windows) + models `list`, LiteLLM model_discovery; field incident (alias removed between PR verify and review → 400) | -| backend/common/storage/object-key-persistence | verified | AWS S3 CompleteMultipartUpload + managed-upload API/source, aws-sdk-js issues #1158/#5656 | -| infrastructure/containers/host-cgroup-visibility | field-tested | cgroup_namespaces(7), Docker `--cgroupns=host`, nsenter, k8s #103363; OrbStack repro | -| infrastructure/observability/missing-container-metrics | verified/field-tested | k8s resource-metrics-pipeline docs, kube-prometheus-stack values, kubernetes-mixin; OrbStack #2217 repro | -| platforms/environment/unicode-text-matching | verified | UAX #15, Unicode core §3.12, APFS FAQ, POSIX grep; local repro (macOS 15/APFS, grep 2.6.0-FreeBSD, Python 3.13) | -| platforms/shells/command-text-inspected-before-execution | verified | Claude Code hooks docs, POSIX shell §2.6; local reproduction | -| platforms/processes/non-interactive-cli-invocation | verified | GNU nohup, OpenBSD ssh/ssh_config, git, timeout man pages; no-request-in-gateway-log field incident | -| qa/document-verification/spec-document-gates | field-tested | ESLint, Google mutation testing, RFC 2119, Vale, markdownlint; 32/32 mutant / 62/62 intact RFC sessions | -| qa/document-verification/editing-a-gated-document | field-tested | pgrep, Vale, markdownlint; in-house editing methodology | -| testing/quality/checks-that-cannot-pass | verified | James Shore AoAD2, POSIX grep exit status, Semgrep rule-testing, pytest exit codes; BSD/ugrep measurement | -| testing/quality/spec-artifact-checks | verified | JSON Schema, ESLint RuleTester, pitest, GFM table spec; local cell-count repro + GitHub renderer cross-check | -| testing/quality/harness-reverse-controls | verified | mutation-testing + CI-control sources; field repro (re-fetched all cited URLs, PASS) | - -Three pages were reconciled from two overlapping PR versions each, keeping the more -complete/better-sourced body and folding in the other's unique cases: -- **completion-response-validation** — #12 body (all five `finish_reason` values, - `tool_calls`/`function_call` carve-out, streaming, Responses API, "reasoning is - scratch, not deliverable") kept in `llm/` (coherent with #6/#13); folded in #6's - DeepSeek first-party edge + the field incident. -- **externally-owned-defaults** — #12 generalized body (any repo-external resource) - in `integrations/`; folded in #6's alias-removed field incident + the - gateway-config-vs-live-upstream nuance. -- **non-interactive-cli-invocation** — #12 body (GNU-nohup extension precision, - ssh -n stdin-detach vs BatchMode, pre-log DNS/TLS/proxy + `curl -v`) kept; folded - in #11's DEBIAN_FRONTEND, pager/color TTY case, wrapper-CLI case, field incident. +**T1 — a differential "agree" verdict is scoped to inputs where the asymmetric state is reachable.** +Checked: McKeeman, *Differential Testing for Software*, Digital Technical Journal +10(1) 1998 pp. 100–107 (the method is a pseudo-oracle: agreement is only as +strong as the inputs that could have exposed a difference); + (RIP — "Finding test cases to kill the alive +mutants in Mutation testing needs to calculate the Reachability, Infection and +Propagation(RIP) conditions": detection requires all three, so an unreached +dimension yields no verdict); +(**No coverage** is "the same as Survived except there were no tests that +exercised the line of code where the mutation was created" — the tools already +keep "never reached" separate from "reached and not detected"); + +(score = "detected / valid * 100"). The session's own evidence reproduces it: +`lnpl diff` on a read-then-create module → `EQUIVALENT` 4/4 on the default seed, +`DIVERGENT` `A=failed B=completed` under `--no-row`. → **verified**. + +**T2 — mutation granularity is per assertion, and the kill is attributed per test.** +Checked: — **Killed** = "a test +caught the mutation successfully", **Survived** = "the mutation was not detected +by the covering test", **No coverage** = no test exercised the line. Those are +exactly the three outcomes of a per-assertion mutation, and the per-test +attribution is what makes "which test reddened" the unit of proof rather than +"did the file redden". Session evidence: renaming `workflow Checkout` reddened +`test_source_compiles_to_the_committed_ir` but not +`test_node_ids_and_order_are_stable`, which pins only the first node id and the +trailing capability ids. → **verified**. + +**T3 — a patch-based control must target a seam the path reads at call time.** +Checked: "Where to patch" +— `patch()` "works by (temporarily) changing the object that a *name* points to +with another one … you must ensure that you patch the name used by the system +under test", and "You patch where an object is *looked up*, which is not +necessarily the same place as where it is defined". The candidate is a genuine +extension of that rule: the name is patched *correctly* and still never read, +because an upstream caller resolved the defaulted value and passed it explicitly, +leaving the callee's `if x is None` branch dead for that path. Session evidence: +patching `backend.seeded_entities` → control green (`AssertionError: True is not +false`); patching `backend.READ_OPS` / `backend._failure_attempts`, both read +unconditionally, → the expected `FAIL 2/4` and `FAIL 3/4`. → **verified**. + +**T4 — enumerate call sites by callee; a positional argument carries no name.** +Checked: (a call binds +positional arguments by position; the parameter name is simply absent from a +positional call site's source text, so a name search structurally cannot see +those callers); and + ("Safe code refactoring for modern +Python … guaranteeing that the resulting code compiles and runs") for the +sourced replacement action — match `Call` nodes in a CST rather than text. +Session evidence: `grep -rn "repo_rows" impl/tests/` → 13 hits, all keyword form; +full suite → `Ran 472 tests / FAILED (failures=11)`, all from 8 positional +`verify()` call sites plus a `rows_for()` helper feeding 5 more. → **verified**. + +**D1 — the bracket-expression false positive, reproduced from scratch.** +Checked: — `-F` "Interpret +patterns as fixed strings, not regular expressions"; `-c` prints "a count of +matching lines"; `-v` "Invert the sense of matching". + — a +bracket expression matches a *single* character from the enclosed set. +**Reproduced live on this machine (2026-08-04, macOS/BSD grep)** on a 3-line +status file with one `[completed]` and two in-flight lines: + +``` +grep -c '[completed]' → 3 (every line matches the character class) +grep -qv '[completed]' → selects nothing → "ALL COMPLETED" ← false positive +grep -cF '[completed]' → 1 vs total 3 → "still running (1/3)" +``` + +→ **verified** (reproducible command sequence, not just the original incident). + +**B1 — candidate claim DISPROVED; directive rewritten.** The candidate asserted +"기관 자체 사이트는 전면 크롤링 거부(`Disallow: /`)" with `www.i-sh.co.kr` as evidence. +Live fetch 2026-08-04 shows that is **wrong as stated**: + +- `www.i-sh.co.kr/robots.txt` gives `User-agent: *` a **path list only** + (`/admin`, `/cert`, `/upload`, `/gcms/brd`, per-district board paths). The + `Disallow: /` lines belong to 15 **named** groups — `GPTBot`, `ChatGPT-User`, + `facebookexternalhit`, `BaiDuSpider`, `MJ12bot`, `OAI-SearchBot`, + `PerplexityBot`, `Google-Extended`, `ClaudeBot`, `Claude-SearchBot`, + `meta-externalAgent`, `Applebot-Extended`, `CCBot`, `aiohttp`, `DuckDuckBot`. + The candidate read the file's last line without its group header. +- `www.gh.or.kr/robots.txt` is **not** `Disallow: /` — it returns **HTTP 410**, + which RFC 9309 classifies as *unavailable*: "the crawler MAY access any + resources on the server". That is the opposite of the candidate's reading. +- Confirmed as stated: `housing.seoul.go.kr` → `User-agent: * / Allow: /`; + `apply.gh.or.kr` → `User-agent: * / Allow: /*`. + +Grounded in : "Crawlers MUST use +case-insensitive matching to find the group that matches the product token and +then obey the rules of the group"; `*` applies only "If no matching group +exists"; multiple matching groups "MUST be combined into one group"; the product +token "should appear as a substring in the crawler's user-agent header" +(so a client on its library's default UA — aiohttp autogenerates one per + — can land in a named +disallowed group); 4xx → "MAY access any resources", 5xx/unreachable → "MUST +assume complete disallow"; five-redirect limit; "SHOULD NOT use the cached +version for more than 24 hours". The page's directive is therefore **"read the +group your product token matches, and branch on the response status"**, with the +republishing-portal fallback as a secondary step — not the candidate's "the +origin blocks crawling". → **verified** (and materially more useful than the +candidate). + +**B2 — mechanism verified, vendor numbers NOT verified → field-tested.** +Sourced the general claims: (the +token endpoint is reached by an ordinary HTTP request from the client, so +credential acquisition consumes the same request budget as data calls) and + (429 "indicates that the user has +sent too many requests in a given amount of time", `Retry-After`). I could +**not** confirm the candidate's specific "한도 2건/초" or a token-issuance quota +from Korea Investment & Securities' official developer portal — public sources I +found cite a different per-second figure, so the page carries **no vendor +numbers** and states the mechanism only (auth refresh inside `_headers()` sits +outside a throttle wrapped around the call layer; a zero-initialized +`_last_request_at` makes the first wait evaluate to zero). The timestamped log +evidence (POST `:00.354` → token `:00.495` → rejected `:00.543`, on two separate +token-issuance days, clean on cached-token days) is real production observation. +→ **field-tested**, labelled as such in frontmatter and stated plainly in the +page's Sources. + +No candidate was dropped, and nothing was upgraded to `verified` without a cited +source or a reproduction. Every cited link was fetched (HTTP 200) or is a +bibliographic citation with no URL (McKeeman 1998). + +--- ## Existing-layer check -Cross-PR and against-main duplication was the focus. Findings and resolutions: - -- **spec-artifact-checks (#8) ≡ document-conformance-checks (#9)** — same case - (coverage-vs-validity split, per-check negative controls, GFM pipe parsing, - ESLint/Semgrep/mutation examples). #9's report predated awareness of #8. → - **#8 kept canonical; #9's page dropped, `testing/docs-as-spec` category not created.** -- **completion-response-validation (#6) ≈ llm-response-completeness (#12)** — ~95% - same case (HTTP 200 ≠ usable output; `length`/blank/reasoning-budget). → - **merged into one `llm/` page; #12's `integrations/` copy dropped.** -- **gateway-model-alias-defaults (#6) ≈ externally-owned-defaults (#12)** — ~80%; - #12 generalizes the model-alias case to any external resource. → - **kept the general `integrations/` page; #6's LLM-only page dropped.** -- **non-interactive-cli-invocation** — created by BOTH #11 and #12 (file collision). - → **single reconciled page.** -- Distinct (no overlap, all landed): checks-that-cannot-pass, harness-reverse-controls, - spec-document-gates, editing-a-gated-document, unicode-text-matching, - command-text-inspected-before-execution, object-key-persistence, context-window-budget, - host-cgroup-visibility, missing-container-metrics. -- Reciprocal `related:` links added on existing pages (tests-that-cannot-fail, - timeouts-and-retries, environment-config, release-gates, background-services, - portable-shell-scripts, timezone-and-locale, paths-case-and-line-endings, - acceptance-criteria, resource-limits-and-probes, logs-metrics-signals, - minimum-case-set). A dropped-page backlink (#6 → gateway-model-alias-defaults on - environment-config and release-gates) was retargeted to externally-owned-defaults. -- Invariants verified programmatically: all `related:`/inline `[id]` references - resolve, every page listed in its domain index, no duplicate ids, no page >120 - body lines. +Routed via `INDEX.md` → domain indexes → every page whose "load when" overlapped. + +**Pages read in full before writing** (dedup + conflict + link decisions): +`wiki/testing/quality/harness-reverse-controls.md`, +`wiki/testing/quality/tests-that-cannot-fail.md`, +`wiki/testing/quality/checks-that-cannot-pass.md`, +`wiki/backend/common/reliability/timeouts-and-retries.md`, plus the `testing`, +`backend`, `debugging`, and `platforms` domain indexes, `INDEX.md`, `AGENTS.md` +and `templates/page.md`. + +**Merged rather than created (merge-before-create):** + +- **T2 → `tests-that-cannot-fail`.** Its step 1 already prescribes manual + mutation testing; the candidate refines the *granularity* of that same case, so + a new page would have split one case in two. Added: a new step 2 (one mutation + per assertion, with a three-outcome table mapping to PIT's + Killed/Survived/No-coverage), a never-fails-pattern row for assertions + inherited into a shared base class/mixin/parameterised harness, two edge-case + rows, two `Instead of` rows, the PIT source, and the field reproduction. + Renumbered the following steps; `last_verified` → 2026-08-04. +- **T3 → `harness-reverse-controls`.** The candidate explicitly lands in that + page's "every case green" failure mode, but the page documented only Stryker's + two *sandbox-mechanics* causes. Added a third cause as a table row, a new Do + item 7 (patch a seam the path consults at call time; require the control red), + an edge-case row for the refactor that creates the dead default, an + `Instead of` row, the `unittest.mock` "Where to patch" source, and the field + reproduction. Page stayed at 97 body lines (limit 120). + +**Overlaps found but judged distinct (kept separate, cross-linked):** + +- `checks-that-cannot-pass` vs **D1** — exact mirror cases: that page is "a gate + only ever observed *failing*"; D1 is a predicate only ever observed + *succeeding*. Its grep material is about exit-status semantics (0/1/>1) and + `-q` masking; D1's is about a literal degrading into a bracket expression + through quoting layers, and its subject is a polling monitor rather than a + gate. Cross-linked both ways via `related`. +- `harness-reverse-controls` vs **T1** — same theme ("a uniform verdict is a + property of the harness"), different case and different remedy: the control + there is a semantics-preserving *mutation* that must survive; here it is an + *input* that forces an unmodelled dimension to decide. Made a separate page and + cross-linked; also added a T1 pointer to `tests-that-cannot-fail`'s edge cases. +- `timeouts-and-retries` vs **B2** — that page owns the *reactive* path (its only + rate-limit content is one row: "429 | Wait the `Retry-After` value…"). B2 is + *proactive* quota pacing and which requests count. Kept separate; B2's step 7 + defers to it for the reactive half and `related`-links it. +- `externally-owned-defaults` vs **B1** — B1's source host/path/id scheme is an + externally-owned default; referenced from B1's edge cases rather than + duplicated. + +**Conflicts flagged:** none. No existing directive is contradicted or +overwritten; the two amendments only extend their pages. + +**Links added:** `differential-run-agreement` ↔ `harness-reverse-controls`, +`tests-that-cannot-fail`, `minimum-case-set`; `completion-predicates` ↔ +`checks-that-cannot-pass`, `tests-that-cannot-fail`, `portable-shell-scripts`, +`background-services`; `robots-txt-and-source-selection` → `timeouts-and-retries`, +`externally-owned-defaults`; `client-side-rate-limiting` → `timeouts-and-retries`, +`jwt-server-side`, `intermittent-failures`; `signature-change-call-sites` → +`tests-that-cannot-fail`, `behavior-not-implementation`. + +**Invariants checked after the change:** all 7 touched pages ≤120 body lines +(74/73/79/75/73/77/97); every `related:` id across the whole wiki resolves to an +existing page id; every inline `[page-id]` reference in the touched pages +resolves; every markdown link in `INDEX.md`, `wiki/testing/index.md` and +`wiki/backend/index.md` resolves to a file; no banned vague qualifier remains in +the touched pages; `log.md` carries the appended `ingest` and `revise` entries. + +--- ## Routing decision -- `backend/common/llm/` (new) — LLM-specific server concerns: completion-response-validation, - context-window-budget. Coherent home shared by #6 and #13. -- `backend/common/integrations/` (new) — general repo-external-dependency concern: - externally-owned-defaults. Kept separate from `llm/` because its scope is any - external resource (bucket/queue/index), not LLM-only. -- `backend/common/storage/` (new) — object-key-persistence. -- `qa/document-verification/` (new) — spec-document-gates, editing-a-gated-document. - Introduced by both #10 and #11; unified into one index section. -- `testing/quality/` (existing) — checks-that-cannot-pass, spec-artifact-checks, - harness-reverse-controls (test/check-authoring discipline, distinct from - qa/document-verification which is release-process gate design). -- `platforms/{environment,shells,processes}/` (existing) — unicode-text-matching, - command-text-inspected-before-execution, non-interactive-cli-invocation. -- `infrastructure/{containers,observability}/` (existing) — host-cgroup-visibility, - missing-container-metrics. - -Source PRs #6–#13 are closed with a disposition comment crediting the author. +| Insight | Domain / category | Page | Note | +|---------|-------------------|------|------| +| T1 | `testing` / `quality` | **NEW** `quality/differential-run-agreement.md` | Domain owns "verifying tests can actually fail"; this is the differential-harness instance of that case | +| T2 | `testing` / `quality` | **MERGE** into `quality/tests-that-cannot-fail.md` | Same case at finer granularity — a new page would have split one case | +| T3 | `testing` / `quality` | **MERGE** into `quality/harness-reverse-controls.md` | The candidate names that page's own failure mode; adds a third documented cause | +| D1 | `testing` / `quality` | **NEW** `quality/completion-predicates.md` | Non-test checks already live in this category (`checks-that-cannot-pass` covers grep gates and "a plan's verification command"), so a polling monitor's predicate belongs here rather than in `debugging` (nothing is being diagnosed) or `platforms/shells` (the bracket-expression behaviour is not OS-specific) | +| B1 | `backend` / `integrations` | **NEW** `common/integrations/robots-txt-and-source-selection.md` | Existing category, language-agnostic, sits beside `externally-owned-defaults` (both are "a resource the repo does not own") | +| B2 | `backend` / `reliability` | **NEW** `common/reliability/client-side-rate-limiting.md` | Existing category; complements `timeouts-and-retries` (proactive pacing vs reactive retry) | +| T4 | `backend` / **`refactoring` (NEW category)** | **NEW** `common/refactoring/signature-change-call-sites.md` | Rationale below | + +**New category: `backend/common/refactoring`.** The candidate arrived tagged +`domain: testing` because a test suite is what caught the miss, but the practice +governs *changing application code*, not writing tests — routing protocol step 1 +sends it to the domain that owns the artifact being changed. No existing category +covers it: `testing/quality` is about what tests assert; +`testing/quality/behavior-not-implementation` covers "a behavior-preserving +refactor broke tests" (test coupling), which is the opposite direction — here the +tests were right and the migration was short; `debugging` is for diagnosing a +failure, and nothing is being diagnosed. Placed under `backend/common` because +`INDEX.md` assigns language-agnostic application-code concerns there, matching +the existing `common/api-design`, `common/orm`, `common/errors` pattern. Category +name follows the lowercase-kebab-noun rule; the page id is +`backend-common-refactoring-signature-change-call-sites`, matching its path. + +`INDEX.md` was updated (the backend and testing "route here when" lines) — no new +domain, so the domain table itself is unchanged. `wiki/backend/index.md` gained +the `refactoring` section plus two rows; `wiki/testing/index.md` gained two rows +and an extended "Route here for" line. + +--- + +**Review note.** Opened for review only; nothing is auto-merged. The one item +worth a maintainer's eye is B2's `field-tested` confidence — the mechanism and +the timestamped production evidence are solid, but the vendor's published quota +could not be confirmed, so the page deliberately carries no numbers. Reject that +page alone if the bar for a `reliability` entry is a cited provider limit. diff --git a/INDEX.md b/INDEX.md index d6d4238..d0bda68 100644 --- a/INDEX.md +++ b/INDEX.md @@ -10,10 +10,10 @@ follow the cross-pointers in their index or take the next matching seeded domain | Domain | Status | Route here when | |--------|--------|-----------------| | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior | -| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | +| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, client-side rate limiting, crawl-source selection via robots.txt, object-storage references, migrating call sites through a signature change) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility | | [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) | -| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) | +| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests, proving tests/harnesses/predicates can report the failing verdict — including differential-run agreement and background-work completion predicates (release-process quality → qa) | | [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing, automated verification of document deliverables (spec/RFC gates) (writing automated test code → testing) | | [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) | | [security](wiki/security/index.md) | **seeded** | Trust-boundary decisions: input validation, session-vs-token auth choice, per-resource authorization (IDOR), secrets hygiene, dependency trust, PII handling (XSS rendering → frontend; CI secrets → infrastructure; JWT implementation → backend/frontend auth) | diff --git a/log.md b/log.md index 1c6293b..703ab87 100644 --- a/log.md +++ b/log.md @@ -37,3 +37,5 @@ Append-only. Format: `## [YYYY-MM-DD] 1 = an error occurred, so a read error and a genuine no-match are distinguishable +- https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap09.html — a bracket expression `[...]` matches a single character from the enclosed set; this is what turns a literal marker into an eight-letter character class +- Field reproduction 2026-08-04 (macOS, BSD grep): a three-line status file with one `[completed]` and two in-flight lines — `grep -c '[completed]'` returned 3 (every line), `grep -qv '[completed]'` selected nothing so the "no unfinished lines" test reported ALL COMPLETED, and `grep -cF '[completed]'` returned 1 against a total of 3. The original incident was an eight-task fan-out with one task complete, reported by its monitor as "ALL 8 TASKS COMPLETED" on the first poll diff --git a/wiki/testing/quality/differential-run-agreement.md b/wiki/testing/quality/differential-run-agreement.md new file mode 100644 index 0000000..fa85e88 --- /dev/null +++ b/wiki/testing/quality/differential-run-agreement.md @@ -0,0 +1,88 @@ +--- +id: testing-quality-differential-run-agreement +domain: testing +category: quality +applies_to: [general] +confidence: verified +sources: + - W. M. McKeeman, "Differential Testing for Software", Digital Technical Journal 10(1), 1998, pp. 100-107 + - https://arxiv.org/abs/2410.21904 + - https://pitest.org/quickstart/basic_concepts/ + - https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/ +last_verified: 2026-08-04 +related: [testing-quality-harness-reverse-controls, testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set] +--- + +# Citing an "Agree" Verdict from a Differential Run + +## When this applies + +You ran two implementations of one spec on the same input, the harness reported +agreement (`EQUIVALENT`, "no diff", "N/N checks pass"), and you are about to cite +that verdict as evidence in a commit, PR, README, or report. Also when the two +sides model different amounts of state — one has a repository, cache, clock, or +session the other stubs out, omits, or hard-codes. + +## Do this + +1. **Before citing agreement, write down the state each side models and diff the + two lists.** Every entry that appears on one side only is a dimension the run + did not compare. A side that holds no repository cannot fail for a repository + reason, so on any input where the repository does not decide the outcome it + agrees with the stateful side by construction. +2. **Re-run with an input that makes each asymmetric dimension decide the + outcome, and report that verdict separately from the default-input one.** + Pick the input from what the dimension is: + +| Asymmetric state | Input that makes it decide | +|------------------|----------------------------| +| Repository / persisted rows | Empty seed, or a seed missing exactly the row the flow reads | +| Uniqueness or key constraint | Two operations writing the same key in one run | +| Cache | Cold cache, then a second run against the warm one | +| Clock / deadline / TTL | A duration past the deadline the flow enforces | +| Session or auth context | A request with the identity absent or belonging to another owner | + +3. **Read the default-input verdict as its literal claim: "the two sides agree on + inputs where the unmodelled state does not decide the outcome."** That is a + real result and a narrow one. Publish it in that form rather than as bare + agreement. +4. **Treat divergence under the forcing input as the run's most informative + output, not as a harness defect.** Record which side stopped and where — the + step at which the two traces separate localizes the fault to one side. +5. **When the state-blind side cannot represent the forcing input at all, + record the dimension as untested.** An input the harness cannot express is a + coverage gap, and calling it agreement asserts something the run never + evaluated. +6. **Ground the choice of input in reachability.** A fault is detected only when + the test reaches it, infects the state, and propagates that state to the + observed output (the RIP conditions). On the default input the asymmetric + dimension is never reached, so no verdict about it is possible — the same + reason a mutation-testing tool reports **no coverage** rather than **survived** + for a mutant no test executes. + +## Edge cases + +| Case | Then | +|------|------| +| Both sides model the state, and they agree | The verdict is informative for that dimension; cite it with the forcing input named so a reader can see which dimension was exercised | +| The state-blind side is a deliberate reference model (a simplified oracle) | Scope the harness's claim to the modelled subset in its own output, and add a separate check that the unmodelled subset is covered elsewhere | +| The forcing input makes both sides fail identically | Confirm they fail for the same reason before counting it as agreement; identical failure text from different causes is a coincidence, not a comparison | +| Agreement is reported across a matrix of inputs, all of them defaults | The matrix size is not evidence; one forcing input per asymmetric dimension outranks a hundred default rows | +| The harness reports agreement and the score has already been published | Re-run with the forcing inputs before defending the number, and correct the citation when a dimension turns out untested | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Cite `EQUIVALENT` on the default input as proof two implementations match | Cite it as agreement on inputs where the unmodelled state does not decide, plus the separate forcing-input verdict | The state-blind side agrees by construction on the happy path, so the verdict certifies nothing about the dimension the harness exists to check | +| Add more default-shaped inputs to raise confidence | Add one input per asymmetric dimension, chosen so that dimension decides the outcome | Inputs that never reach the asymmetry re-prove the same fact; RIP requires reachability before any detection is possible | +| Read a divergence under a forcing input as the harness being too strict | Read it as the finding, and localize by the step at which the traces separate | The forcing input is the only place the harness can produce information about that dimension | +| Report one combined verdict for a run with mixed inputs | Report the default verdict and the forcing verdict as two lines | A single line lets a narrow agreement read as full coverage | + +## Sources + +- W. M. McKeeman, "Differential Testing for Software", Digital Technical Journal 10(1), 1998, pp. 100-107 — the method: run two or more independent implementations of one specification on identical inputs and flag discrepancies; agreement is a pseudo-oracle, and its strength depends entirely on inputs that can expose a difference +- https://arxiv.org/abs/2410.21904 — "Finding test cases to kill the alive mutants in Mutation testing needs to calculate the Reachability, Infection and Propagation(RIP) conditions"; a fault is detected only when all three hold, so an unreached dimension yields no verdict +- https://pitest.org/quickstart/basic_concepts/ — **No coverage** is "the same as Survived except there were no tests that exercised the line of code where the mutation was created": the tool keeps "never reached" separate from "reached and not detected" +- https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/ — the score is "detected / valid * 100", so cases that never exercised the rule leave the denominator instead of counting as agreement +- Field reproduction 2026-08-04 (linkly `lnpl diff`, a read-then-create module): the default seed reported `EQUIVALENT` with 4/4 checks passing, because mode B's native backend models no repository; the same module under `--no-row` reported `DIVERGENT` with `A=failed B=completed` — mode A stopped at the failed read while mode B ran all four steps diff --git a/wiki/testing/quality/harness-reverse-controls.md b/wiki/testing/quality/harness-reverse-controls.md index 088268a..8106b0b 100644 --- a/wiki/testing/quality/harness-reverse-controls.md +++ b/wiki/testing/quality/harness-reverse-controls.md @@ -11,8 +11,9 @@ sources: - https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/ - https://stryker-mutator.io/docs/mutation-testing-elements/equivalent-mutants/ - https://testing.googleblog.com/2021/04/mutation-testing.html -last_verified: 2026-08-02 -related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set] + - https://docs.python.org/3/library/unittest.mock.html +last_verified: 2026-08-04 +related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, testing-quality-differential-run-agreement] --- # Citing a Verification Harness's Own Score @@ -41,6 +42,7 @@ come out uniform across every case: every mutant caught, or every one surviving. | Mixed verdicts, and the no-op control survived | The harness discriminates | Cite the score together with the control's result | | Every case caught / red, including the no-op control | Cases die before the rule executes — a broken isolated environment (missing input files, absent dependency, wrong working directory) | Fix the environment, then re-run the control; a 100% catch rate here is a 0% detection rate | | Every case survives / green | The harness never applied the mutation or never reached the rule — Stryker's troubleshooting carries two distinct "All mutants survive" sections whose documented causes are both sandbox mechanics, not weak tests (the Jest runner cannot run in a hidden temp directory; sandboxing does not support `module-alias/register`) | Verify one mutation reaches the artifact by hand before adjusting the rules | +| One deliberate-fault control is green while others in the same harness redden | The patched seam is not on the path that control exercises — the name is patched correctly and never read | Patch a seam the path consults at call time (next item), then re-run and require red | 3. **Observe the harness produce a verdict on the unmutated artifact first.** Stryker makes this a named phase — "Initial test run fails" is its own documented failure @@ -64,7 +66,17 @@ come out uniform across every case: every mutant caught, or every one surviving. "detected / valid * 100". A case that blew up before reaching the rule is invalid, not a detection — folding the two together is the arithmetic that turns a broken environment into a perfect score. -7. **Publish the score with the control alongside it** — "34/36 caught; no-op +7. **When a control works by patching a symbol, patch a seam the path under test + consults at call time, and require the control to go red before trusting it.** + Patching is name rebinding — "you patch where an object is *looked up*, which + is not necessarily the same place as where it is defined" — and a name the path + never reads is a no-op that reports green. Two resolution sites are the common + trap: a parameter that defaults to computing a value (`f(x=None)` → `x = rule()` + inside) is dead for any caller that resolves the value itself and passes it + explicitly, so patching the callee's copy of `rule` mutates nothing on that + path. Pick a symbol the path reads unconditionally, and confirm the control's + red run before citing any score derived from it. +8. **Publish the score with the control alongside it** — "34/36 caught; no-op control survived" — so the number carries its own proof of discrimination. ## Edge cases @@ -76,6 +88,7 @@ come out uniform across every case: every mutant caught, or every one surviving. | The harness genuinely catches every real mutant (small rule set, exhaustive cases) | The no-op control is then the only evidence separating that from a broken harness — report it explicitly rather than the bare percentage | | A mutation changes behavior in a way outside what the suite is meant to cover | PIT's second undetectable class (it excludes logging code for this reason) — exclude that region from the mutation set instead of adding a test to chase it | | The score is already published and cited | Re-run with the control before defending the number, and correct the citation when the control fails | +| A refactor moved value resolution from the callee's default into the caller | Re-verify every control that patches that symbol — each one silently became a no-op on the path it was written for, while still passing | | Individual checks each have a negative control already | Add the harness-level control too — a per-check control asks "can this check go red", the harness control asks "can this harness go green"; the second failure mode survives the first | ## Instead of @@ -86,6 +99,7 @@ come out uniform across every case: every mutant caught, or every one surviving. | Read a uniform 100% detection rate as strength | Treat uniformity as the fault signal and run the control | Discriminating measurement produces mixed results; a single verdict for every input is what a constant function looks like | | Tighten the rules when every case comes back red | Verify one case reaches the rule, then re-run the control | Rules are not what fails when the environment is missing the inputs the cases need | | Copy only the directory under test into the harness's temp tree | Copy the repository, or fail the case on a missing input | Tests that resolve paths from the repo root read as detections when they die on a missing file | +| Accept a green deliberate-fault control as "the harness cannot be fooled" | Require that control to go red first, patching a seam the path reads at call time | A patch applied to a name the path never looks up mutates nothing, so the control reports the harness's health without ever testing it | ## Sources @@ -95,4 +109,6 @@ come out uniform across every case: every mutant caught, or every one surviving. - https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/ — the state set (`Killed`, `Survived`, `No coverage`, `Timeout`, `Runtime error`, `Compile error`, `Ignored`) and the score as "detected / valid * 100", so errored cases leave the denominator rather than counting as catches - https://stryker-mutator.io/docs/mutation-testing-elements/equivalent-mutants/ — an equivalent mutant cannot be killed and "There is no definitive way for Stryker to find and ignore them", which is why a surviving no-op is the correct control verdict - https://testing.googleblog.com/2021/04/mutation-testing.html — inserting faults and requiring test failure is what measures detection, as opposed to coverage +- https://docs.python.org/3/library/unittest.mock.html — "Where to patch": `patch()` "works by (temporarily) changing the object that a *name* points to with another one … you must ensure that you patch the name used by the system under test", and "You patch where an object is *looked up*, which is not necessarily the same place as where it is defined" — patching a name the path does not read leaves the run unmutated +- Field reproduction 2026-08-04 (linkly `impl/lnpl/differential.py`): `verify` resolved `seeded_entities(...)` itself and passed the result into `observe_mode_b` → `backend.build(seeded=...)`, so `backend._lnpl_ops`'s own `if seeded is None` default never executed on that path. A control patching `backend.seeded_entities` reported `AssertionError: True is not false` — no divergence, control green. Repatching `backend.READ_OPS` and `backend._failure_attempts`, both read unconditionally inside the scan, produced the expected `FAIL 2/4` and `FAIL 3/4` - Field reproduction 2026-07-31 (Python rule-conformance harness): the harness copied only the implementation directory into its temp tree while the tests resolved `examples/*.json` from the repo root, so all 105 tests died on `FileNotFoundError` and every mutation reported as caught — "36/36" was cited in five commits and a README. A docstring-capitalization no-op reproduced the red verdict and exposed it; copying the full repository plus adding the no-op control moved the score to 34/36 and surfaced two rules no test asserted diff --git a/wiki/testing/quality/tests-that-cannot-fail.md b/wiki/testing/quality/tests-that-cannot-fail.md index 884ff61..47965da 100644 --- a/wiki/testing/quality/tests-that-cannot-fail.md +++ b/wiki/testing/quality/tests-that-cannot-fail.md @@ -10,8 +10,9 @@ sources: - https://testing.googleblog.com/2021/04/mutation-testing.html - https://martinfowler.com/bliki/TestCoverage.html - https://testing.googleblog.com/2013/05/testing-on-toilet-dont-overuse-mocks.html -last_verified: 2026-07-10 -related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-checks-that-cannot-pass, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, qa-document-verification-spec-document-gates] + - https://pitest.org/quickstart/basic_concepts/ +last_verified: 2026-08-04 +related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-checks-that-cannot-pass, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, testing-quality-differential-run-agreement, testing-quality-completion-predicates, qa-document-verification-spec-document-gates] --- # Proving a Test Can Fail @@ -29,7 +30,20 @@ suite reported as covered, or you are auditing a suspiciously green suite. it stays green, the test is decoration — locate its defect in the table below and fix the test, then re-verify red before restoring the code. This is manual mutation testing; run it whenever a test's value is in doubt. -2. Fix each never-fails pattern with its replacement: +2. **Seed one mutation per assertion, not one per file, and require exactly the + test that owns that assertion to redden.** Choose each mutation from what its + assertion actually reads. A file-level red proves the file executes and that + *some* assertion discriminates; it says nothing about the others, which is why + mutation tools attribute a kill to the covering test rather than to the file. + Read the outcome per assertion: + +| Mutation outcome | Read it as | Do | +|------------------|------------|-----| +| Exactly the expected test reddens | That assertion discriminates on that input | Record the pair (mutation → test) and move to the next assertion | +| The file reddens but the target test stays green | The target assertion is unproven; another one caught the mutation | Find the input this assertion owns — mutate what it reads, not what its name suggests | +| No test reddens | Nothing asserts that behavior | Add the assertion, then re-run the mutation | + +3. Fix each never-fails pattern with its replacement: | Never-fails pattern | Fix | |---------------------|-----| @@ -39,8 +53,9 @@ suite reported as covered, or you are auditing a suspiciously green suite. | Always-true assertion (`toBeDefined`/`toBeTruthy` on a value that is always defined, `expect(arr.length).toBeGreaterThanOrEqual(0)`) | Assert the specific expected value or shape — the observable-outcome rule in [testing-quality-minimum-case-set] | | Testing the mock instead of the code (mock returns X, test asserts X came back) | Assert the unit's transformation of its inputs, not the pass-through; when no transformation exists at this layer, test the layer that has one ([testing-mocking-what-to-mock]) | | Copied test body with the name changed but identical inputs and expectation | Give each case distinct inputs and its own expectation; delete exact duplicates — a renamed copy re-proves the same fact and guards nothing new | +| Assertion inherited from a shared base class, mixin, or parameterised harness, whose name announces the new subject's whole shape while its body pins the original narrow scope | Read the inherited body and list what it compares; add a subject-specific assertion for each part of the shape the name claims, then prove each one with its own mutation | -3. **Coverage note:** a covered line is only an executed line. Use coverage to +4. **Coverage note:** a covered line is only an executed line. Use coverage to find untested code; it cannot certify tested behavior. The proof a test works is the red run from step 1, not the coverage report. @@ -53,6 +68,8 @@ suite reported as covered, or you are auditing a suspiciously green suite. | The mutation run is your own script rather than PIT/Stryker | Prove the harness discriminates before citing its score — a semantics-preserving no-op must survive ([testing-quality-harness-reverse-controls]) | | A test intentionally has no outcome assertion (smoke test: module loads, page renders) | Keep it only when the regression it guards manifests as a throw; name it as a smoke test so reviewers do not count it as behavior coverage | | The always-green test is a snapshot approved without reading | Snapshot rules → [testing-quality-behavior-not-implementation] | +| Assertions were just pulled into a shared contract (base class, mixin, parameterised suite) so several subjects now run them | Re-prove each assertion against each subject: the assertion's scope stayed where it was written while its name now speaks for every subject, so one mutation per (assertion, subject) pair is the granularity | +| The two implementations under comparison model different amounts of state | An agreement verdict on the default input cannot fail for the unmodelled dimension → [testing-quality-differential-run-agreement] | ## Instead of @@ -62,6 +79,8 @@ suite reported as covered, or you are auditing a suspiciously green suite. | Prove an error path with `try { await f() } catch (e) { expect(e.message)... }` alone | Use `rejects`/`assertThrows`-style assertion, or add `expect.assertions(1)` above the try | When `f()` succeeds, the catch never runs and the test passes with zero assertions | | Trust "green suite + high coverage" as proof an area is tested | Break the behavior once and require a red run | Coverage counts execution, not detection; high numbers are reachable with assertion-free tests | | Delete a suspicious always-green test to clean up | Fix it via the table above, then re-verify it can fail | The test names a behavior someone meant to guard; deletion drops the intent along with the defect | +| Prove a test file can fail by seeding one mutation and watching the file go red | Seed one mutation per assertion and require exactly the owning test to redden | A file-level red is produced by whichever assertion happens to be strictest; the silent ones remain unproven | +| Pick a mutation from what the test's name says it covers | Pick it from what the assertion body actually reads | An assertion inherited into a shared contract keeps its original narrow scope, so a reasonable-looking mutation sails past it | ## Sources @@ -70,3 +89,5 @@ suite reported as covered, or you are auditing a suspiciously green suite. - https://testing.googleblog.com/2021/04/mutation-testing.html — inserting faults and requiring test failure measures whether tests detect bugs; coverage alone does not - https://martinfowler.com/bliki/TestCoverage.html — coverage finds untested code; it is not a measure of test quality - https://testing.googleblog.com/2013/05/testing-on-toilet-dont-overuse-mocks.html — mock-heavy tests can pass while the real code is broken +- https://pitest.org/quickstart/basic_concepts/ — a kill is attributed per covering test: **Killed** is "a test caught the mutation successfully", **Survived** is "the mutation was not detected by the covering test", and **No coverage** is "the same as Survived except there were no tests that exercised the line of code where the mutation was created" — the three outcomes of the per-assertion table above +- Field reproduction 2026-08-04 (linkly `test_golden.py`, 8 mutations mapped to their owning tests): renaming `workflow Checkout` to `CheckoutX` reddened `test_source_compiles_to_the_committed_ir` while `test_node_ids_and_order_are_stable` stayed green — that assertion pins only the first node id and the trailing capability ids, so reddening it required a capability-order swap instead. The file-level red would have credited both From 1ccb0837565971fd9ee02be9fbe9ca35f8e10a96 Mon Sep 17 00:00:00 2001 From: dch0202 Date: Tue, 4 Aug 2026 15:41:09 +0900 Subject: [PATCH 2/2] knowledge: fold 2 mid-flush insights into the same batch (stream-fed prompt tests, executable fixtures) --- .dev-loop/INGEST_REPORT.md | 95 ++++++++++++++++++- log.md | 1 + .../filesystems/permissions-and-exec-bits.md | 8 +- wiki/platforms/index.md | 2 +- wiki/testing/async/async-testing.md | 6 ++ wiki/testing/data/test-data-and-isolation.md | 10 +- wiki/testing/index.md | 4 +- 7 files changed, 117 insertions(+), 9 deletions(-) diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 9174533..fa4b82a 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,7 +1,9 @@ -# Knowledge flush — 7 insight(s) +# Knowledge flush — 9 insight(s) Drained the whole pending queue (`~/.dev-loop/queue/`, 7 rows across 5 session -files). Result: **5 new pages, 2 amendments, 1 new category**. Two candidate +files), then a **second pass** for 2 further candidates that arrived mid-flush +(15:29 / 15:30) — folded into this same branch so the flush stays one PR. +Result: **5 new pages, 5 amendments, 1 new category**. Two candidate claims were **disproved during verification** and the resulting directives were rewritten before ingest — details under *Verified best-practice* (B1). @@ -14,6 +16,8 @@ rewritten before ingest — details under *Verified best-practice* (B1). | D1 | Monitor whose grep completion predicate reports done immediately | NEW `testing/quality/completion-predicates` | verified | | B1 | Picking a crawl source for Korean public-institution notices | NEW `backend/common/integrations/robots-txt-and-source-selection` (**claim corrected**) | verified | | B2 | Client throttle in place, still rate-limited on the first call | NEW `backend/common/reliability/client-side-rate-limiting` | field-tested | +| A1 | Feeding a readline-style stream consumer from a test double | MERGE → `testing/async/async-testing` | verified | +| A2 | A test fixture that must carry the executable bit, under EDR | MERGE → `testing/data/test-data-and-isolation` + `platforms/filesystems/permissions-and-exec-bits` | verified (EDR rationale: field-tested) | --- @@ -252,3 +256,90 @@ worth a maintainer's eye is B2's `field-tested` confidence — the mechanism and the timestamped production evidence are solid, but the vendor's published quota could not be confirmed, so the page deliberately carries no numbers. Reject that page alone if the bar for a `reliability` entry is a cited provider limit. + +--- + +# Addendum — second pass (A1, A2) + +Two candidates were written to the queue at 15:29 and 15:30, after the first +drain. They are folded into this same branch so the flush remains one PR. + +## Verified best-practice — second pass + +**A1 — feeding a line-oriented stream consumer from a test double.** The claim was +that batching several answers into one `write` silently loses all but the first. +**Reproduced live, Node v25.8.1**, `readline.createInterface` over a `PassThrough` +with three sequential `question` calls: + +``` +batched input.write('one\ntwo\nthree\n') → ["one"], then hangs +one-per-tick write(line+'\n') + await setImmediate ×3 → ["one","two","three"] +``` + +Mechanism as the candidate described it: a readable hands the consumer one +concatenated chunk, and readline walks every newline in that chunk synchronously, +emitting `'line'` for input no `question` callback is waiting on — those lines are +gone. Documented API surface: . The +"one shared interface per interaction" half is the same root cause seen from the +other side (a second interface competes for the same buffered data). +→ **verified** by reproduction. + +**A2 — an executable test fixture: mode at creation, and where to put it.** +**Measured live, Node v25.8.1 / macOS / umask 022**: + +``` +writeFileSync(p, body, { mode: 0o755 }) → stat 755 +writeFileSync(p, body2, { mode: 0o644 }) → stat 755 (mode applies at creation only) +``` + +That second line is a caveat the candidate did not state and the page now carries: +rewriting an existing fixture keeps the original mode, so a fixture that changes +mode between tests must be deleted and recreated. API: + (`writeFileSync` `mode`, `mkdtemp`). +The **EDR rationale is not vendor-documented** — "a +x file created under a system +temp dir is a dropper heuristic" is operational practice, and the page labels it +as such rather than citing a detection vendor. What is verifiable and stated: the +fixture needs the bit, not the location, and a repo-local gitignored build-output +path additionally leaves `git status` clean when a run crashes (field evidence: +25 fixtures moved, suite green 59/59, build dir empty afterwards). +→ **verified** for the mechanics, **field-tested** for the EDR rationale. + +## Existing-layer check — second pass + +Read before writing: `wiki/testing/async/async-testing.md` (72 body lines), +`wiki/testing/data/test-data-and-isolation.md` (69), +`wiki/platforms/filesystems/permissions-and-exec-bits.md` (68). + +Both candidates **merged, no new pages**: + +- **A1 → `testing/async/async-testing`.** Its "Async shape" table is exactly the + routing structure this case needs, and its root rule ("the test must not finish + before the work does") is the same principle. Added one table row, two edge-case + rows (hang after the first record; a consumer rebuilt per prompt), one + `Instead of` row, the readline source and the reproduction. A separate page + would have duplicated the table. +- **A2 → two pages, split by ownership.** The *fixture* half (where to create it, + per-test directory, teardown) extends `test-data-and-isolation`'s existing + "Filesystem / temp files" row — one new Do row plus two edge cases. The + *mode mechanics* half (set at creation, umask, creation-only semantics, EDR + location) extends `permissions-and-exec-bits`, whose Do table already owns + exec-bit rules — two new Do rows. Cross-linked via `related` in both + directions rather than repeating the content. + +Conflicts: none. Neither amendment contradicts an existing directive; both extend +tables their pages already own. + +Index lines updated for all three pages (`wiki/testing/index.md` ×2, +`wiki/platforms/index.md` ×1) so the new use cases are routable. Re-ran the +invariant checks over the full wiki after these edits: every `related:` id and +inline `[page-id]` resolves, every index link resolves, all touched pages remain +under 120 body lines. + +## Routing decision — second pass + +| Insight | Domain / category | Target | Note | +|---------|-------------------|--------|------| +| A1 | `testing` / `async` | **MERGE** `async/async-testing.md` | Stream-fed prompt tests are async-shape routing, which that page's table already owns; no new category | +| A2 | `testing` / `data` **and** `platforms` / `filesystems` | **MERGE** `data/test-data-and-isolation.md` + `filesystems/permissions-and-exec-bits.md` | Split by artifact: fixture lifecycle belongs to the testing page, file-mode mechanics to the platforms page — matching the domain rule that each page owns the artifact it changes | + +No new categories in this pass. diff --git a/log.md b/log.md index 703ab87..4464470 100644 --- a/log.md +++ b/log.md @@ -39,3 +39,4 @@ Append-only. Format: `## [YYYY-MM-DD] setImmediate(r))` — and share one reader instance across the whole interaction rather than constructing one per prompt | 3. **Contain leaked work.** A promise or timer that outlives its test corrupts the next test's state and assertions. In each test's teardown, unsubscribe @@ -59,6 +60,8 @@ un-awaited promises; or an async test intermittently interferes with the next te | A condition wait (`waitFor`) is needed while fake timers are active | Advance the fake clock explicitly before/while awaiting the condition, or use the wait utility's fake-timer-aware mode; otherwise the poll's own timers never fire | | Runner reports an unhandled rejection after the suite passes | A promise was created without `await`/`return` — find it and await it; do not silence the warning | | Assertions run inside a `.then`/callback the test never awaits | Add `expect.assertions(n)` / `expect.hasAssertions()` so the test fails when the callback is skipped, then restructure to await-then-assert | +| A stream-fed test hangs after consuming the first record, with the later records never delivered | The records arrived in one chunk: a readable concatenates buffered writes, and a line-oriented consumer walks every delimiter in that chunk synchronously, discarding the lines no reader is waiting for. Write one record per turn (table row above) and re-run | +| The consumer is rebuilt per prompt (a new interface inside a retry loop) | Construct it once per interaction and reuse it — a second instance attached to the same stream competes for the same buffered data, so records land in whichever instance reads first | ## Instead of @@ -68,6 +71,7 @@ un-awaited promises; or an async test intermittently interferes with the next te | Assert inside `.then()` without returning the promise | `await` the promise, then assert on the main path | The test finishes before the callback runs; it passes with zero assertions executed | | Catch an expected rejection with `try/catch` and assert loosely in the catch | `await expect(...).rejects.toThrow(...)`-style API | The catch-block assert never runs when the code succeeds, and the test still passes | | Enable fake timers globally for the whole suite | Enable per test/suite that advances them; restore real timers in teardown | Frozen time deadlocks unrelated tests' timeouts, polling waits, and library internals | +| Preload every scripted answer into the test's input stream in one write | Write one line per macrotask turn, awaiting `setImmediate` between them | The stream hands the consumer one concatenated chunk, and lines with no waiting reader are emitted and dropped — the test hangs on the second prompt | ## Sources @@ -76,3 +80,5 @@ un-awaited promises; or an async test intermittently interferes with the next te - https://jestjs.io/docs/expect — `expect.assertions` / `expect.hasAssertions` verify callback assertions ran - https://testing-library.com/docs/dom-testing-library/api-async/ — `waitFor`/`findBy`: polling on a condition with interval and bounded timeout - https://martinfowler.com/articles/nonDeterminism.html — poll/callback on the completion condition instead of bare sleeps +- https://nodejs.org/api/readline.html — `readline.createInterface` consumes an input stream and delivers a line per `question` callback, emitting `'line'` for input no callback is waiting on +- Field reproduction 2026-08-04 (Node v25.8.1, `readline` over a `PassThrough`, three sequential `question` calls): a single `input.write('one\ntwo\nthree\n')` yielded `["one"]` and then hung; writing the same three lines one per `setImmediate` turn yielded `["one","two","three"]` and completed diff --git a/wiki/testing/data/test-data-and-isolation.md b/wiki/testing/data/test-data-and-isolation.md index 994a602..4694c21 100644 --- a/wiki/testing/data/test-data-and-isolation.md +++ b/wiki/testing/data/test-data-and-isolation.md @@ -7,8 +7,9 @@ confidence: verified sources: - https://martinfowler.com/articles/nonDeterminism.html - https://abseil.io/resources/swe-book/html/ch12.html -last_verified: 2026-07-10 -related: [testing-flaky-diagnosing-flaky-tests, testing-strategy-test-level-choice] + - https://nodejs.org/api/fs.html +last_verified: 2026-08-04 +related: [testing-flaky-diagnosing-flaky-tests, testing-strategy-test-level-choice, platforms-filesystems-permissions-and-exec-bits] --- # Owning Test Data and Isolating Test State @@ -39,6 +40,7 @@ state-leak symptom. | Time-dependent logic (expiry, scheduling, "created today") | Inject a clock/time source and freeze it in the test; assert against the frozen instant | | Unique-constrained values (emails, usernames, external ids) | Generate per test (counter, UUID suffix) inside the factory — hardcoded constants collide across tests and across parallel runs | | Filesystem / temp files | Create a fresh per-test temp directory and remove it in teardown | +| A fixture file must carry the executable bit (permission checks, PATH/binary-resolution code) | Create it with the mode set at creation time (`writeFileSync(p, body, { mode: 0o755 })`, `open` with a mode) inside a per-test directory under an already-gitignored build-output path of the repo, and remove it in teardown | | Global config / environment variables / singletons mutated by a test | Set in setup, restore in teardown that runs on failure too (`finally`/fixture teardown) | 4. Keep fixture data **minimal**: create only the entities the behavior under @@ -53,6 +55,8 @@ state-leak symptom. | Suite is too slow because every test builds a deep object graph | Move the invariant graph into a per-suite setup that tests never mutate; keep mutated entities per-test | | Failure appears only in the full suite, never alone | Run the suite in random order to expose the order dependency, then bisect to the polluting test; fix the polluter's ownership, not the victim ([testing-flaky-diagnosing-flaky-tests]) | | Test needs "now"-relative data but the code reads the system clock directly | Refactor the code to accept an injected clock; that seam is the fix — assertions with tolerance windows around real time stay flaky | +| The executable fixture is rewritten between tests | Delete and recreate it: a write to an existing path keeps the original mode, so a second write with a different mode leaves the first one in place (measured on Node v25.8.1) | +| The machine runs endpoint security (EDR) that flags an executable created under the system temp directory | Keep the fixture inside the repo's gitignored build-output tree and set the bit at creation; that path is what a "+x file dropped in a temp dir" heuristic looks for, and the fixture only needs the bit, not the location ([platforms-filesystems-permissions-and-exec-bits]) | ## Instead of @@ -67,3 +71,5 @@ state-leak symptom. - https://martinfowler.com/articles/nonDeterminism.html — isolation between tests, wrapping the system clock, callbacks/polling over bare sleeps - https://abseil.io/resources/swe-book/html/ch12.html — tests should contain the values they depend on; clarity over shared magic setup +- https://nodejs.org/api/fs.html — `fs.writeFileSync(file, data, { mode })` and `fs.mkdtemp(prefix)` create the file/directory with the mode supplied at creation +- Field measurement 2026-08-04 (Node v25.8.1, macOS, umask 022): `writeFileSync(p, body, { mode: 0o755 })` produced mode `755`; a second `writeFileSync(p, body2, { mode: 0o644 })` on the same path left the file at `755` — the mode argument applies at creation only. Field context: a 25-fixture suite moved off `tmpdir()` + `chmod +x` onto a gitignored build-output directory ran green (59/59) with the build directory empty and `git status --short` clean afterwards diff --git a/wiki/testing/index.md b/wiki/testing/index.md index 9eb23f3..d9685b9 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -34,7 +34,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| -| [test-data-and-isolation](data/test-data-and-isolation.md) | Tests need fixture data and you are choosing how to create it; tests pass alone but fail together (or vice versa); DB cleanup, shared fixtures, time-dependent logic, or unique-value collisions | +| [test-data-and-isolation](data/test-data-and-isolation.md) | Tests need fixture data and you are choosing how to create it; tests pass alone but fail together (or vice versa); DB cleanup, shared fixtures, time-dependent logic, or unique-value collisions; a fixture file must carry the executable bit, or must live somewhere endpoint security and `git status` both tolerate | ## mocking @@ -52,7 +52,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| -| [async-testing](async/async-testing.md) | Testing async code — promises, timers, retries, debounce, event-driven flows; the runner warns about assertions after completion or un-awaited promises; an async test intermittently interferes with the next test; deciding between fake timers and condition waits | +| [async-testing](async/async-testing.md) | Testing async code — promises, timers, retries, debounce, event-driven flows; the runner warns about assertions after completion or un-awaited promises; an async test intermittently interferes with the next test; deciding between fake timers and condition waits; driving a stream-consuming reader (readline prompts, a line-delimited protocol) from an in-memory test double, and a stream-fed test that hangs after the first record | ## e2e