From ae333dad58b7f72879fc90dd7a93eaa7f14e67d6 Mon Sep 17 00:00:00 2001 From: dch0202 Date: Thu, 6 Aug 2026 13:04:21 +0900 Subject: [PATCH] knowledge: ingest 6 insights into existing pages, retire 5 duplicates --- .dev-loop/INGEST_REPORT.md | 149 +++++++++++++----- log.md | 1 + .../control-signals-vs-primary-artifacts.md | 5 +- .../worktree-isolated-workers.md | 4 +- wiki/infrastructure/index.md | 4 +- .../filesystems/permissions-and-exec-bits.md | 2 +- wiki/testing/index.md | 4 +- .../quality/guard-shape-vs-consequence.md | 8 +- .../testing/quality/tests-that-cannot-fail.md | 6 +- 9 files changed, 130 insertions(+), 53 deletions(-) diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 55ccfd1..331397b 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,53 +1,116 @@ -# Knowledge consolidation — 15 open PRs (#17–#40) → one reconciled state +# Knowledge flush — 11 insight(s) -The 15 open `knowledge/*` PRs (created 2026-08-04 → 2026-08-05, before the -harvest processed-store dedupe fix in #41) contained 123 file-versions of ~75 -unique pages, with the same insight landing at up to 3 different paths across -up to 8 PRs. Per-PR review would re-import those duplicates, so — as with the -#6–#13 consolidation — this branch carries the reconciled end-state and the 15 -PRs are closed in its favor. +Drained 9 queue files (11 candidate rows) from `~/.dev-loop/queue/`. 5 candidates +were exact duplicates of knowledge already ingested by the #42–#46 consolidation +and were retired without edits; 6 were merged into existing pages (merge-before-create +— no new pages, no new categories). Open-PR dedup check: `gh pr list --label +dev-loop:knowledge --state open` returned zero open PRs, so nothing here overlaps +an in-flight review. ## Verified best-practice -Every adopted page's sources were carried from its originating PR's flush, where -they were live-verified at flush time; no new URLs were introduced during -consolidation (checked mechanically: every `http(s)` URL in every merged page -appears in a source PR's diff; every added body line in amended pages traces to -a source PR hunk — orphan-line verification). Confidence fields were kept as the -originating flushes set them, except client-side-rate-limiting where the union -of provider-doc citations (Okta, Auth0, GitHub, OpenAI, RFC 6585) supports -`verified` for the load-bearing claims. One subagent's fabricated content (12 -files matching neither main nor any PR, with invented source URLs) was detected -by the same verification and replaced with true PR content. +**1. Bats assertions as `[[ ]]` mid-test are decoration on bash 3.2 → `verified`.** +Claim: under bats on macOS system bash (3.2.57), a false `[[ ]]` that is not the +test's last command does not fail the test; `[ ]` and `grep -qF` pipelines fail at +any position. Verified this session by fresh local reproduction (Bats 1.14.0, GNU +bash 3.2.57, arm64): 4-test probe file gave `ok` for mid-test false `[[ ]]`, +`not ok` for mid-test `[ ]`, mid-test grep, and last-line `[[ ]]`. Mechanism +isolated outside bats: `bash -ec '[[ … ]]; echo survived'` exits 0 while the `[ ]` +form aborts — a pre-4.0 errexit semantic, not a bats defect. Sources: +https://tiswww.case.edu/php/chet/bash/COMPAT (bash-4.0 changed `set -e` to exit on +compound-command failure), https://bats-core.readthedocs.io/en/stable/gotchas.html +and https://www.shellcheck.net/wiki/SC2314 (the documented same-shape gotcha for +negated `!` commands: they "can never fail when used in the middle of a test"). + +**2. Widened scan surface turning old tests red → triage as first true positive → `field-tested`.** +Claim: when a leak/masking detector gains a previously unscanned output channel and +an existing test reddens, diff what the new surface saw against what the fixture +declares, and fix the fixture's smuggled data rather than the detector. Evidence is +the session reproduction (linkly #43: `result.bindings` widening reddened +`test_when_guard_removed_diverges`; fixture carried an undeclared `password` key; +narrowing to declared fields → 1218 green). No independent external source claims +this exact triage order, so it stays field-tested; it is consistent with the page's +existing sourced principle (change-detector tests / "the guard reddens on a genuine +S-and-C artifact means the guard is working"). + +**3. Worker usage-limit pause looks alive → check pane tail for the limit marker → `field-tested`.** +Claim: simultaneous quiet workers with green liveness are a usage-limit pause; +find the `You've hit your session limit · resets HH:MM` marker, then after reset +send a resume prompt ordering state-recheck → remaining DoD → completion signal. +Evidence: 2026-08-06 run, three workers paused on one reset with identical markers; +the structured resume prompt recovered all three at their exact interruption point. +Vendor docs do not document the marker string, so no external citation is possible — +kept field-tested with the context described on the page. + +**4. Dispatch after `worker_done` needs a substrate-idle wait; a failed dispatch consumes the task → `field-tested`.** +Evidence: reproduced twice in the 2026-08-06 run (immediate dispatch → +`runtime_unavailable` + task consumed; dispatch after `orca terminal wait --for +tui-idle` succeeded first try). Orca is an internal tool; no external source exists. + +**5. Guardrail `worktree_escape` can escalate on read-only cross-worktree access → `field-tested`.** +Evidence: two read-only commands (`awk`/`grep` over an upstream FINDINGS file, +`git status`) each raised `ask` and stopped the watch with exit 5; both approved +after review. This **conflicts** with the existing page's 1.0.0 reproduction where +reads passed — handled as a condition-dependent (rule-version) edge case, not an +overwrite (see below). + +**6. `sh "$SCRIPT"` stub seam under EDR — second reproduction → enriches existing `field-tested` row.** +Directive already on the page from the #42–#43 reconciliation; this flush adds the +independent second reproduction (8 stall-handler stubs injected without any +`chmod`, bats suite 331/331 green under SentinelOne). + +**Dropped as exact duplicates (no edit, retired from queue):** gate quoting-form +parsing (already `command-text-inspected-before-execution` step 8 + Instead-of row ++ field context citing the same bats tests 12–13), stderr-warnings-with-exit-0 +capture incl. redirection order (already the whole of +`tool-diagnostics-without-a-failing-exit-code`), Homebrew clang `-isysroot +$(xcrun --show-sdk-path)` (already the whole of `compiler-sysroot-on-macos`, incl. +the same 69-failure repro), temp-artifact prefix counting + AST-rule enforcement + +RED-first guard (already `artifact-leakage-from-a-suite` steps 1/4/5), and +vacuously-green pre-implementation usage-error test proven by guard mutation +(already a `checks-that-cannot-pass` edge row describing the identical +unknown-subcommand/exit-1 case). ## Existing-layer check -- Merged-main near-dup scan before consolidation: pairwise Jaccard over - title + "When this applies" across all 141 merged pages → **0 flagged pairs**; - previously merged content carries no duplication. -- Cross-PR dedup during consolidation: 10 duplicate clusters collapsed to one - canonical page each (rate limiting 8→1, call-site enumeration 7→folded into - the canonical merged in #20, stderr/exit-0 diagnostics 4→1, sysroot 2→1, - env-off-switch 2→1, completion predicates 2→1, robots.txt 2→1, - harness-mediated results 2→1, leaked artifacts 2→1, orchestration category - naming unified). Three near-pairs kept distinct after trigger comparison, - with mutual `related:` links (differential setup vs interpretation; expansion - semantics vs off-switch design; import-time tactics vs level choice). -- 24 existing pages received union-merged amendments; additions already present - in main (from #16/#20) were skipped, and all non-canonical `related:` ids - were remapped to canonical page ids (post-merge broken-link scan: 0). +Read before deciding: root `INDEX.md`; domain indexes for testing, platforms, +infrastructure; and the nine candidate-overlapping pages +(`command-text-inspected-before-execution`, +`tool-diagnostics-without-a-failing-exit-code`, `compiler-sysroot-on-macos`, +`artifact-leakage-from-a-suite`, `worktree-isolated-workers`, +`tests-that-cannot-fail`, `checks-that-cannot-pass`, `guard-shape-vs-consequence`, +`control-signals-vs-primary-artifacts`, `permissions-and-exec-bits`, +`destructive-operations-on-shared-daemons`). + +- **Merged, not created:** all 6 surviving insights landed as edge-case/table rows + and evidence on existing pages. No new page, no new category. +- **Conflict flagged and resolved as condition-dependent:** + `worktree-isolated-workers` states reads pass the guardrail (1.0.0 repro); the + new observation shows a rule version escalating on reads. Added as an edge row + ("guardrail rules differ by version — probe one read before fanning out") and + noted in the log entry; the Do-this table was not overwritten. +- **Duplicates:** the 5 dropped candidates matched existing pages + trigger-for-trigger and directive-for-directive (the #42–#46 reconciliation had + already ingested earlier harvests of the same sessions' insights). +- **Related-links:** no new cross-links needed — every edited page already links + the pages the new rows reference (e.g. `tests-that-cannot-fail` ↔ + `checks-that-cannot-pass`, `control-signals` ↔ `worktree-isolated-workers` via + the agent-orchestration index). ## Routing decision -- New categories: `infrastructure/agent-orchestration` (5 pages; unified the - competing `orchestration`/`agent-orchestration` names), `databases/data-survey` - (1), `qa/deliverables` (1). All other pages route into existing categories. -- Canonical-path decisions: rate limiting → `backend/common/reliability/` - (sits beside timeouts-and-retries; 6 of 8 variants chose it); stderr - diagnostics → `platforms/processes/` (concern spans beyond shells); leaked - artifacts → `testing/data/artifact-leakage-from-a-suite`; call-site - enumeration → the existing `backend/common/change-impact/` page. -- All 38 new pages listed in their domain indexes (nearest-index rule; backend - routes via its python sub-index for bytecode-cache-staleness); INDEX.md domain - summaries updated for infrastructure/qa/databases. Full-wiki lint: frontmatter, - ids, related-links, index coverage, size, qualifiers, staleness → 0 findings. +| Insight | Target (existing page) | Why this page | +|---------|------------------------|---------------| +| bats/bash-3.2 mid-test `[[ ]]` | `testing/quality/tests-that-cannot-fail` — new never-fails-pattern row + sources | The page owns "assertions that cannot detect a defect"; this is a shell-level instance of that exact class | +| widened-scan-surface red | `testing/quality/guard-shape-vs-consequence` — When-this-applies clause + edge row + field evidence | The page owns guard-red triage; existing edge row already covered "guard reddens on genuine S-and-C"; this adds the widened-surface trigger and fixture-diff triage | +| usage-limit worker stall | `infrastructure/agent-orchestration/control-signals-vs-primary-artifacts` — edge row + field evidence | The page owns done/alive/stalled/dead verdicts; this is a new stalled-state cause with recovery protocol | +| dispatch-after-done timing | same page — edge row | "Done signal ≠ substrate release" is precisely the page's signal-vs-artifact distinction | +| read-only guardrail escalation | `infrastructure/agent-orchestration/worktree-isolated-workers` — edge row (condition-dependent conflict) | The page owns the guardrail's read/write asymmetry; the conflicting observation must sit next to the claim it qualifies | +| EDR stub second repro | `platforms/filesystems/permissions-and-exec-bits` — evidence sentence | Directive already lives there; only evidence strengthened | + +Domain hints from the queue were respected except where a page already owned the +case: the "testing"-hinted EDR-stub insight routes to platforms (the page that owns +exec-bit/EDR invocation style), and the "platforms"-hinted orchestration insights +route to infrastructure/agent-orchestration (dedicated category), consistent with +prior flushes. Index "load when" lines updated for the four pages whose routing +surface grew. diff --git a/log.md b/log.md index c930fc2..6b31cb1 100644 --- a/log.md +++ b/log.md @@ -43,3 +43,4 @@ Append-only. Format: `## [YYYY-MM-DD] -backup/…`) | The guardrail does not fire — the match requires a path separator after the main root — but the write is still outside the worktree; keep it out of the brief | +| A read-only command (`ls`, `grep`, `awk`, `git status`) naming the main checkout's or another worktree's absolute path raises an `ask` escalation anyway, halting the watch | Guardrail rules differ by version: a conservative rule treats any cross-worktree path reference in command text as a potential write, reads included. Extend the step-6 dry run with one read probe to learn which behavior you have; when reads escalate, budget the round-trip into the phase (read the escalation record → approve or deny → clear the escalation state → restart the watch) and state in the worker's first briefing which reads are pre-approved and that writes and system-temp use stay forbidden — this cuts repeat escalations for the same access | ## Instead of @@ -69,4 +70,5 @@ wait loop keeps escalating with no error from the task itself. - https://git-scm.com/docs/git-worktree — linked worktrees are separate checkouts sharing one repository; each has its own working directory - Field reproduction 2026-08-05 (groundwork guardrails 1.0.0 `hooks/bash-guard.sh`, `worktree_escape` rule, macOS): from a linked worktree, `cp ./a /b` and `echo z > /f` were both stopped; `cat /f`, `ls /.orchestration`, and `grep -n x /f` all passed. The rule matches an absolute main-root mention together with a write verb (`rm|mv|cp|tee|mkdir|touch|install|dd`) or a redirect to an absolute path +- Field evidence 2026-08-06 (dev-loop orchestrate, Wave 2 worker consuming an upstream worktree's FINDINGS file): a read-only `awk`/`grep` verification and a `git status` check each raised `worktree_escape` as `ask` and stopped the coordinator's watch with exit 5; both were confirmed read-only and approved. This rule version fired on reads, unlike the 1.0.0 reproduction above where bare `cat`/`ls`/`grep` passed — the read/write asymmetry in the Do-this table is version-dependent, so probe before fanning out - Field context: a parallel run stalled at the same phase for two workers whose brief's `` named a main-checkout absolute path; the coordinator's wait loop returned its escalation status repeatedly. Rewriting the contract to worktree-relative paths let the remaining workers record their plans locally diff --git a/wiki/infrastructure/index.md b/wiki/infrastructure/index.md index 36a0771..7483121 100644 --- a/wiki/infrastructure/index.md +++ b/wiki/infrastructure/index.md @@ -12,11 +12,11 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| -| [control-signals-vs-primary-artifacts](agent-orchestration/control-signals-vs-primary-artifacts.md) | An orchestrator is about to restart, discard, merge, or keep waiting on a worker based on a status file, a watcher's exit code, or a heartbeat; a monitor reports a worker dead while it is committing; a worker's status write produced no output and you must decide whether it landed; distinguishing alive-and-progressing from stalled from dead | +| [control-signals-vs-primary-artifacts](agent-orchestration/control-signals-vs-primary-artifacts.md) | An orchestrator is about to restart, discard, merge, or keep waiting on a worker based on a status file, a watcher's exit code, or a heartbeat; a monitor reports a worker dead while it is committing; a worker's status write produced no output and you must decide whether it landed; distinguishing alive-and-progressing from stalled from dead; several workers went quiet at once while every liveness check passes (usage-limit pause); a dispatch issued right after a worker's done signal fails runtime-unavailable | | [shared-run-state](agent-orchestration/shared-run-state.md) | Several agent/worker sessions coordinate through files in one repository (status directory, briefs, escalations, claim files); choosing the path layout for that state; starting an orchestration in a repo that may already have one running; a watcher woke on a task id it did not create; the default branch moved during a run | | [pane-delivery-confirmation](agent-orchestration/pane-delivery-confirmation.md) | An orchestrator drives another program through a terminal multiplexer (`tmux send-keys` + `capture-pane`) and must decide whether the input was consumed, retry, or escalate; a pane diff is being used as delivery evidence; the target echoes but never runs the input | | [session-completion-gates](agent-orchestration/session-completion-gates.md) | Writing a Stop/completion hook that blocks a worker session from ending while its phase is non-terminal; the gate fires on a worker that followed its own prompt; deciding the terminal phase set, the unknown-phase default, and how the gate bounds its own repetition | -| [worktree-isolated-workers](agent-orchestration/worktree-isolated-workers.md) | Authoring the brief/output contract for parallel workers each confined to its own git worktree; workers stall at the same phase with no task-level error; deciding where shared or produced artifacts live and which direction (read vs write) a worktree guardrail stops | +| [worktree-isolated-workers](agent-orchestration/worktree-isolated-workers.md) | Authoring the brief/output contract for parallel workers each confined to its own git worktree; workers stall at the same phase with no task-level error; deciding where shared or produced artifacts live and which direction (read vs write) a worktree guardrail stops; a guardrail escalates on read-only access to another worktree | ## ci-cd diff --git a/wiki/platforms/filesystems/permissions-and-exec-bits.md b/wiki/platforms/filesystems/permissions-and-exec-bits.md index 5774423..879e833 100644 --- a/wiki/platforms/filesystems/permissions-and-exec-bits.md +++ b/wiki/platforms/filesystems/permissions-and-exec-bits.md @@ -71,4 +71,4 @@ channels that preserve modes. - https://docs.docker.com/engine/containers/run/ — container default user is root (uid 0); `--user`/`-u` overrides with `uid:gid` - https://docs.docker.com/engine/storage/bind-mounts/ — bind-mount mechanics (host uid/gid visibility rows are field practice, not stated on this page) - https://nodejs.org/api/fs.html — `fs.writeFileSync(file, data, { mode })` sets the mode when the file is created -- Field measurement 2026-08-04 (Node v25.8.1, macOS, umask 022): `writeFileSync` with `mode: 0o755` produced mode `755`; a second write to the same path with `mode: 0o644` left it at `755`, confirming the creation-only semantics. The EDR row is operational practice, not vendor-documented — a 25-fixture suite was moved off `tmpdir()` + `chmod +x` to a gitignored build-output directory and ran green (59/59) with no alert +- Field measurement 2026-08-04 (Node v25.8.1, macOS, umask 022): `writeFileSync` with `mode: 0o755` produced mode `755`; a second write to the same path with `mode: 0o644` left it at `755`, confirming the creation-only semantics. The EDR row is operational practice, not vendor-documented — a 25-fixture suite was moved off `tmpdir()` + `chmod +x` to a gitignored build-output directory and ran green (59/59) with no alert. Second reproduction 2026-08-06 (bats, macOS + SentinelOne): eight stall-handler stub scripts injected through an env-var seam as plain non-executable files into a caller invoking `sh "$SCRIPT"` — full suite green (331/331) with no `chmod` anywhere in the tests diff --git a/wiki/testing/index.md b/wiki/testing/index.md index 5f3ed78..7c4c010 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -24,12 +24,12 @@ Match your situation to a "load when" line; load only matching pages. |------|-----------| | [completion-predicates](quality/completion-predicates.md) | Writing the "everything is done" condition a monitor, wait loop, or polling script uses to decide background work has finished; a monitor declared completion far sooner than the work could have finished; matching a status marker that contains regex metacharacters, or passing that pattern through wrapper/ssh/send-keys quoting layers; deciding completion by counting rather than by absence | | [differential-run-agreement](quality/differential-run-agreement.md) | Two implementations of one spec were run on the same input and the harness reported agreement (EQUIVALENT / no diff / N-of-N checks pass) and you are about to cite it; the two sides model different amounts of state (one stubs out a repository, cache, clock, or session); choosing the input that forces an asymmetric dimension to decide the outcome | -| [guard-shape-vs-consequence](quality/guard-shape-vs-consequence.md) | A repo-wide guard asserting that no shipped artifact (example, config, migration, fixture) has a structural shape has gone red on a legitimate new artifact; authoring such a scanning guard; deciding between exempting an artifact, deleting the guard, and sharpening it; an existing guard has accumulated an exemption/allow list | +| [guard-shape-vs-consequence](quality/guard-shape-vs-consequence.md) | A repo-wide guard asserting that no shipped artifact (example, config, migration, fixture) has a structural shape has gone red on a legitimate new artifact; authoring such a scanning guard; deciding between exempting an artifact, deleting the guard, and sharpening it; an existing guard has accumulated an exemption/allow list; a widened scan surface (a detector now scans a new output channel) turned an existing test red | | [injected-clock-duration-assertions](quality/injected-clock-duration-assertions.md) | Asserting an elapsed duration between two readings of an injected/fake float clock (rate-limit interval, backoff, debounce, TTL); choosing that fake clock's start value; a single duration test fails on correct code by a margin in the far decimal places; choosing a comparison tolerance, or deciding between float seconds and integer nanoseconds | | [write-path-assertions](quality/write-path-assertions.md) | Writing an HTTP-level test for an endpoint that persists something (form submit, create/update, onboarding step) and choosing what to assert beyond the status code; such a test is green while the records are empty or defaulted; sending repeated form fields from a client (httpx/TestClient) and deciding the `data=` shape | | [minimum-case-set](quality/minimum-case-set.md) | Writing tests for a function/endpoint/change and choosing which cases to cover; reviewing whether coverage suffices; picking boundary values by input type; adding a regression test for a bug fix | | [behavior-not-implementation](quality/behavior-not-implementation.md) | Deciding what a test should assert; a behavior-preserving refactor broke tests; tempted to expose privates for testing; deciding whether a snapshot test is appropriate | -| [tests-that-cannot-fail](quality/tests-that-cannot-fail.md) | Reviewing tests that always pass; a bug shipped through an area the suite reported as covered; auditing a suspiciously green suite; judging whether an assertion, error-path test, or mock-based test can actually detect a defect | +| [tests-that-cannot-fail](quality/tests-that-cannot-fail.md) | Reviewing tests that always pass; a bug shipped through an area the suite reported as covered; auditing a suspiciously green suite; judging whether an assertion, error-path test, or mock-based test can actually detect a defect; a shell-test (bats) assertion passes mid-test on code it should fail | | [checks-that-cannot-pass](quality/checks-that-cannot-pass.md) | Authoring a check whose target does not exist yet (grep/regex gate on an unwritten file or doc section, lint/scan rule, schema assertion on an unbuilt endpoint, a plan's verification command) and it has only ever been observed failing; reviewing a plan's gates before adopting them; separating "target missing" from "content missing" in a gate's exit status | | [spec-artifact-checks](quality/spec-artifact-checks.md) | Writing or reviewing an automated check that a mapping table covers every rule/field/enum case, or that ids resolve across documents; deciding whether a green check earned "verified" or only "present"; designing one negative control per check in a multi-check harness; parsing Markdown table rows programmatically in a doc-as-spec repo | | [schema-additions-under-a-golden-gate](quality/schema-additions-under-a-golden-gate.md) | Adding a node kind, variant, discriminator value, or field to a document format (IR, JSON Schema, spec artifact) whose only automated gate builds its negatives by mutating one committed golden example; the gate or the whole suite comes back green right after a schema change; deciding which negative each new schema keyword needs, and whether a green suite that never loads the schema is evidence at all | diff --git a/wiki/testing/quality/guard-shape-vs-consequence.md b/wiki/testing/quality/guard-shape-vs-consequence.md index e3875e8..8a32c44 100644 --- a/wiki/testing/quality/guard-shape-vs-consequence.md +++ b/wiki/testing/quality/guard-shape-vs-consequence.md @@ -7,7 +7,7 @@ confidence: field-tested sources: - https://testing.googleblog.com/2015/01/testing-on-toilet-change-detector-tests.html - https://pitest.org/ -last_verified: 2026-08-04 +last_verified: 2026-08-06 related: [testing-quality-tests-that-cannot-fail, testing-quality-behavior-not-implementation, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, qa-process-regression-scope] --- @@ -19,7 +19,9 @@ A guard test scans every shipped artifact of a kind — example files, configs, migrations, fixtures, schema docs — and asserts that none of them has a structural shape S. A newly added, legitimate artifact now has S, so the guard is red and you are deciding what to do about it. Also applies when authoring -such a guard, before the first legitimate collision happens. +such a guard, before the first legitimate collision happens — and when you have +just widened a guard's scan surface (a leak/masking detector now scans a +previously unscanned output channel) and a previously green test reddens. Reviewing a guard that has never been red → [testing-quality-tests-that-cannot-fail]. @@ -66,6 +68,7 @@ Reviewing a guard that has never been red → [testing-quality-tests-that-cannot | Reusing the production derivation means a bug in that derivation silently greens the guard | Accept the coupling — it is what keeps the guard's meaning in sync — and keep the step-4 fixture as the independent control that would catch the greening | | Computing C over every artifact is expensive | Keep the shape check as a cheap prefilter and compute C only for the artifacts S matched; the assertion stays "S and C" | | The guard reddens on an artifact that has S and genuinely has C | This is the guard working — fix the artifact, not the guard | +| The guard's scan surface was just widened and an existing test reddens | Triage it as the widened guard's first true positive before touching guard or production code: diff what the new surface saw against what the fixture declares. Fixtures written before the surface existed often smuggle realistic data through paths the type system never classified (undeclared keys, copied payloads) — narrow the fixture to its declared fields, preserving the test's original intent. Re-greening old tests by loosening the detector reinstates the blind spot the widening closed | ## Instead of @@ -80,4 +83,5 @@ Reviewing a guard that has never been red → [testing-quality-tests-that-cannot - https://testing.googleblog.com/2015/01/testing-on-toilet-change-detector-tests.html — Alex Eagle, "Testing on the Toilet: Change-Detector Tests Considered Harmful" (2015-01-27): "Change-detector tests do not add clarity, and you cannot safely refactor code if you know you need to adapt the tests afterwards to get them passing again." A shape-only guard that must be exempted for each new legitimate artifact is this failure mode at repo scope - https://pitest.org/ — "Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived" — the basis for step 4's required-red fixture +- Field evidence (linkly #43, 2026-08-06): adding `result.bindings` to a differential masking surface immediately reddened `test_when_guard_removed_diverges`; the repro showed the fixture's seeded row carrying an undeclared `password` key raw through the new channel while the entity declared only `id`/`email`/`token`. Narrowing the payload to declared fields preserved the test's guard-divergence intent and returned 1218 tests green with the widened detector intact - Field evidence (linkly #35, 2026-08-04): `test_no_shipped_example_has_a_guarded_repository_call` asserted that no shipped `.lnpl` example contained a repository call under a guard. `examples/checkout.lnpl` legitimately added a `create` under `when stock > 0` — the issue's own reproduction shape — turning the guard permanently red. Re-expressing it as "a guarded call that could actually fail", with the conflict/miss decision taken from the production `_lnpl_ops` derivation via `seeded_entities`/`repository_calls`, returned the suite to `Ran 518 tests / OK` while a fixture holding a guarded-and-can-fail create still drove the guard red diff --git a/wiki/testing/quality/tests-that-cannot-fail.md b/wiki/testing/quality/tests-that-cannot-fail.md index b0534af..c95fc99 100644 --- a/wiki/testing/quality/tests-that-cannot-fail.md +++ b/wiki/testing/quality/tests-that-cannot-fail.md @@ -15,7 +15,7 @@ sources: - https://www.gnu.org/software/sed/manual/html_node/Exit-status.html - https://git-scm.com/docs/git-checkout - https://git-scm.com/docs/git-restore -last_verified: 2026-08-05 +last_verified: 2026-08-06 related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-checks-that-cannot-pass, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, testing-quality-schema-additions-under-a-golden-gate, testing-quality-differential-run-agreement, testing-quality-completion-predicates, testing-quality-guard-shape-vs-consequence, testing-quality-injected-clock-duration-assertions, testing-quality-write-path-assertions, backend-common-change-impact-call-site-enumeration, platforms-shells-portable-shell-scripts, qa-document-verification-spec-document-gates] --- @@ -73,6 +73,7 @@ suite reported as covered, or you are auditing a suspiciously green suite. | Testing the mock instead of the code (mock returns X, test asserts X came back) | Assert the unit's transformation of its inputs, not the pass-through; when no transformation exists at this layer, test the layer that has one ([testing-mocking-what-to-mock]) | | Copied test body with the name changed but identical inputs and expectation | Give each case distinct inputs and its own expectation; delete exact duplicates — a renamed copy re-proves the same fact and guards nothing new | | Assertion inherited from a shared base class, mixin, or parameterised harness, whose name announces the new subject's whole shape while its body pins the original narrow scope | Read the inherited body and list what it compares; add a subject-specific assertion for each part of the shape the name claims, then prove each one with its own mutation | +| Bats assertion written as `[[ … ]]` anywhere but the test's last command, when bats resolves to bash 3.2 (macOS system bash) — a false `[[ ]]` mid-test does not fail the test | Write bats assertions as simple commands — `[ … ]` or `printf '%s\n' "$output" \| grep -qF "expected"` — which fail at any position; before bash 4.0, `set -e` ignores a failing compound command, so a mid-test `[[ ]]` is decoration on that shell (same shape as the documented bats `!`-negation gotcha) | 6. **Coverage note:** a covered line is only an executed line. Use coverage to find untested code; it cannot certify tested behavior. The proof a test @@ -119,3 +120,6 @@ suite reported as covered, or you are auditing a suspiciously green suite. - https://man7.org/linux/man-pages/man2/execve.2.html — the shebang is honoured only on direct execution, not when a file is passed to an interpreter - https://www.gnu.org/software/sed/manual/html_node/Exit-status.html — a `sed` expression that matches nothing still exits 0 - https://git-scm.com/docs/git-checkout, https://git-scm.com/docs/git-restore — `checkout -- ` restores the index copy, discarding unstaged changes; measured 2026-08-05: with the fix unstaged the checkout removed fix and mutation together, and the lost import surfaced as `Ran 1042 … errors=1` where the intact tree ran 1098 +- https://tiswww.case.edu/php/chet/bash/COMPAT — bash-4.0 changed `set -e` handling so the shell exits when a compound command fails; bash-3.2 and earlier do not, which is what lets a false mid-test `[[ ]]` pass silently under bats on macOS system bash +- https://bats-core.readthedocs.io/en/stable/gotchas.html, https://www.shellcheck.net/wiki/SC2314 — the documented same-shape gotcha: bats commands whose failure is excluded from errexit (negated `!` commands) "can never fail when used in the middle of a test" +- Local reproduction 2026-08-06 (Bats 1.14.0, GNU bash 3.2.57, macOS arm64): a false `[[ "a" == *"zzz"* ]]` mid-test → `ok`; the same false comparison as `[ "a" = "zzz" ]` or piped `grep -qF` mid-test → `not ok`; the `[[ ]]` as the test's last line → `not ok`. Outside bats, `bash -ec '[[ … ]]; echo survived'` printed and exited 0 while the `[ ]` form aborted — bash-3.2 errexit semantics, not a bats defect