diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 7172a68..570b40c 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,192 +1,79 @@ -# Knowledge flush — 3 insight(s) +# Knowledge flush — 5 insight(s) -Queue drained: `~/.dev-loop/queue/0c6a5439-….jsonl` (1 row), -`~/.dev-loop/queue/fb7e7221-….jsonl` (2 rows). All three were harvested -2026-08-04 from the `linkly-t1-spec-notation` and `linkly-t1-repo-policy` repos. +Drained 5 pending rows from `~/.dev-loop/queue` in two rounds (session files: qa-t1-inventory-order ×1, qa-t4-rate-notify ×2, then qa-t2-payment-refund ×2 that were harvested while the first round was in flight). Result: **5 new pages (qa ×4, security ×1), 7 reverse related-links, 0 dropped, all `confidence: verified`.** One candidate's stated evidence wording was corrected during verification (I-3), and one queue domain hint was re-routed (I-5: `debugging` → `qa`). -Result: **2 new pages, 1 merge into an existing page, 1 new category.** - -| # | Insight (trigger → directive) | Outcome | Confidence | -|---|-------------------------------|---------|------------| -| 1 | Adding a node kind to a format whose only gate mutates a golden example → commit a fixture holding the new kind, one negative per keyword, verify each reddens | **New page** `testing/quality/schema-additions-under-a-golden-gate` | verified | -| 2 | Enumerating call sites before a contract change → enumerate by callee name; a parameter-name search is a partial index | **New page + new category** `backend/common/change-impact/call-site-enumeration` | verified | -| 3 | A fixture helper whose shape depends on a value the test also passes to the SUT → put that value in the helper's signature | **Merged** into `testing/data/test-data-and-isolation` | verified | - ---- +| # | Insight | Target page | Confidence | +|---|---------|-------------|-----------| +| 1 | `git status --porcelain` needs `-uall` before path-filtering for scope purity | `wiki/qa/process/scope-purity-checks.md` (new) | verified | +| 2 | Run a control pair before trusting a name-based value-override matrix | `wiki/qa/exploratory/override-control-pairs.md` (new) | verified | +| 3 | Execute every guard-true path once when static stages don't resolve references | `wiki/qa/exploratory/guard-true-path-coverage.md` (new) | verified | +| 4 | Verify masking per output channel with a planted secret + unmasked control field | `wiki/security/data/masking-verification.md` (new) | verified | +| 5 | Count stacked declarations in the lowered IR before trusting runtime | `wiki/qa/exploratory/lowered-declaration-survival.md` (new) | verified | ## Verified best-practice -Every URL below was fetched during this flush; the quotes are from those -fetches, not from memory. Nothing was cited that I did not open. +### 1 — Scope-purity checks over `git status` output need `-uall` -### Insight 1 — a golden-derived negative corpus cannot reach a newly added schema branch +**Claim as queued:** default porcelain collapses an entirely-untracked directory to one `?? qa/` line, so a per-file path filter (`^?? qa/cases/...`) mis-matches and the purity gate reports a false violation; `-uall` expands to per-file lines. -*Claim under test:* when a format's only gate builds negatives by mutating one -committed golden example, adding a new node kind to the schema produces a green -run that proves nothing about the addition. +**Verified two ways:** +- **Official docs** — [git-status](https://git-scm.com/docs/git-status): mode `normal` "Show untracked files and directories", mode `all` "Also show individual files in untracked directories"; the default equals `normal` and is user-configurable via `status.showUntrackedFiles`. The config point yields an addition the candidate lacked: a checkout with `showUntrackedFiles=no` hides untracked files entirely, turning the same gate into a **false pass** — so scripts must pass `-uall` explicitly, never rely on the ambient default. Also doc-verified for the page's edge cases: porcelain v1 rename lines are `R -> ` (two paths, one line), special-character paths are C-string-quoted unless `-z`, and ignored files are omitted unless `--ignored=matching`. +- **Local reproduction** (git 2.50.1 Apple Git-155, scratch repo, 2026-08-05): default porcelain printed exactly `?? qa/` and the per-file filter left it as a "violation" line (grep exit 0); `-uall` printed the three real file paths and the filter passed (grep exit 1). Matches the t1 session evidence (14 files proven in scope only after `-uall`). -| Source checked | What it establishes | -|----------------|---------------------| -| https://json-schema.org/understanding-json-schema/reference/conditionals | The mechanism, verbatim: *"If `if` is invalid, `else` must also be valid (and `then` is ignored)"* — a branch keyed on the new kind is simply **not applied** to an instance that lacks it. So a mutant of a golden without the new kind cannot exercise the new branch. | -| https://json-schema.org/understanding-json-schema/reference/object | Why the new branch also needs constraining before a negative can even exist: *"By default any additional properties are allowed"* and *"By default, the properties defined by the `properties` keyword are not required"*. | -| https://json-schema.org/draft/2020-12/json-schema-core | Sibling applicators *"MUST NOT impact the results of sibling subschemas"*; and *"Unknown keywords SHOULD be treated as annotations"* — a misspelled keyword in a new branch is ignored rather than rejected, a second silent-pass mode. | -| https://pitest.org/quickstart/basic_concepts/ | The same gap has a name in mutation tooling: *"**No coverage**: the same as **Survived** except there were no tests that exercised the line of code where the mutation was created"* — cited so the page tells a reader using PIT/Stryker what the symptom looks like there. | +### 2 — A control pair before trusting a value-override matrix -*Session evidence (kept as a field observation, not as the basis of the -directive):* in `linkly-t1-spec-notation`, `grep -rln "lir.schema\|jsonschema" -impl/tests/` returned 0 of 447 tests, and the only schema gate over `*.lir.json` -was `scripts/validate_ir.py --self-test`, whose three negatives are all -`copy.deepcopy` mutations of `examples/login.lir.json`. +**Claim as queued:** when a CLI takes name-based runtime value injection (`--field key=value`) and ignores unknown keys, run a control pair that flips an observable before trusting any value matrix; uniform output across variants means "key silently ignored", not "behavior stable". -**Confidence: `verified`** — the directive's mechanism is stated in the official -JSON Schema documentation; the incident is corroborating, not load-bearing. +**Verified:** +- **Session reproduction with archived raw output** (lnpl 0.2.0, qa-t4-rate-notify, `qa/cases/rate-notify/evidence/05-modeB.md` + `evidence/raw/modeB-b*.txt`): five runs with bare names (`--field value=150/50/100...`) produced effectively identical step traces — guarded create step never fired, until-loop always hit its 16-round cap — with **exit 0 and zero warnings**. The tool's own help documents the policy: "Fields the workflow does not compare on are ignored; omitted ones default to 0." Switching to the canonical dotted name (`--field measurement.value=150`) flipped the guarded step, proving the lever, after which the matrix produced differentiated, meaningful rows. +- **Mechanism prevalence** — [pydantic models docs](https://pydantic.dev/docs/validation/latest/concepts/models/): "By default, Pydantic models won't error when you provide extra data, and these values will simply be ignored"; `ConfigDict(extra=...)` = `ignore` (default) / `allow` / `forbid`. This grounds the page's directive to prefer strict/forbid modes for measurement runs. +- **Citation dropped, not approximated:** I attempted to add Kohavi's A/A-test literature as a third source; the candidate URL (kdd.org PDF) could not be content-verified by fetch, so it is **not cited**. The in-repo page [testing-quality-harness-reverse-controls] (mutation-testing-sourced) carries the same "prove the instrument discriminates" principle and is linked instead. -### Insight 2 — enumerate by callee name, not by parameter name +### 3 — Guard-true path coverage when static stages skip reference resolution -*Claim under test:* a keyword-argument search (`repo_rows=`) is structurally -incapable of finding call sites that pass the same argument positionally. +**Claim as queued:** steps behind guards must be executed with the guard true at least once, because a pipeline whose compile/validation doesn't resolve cross-node references defers those errors to runtime, and guard-false runs return rc=0 forever. -| Source checked | What it establishes | -|----------------|---------------------| -| https://docs.python.org/3/glossary.html | *positional-or-keyword* — *"specifies an argument that can be passed either positionally or as a keyword argument. **This is the default kind of parameter**"*. The blindness is therefore the default case, not an edge case. | -| https://docs.python.org/3/library/ast.html | `ast.Call`: *"`args` holds a list of the arguments passed by position"*, *"`keywords` holds a list of `keyword` objects representing arguments passed by keyword"* — the two forms live in **separate fields**, so a keyword-name text search reads only one of them. | -| https://peps.python.org/pep-0570/ | The `/` and `*` markers, which is what makes the page's "declare it keyword-only so a stale positional call errors instead of rebinding" edge case actionable. | +**Verified:** +- **Session reproduction** (qa-t4-rate-notify, `qa/cases/rate-notify/evidence/04-modeA.md`): an `emit` referencing an undeclared event passed compile (0 errors) and IR validation (PASS); 6 of 7 runtime runs failed at the emit step with "EventEmit references undeclared event 'event.notification'"; the single run where a presence guard skipped emit exited 0. **Correction to the queue row:** the failure text is the above, not "RunError" — the page and this report quote the actual error. The same evidence file's bidirectional guard table (guard.1/2/3, true and false runs each, all three discriminating) is the template for the page's contrast-table directive, and its guard.3 zero-round observation (0-iteration `until` absent from the skipped list) became the "assert on executed steps, not skip markers" directive. +- **External grounding** — [ISTQB glossary, branch coverage](https://istqb-glossary.page/branch-coverage/): "The percentage of branches that have been exercised by a test suite. 100% branch coverage implies both 100% decision coverage and 100% statement coverage." The page applies this at whole-program QA level: a guard is a branch; N green runs down one side accumulate no evidence about the other. -*Reproduction run this session* (Python 3.14.6, macOS, no files written — piped -to `python3` on stdin): over four call sites of `verify(...)` of which one passes -`repo_rows=` by keyword, a regex search for `repo_rows\s*=` matches **1** while -an AST pass over `Call` nodes named `verify` finds **4** — 3 sites invisible to -the keyword search. This is the minimal version of the reported incident. +### 4 — Masking verified per output channel, with a negative control -*Session evidence:* recon reported "13 call sites, 7 need editing"; 8 further -seeds passed the value as `verify()`'s 4th positional argument, and the suite the -session had reported green then ran `472 tests / FAILED (failures=11)`. +**Claim as queued:** masking is implemented per-channel, so a check that passes on one channel proves presence, not enforcement — enumerate every output channel, grep the raw secret in each, and pair with a control field that must appear unmasked. -**Confidence: `verified`** — official language reference plus a reproduction. +**Verified by my own fresh reproduction** (the t2 worktree that produced the candidate was blocked by the session-isolation guardrail, so I rebuilt the case from scratch in my own worktree, lnpl 0.2.0, 2026-08-05): a `Password`-typed field fed the planted value `4111111111111111` through a run with `--json`. One output document contained **the raw card number at `.result.bindings.account.cardSecret` and `***` at `.trace.logs[0].payload.cardSecret`**; the unmasked control field `label` appeared in both channels (proving both channels were captured). This independently confirms the queued evidence, including its sharpest claim — the platform's own differential check reported "PASS 4/4 masking" because it compares only the masked-clean channels. -### Insight 3 — a fixture factory takes the value the test also passes to the SUT +- **External grounding** — [OWASP Logging Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html): "Authentication passwords" and "Bank account or payment card holder data" are data to be "removed, masked, sanitized, hashed, or encrypted" rather than logged — masking framed as a *logging-layer* responsibility, which is exactly why the result-payload channel stays uncovered. -*Claim under test:* when a fixture helper's shape depends on a value the test -also feeds the system under test, that value belongs in the helper's signature -rather than in a module-level default. +### 5 — Stacked declarations counted in the lowered artifact -| Source checked | What it establishes | -|----------------|---------------------| -| https://abseil.io/resources/swe-book/html/ch12.html | A test is *complete* when *"its body contains all of the information a reader needs in order to understand how it arrives at its result"*; DAMP over DRY; and, directly on point, that engineers should *"use helper methods with descriptive parameters that make dependencies explicit"* rather than reusing shared constants. | -| https://testing.googleblog.com/2017/01/testing-on-toilet-keep-cause-and-effect.html | Keep the inputs a result depends on visible in the test method rather than in shared setup, so cause and effect is readable without jumping elsewhere. | +**Claim as queued:** a compiler/DSL can silently keep only the last of consecutive declarations with exit 0 and no diagnostic — inspect the lowered IR/AST and count that every declared item survived before trusting runtime. -*Session evidence:* `rows_for(doc)` seeded from the module constant `PAYLOAD` -while its tests ran payload `{}`; a shape-only migration fixed 1 of 11 failures, -moving the payload into the signature fixed 11 of 11. +**Verified by my own fresh reproduction** (same guardrail note as I-4; rebuilt from scratch, lnpl 0.2.0, 2026-08-05): a workflow with `when approval.amount > 100` directly followed by `when approval.amount < 0` compiled with **rc=0 and zero diagnostics**; both the Semantic IR JSON (`lnpl compile`) and the lowered MLIR contained **exactly one Guard node — `wf.approve.refund.guard.1`, carrying the second guard's condition under the first guard's id**. That id-reuse detail (absent from the queue row) became page directive 3: count by condition content, not by node-id presence. The runtime consequence (amounts `0` and `-1` approved past the dropped guard) is carried from the originating session as described context. -**Confidence: `verified`** — both sources are prescriptive on the exact point. +## Existing-layer check -Note: the canonical name for this smell is xUnit Patterns' *Mystery Guest*. -`xunitpatterns.com` is HTTP-only and could not be fetched over HTTPS this -session, so **it is deliberately not cited** — an unfetchable URL is worse than -none, per the ingest rules. +**Read:** root `INDEX.md`; `wiki/qa/index.md` (all categories); `wiki/qa/exploratory/exploratory-sessions.md` (full); `wiki/testing/index.md`; `wiki/testing/quality/harness-reverse-controls.md`; `wiki/testing/quality/minimum-case-set.md` (full); `wiki/testing/quality/checks-that-cannot-pass.md` (trigger sections); `wiki/security/index.md`; `wiki/security/data/pii-handling.md`; `wiki/debugging/index.md`; plus repo-wide greps for `porcelain|untracked|-uall`, `control`, `branch coverage|guard|unknown key`, `mask`. ---- +**Overlaps found, all resolved as adjacency (cross-link), not duplication:** +- I-1 vs [testing-quality-checks-that-cannot-pass] — that page validates a check against known-good input; I-1 is a specific gate whose false verdict comes from git's output mode, and its directive 3 routes to that page for the both-ways control. Linked both ways. No existing page mentions porcelain/untracked mechanics (grep confirmed). +- I-2 vs [testing-quality-harness-reverse-controls] — same principle ("uniform verdict is a property of the instrument; run a control"), different situation: that page is about citing a *scoring harness's* verdict, I-2 about a *measurement matrix* through value injection. Kept separate, linked both ways. +- I-3 vs [testing-quality-minimum-case-set] — that page selects cases for automated tests of a function; I-3 covers runtime QA of guarded whole-program paths. Linked both ways. +- I-4 vs [security-data-pii-handling] — that page *designs* PII handling (mask at the logger layer, one central filter); I-4 *verifies* that such masking actually holds per channel. Complementary, linked both ways; I-4's error-report edge row routes to pii-handling's scrub-hook row. +- I-5 vs I-3 — siblings: I-5 counts declarations that survived lowering; I-3 executes the survivors. Cross-linked; also linked to I-2 (all three are probing techniques from the same QA campaign family). -## Existing-layer check +**Open-PR overlap scan (13 open `dev-loop:knowledge` PRs by title, closest by body: #32, #34, #24, #23):** none covers these five. Flagged as *cousins, not duplicates*: PR #32 `infrastructure/config/keys-ahead-of-their-consumer` and PR #34 `backend/common/api-design/unenforced-declarations` share the "accepted but not acted on" mechanism family — #34's page is the *designer's* side (closed-table lookup, strictness levels, K8s KEP-2885) while I-2/I-5 are the *consumer/QA* side (prove your lever connects; count survivors); no content conflict, and a follow-up `related:` link between them is worth adding after whichever merges second. PR #34's `testing/strategy/differential-testing` is adjacent to I-4's "ask which channels the differential check compares". PR #24/#22 completion-predicate pages share I-2's control-run spirit; PR #23 `guard-shape-vs-consequence` concerns test-artifact guards, unrelated to runtime `when`/`until` guards. -Routed via `INDEX.md` → domain `index.md` → every page whose "load when" line -overlapped. Pages read in full before deciding: all six of `testing/quality/`, -`testing/data/test-data-and-isolation`, `testing/index`, `qa/process/regression-scope`, -`qa/index`, `debugging/index`, `backend/index`, `backend/python/index`, plus a -repo-wide search for prior coverage (`grep -rliE "call site|callee|positional -argument|keyword argument"` and a `\bgrep\b` sweep over `wiki/`). - -### Insight 1 — nearest neighbours, and why it is not a duplicate - -| Page read | Overlap | Decision | -|-----------|---------|----------| -| `testing/quality/spec-artifact-checks` | Closest. Already owns *"one negative control per check, mutating only what that check owns"* and the coverage-vs-validity split. | **Not a merge.** Its trigger is *authoring a check*; this insight's trigger is *evolving the artifact the check validates* — the negatives already exist and are individually sound, yet the corpus as a whole cannot reach the new shape. Per the wiki's one-case-per-page rule this is a new trigger. Linked both ways. | -| `testing/quality/tests-that-cannot-fail` | Owns the break-the-code red-run rule the new page's step 4 depends on. | Referenced inline; `related:` added both ways. | -| `testing/quality/harness-reverse-controls` | Owns the *harness-level* control (can this harness go green). | Complementary, not overlapping: that page asks whether the harness discriminates at all, this one asks whether the fixture corpus reaches a newly added shape. `related:` added both ways. | -| `testing/quality/checks-that-cannot-pass` | Trigger is a check whose **target does not exist yet**. | Distinct — here the target exists and the gate is green. No edit. | -| `qa/document-verification/spec-document-gates` | Release-gate altitude for document deliverables. | Kept as a one-way `related:` from the new page. | - -**No conflicts found.** Nothing in the wiki contradicts the new directive. - -### Insight 2 — no existing coverage anywhere - -The repo-wide search found **zero** pages discussing call-site enumeration, -callee-name search, or positional-vs-keyword arguments as a recon concern. The -single adjacent line is `qa/process/regression-scope`'s edge-case row *"The -change is in code with no test coverage and unclear callers | Trace callers -before scoping"* — which names the need and does not say how. That row now -points at the new page, and `regression-scope` gained the new page in -`related:` (both directions). - -Checked and rejected as homes: `debugging/*` (diagnosis of a failure, not -pre-change recon), `backend/python/language/mutable-state-traps` (mutable -defaults and shared state — a different mechanism), `platforms/tools/bsd-vs-gnu-cli` -(grep *flag* portability, not search strategy). - -### Insight 3 — merged, not created - -`testing/data/test-data-and-isolation` already owns fixture construction and -carries the adjacent rule *"pass explicitly only the fields the test's behavior -depends on"* plus a row for shared **mutable** fixture objects. This insight is -the same trigger with a directive that extends it (a *defaulted* constant, which -is not mutated and so is not covered by the existing row). Merge-before-create -applied — no new page. Added: 1 `Do` row, 1 edge-case row (the symptom is a -lookup miss far from the helper), 1 `Instead of` row, 2 sources, and -`last_verified` bumped to 2026-08-04. - ---- +**Conflicts flagged:** none — no existing directive contradicts any of the five. ## Routing decision -| Insight | Target | Rationale | -|---------|--------|-----------| -| 1 | `testing` / `quality` / **new page** `schema-additions-under-a-golden-gate` | `INDEX.md` routes "writing or structuring automated tests … verifying tests can actually fail" to `testing`; within it, `quality` already holds the five pages about whether a check proves anything. Existing category, no structural change. | -| 3 | `testing` / `data` / **merge** into `test-data-and-isolation` | Same trigger as the page's own ("tests need fixture data and you are choosing how to create it"); directive extends step 1. | -| 2 | `backend` / `common` / **NEW category `change-impact`** / `call-site-enumeration` | See below. | - -### New category: `backend/common/change-impact/` - -**Why `backend`:** `AGENTS.md`'s routing protocol resolves a multi-domain match -by "the domain that owns **the artifact you will change**" — here, application -code. **Why `common`:** the directive is language-agnostic (any language with -optional or positional arguments; in a language with no keyword arguments at all -the parameter-name search returns nothing whatsoever). The Python mechanics are -cited as the mechanism, not as the scope. - -**Why not an existing category** — all eleven `backend/common` categories were -re-checked by name before creating a new one: - -- `api-design` is the only near miss, and all three of its pages are HTTP-shaped - (status codes, endpoint idempotency, list-endpoint pagination). Its "load when" - lines are written about endpoints; filing an in-process function-signature - concern there would make the category's routing lines contradict its contents, - which invariant 1 forbids. -- `reliability`, `caching`, `jobs`, `errors`, `auth`, `orm`, `concurrency`, - `llm`, `integrations`, `storage` — all runtime-behaviour categories; none - covers a design-time change-impact question under any other name. - -`change-impact` is the noun for the concern, leaves room for sibling pages -(schema/event-contract consumers, deprecation windows), and is registered in -`wiki/backend/index.md` plus the root `INDEX.md` backend row. - -### Plumbing - -- `wiki/testing/index.md` — new `quality` row with a use-case-enumerating "load when" line. -- `wiki/backend/index.md` — new `change-impact` section + subtree summary row updated. -- `INDEX.md` — backend row's `common/` concern list updated. -- `log.md` — `## [2026-08-04] ingest | …` entry appended. -- `related:` added both ways for all four adjacent pages. - -### Invariants verified mechanically (not by eye) - -Ran over all **141** pages after the edits: - -- every page is listed in an ancestor `index.md` → **0 unlisted** -- every `related:` id and inline `[page-id]` reference resolves → **0 broken** -- no page exceeds 120 body lines → **0 over** -- banned vague qualifiers (`usually`, `consider`, `might want to`, `generally`, - `as appropriate`) in the three touched pages → **0 hits** -- every "don't"-shaped statement in the new pages is descriptive prose, not a - bare prohibition; anti-patterns appear only in `Instead of` tables, each paired - with its replacement. +- **I-1 → `qa/process/scope-purity-checks`.** Queue hint `qa` confirmed: the artifact is a release/session gate ("did this run stay in its lane"), owned by qa/process alongside release-gates. Not `platforms` (no OS variance) and not `testing` (nothing here writes test code); the git mechanics are the page's evidence, not its owner. +- **I-2 → `qa/exploratory/override-control-pairs`.** Queue hint `qa` confirmed: probing a live system during exploratory QA. The alternative home — `testing/quality` next to harness-reverse-controls — was rejected because that category governs authoring/citing automated checks, while this governs how to *measure* a system by hand; the shared principle is carried by the two-way `related:` link. +- **I-3 → `qa/exploratory/guard-true-path-coverage`.** Same category as I-2 (same probing activity, sibling pages). +- **I-4 → `security/data/masking-verification`.** Queue hint `security` confirmed: the subject is a security control's enforcement, placed beside `pii-handling` (which owns the design side). `qa` was rejected — the channel table and planted-secret method are specific to sensitive-data controls, and security/data readers are the ones about to claim "masking works". +- **I-5 → `qa/exploratory/lowered-declaration-survival`.** **Queue hint `debugging` overridden**: debugging's charter is "diagnosing a failure — finding what is wrong and why", but this directive fires *before* any failure is observed (pre-trust verification during QA); its natural siblings are I-2/I-3 in `qa/exploratory`. If it had been filed as "why did runtime approve amount 0?", the debugging methodology pages would route the investigation — the lesson worth persisting is the preventive count. +- **No new categories.** `qa/exploratory` grows 1→4 pages (its charter — manual/exploratory probing techniques — covers all three newcomers); `security/data` grows 1→2. +- **Plumbing:** `wiki/qa/index.md` +4 rows, `wiki/security/index.md` +1 row; reverse `related:` on `checks-that-cannot-pass`, `harness-reverse-controls`, `minimum-case-set`, `exploratory-sessions`, `pii-handling`, plus cross-links among the three exploratory siblings; `log.md` +2 ingest entries. + +**Invariants (checked mechanically):** body lines 53/53/56/56/52 (≤120) · id matches path 5/5 · all `related:` ids resolve · 0 banned vague qualifiers (one "usually" inside an OWASP quotation was caught and the quote tightened) · every Instead-of row pairs prohibition with replacement · new pages listed in domain index 5/5. diff --git a/log.md b/log.md index c519858..a964dfd 100644 --- a/log.md +++ b/log.md @@ -39,3 +39,5 @@ Append-only. Format: `## [YYYY-MM-DD] /bin` (verified: `which mlir-opt` not found vs `/opt/homebrew/opt/llvm/bin/mlir-opt` → LLVM 22.1.8; source docs.brew.sh/FAQ). platforms/processes/non-interactive-cli-invocation +bracketed-paste edge: injecting a long/multiline prompt into a REPL (tmux `send-keys -l`) stalls at `❯ [Pasted text #1]` because the input is one bracketed-paste block (ESC[200~…201~) whose embedded newline is not submit — send Enter as a separate keystroke a beat later (source en.wikipedia.org/wiki/Bracketed-paste + claude-code#43169). testing/strategy/test-level-choice +import-side-effect edge/instead-of: a "pure" function's test is not dependency-free if its module runs I/O at import; `@pytest.mark.skipif` evaluates after the module import so it can't gate it — use `importorskip`/`skip(allow_module_level=True)` or move the function to a side-effect-free module (source docs.pytest.org skipping). Harvested "infrastructure" hint for the keg-only insight re-routed to platforms/environment (dedicated PATH page). Confidence: keg-only & pytest verified vs official docs; bracketed-paste mechanism doc-verified, the claude-CLI submit specifics field-tested. +## [2026-08-05] ingest | knowledge-flush of 3 queued insights (qa +3, all verified). New: process/scope-purity-checks (default porcelain collapses an entirely-untracked directory to `?? dir/` so per-file path filters false-flag it — pass `-uall` explicitly since `status.showUntrackedFiles` can shift the ambient default; rename lines carry two paths; `-z` for quoting; validated by scratch-repo reproduction on git 2.50.1 + git-scm docs), exploratory/override-control-pairs (before trusting a `--field k=v`/env override matrix into an ignore-unknown-keys consumer, run a control pair that flips an observable; uniform output = lever not connected, not stable behavior — reproduced against a workflow runner where bare key names were silently ignored at exit 0 and the canonical dotted name flipped the guard; pydantic extra-ignored default cited as mechanism prevalence), exploratory/guard-true-path-coverage (run every guard's true and false direction once and assert on the executed-step list — when compile/validation do not resolve cross-node references, the guard-true run is the only reference check; reproduced: undeclared event emit passed compile 0-errors + validate PASS and failed only in runs where the guard let emit execute; ISTQB branch coverage). Reverse related-links added to checks-that-cannot-pass, harness-reverse-controls, minimum-case-set, exploratory-sessions. +## [2026-08-05] ingest | knowledge-flush round 2, same PR: 2 more queued insights (security +1, qa +1, both verified by local reproduction). New: security/data/masking-verification (masking is per-channel — enumerate every output channel and grep the planted raw secret in each, paired with an unmasked negative-control field; reproduced: one lnpl `--json` run held the raw card number at result.bindings while trace/logs showed `***`, and the platform's differential check compares only masked-clean channels), qa/exploratory/lowered-declaration-survival (a DSL accepting stacked declarations at exit 0 can silently keep only the last — count survivors in the lowered IR/AST by condition content, not node id, before trusting runtime; reproduced: two stacked `when` guards lowered to a single Guard node carrying the first guard's id and the second guard's condition). Queue hint `debugging` for the second re-routed to qa/exploratory (pre-failure QA verification, not failure diagnosis). Backlinks: pii-handling, guard-true-path-coverage, override-control-pairs. diff --git a/wiki/qa/exploratory/exploratory-sessions.md b/wiki/qa/exploratory/exploratory-sessions.md index 3dac70b..f6e04e8 100644 --- a/wiki/qa/exploratory/exploratory-sessions.md +++ b/wiki/qa/exploratory/exploratory-sessions.md @@ -8,7 +8,7 @@ sources: - https://www.satisfice.com/download/session-based-test-management - https://developsense.com/blog/2009/08/testing-vs-checking last_verified: 2026-07-10 -related: [qa-bug-reports-reproducible-reports, qa-process-release-gates] +related: [qa-bug-reports-reproducible-reports, qa-process-release-gates, qa-exploratory-guard-true-path-coverage] --- # Running a Chartered Exploratory Testing Session diff --git a/wiki/qa/exploratory/guard-true-path-coverage.md b/wiki/qa/exploratory/guard-true-path-coverage.md new file mode 100644 index 0000000..543b153 --- /dev/null +++ b/wiki/qa/exploratory/guard-true-path-coverage.md @@ -0,0 +1,69 @@ +--- +id: qa-exploratory-guard-true-path-coverage +domain: qa +category: exploratory +applies_to: [general] +confidence: verified +sources: + - https://istqb-glossary.page/branch-coverage/ + - "Field reproduction (lnpl 0.2.0, 2026-08-05): undeclared event reference passed compile (0 errors) and IR validation (PASS); failed only in runs that executed the guarded emit step; the one run where a presence guard skipped the step exited 0" +last_verified: 2026-08-05 +related: [qa-exploratory-override-control-pairs, testing-quality-minimum-case-set, qa-exploratory-exploratory-sessions, qa-exploratory-lowered-declaration-survival] +--- + +# Executing Guard-True Paths When Static Stages Skip Reference Resolution + +## When this applies + +You are QA-ing a program or workflow whose steps hide behind guards (`when` / +`until` conditions, optional branches) in a pipeline whose compile or schema +validation does not resolve cross-node references; or a guarded step has never +executed in any green run you are about to cite. + +## Do this + +1. **Enumerate every guard and run each direction at least once** — one run + where the guard is true and one where it is false — and record which run + exercised which direction in a bidirectional contrast table (guard, true + run, false run, observed signal on each side). This is branch coverage + ("the percentage of branches that have been exercised by a test suite" — + ISTQB) applied at the whole-program QA level. +2. **Treat exit 0 from a run that skipped a guarded step as evidence about the + skip path only.** The guarded step's body is unexecuted code of unknown + validity; repeated green runs that all skip it accumulate no evidence about + it. +3. **Establish what the static stages actually check.** When compile and + validation do not resolve references between nodes, a dangling reference + inside a guarded step (an undeclared event, a missing target id) survives + every static stage and surfaces only when the guard lets the step run — so + the guard-true run is the *only* reference check that exists. Run it before + calling the artifact shippable. +4. **For loop guards (`until`, retry conditions), the zero-iteration case is + also a branch**: include one run where the condition is satisfied + immediately, alongside the run that iterates. +5. **Assert on the executed-step list, not on skip markers alone.** Observed + asymmetry in the field reproduction: a zero-round `until` loop was absent + from the skipped list even though its body never ran — skip markers and + execution records can disagree. + +## Edge cases + +| Case | Then | +|------|------| +| A guard-true state is unreachable through the program's inputs | Record it explicitly as a coverage gap in the QA report; do not claim full guard coverage silently | +| Two guards cannot both be true in one run | Cover them in separate runs; one row per guard in the contrast table, each with its own true/false pair | +| The guard-true run fails late in QA | That is the mechanism working — the error was latent behind the guard; fix, then re-run both directions of that guard | +| Static validation claims to check references | Verify with a planted dangling reference that it actually fails validation; if it passes, treat reference resolution as runtime-only | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Ship after N green runs that all skipped a guarded step | Force one guard-true run per guard first | Compile 0 errors + validation PASS + exit 0 never executed the step body; an error inside it stays latent until production hits the guard-true state | +| Cite validator PASS as proof references resolve | Check whether the validator resolves references; if not, count only guard-true runs as reference checks | Validation scope and reference resolution are separate concerns; PASS on the former says nothing about the latter | +| Infer skipped steps from a skipped-markers list | Assert on the executed-step list | Zero-iteration loops can be missing from both lists — only the execution record is authoritative | + +## Sources + +- https://istqb-glossary.page/branch-coverage/ — "The percentage of branches that have been exercised by a test suite. 100% branch coverage implies both 100% decision coverage and 100% statement coverage" +- Field reproduction (2026-08-05, lnpl 0.2.0 workflow runner): an `emit` referencing an undeclared event passed compile (0 errors) and IR validation (PASS); 6 of 7 runs failed at runtime at the emit step ("EventEmit references undeclared event"), while the single run where a presence guard skipped emit exited 0 — a guard-skipping input masks the defect indefinitely. Raw run outputs archived alongside the QA case diff --git a/wiki/qa/exploratory/lowered-declaration-survival.md b/wiki/qa/exploratory/lowered-declaration-survival.md new file mode 100644 index 0000000..73d589e --- /dev/null +++ b/wiki/qa/exploratory/lowered-declaration-survival.md @@ -0,0 +1,64 @@ +--- +id: qa-exploratory-lowered-declaration-survival +domain: qa +category: exploratory +applies_to: [general] +confidence: verified +sources: + - "Local reproduction (lnpl 0.2.0, 2026-08-05): two stacked `when` guards compiled rc=0 with zero diagnostics; Semantic IR and MLIR both held exactly one Guard node — the first guard's id carrying the second guard's condition" +last_verified: 2026-08-05 +related: [qa-exploratory-guard-true-path-coverage, qa-exploratory-override-control-pairs] +--- + +# Counting Stacked Declarations in the Lowered Artifact + +## When this applies + +A compiler/DSL accepts multiple stacked declarations (consecutive guards, +policies, annotations) with exit 0 and no diagnostic, and you are about to +trust runtime behavior that depends on all of them; or a run honored only one +of several declared rules. + +## Do this + +1. **Dump the lowered artifact and count that every declared item survived**, + before any runtime trust: semantic IR (compile stage output), MLIR/AST dump, + or the generated config — whichever lowered form the toolchain exposes. Key + the count to source constructs: one node per declared guard, each with its + own condition text. +2. **Read "exit 0 + no diagnostic" as absence of rejection, not as + acceptance.** Lowering can keep only the last of consecutive declarations + silently: in the reproduction, two stacked `when` guards produced a single + Guard node, and the loss was invisible in every compiler message. +3. **Count by content, not by id presence.** The surviving node can carry the + *first* item's id with the *second* item's payload (observed: + `…guard.1` holding the second guard's condition) — so "an id exists per + declaration" is a weaker check than "each declared condition string appears + once". +4. **When a declaration is missing from the artifact, probe runtime with an + input only the dropped rule would reject** to size the impact (in the + originating case, runtime approved amounts `0` and `-1` that the dropped + guard existed to block), then report the silent drop as a finding + ([qa-exploratory-guard-true-path-coverage] owns exercising the surviving + guards). + +## Edge cases + +| Case | Then | +|------|------| +| The CLI has no explicit IR-dump flag | Use whatever the build leaves behind — a compile stage that prints IR JSON, `.mlir`/`.ll` files in the build workdir — any lowered form supports the count | +| The language documents merge semantics for stacked declarations | Check the documented merge rule first and verify the merged node matches it; an undocumented merge or last-wins is the finding | +| Same-kind declarations live in different scopes (per-service, per-workflow) | Count within each scope separately — cross-scope totals hide a drop in one scope | +| The artifact nests or renames declarations during lowering | Match on the invariant part (condition text, rule value), not on source-level names | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Trust exit 0 as proof all declarations took effect | Count survivors in the lowered artifact first | Last-wins lowering drops predecessors with no warning; the loss surfaces only when an input the dropped rule guarded slips through | +| Verify the declarations with one runtime input | Pick one input per declared rule, including inputs only a dropped rule would reject | A single input can satisfy the surviving rule and still say nothing about the dropped ones | +| Check that a node id exists per declaration | Check each declared condition/value appears in the artifact | The surviving node can reuse a dropped declaration's id | + +## Sources + +- Local reproduction (2026-08-05, lnpl 0.2.0): a workflow with `when approval.amount > 100` directly followed by `when approval.amount < 0` compiled with rc=0 and zero diagnostics; the Semantic IR JSON and the lowered MLIR each contained exactly one Guard node, `wf.approve.refund.guard.1`, with condition `approval.amount < 0` — the second declaration's condition under the first declaration's id. The originating QA session observed the runtime consequence: amounts `0` and `-1` were approved past the dropped guard diff --git a/wiki/qa/exploratory/override-control-pairs.md b/wiki/qa/exploratory/override-control-pairs.md new file mode 100644 index 0000000..0d64480 --- /dev/null +++ b/wiki/qa/exploratory/override-control-pairs.md @@ -0,0 +1,66 @@ +--- +id: qa-exploratory-override-control-pairs +domain: qa +category: exploratory +applies_to: [general] +confidence: verified +sources: + - https://pydantic.dev/docs/validation/latest/concepts/models/ + - "Field reproduction (lnpl 0.2.0, 2026-08-05): 5-run override matrix flat under bare key names, exit 0, no warning; canonical dotted name flipped the guarded step — raw traces archived with the QA case" +last_verified: 2026-08-05 +related: [qa-exploratory-guard-true-path-coverage, testing-quality-harness-reverse-controls, testing-quality-tests-that-cannot-fail, qa-exploratory-lowered-declaration-survival] +--- + +# A Control Pair Before Trusting a Value-Override Matrix + +## When this applies + +You are measuring branch/guard behavior by feeding a matrix of values through +name-based runtime injection — CLI `--field key=value`, environment overrides, +config overlays — into a consumer whose policy for unknown keys is "ignore". +Also whenever every variant in such a matrix returns identical observations. + +## Do this + +1. **Before running the matrix, run one control pair**: two runs whose injected + values are chosen to flip a concrete observable (a guard fires in one and + skips in the other). Require the flip. Until the flip is observed, no run of + the matrix measures anything. +2. **Read uniform output across all variants as "lever not connected", not + "behavior stable".** Ignore-unknown-keys is a common default — pydantic, for + example: "By default, Pydantic models won't error when you provide extra + data, and these values will simply be ignored" — so a mis-named key produces + the default-value branch on every run, with exit 0 and no warning. +3. **Look up the canonical internal key name** (normalized dotted path, IR + field id) from the tool's trace output, IR dump, or docs — not from the + spelling you used in source. A bare name that normalizes differently lands + in the "not compared" bucket silently. +4. **When the consumer offers a strict mode** (error/warn on unknown keys — + pydantic `extra='forbid'`, schema `additionalProperties: false`), turn it on + for the measurement run so a mis-named key fails loudly instead of + defaulting. +5. **Do not count exit codes as evidence the values landed.** The failure mode + is exit 0 across the whole matrix. + +## Edge cases + +| Case | Then | +|------|------| +| The tool's help text documents the ignore policy | The run is still silent — read the key-handling policy before building the matrix, and pick key names from the documented canonical form | +| The chosen observable is insensitive to the value (aggregate status, summary count) | Switch to an observable the value provably drives: executed-step list, trace line, emitted record — then re-run the control pair | +| No injected value flips anything observable | Instrument first (verbose/trace mode, debug output) before measuring; a matrix without any observable lever produces only noise | +| The control pair flips but a later matrix cell looks impossible | Re-run that cell's control neighbor — key handling can differ per field (compared vs non-compared fields) | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Conclude "behavior is stable" from identical outputs across a value matrix | Run a control pair and require the observable to flip first | Silently ignored keys make misconfiguration masquerade as coverage — every run exercised the same default branch | +| Trust the key spelling you wrote in source files | Find the normalized internal name from trace/IR output | Normalized (dotted) and bare names diverge; the bare name is classified as a field the run does not compare on | +| Cite the matrix's uniform exit 0 as passing evidence | Cite the flipped observable from the control pair, then per-cell observations | Exit status stays 0 while the injection is ignored | + +## Sources + +- https://pydantic.dev/docs/validation/latest/concepts/models/ — "By default, Pydantic models won't error when you provide extra data, and these values will simply be ignored"; `ConfigDict(extra=...)` values `ignore` (default) / `allow` / `forbid` — the ignore-unknown default this page defends against +- Field reproduction (2026-08-05, lnpl 0.2.0 workflow runner): five `--field value=N` runs produced byte-identical step traces (guard always false, exit 0, no diagnostics); the tool's own help stated "Fields the workflow does not compare on are ignored"; switching to the canonical dotted name `measurement.value` flipped the guarded create step. Raw run outputs archived alongside the QA case +- [testing-quality-harness-reverse-controls] — the same principle applied to harnesses that *score* verification: a uniform verdict is a property of the instrument, proven otherwise only by a control run diff --git a/wiki/qa/index.md b/wiki/qa/index.md index c80140c..ccb8f4a 100644 --- a/wiki/qa/index.md +++ b/wiki/qa/index.md @@ -17,6 +17,7 @@ Match your situation to a "load when" line; load only matching pages. | [regression-scope](process/regression-scope.md) | Choosing what to re-test for a release/change when full regression is too expensive; reviewing someone else's proposed regression scope | | [severity-and-priority](process/severity-and-priority.md) | Triaging a bug — deciding how bad it is and when it gets fixed; a triage stalled on a severity debate | | [post-release-verification](process/post-release-verification.md) | A release just deployed to production; defining what "released safely" means; an incident revealed a release was broken for hours before anyone noticed | +| [scope-purity-checks](process/scope-purity-checks.md) | Proving a change/session/agent run touched nothing outside an allowed path set by filtering `git status --porcelain` output; a purity gate flags `?? dir/` for a directory that is wholly in scope; writing such a gate for an orchestration/CI workflow | ## document-verification @@ -42,3 +43,6 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [exploratory-sessions](exploratory/exploratory-sessions.md) | A new feature needs testing beyond its scripted checks; you have test time available and want maximum new information per hour | +| [override-control-pairs](exploratory/override-control-pairs.md) | Measuring branch/guard behavior through a matrix of name-based runtime value overrides (`--field k=v`, env overlays) into a consumer that ignores unknown keys; every variant in the matrix returns identical observations | +| [guard-true-path-coverage](exploratory/guard-true-path-coverage.md) | QA-ing a program whose steps hide behind `when`/`until` guards in a pipeline whose compile/validation stages do not resolve cross-node references; a guarded step has never executed in any green run being cited | +| [lowered-declaration-survival](exploratory/lowered-declaration-survival.md) | A compiler/DSL accepted stacked declarations (consecutive guards, policies, annotations) with exit 0 and no diagnostic; about to trust runtime behavior that depends on all of them; a run honored only one of several declared rules | diff --git a/wiki/qa/process/scope-purity-checks.md b/wiki/qa/process/scope-purity-checks.md new file mode 100644 index 0000000..c1a457a --- /dev/null +++ b/wiki/qa/process/scope-purity-checks.md @@ -0,0 +1,66 @@ +--- +id: qa-process-scope-purity-checks +domain: qa +category: process +applies_to: [general] +confidence: verified +sources: + - https://git-scm.com/docs/git-status + - "Local reproduction, git 2.50.1 (Apple Git-155), 2026-08-05: collapsed `?? qa/` vs -uall per-file expansion" +last_verified: 2026-08-05 +related: [testing-quality-checks-that-cannot-pass, testing-quality-harness-reverse-controls] +--- + +# Proving Scope Purity from `git status` Output + +## When this applies + +You must prove that a change, session, or agent run touched nothing outside an +allowed path set by filtering `git status --porcelain` lines; a purity gate +reports a violation on a line like `?? qa/` for a directory that is wholly in +scope; or you are writing such a gate for an orchestration/CI workflow. + +## Do this + +1. **Run `git status --porcelain -uall` whenever the output will be filtered by + path.** The untracked-files mode decides whether your filter can see real + paths at all: + +| Mode | Output for an entirely-untracked directory | Effect on a path-filter gate | +|------|--------------------------------------------|------------------------------| +| `-uno` | nothing | out-of-scope untracked files are invisible — false pass | +| default (`-unormal`) | one collapsed `?? dir/` line | a per-file filter (`^\?\? qa/cases/…`) never matches the collapsed line — false violation | +| `-uall` | one line per file ("Also show individual files in untracked directories") | filter sees real paths — correct verdict | + +2. **Pass the mode flag explicitly in scripts; never rely on the ambient + default.** The default is user-configurable via `status.showUntrackedFiles` + — a checkout where it is set to `no` makes the same gate silently pass with + untracked out-of-scope files present. The command-line flag overrides the + config. +3. **Validate the gate in both directions before trusting its first verdict**: + run it against a tree whose changes are all in scope (must pass) and against + the same tree with one planted out-of-scope file (must fail). A gate first + observed only failing — or only passing — has not demonstrated it can tell + the two apart ([testing-quality-checks-that-cannot-pass]). + +## Edge cases + +| Case | Then | +|------|------| +| Staged renames | Porcelain v1 prints `R -> ` — one line, two paths; the filter must accept the line only when **both** sides are in scope | +| Paths with whitespace/nonprintable characters | Porcelain v1 quotes them as C string literals, so a plain path prefix no longer matches — use `-z` (NUL-terminated, no quoting) and split on NUL | +| Purity must also cover ignored artifacts (build outputs, caches) | `git status` omits ignored files entirely; add `--ignored=matching` to list paths matching ignore patterns | +| Gate runs in a fresh worktree/clone | Config differences travel with `$HOME`, not the repo — the explicit `-uall` flag is still required | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Filter default `git status --porcelain` output with per-file path patterns | Add `-uall` first | An entirely-untracked directory collapses to `?? dir/`, which file-level patterns cannot match | +| Rely on the repo's ambient untracked-files default | Pass `-uall` explicitly in the gate script | `status.showUntrackedFiles=no` in any user config hides untracked files and turns the gate into a rubber stamp | +| Adopt the gate after seeing it fail once on real output | Run known-in-scope and planted-out-of-scope controls | Every mistyped filter also produces a failing run; only the pass/fail pair shows the gate discriminates | + +## Sources + +- https://git-scm.com/docs/git-status — `-u` modes ("normal — Shows untracked files and directories", "all — Also show individual files in untracked directories"), `status.showUntrackedFiles`, porcelain v1 rename format (` -> `), C-string quoting vs `-z`, `--ignored=matching` +- Local reproduction (git 2.50.1, 2026-08-05): scratch repo with `qa/cases/x/{a,b}.md`; default porcelain printed the single line `?? qa/`, which a `^\?\? qa/…` per-file filter treated as a violation; `-uall` expanded to three file lines and the filter passed diff --git a/wiki/security/data/masking-verification.md b/wiki/security/data/masking-verification.md new file mode 100644 index 0000000..146bce9 --- /dev/null +++ b/wiki/security/data/masking-verification.md @@ -0,0 +1,69 @@ +--- +id: security-data-masking-verification +domain: security +category: data +applies_to: [general] +confidence: verified +sources: + - https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html + - "Local reproduction (lnpl 0.2.0, 2026-08-05): one `--json` run held the raw planted card number at result.bindings while the trace/log channel showed `***`; an unmasked control field appeared in both" +last_verified: 2026-08-05 +related: [security-data-pii-handling, testing-quality-harness-reverse-controls, qa-exploratory-override-control-pairs] +--- + +# Verifying Masking Across Every Output Channel + +## When this applies + +You are about to claim that masking/redaction of a sensitive field works — a +logger filter, type-driven DSL masking, a serializer — or a masking check has +passed on one channel, or you are reviewing an automated/differential "masking +PASS" verdict from the platform itself. + +## Do this + +1. **Enumerate every distinct output channel the runtime produces**, then grep + the raw secret value in each one: + +| Channel | Where masking is typically applied | +|---------|-------------------------------------| +| Result payload / API response | Not by a logger filter — the response never passes through it | +| Trace / log stream | The usual home of masking (logger filter, log-record rewrite) | +| Human-readable / pretty output | Separate serializer from the JSON path — check both | +| Generated artifacts (OpenAPI examples, docs) | Generated from schema/IR, not through the logger | +| Error reports / crash dumps | Third-party SDK payloads need their own scrub hooks ([security-data-pii-handling]) | + + Masking is implemented per-channel; a check that passes on one channel + proves presence there, not enforcement anywhere else. +2. **Plant a known, distinctive raw value** (a test card number such as + `4111111111111111`) so the grep is exact and cannot false-match. +3. **Pair every channel's grep with a negative-control field that must appear + unmasked in the same channel.** A channel where the control is also absent + was not captured at all — the "no raw value" result is vacuous + ([testing-quality-harness-reverse-controls]). +4. **Treat the platform's own masking verdict as scoped to the channels it + compares.** Ask which channels the check reads before accepting its PASS: a + differential check that only compares masked-clean channels reports PASS + while the raw value sits in the result payload. + +## Edge cases + +| Case | Then | +|------|------| +| The value is transformed before output (formatted, truncated, chunked) | Grep distinctive substrings of the raw value as well as the whole | +| Structured and human-readable output serialize separately | Run the channel sweep once per serialization mode (`--json` and default) | +| Masking is type-driven (a `Password`-typed field) | Verify per channel anyway — the type triggers masking only in the layers wired to honor it | +| A new output channel is added later (export, webhook, metrics label) | Re-run the sweep; channel enumeration is release-scoped, not one-time | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Claim masking works after seeing `***` in the log | Grep the raw value in every enumerated channel of the same run | The log is one channel; the result payload in the same output file can carry the raw value | +| Accept the tool's "masking PASS" verdict | Determine which channels the verdict compares, then sweep the rest yourself | The verdict is a claim about the compared channels only | +| Check only that the raw value is absent | Also require the negative-control field present in each channel | An empty or uncaptured channel makes absence meaningless | + +## Sources + +- https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html — lists "Authentication passwords" and "Bank account or payment card holder data" as data to be "removed, masked, sanitized, hashed, or encrypted" rather than recorded in logs; sanitization is framed as a logging-layer responsibility — which is exactly why other channels stay uncovered +- Local reproduction (2026-08-05, lnpl 0.2.0 runner): a `Password`-typed field fed the planted value `4111111111111111`; one `--json` output held the raw value at `.result.bindings.account.cardSecret` and `***` at `.trace.logs[0].payload.cardSecret`; the unmasked control field `label` appeared in both channels. Matches the originating QA case, where the platform's differential check reported "PASS 4/4 masking" while the raw card number sat in `result.bindings` diff --git a/wiki/security/data/pii-handling.md b/wiki/security/data/pii-handling.md index 9d5466b..77767ee 100644 --- a/wiki/security/data/pii-handling.md +++ b/wiki/security/data/pii-handling.md @@ -9,7 +9,7 @@ sources: - https://gdpr-info.eu/art-17-gdpr/ - https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html last_verified: 2026-07-10 -related: [databases-schema-design-soft-delete] +related: [databases-schema-design-soft-delete, security-data-masking-verification] --- # Handling Personal Data (PII) in a Feature diff --git a/wiki/security/index.md b/wiki/security/index.md index b25d79b..9c6ec2c 100644 --- a/wiki/security/index.md +++ b/wiki/security/index.md @@ -49,3 +49,4 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [pii-handling](data/pii-handling.md) | A feature stores/processes personal data (emails, names, phones, addresses, government ids); reviewing a log/analytics/export/URL path that can carry PII; designing retention/erasure or handling an erasure request; choosing staging/test data for tables holding PII | +| [masking-verification](data/masking-verification.md) | About to claim masking/redaction of a sensitive field works (logger filter, type-driven masking, serializer); a masking check passed on one output channel; reviewing an automated "masking PASS" verdict | diff --git a/wiki/testing/quality/checks-that-cannot-pass.md b/wiki/testing/quality/checks-that-cannot-pass.md index e371894..306cf76 100644 --- a/wiki/testing/quality/checks-that-cannot-pass.md +++ b/wiki/testing/quality/checks-that-cannot-pass.md @@ -10,7 +10,7 @@ sources: - https://docs.semgrep.dev/writing-rules/testing-rules - https://docs.pytest.org/en/stable/reference/exit-codes.html last_verified: 2026-07-29 -related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set] +related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, qa-process-scope-purity-checks] --- # Validating a Check Whose Target Does Not Exist Yet diff --git a/wiki/testing/quality/harness-reverse-controls.md b/wiki/testing/quality/harness-reverse-controls.md index 2fcf98c..0069db7 100644 --- a/wiki/testing/quality/harness-reverse-controls.md +++ b/wiki/testing/quality/harness-reverse-controls.md @@ -12,7 +12,7 @@ sources: - https://stryker-mutator.io/docs/mutation-testing-elements/equivalent-mutants/ - https://testing.googleblog.com/2021/04/mutation-testing.html last_verified: 2026-08-02 -related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, testing-quality-schema-additions-under-a-golden-gate] +related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, testing-quality-schema-additions-under-a-golden-gate, qa-exploratory-override-control-pairs] --- # Citing a Verification Harness's Own Score diff --git a/wiki/testing/quality/minimum-case-set.md b/wiki/testing/quality/minimum-case-set.md index 8b1287e..1540584 100644 --- a/wiki/testing/quality/minimum-case-set.md +++ b/wiki/testing/quality/minimum-case-set.md @@ -9,7 +9,7 @@ sources: - https://abseil.io/resources/swe-book/html/ch12.html - https://martinfowler.com/bliki/TestDrivenDevelopment.html last_verified: 2026-07-10 -related: [testing-strategy-test-level-choice, testing-quality-behavior-not-implementation, testing-quality-checks-that-cannot-pass] +related: [testing-strategy-test-level-choice, testing-quality-behavior-not-implementation, testing-quality-checks-that-cannot-pass, qa-exploratory-guard-true-path-coverage] --- # Selecting the Minimum Case Set for a Function or Endpoint