From aa6794d1b70d37514f0359c0e2d5de23e07c6268 Mon Sep 17 00:00:00 2001 From: choiyounggi <74581798+choiyounggi@users.noreply.github.com> Date: Sat, 8 Aug 2026 01:41:39 +0900 Subject: [PATCH] knowledge: ingest 2 verified insight(s) --- .dev-loop/INGEST_REPORT.md | 247 +++++++++++++++--- log.md | 3 + .../non-interactive-cli-invocation.md | 2 +- ...diagnostics-without-a-failing-exit-code.md | 2 +- .../command-transcripts-in-a-document.md | 82 ++++++ ...nerated-artifacts-as-deliverable-source.md | 2 +- .../spec-document-gates.md | 12 +- wiki/qa/index.md | 3 +- 8 files changed, 305 insertions(+), 48 deletions(-) create mode 100644 wiki/qa/deliverables/command-transcripts-in-a-document.md diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 55ccfd1..19e4a01 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,53 +1,214 @@ -# Knowledge consolidation — 15 open PRs (#17–#40) → one reconciled state +# Knowledge flush — 5 candidates → 2 ingested, 3 dropped -The 15 open `knowledge/*` PRs (created 2026-08-04 → 2026-08-05, before the -harvest processed-store dedupe fix in #41) contained 123 file-versions of ~75 -unique pages, with the same insight landing at up to 3 different paths across -up to 8 PRs. Per-PR review would re-import those duplicates, so — as with the -#6–#13 consolidation — this branch carries the reconciled end-state and the 15 -PRs are closed in its favor. +Queue: `~/.dev-loop/queue/` held 5 pending rows across 2 session files +(`dc029c0c…` — 2 qa rows from the `fix-i55-modeb` worktree; `f1a3ae46…` — 3 +infrastructure rows from the `linkly` orchestration session). ## Verified best-practice -Every adopted page's sources were carried from its originating PR's flush, where -they were live-verified at flush time; no new URLs were introduced during -consolidation (checked mechanically: every `http(s)` URL in every merged page -appears in a source PR's diff; every added body line in amended pages traces to -a source PR hunk — orphan-line verification). Confidence fields were kept as the -originating flushes set them, except client-side-rate-limiting where the union -of provider-doc citations (Okta, Auth0, GitHub, OpenAI, RFC 6585) supports -`verified` for the load-bearing claims. One subagent's fabricated content (12 -files matching neither main nor any PR, with invented source URLs) was detected -by the same verification and replaced with true PR content. +### 1. A doc gate over a copied table must resolve the owning code constant → `verified` + +**Claim.** When a document's table restates values owned by code (a severity map, +a schema enum, a config default), the gate must compare each cell against the +owning symbol. A gate that asserts the invariant the table itself states ("all +five are warnings") uses the document as both subject and oracle, so it stays +green after the constant moves — the check built to catch drift is what pins the +stale claim. + +**How verified — local reproduction, 2026-08-08, `linkly` @ `fix/rerun-gaps`.** +Resolved every row of `docs/ENFORCEMENT-MATRIX.md` §C against +`lnpl.diagnostics.SEVERITY_OF` and printed the pairs: + +``` +ok unknown-verb: documented='warning' code='warning' +MISMATCH declared-not-enforced: documented='warning' code='info' +MISMATCH declared-measured-only: documented='warning' code='info' +MISMATCH authorization-not-verified: documented='warning' code='info' +ok guard-skipped-steps: documented='warning' code='warning' +``` + +The document's severity column reads `warning` in all five rows and its prose +says so outright ("전부 `warning`이고"), so the self-referential form of the check +passes on exactly this pair; the constant grades three of them `info`. The same +column in the plugin's `references/declarations.md`, which is *generated* from the +constant, agreed — the drift is specific to the hand-maintained copy. The +repo's own `impl/tests/test_plugin_references.py:96` already records the +corrected form ("against `SEVERITY_OF` itself, which is the only comparison +that…"), which is independent confirmation of the directive rather than of my +reproduction alone. + +**Sources checked.** +- https://google.github.io/styleguide/docguide/best_practices.html — "Change your + documentation in the same CL as the code change"; where a fact lives elsewhere, + "Link to it instead" of restating it. Supports the external-agreement axis as + the enforcement mechanism when a table restates the fact anyway. (Fetched + 2026-08-08.) +- Already on the target page and re-used, not re-fetched: + https://testing.googleblog.com/2021/04/mutation-testing.html (a check is + measured by whether a planted defect makes it fail), + https://eslint.org/docs/latest/extend/custom-rule-tutorial (a checker needs a + must-pass and a must-fail case). + +### 2. Re-capture CLI transcripts at paste time; give each stream its own block → `verified` + +**Claim.** A transcript pasted into a document must be re-captured from the +current tree at paste time, and stdout/stderr must appear as two labelled blocks. +A `2>&1` capture records the environment's flush order, not the program's write +order. + +**How verified — local reproduction, 2026-08-08, CPython 3.13 / macOS.** One +program printing `STDOUT-1, STDERR-1, STDOUT-2, STDERR-2` in that order, captured +three ways: + +| Capture | Recorded order | +|---------|----------------| +| `python prog.py 2>&1 \| cat` | `STDERR-1, STDERR-2, STDOUT-1, STDOUT-2` — **reversed** | +| `python -u prog.py 2>&1 \| cat` | program order | +| both streams on a pty (`pty.openpty()`) | program order | + +One unchanged binary, three environments, two different documented orders — so a +merged block cannot support an ordering claim in either direction. + +**Sources checked (all fetched 2026-08-08).** +- https://pubs.opengroup.org/onlinepubs/9799919799/functions/stdin.html — "When + opened, `stderr` shall not be fully buffered"; "`stdout` shall be fully buffered + if and only if the file descriptor associated with the stream is determined not + to be associated with an interactive device." This is the mechanism behind the + table above. +- https://docs.python.org/3/using/cmdline.html — `-u`: "Force the stdout and + stderr streams to be unbuffered"; `PYTHONUNBUFFERED` "is equivalent to + specifying the `-u` option." +- https://docs.python.org/3/library/doctest.html — the executable-transcript + mechanism ("executes those sessions to verify that they work exactly as + shown"), and the constraint that makes stream separation matter for gating: + "Output to stdout is captured, but not output to stderr." +- https://google.github.io/styleguide/docguide/best_practices.html — same-change + documentation updates, applied to transcripts as capture-at-paste-time. + +The staleness half is `field-tested` in origin (RFC-0022's `build --run` +transcript omitted a `validation-sample-derived` block added two tasks after +capture, with no elision mark) and is recorded in the page as a dated field +incident, distinct from the doc-sourced buffering half. + +### 3–5. Three orchestration candidates → not verified here, dropped as pending duplicates + +See **Open-PR check**. No confidence upgrade was applied to any of them; they were +retired from the queue unchanged because open PRs already carry them in equal or +better form. ## Existing-layer check -- Merged-main near-dup scan before consolidation: pairwise Jaccard over - title + "When this applies" across all 141 merged pages → **0 flagged pairs**; - previously merged content carries no duplication. -- Cross-PR dedup during consolidation: 10 duplicate clusters collapsed to one - canonical page each (rate limiting 8→1, call-site enumeration 7→folded into - the canonical merged in #20, stderr/exit-0 diagnostics 4→1, sysroot 2→1, - env-off-switch 2→1, completion predicates 2→1, robots.txt 2→1, - harness-mediated results 2→1, leaked artifacts 2→1, orchestration category - naming unified). Three near-pairs kept distinct after trigger comparison, - with mutual `related:` links (differential setup vs interpretation; expansion - semantics vs off-switch design; import-time tactics vs level choice). -- 24 existing pages received union-merged amendments; additions already present - in main (from #16/#20) were skipped, and all non-canonical `related:` ids - were remapped to canonical page ids (post-merge broken-link scan: 0). +Routed via `INDEX.md` → `qa` ("automated verification of document deliverables") +and `infrastructure` ("multi-agent orchestration"), then read both domain indexes +and every page whose "load when" line overlapped. + +Pages read: qa-document-verification-spec-document-gates, +qa-document-verification-editing-a-gated-document, +qa-deliverables-generated-artifacts-as-deliverable-source, +testing-quality-tests-that-cannot-fail, +platforms-processes-tool-diagnostics-without-a-failing-exit-code, +platforms-processes-non-interactive-cli-invocation, +infrastructure-agent-orchestration-worktree-isolated-workers, +infrastructure-agent-orchestration-pane-delivery-confirmation, +infrastructure-agent-orchestration-control-signals-vs-primary-artifacts, +infrastructure-agent-orchestration-shared-run-state + +**Insight 1 — merged, not created.** `spec-document-gates` already owns the +trigger (automated checks deciding whether a spec document meets its +requirements) and already has a four-axis table. Its `Cross-reference` axis stops +at the document boundary — "assert that a statement in one section implies its +counterpart elsewhere, and recompute a derived value from its inputs" — so a +doc-vs-code comparison had no axis. Added a fifth axis row (`External +agreement`), four edge-case rows (summary invariants over a copied column; +checks that restate the expected values as their own literals; docs-only CI where +the owning code cannot be imported; a row present on one side only), one +`Instead of` row, one source, and the dated reproduction. **No conflict** — the +addition extends the axis table rather than contradicting any existing row. +`confidence` deliberately left at `field-tested`: the page's older four-axis +content is field-distilled, and the page-level field takes the lower of the two +rather than being upgraded on the strength of the new section alone. + +**Insight 2 — new page.** No existing page carries the trigger. Closest +neighbours and why each is distinct: +- `generated-artifacts-as-deliverable-source` — hand-writing a document the repo + already generates. A transcript is captured, not generated; overlaps only in + the "re-run the generator" edge case, which the new page defers to it. +- `tool-diagnostics-without-a-failing-exit-code` — how a *harness* captures + stderr from a tool that exits 0. Same streams, different consumer (a gate, not + a reader), and it does not cover documents. +- `non-interactive-cli-invocation` — a TTY-detecting tool changing its output + format under automation. The new page cites it from the edge-case table rather + than restating it. +- `qa-deliverables-quantitative-claims-in-a-published-document` (in open PR #51, + not merged) — numbers in a published document. Same drift mechanism, different + claim shape; **flagged here rather than merged into**, because it does not exist + on `main` and cross-PR edits are what caused the #17–#40 pile-up. If #51 lands + first, the two pages should gain reciprocal `related:` links; neither + contradicts the other. + +Reciprocal `related:` links added both ways on +`generated-artifacts-as-deliverable-source`, +`tool-diagnostics-without-a-failing-exit-code`, and +`non-interactive-cli-invocation`; the qa index "load when" line for +`spec-document-gates` was widened to name the new axis so index and trigger stay +in agreement. + +Lint (changed-page pass): 0 errors, 0 warnings — sources-vs-confidence, +prohibitions outside `Instead of`, related/inline id resolution, index presence +and trigger agreement, vague qualifiers, body length (62 and 85 body lines). + +## Open-PR check + +Listed with `gh pr list --repo choiyounggi/dev-loop --state open --search +"head:knowledge/"` — 12 open heads (#47, #49, #50, #51, #52, #55, #56, #57, #58, +#61, #62, #64). Fetched and diffed the three whose titles touched orchestration +or document gating. + +| Candidate | Overlapping head | Verdict | +|-----------|------------------|---------| +| 1 — doc-as-spec gate vs code constant | none (checked #51, #56, #58, #61 — #51 touches qa/deliverables, not document-verification) | **new** | +| 2 — CLI transcripts, re-capture + split streams | none (#51 adds `quantitative-claims-in-a-published-document`, a different claim shape — see Existing-layer check) | **new** | +| 3 — `worktree_escape` fires `ask` on read-only cross-worktree reads; budget the escalation round trip | #51 `knowledge/dch0202-20260806-183029` | **drop** | +| 4 — Orca dispatch-binding taxonomy (`runtime_unavailable` vs `agent_unconfigured`; pass the worktree with the pane) | #51 same head | **drop** | +| 5 — tmux worker wedged on a numbered in-band chooser | #64 `knowledge/choiyounggi-20260808-004155` | **drop** | + +Evidence for the three drops, from `git diff origin/main origin/ -- wiki/`: + +- #51's `worktree-isolated-workers` hunk already carries the candidate's directive + nearly verbatim — "Budget the round trip (read the recorded escalation → + approve → clear `escalations/` → restart the watcher) and state in the worker's + first brief that reads are approved and only writes outside the worktree are + refused" — plus three rows my candidate did not have (sibling-worktree path + survives the strip; a write verb anywhere on the line fires independently of + what is read; redirect-to-absolute fires regardless), backed by a fuller + reproduction against guardrails 1.2.0. Nothing unique to push. +- #51's `pane-delivery-confirmation` hunk carries the full taxonomy, including the + `terminal_worktree_mismatch` resolution and the "a failed unit is replaced, not + retried in place" consequence. Nothing unique to push. +- #64's new `unattended-worker-questions` page supersedes candidate 5: it adds the + classify-by-terminal-tail table, the allowlisted-key protocol with the + selection-then-confirmation caveat, the re-send-the-in-flight-prompt step, and + the out-of-band question channel the candidate only gestured at. Nothing unique + to push. + +This trio has now re-entered the queue on several consecutive flushes. Merging +#51 and #64 retires it at the source. ## Routing decision -- New categories: `infrastructure/agent-orchestration` (5 pages; unified the - competing `orchestration`/`agent-orchestration` names), `databases/data-survey` - (1), `qa/deliverables` (1). All other pages route into existing categories. -- Canonical-path decisions: rate limiting → `backend/common/reliability/` - (sits beside timeouts-and-retries; 6 of 8 variants chose it); stderr - diagnostics → `platforms/processes/` (concern spans beyond shells); leaked - artifacts → `testing/data/artifact-leakage-from-a-suite`; call-site - enumeration → the existing `backend/common/change-impact/` page. -- All 38 new pages listed in their domain indexes (nearest-index rule; backend - routes via its python sub-index for bytecode-cache-staleness); INDEX.md domain - summaries updated for infrastructure/qa/databases. Full-wiki lint: frontmatter, - ids, related-links, index coverage, size, qualifiers, staleness → 0 findings. +| Insight | Domain / category / page | New category? | +|---------|--------------------------|---------------| +| 1 — doc gate resolves the owning code constant | `qa` / `document-verification` / **merged into** `spec-document-gates.md` (5th axis + 4 edge cases + 1 `Instead of` + source + reproduction) | no — merge-before-create; the page owns the trigger and the addition is a new axis on its existing table | +| 2 — CLI transcripts in a document | `qa` / `deliverables` / **new** `command-transcripts-in-a-document.md` | no new category — `deliverables` already covers "documents produced for a reader, and where their content comes from"; this is the captured-run source alongside the generated-artifact source | + +Why `qa/deliverables` and not `platforms/processes` for insight 2: the buffering +mechanism is a platforms fact, but the decision the page governs is what goes into +a document. `platforms` is routed to when code or scripts break across machines; +the transcript rule applies when nothing is broken and a document is being +written. The mechanism is cited from POSIX and the two adjacent `platforms/processes` +pages are cross-linked both ways rather than duplicated. + +Nothing was left `unverified`. Both promoted insights carry a dated local +reproduction plus fetched official-doc citations; the staleness half of insight 2 +is labelled as a dated field incident inside the page rather than presented as +doc-sourced. diff --git a/log.md b/log.md index c930fc2..f3cc96d 100644 --- a/log.md +++ b/log.md @@ -43,3 +43,6 @@ Append-only. Format: `## [YYYY-MM-DD] &1` capture records the environment's flush order, not the program's write order; local three-way reproduction on CPython 3.13/macOS: through a pipe the merged order was all-stderr-then-all-stdout, under `python -u` and on a pty it was the program's order, one unchanged binary; POSIX stdin.html — stderr "shall not be fully buffered", stdout fully buffered iff not on an interactive device; doctest captures stdout but not stderr). Merged: qa/document-verification/spec-document-gates +5th "External agreement" axis / +4 edge cases / +1 Instead-of (when a documented table's cells are copies of a code constant, resolve the owning symbol and compare per row — asserting the invariant the table itself states makes the document its own oracle, so the gate built to catch drift is what pins the stale claim green; reproduced 2026-08-08 in `linkly`: `docs/ENFORCEMENT-MATRIX.md` §C reads `warning` in all five rows while `SEVERITY_OF` grades three `info`). Reciprocal related-links added on generated-artifacts-as-deliverable-source, tool-diagnostics-without-a-failing-exit-code, non-interactive-cli-invocation. +## [2026-08-08] dedup | Dropped 3 orchestration candidates as pending duplicates of open PRs, not re-ingested: worktree_escape fires `ask` on read-only cross-worktree access and the coordinator must budget the escalation round trip → already in #51's worktree-isolated-workers hunk (identical directive, richer reproduction); the Orca dispatch-binding taxonomy (runtime_unavailable = occupied turn tail → wait and rebind; agent_unconfigured = dead agent → replace; always pass the worktree with the pane) → already in #51's pane-delivery-confirmation hunk; a tmux worker wedged on a numbered in-band chooser → already in #64's unattended-worker-questions (classify by terminal tail, unblock by allowlisted key, re-send the in-flight prompt). Merging #51 and #64 retires this recurring trio. +## [2026-08-08] lint | 0 errors fixed, 0 reported — changed-page pass over qa/deliverables/command-transcripts-in-a-document (new, 62 body lines, verified with 4 cited sources), qa/document-verification/spec-document-gates (85 body lines), and the three pages that gained reciprocal related-links. Checks run: sources-vs-confidence, prohibition-outside-Instead-of, related/inline id resolution, domain-index presence and trigger agreement, vague qualifiers, body length. diff --git a/wiki/platforms/processes/non-interactive-cli-invocation.md b/wiki/platforms/processes/non-interactive-cli-invocation.md index 2315770..e3df85d 100644 --- a/wiki/platforms/processes/non-interactive-cli-invocation.md +++ b/wiki/platforms/processes/non-interactive-cli-invocation.md @@ -12,7 +12,7 @@ sources: - https://man7.org/linux/man-pages/man1/timeout.1.html - https://en.wikipedia.org/wiki/Bracketed-paste last_verified: 2026-08-04 -related: [platforms-processes-background-services, platforms-tools-bsd-vs-gnu-cli, platforms-shells-portable-shell-scripts, debugging-methodology-hypothesis-testing, platforms-processes-parsing-cli-structured-output, platforms-shells-env-var-off-switches, infrastructure-agent-orchestration-pane-delivery-confirmation] +related: [platforms-processes-background-services, platforms-tools-bsd-vs-gnu-cli, platforms-shells-portable-shell-scripts, debugging-methodology-hypothesis-testing, platforms-processes-parsing-cli-structured-output, platforms-shells-env-var-off-switches, infrastructure-agent-orchestration-pane-delivery-confirmation, qa-deliverables-command-transcripts-in-a-document] --- # Invoking a Prompt-Capable CLI from a Script or Agent Harness diff --git a/wiki/platforms/processes/tool-diagnostics-without-a-failing-exit-code.md b/wiki/platforms/processes/tool-diagnostics-without-a-failing-exit-code.md index bc65b52..5d653ee 100644 --- a/wiki/platforms/processes/tool-diagnostics-without-a-failing-exit-code.md +++ b/wiki/platforms/processes/tool-diagnostics-without-a-failing-exit-code.md @@ -10,7 +10,7 @@ sources: - https://www.gnu.org/software/bash/manual/bash.html#Redirections - https://code.claude.com/docs/en/hooks last_verified: 2026-08-06 -related: [platforms-processes-non-interactive-cli-invocation, platforms-shells-command-text-inspected-before-execution, testing-quality-checks-that-cannot-pass] +related: [platforms-processes-non-interactive-cli-invocation, platforms-shells-command-text-inspected-before-execution, testing-quality-checks-that-cannot-pass, qa-deliverables-command-transcripts-in-a-document] --- # Feeding a Tool's Warnings Back When It Exits 0 diff --git a/wiki/qa/deliverables/command-transcripts-in-a-document.md b/wiki/qa/deliverables/command-transcripts-in-a-document.md new file mode 100644 index 0000000..31dfeae --- /dev/null +++ b/wiki/qa/deliverables/command-transcripts-in-a-document.md @@ -0,0 +1,82 @@ +--- +id: qa-deliverables-command-transcripts-in-a-document +domain: qa +category: deliverables +applies_to: [general] +confidence: verified +sources: + - https://pubs.opengroup.org/onlinepubs/9799919799/functions/stdin.html + - https://docs.python.org/3/using/cmdline.html + - https://docs.python.org/3/library/doctest.html + - https://google.github.io/styleguide/docguide/best_practices.html +last_verified: 2026-08-08 +related: + [ + qa-deliverables-generated-artifacts-as-deliverable-source, + qa-document-verification-spec-document-gates, + platforms-processes-tool-diagnostics-without-a-failing-exit-code, + platforms-processes-non-interactive-cli-invocation, + ] +--- + +# A Pasted CLI Transcript Inside a Document + +## When this applies + +You are putting a captured command run — an RFC's worked example, a README +quickstart, a design doc's "here is what it prints" — into a document, and the +transcript shows both the program's output and its diagnostics. Also when a +reviewer re-runs a documented example and gets a different order or extra lines. + +## Do this + +1. **Re-capture the transcript from the current tree immediately before the + paste lands.** A transcript captured earlier in the work describes the build + at capture time; anything added since is missing from a block presented as a + measurement. Capture and paste in the same step, and record the commit and + the exact command line above the block. +2. **Give each stream its own block, labelled.** Capture them separately — + `cmd >out.txt 2>err.txt` — and paste `out.txt` and `err.txt` as two blocks. + The document then states what the program produced on each stream, which is a + property of the program. +3. **When the document's point is the ordering** (a diagnostic must appear + before/after a result), state the ordering as a sentence about the streams + and show the two blocks, rather than showing one merged block as the proof. +4. **Force unbuffered output for any transcript you do capture merged** + (`python -u`, `PYTHONUNBUFFERED=1`, `stdbuf -o0 -e0`) and say in the document + that it was captured that way, so a reader who re-runs it plainly and sees a + different order knows why. +5. **Make the transcript executable where the toolchain has a mechanism for it** + — `doctest`, rustdoc doctests, a snippet-extraction test. An executed + transcript cannot go stale silently. Note that `doctest` checks stdout only: + "Output to stdout is captured, but not output to stderr", so a merged block + is not checkable by it either. + +## Edge cases + +| Case | Then | +|------|------| +| The tool writes everything to one stream by design | Say so in the document and show one block; the rule is about not *merging* two streams, not about splitting one | +| The tool changes its output when it is not attached to a terminal (colour, progress bars, plain-vs-rich format) | Capture the form your reader will get, name which one it is, and keep the capture command in the document ([platforms-processes-non-interactive-cli-invocation]) | +| The transcript is long and you abridge it | Mark the elision (`…`) at the point it happens; an unmarked cut reads as complete output and is what makes a later-added block go unnoticed | +| The output contains timings, temp paths, or ids that change per run | Replace them with a stable placeholder and say the substitution was made, so a re-runner diffs the shape rather than the values | +| The document is generated from the run | Re-run the generator instead of editing the block ([qa-deliverables-generated-artifacts-as-deliverable-source]) | +| A gate greps the transcript block | Anchor it on the stream label and the command line, not on line adjacency between the two streams ([qa-document-verification-spec-document-gates]) | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Paste a `cmd 2>&1` capture as the record of what the program prints | Capture the two streams separately and paste two labelled blocks | With stdout to a pipe the C runtime makes it fully buffered while stderr is not, so the merged order is the environment's flush order, not the program's write order | +| Reuse a transcript captured earlier in the same piece of work | Re-run and re-capture at paste time | Output added between capture and paste is absent from a block the document presents as measured, and nothing in the block shows the omission | +| Argue an ordering claim ("the diagnostic comes after the result") from a merged block | State the claim about the streams and show them separately | The same binary produces both orders depending on whether stdout is a pipe or a tty — the merged block cannot support either claim | +| Reproduce the transcript once by hand and call it verified | Re-run it under the reader's conditions (piped, unbuffered, and on a tty) and reconcile | A single capture cannot distinguish a program-order claim from a buffering artefact | + +## Sources + +- https://pubs.opengroup.org/onlinepubs/9799919799/functions/stdin.html — "When opened, `stderr` shall not be fully buffered"; "`stdout` shall be fully buffered if and only if the file descriptor associated with the stream is determined not to be associated with an interactive device" — redirecting a transcript into a pipe or file changes stdout's buffering and therefore the interleaving, while stderr's is unchanged +- https://docs.python.org/3/using/cmdline.html — `-u`: "Force the stdout and stderr streams to be unbuffered"; `PYTHONUNBUFFERED` "is equivalent to specifying the `-u` option" +- https://docs.python.org/3/library/doctest.html — doctest "executes those sessions to verify that they work exactly as shown" (the executable-transcript mechanism), and "Output to stdout is captured, but not output to stderr" +- https://google.github.io/styleguide/docguide/best_practices.html — "Change your documentation in the same CL as the code change" — the capture-at-paste-time rule applied to transcripts +- Local reproduction 2026-08-08 (CPython 3.13, macOS), one program printing `STDOUT-1, STDERR-1, STDOUT-2, STDERR-2` in that order: through a pipe (`2>&1 | cat`) the merged capture came out `STDERR-1, STDERR-2, STDOUT-1, STDOUT-2` — stdout fully buffered, flushed at exit; the same command with `python -u` gave the program's order; and with both streams on a pty (`pty.openpty()`) it also gave the program's order. Three environments, two different documented orders, one unchanged program +- Field incident 2026-08-07 (`linkly`, RFC-0022 draft): a `build --run` transcript stated the reverse of the real diagnostic/output ordering and omitted a `validation-sample-derived` block added two tasks after the capture, with no elision mark. A reviewer re-running under `python -u` and a pty found both; rewritten with the streams separated, all four transcripts in the RFC then matched their re-runs byte for byte diff --git a/wiki/qa/deliverables/generated-artifacts-as-deliverable-source.md b/wiki/qa/deliverables/generated-artifacts-as-deliverable-source.md index 96aa588..5b2678c 100644 --- a/wiki/qa/deliverables/generated-artifacts-as-deliverable-source.md +++ b/wiki/qa/deliverables/generated-artifacts-as-deliverable-source.md @@ -10,7 +10,7 @@ sources: - https://google.github.io/styleguide/docguide/best_practices.html — keep documentation sources close to the code they describe so they change together last_verified: 2026-08-04 related: - [frontend-state-derived-state, qa-document-verification-spec-document-gates] + [frontend-state-derived-state, qa-document-verification-spec-document-gates, qa-deliverables-command-transcripts-in-a-document] --- # A Requested Deliverable Duplicating an Artifact the Repo Already Generates diff --git a/wiki/qa/document-verification/spec-document-gates.md b/wiki/qa/document-verification/spec-document-gates.md index 9c2fea6..06dfeff 100644 --- a/wiki/qa/document-verification/spec-document-gates.md +++ b/wiki/qa/document-verification/spec-document-gates.md @@ -13,13 +13,15 @@ sources: - https://docs.vale.sh/checks/occurrence - https://github.com/DavidAnson/markdownlint/blob/main/doc/md056.md - https://github.com/DavidAnson/markdownlint/issues/1206 -last_verified: 2026-07-30 + - https://google.github.io/styleguide/docguide/best_practices.html +last_verified: 2026-08-08 related: [ qa-process-acceptance-criteria, testing-quality-tests-that-cannot-fail, platforms-environment-unicode-text-matching, qa-deliverables-generated-artifacts-as-deliverable-source, + qa-deliverables-command-transcripts-in-a-document, ] --- @@ -51,6 +53,7 @@ requirement unmet; you are fixing gate patterns for a document that is not writt | Modality and polarity | Within one sentence scope, assert the requirement is neither negated nor demoted (MUST→SHOULD, 필수→권장/원칙적으로) | "X is not required" and "X is recommended" both contain every keyword | | Set completeness | Assert the exact member count of a closed set (`enum has exactly 5 rows`) | One enum row is deleted; the token count is still ≥ 1 | | Cross-reference | Assert that a statement in one section implies its counterpart elsewhere, and recompute a derived value from its inputs | Two sections disagree, or an `Examples` block silently stands in for the deleted normative rule | +| External agreement | For a cell that is a copy of a value owned outside the document (a code constant, schema enum, config default), resolve the owning symbol and compare cell against symbol, per row | Every cell still matches the document's own summary sentence while the owning constant has moved | 4. **Fail closed when the anchor is missing.** When the section heading, table, or derivation input a check needs cannot be located, report FAIL. A check that @@ -69,6 +72,10 @@ requirement unmet; you are fixing gate patterns for a document that is not writt | The check counts delimiters (`\|`) as a stand-in for parsing | Use a Markdown parser — markdownlint's own MD056 misreports when a pipe appears inside backticks (issue #1206) | | The document deliberately relaxes a requirement | Change the gate and the acceptance criteria in the same commit, and record the relaxation in the PR ([qa-process-acceptance-criteria]) | | The deliverable is code, not a document | Apply [testing-quality-tests-that-cannot-fail] — same red-run proof, expressed as tests | +| The document states a summary invariant over a copied column ("all five are warnings", "every endpoint is POST") | Assert the invariant against the owning constant, then assert each cell against it. Asserting the invariant over the parsed column compares the document to itself and holds whatever the code now does | +| The check restates the expected values as its own literals | Import or read the owning symbol and iterate it. A literal list is a third copy that drifts with the other two, and it makes the check pass when the constant changes but the document does not | +| The check must run where the owning code cannot be imported (docs-only CI job) | Have the code-side job emit the constant as a data file and diff the document against that file; keep the emit step in the same pipeline as the code | +| A row exists in the document but not in the constant (or vice versa) | FAIL naming the direction — a removed constant leaves a documented row nothing checks, which reads as covered | ## Instead of @@ -79,6 +86,7 @@ requirement unmet; you are fixing gate patterns for a document that is not writt | Assert a token appears at least once for a closed enum | Parse the table and assert the exact row count | Deleting one member leaves the count ≥ 1, so the gate stays green on an incomplete set | | Skip a check whose anchor sentence was not found | Report FAIL and name the missing anchor | A skipped check is indistinguishable from a passed one in the summary line | | Verify a cross-section value by matching the number as written | Recompute it from its inputs and compare | Matching the written number passes when both sections were edited to the same wrong value | +| Assert a documented table satisfies the property the table itself claims | Resolve the owning constant and compare the table's cells to it, row by row | The document is both subject and oracle, so the check is a tautology; when the constant moves, the gate built to catch drift is what pins the stale claim green | ## Sources @@ -90,6 +98,8 @@ requirement unmet; you are fixing gate patterns for a document that is not writt - https://docs.vale.sh/checks/occurrence — enforces the minimum/maximum number of times a token appears "in a given scope" — count and scope, not bare existence - https://github.com/DavidAnson/markdownlint/blob/main/doc/md056.md — MD056 flags tables whose rows disagree with the header's column count (structural table checking) - https://github.com/DavidAnson/markdownlint/issues/1206 — MD056 counts pipes inside backticks as separators: a delimiter count is not a parse +- https://google.github.io/styleguide/docguide/best_practices.html — "Change your documentation in the same CL as the code change"; and where a fact lives elsewhere, "Link to it instead" of restating it — the external-agreement axis is what enforces that when a table restates the fact anyway +- Local reproduction 2026-08-08 (`linkly`, `docs/ENFORCEMENT-MATRIX.md` §C vs `lnpl.diagnostics.SEVERITY_OF`): the document's severity column reads `warning` in all five rows and its summary sentence says so, while the constant grades three of them `info`. A gate asserting "every documented code is a warning" passes on this pair; resolving each row against `SEVERITY_OF` returns three mismatches (`declared-not-enforced`, `declared-measured-only`, `authorization-not-verified`: documented `warning`, code `info`). The repo's `references/declarations.md`, whose same column is generated from the constant, agreed ## Field context diff --git a/wiki/qa/index.md b/wiki/qa/index.md index 2278dfd..f14af2e 100644 --- a/wiki/qa/index.md +++ b/wiki/qa/index.md @@ -24,12 +24,13 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [generated-artifacts-as-deliverable-source](deliverables/generated-artifacts-as-deliverable-source.md) | Asked to produce a document (ERD, schema reference, API surface list, dependency inventory) for a hand-off, review, or external partner when the repo already generates that content from code; deciding whether to re-run a stale generator or hand-write the deliverable; a hand-written reference document disagrees with the live system (checks that gate a document → document-verification) | +| [command-transcripts-in-a-document](deliverables/command-transcripts-in-a-document.md) | Pasting a captured CLI run into an RFC, README, or design doc as a worked example; the transcript shows program output and diagnostics together; a reviewer re-running a documented example gets a different order or extra lines; a documented example is being used to argue that one line appears before another | ## document-verification | Page | Load when | |------|-----------| -| [spec-document-gates](document-verification/spec-document-gates.md) | Writing or reviewing automated checks (grep/script) that decide whether a spec/RFC/schema document meets its requirements; a document passed its checklist but the requirement is still unmet; choosing what a doc gate must assert beyond keyword presence (table structure, MUST-vs-SHOULD demotion, closed-set completeness, cross-section consistency); validating a gate pattern for a document that does not exist yet | +| [spec-document-gates](document-verification/spec-document-gates.md) | Writing or reviewing automated checks (grep/script) that decide whether a spec/RFC/schema document meets its requirements; a document passed its checklist but the requirement is still unmet; choosing what a doc gate must assert beyond keyword presence (table structure, MUST-vs-SHOULD demotion, closed-set completeness, cross-section consistency, agreement with a code constant the document copies); a gate over a table asserts the property the table itself claims; validating a gate pattern for a document that does not exist yet | | [editing-a-gated-document](document-verification/editing-a-gated-document.md) | Editing or rewording a document that grep/regex gates or a lint config check; a gate fails on wording whose meaning did not change; describing what an upstream spec says without tripping a "do not redefine it" gate; a check matches the pattern your own document quotes; recording an audit verdict inside the document that was audited; deciding which checks to re-run after editing a gated document | ## environments