diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index c375750..db5803f 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,95 +1,204 @@ -# Consolidated review — knowledge PRs #6–#13 +# Knowledge flush — 4 insight(s) -Eight fork PRs (`dch0202-rsquare`, 2026-07-28 → 2026-08-02) were reviewed together -against `AGENTS.md`. Each PR was audited by an independent reviewer (format rules, -sources, vague-qualifier ban, ≤120 body lines, index/log invariants), then -cross-compared to catch duplication the per-PR flushes could not see — they branched -independently off the same main and rewrote the same shared index/log files. Fork -branches can't be edited from here and several PRs needed content changes (drop a -duplicate, merge a colliding page), so this branch carries the reconciled end-state -rather than merging each PR as-is (which would import the duplicates). +Drained 4 pending candidates from `~/.dev-loop/queue`. All four survived +research and are ingested: **4 new pages, 5 amendments.** Three are +`confidence: verified`; one is `field-tested` and says so (see below). + +| # | Insight | Target page | Confidence | +|---|---------|-------------|-----------| +| 1 | Duration assertions against an injected float clock | `testing/quality/injected-clock-duration-assertions.md` (new) | verified | +| 2 | A path-valued config key must be absolute | `infrastructure/config/path-valued-config.md` (new) | verified | +| 3 | Enumerate call sites by callee, not parameter name | `backend/common/refactoring/call-site-enumeration.md` (new) | verified | +| 4 | Sharpen an artifact guard from shape to consequence | `testing/quality/guard-shape-vs-consequence.md` (new) | field-tested | + +--- ## Verified best-practice -Sources are per-page and were live-verified in each originating PR's flush; the -independent re-reviews re-checked them. Landed pages and their evidence base: - -| Page | Confidence | Source basis | -|------|-----------|--------------| -| backend/common/llm/completion-response-validation | verified | OpenAI reasoning guide + chat `object` spec (5 `finish_reason` values), vLLM/LiteLLM reasoning fields; field incident (200/`length`/empty content/8,173-char reasoning) | -| backend/common/llm/context-window-budget | verified | Claude context-window docs, LiteLLM exception mapping, vLLM/Claude Code env-var docs | -| backend/common/integrations/externally-owned-defaults | verified | OpenAI deprecations (notice windows) + models `list`, LiteLLM model_discovery; field incident (alias removed between PR verify and review → 400) | -| backend/common/storage/object-key-persistence | verified | AWS S3 CompleteMultipartUpload + managed-upload API/source, aws-sdk-js issues #1158/#5656 | -| infrastructure/containers/host-cgroup-visibility | field-tested | cgroup_namespaces(7), Docker `--cgroupns=host`, nsenter, k8s #103363; OrbStack repro | -| infrastructure/observability/missing-container-metrics | verified/field-tested | k8s resource-metrics-pipeline docs, kube-prometheus-stack values, kubernetes-mixin; OrbStack #2217 repro | -| platforms/environment/unicode-text-matching | verified | UAX #15, Unicode core §3.12, APFS FAQ, POSIX grep; local repro (macOS 15/APFS, grep 2.6.0-FreeBSD, Python 3.13) | -| platforms/shells/command-text-inspected-before-execution | verified | Claude Code hooks docs, POSIX shell §2.6; local reproduction | -| platforms/processes/non-interactive-cli-invocation | verified | GNU nohup, OpenBSD ssh/ssh_config, git, timeout man pages; no-request-in-gateway-log field incident | -| qa/document-verification/spec-document-gates | field-tested | ESLint, Google mutation testing, RFC 2119, Vale, markdownlint; 32/32 mutant / 62/62 intact RFC sessions | -| qa/document-verification/editing-a-gated-document | field-tested | pgrep, Vale, markdownlint; in-house editing methodology | -| testing/quality/checks-that-cannot-pass | verified | James Shore AoAD2, POSIX grep exit status, Semgrep rule-testing, pytest exit codes; BSD/ugrep measurement | -| testing/quality/spec-artifact-checks | verified | JSON Schema, ESLint RuleTester, pitest, GFM table spec; local cell-count repro + GitHub renderer cross-check | -| testing/quality/harness-reverse-controls | verified | mutation-testing + CI-control sources; field repro (re-fetched all cited URLs, PASS) | - -Three pages were reconciled from two overlapping PR versions each, keeping the more -complete/better-sourced body and folding in the other's unique cases: -- **completion-response-validation** — #12 body (all five `finish_reason` values, - `tool_calls`/`function_call` carve-out, streaming, Responses API, "reasoning is - scratch, not deliverable") kept in `llm/` (coherent with #6/#13); folded in #6's - DeepSeek first-party edge + the field incident. -- **externally-owned-defaults** — #12 generalized body (any repo-external resource) - in `integrations/`; folded in #6's alias-removed field incident + the - gateway-config-vs-live-upstream nuance. -- **non-interactive-cli-invocation** — #12 body (GNU-nohup extension precision, - ssh -n stdin-detach vs BatchMode, pre-log DNS/TLS/proxy + `curl -v`) kept; folded - in #11's DEBIAN_FRONTEND, pager/color TTY case, wrapper-CLI case, field incident. +### 1 — Duration assertions against an injected float clock + +**Claim as queued:** with a fake monotonic clock started at a large value (e.g. +`1000.0`), `assert gap >= interval` fails on correct code; add a small tolerance +or start the clock at `0.0`. + +**Verified — and the claim was too narrow.** Reproduced locally (CPython 3.14.6, +macOS) across start values for a `1.05` step: + +| start | `start + 1.05 - start` | `>= 1.05`? | +|-------|------------------------|-----------| +| `0.0` | `1.05` | yes | +| `1.0` | `1.0499999999999998` | no | +| `100.0` | `1.0499999999999972` | no | +| `1000.0` | `1.0499999999999545` | no | +| `1e6` | `1.0500000000465661` | yes (rounded **up**) | +| `1e9` | `1.0499999523162842` | no | + +Two corrections folded into the page: a start of `1.0` already breaks the exact +comparison (not just "large" values), and the error is **not always negative** — +at `1e6` the gap came out larger than the interval, so equality and upper-bound +assertions need tolerance on both sides. Every listed start satisfies +`gap >= 1.05 - 1e-6`. + +**Sources checked:** +- [PEP 564](https://peps.python.org/pep-0564/) — the strongest confirmation of + the "start at 0.0" branch, in CPython's own words: *"Internally, Python starts + `monotonic()` and `perf_counter()` clocks at zero on some platforms which + indirectly reduce the precision loss."* Also *"the `float` type starts to lose + nanoseconds after 104 days."* +- [Python floating-point tutorial](https://docs.python.org/3/tutorial/floatingpoint.html) + — *"most decimal fractions cannot be represented exactly as binary fractions."* +- [`math.isclose`](https://docs.python.org/3/library/math.html#math.isclose) — + signature `(a, b, *, rel_tol=1e-09, abs_tol=0.0)`; **symmetric**, which is why + the page routes it to the equality row only: on a lower bound `isclose` also + accepts a gap that is too *short*, the exact defect a rate-limit test guards. +- [`pytest.approx`](https://docs.pytest.org/en/stable/reference/reference.html#pytest-approx) + — default rel tol `1e-6`, abs tol `1e-12`; justifies the page's `1e-6` default. + +### 2 — A path-valued config key must be absolute + +**Claim as queued:** when a launcher owns the CWD, reject a relative path env var +with `ValueError` rather than resolving it against CWD; a missing dir globs to +`[]` and looks like "no work today". + +**Verified by controlled experiment.** Installed a LaunchAgent with +`ProgramArguments` + `RunAtLoad` and **no** `WorkingDirectory` key; it recorded +`cwd=/`, `PWD=/`, and a relative `./data/signals` lookup reported "No such file +or directory". Plist booted out and removed afterwards. Independently, +launchd-spawned `loginwindow` also reports cwd `/` under `lsof`. + +The silent-failure half also reproduced: `glob.glob("/nonexistent-xyz/*.json")` +returns `[]` and `Path(...).glob(...)` yields nothing — neither raises — while +`os.listdir` on the same path raises `FileNotFoundError`. And +`Path("~/data").expanduser().is_absolute()` is `True` while +`Path("./data").expanduser().is_absolute()` is `False`, which is what makes the +queued implementation's `expanduser()`-then-`is_absolute()` order correct. + +**Sources checked:** +- [systemd.exec(5)](https://man7.org/linux/man-pages/man5/systemd.exec.5.html) — + `WorkingDirectory=`: *"If not set, defaults to the root directory when systemd + is running as a system instance and the respective user's home directory if run + as user."* This adds a nuance the candidate did not have: **systemd user units + default to `$HOME`, not `/`** — so the same relative path resolves to three + different places across launchd / systemd-system / systemd-user. That nuance is + now in the page and in the `background-services` edge row. +- `launchd.plist(5)` (local man page) — `WorkingDirectory` is *"This optional key + is used to specify a directory to chdir(2) to before running the job"*: optional, + nothing inherited from the installer. +- [Apple, Creating launchd jobs](https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html), + [12factor config](https://12factor.net/config). + +### 3 — Enumerate call sites by callee, not parameter name + +**Claim as queued:** grep the callee (`verify(`), not the parameter name +(`repo_rows=`), because positional calls carry no parameter name; sweep test +helpers separately. + +**Verified, mechanism reproduced.** For a file holding both a keyword call and a +positional call, `grep -n "repo_rows"` returns 2 hits (definition + keyword call) +while `grep -n "verify("` returns 3 (definition + both calls) — the positional +call is invisible to the parameter-name search. + +**Two additions the candidate did not have**, both verified: +- **Find-references beats both greps.** [LSP 3.17](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/) + `textDocument/references` resolves the symbol, so it returns positional and + keyword calls alike and does not over-match a same-named function on another + type. Callee grep is now the documented *fallback*. +- **Make stale calls loud.** Marking the parameter keyword-only + ([PEP 3102](https://peps.python.org/pep-3102/)) turns an unmigrated positional + call into `TypeError: verify() takes 3 positional arguments but 4 were given` + instead of silently binding into a neighbouring parameter. Verified locally. + +Also cited: [Python calls reference](https://docs.python.org/3/reference/expressions.html#calls) +for the positional/keyword binding rules. The original field evidence (linkly: +13 keyword hits → `Ran 472 tests / FAILED (failures=11)`, plus a `rows_for()` +helper feeding 5 more sites) is preserved in the page's Sources. + +### 4 — Sharpen an artifact guard from shape to consequence + +**Claim as queued:** when a repo-wide guard asserting "no artifact has shape S" +fires on a legitimate artifact, rewrite it as "no artifact has S *and* the +consequence C", computing C via the production derivation. + +**Kept `confidence: field-tested`, deliberately.** The *failure mode* is +sourced — Google Testing Blog, +[Change-Detector Tests Considered Harmful](https://testing.googleblog.com/2015/01/testing-on-toilet-change-detector-tests.html) +(Alex Eagle, 2015-01-27): *"Change-detector tests do not add clarity, and you +cannot safely refactor code if you know you need to adapt the tests afterwards to +get them passing again."* A shape-only guard needing an exemption per legitimate +artifact is that failure at repo scope. Step 4's required-red fixture rests on +[pitest.org](https://pitest.org/)'s mutation mechanic (already cited elsewhere in +this wiki). + +But the specific technique — compute C from the *production* derivation, assert +the exemption's reason — rests on **one** real case (linkly #35: the guard fired +on `examples/checkout.lnpl`; sharpening to "guarded call that could actually +fail" via `_lnpl_ops`' `seeded_entities`/`repository_calls` returned the suite to +`Ran 518 tests / OK` while a guarded-and-can-fail fixture still drove it red). +One case is field evidence, not verification, so the page says `field-tested` and +describes that context. + +**Honest caveat on this source:** the change-detector article's body would not +render through fetch (only header/comments returned). The quoted sentence is the +one confirmed via search snippet; the URL itself is already cited by +`testing-quality-behavior-not-implementation` in this wiki. No other sentence +from that article is quoted. + +--- ## Existing-layer check -Cross-PR and against-main duplication was the focus. Findings and resolutions: - -- **spec-artifact-checks (#8) ≡ document-conformance-checks (#9)** — same case - (coverage-vs-validity split, per-check negative controls, GFM pipe parsing, - ESLint/Semgrep/mutation examples). #9's report predated awareness of #8. → - **#8 kept canonical; #9's page dropped, `testing/docs-as-spec` category not created.** -- **completion-response-validation (#6) ≈ llm-response-completeness (#12)** — ~95% - same case (HTTP 200 ≠ usable output; `length`/blank/reasoning-budget). → - **merged into one `llm/` page; #12's `integrations/` copy dropped.** -- **gateway-model-alias-defaults (#6) ≈ externally-owned-defaults (#12)** — ~80%; - #12 generalizes the model-alias case to any external resource. → - **kept the general `integrations/` page; #6's LLM-only page dropped.** -- **non-interactive-cli-invocation** — created by BOTH #11 and #12 (file collision). - → **single reconciled page.** -- Distinct (no overlap, all landed): checks-that-cannot-pass, harness-reverse-controls, - spec-document-gates, editing-a-gated-document, unicode-text-matching, - command-text-inspected-before-execution, object-key-persistence, context-window-budget, - host-cgroup-visibility, missing-container-metrics. -- Reciprocal `related:` links added on existing pages (tests-that-cannot-fail, - timeouts-and-retries, environment-config, release-gates, background-services, - portable-shell-scripts, timezone-and-locale, paths-case-and-line-endings, - acceptance-criteria, resource-limits-and-probes, logs-metrics-signals, - minimum-case-set). A dropped-page backlink (#6 → gateway-model-alias-defaults on - environment-config and release-gates) was retargeted to externally-owned-defaults. -- Invariants verified programmatically: all `related:`/inline `[id]` references - resolve, every page listed in its domain index, no duplicate ids, no page >120 - body lines. +**Pages read in full for overlap:** `testing/index.md`, `infrastructure/index.md`, +`platforms/index.md`, `qa/index.md`, `debugging/index.md`, `backend/index.md`, +`testing/quality/tests-that-cannot-fail.md`, +`testing/quality/spec-artifact-checks.md`, +`testing/quality/behavior-not-implementation.md`, +`testing/async/async-testing.md`, `infrastructure/config/environment-config.md`, +`platforms/processes/background-services.md`, plus a repo-wide grep for +`floating.point|tolerance|isclose|approx|monotonic|fake clock` and +`absolute path|is_absolute|working directory|launchd|relative path|fail-fast`. + +| Insight | Overlap found | Resolution | +|---------|---------------|-----------| +| 1 | `test-data-and-isolation` already says "refactor the code to accept an injected clock; that seam is the fix". `async-testing` covers fake timers vs condition waits. | **Complementary, not duplicate** — those pages get you *to* an injected clock; this page is what to do once you have one and must compare its readings. Created new; linked both ways. | +| 2 | `environment-config` already mandates full-schema startup validation + "required keys get NO default". `background-services` covers minimal environment and absolute paths **for binaries** — but nowhere states the CWD default. | **Merged the fact where it belongs, created the directive page.** The CWD-is-`/` fact went to `background-services` (platforms owns it). The validation directive is a new sibling page under `infrastructure/config/`. | +| 3 | `behavior-not-implementation` covers "a refactor broke tests"; `qa/process/regression-scope` covers what to re-test. | Both are adjacent, neither owns *enumerating call sites*. Created new; linked to both. | +| 4 | `tests-that-cannot-fail` (a test that can't detect) and `spec-artifact-checks` (per-check negative controls) are close cousins. | **Distinct trigger**: those pages are about a guard that never fires; this is a guard that fires on a *legitimate* artifact. It is the mirror case. Created new; linked to both. | + +**Conflicts flagged:** none. No new directive contradicts an existing one. +Insight 2's directive *sharpens* `environment-config` rule 3 ("crash on any +missing or invalid key") for the path case rather than opposing it. + +**Amendments to existing pages (5):** +- `infrastructure/config/environment-config.md` — new edge row (path-valued keys) + + `related:` link. +- `platforms/processes/background-services.md` — new edge row (working directory + defaults per manager), `systemd.exec(5)` added to `sources:` with the verbatim + default quote, `last_verified` → 2026-08-04, `related:` link. +- `testing/quality/tests-that-cannot-fail.md` — reciprocal `related:` links. +- `testing/quality/behavior-not-implementation.md` — reciprocal `related:` links. +- `testing/async/async-testing.md` — reciprocal `related:` link. + +--- ## Routing decision -- `backend/common/llm/` (new) — LLM-specific server concerns: completion-response-validation, - context-window-budget. Coherent home shared by #6 and #13. -- `backend/common/integrations/` (new) — general repo-external-dependency concern: - externally-owned-defaults. Kept separate from `llm/` because its scope is any - external resource (bucket/queue/index), not LLM-only. -- `backend/common/storage/` (new) — object-key-persistence. -- `qa/document-verification/` (new) — spec-document-gates, editing-a-gated-document. - Introduced by both #10 and #11; unified into one index section. -- `testing/quality/` (existing) — checks-that-cannot-pass, spec-artifact-checks, - harness-reverse-controls (test/check-authoring discipline, distinct from - qa/document-verification which is release-process gate design). -- `platforms/{environment,shells,processes}/` (existing) — unicode-text-matching, - command-text-inspected-before-execution, non-interactive-cli-invocation. -- `infrastructure/{containers,observability}/` (existing) — host-cgroup-visibility, - missing-container-metrics. - -Source PRs #6–#13 are closed with a disposition comment crediting the author. +| Insight | Domain / category / page | Rationale | +|---------|--------------------------|-----------| +| 1 | `testing` / `quality` / `injected-clock-duration-assertions` | Existing category. It is assertion-design: a test that **fails on correct code**, the mirror of `tests-that-cannot-fail`. Considered `async` (fake timers) and `data` (time-dependent fixtures); both own adjacent concerns and are linked instead. | +| 2 | `infrastructure` / `config` / `path-valued-config` | Existing category, **new page rather than a row in `environment-config`**. Distinct trigger ("a config value is a path and the CWD is not mine" vs "config differs per environment"), and AGENTS.md rule 1 is one case per page. The platform *fact* (CWD default) was merged into `platforms/processes/background-services` instead of duplicated. | +| 3 | `backend` / **new category `refactoring`** / `call-site-enumeration` | Routing protocol says route to the domain owning the artifact you change — this changes application code → `backend`, and it is language-agnostic → `common/`. **New category justified:** the 11 existing `backend/common` categories are all runtime concerns (api-design, reliability, caching, jobs, errors, auth, orm, concurrency, llm, integrations, storage); none covers changing existing code. Filed under `common/` because positional/keyword argument binding is not Python-specific. | +| 4 | `testing` / `quality` / `guard-shape-vs-consequence` | Existing category, alongside the other four guard/check-design pages. | + +**New categories created: 1** (`backend/common/refactoring/`). +`INDEX.md` and `wiki/backend/index.md` updated for it; `INDEX.md` and +`wiki/infrastructure/index.md` updated for insight 2. + +--- + +## Verification of the change itself + +A structural lint over all **143** pages passes: no duplicate ids, no id/path +mismatch, no page over 120 body lines, every page carries `sources:` and the +required sections, every `related:` id and inline `[page-id]` reference resolves, +and every page is listed in its domain index. One vague-qualifier hit in a new +page was fixed; the two remaining hits are pre-existing and untouched. diff --git a/INDEX.md b/INDEX.md index d6d4238..34e0144 100644 --- a/INDEX.md +++ b/INDEX.md @@ -10,9 +10,9 @@ follow the cross-pointers in their index or take the next matching seeded domain | Domain | Status | Route here when | |--------|--------|-----------------| | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior | -| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | +| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references, migrating call sites of a changed signature) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility | -| [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) | +| [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, per-environment and path-valued configuration, rollout/rollback strategy, observability (logs/metrics/alerting) | | [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) | | [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing, automated verification of document deliverables (spec/RFC gates) (writing automated test code → testing) | | [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) | diff --git a/log.md b/log.md index 1c6293b..91ce11c 100644 --- a/log.md +++ b/log.md @@ -37,3 +37,4 @@ Append-only. Format: `## [YYYY-MM-DD] ` for developer convenience | No default; validate presence and absoluteness at startup | The dev-friendly relative default becomes the production value the moment the real one is missing | + +## Sources + +- https://man7.org/linux/man-pages/man5/systemd.exec.5.html — `WorkingDirectory=`: "If not set, defaults to the root directory when systemd is running as a system instance and the respective user's home directory if run as user" +- https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html — LaunchAgent plist keys; `WorkingDirectory` is documented in `launchd.plist(5)` as "This optional key is used to specify a directory to chdir(2) to before running the job" — optional, with no directory inherited from the installer +- https://12factor.net/config — config lives in the environment and is what varies between deploys +- Local reproduction 2026-08-04 (macOS 25.1, launchd): a LaunchAgent with `ProgramArguments` and `RunAtLoad` and **no** `WorkingDirectory` key recorded `cwd=/` and `PWD=/`; a relative `./data/signals` lookup from that job reported "No such file or directory". An independently launchd-spawned process (`loginwindow`) also reports cwd `/` under `lsof` +- Local reproduction 2026-08-04 (CPython 3.14.6): `glob.glob("/nonexistent-xyz/*.json")` returns `[]` and `Path("/nonexistent-xyz").glob("*.json")` yields nothing, both without raising, while `os.listdir` on the same path raises `FileNotFoundError` — the empty-scan result is what makes a bad path silent. `Path("~/data").expanduser().is_absolute()` is `True` while `Path("./data").expanduser().is_absolute()` is `False` diff --git a/wiki/infrastructure/index.md b/wiki/infrastructure/index.md index 2bff59c..6d020fa 100644 --- a/wiki/infrastructure/index.md +++ b/wiki/infrastructure/index.md @@ -20,6 +20,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [environment-config](config/environment-config.md) | Adding configuration that differs per environment; a bug traced to a dev/stg/prd config difference; config sprawled across hardcoded values, files, and env vars; reviewing how a service gets its settings | +| [path-valued-config](config/path-valued-config.md) | A config key or env var holds a filesystem path (spool/input/output directory, data file, socket) for a process whose working directory is set by launchd/systemd/cron/a container entrypoint/CI; deciding whether to accept a relative path, expand `~`, or crash at startup; a correctly-deployed service processes nothing and reports no error; writing the loader's rejection tests | ## containers diff --git a/wiki/platforms/processes/background-services.md b/wiki/platforms/processes/background-services.md index 22b1e9a..71fe562 100644 --- a/wiki/platforms/processes/background-services.md +++ b/wiki/platforms/processes/background-services.md @@ -7,11 +7,12 @@ confidence: verified sources: - https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html - https://man7.org/linux/man-pages/man5/systemd.service.5.html + - https://man7.org/linux/man-pages/man5/systemd.exec.5.html - https://man7.org/linux/man-pages/man5/crontab.5.html - https://man7.org/linux/man-pages/man1/nohup.1.html - https://man7.org/linux/man-pages/man1/loginctl.1.html -last_verified: 2026-07-10 -related: [platforms-toolchains-version-management, platforms-shells-portable-shell-scripts, platforms-processes-non-interactive-cli-invocation] +last_verified: 2026-08-04 +related: [platforms-toolchains-version-management, platforms-shells-portable-shell-scripts, platforms-processes-non-interactive-cli-invocation, infrastructure-config-path-valued-config] --- # Keeping a Process Running Beyond the Terminal Session @@ -56,6 +57,7 @@ Then apply all four of these regardless of mechanism: | Case | Then | |------|------| | Job works in a terminal, fails under cron/launchd | Environment gap. Reproduce with `env -i /bin/sh -c ''`; fix by absolute-pathing binaries and exporting env in the unit | +| The job reads or writes a relative path | The working directory is the manager's, not the install directory: `/` for launchd and systemd **system** units, the user's home for systemd **user** units. Set `WorkingDirectory` in the unit/plist, or take absolute paths from config ([infrastructure-config-path-valued-config]) | | Service runs a version-manager-installed binary (nvm node, pyenv python) | Shims and lazy-loaders need rc files that services never load — invoke the real binary's absolute path (see platforms-toolchains-version-management) | | Process started as an agent-harness background task (e.g. run_in_background) must outlive the session | The harness kills its background tasks when the session ends — detach with `nohup … & disown` or promote to a service unit | | Linux user service must survive logout | `loginctl enable-linger ` — user units otherwise stop when the last session closes | @@ -73,6 +75,7 @@ Then apply all four of these regardless of mechanism: - https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html — LaunchAgent plists, KeepAlive, StartCalendarInterval - https://man7.org/linux/man-pages/man5/systemd.service.5.html — `Restart=on-failure` semantics +- https://man7.org/linux/man-pages/man5/systemd.exec.5.html — `WorkingDirectory=`: "If not set, defaults to the root directory when systemd is running as a system instance and the respective user's home directory if run as user" - https://man7.org/linux/man-pages/man5/crontab.5.html — cron-set environment (SHELL/HOME/LOGNAME), overridable in the crontab - https://man7.org/linux/man-pages/man1/nohup.1.html — run a command immune to hangups - https://man7.org/linux/man-pages/man1/loginctl.1.html — `enable-linger` keeps user services running while logged out diff --git a/wiki/testing/async/async-testing.md b/wiki/testing/async/async-testing.md index a89968e..8604e29 100644 --- a/wiki/testing/async/async-testing.md +++ b/wiki/testing/async/async-testing.md @@ -11,7 +11,7 @@ sources: - https://testing-library.com/docs/dom-testing-library/api-async/ - https://martinfowler.com/articles/nonDeterminism.html last_verified: 2026-07-10 -related: [testing-quality-tests-that-cannot-fail, testing-flaky-diagnosing-flaky-tests, testing-data-test-data-and-isolation] +related: [testing-quality-tests-that-cannot-fail, testing-flaky-diagnosing-flaky-tests, testing-data-test-data-and-isolation, testing-quality-injected-clock-duration-assertions] --- # Testing Asynchronous Code Deterministically diff --git a/wiki/testing/index.md b/wiki/testing/index.md index 2cbad76..3c69c08 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -25,6 +25,8 @@ Match your situation to a "load when" line; load only matching pages. | [tests-that-cannot-fail](quality/tests-that-cannot-fail.md) | Reviewing tests that always pass; a bug shipped through an area the suite reported as covered; auditing a suspiciously green suite; judging whether an assertion, error-path test, or mock-based test can actually detect a defect | | [checks-that-cannot-pass](quality/checks-that-cannot-pass.md) | Authoring a check whose target does not exist yet (grep/regex gate on an unwritten file or doc section, lint/scan rule, schema assertion on an unbuilt endpoint, a plan's verification command) and it has only ever been observed failing; reviewing a plan's gates before adopting them; separating "target missing" from "content missing" in a gate's exit status | | [spec-artifact-checks](quality/spec-artifact-checks.md) | Writing or reviewing an automated check that a mapping table covers every rule/field/enum case, or that ids resolve across documents; deciding whether a green check earned "verified" or only "present"; designing one negative control per check in a multi-check harness; parsing Markdown table rows programmatically in a doc-as-spec repo | +| [injected-clock-duration-assertions](quality/injected-clock-duration-assertions.md) | Asserting an elapsed duration between two readings of an injected/fake float clock (rate-limit interval, backoff, debounce, TTL); choosing that fake clock's start value; a single duration test fails on correct code by a margin in the far decimal places; choosing a comparison tolerance, or deciding between float seconds and integer nanoseconds | +| [guard-shape-vs-consequence](quality/guard-shape-vs-consequence.md) | A repo-wide guard asserting that no shipped artifact (example, config, migration, fixture) has a structural shape has gone red on a legitimate new artifact; authoring such a scanning guard; deciding between exempting an artifact, deleting the guard, and sharpening it; an existing guard has accumulated an exemption/allow list | | [harness-reverse-controls](quality/harness-reverse-controls.md) | You built a harness that scores how well something is verified (mutation run, doc/spec gate suite, CI check matrix) and are about to cite its score in a commit, PR, README, or report; its verdicts come out uniform (every case caught, or every case green); deciding what control run proves the harness discriminates, how to score errored/never-ran cases, and what the harness's isolated working tree must contain | ## data diff --git a/wiki/testing/quality/behavior-not-implementation.md b/wiki/testing/quality/behavior-not-implementation.md index 3436612..be83070 100644 --- a/wiki/testing/quality/behavior-not-implementation.md +++ b/wiki/testing/quality/behavior-not-implementation.md @@ -9,7 +9,7 @@ sources: - https://abseil.io/resources/swe-book/html/ch12.html - https://testing.googleblog.com/2015/01/testing-on-toilet-change-detector-tests.html last_verified: 2026-07-10 -related: [testing-quality-minimum-case-set, testing-mocking-what-to-mock] +related: [testing-quality-minimum-case-set, testing-mocking-what-to-mock, testing-quality-guard-shape-vs-consequence, backend-common-refactoring-call-site-enumeration] --- # Asserting Behavior Through the Public Interface diff --git a/wiki/testing/quality/guard-shape-vs-consequence.md b/wiki/testing/quality/guard-shape-vs-consequence.md new file mode 100644 index 0000000..e3875e8 --- /dev/null +++ b/wiki/testing/quality/guard-shape-vs-consequence.md @@ -0,0 +1,83 @@ +--- +id: testing-quality-guard-shape-vs-consequence +domain: testing +category: quality +applies_to: [general] +confidence: field-tested +sources: + - https://testing.googleblog.com/2015/01/testing-on-toilet-change-detector-tests.html + - https://pitest.org/ +last_verified: 2026-08-04 +related: [testing-quality-tests-that-cannot-fail, testing-quality-behavior-not-implementation, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, qa-process-regression-scope] +--- + +# A Repo-Wide Guard That Fires on a Legitimate Artifact + +## When this applies + +A guard test scans every shipped artifact of a kind — example files, configs, +migrations, fixtures, schema docs — and asserts that none of them has a +structural shape S. A newly added, legitimate artifact now has S, so the guard +is red and you are deciding what to do about it. Also applies when authoring +such a guard, before the first legitimate collision happens. + +Reviewing a guard that has never been red → [testing-quality-tests-that-cannot-fail]. + +## Do this + +1. **Write down the consequence C that S was a proxy for.** The guard was never + about the shape; it was about an outcome the shape stands in for — "this + call can fail at runtime", "this migration takes an exclusive lock", "this + config exposes a port". State C as a property you could compute from the + artifact. + +2. **Re-express the assertion as "no artifact has S *and* C".** Compute C by + feeding the artifact to the **production derivation that already decides C** — + the same resolver, planner, or rule engine the real system uses — rather than + re-implementing the rule inside the test. + +3. **Assert the specific reason the exempt artifact is exempt.** Name the + property that makes it safe (its precondition is guaranteed, its target is + seeded, its lock is already held). This keeps the guard's answer meaningful + instead of merely quiet. + +4. **Prove the sharpened guard still reddens.** Add a fixture artifact that has + both S and C and require the guard to fail on it — a seeded fault that + produces no failure means the guard stopped measuring when you sharpened it. + +5. **When C cannot be computed at check time, narrow the guard's scope instead + of exempting an artifact.** Restrict it to a directory or naming convention + where S is always wrong, so membership is decided by location rather than by + a list of names. + +| Case | Do | +|------|----| +| Production code already derives C | Import that derivation and call it from the guard | +| C needs inputs the artifact does not carry | Synthesize the minimal input in the guard and state that assumption in the test name | +| C is genuinely uncomputable at check time | Narrow the guard's scope (step 5); keep the shape check inside that scope | +| The colliding artifact is the bug's own reproduction case | Move it into a fixture directory the guard excludes by scope — a reproduction is meant to have the shape | + +## Edge cases + +| Case | Then | +|------|------| +| An exemption/allow list already exists on the guard | Each entry is a case the guard stopped measuring; convert the guard to the consequence form and delete the list, or move those artifacts out of scope by location | +| The sharpened guard goes green immediately and no fixture has both S and C | It is unproven, not passing — add the fixture from step 4 before trusting it ([testing-quality-tests-that-cannot-fail]) | +| Reusing the production derivation means a bug in that derivation silently greens the guard | Accept the coupling — it is what keeps the guard's meaning in sync — and keep the step-4 fixture as the independent control that would catch the greening | +| Computing C over every artifact is expensive | Keep the shape check as a cheap prefilter and compute C only for the artifacts S matched; the assertion stays "S and C" | +| The guard reddens on an artifact that has S and genuinely has C | This is the guard working — fix the artifact, not the guard | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Delete the guard because a legitimate artifact now has the shape | Sharpen it to "shape and consequence" and keep it | The regression it was written for is still possible; deleting the guard drops the intent along with the false positive | +| Append the artifact to an exemption list | Compute the consequence and let the artifact pass on its merits | An exemption list grows by one on every collision, and a guard that exempts each new case measures nothing | +| Re-implement the consequence rule inside the test | Call the production derivation the real system uses | A copied rule drifts from the system it describes, so the guard eventually reports on a policy nobody ships | +| Ship the sharpened guard because the suite is green | Add a fixture with both the shape and the consequence and require red first | Sharpening an assertion is the easiest way to accidentally narrow it to nothing | + +## Sources + +- https://testing.googleblog.com/2015/01/testing-on-toilet-change-detector-tests.html — Alex Eagle, "Testing on the Toilet: Change-Detector Tests Considered Harmful" (2015-01-27): "Change-detector tests do not add clarity, and you cannot safely refactor code if you know you need to adapt the tests afterwards to get them passing again." A shape-only guard that must be exempted for each new legitimate artifact is this failure mode at repo scope +- https://pitest.org/ — "Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived" — the basis for step 4's required-red fixture +- Field evidence (linkly #35, 2026-08-04): `test_no_shipped_example_has_a_guarded_repository_call` asserted that no shipped `.lnpl` example contained a repository call under a guard. `examples/checkout.lnpl` legitimately added a `create` under `when stock > 0` — the issue's own reproduction shape — turning the guard permanently red. Re-expressing it as "a guarded call that could actually fail", with the conflict/miss decision taken from the production `_lnpl_ops` derivation via `seeded_entities`/`repository_calls`, returned the suite to `Ran 518 tests / OK` while a fixture holding a guarded-and-can-fail create still drove the guard red diff --git a/wiki/testing/quality/injected-clock-duration-assertions.md b/wiki/testing/quality/injected-clock-duration-assertions.md new file mode 100644 index 0000000..ec1a227 --- /dev/null +++ b/wiki/testing/quality/injected-clock-duration-assertions.md @@ -0,0 +1,83 @@ +--- +id: testing-quality-injected-clock-duration-assertions +domain: testing +category: quality +applies_to: [general] +confidence: verified +sources: + - https://peps.python.org/pep-0564/ + - https://docs.python.org/3/tutorial/floatingpoint.html + - https://docs.python.org/3/library/math.html#math.isclose + - https://docs.pytest.org/en/stable/reference/reference.html#pytest-approx +last_verified: 2026-08-04 +related: [testing-async-async-testing, testing-data-test-data-and-isolation, testing-flaky-diagnosing-flaky-tests, testing-quality-tests-that-cannot-fail] +--- + +# Asserting a Duration Between Two Readings of an Injected Clock + +## When this applies + +Code under test records two readings of an injected/fake clock that returns a +float of seconds, and the test asserts the elapsed gap against an exact bound — +a rate limiter's minimum interval, a retry backoff, a debounce window, a TTL. +The assertion is `gap >= interval`, `gap == interval`, or `gap <= interval`. + +Choosing the seam (injecting the clock at all) → [testing-data-test-data-and-isolation]. + +## Do this + +1. **Put the tolerance on the bound, and pick the bound's direction + deliberately.** A float clock makes `start + interval - start` differ from + `interval`, so an exact comparison fails on correct code: + +| Assertion you mean | Write | +|--------------------|-------| +| "at least `interval` elapsed" (rate limit, min backoff) | `assert gap >= interval - TOL` | +| "at most `interval` elapsed" (deadline, max wait) | `assert gap <= interval + TOL` | +| "exactly `interval` elapsed" (computed duration) | `math.isclose(gap, interval)` / `gap == pytest.approx(interval)` | + + Use a symmetric closeness helper only for the equality row. On a lower bound, + `isclose` also accepts a gap that is *smaller* than the interval — which is + the defect a rate-limiter test exists to catch. + +2. **Start the fake clock at `0.0`.** `0.0 + x - 0.0` is exact for every `x`, so + the arithmetic error disappears at the source. This is what CPython itself + does: PEP 564 records that Python "starts `monotonic()` and `perf_counter()` + clocks at zero on some platforms which indirectly reduce the precision loss". + +3. **Set `TOL` from the magnitudes, not by widening until green.** Pick a value + at least three orders of magnitude below the smallest interval the test must + distinguish, and above the representation error at your clock's magnitude. + `1e-6` seconds satisfies both for second-scale intervals, and matches + `pytest.approx`'s default relative tolerance of `1e-6`. + +4. **When the assertion must be exact, count integer nanoseconds** and keep the + fake clock's counter an `int`. Integers do not lose precision, so no + tolerance is needed and the test states an exact fact. + +## Edge cases + +| Case | Then | +|------|------| +| The fake clock's start value was chosen to look "realistic" (an epoch, a machine uptime, `1000.0`) | Set it to `0.0` and keep the tolerance. Realism buys the test nothing and is exactly what reintroduces the error — measured below, a start of `1.0` is already enough to break an exact `>=` | +| The gap comes out *larger* than the interval, not smaller | Expected — the rounding direction depends on the start value's exponent (measured: start `1e6` yields `1.0500000000465661` for a `1.05` step). Tolerate both directions on an equality or upper-bound assertion, not just the low side | +| The test seeds the fake clock from `time.monotonic()` | Seed it from `0.0` instead; a real monotonic reading is a large float (measured 90474.0 on an ordinary session) and carries the same error class | +| Only one test in the suite fails after a clock change, with a gap that differs from the bound in the 13th decimal | Read it as a representation artifact, not a behavior regression — confirm by checking the delta's magnitude before touching the implementation | +| The chosen `TOL` is within an order of magnitude of the interval | The assertion no longer distinguishes "waited" from "did not wait"; shrink `TOL` or assert on an integer-nanosecond clock ([testing-quality-tests-that-cannot-fail]) | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Write `assert gap >= interval` against a float fake clock | `assert gap >= interval - TOL` with `TOL` chosen by magnitude | Most decimal fractions have no exact binary representation, so a correct implementation produces a gap a few ulps short and the test fails on working code | +| Give the fake clock a large "realistic" start value | Start it at `0.0` | `0.0 + x - 0.0` is exact; any other start makes the recoverable gap depend on that start's exponent | +| Widen the tolerance until the suite goes green | Compute the tolerance from the smallest interval the test must distinguish | A tolerance sized to silence a failure eventually exceeds the defect and the test stops detecting it | +| Use `math.isclose(gap, interval)` for an "at least" assertion | One-sided `gap >= interval - TOL` | Symmetric closeness also accepts a too-short gap — the exact violation a minimum-interval test guards | + +## Sources + +- https://peps.python.org/pep-0564/ — "Internally, Python starts `monotonic()` and `perf_counter()` clocks at zero on some platforms which indirectly reduce the precision loss"; "The problem is that the `float` type starts to lose nanoseconds after 104 days" +- https://docs.python.org/3/tutorial/floatingpoint.html — "Unfortunately, most decimal fractions cannot be represented exactly as binary fractions… the decimal floating-point numbers you enter are only approximated by the binary floating-point numbers actually stored in the machine" +- https://docs.python.org/3/library/math.html#math.isclose — `math.isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0)`; symmetric closeness, with `abs_tol` required for comparisons against zero +- https://docs.pytest.org/en/stable/reference/reference.html#pytest-approx — default relative tolerance `1e-6`, default absolute tolerance `1e-12`; equal if either tolerance is met +- Local reproduction 2026-08-04 (CPython 3.14.6, macOS): for a `1.05` step, `start + 1.05 - start` yields `1.05` at `start=0.0`; `1.0499999999999998` at `1.0`; `1.0499999999999972` at `100.0`; `1.0499999999999545` at `1000.0`; `1.0500000000465661` at `1e6`; `1.0499999523162842` at `1e9`. Only `start=0.0` satisfies `gap >= 1.05`; every listed start satisfies `gap >= 1.05 - 1e-6` diff --git a/wiki/testing/quality/tests-that-cannot-fail.md b/wiki/testing/quality/tests-that-cannot-fail.md index 884ff61..8643b60 100644 --- a/wiki/testing/quality/tests-that-cannot-fail.md +++ b/wiki/testing/quality/tests-that-cannot-fail.md @@ -11,7 +11,7 @@ sources: - https://martinfowler.com/bliki/TestCoverage.html - https://testing.googleblog.com/2013/05/testing-on-toilet-dont-overuse-mocks.html last_verified: 2026-07-10 -related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-checks-that-cannot-pass, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, qa-document-verification-spec-document-gates] +related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-checks-that-cannot-pass, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, qa-document-verification-spec-document-gates, testing-quality-guard-shape-vs-consequence, testing-quality-injected-clock-duration-assertions] --- # Proving a Test Can Fail