Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
321 changes: 278 additions & 43 deletions .dev-loop/INGEST_REPORT.md

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions log.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,5 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
## [2026-08-06] revise | Union-merged amendments from the same 15 PRs into 24 existing pages (largest: tests-that-cannot-fail — 7 PRs: per-assertion mutation granularity, restore-mechanism-by-commit-state, suite-total-drop detection; portable-shell-scripts — 8 PRs: POSIX inline set-- reordering, ${VAR:-} vs ${VAR-} colon semantics, quote-by-what-the-text-is; test-data-and-isolation — 6 PRs: env-derived write paths, absent-variable cases, exec-bit fixtures, leak attribution; change-impact/call-site-enumeration — 8 unique additions from 7 path-variants: rename-in-place, same-type reorder, *args forwarding, two-searches-agree fallacy, codemod, coverage completion check). All non-canonical cross-page ids remapped to canonical pages; every added line traced to its source PR hunk (fabricated worker output was detected by orphan-line verification and replaced with true PR content).
## [2026-08-06] dedup | Cross-PR duplicate clusters collapsed: client-side rate limiting (8 PR versions at 3 paths → 1), call-site enumeration (7 versions at 7 paths → additions folded into the already-merged canonical page), stderr/exit-0 diagnostics (4 versions at 4 paths → 1), macOS sysroot (2→1), env-var off switch (2→1), completion predicates (2→1), robots.txt (2→1), harness-mediated results (2→1), leaked test artifacts (2→1), orchestration category naming (orchestration vs agent-orchestration → agent-orchestration). Kept distinct after trigger comparison: differential-testing vs differential-run-agreement (setup vs interpretation), unset-versus-empty-parameters vs env-var-off-switches (semantics vs design, cross-linked), import-time-side-effects vs test-level-choice edge (tactics vs level choice, cross-linked). Merged-main near-dup scan (Jaccard over title+trigger, 141 pages): 0 pairs — no duplication among previously merged content.
## [2026-08-06] ingest | Reconciled post-consolidation flush PRs #42–#43 (created against pre-#44 main). Folded unique content: platforms/filesystems/permissions-and-exec-bits (+`sh "$SCRIPT"` interpreter-invocation rows — stubs need no exec bit, avoids EDR chmod+x heuristics), platforms/shells/command-text-inspected-before-execution (+gate-AUTHOR step 8: parse all three POSIX quoting forms, expand only ~/$HOME/${HOME}; red-then-green bats evidence), testing/quality/checks-that-cannot-pass (+vacuously-green pre-implementation test row), platforms/processes/tool-diagnostics-without-a-failing-exit-code (+repeat-suppression edge, +clang reproduction source), infrastructure/ci-cd/secrets-handling (+independent credential channels row: gh token vs git SSH vs API token, git push --dry-run / author:@me probes), testing/mocking/what-to-mock (+ESM read-only-bindings DI rows, +PATH-emptied tripwire negative-proof). Dropped as duplicates of the #44 state: #42's rate-limit row (already in timeouts-and-retries + client-side-rate-limiting), #42's warning-only-diagnostics page (same trigger as tool-diagnostics-without-a-failing-exit-code), #42's leak-attribution row (already in test-data-and-isolation).
## [2026-08-07] ingest | knowledge-flush of 4 queued insights — 4 new pages, 7 existing pages cross-linked. New: testing/quality/surviving-mutant-equivalence-triage (classify a live mutant as missing-test / equivalent / no-coverage before writing anything; when equivalent, delete the redundant branch and correct the justification comment that the equivalence just disproved — Stryker's documented remedy is "try to rewrite the code so it won't occur"), testing/quality/source-text-wiring-assertions (a `>= n` / `toHaveLength(n)` count assertion over a call appearing at several sites stays green when the one site the guard was written for is deleted — enumerate the sites and bind each to a bounded order anchor that occurs exactly once in the file, or to a function-body slice; prove each by deleting only its own site, and run a reformat control. Measured: greedy vs lazy quantifiers give identical verdicts, so the bound and the anchor's uniqueness are what constrain the match), frontend/data-fetching/query-state-vs-fetch-state (a `data | undefined` component prop collapses TanStack Query's two orthogonal axes; a disabled or offline-paused query is `status: pending` with `isLoading === false` and `isError === false`, so "undefined means loading" renders a spinner no fetch will resolve — pass status+fetchStatus or an explicit union and test one case per cell), backend/python/language/default-encoding-in-text-io (a byte round-trip cannot prove an `encoding=` fix on a UTF-8 locale — run the real entry point under `-X warn_default_encoding -W always::EncodingWarning` and assert zero warning lines naming that file). Merged into existing: tests-that-cannot-fail (whole-suite edge row now routes surviving mutants through classification instead of reading them all as missing tests), harness-reverse-controls, behavior-not-implementation, guard-shape-vs-consequence, async-ui-states (+disabled/paused edge row), bytecode-cache-staleness, timezone-and-locale — related links both ways. All cited URLs opened this session; two local reproductions (CPython 3.14.6 EncodingWarning discriminator vs byte-identical round-trip; `@tanstack/query-core@5.100.14` queryObserver.js:308-332 `isLoading = isPending && isFetching`).
## [2026-08-07] revise | testing/quality/guard-shape-vs-consequence — corrected a misattributed citation found by the pre-PR adversarial pass: the sentence "you cannot safely refactor code if you know you need to adapt the tests afterwards to get them passing again" was presented as the Google Testing Blog article's own, but re-fetching the page shows it is a reader comment (2015-02-04) with different wording ("refactor stuff", "know for sure"). The article body was not retrievable in full, so the bullet now cites the URL for the change-detector category without quoting it, and states the correction inline. The same quote had been copied into a new page in this flush before verification — the lesson being that a citation already present in the wiki is not a verified citation.
1 change: 1 addition & 0 deletions wiki/backend/python/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,4 @@ Match your situation to a "load when" line; load only matching pages.
|------|-----------|
| [mutable-state-traps](language/mutable-state-traps.md) | State persists or leaks across calls/requests in a long-lived Python process — one user's data appears for another, values "remembered" between calls; loop-built callbacks all use the last value; reviewing function signatures (mutable defaults), class bodies (class attributes), or module-level objects for hidden sharing; choosing contextvars vs thread-locals for request context |
| [bytecode-cache-staleness](language/bytecode-cache-staleness.md) | A script or harness rewrites `.py` files and re-runs them in a loop (mutation testing, edit/test/revert, codegen check, bisect) and the result stops tracking what is on disk — a revert that `git diff` reports clean still fails, or an injected change has no effect; choosing between clearing `__pycache__`, refreshing mtime, and hash-based `.pyc` (PEP 552); designing byte-length-preserving mutations |
| [default-encoding-in-text-io](language/default-encoding-in-text-io.md) | Python opens a text file without `encoding=` (`open`, `Path.read_text`, `subprocess` text mode) and you are adding the argument or writing the regression test that keeps it there; a file-writing bug reproduces on Windows, a `LANG=C` container, or a cp949/cp932 desktop but not on your machine; choosing a test discriminator that does not depend on the runner's locale |
2 changes: 1 addition & 1 deletion wiki/backend/python/language/bytecode-cache-staleness.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ sources:
- https://peps.python.org/pep-0552/
- https://docs.python.org/3/library/py_compile.html
last_verified: 2026-08-04
related: [testing-quality-harness-reverse-controls, testing-quality-tests-that-cannot-fail, backend-python-language-mutable-state-traps]
related: [backend-python-language-default-encoding-in-text-io, testing-quality-harness-reverse-controls, testing-quality-tests-that-cannot-fail, backend-python-language-mutable-state-traps]
---

# Edited Python Source the Interpreter Keeps Ignoring
Expand Down
107 changes: 107 additions & 0 deletions wiki/backend/python/language/default-encoding-in-text-io.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
---
id: backend-python-language-default-encoding-in-text-io
domain: backend
category: language
applies_to: [python]
confidence: verified
sources:
- https://peps.python.org/pep-0597/
- https://peps.python.org/pep-0686/
- https://docs.python.org/3/library/functions.html
last_verified: 2026-08-07
related:
[
backend-python-language-bytecode-cache-staleness,
platforms-environment-timezone-and-locale,
testing-quality-tests-that-cannot-fail,
testing-quality-minimum-case-set,
]
---

# Text I/O Whose Encoding Comes from the Machine's Locale

## When this applies

Python code opens a text file without `encoding=` — `open(p)`, `open(p, "w")`,
`Path.read_text()`, `csv`/`json` wrappers built on them — and you are adding the
argument, or writing the regression test that keeps it there. Also when a
file-writing bug reproduces on one machine (Windows, a `LANG=C` container, a
cp949/cp932 desktop) and not on yours.

Timezone and locale as hidden inputs across dates and text →
[platforms-environment-timezone-and-locale].

## Do this

1. **Pass `encoding=` at every text-mode call site.** The default is the
machine's: "The default encoding is platform dependent (whatever
`locale.getencoding()` returns)". Choose the value from what the file is:

| The file is | Pass |
| --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| A format with a defined encoding (JSON, TOML, YAML, Markdown, source) | `encoding="utf-8"` |
| Written and read only by this program | `encoding="utf-8"` |
| Produced by a tool bound to the OS console encoding, deliberately | `encoding=locale.getencoding()`, stated explicitly so the dependency is visible |
| Raw bytes | Binary mode with no `encoding` — "For reading and writing raw bytes use binary mode and leave _encoding_ unspecified" |

2. **Make the regression test run the real entry point under the interpreter's
own diagnostic, and assert zero warnings naming the file you fixed:**

```sh
python3 -X warn_default_encoding -W always::EncodingWarning <entrypoint> <args>
```

`EncodingWarning` "is emitted when the `encoding` argument to `open()` is
omitted and the default locale-specific encoding is used", and the flag (or
`PYTHONWARNDEFAULTENCODING`) is what enables it. Filter the captured stderr
to the file under test by name, so unfixed call sites elsewhere in the
codebase do not redden this test.

3. **Assert on the warning lines, not on the produced bytes.** The warning is
emitted at the call site regardless of what the locale happens to be, so it
is the same verdict on your laptop and in CI.

4. **Prove the check reddens before trusting it.** Without `-X
warn_default_encoding` the warning is silent, so a runner that drops the flag
reports green on the reintroduced defect and looks identical to a pass. Seed a
deliberately unencoded `open()` in the file under test, require red, then
restore ([testing-quality-tests-that-cannot-fail]).

5. **Widen the flag from the one test invocation to the whole CI run once every
call site is clean**, so a new omission is caught where it is written rather
than at the next locale change. Until then the filter in step 2 is what keeps
the unfixed sites from reddening this test.

6. **Keep a value assertion for the encodings you set explicitly.**
`EncodingWarning` fires only on an *omitted* argument, so it says nothing about
`encoding="latin-1"` or a deliberate `encoding=locale.getencoding()`. For those
call sites, assert the bytes the file should contain, and run that assertion
under a non-UTF-8 locale (`LANG=C`, or a cp949/cp932 job) where a wrong value
changes the output.

## Edge cases

| Case | Then |
| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The entry point is a library function, not a script | Run it through a one-line driver under the same flags; the warning is attributed to the frame that called `open()`, so the driver's own lines do not mask it |
| A dependency emits `EncodingWarning` from its own files | Filter by filename as in step 2 and record the dependency in the test's name, so the filter states what it is excluding |
| The code targets Python 3.15 or later, where UTF-8 mode is on by default (PEP 686) | Keep the explicit `encoding=`: the argument states the file's contract and is what makes the call correct under an inherited `PYTHONUTF8=0` or an older runtime |
| Running under `PYTHONUTF8=1` / UTF-8 mode already | The warning still fires on the omitted argument, so the test keeps working; the mode changes the value used, not whether the argument was passed |
| `subprocess` output is being decoded | The same default applies to its text mode — pass `encoding="utf-8"` there, and include it in the call-site sweep |
| The harness rewrites the file between runs to seed the missing-`encoding` mutation | Clear the bytecode cache between iterations ([backend-python-language-bytecode-cache-staleness]) |

## Instead of

| If you are about to | Do this instead | Why |
| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Prove an omitted-`encoding` fix with a round-trip assertion alone (write non-ASCII text, read it back, compare) | Assert zero `EncodingWarning` lines naming the file, under `-X warn_default_encoding`, and keep the round-trip for the explicitly-set encodings (step 6) | On a UTF-8 locale the encoded bytes are identical with and without the argument, so the round-trip is green on the defect — it discriminates only on a runner whose locale encoding is not UTF-8, which is not the default on macOS or on most Linux CI images |
| Assert the output file's declared charset (`<meta charset>`, an XML declaration) | Assert the warning count | A declaration is a literal in the template — it is written correctly by code that encoded the body wrongly |
| Set `LANG`/`PYTHONUTF8` in the test environment to make the behavior deterministic | Fix the call sites and assert the warning | Pinning the environment makes the test pass by removing the input the defect depends on, so the defect ships and fails on the machines that do not inherit that environment |
| Read "it works on macOS and Linux" as evidence the encoding is right | Run the warning check | PEP 686: "many Python developers using Unix forget that the default encoding is platform dependent … Inconsistent default encoding causes many bugs"; "this change mostly affects Windows users" |

## Sources

- https://peps.python.org/pep-0597/ — `EncodingWarning` "is emitted when the `encoding` argument to `open()` is omitted and the default locale-specific encoding is used"; "The `-X warn_default_encoding` option and the `PYTHONWARNDEFAULTENCODING` environment variable are added. They are used to enable `EncodingWarning`"; "When the flag is set, `io.TextIOWrapper()`, `open()` and other modules using them will emit `EncodingWarning` when the `encoding` argument is omitted"; "Developers using macOS or Linux may forget that the default encoding is not always UTF-8"
- https://peps.python.org/pep-0686/ — enabling UTF-8 mode by default targets Python 3.15; "many Python developers using Unix forget that the default encoding is platform dependent. They omit to specify `encoding="utf-8"` … Inconsistent default encoding causes many bugs"; "Most Unix systems use UTF-8 locale … So this change mostly affects Windows users"
- https://docs.python.org/3/library/functions.html — `open()`: "The default encoding is platform dependent (whatever `locale.getencoding()` returns)"; "In text mode, if _encoding_ is not specified the encoding used is platform-dependent"; "For reading and writing raw bytes use binary mode and leave _encoding_ unspecified"
- Reproduction 2026-08-07 (CPython 3.14.6, macOS, `locale.getpreferredencoding(False) == 'UTF-8'`): a script with one `open(p, "w")` and one `open(p, "w", encoding="utf-8")` produced byte-identical output — a round-trip assertion cannot distinguish them. `python3 -X warn_default_encoding -W always::EncodingWarning script.py out.txt` emitted exactly one line, naming the unencoded call by file and line number; the same run without the flag emitted nothing
3 changes: 2 additions & 1 deletion wiki/frontend/data-fetching/async-ui-states.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ sources:
- https://tanstack.com/query/latest/docs/framework/react/guides/optimistic-updates
- https://react.dev/reference/react/Component
last_verified: 2026-07-10
related: [frontend-state-client-vs-server-state, frontend-data-fetching-race-conditions]
related: [frontend-state-client-vs-server-state, frontend-data-fetching-race-conditions, frontend-data-fetching-query-state-vs-fetch-state]
---

# Designing Loading, Error, Empty, and Data States for an Async View
Expand Down Expand Up @@ -59,6 +59,7 @@ Then apply these to the transitions between states:
| List is empty because the user's filters excluded everything | Say so, and offer "clear filters" — the generic empty state ("add your first item") misleads |
| Response resolves fast enough that the skeleton only flashes | Keep the reserved space but suppress indicator animation for sub-second responses — feedback that fast is distraction, not information |
| Mutation has no inverse (send email, submit payment) | No optimistic update — render an explicit pending state until the server confirms |
| The query can be disabled (`enabled: false`) or paused offline, so it has no data and is not fetching | The four states above do not cover it — branch on the cache's status/fetchStatus pair ([frontend-data-fetching-query-state-vs-fetch-state]) |

## Instead of

Expand Down
Loading
Loading