Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
220 changes: 44 additions & 176 deletions .dev-loop/INGEST_REPORT.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions log.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,4 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
## [2026-08-03] dedup | Dropped 3 candidate pages as duplicates/superseded during the #6–#13 consolidation: testing/docs-as-spec/document-conformance-checks (#9 — same case as testing/quality/spec-artifact-checks from #8: coverage-vs-validity split, per-check negative controls, GFM pipe parsing; #8 kept as canonical, docs-as-spec category not created); backend/common/llm/gateway-model-alias-defaults (#6 — subsumed by the generalized integrations/externally-owned-defaults; the model-alias case is one instance); backend/common/integrations/llm-response-completeness (#12 — folded into llm/completion-response-validation, kept in llm/ for category coherence with context-window-budget).
## [2026-08-04] ingest | knowledge-flush of 3 queued insights. New: testing/quality/schema-additions-under-a-golden-gate (a new node kind absent from the golden example is untouched by every mutant derived from it — commit a minimal conforming fixture, one negative per keyword the new branch adds, restore each and require red; judge the suite by whether anything loads the schema at all), backend/common/change-impact/call-site-enumeration (NEW category change-impact: enumerate by callee name, treat a parameter-name search as a partial index — positional-or-keyword is Python's default parameter kind and ast.Call keeps args/keywords in separate fields; re-run the enumeration after the edit). Merged: testing/data/test-data-and-isolation +1 Do row / +1 edge case / +1 Instead-of (a factory whose shape depends on a value the test also passes to the code under test takes it as a parameter, never as a module-level default). All cited URLs live-verified this session; one local reproduction (Python 3.14.6: keyword search 1 hit vs AST callee enumeration 4).
## [2026-08-04] ingest | knowledge-flush: 3 queued session insights merged into existing pages (no new pages — merge-before-create). platforms/environment/path-resolution +keg-only case: a package manager reports a tool installed but `command -v`/`which` find nothing because keg-only/unlinked formulae (llvm, openssl, curl) are deliberately off PATH — run from `$(brew --prefix)/opt/<f>/bin` (verified: `which mlir-opt` not found vs `/opt/homebrew/opt/llvm/bin/mlir-opt` → LLVM 22.1.8; source docs.brew.sh/FAQ). platforms/processes/non-interactive-cli-invocation +bracketed-paste edge: injecting a long/multiline prompt into a REPL (tmux `send-keys -l`) stalls at `❯ [Pasted text #1]` because the input is one bracketed-paste block (ESC[200~…201~) whose embedded newline is not submit — send Enter as a separate keystroke a beat later (source en.wikipedia.org/wiki/Bracketed-paste + claude-code#43169). testing/strategy/test-level-choice +import-side-effect edge/instead-of: a "pure" function's test is not dependency-free if its module runs I/O at import; `@pytest.mark.skipif` evaluates after the module import so it can't gate it — use `importorskip`/`skip(allow_module_level=True)` or move the function to a side-effect-free module (source docs.pytest.org skipping). Harvested "infrastructure" hint for the keg-only insight re-routed to platforms/environment (dedicated PATH page). Confidence: keg-only & pytest verified vs official docs; bracketed-paste mechanism doc-verified, the claude-CLI submit specifics field-tested.
## [2026-08-06] ingest | knowledge-flush of 6 queued insights (1 new page, 5 merges; 1 candidate dropped as tool-specific). New: platforms/shells/warning-only-diagnostics (tools that warn on stderr while exiting 0 — branch on captured stderr not exit code, `2>&1 >/dev/null` order per POSIX 2.7, PostToolUse exit-2 forwarding; verified vs code.claude.com/hooks + POSIX + local cc -Wall reproduction). Merged: platforms/shells/command-text-inspected-before-execution + gate-author step 6 (parse bare/single/double-quoted argument forms, expand only ~/$HOME/${HOME}; bats 12–13 re-run green 2026-08-06); platforms/filesystems/permissions-and-exec-bits + EDR/test-stub case (design callers to invoke helpers via `sh "$SCRIPT"` — POSIX "need not be executable", reproduced exit 0 vs 126); testing/quality/checks-that-cannot-pass + vacuously-green pre-implementation test case (expected failure identical to the unimplemented path's behavior → record vacuous green, prove post-green by guard mutation; t2-send-keys sed-mutation reproduction); testing/data/test-data-and-isolation + temp-artifact-accumulation attribution case (prefix histogram vs creation sites, then static cleanup guard observed red first; linkly 998-dir incident, tempfile docs); backend/common/reliability/timeouts-and-retries + client-throttle case (count token/auth interceptor requests in the throttle, stamp at actual send, init state must not satisfy spacing; stock-trader KIS cold-token incident — field-tested row). Dropped: dev-loop orchestrate worktree_escape escalation budgeting — specific to this repo's own orchestrate guardrails, no general wiki domain/category owns it; candidate retired, noted in INGEST_REPORT for owner follow-up (orchestrate skill docs are schema-layer, owner-approval only).
4 changes: 3 additions & 1 deletion wiki/backend/common/reliability/timeouts-and-retries.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ sources:
- https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
- https://sre.google/sre-book/addressing-cascading-failures/
- https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
last_verified: 2026-07-10
last_verified: 2026-08-06
related: [backend-common-api-design-idempotency, backend-common-llm-completion-response-validation]
---

Expand Down Expand Up @@ -63,6 +63,7 @@ retry storms, or requests that hang until a client gives up.
| Choosing the timeout value | Derive from the dependency's observed p99 latency plus headroom, not a guess; alert when timeouts actually fire so drift is visible |
| Batch/scheduled job calling an API in a loop | Same rules per call, plus one overall deadline per run so a hung run does not overlap the next schedule |
| Dependency is a DB with its own driver timeout | Set both the driver statement timeout and your outer deadline; the outer one must be the larger of the two, or you cancel work the DB has almost finished |
| The dependency enforces a requests-per-second cap and your client-side throttle still trips it on a fresh process's first call | Route every outbound HTTP request through the throttle — including token/credential issuance fired from header builders or auth interceptors (those are rate-limited requests too); stamp the throttle's timestamp immediately before the actual send; and check the initial state (`last_request_at = 0`) cannot satisfy the spacing check. An interceptor-issued token POST landing in the same second as the first API call exceeds a 2-req/s cap deterministically, but only on days the token cache is cold — which presents as an intermittent failure |

## Instead of

Expand All @@ -77,3 +78,4 @@ retry storms, or requests that hang until a client gives up.
- https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/ — timeouts on every remote call, backoff, jitter, idempotency precondition for retries
- https://sre.google/sre-book/addressing-cascading-failures/ — deadlines and propagation, retry budgets, per-request retry limits, failing fast under overload
- https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ — jittered backoff outperforms plain exponential backoff; full-jitter formula
- Field incident 2026-08-06 (`stock-trader`, KIS API, 2-req/s cap; this row is field-tested): `_headers()` called `_throttle()` and then `_get_token()`, so the token POST bypassed the throttle; provider logs on the two token-issuance days showed token POST at :00.354, issuance at :00.495, and the first API call rate-limited at :00.543, while cache-valid days ran clean — misdiagnosed as intermittent until the interceptor path was counted
2 changes: 1 addition & 1 deletion wiki/backend/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Match your situation to a "load when" line; load only matching pages.

| Page | Load when |
|------|-----------|
| [timeouts-and-retries](common/reliability/timeouts-and-retries.md) | Your service calls another service/external API/DB over the network — setting timeouts and deadlines, deciding what to retry per failure type, backoff/jitter, capping concurrency against a slow dependency; debugging pool exhaustion or retry storms |
| [timeouts-and-retries](common/reliability/timeouts-and-retries.md) | Your service calls another service/external API/DB over the network — setting timeouts and deadlines, deciding what to retry per failure type, backoff/jitter, capping concurrency against a slow dependency; debugging pool exhaustion or retry storms; a client-side throttle that still trips the provider's per-second rate limit (auth/token calls bypassing it) |

### caching

Expand Down
5 changes: 4 additions & 1 deletion wiki/platforms/filesystems/permissions-and-exec-bits.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ sources:
- https://man7.org/linux/man-pages/man7/inode.7.html
- https://docs.docker.com/engine/containers/run/
- https://docs.docker.com/engine/storage/bind-mounts/
last_verified: 2026-07-10
last_verified: 2026-08-06
related: [platforms-filesystems-paths-case-and-line-endings]
---

Expand Down Expand Up @@ -49,6 +49,7 @@ channels that preserve modes.
| You need a mode other than 755/644 tracked in git | git tracks only executable-or-not (`100755`/`100644`) — enforce fuller modes (setgid, 600 secrets) in a deploy/entrypoint step, not via git |
| Exec bit committed but Windows-checkout users still can't run it | Windows doesn't consume the POSIX exec bit; invoke via the interpreter there. Line-ending/casing breakage on the same journey: [platforms-filesystems-paths-case-and-line-endings] |
| Rootless Docker / userns-remap in play | uids are remapped, so host-uid matching arithmetic changes — verify with `ls -ln` on the host and `id` inside the container before choosing `--user` |
| An endpoint-security agent (EDR, e.g. SentinelOne) flags `chmod +x` on temp or test-injected files, or tests must swap in stub scripts via an env var | Design the calling code to invoke the helper as `sh "$SCRIPT"` (POSIX: the command file "need not be executable") — stubs are then plain read-only files needing no `chmod`, and the same call keeps working when a distribution path (plugin cache, artifact store) strips modes |

## Instead of

Expand All @@ -57,6 +58,7 @@ channels that preserve modes.
| `chmod 777` to make a permission error go away | Identify WHICH user/process needs WHICH access (`ls -ln` + the failing process's uid) and grant exactly that — owner change, group+setgid, or `--user` | 777 is an incident deferred: any local user/process can now modify or replace the file |
| `chmod +x` locally and moving on | `git update-index --chmod=+x <file>` and commit | The local bit doesn't reach the repo; every fresh clone and CI run re-breaks |
| Running the container as root because the mount "just works" that way | `--user` matching the host owner, or entrypoint `chown` | Root-in-container writes root-owned files onto the host and widens container-escape blast radius |
| `chmod +x` a test stub so the code under test can exec it | Have the code call helpers via `sh "$SCRIPT"` and inject the stub as a plain file | The exec-bit requirement is a property of the caller's invocation style; interpreter invocation removes it for stubs and deployed copies alike, and avoids EDR rules that treat `chmod +x` on fresh files as malicious |

## Sources

Expand All @@ -66,3 +68,4 @@ channels that preserve modes.
- https://man7.org/linux/man-pages/man7/inode.7.html — setgid on a directory: files created inside inherit the directory's group
- https://docs.docker.com/engine/containers/run/ — container default user is root (uid 0); `--user`/`-u` overrides with `uid:gid`
- https://docs.docker.com/engine/storage/bind-mounts/ — bind-mount mechanics (host uid/gid visibility rows are field practice, not stated on this page)
- https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html — `sh command_file`: "the file need not be executable". Reproduced 2026-08-06 (macOS zsh): `sh` on a mode-644 script ran (exit 0); direct `./` invocation → "permission denied" (exit 126). EDR/test-stub row is field practice: 8 stub scripts injected without `chmod` via an env-var seam, 331/331 bats green (dev-loop `watch-status.sh`, 2026-08-05)
5 changes: 3 additions & 2 deletions wiki/platforms/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ Match your situation to a "load when" line; load only matching pages.
| Page | Load when |
|------|-----------|
| [portable-shell-scripts](shells/portable-shell-scripts.md) | Writing a shell script that must run on more than one machine/OS/shell or in CI; a script that works locally fails elsewhere; choosing a shebang (bash vs sh); a bash script misbehaves in zsh or vice versa (unquoted vars, `=word`, array indexing); deciding how `set -euo pipefail` protects (and doesn't); building argument lists safely |
| [command-text-inspected-before-execution](shells/command-text-inspected-before-execution.md) | A hook, policy gate, allow-list, or audit rule blocked a command that is correct as written; composing a command that must satisfy such a gate first try; deciding whether to write a path literally or as `"$VAR"` in an inspected argument; a gate reports an argument missing or a file nonexistent though both are right; a gate must read a file your command creates; prose containing a dangerous-looking command (release notes, docs, fixtures) trips a text scanner |
| [command-text-inspected-before-execution](shells/command-text-inspected-before-execution.md) | A hook, policy gate, allow-list, or audit rule blocked a command that is correct as written; composing a command that must satisfy such a gate first try; **authoring** such a gate's argument-extraction pattern (quoted forms, `~`/`$HOME` expansion); deciding whether to write a path literally or as `"$VAR"` in an inspected argument; a gate reports an argument missing or a file nonexistent though both are right; a gate must read a file your command creates; prose containing a dangerous-looking command (release notes, docs, fixtures) trips a text scanner |
| [warning-only-diagnostics](shells/warning-only-diagnostics.md) | Wiring a compiler/linter/validator into a feedback hook or CI gate when the tool prints warnings to stderr but exits 0; a PostToolUse-style hook stays silent though the tool printed diagnostics; choosing the redirection order to capture stderr without the tool's stdout product |

## tools

Expand All @@ -37,7 +38,7 @@ Match your situation to a "load when" line; load only matching pages.
| Page | Load when |
|------|-----------|
| [paths-case-and-line-endings](filesystems/paths-case-and-line-endings.md) | A repo moves between macOS/Windows/Linux and files disappear or collide; an import resolves locally but fails on Linux CI (casing); renaming only the case of a file; diffs show every line changed or a script dies with `bad interpreter: ^M` (CRLF); setting up `.gitattributes` line-ending policy; generating file names or paths that must be valid on Windows (reserved names, path length) |
| [permissions-and-exec-bits](filesystems/permissions-and-exec-bits.md) | "Permission denied" running a script that exists; a script loses its executable bit through git/Windows/zip/CI artifacts; surprise file-mode diffs in git (`core.fileMode`); docker bind-mount files root-owned or unreadable (host/container uid mismatch); pipeline stages can't read each other's artifacts (umask); setting up a shared directory for several users/daemons; reviewing file-permission handling in a repo or pipeline |
| [permissions-and-exec-bits](filesystems/permissions-and-exec-bits.md) | "Permission denied" running a script that exists; a script loses its executable bit through git/Windows/zip/CI artifacts; surprise file-mode diffs in git (`core.fileMode`); docker bind-mount files root-owned or unreadable (host/container uid mismatch); pipeline stages can't read each other's artifacts (umask); setting up a shared directory for several users/daemons; tests must inject stub scripts where an EDR flags `chmod +x` or a distribution path strips modes; reviewing file-permission handling in a repo or pipeline |

## processes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ confidence: verified
sources:
- https://code.claude.com/docs/en/hooks
- https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
last_verified: 2026-07-30
related: [platforms-shells-portable-shell-scripts, platforms-environment-path-resolution]
last_verified: 2026-08-06
related: [platforms-shells-portable-shell-scripts, platforms-environment-path-resolution, platforms-shells-warning-only-diagnostics]
---

# Commands Read as Text by a Gate Before the Shell Runs Them
Expand Down Expand Up @@ -58,6 +58,16 @@ on the first attempt.
your exact command string before rewriting anything else — one run tells you
whether you are in the missing-argument or nonexistent-file mode above.

6. **When you author the gate, accept all three shell quoting forms and expand
only prefixes the gate can resolve from its own environment.** Correct shell
style quotes paths, so an extraction pattern that excludes quote characters
(`[^ '"]+`) denies exactly the well-formed commands, with a misleading
missing-argument error. Parse the argument bare, single-quoted, and
double-quoted (POSIX 2.2 defines only these three forms), expand `~`, `$HOME`,
and `${HOME}` against the gate's own environment, and state in the gate's
error message that any other variable must be written as a literal path.
Prove the parser with one regression test per form before relying on it.

## Edge cases

| Case | Then |
Expand All @@ -76,6 +86,7 @@ on the first attempt.
| Assume a blocked command means the deliverable is wrong | Reproduce the gate's extraction pattern against your literal command string first | A quoting-level extraction failure and a genuinely incomplete deliverable produce the same refusal, so fixing content wastes the round |
| Build the file the gate checks with a heredoc in the same command | Write it in a prior command and reference the path | The gate is evaluated before execution, so the file is absent at decision time |
| Reword prose to get a dangerous-looking string past a scanner | Put the prose in a file and pass `--notes-file`/`--body-file` | Editing meaning to satisfy a text scanner degrades the artifact; a file is not scanned as a command |
| Extract a gate's file argument with a bare-token pattern like `[^ '"]+` | Parse bare, single-quoted, and double-quoted forms and expand `~`/`$HOME`/`${HOME}` yourself (step 6) | The quote characters callers are taught to use land inside the match window, so the extractor returns empty and the gate reports a present argument as missing |

## Sources

Expand All @@ -92,3 +103,10 @@ missing), `--body-file $REPO/…` → literal `$REPO/…` (blocked as nonexisten
`--body-file /abs/…` and `--body-file=/abs/…` extracted correctly. A same-command
heredoc body-file was separately blocked as not-yet-existing until moved to a
preceding call.

Gate-author side reproduced 2026-08-05/06 on the same gate: the original
bare-token extractor denied a double-quoted `--body-file` path as missing;
after adding quoted-form parsing plus `~`/`$HOME`/`${HOME}` expansion, the bats
regressions covering a double-quoted path and a `$HOME`-prefixed path
(`tests/pre-flush-pr-gate.bats` tests 12–13) went red-then-green and still pass
(re-run 2026-08-06).
2 changes: 1 addition & 1 deletion wiki/platforms/shells/portable-shell-scripts.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ sources:
- https://google.github.io/styleguide/shellguide.html
- https://www.shellcheck.net/
last_verified: 2026-07-10
related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management, platforms-shells-command-text-inspected-before-execution]
related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management, platforms-shells-command-text-inspected-before-execution, platforms-shells-warning-only-diagnostics]
---

# Shell Scripts That Must Run on More Than One Machine or Shell
Expand Down
Loading
Loading