choiyounggi · choiyounggi · Aug 4, 2026 · Aug 4, 2026
@@ -10,10 +10,10 @@ follow the cross-pointers in their index or take the next matching seeded domain
 | Domain | Status | Route here when |
 |--------|--------|-----------------|
 | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior |
-| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
+| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, client-side rate limiting, crawl-source selection via robots.txt, object-storage references, migrating call sites through a signature change) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
 | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility |
 | [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) |
-| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) |
+| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests, proving tests/harnesses/predicates can report the failing verdict — including differential-run agreement and background-work completion predicates (release-process quality → qa) |
 | [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing, automated verification of document deliverables (spec/RFC gates) (writing automated test code → testing) |
 | [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) |
 | [security](wiki/security/index.md) | **seeded** | Trust-boundary decisions: input validation, session-vs-token auth choice, per-resource authorization (IDOR), secrets hygiene, dependency trust, PII handling (XSS rendering → frontend; CI secrets → infrastructure; JWT implementation → backend/frontend auth) |

@@ -37,3 +37,6 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-08-03] ingest | Consolidated review of knowledge PRs #6–#13 (8 fork PRs) into 12 pages. New: backend/common/llm (completion-response-validation, context-window-budget), backend/common/integrations (externally-owned-defaults), backend/common/storage (object-key-persistence), infrastructure/containers/host-cgroup-visibility, infrastructure/observability/missing-container-metrics, platforms/environment/unicode-text-matching, platforms/shells/command-text-inspected-before-execution, platforms/processes/non-interactive-cli-invocation, qa/document-verification (spec-document-gates, editing-a-gated-document), testing/quality (checks-that-cannot-pass, spec-artifact-checks, harness-reverse-controls). All cited URLs are per-PR live-verified; three pages were reconciled from two overlapping PR versions each (see revise/dedup entries below).
 ## [2026-08-03] revise | Reconciled 3 pages from overlapping PR pairs, taking the more complete/better-sourced body and folding in the other's unique cases: backend/common/llm/completion-response-validation (#12 body — tool_calls/function_call carve-out, streaming, Responses API status==incomplete, "reasoning is scratch, not deliverable" — kept in llm/ per #6/#13 category, folded in #6's DeepSeek-first-party edge + the 8,173-char reasoning_content field incident); backend/common/integrations/externally-owned-defaults (#12 generalized body — any repo-external resource — folded in #6's LiteLLM-alias-removed field incident + gateway-config-vs-live-upstream nuance); platforms/processes/non-interactive-cli-invocation (#12 body — GNU-nohup extension precision, ssh -n stdin-detach vs BatchMode, pre-log DNS/TLS/proxy + curl -v — folded in #11's DEBIAN_FRONTEND, pager/color TTY case, wrapper-CLI case, and the no-request-in-gateway-log field incident).
 ## [2026-08-03] dedup | Dropped 3 candidate pages as duplicates/superseded during the #6–#13 consolidation: testing/docs-as-spec/document-conformance-checks (#9 — same case as testing/quality/spec-artifact-checks from #8: coverage-vs-validity split, per-check negative controls, GFM pipe parsing; #8 kept as canonical, docs-as-spec category not created); backend/common/llm/gateway-model-alias-defaults (#6 — subsumed by the generalized integrations/externally-owned-defaults; the model-alias case is one instance); backend/common/integrations/llm-response-completeness (#12 — folded into llm/completion-response-validation, kept in llm/ for category coherence with context-window-budget).
+## [2026-08-04] ingest | 7 queued insights researched and routed: 5 new pages (testing/quality/differential-run-agreement — an agreement verdict from a differential run is scoped to inputs where the unmodelled state does not decide, forcing-input table, RIP grounding; testing/quality/completion-predicates — prove the monitor reports not-done first, count-equality over absence, `grep -F` for markers carrying regex metacharacters; backend/common/integrations/robots-txt-and-source-selection — obey the group matching your product token, 4xx=unavailable vs 5xx=complete disallow per RFC 9309, republishing-portal fallback; backend/common/reliability/client-side-rate-limiting — count the token request against the quota, throttle at the transport layer, zero-initialized last-request timestamp disables the first wait; NEW category backend/common/refactoring + signature-change-call-sites — enumerate by callee, positional callers are invisible to a parameter-name search, migrate test helpers first) and 2 amendments (tests-that-cannot-fail: per-assertion mutation granularity with a three-outcome table, inherited-shared-contract never-fails row; harness-reverse-controls: a monkeypatched control must target a seam the path consults at call time — a caller that resolves a defaulted value itself makes the callee's default dead, and the control reports green having mutated nothing).
+## [2026-08-04] revise | Two candidate claims disproved during source verification and corrected before ingest: (1) "기관 자체 사이트는 전면 크롤링 거부" — www.i-sh.co.kr's `User-agent: *` group disallows a path list only; the `Disallow: /` belongs to 15 named groups (GPTBot, ClaudeBot, CCBot, aiohttp, …), so the page's directive became "read the group your product token matches" rather than "the origin blocks crawling"; (2) "www.gh.or.kr robots.txt is `Disallow: /`" — it returns HTTP 410, which RFC 9309 classifies as unavailable ("the crawler MAY access any resources on the server"), the opposite of a refusal. Both live-fetched 2026-08-04.
+## [2026-08-04] ingest | 2 further insights queued mid-flush, folded into the same PR as amendments (no new pages): testing/async/async-testing gains the stream-consumer row — drive a readline/line-delimited reader from a test double one record per macrotask turn (`setImmediate` between writes) and share one reader instance across prompts, plus two edge cases (hang after the first record; a reader rebuilt per prompt) and an Instead-of row; testing/data/test-data-and-isolation + platforms/filesystems/permissions-and-exec-bits gain the executable-fixture rows — set the mode at creation (`writeFileSync(p, body, {mode: 0o755})`, `install -m 755`) inside a gitignored build-output directory rather than `chmod +x` under the system temp dir, with the creation-only caveat (a rewrite of an existing path keeps the original mode). Both reproduced locally on Node v25.8.1: batched `input.write('one\ntwo\nthree\n')` yielded `["one"]` then hung while one-line-per-turn yielded all three; `writeFileSync` mode 0o755 → 755, second write with 0o644 → still 755. The EDR-heuristic rationale is labelled operational practice, not vendor-documented.
@@ -0,0 +1,90 @@
+---
+id: backend-common-integrations-robots-txt-and-source-selection
+domain: backend
+category: integrations
+applies_to: [general]
+confidence: verified
+sources:
+  - https://www.rfc-editor.org/rfc/rfc9309.html
+  - https://docs.aiohttp.org/en/stable/client_reference.html
+last_verified: 2026-08-04
+related: [backend-common-integrations-externally-owned-defaults, backend-common-reliability-timeouts-and-retries]
+---
+
+# Choosing a Source to Crawl by Reading robots.txt
+
+## When this applies
+
+You are picking which site to fetch a published dataset from (notices, listings,
+filings, prices) and are reading its `/robots.txt` to decide whether your client
+may crawl it. Also when a file you read as a blanket refusal contains a
+`Disallow: /`, or when the file did not return 200.
+
+## Do this
+
+1. **Select the group whose product token matches your client, then obey only
+   that group.** RFC 9309: "Crawlers MUST use case-insensitive matching to find
+   the group that matches the product token and then obey the rules of the
+   group"; the `*` group applies only "If no matching group exists". When several
+   groups match your token, "the matching groups' rules MUST be combined into one
+   group".
+2. **Attribute every `Disallow: /` you see to the group it sits under before
+   drawing a conclusion.** A file can leave `User-agent: *` open to everything
+   except a few admin paths while giving named crawlers a full block. Read the
+   file as a list of groups, not as a list of rules.
+3. **Look up your own User-Agent string in the file.** The product token "should
+   appear as a substring in the crawler's user-agent header", so a client running
+   on its library's default UA carries that library's name as a token. Set an
+   explicit User-Agent that identifies you and a contact, then search the file for
+   that token as well as for your HTTP library's name.
+4. **Branch on the response status, not just the body:**
+
+| robots.txt response | Do |
+|---------------------|-----|
+| 2xx with a parseable body | Follow the rules of your matching group |
+| 3xx | Follow up to five consecutive redirects; past five, treat the file as unavailable |
+| 4xx (404, 410, and the rest) | Treat the file as unavailable — the crawler "MAY access any resources on the server" |
+| 5xx, connection or DNS failure | Assume complete disallow while the failure persists |
+| 5xx continuing past 30 days | Treat as unavailable (4xx handling) |
+
+5. **When your matching group disallows the path, enumerate the other publishers
+   of the same records before concluding the data is unreachable.** A source
+   organization and the portal that republishes its records are separate
+   authorities, so each serves its own robots.txt and the two can differ. Fetch
+   the portal's file, match your token against it, and compare record identity
+   (same notice id, same fields) before substituting one source for the other.
+6. **Apply the most-specific-match rule inside the chosen group**: "The most
+   specific match is the match that has the most octets", and an `allow` that ties
+   a `disallow` wins.
+7. **Re-fetch robots.txt at least daily.** Crawlers "SHOULD NOT use the cached
+   version for more than 24 hours", so a long-running collector re-reads before
+   each run rather than pinning a value read at deployment.
+8. **Settle the terms-of-service and licensing question separately.** robots.txt
+   states crawler access rules; record which document grants you the right to
+   store and redistribute the content, and cite it next to the source config.
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| The subdomain differs from the marketing site (`apply.`, `housing.`, `open.`) | Fetch each host's own robots.txt — the file is per-authority (scheme, host, port), so a sibling host's rules say nothing about this one |
+| Your matching group is a named block and the `*` group is open | Follow your named group; the specific match is what RFC 9309 requires you to obey |
+| The site publishes an official API or bulk download for the same records | Take that path and record it in the source config; it removes the crawl question entirely |
+| The permitted portal renders the table server-side with a stable detail id | Key your stored records on that id, so re-collection is idempotent rather than position-based |
+| robots.txt is unreachable intermittently | Hold the previous rules and retry with backoff ([backend-common-reliability-timeouts-and-retries]); treat a sustained failure as complete disallow |
+| The permitted portal's default is one your repo does not own (host, path, id scheme) | Add a startup check that the source still resolves and still returns the expected shape ([backend-common-integrations-externally-owned-defaults]) |
+
+## Instead of
+
+| If you are about to | Do this instead | Why |
+|---------------------|-----------------|-----|
+| Conclude "this site blocks crawling" from a `Disallow: /` seen anywhere in the file | Find the group your product token matches and read that group's rules | The blanket block frequently belongs to named AI/library crawler groups while `User-agent: *` is open except for a handful of admin paths |
+| Ship a collector on its HTTP library's default User-Agent | Set an explicit UA naming your crawler and a contact, then check that token against the file | Sites publish groups naming library tokens; the library default can place you in a fully disallowed group you never looked for |
+| Treat a 404 or 410 on robots.txt as a refusal | Treat 4xx as unavailable and proceed under the site's other terms | RFC 9309 assigns "MAY access any resources" to unavailable status; 5xx is the status that means complete disallow |
+| Give up on a dataset because the originating agency's site restricts your token | Check the portals that republish the same records and compare record identity | Republishing portals exist to distribute the records and commonly permit what the origin restricts |
+
+## Sources
+
+- https://www.rfc-editor.org/rfc/rfc9309.html — group selection ("Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group"; `*` only "If no matching group exists"; "If there is more than one group matching the user-agent, the matching groups' rules MUST be combined into one group"); the product token contains "only uppercase and lowercase letters ('a-z' and 'A-Z'), underscores ('_'), and hyphens ('-')" and "should appear as a substring in the crawler's user-agent header"; access results ("If a server status code indicates that the robots.txt file is unavailable to the crawler, then the crawler MAY access any resources on the server"; unreachable "means the robots.txt file is undefined and the crawler MUST assume complete disallow"; five-redirect limit; 30-day rule); "The most specific match found MUST be used. The most specific match is the match that has the most octets"; "Crawlers SHOULD NOT use the cached version for more than 24 hours"
+- https://docs.aiohttp.org/en/stable/client_reference.html — aiohttp autogenerates a `User-Agent` header when none is passed, so a client that sets no UA advertises the library name; `headers` on `ClientSession` sets an explicit one
+- Live observation 2026-08-04 (Korean public-housing notices, fetched directly): `www.i-sh.co.kr/robots.txt` gives `User-agent: *` a path list only (`/admin`, `/cert`, `/upload`, `/gcms/brd`, per-district board paths) and reserves `Disallow: /` for named groups — `GPTBot`, `ChatGPT-User`, `facebookexternalhit`, `BaiDuSpider`, `MJ12bot`, `OAI-SearchBot`, `PerplexityBot`, `Google-Extended`, `ClaudeBot`, `Claude-SearchBot`, `meta-externalAgent`, `Applebot-Extended`, `CCBot`, `aiohttp`, `DuckDuckBot`. `housing.seoul.go.kr/robots.txt` is `User-agent: * / Allow: /` and republishes the same notices with a server-rendered table and a stable detail id; `apply.gh.or.kr/robots.txt` is `User-agent: * / Allow: /*`; `www.gh.or.kr/robots.txt` returns HTTP 410, which RFC 9309 classifies as unavailable rather than as a refusal