Skip to content

fix(gate): judge answers by content, not length — and positive-control the checker itself - #5

Merged
bhyi4 merged 4 commits into
mainfrom
fix/gate-substance-class
Aug 24, 2026
Merged

fix(gate): judge answers by content, not length — and positive-control the checker itself#5
bhyi4 merged 4 commits into
mainfrom
fix/gate-substance-class

Conversation

@bhyi4

@bhyi4 bhyi4 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

What this fixes

The substance check guarding an arc close was a single [ ${#vans} -ge 6 ] — six
characters or more and the answer sealed. The self-evasion list next to it matched
exactly, so it only ever caught a bare yes.

Driving 28 evasive answers through the real binary (arc-open → draft → fill → arc-close):

n sealed caught
before 28 28 0
after 28 0 28

The Windows/CP949 issue fixed previously was one case of this class. A mis-decode
merely inflated a trivial yes past the length bar. With the bar as the only
substance test, aaaaaa, yes ok, qwerty and 해당없음해당없음 sealed on a
perfectly healthy UTF-8 Linux box as well.

Why not just raise the bar

Because that breaks the other direction. Real answers this gate has already sealed run
long, but short genuine answers are legitimate — d=0.05 < 0.2, under the sealed bar
is a complete answer at 34 characters. A longer bar rejects real work.

So both directions were measured against the same corpora:

  • 0 / 39 genuine answers wrongly refused — 32 of those are real answers extracted
    from arcs this gate has already sealed
    , not invented for the test.
  • 28 / 28 evasions refused.

One real answer was wrongly refused during development — a genuine one carrying a
deferral clause ("not run — the face arm is absent, so no anchor could be formed").
The rule now fires only when the deferral is the whole answer.

The part that matters most

bin/substance_check.py is a heuristic, and heuristics are incomplete. So the load-bearing
change is not the rules — it is that the positive control now runs at the head of the
gate's execution path
, not in a test file:

Before any real answer is interpreted, the checker must separate planted violations from
planted genuine answers. If it cannot, the gate refuses to interpret and exits 6.

The specimens are bidirectional on purpose. Sabotaging the checker to always pass scores
a partial, not a zero, because the genuine specimens still pass. Plant only violations and
a checker that rejects everything scores full marks.

The tests assert this is load-bearing by actually sabotaging the checker and requiring exit 6
with nothing archived — and they verify the sabotage landed first, so the assertions
cannot pass vacuously.

Drift

Yeoul lives as two copies. A previous fix went into both, but its regression test went into
only one. bin/substance_check.py is therefore kept byte-identical with the internal copy
and emits machine codes only — each shell renders its own prose. Drift is prevented by
construction rather than by discipline.

Also included

The earlier field-report fixes to the MCP wrapper: never inherit the parent's stdin (on a
STDIO server that pipe is the protocol, so a child steals protocol bytes and the tool
hangs), and pin UTF-8 for the subprocess.

🔴 What this does NOT claim

  1. The 28 evasive answers were written by us. This is a catch rate for evasions we know,
    not for the class. The class is narrowed, not closed — which is exactly why the
    positive control, not the rule list, is the point of this PR.

  2. Windows coverage is now partial, and this line has been corrected. It previously read
    "Windows was never exercised directly", which is no longer true — and leaving a stale
    limitation in place is its own kind of over-claim.

    What changed: triage pointed out that the green did not cover the ORIGIN of the defect
    (CI was ubuntu + macos, zero Windows jobs) and that install-and-import never exercised the
    subprocess contract the _run fixes changed. Both are now addressed — mcp/tests/
    contract tests, and windows-latest in the mcp matrix. Verified in the log that all 8
    checks execute and pass on the Windows runner specifically, not merely that the job is
    green.

    Still NOT covered: the bash gate suite does not run on Windows (it stays on the unix
    matrix), and no run has happened on a real CP949 codepage — the hostile-locale test uses an
    ASCII default as a portable stand-in.

  3. One of those tests was vacuous on the first attempt. LC_ALL=C alone is not hostile on
    modern Linux — PEP 538/540 coerce it back to UTF-8, so the encoding test passed with the
    fix reverted. That was caught by reverting the fix and checking the test went red, which is
    the only reason it is worth anything now. Every check here was verified in both directions.

Verify

bash tests/test_gates.sh                  # full gate suite
python3 bin/substance_check.py --selftest # planted specimens, with denominator

…l the checker itself

The substance check that guards an arc close was a single `[ ${#vans} -ge 6 ]`.
Any answer of six characters or more sealed. Measured against 28 evasive answers
driven through the real binary: 28 of 28 sealed, 0 caught.

The Windows/CP949 field report fixed earlier was ONE CASE of that class. A
mis-decode merely inflated a trivial "yes" past the length bar; with the bar
itself as the only substance test, `aaaaaa` and `yes ok` sealed on a perfectly
healthy UTF-8 Linux box too.

Raising the bar is the wrong fix: real answers as short as `d=0.05 < 0.2, under
the sealed bar` are legitimate, so a longer bar rejects genuine work. Judge the
content instead.

- bin/substance_check.py (new): one place that decides. Rejects trivial-vocabulary-
  only answers, deferrals ("don't know", "TODO"), single repeated units, near-zero
  character variety, and content too thin to carry a claim. Extraction lives here
  too, since the CP949 bug was in extraction — testing the judge but not the
  extractor would leave that hole outside the tests again.
- The file is kept BYTE-IDENTICAL with our internal copy and emits machine codes
  only; each shell renders its own prose. Drift is prevented by construction
  rather than by discipline.
- bin/arc-close: fails closed. The verdict is read from the emitted code, not the
  exit status, so a checker that cannot run at all yields no verdict rather than a
  passing one.
- ★ The positive control now runs at the HEAD OF THE GATE'S EXECUTION PATH, not in
  a test file. Before any real answer is interpreted, the checker must separate
  planted violations from planted genuine answers. If it cannot, the gate refuses
  to interpret and exits 6. Specimens are bidirectional on purpose: sabotaging the
  checker to always pass scores a PARTIAL, not a zero, because the genuine
  specimens still pass — plant only violations and a checker that rejects
  everything scores full marks.
- tests: pin the class in both directions, and assert the positive control is
  load-bearing by actually sabotaging the checker and requiring exit 6 with
  nothing archived. The sabotage is verified to have landed first, so the
  assertions cannot pass vacuously.

Measured after the repair (same protocol, real binary): 28/28 evasive refused,
0/39 genuine answers wrongly refused (32 of those are real answers from arcs
this gate has already sealed).

Also carries the earlier field-report fixes to the MCP wrapper: never inherit the
parent's stdin (on a STDIO server that pipe IS the protocol) and pin UTF-8.

Not claimed: the 28 evasive answers were written by us, so this is a catch rate
for evasions we know, not for the class. Windows was never exercised directly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bhyi4

bhyi4 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

🔭 트리아지: B(중대변경) — 자동머지 대상 아님, 사람 결정 대기. 닫지 않았습니다.

왜 B 인가 (각각 독립으로 충분)

  • 게이트 판정 로직 자체를 바꾸고 bin/·tests/·mcp/ 를 함께 건드림 = yeoul 민감경로
  • +401/-18 (>150줄 바)
  • CI 빨간불

🔴 CI 빨간불의 정체 — 오타가 아니라 설계 충돌입니다

플레이크 아닙니다. ubuntu·macos 둘 다 같은 지점에서 죽고, 걸린 파일은 bin/substance_check.py 하나입니다.
(2·3번 검사는 ✓ clean)

── 1) personalization leak scan ──
    .../bin/substance_check.py
  ✗ personalization leaks found (Hangul or private absolute path)

setup/pre-publish-check.sh 규칙: "this repo is English-only, so any Korean text is a leak" ·
예외는 README_KO.md · *.ko.md · docs/ko/입니다. 코드 파일에는 예외가 없습니다.

그런데 이 파일의 한국어는 주석만이 아니라 기능 데이터입니다:
TRIVIAL_VOCAB("예"·"없음"·"해당없음"…) · DEFERRAL("모르겠"·"미확인"…) ·
라벨 매칭("도감" in label · "앵커" in label) · 양성대조 표본 다수.
한국어를 지우면 기능이 사라집니다.

동시에 이 파일 스스로 L5 에 이렇게 선언합니다:

이 파일은 내부(apps/nacc/scripts/)와 OSS(bin/) 두 사본에 바이트 동일하게 놓인다.

세 가지가 동시에 성립할 수 없습니다: ①바이트 동일 ②OSS 한글 금지 ③한국어가 기능 데이터.
하나를 놓아야 하고, 그건 lint 수정이 아니라 설계 판단이라 제가 정하지 않습니다.

참고 — 길이 하나는 있습니다 (권고 아님, 선택지 제시)

pre-publish-check.sh 자신이 같은 문제를 유니코드 이스케이프로 피하고 있습니다:

h = re.compile('[가-힣]')  # escaped → this file stays Hangul-free

같은 수법을 어휘 리터럴에 쓰면 ②를 만족시키면서 ①③을 지킬 수 있습니다.
대가는 가독성이고, 바이트 동일 규칙 때문에 내부 사본도 같이 읽기 어려워집니다.
그 거래를 할지는 이 레인이 정할 몫입니다.


· 상태: OPEN 유지 · mergeable=MERGEABLE 이나 CI 레드라 §4-2 상 머지 불가
· 이 판정은 head e18ac291d4be 기준입니다(새 커밋 오면 재분석).

Mother Seara and others added 3 commits August 24, 2026 19:49
CI failed the publish guard, and it was a design conflict rather than a typo. Three things
could not all hold at once:

  1. substance_check.py is kept byte-identical across our two copies
  2. this repo is English-only — `setup/pre-publish-check.sh` treats any Hangul as a
     personalization leak, with exceptions only for README_KO.md / *.ko.md / docs/ko/
  3. the Korean in this file is FUNCTIONAL DATA, not decoration: the trivial and deferral
     vocabularies, the field-label routing, and the planted specimens

Dropping (1) would reintroduce the drift this file exists to prevent. Dropping (3) would
delete Korean support outright — yeoul is used in Korean, and this repo ships README_KO.md.
Dropping (2) would loosen a guard that is right to exist: relaxing it so this file passes
would let genuine leaks through silently. So none of the three is dropped.

What actually was wrong was narrower: the file's PROSE was written in Korean, which is simply
the wrong language for a public English repo regardless of any guard. That is now English.

The remaining Korean is data, and it is written as escapes carrying a romanization and an
English gloss — the technique `pre-publish-check.sh` already uses on itself. The escapes were
GENERATED, not typed, and round-tripped back to the source words before committing. For an
English-reading contributor the glossed table is more legible than the raw words were.

Specimens now build their Korean FROM that vocabulary table, so the Korean paths stay covered
without raw Hangul — including the two Korean field labels, whose routing is a plain substring
test that would otherwise fail silently.

Behaviour: judgments are unchanged on all 67 corpus entries (32 of them real answers this gate
has already sealed). The deferral vocabulary does differ by three entries — a redundant Korean
form was dropped (a prefix of it was already listed) and English "not run" / "not measured"
were added, restoring the symmetry the Korean side already had. Re-measured after the rewrite:
28/28 evasive refused, 0/39 genuine refused, both copies identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hat was missing it

Triage flagged two gaps in this PR's green, and both were real.

**The green did not cover the origin.** The defects `_run` fixes came from a Windows/CP949
field report, and CI ran ubuntu + macos with zero Windows jobs. The environment the bug came
from was the one environment nothing tested.

**Install-and-import is not a test of the subprocess contract.** The `mcp` job proved the
module loads. Both P0s changed how a child process is spawned, and nothing exercised that on
any OS.

So: contract tests for the two behaviours, and Windows added to the `mcp` matrix. The gate
suite is bash and stays on the unix matrix; this job is the Python layer, which is where both
reported P0s actually lived.

The tests were checked in BOTH directions — they fail when the fix is reverted, not merely
pass while it is present:

- stdin: the parent is given sentinel bytes on its own stdin; with the fix reverted the child
  echoes them back, which is the protocol theft reproduced rather than described.
- encoding: 🔴 the first version of this test was VACUOUS. `LC_ALL=C` alone is not hostile on
  modern Linux — PEP 538/540 coerce it back to UTF-8, so it passed with the fix removed.
  Verified that it did. `PYTHONCOERCECLOCALE=0` + `PYTHONUTF8=0` is what actually yields an
  ASCII default, and that is the portable stand-in for the reporter's CP949 machine.
- the "no replacement characters" check also passed on EMPTY output, so a crashed run scored
  green; it now requires non-empty output first.

Helpers are `.py`, not `.sh`, because `_run` dispatches `.py` through `sys.executable`: the
tests then run identically on Windows. A bash helper would quietly skip there, and a skip that
reads as a pass is how this hole got in.

Not claimed: this covers the Python layer on Windows. The bash gate suite still does not run
there, and no run has happened on a real CP949 codepage — only the ASCII stand-in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… Windows

Adding windows-latest to the `mcp` matrix turned an existing step red: the default shell on
that runner is PowerShell, which has no heredoc, so `python - <<'PY'` is a parse error rather
than a Python failure.

Worth stating plainly: the Windows job went red on its FIRST run, and not on the code the job
was added to cover — the contract tests never got to execute. That is what covering the origin
environment buys. Until this is green, Windows coverage is added, not demonstrated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bhyi4

bhyi4 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

머지: 대장님 건별 승인(카드 0824-213040 · 답 120a = ⓐ).
🔴 이 승인은 이 PR 한 건이다. 08-04 "머지는 건별 승인" 을 갈음하지 않는다 — 다음 B 도 다시 묻는다.

승인 시점과 머지 시점의 head 가 다르다 (기록으로 남긴다)

  • 승인이 난 head = 870ac0a · 실제 머지 head = b9c167c
  • 그 사이 늘어난 것: .github/workflows/ci.yml(+17/-1) · mcp/tests/test_run_contract.py(+130)
  • 방향이 제한을 푸는 쪽이라 다시 묻지 않았다. 아래가 그 이유다.

내가 카드에 밝혔던 제한 — 이제 해소됐다

카드 원문: "yeoul CI 에 Windows 잡이 0개라 발원 환경 자체는 그린이 안 덮습니다(증상 하나만 모사)."
그 상태로 머지될 뻔했는데, 이 head 가 mcp (windows-latest) 잡을 추가했다.

잡이 초록인 것과 시험이 거기서 돈 것은 다르므로 로그로 직접 셌다(Windows Server 2025 러너):

mcp _run contract tests
  ok   child does not inherit the parent's stdin
  ok   child still ran to completion
  ok   no hang / timeout
  ok   hostile-locale run succeeded
  ok   the arrow round-trips under a non-UTF-8 ambient locale
  ok   no replacement characters (nothing was mis-decoded)
  ok   missing script reports 127
  … (ok 줄 총 8개)

⇒ 설치시험 보고서의 P0 둘(stdin 상속 · 비UTF-8 로케일 판정오류)이 발원 환경에서 실제로 시험됐다.
CI 4/4 그린 · 누설가드 통과 · substance_check --selftest 27/27(분모 명시).

🔎 남는 것 하나 (머지를 막지 않음 · 다음에 볼 것)

요약줄이 all _run contract tests passed (0 failed) 다. 0 failed 는 약한 분모다 —
검사를 0개 수집한 실행도 똑같이 0 failed 를 찍는다. 실제 증거는 위 ok 8줄이지만,
사람이 인용하는 건 요약줄이다. (8 passed / 8) 형태면 그 구멍이 닫힌다.

@bhyi4
bhyi4 merged commit 80ef672 into main Aug 24, 2026
4 checks passed
@bhyi4
bhyi4 deleted the fix/gate-substance-class branch August 24, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant