fix(gate): judge answers by content, not length — and positive-control the checker itself - #5
Conversation
…l the checker itself
The substance check that guards an arc close was a single `[ ${#vans} -ge 6 ]`.
Any answer of six characters or more sealed. Measured against 28 evasive answers
driven through the real binary: 28 of 28 sealed, 0 caught.
The Windows/CP949 field report fixed earlier was ONE CASE of that class. A
mis-decode merely inflated a trivial "yes" past the length bar; with the bar
itself as the only substance test, `aaaaaa` and `yes ok` sealed on a perfectly
healthy UTF-8 Linux box too.
Raising the bar is the wrong fix: real answers as short as `d=0.05 < 0.2, under
the sealed bar` are legitimate, so a longer bar rejects genuine work. Judge the
content instead.
- bin/substance_check.py (new): one place that decides. Rejects trivial-vocabulary-
only answers, deferrals ("don't know", "TODO"), single repeated units, near-zero
character variety, and content too thin to carry a claim. Extraction lives here
too, since the CP949 bug was in extraction — testing the judge but not the
extractor would leave that hole outside the tests again.
- The file is kept BYTE-IDENTICAL with our internal copy and emits machine codes
only; each shell renders its own prose. Drift is prevented by construction
rather than by discipline.
- bin/arc-close: fails closed. The verdict is read from the emitted code, not the
exit status, so a checker that cannot run at all yields no verdict rather than a
passing one.
- ★ The positive control now runs at the HEAD OF THE GATE'S EXECUTION PATH, not in
a test file. Before any real answer is interpreted, the checker must separate
planted violations from planted genuine answers. If it cannot, the gate refuses
to interpret and exits 6. Specimens are bidirectional on purpose: sabotaging the
checker to always pass scores a PARTIAL, not a zero, because the genuine
specimens still pass — plant only violations and a checker that rejects
everything scores full marks.
- tests: pin the class in both directions, and assert the positive control is
load-bearing by actually sabotaging the checker and requiring exit 6 with
nothing archived. The sabotage is verified to have landed first, so the
assertions cannot pass vacuously.
Measured after the repair (same protocol, real binary): 28/28 evasive refused,
0/39 genuine answers wrongly refused (32 of those are real answers from arcs
this gate has already sealed).
Also carries the earlier field-report fixes to the MCP wrapper: never inherit the
parent's stdin (on a STDIO server that pipe IS the protocol) and pin UTF-8.
Not claimed: the 28 evasive answers were written by us, so this is a catch rate
for evasions we know, not for the class. Windows was never exercised directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🔭 트리아지: B(중대변경) — 자동머지 대상 아님, 사람 결정 대기. 닫지 않았습니다. 왜 B 인가 (각각 독립으로 충분)
🔴 CI 빨간불의 정체 — 오타가 아니라 설계 충돌입니다플레이크 아닙니다. ubuntu·macos 둘 다 같은 지점에서 죽고, 걸린 파일은
그런데 이 파일의 한국어는 주석만이 아니라 기능 데이터입니다: 동시에 이 파일 스스로 L5 에 이렇게 선언합니다:
⇒ 세 가지가 동시에 성립할 수 없습니다: ①바이트 동일 ②OSS 한글 금지 ③한국어가 기능 데이터. 참고 — 길이 하나는 있습니다 (권고 아님, 선택지 제시)
h = re.compile('[가-힣]') # escaped → this file stays Hangul-free같은 수법을 어휘 리터럴에 쓰면 ②를 만족시키면서 ①③을 지킬 수 있습니다. · 상태: OPEN 유지 · |
CI failed the publish guard, and it was a design conflict rather than a typo. Three things
could not all hold at once:
1. substance_check.py is kept byte-identical across our two copies
2. this repo is English-only — `setup/pre-publish-check.sh` treats any Hangul as a
personalization leak, with exceptions only for README_KO.md / *.ko.md / docs/ko/
3. the Korean in this file is FUNCTIONAL DATA, not decoration: the trivial and deferral
vocabularies, the field-label routing, and the planted specimens
Dropping (1) would reintroduce the drift this file exists to prevent. Dropping (3) would
delete Korean support outright — yeoul is used in Korean, and this repo ships README_KO.md.
Dropping (2) would loosen a guard that is right to exist: relaxing it so this file passes
would let genuine leaks through silently. So none of the three is dropped.
What actually was wrong was narrower: the file's PROSE was written in Korean, which is simply
the wrong language for a public English repo regardless of any guard. That is now English.
The remaining Korean is data, and it is written as escapes carrying a romanization and an
English gloss — the technique `pre-publish-check.sh` already uses on itself. The escapes were
GENERATED, not typed, and round-tripped back to the source words before committing. For an
English-reading contributor the glossed table is more legible than the raw words were.
Specimens now build their Korean FROM that vocabulary table, so the Korean paths stay covered
without raw Hangul — including the two Korean field labels, whose routing is a plain substring
test that would otherwise fail silently.
Behaviour: judgments are unchanged on all 67 corpus entries (32 of them real answers this gate
has already sealed). The deferral vocabulary does differ by three entries — a redundant Korean
form was dropped (a prefix of it was already listed) and English "not run" / "not measured"
were added, restoring the symmetry the Korean side already had. Re-measured after the rewrite:
28/28 evasive refused, 0/39 genuine refused, both copies identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hat was missing it Triage flagged two gaps in this PR's green, and both were real. **The green did not cover the origin.** The defects `_run` fixes came from a Windows/CP949 field report, and CI ran ubuntu + macos with zero Windows jobs. The environment the bug came from was the one environment nothing tested. **Install-and-import is not a test of the subprocess contract.** The `mcp` job proved the module loads. Both P0s changed how a child process is spawned, and nothing exercised that on any OS. So: contract tests for the two behaviours, and Windows added to the `mcp` matrix. The gate suite is bash and stays on the unix matrix; this job is the Python layer, which is where both reported P0s actually lived. The tests were checked in BOTH directions — they fail when the fix is reverted, not merely pass while it is present: - stdin: the parent is given sentinel bytes on its own stdin; with the fix reverted the child echoes them back, which is the protocol theft reproduced rather than described. - encoding: 🔴 the first version of this test was VACUOUS. `LC_ALL=C` alone is not hostile on modern Linux — PEP 538/540 coerce it back to UTF-8, so it passed with the fix removed. Verified that it did. `PYTHONCOERCECLOCALE=0` + `PYTHONUTF8=0` is what actually yields an ASCII default, and that is the portable stand-in for the reporter's CP949 machine. - the "no replacement characters" check also passed on EMPTY output, so a crashed run scored green; it now requires non-empty output first. Helpers are `.py`, not `.sh`, because `_run` dispatches `.py` through `sys.executable`: the tests then run identically on Windows. A bash helper would quietly skip there, and a skip that reads as a pass is how this hole got in. Not claimed: this covers the Python layer on Windows. The bash gate suite still does not run there, and no run has happened on a real CP949 codepage — only the ASCII stand-in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… Windows Adding windows-latest to the `mcp` matrix turned an existing step red: the default shell on that runner is PowerShell, which has no heredoc, so `python - <<'PY'` is a parse error rather than a Python failure. Worth stating plainly: the Windows job went red on its FIRST run, and not on the code the job was added to cover — the contract tests never got to execute. That is what covering the origin environment buys. Until this is green, Windows coverage is added, not demonstrated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
✅ 머지: 대장님 건별 승인(카드 승인 시점과 머지 시점의 head 가 다르다 (기록으로 남긴다)
내가 카드에 밝혔던 제한 — 이제 해소됐다카드 원문: "yeoul CI 에 Windows 잡이 0개라 발원 환경 자체는 그린이 안 덮습니다(증상 하나만 모사)." 잡이 초록인 것과 시험이 거기서 돈 것은 다르므로 로그로 직접 셌다(Windows Server 2025 러너): ⇒ 설치시험 보고서의 P0 둘(stdin 상속 · 비UTF-8 로케일 판정오류)이 발원 환경에서 실제로 시험됐다. 🔎 남는 것 하나 (머지를 막지 않음 · 다음에 볼 것)요약줄이 |
What this fixes
The substance check guarding an arc close was a single
[ ${#vans} -ge 6 ]— sixcharacters or more and the answer sealed. The self-evasion list next to it matched
exactly, so it only ever caught a bare
yes.Driving 28 evasive answers through the real binary (
arc-open → draft → fill → arc-close):The Windows/CP949 issue fixed previously was one case of this class. A mis-decode
merely inflated a trivial
yespast the length bar. With the bar as the onlysubstance test,
aaaaaa,yes ok,qwertyand해당없음해당없음sealed on aperfectly healthy UTF-8 Linux box as well.
Why not just raise the bar
Because that breaks the other direction. Real answers this gate has already sealed run
long, but short genuine answers are legitimate —
d=0.05 < 0.2, under the sealed baris a complete answer at 34 characters. A longer bar rejects real work.
So both directions were measured against the same corpora:
from arcs this gate has already sealed, not invented for the test.
One real answer was wrongly refused during development — a genuine one carrying a
deferral clause ("not run — the face arm is absent, so no anchor could be formed").
The rule now fires only when the deferral is the whole answer.
The part that matters most
bin/substance_check.pyis a heuristic, and heuristics are incomplete. So the load-bearingchange is not the rules — it is that the positive control now runs at the head of the
gate's execution path, not in a test file:
The specimens are bidirectional on purpose. Sabotaging the checker to always pass scores
a partial, not a zero, because the genuine specimens still pass. Plant only violations and
a checker that rejects everything scores full marks.
The tests assert this is load-bearing by actually sabotaging the checker and requiring exit 6
with nothing archived — and they verify the sabotage landed first, so the assertions
cannot pass vacuously.
Drift
Yeoul lives as two copies. A previous fix went into both, but its regression test went into
only one.
bin/substance_check.pyis therefore kept byte-identical with the internal copyand emits machine codes only — each shell renders its own prose. Drift is prevented by
construction rather than by discipline.
Also included
The earlier field-report fixes to the MCP wrapper: never inherit the parent's stdin (on a
STDIO server that pipe is the protocol, so a child steals protocol bytes and the tool
hangs), and pin UTF-8 for the subprocess.
🔴 What this does NOT claim
The 28 evasive answers were written by us. This is a catch rate for evasions we know,
not for the class. The class is narrowed, not closed — which is exactly why the
positive control, not the rule list, is the point of this PR.
Windows coverage is now partial, and this line has been corrected. It previously read
"Windows was never exercised directly", which is no longer true — and leaving a stale
limitation in place is its own kind of over-claim.
What changed: triage pointed out that the green did not cover the ORIGIN of the defect
(CI was ubuntu + macos, zero Windows jobs) and that install-and-import never exercised the
subprocess contract the
_runfixes changed. Both are now addressed —mcp/tests/contract tests, and
windows-latestin themcpmatrix. Verified in the log that all 8checks execute and pass on the Windows runner specifically, not merely that the job is
green.
Still NOT covered: the bash gate suite does not run on Windows (it stays on the unix
matrix), and no run has happened on a real CP949 codepage — the hostile-locale test uses an
ASCII default as a portable stand-in.
One of those tests was vacuous on the first attempt.
LC_ALL=Calone is not hostile onmodern Linux — PEP 538/540 coerce it back to UTF-8, so the encoding test passed with the
fix reverted. That was caught by reverting the fix and checking the test went red, which is
the only reason it is worth anything now. Every check here was verified in both directions.
Verify