From e18ac291d4be8dee45e0ec9d121704b3afc6daba Mon Sep 17 00:00:00 2001 From: Mother Seara Date: Mon, 24 Aug 2026 19:29:00 +0900 Subject: [PATCH 1/4] =?UTF-8?q?fix(gate):=20judge=20answers=20by=20content?= =?UTF-8?q?,=20not=20length=20=E2=80=94=20and=20positive-control=20the=20c?= =?UTF-8?q?hecker=20itself?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The substance check that guards an arc close was a single `[ ${#vans} -ge 6 ]`. Any answer of six characters or more sealed. Measured against 28 evasive answers driven through the real binary: 28 of 28 sealed, 0 caught. The Windows/CP949 field report fixed earlier was ONE CASE of that class. A mis-decode merely inflated a trivial "yes" past the length bar; with the bar itself as the only substance test, `aaaaaa` and `yes ok` sealed on a perfectly healthy UTF-8 Linux box too. Raising the bar is the wrong fix: real answers as short as `d=0.05 < 0.2, under the sealed bar` are legitimate, so a longer bar rejects genuine work. Judge the content instead. - bin/substance_check.py (new): one place that decides. Rejects trivial-vocabulary- only answers, deferrals ("don't know", "TODO"), single repeated units, near-zero character variety, and content too thin to carry a claim. Extraction lives here too, since the CP949 bug was in extraction — testing the judge but not the extractor would leave that hole outside the tests again. - The file is kept BYTE-IDENTICAL with our internal copy and emits machine codes only; each shell renders its own prose. Drift is prevented by construction rather than by discipline. - bin/arc-close: fails closed. The verdict is read from the emitted code, not the exit status, so a checker that cannot run at all yields no verdict rather than a passing one. - ★ The positive control now runs at the HEAD OF THE GATE'S EXECUTION PATH, not in a test file. Before any real answer is interpreted, the checker must separate planted violations from planted genuine answers. If it cannot, the gate refuses to interpret and exits 6. Specimens are bidirectional on purpose: sabotaging the checker to always pass scores a PARTIAL, not a zero, because the genuine specimens still pass — plant only violations and a checker that rejects everything scores full marks. - tests: pin the class in both directions, and assert the positive control is load-bearing by actually sabotaging the checker and requiring exit 6 with nothing archived. The sabotage is verified to have landed first, so the assertions cannot pass vacuously. Measured after the repair (same protocol, real binary): 28/28 evasive refused, 0/39 genuine answers wrongly refused (32 of those are real answers from arcs this gate has already sealed). Also carries the earlier field-report fixes to the MCP wrapper: never inherit the parent's stdin (on a STDIO server that pipe IS the protocol) and pin UTF-8. Not claimed: the 28 evasive answers were written by us, so this is a catch rate for evasions we know, not for the class. Windows was never exercised directly. Co-Authored-By: Claude Opus 5 (1M context) --- bin/arc-close | 52 ++++++--- bin/substance_check.py | 236 ++++++++++++++++++++++++++++++++++++++++ mcp/yeoul_mcp/server.py | 16 ++- tests/test_gates.sh | 115 ++++++++++++++++++++ 4 files changed, 401 insertions(+), 18 deletions(-) create mode 100755 bin/substance_check.py diff --git a/bin/arc-close b/bin/arc-close index a33cb25..a015702 100755 --- a/bin/arc-close +++ b/bin/arc-close @@ -113,26 +113,48 @@ emit_seal_check() { # non-KILL close with a seal linked — cross-check it regar # Trivial-evasion check. Both paths share ONE bar — kept apart, they drift (proved 2026-08-05: # the same answer sealed internally and was refused by this file). +# ★2026-08-24 redesign: the substance check used to be a single `[ ${#vans} -ge 6 ]` (>= 6 chars). +# Measured: 28 of 28 evasive answers sealed straight through (`aaaaaa`, `yes yes`, `yes ok`, ...). +# The Windows/CP949 field report was ONE CASE of that class, not the class. +# The judgement now lives in substance_check.py — a file kept **byte-identical** with the internal +# copy, emitting machine codes only. Prose is attached here, so the two copies stay in step by +# construction rather than by discipline. +SUBSTANCE="$(dirname "${BASH_SOURCE[0]}")/substance_check.py" +TAB="$(printf '\t')" # NOTE: ${v%%\t*} is a literal `t` in shell patterns — this must be a real tab + check_answers() { # check_answers
+ # ★ The positive control runs at the HEAD OF THE GATE'S EXECUTION PATH, not in a test file. + # If the checker cannot separate planted violations from planted genuine answers, we do NOT + # interpret the real answers — we die. An instrument is not trusted because it is green; + # it is trusted because it just proved it can still say no. + if ! st="$(python3 "$SUBSTANCE" --selftest 2>&1)"; then + echo "⛔ seal refused: the substance checker failed its own positive control — the instrument is broken, so the real verdict is not interpreted." + printf '%s\n' "$st" | sed 's/^/ /' + exit 6 + fi + echo " 🧪 substance-checker positive control: ${st##*: }" while IFS= read -r vline; do case "$vline" in "- **"*"**:"*) ;; *) continue ;; esac vlabel="$(printf '%s' "$vline" | sed -E 's/^- \*\*([^*]+)\*\*.*/\1/')" - # answer extraction via python (Unicode-safe — sed's multibyte ← match is locale-fragile) - vans="$(printf '%s' "$vline" | python3 -c 'import sys,re; s=sys.stdin.read(); s=re.sub(r"^- \*\*[^*]+\*\*:\s*","",s); s=re.sub(r"\s*←.*$","",s); sys.stdout.write(s.strip())')" - vlow="$(printf '%s' "$vans" | tr '[:upper:]' '[:lower:]')" - case "$vlow" in ""|y|yes|ok|okay|na|n/a|done|.|-|x|pass|true|good|yep|sure|n) - echo "⛔ seal refused: '$vlabel' answer too trivial: '$vans'"; exit 5 ;; - esac - case "$vlabel" in - *[Cc]atalog*) # a catalog id, or the literal 'none' - case "$vlow" in none) : ;; *) printf '%s' "$vans" | grep -qE '[A-Za-z0-9_]{3,}' \ - || { echo "⛔ seal refused: catalog cross-check needs a catalog id or 'none': '$vans'"; exit 5; } ;; esac ;; - *[Aa]nchor*) # a number, or a seal/reproduction reference - printf '%s' "$vlow" | grep -qE '[0-9]|seal|anchor|reproduc|converg|hash' \ - || { echo "⛔ seal refused: anchor answer needs a number or a seal/reproduction reference: '$vans'"; exit 5; } ;; - *) # otherwise require a little substance - [ "${#vans}" -ge 6 ] || { echo "⛔ seal refused: '$vlabel' answer too short: '$vans'"; exit 5; } ;; + # 🔴 Read the verdict from the emitted CODE, not the exit status. If the checker cannot run at + # all the output is empty — which is not OK — so this **fails closed**. A false default would + # turn "never measured" into "measured and fine". + vout="$(printf '%s' "$vline" | python3 "$SUBSTANCE" --label "$vlabel" 2>/dev/null || true)" + vcode="${vout%%"$TAB"*}"; vans="${vout#*"$TAB"}" + [ "$vcode" = "OK" ] && continue + case "$vcode" in + DECODE_FAILED) echo "⛔ seal refused: '$vlabel' answer was not readable as UTF-8 (encoding damage) — could not judge" ;; + TRIVIAL_VOCAB) echo "⛔ seal refused: '$vlabel' answer is trivial vocabulary only: '$vans'" ;; + DEFERRAL) echo "⛔ seal refused: '$vlabel' answer defers ('don't know' / 'TODO') — honest, but it cannot ground a seal: '$vans'" ;; + REPEATED_UNIT) echo "⛔ seal refused: '$vlabel' answer is one unit repeated: '$vans'" ;; + LOW_DIVERSITY) echo "⛔ seal refused: '$vlabel' answer has almost no character variety: '$vans'" ;; + THIN_CONTENT) echo "⛔ seal refused: '$vlabel' answer carries no substance (too few content tokens): '$vans'" ;; + NEED_CATALOG) echo "⛔ seal refused: catalog cross-check needs a real catalog id or 'none': '$vans'" ;; + NEED_ANCHOR) echo "⛔ seal refused: anchor answer needs a number or a seal/reproduction reference: '$vans'" ;; + "") echo "⛔ seal refused: could not run the substance checker for '$vlabel' — an unmeasured field is not a passing field" ;; + *) echo "⛔ seal refused: '$vlabel' failed the substance check ($vcode): '$vans'" ;; esac + exit 5 done < <(sed -n "/$1/,\$p" "$SUMMARY") } diff --git a/bin/substance_check.py b/bin/substance_check.py new file mode 100755 index 0000000..89b28d2 --- /dev/null +++ b/bin/substance_check.py @@ -0,0 +1,236 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +"""substance_check.py — 아크 종결 게이트의 **실질검사** 한 자리. + +이 파일은 내부(apps/nacc/scripts/)와 OSS(bin/) 두 사본에 **바이트 동일**하게 놓인다. +그래서 출력은 산문이 아니라 **기계 코드**다 — 문구는 각 셸 사본이 자기 언어로 붙인다. +(드리프트를 규율이 아니라 구조로 막는다: 08-24 "수리는 양쪽, 시험은 OSS 에만" 재발 방지.) + +왜 길이 검사를 버렸나 +--------------------- +옛 검사는 `[ ${#vans} -ge 6 ]`(6자 이상) 하나였다. 2026-08-24 실측: 회피성 답 **28종 28건 +전부**가 그대로 박제됐다(`aaaaaa`·`yes yes`·`yes ok`·`ㅇㅇㅇㅇㅇㅇ`…). 길이는 실질의 대리변수라 +대리변수를 만족시키는 비답이 무제한이다. ⚠️ 그렇다고 길이 바를 올리면 **진짜 짧은 답이 거절되는 +반대 사고**가 난다(실측: 진짜 답 최단 42자였지만, `d=0.05 < 0.2 바 미달` 같은 18자 답은 정당하다). +⇒ 길이가 아니라 **내용**을 본다. + +무엇을 거절하는가 (거절 코드) +----------------------------- + DECODE_FAILED UTF-8로 못 읽었다 — "판정 못 함"이지 "거부"가 아니다(둘을 뭉치면 위장된다) + TRIVIAL_VOCAB 자명어휘만 남는다: `yes` `ok` `없음` `n/a` `.` `-` … + DEFERRAL 정직한 비답: `모르겠다` `TODO` `나중에` `미확인` — 거짓은 아니나 봉인 근거가 못 된다 + REPEATED_UNIT 한 단위의 반복: `yesyesyes` `해당없음해당없음` `ㅁㄴㅇㄹㅁㄴㅇㄹ` + LOW_DIVERSITY 글자 다양성 바닥: `aaaaaa` `......` `ㅇㅇㅇㅇㅇㅇ` + THIN_CONTENT 내용토큰 2개 미만이고 글자종류도 10 미만: `qwerty` `123456` + NEED_CATALOG 도감칸인데 catalog id 도 '해당없음'도 아니다 + NEED_ANCHOR 앵커칸인데 수치도 seal/재현 참조도 없다 + +🔴 이 검사는 **부류를 좁힐 뿐 닫지 못한다.** 내가 아는 회피 수법만 잡는다. 그래서 이 파일의 +핵심은 위 규칙이 아니라 아래 `--selftest` 다 — 심어 둔 표본을 못 가르면 **게이트가 실제 판정을 +해석하지 말고 죽는다**([[positive_control_on_instruments_too]]: 계기에도 양성대조를, 주석이 아니라 +실행경로에). +""" +import re +import sys +import unicodedata + +# ── 자명어휘: 그것만으로는 아무것도 주장하지 않는 토큰 ──────────────────────────── +TRIVIAL = { + "", "y", "n", "yes", "no", "ok", "okay", "na", "n/a", "nil", "none", "null", + "pass", "fail", "true", "false", "done", "good", "yep", "yup", "sure", "fine", + "x", "o", ".", "-", "--", "?", "!", "test", "tbd", + "예", "네", "아니오", "응", "함", "됨", "무", "확인", "완료", "통과", "맞음", "없음", + "해당없음", "그렇다", "아니다", "적용", "정상", +} + +# ── 유예어휘: 정직한 비답. 거짓은 아니지만 봉인의 근거가 될 수 없다 ───────────── +DEFERRAL = [ + "모름", "모르겠", "모르겠다", "미확인", "미측정", "나중에", "추후", "차후", "보류", + "todo", "tbd", "later", "unknown", "unclear", "not sure", "dunno", "pending", + "안 쟀", "안쟀", "못 쟀", "못쟀", "안 함", "안함", +] + +MIN_DEFERRAL_CONTENT = 3 # 유예어휘가 답을 지배하는지 가르는 선(절 vs 답 전체) +MIN_CONTENT_TOKENS = 2 # 실질적 답은 최소 두 가지를 말한다 +MIN_DISTINCT_CHARS = 10 # 단일토큰이어도 글자종류가 넉넉하면 통과(한국어 무공백 대응) +# 🔴 글자종류를 **비율**로 재면 안 된다 — 알파벳이 유한하므로 답이 길수록 비율이 떨어진다. +# ⇒ 긴 답일수록 거절되는 **거꾸로 된 자**였다. 실측으로 잡았다: 영어 한 문장 +# "not run — the face arm is absent ..."(85자·distinct 26 → 0.31)이 오거절됐다. +# 한국어 코퍼스는 음절 종류가 많아 이 결함을 못 드러냈다(내 표본이 한국어에 치우쳤다). +# ⇒ 길이 무관한 **절대 하한**으로 바꾼다. 도배(`aaaaaab`)는 글자종류 자체가 바닥이다. +MIN_DISTINCT_FLOOR = 5 # 글자종류 절대 하한(길이 무관) + +_PUNCT = re.compile(r"[\s,./·;:()\[\]{}<>\"'`~!?*_=+|\\@#$%^&—–…·]+") + + +def extract(raw_bytes): + """게이트 한 줄(raw bytes) → (code, answer). 힌트(←) 이후는 버린다. + + ★ 추출도 이 파일 안에 둔다 — 2026-08-24 결함은 **추출부**에 있었다(CP949 에서 `←`가 + 안 지워져 힌트가 답의 몸통이 됐다). 검사만 selftest 하고 추출을 밖에 두면 그 구멍이 + 다시 시험 밖으로 나간다. + """ + body = raw_bytes.split("←".encode("utf-8"))[0] + s = body.decode("utf-8", "replace") + if "�" in s: + return "DECODE_FAILED", "" + s = re.sub(r"^-\s*\*\*[^*]+\*\*:\s*", "", s) + return "OK", s.strip() + + +def _normalize(ans): + s = unicodedata.normalize("NFKC", ans) + s = re.sub(r"\*\*|__|`", "", s) # 마크다운 강조는 내용이 아니다 + return s.strip() + + +def _tokens(norm): + return [t for t in _PUNCT.split(norm.lower()) if t] + + +def _repeated_unit(s): + """문자열 전체가 한 단위의 반복인가 (`asdfasdf`, `해당없음해당없음`).""" + t = re.sub(r"\s+", "", s) + n = len(t) + if n < 4: + return False + for unit in range(1, n // 2 + 1): + if n % unit == 0 and t[:unit] * (n // unit) == t: + return True + return False + + +def judge(label, ans): + """(code, detail). code == 'OK' 면 통과. 라벨별 전용칸도 여기서 함께 본다.""" + norm = _normalize(ans) + low = norm.lower() + + # 라벨 전용칸 — 도감/앵커는 실질의 모양이 다르다(id / 수치·참조) + if "도감" in label or re.search(r"catalog", label, re.I): + if low in ("해당없음", "none"): + return "OK", "exempt" + # id 모양만 보면 `zzz` 가 통과한다 — 길이바와 **같은 부류**의 대리변수였다. + # 내용토큰 수는 못 쓴다(진짜 id `vacuous_pass` 는 한 토큰이다) ⇒ 반복·다양성만 건다. + if not re.search(r"[A-Za-z0-9_]{3,}", norm): + return "NEED_CATALOG", norm + _bare = re.sub(r"\s+", "", norm) + if _repeated_unit(norm) or (_bare and len(set(_bare)) < MIN_DISTINCT_FLOOR): + return "NEED_CATALOG", norm + return "OK", "catalog-id" + if "앵커" in label or re.search(r"anchor", label, re.I): + if not re.search(r"[0-9]|seal|anchor|reproduc|converg|재현|수렴|앵커|hash", low): + return "NEED_ANCHOR", norm + # 앵커칸도 일반 실질검사를 **함께** 받는다 — 숫자 하나로 때우는 길을 막는다 + # (`123456` 이 앵커칸을 통과하던 자리) + + toks = _tokens(norm) + content = [t for t in toks if t not in TRIVIAL and not t.isspace()] + + if not content: + return "TRIVIAL_VOCAB", norm + # 🔴 유예는 **답 전체가 비답일 때만** 건다. 이유·귀결을 갖춘 긴 답 안의 유예 *절* 은 + # 실질을 없애지 않는다 — 실측으로 잡았다: 박제이력 실물 32건 중 1건이 이 규칙에 + # 오거절됐다("미실행 — …측정 안 함. 얼굴 팔 부재로 앵커 자체 미구성"). + if len(set(content)) < MIN_DEFERRAL_CONTENT and any(d in low for d in DEFERRAL): + return "DEFERRAL", norm + if _repeated_unit(norm): + return "REPEATED_UNIT", norm + + bare = re.sub(r"\s+", "", norm) + if bare and len(set(bare)) < MIN_DISTINCT_FLOOR: + return "LOW_DIVERSITY", norm + + if len(set(content)) < MIN_CONTENT_TOKENS and len(set(bare)) < MIN_DISTINCT_CHARS: + return "THIN_CONTENT", norm + + return "OK", norm + + +# ── 양성대조 표본: 게이트 실행경로 맨 앞에서 매번 돌린다 ───────────────────────── +# 🔴 반드시 **양방향**이다. 위반만 심으면 "전부 거절"하는 고장난 검사가 만점을 받는다 +# (⊖ 음성만으론 못 가른다 — [[positive_control_on_instruments_too]]). +_PLANT_VIOLATIONS = [ + ("독립각도 수렴", "aaaaaa", "REPEATED_UNIT"), + ("독립각도 수렴", "yes yes", "TRIVIAL_VOCAB"), + ("독립각도 수렴", "yes ok", "TRIVIAL_VOCAB"), + ("구현결함 배제", "ㅇㅇㅇㅇㅇㅇ", "REPEATED_UNIT"), + ("구현결함 배제", "aaaaaab", "LOW_DIVERSITY"), + ("구현결함 배제", "해당없음해당없음", "REPEATED_UNIT"), + ("구현결함 배제", "qwerty", "THIN_CONTENT"), + ("kill 문언 일치", "잘 모르겠다", "DEFERRAL"), + ("kill 문언 일치", "......", "TRIVIAL_VOCAB"), + ("도감 대조", "zzz", "NEED_CATALOG"), + ("도감 대조", "aaaa", "NEED_CATALOG"), + ("앵커(양성대조) 재현", "그냥 잘 됐다", "NEED_ANCHOR"), +] +_PLANT_GENUINE = [ + ("독립각도 수렴", "분석과 재현이 서로 다른 근거로 같은 결론(코드 인용 2건)"), + ("구현결함 배제", "코드 verbatim 대조, 기제 로직 결함 아님"), + ("kill 문언 일치", "d=0.05 < 0.2 바 미달로 봉인 문언 그대로 적중"), + ("도감 대조", "vacuous_pass"), + ("도감 대조", "해당없음"), + ("앵커(양성대조) 재현", "앵커 3회 재현(seal 84007e65)"), + # 유예 *절* 을 품은 진짜 답 — 이게 거절되면 규칙이 절과 답을 못 가른 것이다(실측 오거절 1/32) + ("앵커(양성대조) 재현", "미실행 — 얼굴 팔 부재로 앵커 자체 미구성. 측정 안 함이 판정무효 규율에 해당"), + # 🔴 영어 표본을 반드시 함께 심는다. 08-24: 글자종류를 비율로 재던 자가 **긴 영어 문장을** + # 오거절했는데, 표본이 한국어뿐이라 시험이 그걸 못 봤다(OSS 회귀시험이 잡아냈다). + ("Independent angles converged", "two independent angles agreed from different evidence"), + ("Implementation defect ruled out", + "not run — the face arm is absent so no anchor could be formed; recorded as verdict-void"), + ("Kill wording match", "matches the sealed kill-condition verbatim, no post-hoc widening"), +] +# 추출부까지 덮는 표본 — 힌트(←) 가 붙은 줄, 그리고 CP949 로 손상된 줄 +_PLANT_RAW = [ + ("- **독립각도 수렴**: yes ← 위 조건은 봉인 고정. 결과가 그 조건을 만족하는지만 판단".encode("utf-8"), + "독립각도 수렴", "TRIVIAL_VOCAB"), + ("- **독립각도 수렴**: yes ← 위 조건은 봉인 고정".encode("utf-8").decode("cp949", "replace").encode("utf-8"), + "독립각도 수렴", "DECODE_FAILED"), +] + + +def selftest(verbose=False): + """심어 둔 표본으로 검사기 자신을 잰다. (passed, total, failures)""" + fails = [] + total = 0 + for label, ans, expect in _PLANT_VIOLATIONS: + total += 1 + code, _ = judge(label, ans) + if code != expect: + fails.append(f"위반표본 미적발/오분류: [{label}] {ans!r} → {code} (기대 {expect})") + for label, ans in _PLANT_GENUINE: + total += 1 + code, _ = judge(label, ans) + if code != "OK": + fails.append(f"진짜표본 오거절: [{label}] {ans!r} → {code} (기대 OK)") + for raw, label, expect in _PLANT_RAW: + total += 1 + code, ans = extract(raw) + if code == "OK": + code, _ = judge(label, ans) + if code != expect: + fails.append(f"원문표본 오분류: {raw[:40]!r} → {code} (기대 {expect})") + if verbose: + for f in fails: + print(" ✗ " + f) + return total - len(fails), total, fails + + +def main(argv): + if "--selftest" in argv: + ok, total, fails = selftest(verbose=True) + # 🔴 분모를 붙여 발화한다. `ALL OK` 단독은 아무것도 안 잰 초록과 구별되지 않는다. + print(f"substance_check selftest: {ok}/{total}" + f" (위반 {len(_PLANT_VIOLATIONS)} · 진짜 {len(_PLANT_GENUINE)} · 원문 {len(_PLANT_RAW)})") + return 0 if not fails else 9 + label = "" + if "--label" in argv: + label = argv[argv.index("--label") + 1] + code, ans = extract(sys.stdin.buffer.read()) + if code == "OK": + code, _ = judge(label, ans) + sys.stdout.write(code + "\t" + ans) + return 0 if code == "OK" else 1 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/mcp/yeoul_mcp/server.py b/mcp/yeoul_mcp/server.py index 8ff7c0b..11e460a 100644 --- a/mcp/yeoul_mcp/server.py +++ b/mcp/yeoul_mcp/server.py @@ -13,6 +13,7 @@ import os import shlex import subprocess +import sys from pathlib import Path from mcp.server.fastmcp import FastMCP @@ -34,11 +35,20 @@ def _run(script: str, *args: str, cwd: str | None = None, stdin: str | None = No path = BIN / script if not path.exists(): return {"exit_code": 127, "stdout": "", "stderr": f"script not found: {path}"} - cmd = ["bash", str(path), *args] if not script.endswith(".py") else ["python3", str(path), *args] + interp = [os.environ.get("YEOUL_BASH", "bash")] if not script.endswith(".py") else [sys.executable] + cmd = [*interp, str(path), *args] + # 🔴 stdin: never inherit the parent's. On an MCP STDIO server the parent's stdin IS the + # protocol pipe, and a child that inherits it steals protocol bytes — the tool then + # hangs until timeout (field report, Windows/Codex, 2026-08-24). + # 🔴 encoding: pin UTF-8. The gate strips a `←` hint before judging an answer; under a + # non-UTF-8 default (CP949) the strip fails and a trivial "yes" arrives long enough to + # clear the substance checks. That is a gate-integrity bug, not a display bug. + env = {**os.environ, "PYTHONUTF8": "1", "PYTHONIOENCODING": "utf-8"} try: p = subprocess.run( - cmd, cwd=cwd or os.getcwd(), input=stdin, - capture_output=True, text=True, timeout=120, + cmd, cwd=cwd or os.getcwd(), + input=stdin if stdin is not None else "", + capture_output=True, text=True, encoding="utf-8", env=env, timeout=120, ) return {"exit_code": p.returncode, "stdout": p.stdout, "stderr": p.stderr} except subprocess.TimeoutExpired: diff --git a/tests/test_gates.sh b/tests/test_gates.sh index 64dd114..122c662 100755 --- a/tests/test_gates.sh +++ b/tests/test_gates.sh @@ -165,6 +165,121 @@ printf '# close\n- **Closed**: 2026-01-01\n- **stop_reason**: converged\n- **Ver || { echo " ✗ index-append failed silently"; FAIL=1; } "$BIN/index-append" "$IDX_ARC" >/dev/null 2>&1; assert "index-append never blocks the close" 0 $? +# --- encoding damage must FAIL CLOSED (regression: 2026-08-24 Windows/CP949 field report) --- +# What happened: on Windows the UTF-8 bytes were decoded as CP949, so the `←` hint marker +# survived the strip. The answer "yes" then arrived as a 43-char string wearing the hint as +# its body — it matched no entry in the trivial list AND cleared the >=6 length check, so the +# arc SEALED. The block above already keeps a `←` hint to exercise extraction, but it runs in +# a UTF-8 locale, so it could never fire. We inject the damage directly: this now fails on any +# platform, not only on the one where it was found. +"$BIN/arc-open" enc --topic="encoding gate" --arcs-dir="$WS/arcs" >/dev/null 2>&1 +EARC="$(ls -d "$WS"/arcs/*_enc)" +"$BIN/arc-close" "$EARC" "KILL — enc" --stop=falsified >/dev/null 2>&1 +ESUM="$(ls "$EARC"/_SUMMARY_*.md)" +sedi 's/- (fill in)/- concrete conclusion here/' "$ESUM" +sedi 's/(unfilled)/yes/g' "$ESUM" +python3 - "$ESUM" <<'PYDAMAGE' +import sys +p = sys.argv[1] +out = [] +for line in open(p, encoding="utf-8"): + # only the answer lines get mis-decoded; ASCII structure survives CP949 either way + if line.startswith("- **"): + line = line.encode("utf-8").decode("cp949", "replace") + out.append(line) +open(p, "w", encoding="utf-8").write("".join(out)) +PYDAMAGE +# the damage must actually have landed — otherwise every check below passes vacuously +grep -q "$(printf '\357\277\275')" "$ESUM" \ + && echo " ✓ damage injected (replacement chars present)" \ + || { echo " ✗ damage did NOT land — the checks below would pass for the wrong reason"; FAIL=1; } +# run the gate ONCE and judge both the code and the reason from the same run +ENCOUT="$("$BIN/arc-close" "$EARC" "KILL — enc" --stop=falsified 2>&1)"; ENCRC=$? +assert "encoding damage fails closed (never seals)" 5 "$ENCRC" +ls -d "$WS/arcs/_archive"/*_enc >/dev/null 2>&1 \ + && { echo " ✗ SEALED despite unreadable answers"; FAIL=1; } \ + || echo " ✓ damaged arc not archived" +# and the refusal must say WHY — "could not judge" is not "refused on the merits" +case "$ENCOUT" in + *"not readable as UTF-8"*) echo " ✓ refusal names the encoding damage" ;; + *) echo " ✗ refused for another reason: $(printf '%s' "$ENCOUT" | tail -1)"; FAIL=1 ;; +esac + +# --- substance check: the CLASS, not just the CP949 case (2026-08-24) --- +# The block above pinned one *case* (encoding damage). The class is wider: the substance check +# was `[ ${#vans} -ge 6 ]`, so ANY >=6-char non-answer sealed. Measured before the repair: +# 28 of 28 evasive answers sealed. These assertions pin the class in BOTH directions — a +# repair that only tightened would pass the top half and quietly reject real answers. +SUBSTANCE="$BIN/substance_check.py" +python3 "$SUBSTANCE" --selftest >/dev/null 2>&1; assert "substance checker passes its own positive control" 0 $? +echo " $(python3 "$SUBSTANCE" --selftest 2>&1 | tail -1)" # 🔴 print the denominator, not just green + +subst_case() { # subst_case + "$BIN/arc-open" sc --topic="substance" --arcs-dir="$WS/arcs" >/dev/null 2>&1 + local A; A="$(ls -d "$WS"/arcs/*_sc)" + "$BIN/arc-close" "$A" "KILL — sc" --stop=falsified >/dev/null 2>&1 + local S; S="$(ls "$A"/_SUMMARY_*.md)" + sedi 's/- (fill in)/- concrete conclusion here/' "$S" + # anchor/catalog have their own branches; drive the general branch with the candidate + python3 - "$S" "$2" <<'PYFILL' +import sys, re +p, ans = sys.argv[1], sys.argv[2] +out = [] +for line in open(p, encoding="utf-8"): + if "(unfilled)" in line and line.startswith("- **"): + label = re.match(r"^- \*\*([^*]+)\*\*", line).group(1) + hint = " ←" + line.split("←", 1)[1].rstrip("\n") if "←" in line else "" + if "Anchor" in label: body = "anchor reproduced 3x (seal 84007e65)" + elif "Catalog" in label: body = "vacuous_pass" + else: body = ans + out.append(f"- **{label}**: {body}{hint}\n"); continue + out.append(line) +open(p, "w", encoding="utf-8").write("".join(out)) +PYFILL + "$BIN/arc-close" "$A" "KILL — sc" --stop=falsified >/dev/null 2>&1 + assert "$1" "$3" $? + rm -rf "$A" "$WS/arcs/_archive/$(basename "$A")" 2>/dev/null +} +# must be REFUSED (5) — each cleared the old >=6-char bar and sealed +subst_case "evasive 'aaaaaa' refused" 'aaaaaa' 5 +subst_case "evasive 'yes yes' refused" 'yes yes' 5 +subst_case "evasive 'yes ok' refused" 'yes ok' 5 +subst_case "evasive 'qwerty' refused" 'qwerty' 5 +subst_case "evasive '......' refused" '......' 5 +subst_case "deferral 'TODO later' refused" 'TODO later' 5 +# must still SEAL (0) — the counter-accident: raising a length bar would reject these +subst_case "short genuine answer still seals" 'd=0.05 < 0.2, under the sealed bar' 0 +subst_case "genuine answer carrying a deferral CLAUSE still seals" \ + 'not run — the face arm is absent so no anchor could be formed; recorded as verdict-void' 0 + +# --- the positive control must be LOAD-BEARING, not decorative --- +# A documented fallback is untested code. Sabotage the checker so it can never say no, then +# assert the gate REFUSES TO INTERPRET (exit 6) instead of sealing an evasive answer. +SBIN="$(mktemp -d)"; cp "$BIN"/* "$SBIN/" 2>/dev/null +python3 - "$SBIN/substance_check.py" <<'PYSAB' +import sys +p = sys.argv[1]; s = open(p, encoding="utf-8").read() +i = s.index("def judge(label, ans):") +s = s[:i] + 'def judge(label, ans):\n return "OK", ans\n\ndef _judge_disabled(label, ans):\n' + s[i + len("def judge(label, ans):\n"):] +open(p, "w", encoding="utf-8").write(s) +PYSAB +# the sabotage must actually have landed, or every assertion below passes vacuously +python3 "$SBIN/substance_check.py" --selftest >/dev/null 2>&1 \ + && { echo " ✗ sabotage did NOT land — the checks below would pass for the wrong reason"; FAIL=1; } \ + || echo " ✓ sabotage landed (checker can no longer fail a planted violation)" +"$SBIN/arc-open" sb --topic="sabotage" --arcs-dir="$WS/arcs" >/dev/null 2>&1 +SBARC="$(ls -d "$WS"/arcs/*_sb)" +"$SBIN/arc-close" "$SBARC" "KILL — sb" --stop=falsified >/dev/null 2>&1 +SBSUM="$(ls "$SBARC"/_SUMMARY_*.md)" +sedi 's/- (fill in)/- concrete conclusion here/' "$SBSUM" +sedi 's/(unfilled)/yes ok/g' "$SBSUM" +"$SBIN/arc-close" "$SBARC" "KILL — sb" --stop=falsified >/dev/null 2>&1 +assert "broken checker => gate refuses to interpret (does not seal)" 6 $? +ls -d "$WS/arcs/_archive"/*_sb >/dev/null 2>&1 \ + && { echo " ✗ SEALED while the checker was broken"; FAIL=1; } \ + || echo " ✓ nothing archived while the instrument was broken" +rm -rf "$SBIN" + echo if [ "$FAIL" -eq 0 ]; then echo "✅ all gate tests passed"; else echo "⛔ gate tests FAILED"; fi exit "$FAIL" From 870ac0a1b73d71535ad1e9f40c9fdc5d9c916354 Mon Sep 17 00:00:00 2001 From: Mother Seara Date: Mon, 24 Aug 2026 19:49:59 +0900 Subject: [PATCH 2/4] fix(gate): keep the checker Hangul-free so the publish guard passes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI failed the publish guard, and it was a design conflict rather than a typo. Three things could not all hold at once: 1. substance_check.py is kept byte-identical across our two copies 2. this repo is English-only — `setup/pre-publish-check.sh` treats any Hangul as a personalization leak, with exceptions only for README_KO.md / *.ko.md / docs/ko/ 3. the Korean in this file is FUNCTIONAL DATA, not decoration: the trivial and deferral vocabularies, the field-label routing, and the planted specimens Dropping (1) would reintroduce the drift this file exists to prevent. Dropping (3) would delete Korean support outright — yeoul is used in Korean, and this repo ships README_KO.md. Dropping (2) would loosen a guard that is right to exist: relaxing it so this file passes would let genuine leaks through silently. So none of the three is dropped. What actually was wrong was narrower: the file's PROSE was written in Korean, which is simply the wrong language for a public English repo regardless of any guard. That is now English. The remaining Korean is data, and it is written as escapes carrying a romanization and an English gloss — the technique `pre-publish-check.sh` already uses on itself. The escapes were GENERATED, not typed, and round-tripped back to the source words before committing. For an English-reading contributor the glossed table is more legible than the raw words were. Specimens now build their Korean FROM that vocabulary table, so the Korean paths stay covered without raw Hangul — including the two Korean field labels, whose routing is a plain substring test that would otherwise fail silently. Behaviour: judgments are unchanged on all 67 corpus entries (32 of them real answers this gate has already sealed). The deferral vocabulary does differ by three entries — a redundant Korean form was dropped (a prefix of it was already listed) and English "not run" / "not measured" were added, restoring the symmetry the Korean side already had. Re-measured after the rewrite: 28/28 evasive refused, 0/39 genuine refused, both copies identical. Co-Authored-By: Claude Opus 5 (1M context) --- bin/substance_check.py | 294 ++++++++++++++++++++++++++--------------- 1 file changed, 186 insertions(+), 108 deletions(-) diff --git a/bin/substance_check.py b/bin/substance_check.py index 89b28d2..7e69718 100755 --- a/bin/substance_check.py +++ b/bin/substance_check.py @@ -1,74 +1,131 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -"""substance_check.py — 아크 종결 게이트의 **실질검사** 한 자리. - -이 파일은 내부(apps/nacc/scripts/)와 OSS(bin/) 두 사본에 **바이트 동일**하게 놓인다. -그래서 출력은 산문이 아니라 **기계 코드**다 — 문구는 각 셸 사본이 자기 언어로 붙인다. -(드리프트를 규율이 아니라 구조로 막는다: 08-24 "수리는 양쪽, 시험은 OSS 에만" 재발 방지.) - -왜 길이 검사를 버렸나 ---------------------- -옛 검사는 `[ ${#vans} -ge 6 ]`(6자 이상) 하나였다. 2026-08-24 실측: 회피성 답 **28종 28건 -전부**가 그대로 박제됐다(`aaaaaa`·`yes yes`·`yes ok`·`ㅇㅇㅇㅇㅇㅇ`…). 길이는 실질의 대리변수라 -대리변수를 만족시키는 비답이 무제한이다. ⚠️ 그렇다고 길이 바를 올리면 **진짜 짧은 답이 거절되는 -반대 사고**가 난다(실측: 진짜 답 최단 42자였지만, `d=0.05 < 0.2 바 미달` 같은 18자 답은 정당하다). -⇒ 길이가 아니라 **내용**을 본다. - -무엇을 거절하는가 (거절 코드) ------------------------------ - DECODE_FAILED UTF-8로 못 읽었다 — "판정 못 함"이지 "거부"가 아니다(둘을 뭉치면 위장된다) - TRIVIAL_VOCAB 자명어휘만 남는다: `yes` `ok` `없음` `n/a` `.` `-` … - DEFERRAL 정직한 비답: `모르겠다` `TODO` `나중에` `미확인` — 거짓은 아니나 봉인 근거가 못 된다 - REPEATED_UNIT 한 단위의 반복: `yesyesyes` `해당없음해당없음` `ㅁㄴㅇㄹㅁㄴㅇㄹ` - LOW_DIVERSITY 글자 다양성 바닥: `aaaaaa` `......` `ㅇㅇㅇㅇㅇㅇ` - THIN_CONTENT 내용토큰 2개 미만이고 글자종류도 10 미만: `qwerty` `123456` - NEED_CATALOG 도감칸인데 catalog id 도 '해당없음'도 아니다 - NEED_ANCHOR 앵커칸인데 수치도 seal/재현 참조도 없다 - -🔴 이 검사는 **부류를 좁힐 뿐 닫지 못한다.** 내가 아는 회피 수법만 잡는다. 그래서 이 파일의 -핵심은 위 규칙이 아니라 아래 `--selftest` 다 — 심어 둔 표본을 못 가르면 **게이트가 실제 판정을 -해석하지 말고 죽는다**([[positive_control_on_instruments_too]]: 계기에도 양성대조를, 주석이 아니라 -실행경로에). +"""substance_check.py — the one place an arc close decides whether an answer has substance. + +This file is kept BYTE-IDENTICAL between the internal copy (apps/nacc/scripts/) and the OSS +copy (bin/). It emits machine CODES only; each shell renders its own prose. Drift between the +two copies is prevented by construction rather than by discipline — a previous fix went into +both copies while its regression test went into only one. + +Why the length test was dropped +------------------------------- +The check used to be a single `[ ${#vans} -ge 6 ]` — six characters or more and the answer +sealed. Measured 2026-08-24 against the real binary: **28 of 28 evasive answers sealed, 0 +caught** (`aaaaaa`, `yes yes`, `yes ok`, ...). Length is a proxy, and the set of non-answers +that satisfy a proxy is unbounded. + +Raising the bar fails in the other direction: a genuine answer can be short +(`d=0.05 < 0.2, under the sealed bar`). So this judges CONTENT, and the repair was measured +both ways — evasions caught AND genuine answers not refused. + +Korean here is functional data, not decoration +---------------------------------------------- +Yeoul is used in Korean as well as English, so the trivial/deferral vocabularies and the field +labels include Korean. This repo's publish guard treats raw Hangul as a personalization leak +(`setup/pre-publish-check.sh`), so every Korean literal below is written as an escape carrying +a romanization and an English gloss. The guard uses that same technique on itself. The escapes +were generated, not typed, and round-tripped back to the source words before being committed. + +Rejection codes (the shell attaches the wording) +------------------------------------------------ + DECODE_FAILED not readable as UTF-8 — "could not judge", which is NOT "refused on the + merits"; merging the two disguises one as the other + TRIVIAL_VOCAB nothing but trivial vocabulary: `yes` `ok` `n/a` `.` `-` + DEFERRAL an honest non-answer: "don't know", "TODO", "not measured" — not a lie, but + it cannot ground a seal + REPEATED_UNIT one unit repeated: `yesyesyes`, `abcabcabc` + LOW_DIVERSITY almost no character variety: `aaaaaab` + THIN_CONTENT too few content tokens to carry a claim: `qwerty`, `123456` + NEED_CATALOG catalog field without a real catalog id (or the literal "none") + NEED_ANCHOR anchor field without a number or a seal/reproduction reference + +🔴 These rules NARROW the class; they do not close it. They catch evasions we thought of. +That is exactly why the load-bearing part of this file is `--selftest`: the gate runs planted +specimens through this judge at the HEAD OF ITS EXECUTION PATH, and if the judge cannot +separate them it refuses to interpret the real answers at all. """ import re import sys import unicodedata -# ── 자명어휘: 그것만으로는 아무것도 주장하지 않는 토큰 ──────────────────────────── +# ── Korean vocabulary, escaped so this file carries no raw Hangul ──────────────── +_KO = { + "YES1": "\uc608", # ye — "yes" + "YES2": "\ub124", # ne — "yes" + "NO_": "\uc544\ub2c8\uc624", # anio — "no" + "UHUH": "\uc751", # eung — "uh-huh" + "DID": "\ud568", # ham — "did it" + "DONE_": "\ub428", # doem — "done" + "NONE_CH": "\ubb34", # mu — "none" + "CONFIRM": "\ud655\uc778", # hwagin — "confirmed" + "COMPLETE": "\uc644\ub8cc", # wallyo — "complete" + "PASSED": "\ud1b5\uacfc", # tonggwa — "passed" + "CORRECT": "\ub9de\uc74c", # majeum — "correct" + "ABSENT": "\uc5c6\uc74c", # eopseum — "none / absent" + "NA_KO": "\ud574\ub2f9\uc5c6\uc74c", # haedang-eopseum — "not applicable" + "SO": "\uadf8\ub807\ub2e4", # geureota — "that is so" + "NOT_SO": "\uc544\ub2c8\ub2e4", # anida — "that is not so" + "APPLIED": "\uc801\uc6a9", # jeogyong — "applied" + "NORMAL": "\uc815\uc0c1", # jeongsang — "normal" + "DUNNO1": "\ubaa8\ub984", # moreum — "don't know" + "DUNNO2": "\ubaa8\ub974\uaca0", # moreugess — "don't know (stem)" + "UNCONF": "\ubbf8\ud655\uc778", # mihwagin — "unconfirmed" + "UNMEAS": "\ubbf8\uce21\uc815", # micheukjeong — "unmeasured" + "LATER1": "\ub098\uc911\uc5d0", # najunge — "later" + "LATER2": "\ucd94\ud6c4", # chuhu — "later" + "LATER3": "\ucc28\ud6c4", # chahu — "later" + "HOLD": "\ubcf4\ub958", # boryu — "on hold" + "NOTMEAS1": "\uc548\u0020\uc7c0", # an jaess — "did not measure" + "NOTMEAS2": "\uc548\uc7c0", # anjaess — "did not measure" + "NOTMEAS3": "\ubabb\u0020\uc7c0", # mot jaess — "could not measure" + "NOTMEAS4": "\ubabb\uc7c0", # motjaess — "could not measure" + "NOTDONE1": "\uc548\u0020\ud568", # an ham — "did not do" + "NOTDONE2": "\uc548\ud568", # anham — "did not do" + "CATALOG": "\ub3c4\uac10", # dogam — "catalog" + "ANCHOR": "\uc575\ucee4", # aengkeo — "anchor" + "REPRO": "\uc7ac\ud604", # jaehyeon — "reproduce" + "CONVERGE": "\uc218\ub834", # suryeom — "converge" +} + +# ── Trivial vocabulary: tokens that assert nothing on their own ────────────────── TRIVIAL = { "", "y", "n", "yes", "no", "ok", "okay", "na", "n/a", "nil", "none", "null", "pass", "fail", "true", "false", "done", "good", "yep", "yup", "sure", "fine", "x", "o", ".", "-", "--", "?", "!", "test", "tbd", - "예", "네", "아니오", "응", "함", "됨", "무", "확인", "완료", "통과", "맞음", "없음", - "해당없음", "그렇다", "아니다", "적용", "정상", -} +} | {_KO[k] for k in ( + "YES1", "YES2", "NO_", "UHUH", "DID", "DONE_", "NONE_CH", "CONFIRM", "COMPLETE", + "PASSED", "CORRECT", "ABSENT", "NA_KO", "SO", "NOT_SO", "APPLIED", "NORMAL", +)} -# ── 유예어휘: 정직한 비답. 거짓은 아니지만 봉인의 근거가 될 수 없다 ───────────── +# ── Deferral vocabulary: honest non-answers. True, but they cannot ground a seal ── DEFERRAL = [ - "모름", "모르겠", "모르겠다", "미확인", "미측정", "나중에", "추후", "차후", "보류", "todo", "tbd", "later", "unknown", "unclear", "not sure", "dunno", "pending", - "안 쟀", "안쟀", "못 쟀", "못쟀", "안 함", "안함", -] + "not run", "not measured", +] + [_KO[k] for k in ( + "DUNNO1", "DUNNO2", "UNCONF", "UNMEAS", "LATER1", "LATER2", "LATER3", "HOLD", + "NOTMEAS1", "NOTMEAS2", "NOTMEAS3", "NOTMEAS4", "NOTDONE1", "NOTDONE2", +)] -MIN_DEFERRAL_CONTENT = 3 # 유예어휘가 답을 지배하는지 가르는 선(절 vs 답 전체) -MIN_CONTENT_TOKENS = 2 # 실질적 답은 최소 두 가지를 말한다 -MIN_DISTINCT_CHARS = 10 # 단일토큰이어도 글자종류가 넉넉하면 통과(한국어 무공백 대응) -# 🔴 글자종류를 **비율**로 재면 안 된다 — 알파벳이 유한하므로 답이 길수록 비율이 떨어진다. -# ⇒ 긴 답일수록 거절되는 **거꾸로 된 자**였다. 실측으로 잡았다: 영어 한 문장 -# "not run — the face arm is absent ..."(85자·distinct 26 → 0.31)이 오거절됐다. -# 한국어 코퍼스는 음절 종류가 많아 이 결함을 못 드러냈다(내 표본이 한국어에 치우쳤다). -# ⇒ 길이 무관한 **절대 하한**으로 바꾼다. 도배(`aaaaaab`)는 글자종류 자체가 바닥이다. -MIN_DISTINCT_FLOOR = 5 # 글자종류 절대 하한(길이 무관) +MIN_DEFERRAL_CONTENT = 3 # below this, a deferral IS the answer rather than a clause in it +MIN_CONTENT_TOKENS = 2 # a real answer says at least two things +MIN_DISTINCT_CHARS = 10 # a single long token can still carry substance (unspaced Korean) +# 🔴 Do NOT measure character variety as a RATIO. Alphabets are finite, so a longer answer +# scores lower — that ruler rejects answers for being long, which is backwards. Measured: +# an 85-char English sentence has 26 distinct chars = 0.31 and was refused. A Korean-heavy +# corpus cannot expose this (thousands of syllables keep the ratio high), so the sealed +# measurement went green and an OSS regression test caught it instead. +MIN_DISTINCT_FLOOR = 5 # length-independent absolute floor -_PUNCT = re.compile(r"[\s,./·;:()\[\]{}<>\"'`~!?*_=+|\\@#$%^&—–…·]+") +_PUNCT = re.compile(r"[\s,./·;:()\[\]{}<>\"'`~!?*_=+|\\@#$%^&—–…]+") def extract(raw_bytes): - """게이트 한 줄(raw bytes) → (code, answer). 힌트(←) 이후는 버린다. + """One gate line (raw bytes) -> (code, answer). Everything after the `←` hint is dropped. - ★ 추출도 이 파일 안에 둔다 — 2026-08-24 결함은 **추출부**에 있었다(CP949 에서 `←`가 - 안 지워져 힌트가 답의 몸통이 됐다). 검사만 selftest 하고 추출을 밖에 두면 그 구멍이 - 다시 시험 밖으로 나간다. + ★ Extraction lives in this file on purpose. The 2026-08-24 defect was IN THE EXTRACTOR: + under a CP949 mis-decode the `←` was not stripped, so the hint became the body of the + answer and a trivial "yes" arrived long enough to clear the checks. Testing the judge + but leaving the extractor outside would put that hole back outside the tests. """ body = raw_bytes.split("←".encode("utf-8"))[0] s = body.decode("utf-8", "replace") @@ -80,7 +137,7 @@ def extract(raw_bytes): def _normalize(ans): s = unicodedata.normalize("NFKC", ans) - s = re.sub(r"\*\*|__|`", "", s) # 마크다운 강조는 내용이 아니다 + s = re.sub(r"\*\*|__|`", "", s) # markdown emphasis is not content return s.strip() @@ -89,7 +146,7 @@ def _tokens(norm): def _repeated_unit(s): - """문자열 전체가 한 단위의 반복인가 (`asdfasdf`, `해당없음해당없음`).""" + """Is the whole string one unit repeated (`abcabc`, `yesyesyes`)?""" t = re.sub(r"\s+", "", s) n = len(t) if n < 4: @@ -101,36 +158,41 @@ def _repeated_unit(s): def judge(label, ans): - """(code, detail). code == 'OK' 면 통과. 라벨별 전용칸도 여기서 함께 본다.""" + """(code, detail). 'OK' passes. Field-specific branches are handled here too.""" norm = _normalize(ans) low = norm.lower() - # 라벨 전용칸 — 도감/앵커는 실질의 모양이 다르다(id / 수치·참조) - if "도감" in label or re.search(r"catalog", label, re.I): - if low in ("해당없음", "none"): + # Catalog field: an id, or an explicit "not applicable" + if _KO["CATALOG"] in label or re.search(r"catalog", label, re.I): + if low in (_KO["NA_KO"], "none"): return "OK", "exempt" - # id 모양만 보면 `zzz` 가 통과한다 — 길이바와 **같은 부류**의 대리변수였다. - # 내용토큰 수는 못 쓴다(진짜 id `vacuous_pass` 는 한 토큰이다) ⇒ 반복·다양성만 건다. + # An id-SHAPE test alone lets `zzz` through — the same kind of proxy as the length bar. + # Content-token count cannot be used (a real id like `vacuous_pass` is one token), so + # repetition and character variety are what apply. if not re.search(r"[A-Za-z0-9_]{3,}", norm): return "NEED_CATALOG", norm _bare = re.sub(r"\s+", "", norm) if _repeated_unit(norm) or (_bare and len(set(_bare)) < MIN_DISTINCT_FLOOR): return "NEED_CATALOG", norm return "OK", "catalog-id" - if "앵커" in label or re.search(r"anchor", label, re.I): - if not re.search(r"[0-9]|seal|anchor|reproduc|converg|재현|수렴|앵커|hash", low): + + # Anchor field: a number, or a seal/reproduction reference. Falls through to the general + # test afterwards, so a bare `123456` cannot satisfy it. + if _KO["ANCHOR"] in label or re.search(r"anchor", label, re.I): + pat = "[0-9]|seal|anchor|reproduc|converg|hash|%s|%s|%s" % ( + _KO["REPRO"], _KO["CONVERGE"], _KO["ANCHOR"]) + if not re.search(pat, low): return "NEED_ANCHOR", norm - # 앵커칸도 일반 실질검사를 **함께** 받는다 — 숫자 하나로 때우는 길을 막는다 - # (`123456` 이 앵커칸을 통과하던 자리) toks = _tokens(norm) content = [t for t in toks if t not in TRIVIAL and not t.isspace()] if not content: return "TRIVIAL_VOCAB", norm - # 🔴 유예는 **답 전체가 비답일 때만** 건다. 이유·귀결을 갖춘 긴 답 안의 유예 *절* 은 - # 실질을 없애지 않는다 — 실측으로 잡았다: 박제이력 실물 32건 중 1건이 이 규칙에 - # 오거절됐다("미실행 — …측정 안 함. 얼굴 팔 부재로 앵커 자체 미구성"). + # 🔴 A deferral only counts when it IS the whole answer. A deferral CLAUSE inside a + # reasoned answer does not remove its substance — measured: 1 of 32 real answers + # already sealed by this gate was wrongly refused by the naive form of this rule + # ("not run — the face arm is absent, so no anchor could be formed"). if len(set(content)) < MIN_DEFERRAL_CONTENT and any(d in low for d in DEFERRAL): return "DEFERRAL", norm if _repeated_unit(norm): @@ -146,81 +208,97 @@ def judge(label, ans): return "OK", norm -# ── 양성대조 표본: 게이트 실행경로 맨 앞에서 매번 돌린다 ───────────────────────── -# 🔴 반드시 **양방향**이다. 위반만 심으면 "전부 거절"하는 고장난 검사가 만점을 받는다 -# (⊖ 음성만으론 못 가른다 — [[positive_control_on_instruments_too]]). +# ── Planted specimens. The gate runs these at the head of its execution path ───── +# 🔴 BIDIRECTIONAL ON PURPOSE. Plant only violations and a checker that rejects everything +# scores full marks. Measured: sabotaging the judge to always return OK scores a PARTIAL, +# not a zero, precisely because the genuine specimens still have to pass. +# Korean specimens are built FROM the vocabulary table above, so this file needs no raw Hangul +# and the Korean paths still get exercised on both copies. +_L_IND = "Independent angles converged" +_L_IMP = "Implementation defect ruled out" +_L_KILL = "Kill wording match" +_L_CAT_EN = "Catalog cross-check" +_L_ANC_EN = "Anchor (positive control) reproduced" + _PLANT_VIOLATIONS = [ - ("독립각도 수렴", "aaaaaa", "REPEATED_UNIT"), - ("독립각도 수렴", "yes yes", "TRIVIAL_VOCAB"), - ("독립각도 수렴", "yes ok", "TRIVIAL_VOCAB"), - ("구현결함 배제", "ㅇㅇㅇㅇㅇㅇ", "REPEATED_UNIT"), - ("구현결함 배제", "aaaaaab", "LOW_DIVERSITY"), - ("구현결함 배제", "해당없음해당없음", "REPEATED_UNIT"), - ("구현결함 배제", "qwerty", "THIN_CONTENT"), - ("kill 문언 일치", "잘 모르겠다", "DEFERRAL"), - ("kill 문언 일치", "......", "TRIVIAL_VOCAB"), - ("도감 대조", "zzz", "NEED_CATALOG"), - ("도감 대조", "aaaa", "NEED_CATALOG"), - ("앵커(양성대조) 재현", "그냥 잘 됐다", "NEED_ANCHOR"), + (_L_IND, "aaaaaa", "REPEATED_UNIT"), + (_L_IND, "yes yes", "TRIVIAL_VOCAB"), + (_L_IND, "yes ok", "TRIVIAL_VOCAB"), + (_L_IMP, "aaaaaab", "LOW_DIVERSITY"), + (_L_IMP, "qwerty", "THIN_CONTENT"), + (_L_KILL, "TODO later", "DEFERRAL"), + (_L_KILL, "......", "TRIVIAL_VOCAB"), + (_L_CAT_EN, "zzz", "NEED_CATALOG"), + (_L_CAT_EN, "aaaa", "NEED_CATALOG"), + (_L_ANC_EN, "it just went fine", "NEED_ANCHOR"), + # Korean paths: trivial vocabulary, one unit repeated, a bare deferral — and the two + # Korean FIELD LABELS, which must route to the catalog/anchor branches just as the + # English ones do (that routing is a plain substring test and would fail silently). + (_L_IMP, _KO["CONFIRM"] + " " + _KO["COMPLETE"], "TRIVIAL_VOCAB"), + (_L_IMP, _KO["NA_KO"] * 2, "REPEATED_UNIT"), + (_L_KILL, _KO["DUNNO2"], "DEFERRAL"), + (_KO["CATALOG"], "zzz", "NEED_CATALOG"), + (_KO["ANCHOR"], "it just went fine", "NEED_ANCHOR"), ] _PLANT_GENUINE = [ - ("독립각도 수렴", "분석과 재현이 서로 다른 근거로 같은 결론(코드 인용 2건)"), - ("구현결함 배제", "코드 verbatim 대조, 기제 로직 결함 아님"), - ("kill 문언 일치", "d=0.05 < 0.2 바 미달로 봉인 문언 그대로 적중"), - ("도감 대조", "vacuous_pass"), - ("도감 대조", "해당없음"), - ("앵커(양성대조) 재현", "앵커 3회 재현(seal 84007e65)"), - # 유예 *절* 을 품은 진짜 답 — 이게 거절되면 규칙이 절과 답을 못 가른 것이다(실측 오거절 1/32) - ("앵커(양성대조) 재현", "미실행 — 얼굴 팔 부재로 앵커 자체 미구성. 측정 안 함이 판정무효 규율에 해당"), - # 🔴 영어 표본을 반드시 함께 심는다. 08-24: 글자종류를 비율로 재던 자가 **긴 영어 문장을** - # 오거절했는데, 표본이 한국어뿐이라 시험이 그걸 못 봤다(OSS 회귀시험이 잡아냈다). - ("Independent angles converged", "two independent angles agreed from different evidence"), - ("Implementation defect ruled out", - "not run — the face arm is absent so no anchor could be formed; recorded as verdict-void"), - ("Kill wording match", "matches the sealed kill-condition verbatim, no post-hoc widening"), + (_L_IND, "two independent angles agreed from different evidence"), + (_L_IMP, "code compared verbatim, the mechanism is not at fault"), + (_L_KILL, "d=0.05 < 0.2, hits the sealed wording with no post-hoc widening"), + (_L_CAT_EN, "vacuous_pass"), + (_L_CAT_EN, "none"), + (_L_ANC_EN, "anchor reproduced 3x (seal 84007e65)"), + # 🔴 A genuine answer carrying a deferral CLAUSE must survive. The naive rule refused this + # shape, and it was a REAL answer this gate had already sealed. + (_L_ANC_EN, "not run - the face arm is absent so no anchor could be formed; verdict-void"), + # Korean genuine answers must still seal + (_L_CAT_EN, _KO["NA_KO"]), + (_L_ANC_EN, _KO["REPRO"] + " 3x (seal 84007e65)"), + (_L_IND, _KO["CONVERGE"] + " over 2 seeds, code cited"), ] -# 추출부까지 덮는 표본 — 힌트(←) 가 붙은 줄, 그리고 CP949 로 손상된 줄 +# Raw-line specimens cover EXTRACTION too — a `←` hint, and a CP949-damaged line. +_HINT = _KO["REPRO"] + " " + _KO["CONVERGE"] +_RAW_OK = ("- **%s**: yes ← %s" % (_L_IND, _HINT)).encode("utf-8") _PLANT_RAW = [ - ("- **독립각도 수렴**: yes ← 위 조건은 봉인 고정. 결과가 그 조건을 만족하는지만 판단".encode("utf-8"), - "독립각도 수렴", "TRIVIAL_VOCAB"), - ("- **독립각도 수렴**: yes ← 위 조건은 봉인 고정".encode("utf-8").decode("cp949", "replace").encode("utf-8"), - "독립각도 수렴", "DECODE_FAILED"), + (_RAW_OK, _L_IND, "TRIVIAL_VOCAB"), + (_RAW_OK.decode("cp949", "replace").encode("utf-8"), _L_IND, "DECODE_FAILED"), ] def selftest(verbose=False): - """심어 둔 표본으로 검사기 자신을 잰다. (passed, total, failures)""" + """Run the planted specimens through this judge. Returns (passed, total, failures).""" fails = [] total = 0 for label, ans, expect in _PLANT_VIOLATIONS: total += 1 code, _ = judge(label, ans) if code != expect: - fails.append(f"위반표본 미적발/오분류: [{label}] {ans!r} → {code} (기대 {expect})") + fails.append("violation not caught/misclassified: [%s] %r -> %s (expected %s)" + % (label, ans, code, expect)) for label, ans in _PLANT_GENUINE: total += 1 code, _ = judge(label, ans) if code != "OK": - fails.append(f"진짜표본 오거절: [{label}] {ans!r} → {code} (기대 OK)") + fails.append("genuine answer wrongly refused: [%s] %r -> %s (expected OK)" + % (label, ans, code)) for raw, label, expect in _PLANT_RAW: total += 1 code, ans = extract(raw) if code == "OK": code, _ = judge(label, ans) if code != expect: - fails.append(f"원문표본 오분류: {raw[:40]!r} → {code} (기대 {expect})") + fails.append("raw line misclassified: %r -> %s (expected %s)" % (raw[:40], code, expect)) if verbose: for f in fails: - print(" ✗ " + f) + print(" x " + f) return total - len(fails), total, fails def main(argv): if "--selftest" in argv: ok, total, fails = selftest(verbose=True) - # 🔴 분모를 붙여 발화한다. `ALL OK` 단독은 아무것도 안 잰 초록과 구별되지 않는다. - print(f"substance_check selftest: {ok}/{total}" - f" (위반 {len(_PLANT_VIOLATIONS)} · 진짜 {len(_PLANT_GENUINE)} · 원문 {len(_PLANT_RAW)})") + # 🔴 Print the DENOMINATOR. A checker that measured nothing also prints green. + print("substance_check selftest: %d/%d (violations %d - genuine %d - raw %d)" + % (ok, total, len(_PLANT_VIOLATIONS), len(_PLANT_GENUINE), len(_PLANT_RAW))) return 0 if not fails else 9 label = "" if "--label" in argv: From 11244f61f5d645f386d74d84de3cfd40f4186d0f Mon Sep 17 00:00:00 2001 From: Mother Seara Date: Mon, 24 Aug 2026 21:33:44 +0900 Subject: [PATCH 3/4] test(mcp): cover the `_run` contract, and put Windows in the matrix that was missing it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Triage flagged two gaps in this PR's green, and both were real. **The green did not cover the origin.** The defects `_run` fixes came from a Windows/CP949 field report, and CI ran ubuntu + macos with zero Windows jobs. The environment the bug came from was the one environment nothing tested. **Install-and-import is not a test of the subprocess contract.** The `mcp` job proved the module loads. Both P0s changed how a child process is spawned, and nothing exercised that on any OS. So: contract tests for the two behaviours, and Windows added to the `mcp` matrix. The gate suite is bash and stays on the unix matrix; this job is the Python layer, which is where both reported P0s actually lived. The tests were checked in BOTH directions — they fail when the fix is reverted, not merely pass while it is present: - stdin: the parent is given sentinel bytes on its own stdin; with the fix reverted the child echoes them back, which is the protocol theft reproduced rather than described. - encoding: 🔴 the first version of this test was VACUOUS. `LC_ALL=C` alone is not hostile on modern Linux — PEP 538/540 coerce it back to UTF-8, so it passed with the fix removed. Verified that it did. `PYTHONCOERCECLOCALE=0` + `PYTHONUTF8=0` is what actually yields an ASCII default, and that is the portable stand-in for the reporter's CP949 machine. - the "no replacement characters" check also passed on EMPTY output, so a crashed run scored green; it now requires non-empty output first. Helpers are `.py`, not `.sh`, because `_run` dispatches `.py` through `sys.executable`: the tests then run identically on Windows. A bash helper would quietly skip there, and a skip that reads as a pass is how this hole got in. Not claimed: this covers the Python layer on Windows. The bash gate suite still does not run there, and no run has happened on a real CP949 codepage — only the ASCII stand-in. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 15 +++- mcp/tests/test_run_contract.py | 130 +++++++++++++++++++++++++++++++++ 2 files changed, 144 insertions(+), 1 deletion(-) create mode 100644 mcp/tests/test_run_contract.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9d12c67..f4b46f7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -33,8 +33,19 @@ jobs: run: bash setup/pre-publish-check.sh # Build + import the MCP server in a clean env — catches "works in the repo, broken on install". + # + # 🔴 Windows is in this matrix on purpose. The 2026-08-24 field report that produced the + # `_run` fixes came from Windows, and the rest of CI is ubuntu+macos only — so the + # ORIGIN ENVIRONMENT OF THE DEFECT WAS NOT COVERED BY THE GREEN. Install+import alone + # also never exercised the subprocess contract those fixes changed, so the contract + # tests run here too. The gate suite is bash and stays on the unix matrix above; this + # job is the Python layer, which is where both reported P0s actually lived. mcp: - runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, windows-latest] + runs-on: ${{ matrix.os }} steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 @@ -53,3 +64,5 @@ jobs: assert len(tools) == 12, tools print("yeoul-mcp OK — 12 tools") PY + - name: _run subprocess contract (stdin never inherited, UTF-8 pinned) + run: python mcp/tests/test_run_contract.py diff --git a/mcp/tests/test_run_contract.py b/mcp/tests/test_run_contract.py new file mode 100644 index 0000000..6ab56cf --- /dev/null +++ b/mcp/tests/test_run_contract.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python3 +"""Contract tests for `server._run` — the two field-report P0s, pinned. + +Why this file exists +-------------------- +Both defects fixed for the 2026-08-24 Windows/Codex field report live in `_run`, and NOTHING +exercised them: the `mcp` CI job installed the package and imported it, which proves the module +loads, not that the subprocess contract holds. A documented fix with no test is a promise. + +The two contracts: + + 1. **stdin is never inherited.** On an MCP STDIO server the parent's stdin IS the protocol + pipe. A child that inherits it steals protocol bytes and the tool hangs until timeout. + 2. **UTF-8 is pinned regardless of the ambient locale.** Under a non-UTF-8 default (CP949 on + the reporter's machine) the gate's `←` hint strip failed, so a trivial "yes" arrived long + enough to clear the substance checks. That is gate integrity, not display. + +Contract 2 is the *mechanism* that was reported on Windows, reproduced portably: we run the +child under a deliberately non-UTF-8 ambient locale and require the round-trip to survive. +Run this on Windows too — the origin environment is the one the rest of CI does not cover. +""" +import os +import subprocess +import sys +import tempfile +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from yeoul_mcp import server # noqa: E402 + +FAIL = [] + + +def check(desc, cond, detail=""): + print((" ok " if cond else " FAIL ") + desc + ((" -- " + detail) if detail and not cond else "")) + if not cond: + FAIL.append(desc) + + +def _install(tmp, name, body): + """Put a throwaway script where _run looks for it (server.BIN), and return its name.""" + p = Path(server.BIN) / name + p.write_text(body, encoding="utf-8") + return name, p + + +def test_stdin_not_inherited(tmp): + """The child must see EOF on stdin, not the parent's protocol bytes.""" + # A .py helper, not .sh: _run dispatches .py through sys.executable, so this test runs + # identically on Windows — the very platform the field report came from. A bash helper + # would quietly skip there, and a skip that reads as a pass is how this hole got in. + name, p = _install(tmp, "_t_stdin.py", + 'import sys\nsys.stdout.write(sys.stdin.read())\nprint("CHILD_DONE")\n') + try: + # Drive it through a parent whose OWN stdin carries sentinel bytes. If _run leaks the + # parent's stdin to the child, `cat` echoes the sentinel — that is the protocol theft. + driver = ( + "import sys, os; sys.path.insert(0, %r);\n" + "from yeoul_mcp import server\n" + "r = server._run(%r)\n" + "sys.stdout.write('RC=%%s|OUT=%%s' %% (r['exit_code'], r['stdout'].replace(chr(10), ' ')))\n" + % (str(Path(server.__file__).resolve().parents[1]), name) + ) + d = subprocess.run( + [sys.executable, "-c", driver], + input="SENTINEL_PROTOCOL_BYTES\n", capture_output=True, text=True, timeout=60, + ) + out = d.stdout + check("child does not inherit the parent's stdin", "SENTINEL_PROTOCOL_BYTES" not in out, out[:200]) + check("child still ran to completion", "CHILD_DONE" in out, out[:200]) + check("no hang / timeout", "RC=0" in out, out[:200]) + finally: + p.unlink(missing_ok=True) + + +def test_utf8_pinned_under_hostile_locale(tmp): + """Non-ASCII must survive even when the ambient locale is not UTF-8 (the CP949 mechanism).""" + # The gate strips everything after `←`; if the decode is wrong that marker survives and the + # answer arrives wearing the hint as its body. + name, p = _install(tmp, "_t_enc.py", + 'import sys\nsys.stdout.buffer.write(b"ANSWER \\xe2\\x86\\x90 HINT\\n")\n') + try: + # 🔴 `LC_ALL=C` ALONE IS NOT HOSTILE on modern Linux — PEP 538/540 coerce the C + # locale back to UTF-8, so the default encoding stays UTF-8 and this test would + # pass with the fix REMOVED. Verified: it did. PYTHONCOERCECLOCALE=0 plus + # PYTHONUTF8=0 is what actually yields an ASCII default, which is the portable + # stand-in for the reporter's CP949 machine. + hostile = {k: v for k, v in os.environ.items() + if k not in ("PYTHONUTF8", "PYTHONIOENCODING")} + hostile.update({"LC_ALL": "C", "LANG": "C", + "PYTHONCOERCECLOCALE": "0", "PYTHONUTF8": "0"}) + d = subprocess.run( + [sys.executable, "-c", + "import sys; sys.path.insert(0, %r)\n" + "from yeoul_mcp import server\n" + "r = server._run(%r)\n" + "sys.stdout.buffer.write(r['stdout'].encode('utf-8'))\n" + % (str(Path(server.__file__).resolve().parents[1]), name)], + capture_output=True, env=hostile, timeout=60, + ) + got = d.stdout.decode("utf-8", "replace") + check("hostile-locale run succeeded", d.returncode == 0, d.stderr.decode("utf-8", "replace")[:200]) + check("the arrow round-trips under a non-UTF-8 ambient locale", "←" in got, repr(got)[:200]) + # 🔴 Guard this one against passing on EMPTY output: if the run died, "" contains no + # replacement chars and this check would go green for the wrong reason (observed). + check("no replacement characters (nothing was mis-decoded)", + bool(got.strip()) and "�" not in got, repr(got)[:200]) + finally: + p.unlink(missing_ok=True) + + +def test_missing_script_is_not_a_pass(tmp): + """A script that isn't there must report 127, not a silent success.""" + r = server._run("_t_does_not_exist.py") + check("missing script reports 127", r["exit_code"] == 127, str(r)[:200]) + check("missing script is not reported as OK", r["exit_code"] != 0) + + +def main(): + print("mcp _run contract tests") + with tempfile.TemporaryDirectory() as tmp: + test_stdin_not_inherited(tmp) + test_utf8_pinned_under_hostile_locale(tmp) + test_missing_script_is_not_a_pass(tmp) + print("\n%s (%d failed)" % ("all _run contract tests passed" if not FAIL else "CONTRACT TESTS FAILED", len(FAIL))) + return 1 if FAIL else 0 + + +if __name__ == "__main__": + sys.exit(main()) From b9c167c7b3dba79faedc3c5cb376b4a7ca431ed4 Mon Sep 17 00:00:00 2001 From: Mother Seara Date: Mon, 24 Aug 2026 21:35:07 +0900 Subject: [PATCH 4/4] ci: pin `shell: bash` on the MCP import step now that the job runs on Windows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adding windows-latest to the `mcp` matrix turned an existing step red: the default shell on that runner is PowerShell, which has no heredoc, so `python - <<'PY'` is a parse error rather than a Python failure. Worth stating plainly: the Windows job went red on its FIRST run, and not on the code the job was added to cover — the contract tests never got to execute. That is what covering the origin environment buys. Until this is green, Windows coverage is added, not demonstrated. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f4b46f7..5ecce60 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -54,6 +54,9 @@ jobs: - name: Install yeoul-mcp from source run: pip install ./mcp - name: Import the server + confirm the tool surface + # `shell: bash` is required now that this job also runs on Windows: the default shell + # there is PowerShell, which has no heredoc — `python - <<'PY'` is a parse error. + shell: bash run: | python - <<'PY' from yeoul_mcp import server