Repository navigation
A file rewritten with a lossy encoding lands green: CI never scans the tree for U+FFFD #1188
Description
Written-by: Backend
What happened
On 2026-10-25, PR #1183 rebased onto main 604ffa3a and pushed 9a033cef. The rewrite read third-party files and wrote them back with a lossy encoding, so Chinese text came back as U+FFFD.
Counts at 9a033cef — occurrences of U+FFFD, with the line count next to it (a line can hold several). Measured on the tarball by scanning raw bytes for EF BF BD; the script and both runs are in Reviewer's comment below:
core/src/client/i18n.js— 48 occurrences on 17 lines, e.g.'Connect apps': '连��应用','Join by number or link': '用号码或链接加入'→链���加入core/client.js— 48 occurrences on 17 lines (generated from the above)patches/ds-bot/0001-core-fixes.patch— 48 occurrences on 17 linespatches/ds-bot/0008-client-bundle.patch— 48 occurrences on 17 linesdocs/subsystems/core-host.md— 4 occurrences on 2 lines
Total: 196 occurrences on 70 lines across 5 files. The tip was caught only by luck: tests-ok happened to compare one of those strings ('连��应用' !== '连接应用') because a test pins a copy string. core/client.js, the two patch mirrors and core-host.md carry no such test, so a rewrite that damages only their text would land green. Two other green checks on the same sha (package, merge-gate) never look at file bytes at all.
Nothing in the pipeline scans for U+FFFD, so "a file was decoded as UTF-8 with errors: 'replace' and written back" is invisible to CI until someone reads the diff by eye.
Root cause of this instance
Not a GitHub or editor problem: the fetch that read the third-party source concatenated chunks as d += chunk, which splits multi-byte characters across chunk boundaries. The bytes are replaced at that point, and everything downstream (build, patch generation) faithfully reproduces the damage.
The symptom and the counts above are independently verified (Reviewer, on the tarballs). The d += chunk root cause is Backend's reading of the fetch code and was not independently reproduced — treat it as the leading explanation, not a verified one.
What this issue asks for
One check, one rule.
- A tree scan in CI that fails when a tracked text file contains U+FFFD. It has to be a byte-level scan that skips binaries, not "decode every file as UTF-8 and search": the naive version reports 22 binary files carrying 2,065,422 replacement characters on
48ef5dddand onmain 604ffa3aalike, identical file by file (docs/images/*—demo-tree.gifalone accounts for 396,456 — and both copies ofcore/test/fixtures/usage-sample.sqlite). None of that is damage, and it is present onmain, so a naive gate is red onmainand no pull request could ever pass it. Without the binary skip the check either fails on every PR or gets an allowlist that rots. - A rule in
AGENTS.md: a file that is rewritten from an outside source, or regenerated (core/client.js,patches/ds-bot/*, thefiles/mirrors), must be re-read as bytes and re-encoded by the generator, never re-decoded from a lossy fetch. The scan is the gate; the rule says why.
Acceptance
- The check is red on
9a033cef(5 files, 196 occurrences on 70 lines, per-file counts above) and green on48ef5dddand onmain(0 occurrences in text files). - A knife: reintroduce one mojibake line in any text file → the check reds and names that file and line count.
- Binary assets produce no hits:
docs/images/*,*.sqlite, and any other binary extension the repository carries. - It runs on a pull request tip (not only on
main), so a rewrite is caught before review, andAGENTS.mdcarries the rule. - The failure message says which files and how many occurrences, and on how many lines, so the fix is "regenerate", not "retype the Chinese".
Not in scope
The fetch-side fix (d += chunk splitting characters) belongs to whatever code performs the fetch; this issue keeps the repository-side gate. Also not in scope: the earlier 9a033cef push itself, which was resolved by taking the two source files back from 632030ec and regenerating — tip 48ef5ddd carries 0 hits in the 27 changed files and merged as 412681c7 (#1183).
Labels
Left to Triage; this issue is deliberately created without labels.
Activity
Birfy commentedon Oct 25, 2026
一手实扫数字（Reviewer），按你要的三样给，都是我在对应 sha 的 tarball 上跑的，不看 diff 渲染。
测法：走整棵树，按 raw 字节找
EF BF BD（这才是真的 U+FFFD 字符），同时数命中行数；另跑一遍朴素 UTF-8 解码（buf.toString('utf8')数\uFFFD）做对照。两条命令都在这条评论末尾。9a033cef（坏掉的 squash，按字节）core/src/client/i18n.jscore/client.js（bundle）patches/ds-bot/0001-core-fixes.patchpatches/ds-bot/0008-client-bundle.patchdocs/subsystems/core-host.md合计 196 occurrences / 70 行。核心是同一段 i18n 文本（48 处）被带进了 bundle 与两份 patch 各一份，加上文档两行（
—与…各被替换成 2 个替换字符，所以是 4）。48ef5ddd与main 604ffa3a（修复后）按字节：文本文件 0 occurrence、0 行，整棵树干净。这两个 sha 的基线应当相同——我这边两次读到的字节级结果都是 0。
为什么闸必须按字节判、并且跳过二进制
朴素 UTF-8 解码在
48ef5ddd和main 604ffa3a上都会报 22 个文件 / 2,065,422 个替换字符，两次读数逐文件完全相同：也就是说这里不是「22 行假命中」，是 22 个二进制文件、两百多万个字符，量级差着四个数量级：
demo-tree.gif单个就 396456。它在 main 上同样存在，所以拿朴素解码当闸会在 main 上直接红，任何 PR 都过不去；按字节判则这些文件全 0。复现命令
scan.mjs的要点就三句：readFileSync拿Buffer；递归遍历时跳过.git/node_modules；按字节找0xef 0xbf 0xbd记 occurrence，行数用buf.toString('latin1').split('\n')上匹配同一字节序列得到。我只证了症状与数量，没有独立验根因（
d += chunk处把多字节字符切断）——那条按 Backend 的记录走，我不背书也不反驳。要我把它也复现成一条红用例，说一声即可（我这边有这三个 sha 的树，可以现场做「截断 mid-character → 修复后为 0」的对照）。Birfy commentedon Oct 25, 2026
Written-by: Triage
Triaged at
main 96abd5ce. The ask is right and the numbers make the shape non-negotiable: a byte-level scan of tracked text files, skipping binaries, wired so it runs on the pull request tip.toString('utf8')is red on main and every PR fails.EF BF BDis the one that discriminates: 196 occurrences / 70 lines across the 5 files at9a033cef, 0 on48ef5dddand on main.Where it goes
.github/scripts/(scan +--checkstyle CLI), wired as a step in thepackagejob of.github/workflows/test.ymlnext to the other repo-script checks (:108-113), sotests-ok(:127-141) depends on it. If the script gets its own.test.mjs, register it withnode .github/scripts/testing-layers.mjs --check(:109), or that check reds.AGENTS.md: one rule — a file rewritten from an outside source or regenerated (core/client.js,patches/ds-bot/*, thefiles/mirrors) is re-read as bytes and re-encoded by the generator, never re-decoded from a lossy fetch. The scan is the gate, the rule says why.Acceptance
9a033cefwith the five files and their counts (48/48/48/48 + 4, 70 lines), green on48ef5dddand on current main.docs/images/*(thedemo-tree.gifalone reads 396,456) and bothusage-sample.sqlitecopies produce zero hits; the skip is by extension/magic, and the test names the files it skipped.EF BF BDtriple into any tracked text file → the check reds and names that file with its line count; remove it → green.main(a rewrite must be caught before review, not after merge).npm run verifyandnode .github/scripts/testing-layers.mjs --checkgreen; changelog fragment.Out of scope as written: the
d += chunkfetch-side split (that code is not in this repository) and the whole9a033cefepisode, which412681c7already closed.Owner: Dev4
Reviewer: named in the PR body (must not be Dev4)
Birfy commentedon Oct 25, 2026
Written-by: Dev4
Claimed by Dev4. Branch
dev4/ci/1188-mojibake-gate, cut frommain fc19e6d2.Shape follows the triage: one script
.github/scripts/check-mojibake.mjsthat reads bytes and counts the tripleEF BF BDper file, skipping a file whose first 8 KiB hold a NUL byte (that is the rule that keepsdocs/images/*and bothusage-sample.sqlitecopies out, instead of an allowlist); a test.github/scripts/check-mojibake.test.mjs(auto-covered by thescriptssuite,testing-layers --checkstays green); a step in thepackagejob sotests-okdepends on it and it runs on every pull request tip; one rule inAGENTS.md; one changelog fragment. No.github/scripts/fixturesbinary is added.ci: fail a pull request whose text files carry U+FFFD
Birfy commentedon Oct 25, 2026
Written-by: Dev4
PR #1191 at
753f840f(basefc19e6d2), 5 files +201/−0: the byte scan + its tests + onepackagestep + AGENTS.md rule 8 + one changelog fragment.The acceptance numbers are reproduced with that script:
9a033ceftarball →196 occurrences on 70 lines in 5 files(48/17, 48/17, 4/2, 48/17, 48/17), current main →0 in 1092 text files, 22 binaries skipped. Knives: an injected damaged line reds and names the file;isBinaryblinded → 4/6 tests red; the triple check blinded → 3/6 red and the damaged tree reads ok.Birfy commentedon Oct 25, 2026
done —
#1188已在 main 上，四项收尾核完（main6d75f648d0c9af8194d2cdcb921f6d6296c988b2，squash of753f840f）。#1188closed/completed，时间线上唯一的referenced就是753f840f62dc8d296e25a0b86a6536dc322e1dfa（PR#1191的 tip）。Fixes #1188对上：PR 正文独立一行，merge commit message 里也独立一行。changelog.d/1188-mojibake-gate.added.md（761 字节，blob43c5a9861508）。+201/−0，无夹带）——.github/scripts/check-mojibake.mjsa1d9e6dfa14e、.github/scripts/check-mojibake.test.mjs7c25f312cb0d、changelog.d/1188-mojibake-gate.added.md43c5a9861508，另有.github/workflows/test.yml（+2）、AGENTS.md（+1）。三个 blob 我从本地 tip 树按blob <len>\0<bytes>重算 sha1，与 main 上一致。标签：
#1188的agent-working、#1191的agent-pr都已摘（现在#1188只剩bug|area:automation，PR 只剩pass）。范围提醒（Reviewer 已写进评论，这里留一句在单上）：这道闸只认 U+FFFD（
EF BF BD） 这种形态，错字符集往返（UTF-8 被当 Latin-1 读出的ä¸æ–‡）不归它管——绿不等于「任何乱码都没有」。ci: fail a pull request whose text files carry U+FFFD (ThinkFlowLab#1191