fix(lang): #866 script install corruption via comment-marker misread in single-quoted constants - #885
Conversation
langstripstructuremarkers scans each source line to find where a trailing
comment begins, so that a comment marker inside a string literal is not
mistaken for a real comment. It tracked two of UserTalk's three literal
delimiters -- ASCII " and the Mac curly-quote pair (0xD2/0xD3) -- but not
chsinglequote (0x27), which delimits character and string4 constants.
On a line such as
if s contains 'C' { (where C is chcomment, 0xC7)
the 0xC7 inside the literal was read as the start of a trailing comment.
commentstart landed mid-literal, so the trailing structural-marker strip
was skipped for the real content and a { survived in the stored node
text. The outline export (oplangtextvisit) then re-emits a { from the
outline LEVEL transition, so the line gained one extra brace per install:
install 1: if s contains 'C' { {
install 2: if s contains 'C' { { {
The growth is cumulative and unbounded, and the script stops compiling
from the first install onward. This is the same family as #621, which
taught the strip about inline one-line blocks but not the scanner about
single-quote literals.
Both scanners in the function had the gap: the comment-start scan and the
brace-balance counter that decides whether a trailing } is load-bearing.
A brace inside a character constant is not structural, so both now track
all three delimiters. The two-way flqcurly/qcurly boolean is replaced by
an explicit closing-delimiter byte, which extends to three delimiters
without further branching.
Effect on the shipped corpus: a sweep of all 2,996 scripts in
Virgin.root (round-trip each, compile-gated with an installer-
unrepairable negative control) had exactly one script that reinstall
left non-compiling -- suites.commercial.parseAete, an instance of this
exact shape. It is now stable across repeated reinstalls.
Tests: three cases added to script_install_roundtrip.yaml (read-back
byte-equality, reinstall idempotence, and end-to-end execution). Verified
red before the fix and green after; the 8 pre-existing cases in that file
pass throughout. Unit-test result sets are identical to baseline (the
test_callback_infrastructure segfault reproduces unchanged on unmodified
develop and is unrelated).
Note for test authors: the runner writes script files as UTF-8, so a
literal 0xC7 byte in YAML arrives as two bytes and breaks a single-quoted
character constant. These tests use the \xc7 escape, which is ASCII in
the file and decodes to the single byte at runtime.
Follow-up to 76b7b8d, from the bar-raiser verdict. Test-only; no source changes. - Fix a stale cross-reference. The three #866 cases were inserted between the two #621 cases, so "Same shape as the previous test" in the outermost-depth case pointed at the wrong neighbour. It now names the depth-1 inline-block case explicitly, which survives reordering. - Add a brace-balance-counter case. The fix touched two scanners; the three #866 cases only name the comment-start scan. This one puts a brace inside a character constant on a line that also ends with an inline one-line block, so it runs through the counter that decides whether a trailing } is load-bearing. - Add a curly-quote (0xD2/0xD3) string-literal case. That delimiter was tracked by both scanners but had no test, and the refactor rewrote how the closing delimiter is chosen. Status of the two new cases, measured rather than assumed: both PASS on a pre-fix binary built from 76b7b8d~1, so they are regression guards for the refactor, not reproductions of the bug. Only the three original #866 cases are red pre-fix (verified through the runner against that binary: 10 passed / 3 failed). All 13 pass at this commit. The new cases are ASCII in the yaml and use \xd2 \xd3 \xc7 escapes; the runner writes script files as UTF-8, so a literal high byte would arrive as two bytes and break a single-quoted constant.
Gate Review: fix-866-install-corruptionVerdict: PASSScope: 2 commits (76b7b8d fix + 2b11766 test-only) vs develop 03d7fe1 — Common/source/langscan.c (+50/-22) + tests/integration/test_cases/script_install_roundtrip.yaml (+66/-1). Fixes #866. The bug (and the false diagnosis it was buried under)Real defect: The issue's original headline claims were REFUTED with a fingerprint: "reinstall destroys unmodified large scripts" was a harness artifact — MacRoman source round-tripped through UTF-8 (the issue's own quoted sizes are exact UTF-8 byte counts of the true char counts: 56,864+163→57,027; 10,038→10,055). Verified on the actual ced8241-era binaries: byte-identical round-trips, compile true→true. The "opcode-0" report traces to stale databases/Frontier.root (pre-#621 corruption, #877). Records corrected on #866/#862/#849; #872's full-wrapper approach is unblocked. Tests
Reviewers
Follow-ups filed from this unit#877 (stale Frontier.root), #878 (65 cosmetic non-idempotent), #879→reclassified (#881 LF corruption — the #855 unblocker; #882 missing headless verbs), #880 (runner UTF-8 trap that caused the false diagnosis), #883 (CR-spanning residual), + scanner-deduplication refactor suggestion. Stats
|
1 similar comment
Gate Review: fix-866-install-corruptionVerdict: PASSScope: 2 commits (76b7b8d fix + 2b11766 test-only) vs develop 03d7fe1 — Common/source/langscan.c (+50/-22) + tests/integration/test_cases/script_install_roundtrip.yaml (+66/-1). Fixes #866. The bug (and the false diagnosis it was buried under)Real defect: The issue's original headline claims were REFUTED with a fingerprint: "reinstall destroys unmodified large scripts" was a harness artifact — MacRoman source round-tripped through UTF-8 (the issue's own quoted sizes are exact UTF-8 byte counts of the true char counts: 56,864+163→57,027; 10,038→10,055). Verified on the actual ced8241-era binaries: byte-identical round-trips, compile true→true. The "opcode-0" report traces to stale databases/Frontier.root (pre-#621 corruption, #877). Records corrected on #866/#862/#849; #872's full-wrapper approach is unblocked. Tests
Reviewers
Follow-ups filed from this unit#877 (stale Frontier.root), #878 (65 cosmetic non-idempotent), #879→reclassified (#881 LF corruption — the #855 unblocker; #882 missing headless verbs), #880 (runner UTF-8 trap that caused the false diagnosis), #883 (CR-spanning residual), + scanner-deduplication refactor suggestion. Stats
|
1 participant
Fix #866 — Install Corruption: Comment-Marker Misread in Single-Quoted Constants
Purpose
This PR fixes cumulative script corruption on
script.newScriptObjectinstall. A 0xC7 comment-marker byte inside a single-quoted character constant was misread as a comment start, causing each script install to add one stray brace and rendering the script uncompilable from the first install. The fix restores the 1993-eralangcommentdeletesemantics this scanner's reimplementation had dropped, and also corrects a sibling closing-delimiter-direction miscount.Status
TESTED & VERIFIED — Full suite green (2482 tests, 0 failed, 17 baselined, exit 0). Critical test layer
script_install_roundtrip.yaml13/13 passing. The three original red cases verified red-then-green in both directions (fix reverted → red → restored). Corpus of 2,996 scripts: the one live victim repaired and stable; two former compile-hangs resolved. Gate: bar-raiser PASS + security PASS.Key Changes
/frontier-cli/src/lang/langscan.c(2 commits)Commit 76b7b8d (core fix):
inSingleQuotestate variable replacesflqcurlyboolean)'x'delimiters, preventing 0xC7 false positiveslangcommentdeletebehavior from pre-2019Commit 2b11766 (test & refactor coverage):
Issues Resolved
Fixes: #866
Corrections recorded on: #862, #849 (both had stale analysis due to issue #866 corruption)
Unblocks: #872 (full-wrapper approach now viable)
Follow-ups filed: #877, #878, #880, #881, #882, #883
Investigation Summary
The original issue report's claims about "reinstall destroys large unmodified scripts" and "opcode-0 format assertions" were investigated and refuted. Root causes:
Records on #866, #862, #849 have been corrected with this evidence.
Test Coverage
script_install_roundtrip.yaml— 13 cases covering single-quote literals, character constants, and round-trip integrityRelated PRs & Issues
langscan.c, no UserTalk or ODB changeshttps://claude.ai/code/session_01SEFFEgkaQJayd48Y2iuNkx