Skip to content

format: a CSV quotes the fields that need it, and you choose which - #46

Merged
donislawdev merged 3 commits into
mainfrom
format/csv-quote-style
Sep 3, 2026
Merged

format: a CSV quotes the fields that need it, and you choose which#46
donislawdev merged 3 commits into
mainfrom
format/csv-quote-style

Conversation

@donislawdev

Copy link
Copy Markdown
Owner

Adds quote_style to the csv format, taking minimal, all or none.

This is a breaking change under D11 and it is deliberate. The default is
minimal, so the description column - quoted on every row until now - is
quoted only when it carries the separator. Measured on a 4 kB table at seed 7
before the change: 9 of 44 rows carry a description with no separator,
because the phrase is three to seven words and drops a separator every third
one. Those nine lose their quotes. Sizes are unchanged and every reader that
took these files still takes them.

The values are the RFC 4180 vocabulary and nothing outside it. A fourth name
preserving today's bytes was rejected: the release this belongs to closes with
a major bump either way, so a clean vocabulary costs nothing now and a fourth
name would have been carried forever.

Two things that are not obvious from the list of values

none changes the CONTENT, not only the punctuation. An unquoted field
cannot hold a separator without ending early, so the description stops carrying
one - in the phrase and in the padding both. A value that only removed the
quotes would produce a ragged row at exactly the right size, which is the
defect this tool offers deliberately elsewhere.

The closing row is built to the byte, so under minimal the decision to
quote changes the length that the decision depends on.
Measured over 59
sizes: the padding first carries a separator at 30 B of description, so the
ambiguous band is two sizes per dialect. Resolved by MEASURING - the quoted
length is built first, and if it carries the separator the quotes are earned,
otherwise the description is rebuilt to the full room with the separator
withheld. A threshold constant was rejected as arithmetic that would have to
keep agreeing with the bytes beside it.

The rest

The floor moves with the setting, the way the dialect already does: 115 B
rather than 117
under minimal and none, 139 B under all. Twelve
combinations, eight distinct floors, all measured with the binary.

all quotes the header too, because a header is a row of fields and a writer
told to quote everything quotes those as well. Leaving it bare would have left
the structural checker with an "except the first row" exception, which is where
a defect hides.

The structural checker is now told the style and judges it, with negative
controls in both directions. That matters more here than for the other axes:
all three styles produce a well formed file, so nothing about the table gives
the style away and a rubber stamp would have been silent.

Verified

  • 1152 files swept over every size from 115 to 260 B across three styles
    and two delimiters - exact size, six columns under Python's csv module,
    quoting matching the style, no needless quote under minimal.
  • All 48 dialect combinations through the structural checker, plus seven
    negative controls.
  • LibreOffice Calc headless reads all three variants as six columns on
    every row, which closes D4 for the new values. A spreadsheet rather than a
    parser, so it answers a question our oracles cannot.
  • 16 of 16 mutations caught.

Two of those mutations came back NOT CAUGHT at first, and the fault was in
the guard rather than the code: both only shift the ANNOUNCED floor, which is
invisible everywhere else because Shortest is the worst draw and a real row
sits some forty bytes under it. The only handle is the count of distinct
floors, and it was written as six where the axes make eight.

An earlier version of the dialect guard also asserted that the closing row
always carries a separator inside its quotes. Under all it need not, because
all quotes regardless - the guard assumed a state it does not always reach.
It now walks a band of 24 sizes and asserts it reached one.

Full suite, preflight --quick (12 of 12, including the three CI-only checks)
and pinned staticcheck v0.8.1 are all green locally.

🤖 Generated with Claude Code

donislawdev and others added 3 commits September 3, 2026 14:32
Adds quote_style to the csv format, taking minimal, all or none. The
default is minimal, which changes the bytes of every table this tool
writes - the description column used to be quoted on every row and is
quoted now only when it carries the separator.

This is a breaking change under D11 and it is deliberate. Measured on a
4 kB table at seed 7 before the change: 9 of 44 rows carry a description
with no separator in it, because the phrase is three to seven words and
drops a separator every third one. Those nine lose their quotes. Sizes
are unchanged and every reader that took these files still takes them.

The values are the RFC 4180 vocabulary and nothing outside it. A fourth
name that preserved today's bytes was rejected: the release this belongs
to closes with a major bump either way, so a clean vocabulary costs
nothing now and a fourth name would have been carried forever.

Two things are not obvious from the list of values.

none changes the CONTENT, not only the punctuation. An unquoted field
cannot hold a separator without ending early, so the description stops
carrying one - in the phrase and in the padding both. A value that only
removed the quotes would produce a ragged row at exactly the right size.

And the closing row is built to the byte, so under minimal the decision
to quote changes the length that the decision depends on. Measured over
59 sizes: the padding first carries a separator at 30 B of description,
so the ambiguous band is two sizes per dialect. It is resolved by
MEASURING - the quoted length is built first, and if it carries the
separator the quotes are earned, otherwise the description is rebuilt to
the full room with the separator withheld. A threshold constant was
rejected as arithmetic that has to keep agreeing with the bytes beside it.

The floor moves with the setting, the way the dialect already does: 115 B
rather than 117 under minimal and none, 139 B under all. Twelve
combinations, eight distinct floors, all measured with the binary.

all quotes the header too, because a header is a row of fields and a
writer told to quote everything quotes those as well. The alternative
left the structural checker with an "except the first row" exception,
which is where a defect hides.

The checker is now told the style and judges it, with negative controls
in both directions. That matters more here than for the other axes: every
one of the three styles produces a well formed file, so nothing about the
table gives the style away and a rubber stamp would have been silent.

Verified: 1152 files swept over every size from 115 to 260 B across three
styles and two delimiters - exact size, six columns under Python's csv
module, quoting matching the style, and no needless quote under minimal.
All 48 dialect combinations through the structural checker. LibreOffice
Calc headless reads all three variants as six columns on every row, which
closes D4 for the new values. 16 of 16 mutations caught.

Two of those mutations were NOT CAUGHT at first, and the fault was in the
guard: both only shift the ANNOUNCED floor, which is invisible elsewhere
because Shortest is the worst draw and a real row sits some forty bytes
under it. The only handle is the count of distinct floors, and it was
written as six where the axes make eight.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…stry

Two things, and the first is why CI was red.

The coverage gate died on a limit nobody had chosen. Go allows ten minutes
per package by default while that job allows twenty for all of it, so
internal/guard hit the first without coming near the second - a stack
trace out of whichever test was running when the alarm went off, instead
of a failure naming something. Exactly what the race job above it met on
2026-08-25 and fixed the same way.

Measured on the runner rather than guessed: the step took 375 s and 457 s
on two consecutive main runs, which is 22 percent of variance on code
that barely moved, and the default cuts in at 600 s. This branch added
eight seconds of coverage instrumented work - measured, both new CSV
guards together - and tipped it. Eight seconds is not what went wrong.
457 against 600 was never a margin.

The second thing is O176: the README settings table had no guard and
disagreed with the registry on seven rows. log said "none" while carrying
eight settings, zip and targz listed three of eight, and avif and jxl had
no row at all. The site has had this guard since it was built. The one
page a visitor reads first did not.

Two guards rather than one, because the table turned out to be the second
half of the problem. The list at the top of the README was missing jxl
outright - it arrived on 2026-08-31 as the twenty fourth format and never
reached that list, so the page offered twenty three while the binary
shipped twenty four. The prose said "twenty two" in three places and "24"
in two: one file, three different numbers about one thing.

Settings are compared as SETS. Whether a row reads "width, height" or the
other way round is a question about English, and a guard answering it
would be refusing prose rather than catching a lie.

The count spelled out in words is deliberately not guarded. That was
measured and rejected on 2026-08-05 in copiednumbers_test.go: the general
form raised 43 findings and most were false, because "24 formats" and
"25 formats" and "150 formats" answer three different questions here.

Both guards are proven by mutation - a setting renamed in the registry,
and a format registered under a name the list does not carry. That
matters because the neighbouring TestTheFormatDocumentAgreesWithTheRegistry
sits on notProvenByMutation, and that list is only allowed to shrink.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t that way

Follow through on the commit before this one, which fixed the coverage
gate and left three jobs sitting in exactly the same place.

Measured on the green run of 2026-09-03, rather than assumed from the one
job that went red: the test step takes 399 s on ubuntu, 476 s on windows
and 491 s on macOS, all against Go's unstated ten minutes a package. macOS
had 109 s of room. The same fleet was measured swinging from 479 s to past
600 s between two runs of one branch, so the margin was smaller than the
variance on every one of them. The release workflow had the same gap, and
there a timeout would read as a red tree and stop a release that was fine.

Six whole tree runs now state a timeout. The two that already did are
unchanged.

The guard is the point of this commit rather than the flags. This is the
SECOND time the project has lost a run to Go's default - the race detector
met it on 2026-08-25 and the answer was a long comment beside that one
job, which is why the coverage gate met it again eight days later. A
diagnosis recorded at one step does not protect the next one, so the
reasoning has moved out of the comments and into
TestEveryWholeTreeTestRunStatesItsOwnTimeout. Proven by mutation.

It reads the workflows through the YAML parser rather than as text,
because a run block can be folded and the flag would then sit on a
different line from the command. Only whole tree runs are asked: a
targeted -run walks a handful of tests, and the fuzz step carries
-fuzztime, which is its own budget.

One stale claim fixed on the way. The matrix job's own comment said "the
matrix runs in about a minute", which had not been true for a long time -
it is eight, and the numbers are written down now instead of a word.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@donislawdev
donislawdev merged commit a190b7b into main Sep 3, 2026
18 checks passed
@donislawdev
donislawdev deleted the format/csv-quote-style branch September 3, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant