Format/archives pad and lock - #32
Merged
Merged
Conversation
…ded before O163, and a breaking change the owner asked for. Sizes do not move and every size that was reachable still is. What moves is where the padding sits inside the gzip header, and with it the bytes of every .tar.gz this tool writes. The fault was invisible for a month and each reason it stayed that way is worth naming, because none of them was carelessness. Go's compress/gzip reads a header comment into a fixed buffer: 511 bytes it takes, 512 it refuses. This format padded through that comment up to four kilobytes, so 4134 of the 11 260 reachable sizes produced an archive no Go program could open. The message it gives is "gzip: invalid header", which reads like a corrupt file rather than like a field the reader will not take, so anybody meeting it would suspect the fixture rather than the tool that made it. Testers write tools in Go. Every reference tool took those files. 7-Zip, GNU tar, bsdtar, Python and node accept a comment of ten megabytes without a word, so the oracle was green on all of them. The channel had been measured carefully in August and written up, and the measurement asked five readers - the language this is written in was not among them. That is the same shape as trusting a document instead of measuring: the question was right and the sample was one entry short. Both fixes named in the observation were measured and rejected. Capping the comment at what Go takes gives zero unreadable archives and takes 831 sizes out of reach, because a tar filler entry cannot be smaller than one 512 byte block and nothing bridges the gap under it. Trading 4134 unreadable files for 831 refusals is not a fix, and it would have looked like one in a summary. So the padding moved to the gzip extra field, which was measured the same day rather than assumed: Go takes 65 531 bytes of it, to the last one, beside a comment, at a cost of exactly two bytes more than it carries. Byte granular, where the filler entry is aligned to 512. The comment keeps the label and stretches by at most one byte, which is the one amount the extra field cannot do, so the pair together reach every size. After: 11 260 reachable, 0 unreachable, 0 unreadable. Reachability identical to before, which is the number that decided it. 7-Zip, GNU tar and Python were run on real output afterwards and all three open it. Three golden hashes moved, recorded through the remeasured list with the measurement, because editing them quietly is the single move that turns that guard into decoration. There is no way back to the old bytes and the changelog says so: the old bytes are the ones Go cannot read, so a switch for them would be a switch for the fault. TestEveryArchiveThisToolWritesCanBeReadByTheStandardLibrary now asks both containers, across the band where the padding channel changes shape, whether the standard library opens what we wrote. 1762 archives. It is the guard that would have caught this on the day it went in, and the reason it did not exist is that every check we had asked somebody else's reader. The sweep that priced all of this is tools/probes/targzpadding, and it asks two questions rather than one: which sizes are REACHABLE and which of those are READABLE. Run it before and after any change to a padding limit. Reachability is the one that gets worse quietly. validCost, commentOfCost and commentCapacity had no readers left afterwards and are gone. The measurement commentCapacity carried moved into the comment on the limit that still binds, because a fact does not stop being true when the constant holding it does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…aving for what it breaks
tfg generate --format zip --size 30kb --set entries=3 \
--set password=Secret123 --set encryption=zipcrypto
It is here for what it does to a reader rather than for what it protects.
Measured on our own output: .NET's ZipFile opens one of these, reports the entry
at its true length of 8192, hands back a stream and fills it with ff c7 04 3e
where the file holds "tfg - txt". It never says the entry was encrypted at all,
so an application built on that library processes noise and calls it data. AES
in the same library throws, which is the safer defect and the less interesting
one. This is the fixture PRESETS.md section 4.12 has been describing since
before there was anything to build it with.
The cipher is not ours and it was not taken on trust. It went into
tools/probes/zipcrypto BEFORE it went into the generator, pointed at an archive
7-Zip had written, and asked to decrypt it - check byte, plaintext CRC and the
bytes themselves all came back right. That order was chosen because of what the
AES work had shown a few hours earlier: a mutation proved the archiver guard
could not see a keystream running backwards, since an AE-2 entry signs its
ciphertext rather than its contents, so a file of noise passes every check a
reader makes.
A defect got through anyway, and the probe is what diagnosed it. Every locked
entry declared method 99 - WinZip AES - while a ZipCrypto entry has to stay
stored, because it changes nothing about how the bytes sit and only puts twelve
in front of them. 7-Zip reported "Data Error in encrypted file. Wrong password?"
which points at the password, the one thing that was right. The probe decrypted
the same file perfectly, and that is what moved the suspicion off the
cryptography and onto the header.
So there is a guard for the header shape now, and it asks both schemes in both
directions. ZipCrypto stores, carries the real plaintext CRC and no extra field.
AES declares 99, carries a CRC of nought and a 0x9901 field. Getting either
backwards produces a file that opens and then fails on something that sounds
like the user's fault.
The cost is named rather than hidden: ZipCrypto puts the high byte of the
plaintext CRC in its header, so the contents have to be known before the first
byte of the entry goes out - and the contents arrive as a stream. Each entry is
therefore generated twice, once to be checksummed and once to be encrypted.
Twice the processor and not a byte more memory, because buffering the entry
would break the guard that says a generator does not hold a whole file.
Three guards stopped carrying a hand written list of methods and read the
registry instead. A list in a test is one somebody has to remember on the day a
fourth scheme arrives, and this was that day.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ne mistake 691 entries, 74.0 minutes, five guards that stayed green while the thing they name was broken. Every one is closed and proven. Three of the five are the same mistake wearing different clothes: a guard asking "is this text in the file" stops meaning anything the day the text appears a second time. This tree has that trap written down and still collected three of them in one run. The window binary. The guard asked whether ci.yml mentions ./cmd/tfg-gui anywhere. It appears three times now - and the third arrived later, in the sbom job, which runs on Linux alone. Removing both builds from the platform matrix left the guard green, so a window that stops compiling on Windows or macOS would have passed in silence, which is the whole thing it watches for. It asks the matrix job now. The release notes. The notes carry two attestation commands and they answer different questions - what is inside a file, which needs the predicate type spelled out, and where a Linux archive came from, which must not carry it. Breaking one left the phrase present in the other. Both are required now, by count and by name. The menu. The mutation turned off the branch that widens a menu box to what a row of its open list needs, and measuring says that branch is unreachable for every menu this window has: the guard prints the same tightest slot with it on and off. It was reachable once - the comment beside it records the day twenty values came out cut off - so the data changed, not the rule. Deleting a defence in the window is not something to do in passing, so it is written up as O164 with three options and the price of each. The mutation moved to what the guard actually names: the room a row gives its words. Four real values come out cut off and it names each. The fourth is this branch's own doing. The comment terminator mutation used to move the floor, and the minimum guard watched the floor. Padding rode in the comment then, so an off by one in commentCost was absorbed - the solve built a comment one byte longer and the archive still came out exact. Padding rides in the extra field now, so the same byte lands on the size instead, and three other guards go red on it. The entry follows it to the arithmetic guard that owns the fact. The fifth is a guard whose sample was short. JXL checks that a named quality still fits the file, across five qualities and three sizes - and the break shows at quality 100 and 5000 B, which was not among them. Two sizes added. That is the same shape as O163 one file over: a measurement proves what its sample covers and says nothing about the gap beside it. CLAUDE.md gets the measured run time, because the sentence there was written at 512 entries. 74.0 min at 691, median 4.64 s an entry, 99% of it inside go test - and a note not to estimate it from the first dozen, which I did twice that day and was wrong by a factor of four both times. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.