format: archives get settings - a password on zip, permissions and an owner on tar.gz - #30
Merged
Merged
Conversation
…t it cannot
tfg generate --format zip --size 30kb --set entries=3 \
--set password=Secret123 --set encryption=aes-256
writes an archive of exactly 30720 B that 7-Zip opens with that password and
refuses without it. AES-128, AES-192 and AES-256 are the methods, and the
manifest carries the password in plain text - a locked fixture nobody can open
is worth nothing, so writing it down is the point rather than a leak.
It is possible at all because of one measurement, in docs/MVP-FORMATS.md
section 2.16. A stream cipher does not change the length of what it encrypts,
so a lock adds a FIXED count per entry - 20, 24 or 28 bytes of data by key
length, plus an eleven byte field in each of the two headers. The size of the
archive therefore stays an exact function of its input and the planning phase
can still state it without building it, which is the guard the whole container
design rests on. Compression has no such property, and that is why this setting
exists and that one still does not.
Two more things were read out of the bytes 7-Zip writes rather than out of a
specification recalled, and each decided half the work:
An AE-2 entry carries a CRC of zero, because the authentication code is what
proves the contents. Nothing about the plaintext has to be known before the
first byte, so an entry is encrypted as it streams and an archive of any size
costs the memory of an archive of one byte.
ZipCrypto carries the real plaintext CRC, so it cannot be written the same
way. It is not in this commit for that reason rather than because it was
forgotten - the note in section 2.16 says what it would take.
The salt is derived from the run seed and never drawn. A salt from crypto/rand
gives a perfectly good archive that differs on every run, which is untouchable
rule 3 broken in the one place where every other check stays green.
Two settings say this together, and the two that disagree are refused. That is
forced by the window rather than chosen: a menu cannot be empty, so encryption
arrives as "none" from every window run whether or not anybody looked at it,
while a box somebody types in arrives empty and is left out. "Password given,
encryption none" therefore cannot be told apart from "password given,
encryption not considered", and both refusals name both halves so nobody is
left staring at a filled in password box.
tar and gzip get their own refusal rather than the generic one. "targz does not
have a property called password" is true and sends somebody looking for the
build that has it, and there is none: neither format has any encryption in it.
Descriptors can now declare what they deliberately cannot take, with the reason,
and the refusal lands on FORMAT rather than USAGE - a deliberate limitation is
not a spelling mistake.
The alternative in the world is worse than either, and it is why this refuses
instead of ignoring. Measured the same day: 7-Zip accepts -p on a tar, exits 0,
prints nothing and writes a PLAINTEXT archive. Somebody asking for a locked
fixture gets an open one and never finds out.
Seven guards, one of which is the whole feature: 7-Zip opens the archive with
the right password and REFUSES the wrong one. A reader that accepted anything
would have passed the first half while proving nothing.
format.go lost its refusals to refusals.go on the way, because the crowding
gate went red at three files in the band against a cap of two. The cut is by
subject - what a format declares in one file, what it says when a setting is
wrong in the other.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n said so Two of the seven mutations written for the locking work came back NOT CAUGHT, and neither was a false alarm. Both guards had a real hole. The archiver guard could not see a broken keystream, and the reason is a property of AE-2 rather than an oversight. An AE-2 entry carries a CRC of zero and its authentication code is computed over the CIPHERTEXT, so a reader checks that nobody edited the encrypted bytes and never that the plaintext is what was put in. Turn the counter the wrong way round and 7-Zip still says "Everything is Ok": the file is well formed, the password verifies, the code matches, and what comes out is noise. The guard now opens the archive for real and compares what comes out against the same archive built without a lock, which holds the same children from the same seed. Both sides are extracted by 7-Zip, so this is still not our own code judging its own arithmetic. The determinism guard could not see a salt that ignored the seed. Comparing whole archives is too blunt for that: the children are seeded from the run too, so two seeds give different contents and therefore different files whatever the salt does. It now reads the sixteen bytes where a WinZip AES entry keeps its salt and asks them directly, which DataOffset makes possible without a second zip parser. Both mutations are caught now. Nine of nine. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tfg generate --format targz --size 12kb --set entry_mode=755 --set entry_owner=root gives an archive where tar -tvzf reads "-rwxr-xr-x root/root". Modes are the ones chmod takes, 000 through 777, and owners are unset, root and user. It costs nothing, and that is why it exists rather than waiting. Every field of a USTAR header is fixed width, so the mode and the owner are written into space already paid for whatever they say - the archive is the same size either way, so this does not touch the exact size the format promises the way compression would. The useful cases are the ones nobody makes by accident. 000 is a file nothing can read after unpacking, 777 is one a scanner should have something to say about, and an archive claiming root owns everything is what a careless extractor turns into a privilege problem. Building one of those by hand needs a machine with the right permissions. Here it is a flag. The defaults are what this format has always written, to the byte, and that is untouchable rule 3 rather than a preference: a default that recorded an owner would move the bytes of every archive already produced, silently, because an archive with an owner in it is just as valid as one without. Both settings sit in the shared archive vocabulary while only targz names them, which is the other direction of the same asymmetry the password showed. Zip declares a lock tar has nowhere to put, tar declares an owner zip records differently, and each format lists what it can actually carry. One finding came out of this that is bigger than the setting, and it was found because a guard tried to read an archive this tool had just written and could not. Go's compress/gzip accepts a header comment of 511 bytes and refuses one of 512, because it reads the field into a fixed buffer. This format pads through that comment, up to four kilobytes of it. Measured across 866 archive sizes from the minimum upwards: 548 readable by Go, 318 not, first failure at 10269 B. Every one of those is fine to 7-Zip, GNU tar and bsdtar, and every one of them fails in a Go program with "gzip: invalid header" - a message that reads like a corrupt file rather than like a field this reader will not take. Not fixed here, because both ways of fixing it move bytes and that is the owner's call under untouchable rule 3. Written up as O163 with the two options and what each one would cost. The guard reads past the gzip header by hand in the meantime, so a test about permissions does not fail on a fault that is not its subject. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI came back red on five jobs and every cause was in this branch. Four of them were mechanical. The fifth was a defect worth more than the fix. The one that mattered: encryption declared zipcrypto among its choices while nothing here writes it. The registry accepted the value because it was in the declared set, the plan carried it, and the write failed at the last moment - so "--set encryption=zipcrypto" produced no file, exit code 5 and a note in the manifest, for a value the tool had listed itself. "tfg formats zip" printed it and the website published it. That is the shape untouchable rule 6 is about, and it is the same thing this branch criticises 7-Zip for one commit earlier. A refusal has to come from the declaration, before anything is attempted, or not be needed at all. Offering something and failing on it later is the worst of the three. Found by reading the diff of the regenerated website rather than by any test, which is why there is now a guard: every encryption a container offers is planned and written, and the check reads the declaration rather than a list of its own, so the day 7Z arrives with a method of its own it is covered without being edited. The mutation for it puts the unimplemented method back. The ZipCrypto constant and its branch in EntryOverhead went with it. Nothing could select the value any more, so the branch was unreachable, and this tree deletes a defence nothing can turn red rather than keeping it for the look of it. The measurement it came from is not lost - it is in MVP-FORMATS.md section 2.16, along with why writing that scheme needs a second pass over each entry. The other four: Two doc comments had been separated from the declarations they were written for, by insertions in this branch: build in zip.go and sevenZip in oracle.go both ended up documenting whatever followed them. Moved back. The website did not know about the four new settings. Regenerated under go1.26.7 with GOTOOLCHAIN pinned to it, checked first, because a regeneration under a newer local compiler writes numbers CI will never agree with. gosec refused crypto/sha1. It is not a choice here: WinZip AES derives its key with PBKDF2-HMAC-SHA1 and signs the ciphertext with HMAC-SHA1, both named in the specification, and an archive built with anything else is one no archiver opens. Silenced with that reason rather than worked around, and it is used for key derivation and a message code, never as a digest trusted to be collision free. Two nolint directives in the guard were unused and are gone. The gzcomment probe joins the index it belongs in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Settings for archives: a password on zip, permissions and an owner on tar.gz.
Three commits, one subject.
A zip can be locked
writes an archive of exactly 30720 B that 7-Zip opens with that password
and refuses without it.
aes-128,aes-192,aes-256.It is possible at all because of one measurement. A stream cipher does not
change the length of what it encrypts, so a lock adds a fixed count per
entry - measured twice over, from the size of the file and from the
compressed size field of what 7-Zip writes:
So the size of the archive stays an exact function of its input and the
planning phase can still state it without building it, which is the guard the
whole container design rests on. Compression has no such property, and that is
why this setting exists and that one still does not.
Two more facts were read out of the bytes 7-Zip writes, and each decided half
the work:
what proves the contents. Nothing about the plaintext has to be known before
the first byte, so an entry is encrypted as it streams and an archive of any
size costs the memory of an archive of one byte.
same way. It is not here for that reason rather than because it was
forgotten -
MVP-FORMATS.mdsection 2.16 says what it would take.The salt is derived from the run seed and never drawn. A salt from
crypto/randgives a perfectly good archive that differs on every run, whichis untouchable rule 3 broken in the one place where every other check stays
green.
The password goes into the manifest in plain text, by the owner's decision.
A locked fixture nobody can open is worth nothing. It has its own guard, because
the next person to read it will assume it is a mistake.
Two settings say it together
A password with
encryption=none, or an encryption with no password, isrefused and both refusals name both halves. That is forced by the window rather
than chosen: a menu cannot be empty, so
encryptionarrives asnonefromevery window run whether or not anybody looked at it, while a box somebody
types in arrives empty and is left out.
tar says plainly that it cannot
targz does not have a property called passwordis true and sends somebodylooking for the build that has it. There is none. Descriptors can now declare
what they deliberately cannot take, with the reason, and the refusal lands
on FORMAT rather than USAGE - a deliberate limitation is not a spelling mistake.
The alternative in the world is worse than either, and it is why this refuses
instead of ignoring. Measured: 7-Zip accepts
-pon a tar, exits 0, printsnothing and writes a plaintext archive. Somebody asking for a locked fixture
gets an open one and never finds out.
A tar.gz can say what its files may do
--set entry_mode=755 --set entry_owner=rootgives an archive wheretar -tvzfreads-rwxr-xr-x root/root. Modes000through777, ownersunset,root,user.It costs no bytes: every field of a USTAR header is fixed width. The useful
cases are the ones nobody makes by accident -
000is a file nothing can readafter unpacking,
777is one a scanner should have something to say about, andan archive claiming root owns everything is what a careless extractor turns
into a privilege problem.
The defaults are what this format has always written, to the byte.
Both settings sit in the shared archive vocabulary while only targz names them,
which is the other direction of the same asymmetry the password showed.
Two findings that were not the task
A mutation found that two of the new guards were proving less than they
said. The archiver guard could not see a broken keystream, and the reason is a
property of AE-2 rather than an oversight: its authentication code is computed
over the ciphertext, so a reader checks that nobody edited the encrypted bytes
and never that the plaintext is what was put in. Turn the counter the wrong way
and 7-Zip still says "Everything is Ok". It now extracts the archive for real
and compares against the same archive built unlocked. The determinism guard
could not see a salt that ignored the seed, because the children are seeded from
the run too - it now reads the salt bytes directly.
Go cannot read a lot of the
.tar.gzfiles this tool writes, and nobody knew.Found because a guard tried to read an archive we had just written and could
not. Measured:
compress/gzipacceptsEvery one of those is fine to 7-Zip, GNU tar and bsdtar, and every one fails in
a Go program with
gzip: invalid header- which reads like a corrupt filerather than like a field this reader will not take. Testers write tools in Go.
Not fixed here. Both ways of fixing it move bytes, which is untouchable rule
3 and the owner's call. Written up as
O163with the two options and what eachcosts. The guard steps past the gzip header by hand meanwhile, so a test about
permissions does not fail on a fault that is not its subject.
Evidence
wrong one and none at all. Contents are extracted and compared against the
unlocked archive, so a scrambled keystream cannot pass.
TestOurOwnGeneratorsHaveNotDriftedgreen: nothing that was not asked formoved.
its reason - it defends a property of the USTAR header rather than of any line
of ours.
D1parity now 97 of 108.