Skip to content

format: archives get settings - a password on zip, permissions and an owner on tar.gz - #30

Merged
donislawdev merged 4 commits into
mainfrom
format/archives-get-settings
Sep 1, 2026
Merged

format: archives get settings - a password on zip, permissions and an owner on tar.gz#30
donislawdev merged 4 commits into
mainfrom
format/archives-get-settings

Conversation

@donislawdev

Copy link
Copy Markdown
Owner

Settings for archives: a password on zip, permissions and an owner on tar.gz.
Three commits, one subject.

A zip can be locked

tfg generate --format zip --size 30kb --set entries=3 \
  --set password=Secret123 --set encryption=aes-256

writes an archive of exactly 30720 B that 7-Zip opens with that password
and refuses without it. aes-128, aes-192, aes-256.

It is possible at all because of one measurement. A stream cipher does not
change the length of what it encrypts, so a lock adds a fixed count per
entry
- measured twice over, from the size of the file and from the
compressed size field of what 7-Zip writes:

data per entry extra field, local + central total
ZipCrypto +12 0 +12
AES-128 +20 11 + 11 +42
AES-192 +24 11 + 11 +46
AES-256 +28 11 + 11 +50

So the size of the archive stays an exact function of its input and the
planning phase can still state it without building it, which is the guard the
whole container design rests on. Compression has no such property, and that is
why this setting exists and that one still does not.

Two more facts were read out of the bytes 7-Zip writes, and each decided half
the work:

  • An AE-2 entry carries a CRC of zero, because the authentication code is
    what proves the contents. Nothing about the plaintext has to be known before
    the first byte, so an entry is encrypted as it streams and an archive of any
    size costs the memory of an archive of one byte.
  • ZipCrypto carries the real plaintext CRC, so it cannot be written the
    same way. It is not here for that reason rather than because it was
    forgotten - MVP-FORMATS.md section 2.16 says what it would take.

The salt is derived from the run seed and never drawn. A salt from
crypto/rand gives a perfectly good archive that differs on every run, which
is untouchable rule 3 broken in the one place where every other check stays
green.

The password goes into the manifest in plain text, by the owner's decision.
A locked fixture nobody can open is worth nothing. It has its own guard, because
the next person to read it will assume it is a mistake.

Two settings say it together

A password with encryption=none, or an encryption with no password, is
refused and both refusals name both halves. That is forced by the window rather
than chosen: a menu cannot be empty, so encryption arrives as none from
every window run whether or not anybody looked at it, while a box somebody
types in arrives empty and is left out.

tar says plainly that it cannot

targz does not have a property called password is true and sends somebody
looking for the build that has it. There is none. Descriptors can now declare
what they deliberately cannot take, with the reason, and the refusal lands
on FORMAT rather than USAGE - a deliberate limitation is not a spelling mistake.

The alternative in the world is worse than either, and it is why this refuses
instead of ignoring. Measured: 7-Zip accepts -p on a tar, exits 0, prints
nothing and writes a plaintext archive.
Somebody asking for a locked fixture
gets an open one and never finds out.

A tar.gz can say what its files may do

--set entry_mode=755 --set entry_owner=root gives an archive where
tar -tvzf reads -rwxr-xr-x root/root. Modes 000 through 777, owners
unset, root, user.

It costs no bytes: every field of a USTAR header is fixed width. The useful
cases are the ones nobody makes by accident - 000 is a file nothing can read
after unpacking, 777 is one a scanner should have something to say about, and
an archive claiming root owns everything is what a careless extractor turns
into a privilege problem.

The defaults are what this format has always written, to the byte.

Both settings sit in the shared archive vocabulary while only targz names them,
which is the other direction of the same asymmetry the password showed.

Two findings that were not the task

A mutation found that two of the new guards were proving less than they
said.
The archiver guard could not see a broken keystream, and the reason is a
property of AE-2 rather than an oversight: its authentication code is computed
over the ciphertext, so a reader checks that nobody edited the encrypted bytes
and never that the plaintext is what was put in. Turn the counter the wrong way
and 7-Zip still says "Everything is Ok". It now extracts the archive for real
and compares against the same archive built unlocked. The determinism guard
could not see a salt that ignored the seed, because the children are seeded from
the run too - it now reads the salt bytes directly.

Go cannot read a lot of the .tar.gz files this tool writes, and nobody knew.
Found because a guard tried to read an archive we had just written and could
not. Measured:

comment length compress/gzip accepts 511 B, refuses from 512
our padding channel up to 4096 B through that comment
archive sizes readable by Go 548 of 866
not readable 318 (37%), first at 10269 B

Every one of those is fine to 7-Zip, GNU tar and bsdtar, and every one fails in
a Go program with gzip: invalid header - which reads like a corrupt file
rather than like a field this reader will not take. Testers write tools in Go.

Not fixed here. Both ways of fixing it move bytes, which is untouchable rule
3 and the owner's call. Written up as O163 with the two options and what each
costs. The guard steps past the gzip header by hand meanwhile, so a test about
permissions does not fail on a fault that is not its subject.

Evidence

  • 7-Zip opens every locked archive with the right password and refuses the
    wrong one and none at all. Contents are extracted and compared against the
    unlocked archive, so a scrambled keystream cannot pass.
  • Exact size held for every method across four sizes, including odd ones.
  • TestOurOwnGeneratorsHaveNotDrifted green: nothing that was not asked for
    moved.
  • Twelve new guards, ten proven by mutation. One is on the unproven list with
    its reason - it defends a property of the USTAR header rather than of any line
    of ours.
  • Code shape gates green. D1 parity now 97 of 108.

donislawdev and others added 4 commits September 1, 2026 10:15
…t it cannot

    tfg generate --format zip --size 30kb --set entries=3 \
      --set password=Secret123 --set encryption=aes-256

writes an archive of exactly 30720 B that 7-Zip opens with that password and
refuses without it. AES-128, AES-192 and AES-256 are the methods, and the
manifest carries the password in plain text - a locked fixture nobody can open
is worth nothing, so writing it down is the point rather than a leak.

It is possible at all because of one measurement, in docs/MVP-FORMATS.md
section 2.16. A stream cipher does not change the length of what it encrypts,
so a lock adds a FIXED count per entry - 20, 24 or 28 bytes of data by key
length, plus an eleven byte field in each of the two headers. The size of the
archive therefore stays an exact function of its input and the planning phase
can still state it without building it, which is the guard the whole container
design rests on. Compression has no such property, and that is why this setting
exists and that one still does not.

Two more things were read out of the bytes 7-Zip writes rather than out of a
specification recalled, and each decided half the work:

  An AE-2 entry carries a CRC of zero, because the authentication code is what
  proves the contents. Nothing about the plaintext has to be known before the
  first byte, so an entry is encrypted as it streams and an archive of any size
  costs the memory of an archive of one byte.

  ZipCrypto carries the real plaintext CRC, so it cannot be written the same
  way. It is not in this commit for that reason rather than because it was
  forgotten - the note in section 2.16 says what it would take.

The salt is derived from the run seed and never drawn. A salt from crypto/rand
gives a perfectly good archive that differs on every run, which is untouchable
rule 3 broken in the one place where every other check stays green.

Two settings say this together, and the two that disagree are refused. That is
forced by the window rather than chosen: a menu cannot be empty, so encryption
arrives as "none" from every window run whether or not anybody looked at it,
while a box somebody types in arrives empty and is left out. "Password given,
encryption none" therefore cannot be told apart from "password given,
encryption not considered", and both refusals name both halves so nobody is
left staring at a filled in password box.

tar and gzip get their own refusal rather than the generic one. "targz does not
have a property called password" is true and sends somebody looking for the
build that has it, and there is none: neither format has any encryption in it.
Descriptors can now declare what they deliberately cannot take, with the reason,
and the refusal lands on FORMAT rather than USAGE - a deliberate limitation is
not a spelling mistake.

The alternative in the world is worse than either, and it is why this refuses
instead of ignoring. Measured the same day: 7-Zip accepts -p on a tar, exits 0,
prints nothing and writes a PLAINTEXT archive. Somebody asking for a locked
fixture gets an open one and never finds out.

Seven guards, one of which is the whole feature: 7-Zip opens the archive with
the right password and REFUSES the wrong one. A reader that accepted anything
would have passed the first half while proving nothing.

format.go lost its refusals to refusals.go on the way, because the crowding
gate went red at three files in the band against a cap of two. The cut is by
subject - what a format declares in one file, what it says when a setting is
wrong in the other.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n said so

Two of the seven mutations written for the locking work came back NOT CAUGHT,
and neither was a false alarm. Both guards had a real hole.

The archiver guard could not see a broken keystream, and the reason is a
property of AE-2 rather than an oversight. An AE-2 entry carries a CRC of zero
and its authentication code is computed over the CIPHERTEXT, so a reader checks
that nobody edited the encrypted bytes and never that the plaintext is what was
put in. Turn the counter the wrong way round and 7-Zip still says "Everything
is Ok": the file is well formed, the password verifies, the code matches, and
what comes out is noise. The guard now opens the archive for real and compares
what comes out against the same archive built without a lock, which holds the
same children from the same seed. Both sides are extracted by 7-Zip, so this is
still not our own code judging its own arithmetic.

The determinism guard could not see a salt that ignored the seed. Comparing
whole archives is too blunt for that: the children are seeded from the run too,
so two seeds give different contents and therefore different files whatever the
salt does. It now reads the sixteen bytes where a WinZip AES entry keeps its
salt and asks them directly, which DataOffset makes possible without a second
zip parser.

Both mutations are caught now. Nine of nine.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
    tfg generate --format targz --size 12kb --set entry_mode=755 --set entry_owner=root

gives an archive where tar -tvzf reads "-rwxr-xr-x root/root". Modes are the
ones chmod takes, 000 through 777, and owners are unset, root and user.

It costs nothing, and that is why it exists rather than waiting. Every field of
a USTAR header is fixed width, so the mode and the owner are written into space
already paid for whatever they say - the archive is the same size either way,
so this does not touch the exact size the format promises the way compression
would.

The useful cases are the ones nobody makes by accident. 000 is a file nothing
can read after unpacking, 777 is one a scanner should have something to say
about, and an archive claiming root owns everything is what a careless
extractor turns into a privilege problem. Building one of those by hand needs a
machine with the right permissions. Here it is a flag.

The defaults are what this format has always written, to the byte, and that is
untouchable rule 3 rather than a preference: a default that recorded an owner
would move the bytes of every archive already produced, silently, because an
archive with an owner in it is just as valid as one without.

Both settings sit in the shared archive vocabulary while only targz names them,
which is the other direction of the same asymmetry the password showed. Zip
declares a lock tar has nowhere to put, tar declares an owner zip records
differently, and each format lists what it can actually carry.

One finding came out of this that is bigger than the setting, and it was found
because a guard tried to read an archive this tool had just written and could
not. Go's compress/gzip accepts a header comment of 511 bytes and refuses one
of 512, because it reads the field into a fixed buffer. This format pads
through that comment, up to four kilobytes of it. Measured across 866 archive
sizes from the minimum upwards: 548 readable by Go, 318 not, first failure at
10269 B. Every one of those is fine to 7-Zip, GNU tar and bsdtar, and every one
of them fails in a Go program with "gzip: invalid header" - a message that
reads like a corrupt file rather than like a field this reader will not take.

Not fixed here, because both ways of fixing it move bytes and that is the
owner's call under untouchable rule 3. Written up as O163 with the two options
and what each one would cost. The guard reads past the gzip header by hand in
the meantime, so a test about permissions does not fail on a fault that is not
its subject.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI came back red on five jobs and every cause was in this branch. Four of them
were mechanical. The fifth was a defect worth more than the fix.

The one that mattered: encryption declared zipcrypto among its choices while
nothing here writes it. The registry accepted the value because it was in the
declared set, the plan carried it, and the write failed at the last moment - so
"--set encryption=zipcrypto" produced no file, exit code 5 and a note in the
manifest, for a value the tool had listed itself. "tfg formats zip" printed it
and the website published it.

That is the shape untouchable rule 6 is about, and it is the same thing this
branch criticises 7-Zip for one commit earlier. A refusal has to come from the
declaration, before anything is attempted, or not be needed at all. Offering
something and failing on it later is the worst of the three.

Found by reading the diff of the regenerated website rather than by any test,
which is why there is now a guard: every encryption a container offers is
planned and written, and the check reads the declaration rather than a list of
its own, so the day 7Z arrives with a method of its own it is covered without
being edited. The mutation for it puts the unimplemented method back.

The ZipCrypto constant and its branch in EntryOverhead went with it. Nothing
could select the value any more, so the branch was unreachable, and this tree
deletes a defence nothing can turn red rather than keeping it for the look of
it. The measurement it came from is not lost - it is in MVP-FORMATS.md section
2.16, along with why writing that scheme needs a second pass over each entry.

The other four:

  Two doc comments had been separated from the declarations they were written
  for, by insertions in this branch: build in zip.go and sevenZip in oracle.go
  both ended up documenting whatever followed them. Moved back.

  The website did not know about the four new settings. Regenerated under
  go1.26.7 with GOTOOLCHAIN pinned to it, checked first, because a regeneration
  under a newer local compiler writes numbers CI will never agree with.

  gosec refused crypto/sha1. It is not a choice here: WinZip AES derives its key
  with PBKDF2-HMAC-SHA1 and signs the ciphertext with HMAC-SHA1, both named in
  the specification, and an archive built with anything else is one no archiver
  opens. Silenced with that reason rather than worked around, and it is used for
  key derivation and a message code, never as a digest trusted to be collision
  free.

  Two nolint directives in the guard were unused and are gone.

The gzcomment probe joins the index it belongs in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@donislawdev
donislawdev merged commit 48b15ca into main Sep 1, 2026
18 checks passed
@donislawdev
donislawdev deleted the format/archives-get-settings branch September 1, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant