format: archives pad where Go can read them, and can be locked with the old scheme too

[donislawdev]

Two commits. One fixes a fault nobody knew about, the other adds the lock that
finds faults in other people's code.

A .tar.gz pads where Go can read it

Breaking. Sizes do not move, every size that was reachable still is, and
every reader that took these files still takes them. What moves is where the
padding sits in the gzip header, and with it the bytes.

Go's compress/gzip reads a header comment into a fixed buffer: 511 bytes it
takes, 512 it refuses. This format padded through that comment up to four
kilobytes, so:

	before	after
reachable sizes	11 260	11 260
unreachable	0	0
unreadable by Go	4134 (37%)	0

The message Go gives is gzip: invalid header, which reads like a corrupt file
rather than like a field the reader will not take - so anybody meeting it would
suspect the fixture rather than the tool that made it. Testers write tools in Go.

Why it went a month unnoticed is worth more than the fix. Every reference
tool took those files: 7-Zip, GNU tar, bsdtar, Python and node accept a comment
of ten megabytes without a word, so the oracle was green on all of them. The
channel HAD been measured, carefully, in August - and the measurement asked five
readers, none of which was the language this is written in.

Both fixes named in the observation were measured and rejected. Capping the
comment at 511 gives zero unreadable archives and takes 831 sizes out of
reach, because a tar filler entry cannot be smaller than one 512 byte block
and nothing bridges the gap under it. Trading 4134 unreadable files for 831
refusals is not a fix, and it would have read like one in a summary.

So the padding moved to the gzip extra field, measured the same day rather
than assumed: Go takes 65 531 bytes of it, to the last one, beside a comment, at
a cost of exactly two bytes more than it carries. Byte granular, where the
filler entry is aligned to 512. The comment keeps the label and stretches by at
most one byte - the one amount the extra field cannot do - so the pair reach
every size between them.

Three golden hashes moved, recorded through the remeasured list with the
measurement. There is no way back and the changelog says so: the old bytes are
the ones Go cannot read.

A zip can be locked with ZipCrypto

--set encryption=zipcrypto. It is here for what it does to a reader rather
than for what it protects, measured on our own output:

archive	what .NET's ZipFile does
ZipCrypto	opens it, reports Length=8192, hands back ff c7 04 3e where the file holds tfg - txt - ciphertext as data, and never a word that the entry was encrypted
AES-256	opens it and throws InvalidDataException

An application built on that library processes noise and calls it data. AES
fails loudly in the same library, which is the safer defect and the less
interesting one. This is the fixture PRESETS.md section 4.12 has been
describing since before there was anything to build it with.

It is not protection and is not offered as any. ZipCrypto has been broken
for decades. aes-256 is the one to use when the point is that the contents are
hard to read.

The cipher was checked against somebody else's file before it was used

It went into tools/probes/zipcrypto before the generator, pointed at an
archive 7-Zip had written, and asked to decrypt it. Check byte, plaintext CRC
and the bytes all came back right.

That order was chosen because of what the AES work had shown hours earlier: a
mutation proved the archiver guard could not see a keystream running backwards,
since an AE-2 entry signs its ciphertext rather than its contents, so a file of
noise passes every check a reader makes.

A defect got through anyway, and the probe is what diagnosed it. Every
locked entry declared method 99 - WinZip AES - while a ZipCrypto entry has to
stay stored. 7-Zip reported Data Error in encrypted file. Wrong password?,
pointing at the password, the one thing that was right. The probe decrypted the
same file perfectly, and that moved the suspicion off the cryptography and onto
the header.

TestALockedEntryDeclaresTheShapeItsSchemeRequires now asks both schemes in
both directions: ZipCrypto stores, carries the real plaintext CRC and no extra
field. AES declares 99, carries a CRC of nought and a 0x9901 field.

The cost, named

ZipCrypto puts the high byte of the plaintext CRC in its header, so the contents
have to be known before the first byte of the entry goes out - and they arrive
as a stream. Each entry is generated twice, once to be checksummed and once
to be encrypted. Twice the processor and not a byte more memory, because
buffering would break the guard that says a generator does not hold a whole
file.

Evidence

• tools/probes/targzpadding priced the padding change and is the reason the
  obvious fix was not taken. Run it before and after any change to a padding
  limit: it asks which sizes are REACHABLE and which are READABLE, and those are
  two questions.
• TestEveryArchiveThisToolWritesCanBeReadByTheStandardLibrary reads 1762
  archives across both containers with the standard library.
• 7-Zip opens every locked archive with the right password and refuses the wrong
  one and none at all, for all four schemes, and the extracted contents are
  compared against the same archive built unlocked.
• Three guards stopped carrying a hand written list of encryption methods and
  read the registry instead.
• Nine mutations across the two commits, all caught. preflight --quick: 11 of
  11, including staticcheck, lint and fidelity.