! nothing but us reads a compressed cabinet we write

[Hawkynt]

cabextract is installed on the build machine and nothing asks it. It refuses
every compressed cabinet we write, in three different ways:

$ cabextract -t ours-mszip.cab
  A.TXT  failed (decompression error)
$ cabextract -t ours-lzx.cab
  A.TXT  failed (error in CAB data format)
$ cabextract -t ours-quantum.cab
  A.TXT  failed (out of memory)

A stored cabinet passes both readers. 7-Zip accepts our MSZIP and refuses our
LZX and Quantum, so the MSZIP case is one where the two readers disagree about
us and only the stricter one notices.

cabextract is libmspack, which is the reference implementation for all three
of these encodings, so where it and we differ the difference is ours.

Leads, untested:

• MSZIP — each block is a raw deflate stream of its own preceded by CK,
  and the history from the previous block carries over into the next. A writer
  that emits one continuous deflate stream, or that resets the window per
  block, produces something a lenient inflater still reads and a strict one
  does not.
• LZX — the same encoding the WIM writer had wrong (! our WIM writer produces something no WIM reader will open #76): ours is a
  dialect that round-trips against itself and nothing else. CAB, CHM and WIM
  all go through LzxCompressor, so one fix covers three formats. The WIM
  half of the evidence is in the sibling issue.
• Quantum — "out of memory" is what libmspack reports for a window level
  it cannot make sense of; the level is packed into the high byte of the
  compression-type word beside the type itself.

The way to work these is the way #76 was worked: cabextract and 7-Zip
disagree with each other as well as with us, so both have to be asked, and a
cabinet the reference tool writes is available for comparison — gcab, or
cabextract on any cabinet already on disk. Compare one block at a time
rather than reasoning about the format from memory.

[Hawkynt]

The MSZIP third of this is fixed, and it was not a cabinet problem at all.

cabextract refused any MSZIP cabinet whose payload ran past about two
kilobytes, and the size threshold was the giveaway: below it our deflate emits
fixed-Huffman blocks, above it dynamic ones. The dynamic blocks declared a
distance tree holding exactly one code, of one bit:

literal/length codes=29   kraft=1     -> complete
distance       codes=1    kraft=1/2   -> INCOMPLETE
distance code lengths: [0,0,0,0,0,0,0,0,0,1]

Half the code space describing nothing. zlib accepts that — the single-code
case is written into its inflater as a special case — and so does 7-Zip, which
is why every cabinet, zip, gzip and PNG we have ever written looked fine. A
decoder that adds the fractions up rejects the block, and libmspack does.

It happens whenever every back-reference in a block falls in the same distance
range, which for a cabinet is most of the time.

Fixed by giving a second symbol a one-bit code whenever a tree would otherwise
hold one, which is what every other encoder does; it costs one code length in
the header and nothing in the block. The fix is in the one place all three of a
dynamic block's trees are built, so the literal/length and code-length trees
are covered too.

Guarded by DeflateTreesAreCompleteTests, which adds the fractions up itself
rather than asking whichever decoder happens to be installed, and
cabextract is now in OurArchivesTheirReadersTests beside 7-Zip.

Still open here: LZX and Quantum cabinets, which fail both readers.

[Hawkynt]

The LZX codec is now correct — four faults fixed in 5c592e9, verified against
both WIM readers across 32 cases. Cabinets still fail, and it is now clear that
what remains is cabinet framing rather than the encoding:

• A cabinet's LZX stream opens with a translation header — one bit, and a
  32-bit size after it if set. WIM has no such header, and LzxCompressor
  writes the WIM form, so a cabinet reader is a bit out of step from the first
  symbol onwards.
• Block sizes are 24 bits in a cabinet, where WIM writes a one-bit
  "the usual 32768" flag and 16 bits otherwise.
• A folder is one continuous stream, not one stream per CFDATA. Our writer
  calls the compressor once per 32 KB block, so each block restarts the window,
  the remembered offsets and the delta-coded trees. A cabinet reader decodes the
  folder as a single stream and will not find a new header there.

So LzxCompressor needs a cabinet variant of its stream and block headers, and
CabWriter needs to compress a folder once and cut CFDATA records from the
result — which means the compressor must report where each 32 KB of output ends
in the compressed bytes.

Quantum is untouched and still reports "out of memory" from cabextract, which is
what libmspack says about a window level it cannot make sense of.

MSZIP is fixed and both readers accept it; cabextract is now wired into
OurArchivesTheirReadersTests beside 7-Zip.

[Hawkynt]

LZX cabinets are fixed in 5609e8a. Quantum is all that remains here.

The last three faults, after the codec ones in #86:

• Pre-tree codes 17 and 18 mean "this many lengths are zero", not "this many
  are unchanged". We used them for runs of zero delta. The two coincide only
  while the previous block's lengths are all zero — the first block of a stream
  and no other — so every second block came back with its trees wiped. Our
  decoder read them the same wrong way, which is why nothing here noticed. WIM
  never showed it: a chunk is 32 KB and never holds a second block.
• A cabinet frames the stream differently: a translation bit at the front,
  and 24-bit block sizes rather than a one-bit "the usual 32768" flag.
• A folder is one stream cut into records, not one stream per record. Every
  record but the last accounts for exactly 32 768 bytes, and each begins on a
  word boundary of the bit stream.

LzxCompressor and LzxDecompressor now take an LzxStreamFormat, CabWriter
compresses a folder once and cuts records from the result, CabReader decodes a
folder as the one stream it is, and no match may straddle a record boundary.

15 of 15 LZX cabinets accepted by both cabextract and 7-Zip across five payload
shapes and four sizes, one shape dense in x86 call opcodes. CabReferenceReaderTests
now exercises store, MSZIP and LZX against both readers, with payloads sized to
span several records.

Quantum: cabextract reports "out of memory", which is libmspack's answer to a
window level it cannot make sense of. Nobody has looked at it yet.

[Hawkynt]

Quantum, now that it is the only one left here, in two parts.

The window field is in the wrong units. We write the folder's compression
word as (level << 8) | 2 with a level of 1–7, our own QuantumConstants
scale. The high byte is the window bits, and libmspack's qtmd_init refuses
anything outside 10–21 by returning nothing, which cabextract reports as "out of
memory" — the message that made this look like a resource problem rather than a
field one. Patching the byte of a cabinet we wrote from 4 to any of 10, 15 or 21
gets past it:

window bits 4  ->  A.TXT  failed (out of memory)
window bits 10 ->  A.TXT  failed (error in CAB data format)
window bits 15 ->  A.TXT  failed (error in CAB data format)
window bits 21 ->  A.TXT  failed (error in CAB data format)

The stream is wrong too. "Error in CAB data format" is what libmspack says
once it is decoding in earnest, so the arithmetic coder and its models do not
agree with the reference either. That is the larger half of the work and nobody
has looked at it.

Both halves belong together: correcting the field alone changes the error
message and nothing else, and would break our own reader, which maps that byte
back onto the 1–7 scale.

The harness is ready for whoever takes it: cabextract -t gives a per-file
verdict, and CabReferenceReaderTests gains Quantum by adding one entry to its
list of compression types once a cabinet passes.