From 6938b25ac3ff78b65637375494b23b4efd0b720b Mon Sep 17 00:00:00 2001 From: Hawkynt Date: Fri, 28 Aug 2026 20:37:44 +0200 Subject: [PATCH] * regenerate the package API references A pipe inside a signature or a summary split the table row it sat in; a NumPy dtype default of "|u1" in Hawkynt.FileFormats.FileSystems is the visible case. The generator escapes pipes in the cell renderer now, and this is its output. Nothing changed but the escaping. --- CHANGELOG.md | 1106 +++++++++++++++++++++ Hawkynt.FileFormats.Archives/README.md | 12 +- Hawkynt.FileFormats.FileSystems/README.md | 14 +- 3 files changed, 1119 insertions(+), 13 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 000000000..feb66be3a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,1106 @@ +# Changelog + +## Nightly 2026-07-08 + +### Added +- aPLib decompression building block `BB_Aplib` (`Compression.Core/Dictionary/Aplib`): + bit-exact clean-room `aP_depack` decoder (MSB-first tag stream; `0` literal / + `10` normal match with γ-offset + inlined low byte and offset-dependent length + bumps / `110` short 7-bit-offset match + end-of-stream marker / `111` 4-bit + single byte + literal-zero) plus a spec-valid greedy encoder for round-tripping. + This is the reference-compatible aPLib — distinct from the older non-standard + `FileFormat.ApLib`/`ApLibStream`, which reads LSB-first and only round-trips + against itself. Tests cover self-round-trip across all input shapes and + hand-traced reference streams for every decoder-only grammar path. +- Real aPLib PE-packer unpack handlers wired into `ExecutablePackerHandlers`: + `fsg`, `aspack`, `pecompact`, `rlpack`, sharing an `AplibSectionPackerHandler` + base that carves each PE section, aPLib-decodes it, and accepts only a + cleanly-terminated expanding stream (rejecting magic-less false positives). + Each emits `decompressed_payload.bin` + diagnostics; runnable rebuild is + reported as loader-version-specific rather than guaranteed (UPX-handler bar). +- Generic `aplib_pe` fallback handler (registered last, low confidence): detects + aPLib-compressed PEs by decode — any section that inflates to a cleanly-terminated + aPLib stream — so aPLib-family packers we don't name individually (JDPack, + Packman, Neolite in aPLib mode, or aPLib output from unknown tools) still unpack. +- Executable-packer coverage map for the packing-box dataset (25 packers): + `docs/EXE-PACKER-COVERAGE.md` — five now at Unpack level (UPX + the four aPLib + packers), the rest Detect/Planned with their compression core noted. + +## Nightly 2026-06-16 + +### Added +- Large-file streaming write (CreateFromStreams) for 11 FS targets (e974533) +- Conversion-matrix KnownConversionGaps quarantine + bucketed gap doc (c5b8cde) +- Old "SZ " (QBasic/pre-SZDD) MS COMPRESS variant: read + write + creatable + SzCompress descriptor: magic 53 5A 20 88 F0 27 33 D1, 12-byte header (8-byte magic + u32 LE uncompressed length, LZSS stream at offset 12); same 4096-byte ring LZSS body as SZDD, which neither our old reader nor 7-Zip groks * SzddStream.Decompress auto-detects either variant (12- vs 14-byte header); + CompressQBasic emits the "SZ " header so the old format round-trips + tests: QBasic round-trip (incl. matches), empty, and the literal path a first 0xFF control byte yields (8 verbatim literals → an MZ EXE header) (2c2a04b) +- Conversion capability + round-trip matrix harness (586ceff) +- Streaming-write CreateFromStreams override for Tar/Cpio/Ar/Zip-store (a143719) +- Reusable cost-based optimal LZ77 parser primitive + BB_Lz77Optimal (6d647e5) +- Lz4 block-size/checksum + Xz/Lzip level+dict option schemas (ae245cb) +- Shared layout-optimizer adapter + wire FatX/ext/JFS1/NILFS2 (d5ac363) +- Neural building block: online-trained MLP predictor + arithmetic coder (4720b13) +- Benchmark-driven cross-block auto-selector + optimizer budgets/caching (37c5564) +- TD0/IMD/HFE/WOZ floppy-emulator images + QOI/DPX image formats (8535137) +- WSL cross-validation for archive writers + FS content checks (76f913a) +- Image-format external-tool interop tests (WSL ImageMagick/exiftool/webp/avif) (66b0df5) +- Audio external-tool interop tests (ffprobe/flac/wavpack/opus) (b8cf07c) +- Codec.SysV: images now mount on the real Linux kernel (mount -t sysv) and the kernel reads our files byte-exact — validated under the Alpine qemu guest (was rejected before) # SysVWriter/Reader/Modifier: place the superblock at block 0 + BLOCK_SIZE/2 (offset 512), not block 1 (offset 1024). The Linux sysv driver's detect_sysv reads the superblock struct at bh->b_data + BLOCK_SIZE/2, so s_magic must land at byte 1016 (512+504); at offset 1024 the kernel never found it # SysVWriter: s_isize is the FIRST DATA ZONE (FirstInodeBlock + ilist blocks), not the ilist size. The kernel reads it as s_firstdatazone and rejects the mount when s_firstdatazone < s_firstinodezone (=2); our old value of 1 always failed. Reader/Modifier reinterpret s_isize accordingly (ilist size = s_isize - FirstInodeBlock) * SysVFormatDescriptor: magic signature offset 1528 -> 1016 to match the relocated superblock; SysV tests updated to the new offsets (7f5ddb9) +- Codec.SquashFs: images now mount on the real Linux kernel (mount -t squashfs) and the kernel reads our files byte-exact — validated under the Alpine qemu guest # SquashFsWriter: pad the image to a 4 KiB device boundary (as mksquashfs does). An unpadded sub-sector image (e.g. 234 bytes for two tiny files) presents a zero-sector block device, so the kernel's first superblock read failed with EIO ("SQUASHFS error: Failed to read block 0x0: -5"). bytes_used in the superblock stays the true unpadded size (afcdd3e) +- Codec.Udf: images now mount on the real Linux kernel (mount -t udf) and the kernel reads our files byte-exact — validated under the Alpine qemu guest (was rejected at every layer before) # UdfWriter LVD: emit the mandatory "*OSTA UDF Compliant" domain EntityID + OSTA CS0 character set, a Type-1 partition map (map_table_length=6, number_of_partition_maps=1) and an implementation EntityID; without the domain id the kernel reports "Not OSTA UDF compliant logical volume descriptor" and without the map "No partition found" # UdfWriter partition descriptor: set partition_flags=allocated, partition_number=0, "+NSR02" contents EntityID and access_type so the partition map resolves # UdfWriter FSD: emit the OSTA domain EntityID + CS0 charsets and interchange levels # UdfWriter File Entry: set ICB strategy type 4 (0 → "unsupported strategy type") and move file_link_count to the ECMA-167 offset 48 (was 28, inside the ICB tag — the kernel saw link count 0 and failed udf_iget on the root directory) (cdee5f9) +- Test harness: run the external filesystem conformance + post-mutation suites natively on a Linux/POSIX host, not only via WSL on Windows. FsInteropToolbox (DetectWsl/RunWsl/WinToWsl) and the per-fs WSL helpers (F2fs/Jfs/ReiserFs) now execute the Linux fsck/mkfs tools directly through /bin/bash when the host is not Windows; Windows behaviour is unchanged (gated on OperatingSystem.IsWindows) * This lights up 104 conformance checks + 14 add/remove/modify post-mutation r/w checks on Linux that previously skipped — validating ext/xfs/btrfs/f2fs/jfs/reiserfs/minix/ntfs/iso/squashfs/cramfs/bfs writers against the real host tools, and confirming F2fs/Jfs/ReiserFs stay fsck-clean after mutation # WinToWsl returns the bare POSIX path on a non-Windows host (a single-quoted form broke the direct-exec qemu-img container tests) (1034a73) +- Codec.Hammer2 WRITE now produces files the DragonFly kernel reads byte-exact — full read/write. Validated via mount_hammer2 @