Skip to content

Latest commit

 

History

History
400 lines (316 loc) · 33.8 KB

File metadata and controls

400 lines (316 loc) · 33.8 KB

Hawkynt.FileFormats.FileSystems

NuGet NuGet downloads License CI Target

Pure-managed filesystem readers/writers and disk-image containers for modern, legacy, virtual-machine, optical, forensic, and retro-computing images without mounting them through the host OS.

📦 Installation

dotnet add package Hawkynt.FileFormats.FileSystems

The package bundles the filesystem and disk-image FileSystem.* / FileFormat.* assemblies while taking Hawkynt.Compression.Core as the shared NuGet dependency.

✨ Features

  • Inspect filesystems in-process without libguestfs, loop mounts, kernel drivers, or elevated privileges.
  • Cross-platform parsing: inspect NTFS from Linux, ext filesystems from Windows, HFS+ from either, and so on.
  • Fresh filesystem-image creation for many formats, with true modification semantics where implemented.
  • True in-place bcachefs add/replace/remove and purge for the supported single-device profile: unchanged file extents stay at the same physical sectors while allocation, freespace, backpointer and accounting metadata is committed in the reserved metadata zone.
  • Disk-image container support for VM, optical, forensic, firmware, and emulator workflows.
  • Layout/cluster/block-size optimization, defragmentation, and unused-space/slack wiping on supporting filesystems.
  • External conformance validation against real filesystem tools where the platform/test environment provides them.

🧩 Support matrix

State Meaning
R Open/read/walk only.
WORM Read plus create a fresh image; no supported mutation of an existing image.
R/W Read plus supported modification semantics. Some implementations rebuild rather than edit blocks in place.
⚠️ Deliberate structural/profile subset.

The descriptor's implemented interfaces and FormatCapabilities are authoritative for the exact state in a particular build. The tables below document the package surface without inferring capabilities from roadmap text.

Disk-image / firmware containers

Container State Scope Reference
VHD WORM Fixed, dynamic, and differencing VHD paths Microsoft VHD overview
VHDX R Hyper-V VHDX reader; current writer state follows its descriptor MS-VHDX
VMDK WORM VMware virtual disks VMware Virtual Disk API
VDI WORM VirtualBox disk images VirtualBox storage documentation
QCOW2 WORM QEMU copy-on-write images QEMU QCOW2 specification
Apple DMG WORM Apple disk-image container Apple disk images
BIN/CUE WORM Raw optical tracks + cue sheet CUE sheet background
CSO R Compressed ISO image CSO overview
Expert Witness Format R EWF/EnCase forensic images libewf documentation
UEFI Firmware Volume R Firmware volume / FFS-oriented inspection UEFI specification
Device Tree Blob R Flattened Device Tree property traversal Devicetree specification
Intel HEX / S-record R Firmware records normalized to payload + metadata Intel HEX description / S-record manual

Microsoft / DOS filesystems

Filesystem State Scope Reference
FAT12/16/32 R/W FAT variants + long filenames Microsoft FAT specification
exFAT R/W exFAT image creation/read/write Microsoft exFAT specification
NTFS R/W MFT/system metadata, supported compression and modification paths MS-FSCC
ReFS R ⚠️ Header/boot-sector oriented subset Microsoft ReFS overview
HPFS R/W Rebuild-based mutation/extent handling OS/2 Museum HPFS
DriveSpace / DoubleSpace R/W Compressed-volume-file workflows DriveSpace overview

Unix / Linux filesystems

Filesystem State Scope Reference
ext2/ext3/ext4 R/W Extended filesystem family Linux ext4 documentation
Btrfs R/W Tree/chunk-based filesystem paths Btrfs on-disk format
XFS R/W XFS v5-oriented image workflows XFS documentation
ReiserFS R/W ReiserFS 3.6; rebuild-based mutation Linux ReiserFS documentation
JFS R/W IBM/Linux JFS paths JFS project
F2FS R/W Flash-Friendly File System Linux F2FS documentation
ZFS R/W Supported OpenZFS-style structures; rebuild-based mutation where documented OpenZFS documentation
JFFS2 R/W Log-structured flash filesystem Linux JFFS2 documentation
UBIFS R Read path only Linux UBIFS documentation
bcachefs R/W ⚠️ Native b-trees; true in-place add/replace/remove + purge; in-place defrag/optimize and wipe/clean; alloc/freespace/backpointer/accounting metadata kept consistent. Mutation is limited to the supported single-device regular-extent profile. bcachefs

Apple / optical / portable filesystems

Filesystem State Scope Reference
HFS R/W Classic Macintosh HFS Inside Macintosh: Files
HFS+ R/W Catalog-tree and nested-directory support Apple TN1150
APFS R/W Supported container/filesystem tree and modification paths Apple File System Reference
ISO 9660 R/W Optical-disc filesystem images ECMA-119
UDF R/W Universal Disk Format images OSTA UDF specifications
SquashFS R/W Compressed filesystem image; mutation is rebuild-oriented SquashFS documentation
CramFS R/W Compressed ROM filesystem; mutation is rebuild-oriented Linux cramfs documentation
EROFS WORM Enhanced read-only filesystem images EROFS documentation

Retro / emulator filesystems

Family State Examples Reference
Commodore disk images R/W D64, D71, D81 and related media VICE disk image docs
Apple DOS / ProDOS R/W Apple II disk filesystems ProDOS technical reference
CP/M R/W Canonical CP/M 2.2 8-inch SSSD geometry; descriptor implements create/modify as well as read CP/M filesystem notes
TR-DOS WORM ZX Spectrum disk images TR-DOS notes
RT-11 R DEC RT-11 filesystem images RT-11 documentation archive

🚀 Quick start

Walk a filesystem image

using FileSystem.Fat;

var image = File.ReadAllBytes("disk.img");
using var fs = new FatReader(image);
foreach (var entry in fs.ListRecursive())
  Console.WriteLine($"{entry.Path} {entry.Size,10} {entry.Modified:O}");

Open a virtual disk and inspect its inner filesystem

using FileFormat.Vhd;
using FileSystem.Ntfs;

using var stream = File.OpenRead("system.vhd");
var inner = new VhdReader().OpenContents(stream);
foreach (var partition in inner.Partitions) {
  if (partition.Type != "NTFS")
    continue;
  var ntfs = new NtfsReader(partition.Open());
  foreach (var path in ntfs.WalkPaths())
    Console.WriteLine(path);
}

🧭 When to use this package

Use it for forensic/archival inspection, cross-platform disk analysis, retro-computing media, cloud/VM image inspection, firmware/test-image construction, filesystem layout experiments, recovery, and conversions where mounting through the host OS is undesirable or impossible.

It is not a replacement for System.IO when the volume is already mounted, and it is not a kernel filesystem driver with live concurrent journaling guarantees.

Creatable filesystem implementations build real nested directory trees rather than flattening paths. Implementations that expose the relevant interfaces can also wipe unused space/cluster-tip slack, defragment extents, and participate in layout/cluster-size optimization. Those are per-descriptor capabilities, not assumptions applied to every format.

📚 Complete disk-image container inventory

Descriptor Details Reference
FileFormat.Vhd Microsoft VHD v1; fixed/dynamic/differencing paths VHD
FileFormat.Vhdx VHDX reader; exact writer capability follows the current descriptor MS-VHDX
FileFormat.Vmdk VMware Virtual Machine Disk VMDK
FileFormat.Vdi Oracle VirtualBox Disk Image VirtualBox
FileFormat.Qcow2 QEMU Copy-On-Write v2 QCOW
FileFormat.Dmg Apple Disk Image Apple Disk Image
FileFormat.Cso Compressed ISO, PSP/homebrew CSO
FileFormat.BinCue CD/DVD raw optical tracks + cue sheet Cue sheet
FileFormat.Mdf Alcohol 120% Media Disc Format Alcohol 120%
FileFormat.Nrg Nero Burning ROM image Nero Burning ROM
FileFormat.Cdi DiscJuggler image DiscJuggler
FileFormat.Pfs0 Nintendo Switch PartitionFS / firmware packaging
FileFormat.UImage U-Boot uImage Das U-Boot
FileFormat.UefiFv UEFI firmware volume UEFI
FileFormat.Ipsw Apple iOS/iPadOS firmware archive IPSW
FileFormat.Ewf Expert Witness Format / EnCase forensic image EnCase
FileFormat.T64 Commodore 64 tape archive; modification is rebuild-oriented Commodore DOS
FileFormat.Tap Sinclair/Commodore tape image; modification is rebuild-oriented
FileFormat.Dtb Device Tree Blob / overlay; walks FDT properties as pseudo-archive Devicetree
FileFormat.FirmwareHex Intel HEX, Motorola S-Record and TI-TXT normalized to firmware.bin + metadata Intel HEX

📚 Complete filesystem inventory

The long-form table preserves the original implementation detail, but avoids freezing stale state letters for every row. For exact current R/W/WORM/R capability, inspect the descriptor's FormatCapabilities/implemented interfaces; where the state is material to a curated row above it is stated explicitly.

Microsoft / Windows

Descriptor Implementation detail Reference
FileSystem.Fat FAT12/FAT16/FAT32, LFN, BPB, 0x55AA signature, FATGEN-oriented writer FAT
FileSystem.ExFat exFAT VBR + boot-checksum handling exFAT
FileSystem.Ntfs NTFS MFT/system metadata, USA fixup, LZNT1 paths NTFS
FileSystem.Refs ReFS header/boot-sector subset ReFS
FileSystem.Hpfs OS/2 HPFS, rebuild-based add/remove, defrag, extent map HPFS
FileSystem.Htfs SCO HTFS: s_magic=0x012FD15D, S5-style superblock/inodes, 16-byte dirents, nested dirs, 512/1024/2048-byte blocks, defrag/purge/layout options HTFS
FileSystem.DoubleSpace DOS 6 DoubleSpace/DriveSpace CVF stored-run paths, rebuild-based modify DriveSpace

Unix / Linux

Descriptor Implementation detail Reference
FileSystem.Btrfs CRC-32C, chunk tree and SYSTEM/METADATA/DATA structures Btrfs
FileSystem.Ext ext2/ext3/ext4 DYNAMIC_REV/FILETYPE-oriented writer ext4
FileSystem.Xfs XFS v5, AGF/AGI/AGFL and B-tree structures XFS
FileSystem.Ext1 1992 ext1 magic 0xEF51; rebuild-oriented modification; no current mkfs.ext1 validator exists Extended FS
FileSystem.ReiserFs ReiserFS 3.6, multi-leaf S+tree, R5-hashed keys, nested directories; rebuild mutation ReiserFS
FileSystem.Reiser4 Empty-filesystem creation path based on reference blocks; full object-tree authoring is not implied Reiser4
FileSystem.Jfs IBM JFS, nested dirs with external dtree B+ pages, secondary AIT/AIM handling JFS
FileSystem.F2fs Superblock/checkpoint/SIT/NAT/SSA and hash-bucket directory blocks F2FS
FileSystem.Zfs fat-ZAP directories, Fletcher-4, big-endian XDR labels; rebuild-based supported mutation ZFS
FileSystem.Ufs BSD UFS reader, fs_magic=0x011954 path UFS
FileSystem.BcacheFs Native b-tree reader/writer plus true in-place CRUD for the supported single-device regular-extent profile. Unchanged file data stays at its physical sectors; add/replace allocate free buckets; remove/purge zero released extents; alloc/freespace/backpointer/accounting trees are committed in the metadata reservation. In-place defrag/optimize and wipe/clean are supported. bcachefs
FileSystem.Ubifs UBIFS log-structured read path; LPT/TNC writer complexity is intentionally not guessed UBIFS
FileSystem.Jffs2 JFFS2 log-structured paths, rebuild-oriented mutation JFFS2
FileSystem.Yaffs2 YAFFS2 rebuild-oriented mutation + defrag paths YAFFS
FileSystem.Bfs BeFS single-AG B+ tree, rebuild-oriented mutation Be File System
FileSystem.Hammer DragonFly HAMMER reader; validator requires DragonFly environment HAMMER
FileSystem.Hammer2 DragonFly HAMMER2 reader HAMMER2
FileSystem.Ocfs2 OCFS2 paths, rebuild-oriented supported mutation OCFS2
FileSystem.Nwfs Novell NetWare filesystem paths NSS
FileSystem.Efs SGI EFS: fs_magic=0x00072959, single-CG inode table, single-extent files, nested dirs, defrag/purge/layout options EFS
FileSystem.Gfs1 Sistina GFS pre-GFS2: multihost-format superblock, dinodes, lock protocol/table options GFS2
FileSystem.Jfs1 OS/2 JFS1 discriminator (JFS1, version 1), 256-byte dinodes, configurable blocks, defrag/purge/layout options JFS

Apple / classic Mac

Descriptor Implementation detail Reference
FileSystem.HfsPlus HFS+ catalog B-tree, TN1150 case-folding order, nested dirs; rebuild mutation HFS+
FileSystem.Hfs Classic HFS catalog/extents trees; rebuild mutation HFS
FileSystem.Apfs Single container/volume path with NXSB/APSB, object map and FS-tree B-tree; supported rebuild mutation APFS
FileSystem.Mfs Macintosh File System (1984), drSigWord=0xD2D7; rebuild mutation MFS

Compressed / embedded / flash

Descriptor Implementation detail Reference
FileSystem.SquashFs zlib/compressed SquashFS paths; supported mutation uses rebuild semantics SquashFS
FileSystem.CramFs CramFS 0x28CD3D45, CRC-32, zlib; rebuild mutation cramfs
FileSystem.RomFs -rom1fs- big-endian ROMFS; rebuild mutation despite the on-disk format's read-only role romfs
FileSystem.MinixFs Minix v1/v2/v3 superblock families; rebuild mutation MINIX FS
FileSystem.Erofs EROFS compact-inode + FLAT_PLAIN creation path with nested directories EROFS
FileSystem.LittleFs LittleFS metadata-pair commit log, CTZ/inline files, nested directories, commit-walking reader littlefs

Optical

Descriptor Implementation detail Reference
FileSystem.Iso ISO 9660 + Joliet, PVD/SVD, UCS-2 long names, multi-sector dirs, L/M path tables; supported mutation uses rebuild and wiping semantics ISO 9660
FileSystem.Udf ECMA-167/UDF, VRS@16-18, AVDP@256, CRC-16-XMODEM; rebuild mutation UDF
FileSystem.Sfs Amiga Smart File System root-block surface; full object-container B+ tree/bitmap/hash-table support is not inferred SFS

Retro / vintage

Descriptor Implementation detail Reference
FileSystem.D64 / D71 / D81 Commodore 1541/1571/1581 directories and rebuild modification Commodore DOS
FileSystem.CbmNibble Raw G64/NIB, writer GCR-encodes a D64-built 1541 image Commodore DOS
FileSystem.AppleDos Apple DOS 3.3, catalog at T17S15; rebuild mutation Apple DOS
FileSystem.ProDos ProDOS storage trees; rebuild mutation ProDOS
FileSystem.Atari8 Atari DOS 2 VTOC/sector model; rebuild mutation Atari DOS
FileSystem.Bbc BBC DFS/ADFS paths; rebuild mutation DFS
FileSystem.Cpm CP/M 2.2 canonical 8-inch SSSD geometry; current descriptor implements list/extract/create/modify/test and additional layout/defrag/wipe surfaces CP/M
FileSystem.CpcDsk Amstrad CPC DSK / MV - CPCEMU Disk-File
FileSystem.TrDos ZX Spectrum TR-DOS image TR-DOS
FileSystem.ZxScl Spectrum SCL, SINCLAIR magic + checksum; rebuild mutation TR-DOS
FileSystem.Adf Amiga Disk Format DOS\1, BSDsum checksums; rebuild mutation ADF
FileSystem.Msa Atari ST Magic Shadow Archive, BE magic 0x0E0F Atari ST

Mainframe / minicomputer and other historical systems

Descriptor Implementation detail Reference
FileSystem.Lif HP Logical Interchange Format, 256-byte sectors LIF
FileSystem.OpenVms OpenVMS Files-11 ODS-2/ODS-5 home-block path Files-11
FileSystem.Os9Rbf Microware OS-9 Random Block File OS-9
FileSystem.Rt11 DEC RT-11 filesystem path RT-11
FileSystem.Vdfs Gothic-engine VDFS archive/filesystem surface Gothic

🕵️ Detection/header-only and opaque-payload tier

These descriptors intentionally do not advertise creation/modification when the available evidence only supports detection, a header subset, or an opaque encrypted/distributed payload. This table preserves the reasons instead of inventing missing on-disk semantics.

Descriptor Current documented scope Why it stops there
FileSystem.Tfs Detection BBN Trans-FS has no usable public on-disk spec in the repository evidence set.
FileSystem.Mfs1 Detection Acorn MFS-1 identification is heuristic/extension-led; deeper support needs period documentation.
FileSystem.Nwfs386 Detection NetWare 386 raw-partition structures are proprietary and not guessed.
FileSystem.Stacker Detection + SCB/opaque inner payload Full upgrade needs Stacker LZS + inner FAT delegation.
FileSystem.DriveSpace3 Detection + MDBPB/opaque compressed region Full upgrade needs the actual DS compression/MDFAT structures.
FileSystem.GsOs Header/wrapper Apple IIgs 2IMG wrapper can delegate to inner ProDOS/HFS/DOS 3.3 readers.
FileSystem.TahoeLafs Detection Share payloads are capability-encrypted by design; no read-cap means no plaintext.
FileSystem.Ecryptfs Detection Payload is encrypted; decryption requires actual key/passphrase/EFEK metadata.
FileSystem.OrangeFs Detection A single PVFS/OrangeFS server object is insufficient to reconstruct a distributed filesystem without cluster config/striping state.

🧪 Filesystem validation matrix

Selected writers/readers are tested against external filesystem utilities where available. Tool absence causes the corresponding external-interop test to skip rather than convert a missing host dependency into a false product failure. The exact current test suite is authoritative; this table retains the long-form verification context.

Filesystem External validation Expected/evidenced behavior
ext4 fsck.ext4 -fnv Clean image path; reverse mkfs.ext4 reader coverage also exists
ext4 dumpe2fs -h Superblock/magic/UUID inspection
FAT12/16/32 fsck.fat -n -V, reverse mkfs.vfat Forward and reverse interoperability paths
FAT FreeDOS CHKDSK under DOSBox-X ([Explicit]) Optional historical-validator path
exFAT fsck.exfat -n Clean forward validation path
SquashFS unsquashfs -s Superblock accepted
XFS v5 xfs_repair -n -f Repair-tool validation path
Btrfs btrfs check --readonly Read-only checker path
JFS fsck.jfs -n -f -v Gated on jfsutils
NTFS ntfsfix --no-action, ntfsinfo, ntfsls, reverse mkfs.ntfs Gated on ntfs-3g
HFS+ fsck.hfsplus -d -f -n, reverse mkfs.hfsplus Gated on hfsprogs
HFS classic hmount / hls Historical notes record a malformed B-tree report against the writer; current tests/source decide present status
ZFS zdb -l Label/NVList parsing path, gated on ZFS userland tools
UFS1/FFS Linux mount when kernel supports UFS; optional FreeBSD fsck_ffs under QEMU Often unavailable on stock WSL kernels
bcachefs bcachefs show-super, bcachefs fsck -n, internal alloc/freespace/backpointer witness tests Fresh images and supported in-place CRUD/defrag/purge metadata commits are checked for b-tree/allocation consistency; external bcachefs tools remain the authority where installed.
Reiser4 fsck.reiser4 / mkfs.reiser4 Empty-FS/reference-block path, gated on reiser4progs
DoubleSpace / DriveSpace DOSBox-X + DOS utilities when legally staged Optional historical-validator path
HAMMER / HAMMER2 DragonFly BSD Linux lacks the canonical validator/mount stack
ext1 soft magic/rejection witness No mkfs.ext1 exists; internal/spec tests provide evidence instead

See docs/FILESYSTEM-VERIFICATION.md and the current Compression.Tests external-filesystem tests for the exact executable gates and assertions.

🧪 Disk-image container validation

qemu-img is used where suitable because it exercises QEMU's real disk-container parsers. The test environment may use the Windows binary or the WSL qemu-utils package.

Container Forward check Raw round-trip Reverse image → package reader
VHD qemu-img check path qemu-img convert -O raw path qemu-img create coverage
VMDK check path raw conversion path reverse-created image coverage
QCOW2 check path raw conversion path reverse-created image coverage
VDI check path raw conversion path reverse-created image coverage
VHDX Reader interoperability path Depends on current writer capability reverse-created image coverage

A forensic-style integration path builds an inner filesystem with known files, wraps it in a disk container, optionally validates the container externally, then walks it back through the package readers and compares extracted file bytes.

🧯 Filesystem-aware recovery

FilesystemCarver in Compression.Analysis scans raw images for known superblock signatures at canonical offsets, asks the matching reader to validate each candidate, and can then extract readable entries. This is useful when a partition table is lost but an inner filesystem superblock survives.

using var fs = File.OpenRead("sdcard.img");
var hits = new FilesystemCarver().CarveStream(fs);
foreach (var c in hits) {
  var result = FilesystemExtractor.ExtractCarved(
    fs,
    c,
    $"out/{c.FormatId}_0x{c.ByteOffset:X}");
  Console.WriteLine($"{c.FormatId}: {result.FilesExtracted} files, {result.FilesFailed} failed");
}

CLI examples:

cwb recover sdcard.img
cwb recover raw.img --mode filesystems --out out/
cwb recover raw.img --mode files --format Jpeg,Png

📚 Write-state model

Filesystem “write support” is deliberately split by what the implementation actually promises.

State Practical meaning
WORM Build a new valid image from files/metadata; useful for tests, firmware, reproducible images, and conversion.
R/W Existing contents can be changed through the package's supported mutation model; some implementations extract/rebuild rather than journal blocks live.
R Inspection only.

This package is an image-manipulation toolkit, not a kernel filesystem driver. R/W does not imply concurrent mount semantics, crash-consistent journaling under arbitrary interruption, or drop-in replacement for the OS driver.

🔖 Versioning

The filesystem package is built against the repository's shared Core version. Release tooling determines concrete package versions; consume mutually compatible package versions rather than relying on a prose prediction.

📚 API reference

Every public and protected member of all 583 types, generated from the built assembly and its XML documentation, is in REFERENCE.md.

🔌 Dependencies

Dependency Role
Hawkynt.Compression.Core Compression, checksums, bit I/O, partition helpers, and shared registry primitives
Host filesystem drivers / libguestfs Not required at runtime.
External fsck/repair/mkfs/qemu tools Optional validation dependencies in tests, not runtime package dependencies

⚠️ Limitations

  • Some modern filesystems are intentionally partial; a readable superblock or WORM creator is not presented as full R/W support.
  • bcachefs mutation is deliberately profile-gated: the in-place writer currently owns single-device, generation-zero, regular pointer extents as emitted by this package. Foreign volumes with extra live b-trees, reused bucket generations, inline/reflink/compressed/other extent-key forms, or unsupported inode/dirent object types are refused for mutation rather than rewritten speculatively; read support remains broader.
  • R/W can be rebuild-based rather than live in-place journaling. Block placement, journals, snapshots, reflinks, quotas, encryption and crash consistency are format-specific capabilities. bcachefs' supported R/W profile is an explicit exception here: its CRUD and layout maintenance paths are true in-place operations.
  • Disk-image container support and inner-filesystem support are separate capabilities.
  • External-validator parity varies by platform and available tooling; tests are the current evidence source.
  • Historical deep-reference prose can become stale as capabilities improve. Descriptor interfaces/FormatCapabilities, code and tests take precedence over an older state label.
  • Unknown proprietary or encrypted structures are not inferred from names or roadmap intent.

❤️ Support

If this project saves you time or money, consider supporting its development:

GitHub Sponsors PayPal

📜 License

Licensed under LGPL-3.0-or-later — see the repository LICENSE.