Skip to content

Promote filesystem R/W gaps and add mount-grade driver architecture - #104

Open
Hawkynt wants to merge 189 commits into
mainfrom
feat/filesystem-rw-gaps
Open

Promote filesystem R/W gaps and add mount-grade driver architecture#104
Hawkynt wants to merge 189 commits into
mainfrom
feat/filesystem-rw-gaps

Conversation

@Hawkynt

@Hawkynt Hawkynt commented Aug 28, 2026

Copy link
Copy Markdown
Owner

What changed

This PR now covers two related but deliberately separate surfaces:

  1. offline image/archive modificationFormatCapabilities.CanModify means an existing image can be edited and verified; the implementation may patch in place, relayout, copy-on-write, or rebuild where that is the correct image-editor strategy;
  2. mount-grade filesystem drivers — a stronger IFilesystemDriverProvider / IFilesystemSession contract with stable object identities, positional handles, block devices, explicit durability models, transactions and fail-closed per-image profiles.

Archive-level R/W does not imply FilesystemDriverProfile.CanMountWritable.

Offline R/W promotions

DMG / UDIF

  • promote the supported raw UDIF profile to R/W
  • preserve exact logical partition lengths instead of mandatory 512-byte padding
  • append replacement payloads and regenerate the affected blkx / plist / koly metadata
  • remove partitions without relocating unrelated payloads
  • reject foreign compressed/signed plist profiles rather than flattening unknown semantics

UEFI Firmware Volume

  • create standalone PI FV/FFS2 images with FV and FFS checksums
  • reserve erased capacity for later edits
  • add/replace/remove FFS files through aligned erased ranges without changing FV size
  • make the reader walk erased holes left by deletion

OrangeFS / PVFS2 DBPF

  • create standalone DBPF storage objects
  • replace/clear object.bin while preserving DBPF metadata
  • expose the proven single-object storage profile as R/W

Flattened Device Tree

  • hierarchy-preserving FDT v17 writer
  • preserve memory reservations and boot_cpuid_phys
  • add/replace/remove properties by node path
  • preserve string-list NUL termination correctly
  • rebuild/relayout is accepted as an image-editor mutation strategy

ReFS

  • expose a deliberately scoped offline-quiescent R/W profile
  • replace existing regular files using allocator-verified new extents, metadata repointing and old-extent release
  • remove regular files / empty directories through CoW B+ rewrites and alternate-checkpoint publication
  • fail closed for sparse/integrity/shared layouts and for unproven new-name object creation
  • keep native mounted-driver transaction/MLog semantics separate from this offline editor path

SquashFS / CramFS / EROFS

  • normalize read-only-on-mount vs. image-editor semantics: a filesystem can be immutable when mounted by an OS yet still support safe offline image rebuilds
  • SquashFS/CramFS: rebuild-backed R/W
  • EROFS: rebuild-backed R/W for the supported uncompressed/inline profile; unsupported compressed content fails closed instead of being replaced by extraction placeholders

EWF

  • expose the acquired medium as semantic media.raw
  • reconstruct chunked stored/compressed data with checksum validation
  • rebuild the EWF image when the semantic medium is replaced/removed
  • keep diagnostic sections distinct from independently mutable filesystem entries

Other stale capability fixes

  • promote MSA and PFS0 where existing create/modify implementations already provided real existing-instance edit semantics

Mount-grade filesystem driver architecture

Common contracts

  • FilesystemNodeId: path-independent object identity
  • IFilesystemSession: lookup/enumerate/stat plus namespace mutation surface
  • IFilesystemFileHandle: positional I/O; no shared mutable Stream.Position
  • IRandomAccessBlockDevice: container-independent logical block I/O
  • IRawTrackDevice: variable-length GCR/MFM/flux-style media layer
  • explicit FilesystemMutationModel
  • explicit read-only/read-write readiness layers and blocker reports

Repository-wide coverage invariant

  • Compression.Lib and Hawkynt.FileFormats.FileSystems now reference all FileSystems/FileSystem.*/*.csproj projects through an exhaustive project glob
  • the source generator marks descriptors from FileSystem.* namespaces and discovers public native sidecars without reflection
  • registry initialization fails if a filesystem descriptor has neither a native provider/sidecar nor a conservative archive-derived read-only path
  • FilesystemDriverCoverageTests makes this a CI invariant for current and future filesystem projects
  • GetFilesystemDriverCoverage() distinguishes native descriptor providers, generated native sidecars and archive projections

Large-file-safe fallback

  • the generic archive-derived session is a compatibility floor, not a native-readiness claim
  • default OpenEntry no longer materializes the whole file into one byte[]; it uses a bounded delete-on-close temporary-file stream when no native entry stream exists
  • SpoolingReadOnlyFileHandle gives readers with correct streaming extraction positional handles without an array-size ceiling

Native filesystem sessions added

  • D64 / CBM nibble media — Commodore filesystem namespace plus sector/raw-track layers; G64/NIB preserve raw-track identities and the GCR sector projector bridges to block semantics
  • ReFS — native decoded metadata/object read session; mounted writes remain gated independently from offline mutation
  • FAT12/16/32 — native directory/cluster-chain read session and allocation map
  • ext2/ext3/ext4 — native inode identity and positional block/extent reads
  • NTFS — MFT-record identity and positional resident/non-resident file reads
  • XFS — inode+generation identities and streaming extent reads
  • APFS — object-ID session with direct positional reads for the proven unencrypted single-extent profile; broader layouts fail closed
  • Btrfs — native inode IDs plus a global logical segment map across all FS-tree leaves; direct inline/physical reads, holes/prealloc as zeroes, compressed/encrypted/multi-stripe profiles fail closed
  • ZFS — native dataset dnode IDs and Fletcher-4-verified streaming reads for the supported v28 single-vdev profile; unsupported compression/checksum/hole/gang/multi-vdev profiles fail closed

Why mounted writes are still gated

Existing archive Add/Remove implementations are intentionally not reused as mounted write transactions. CanMountWritable stays false until each native driver can prove its complete durability model.

Examples of remaining write blockers:

  • FAT: ordered allocation/directory/size publication and crash-safe rollback semantics
  • ext: allocator + inode/dir mutation integrated with ext3/ext4 journal ordering/replay
  • NTFS: full file references, directory indexes, run-list writes, $LogFile publication/replay and metadata/security semantics
  • XFS: allocation-group btrees, log transactions, delayed allocation/refcount/reflink and recovery
  • APFS: spaceman/OMAP/tree CoW, checkpoint publication, snapshots/clones/encryption semantics
  • Btrfs: CoW tree paths, delayed refs, extent/checksum/free-space trees, transaction generations, log-tree replay, shared extents/RAID/compression
  • ZFS: metaslabs/space maps/deferred frees, complete ZPL/ZAP/dnode mutation, TXG/uberblock publication, ZIL commit/replay, snapshots/clones and multi-vdev semantics
  • ReFS: full native mounted CoW/MLog/checkpoint concurrency and recovery semantics beyond the offline-quiescent editor profile

Tests

In addition to the existing format round-trips, this PR adds driver-level coverage for:

  • repository-wide filesystem driver-path completeness
  • D64/CBM raw-track/sector/native filesystem paths
  • FAT/ext native sessions
  • NTFS native stable identity + positional reads
  • XFS native stable identity + positional reads
  • APFS native object identity + direct positional reads
  • Btrfs native inode identity + regular and inline positional reads
  • ZFS native dnode identity + positional reads
  • R/W promotion round-trips for CramFS, SquashFS, EROFS, MSA, PFS0 and EWF
  • scoped ReFS fail-closed behavior and offline mutation helpers

docs/FILESYSTEM-DRIVER-ARCHITECTURE.md documents the layer boundaries, coverage invariant, current native tier and the requirements that must be met before any driver advertises mounted write support.

@Hawkynt Hawkynt changed the title Add real R/W paths for DMG and UEFI firmware volumes Promote mutable filesystem containers from RO/WORM to R/W Aug 28, 2026
@Hawkynt Hawkynt changed the title Promote mutable filesystem containers from RO/WORM to R/W Promote mutable containers and expose maintenance capabilities Aug 28, 2026
@Hawkynt Hawkynt changed the title Promote mutable containers and expose maintenance capabilities Promote filesystem R/W gaps and add mount-grade driver architecture Aug 29, 2026
@Hawkynt
Hawkynt force-pushed the feat/filesystem-rw-gaps branch 3 times, most recently from ef10fd4 to 3112e33 Compare August 29, 2026 13:33
@Hawkynt
Hawkynt force-pushed the feat/filesystem-rw-gaps branch from 3112e33 to 214ae8e Compare August 29, 2026 13:43
Hawkynt and others added 19 commits August 29, 2026 15:51
…t see

optimizer-wire.py inserts a generic multi-entry optimizer into
ArchiveOperations.Optimize naming FormatRegistry, IArchiveCreatable,
IFormatOptionsSchema and FormatOptionKind unqualified. The file's only using is
the F alias for FormatDetector.Format, and every one of its existing references
to that namespace is written out in full, so the patched file did not compile
and the prepare job failed at "Build tests" before any of the maintenance tests
could run.

The generated block now qualifies those four the way the rest of the file does.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant