diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ef83854..92efb2b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -63,6 +63,16 @@ jobs: fi done + # littlefs has no packaged fsck; the reference C implementation is + # reached through its Python binding instead (tests/littlefs_external.rs + # skips itself when the module is missing). + - name: Install littlefs reference implementation + run: | + python3 -m pip install --user --break-system-packages littlefs-python \ + || python3 -m pip install --user littlefs-python \ + || echo "littlefs-python unavailable (littlefs cross-checks will skip)" + python3 -c "import littlefs; print('littlefs-python ok')" || true + - uses: dtolnay/rust-toolchain@stable with: components: rustfmt, clippy diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ff446f..118312b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,12 +7,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.4.22](https://github.com/KarpelesLab/fstool/compare/v0.4.21...v0.4.22) - 2026-08-17 + ### Added +- *(littlefs)* read, write and in-place edits for lfs2 images +- *(web)* create, edit and download images in the browser +- *(memedit)* in-memory authoring — blank filesystems, partitioned disks + +### Fixed + +- *(ext)* honour journal revoke records across transactions +- *(ext)* decode HTree directory roots and honour journal revoke blocks ([#32](https://github.com/KarpelesLab/fstool/pull/32)) + +### Other + +- *(littlefs)* redraw the tag and skip-list diagrams in our own form + +### Added + +- *(littlefs)* new backend for the embedded-flash filesystem (`lfs2`, disk + versions 2.0 and 2.1): read, write, and in-place edits. Metadata pairs are + replayed from their CRC-committed logs (tags, splices, tails, global-state + deltas, lfs2.1 forward-CRCs) and written back as compactions; files live + inline in metadata or in CTZ skip-lists, which are rebuilt only from the + first changed block onwards so a partial write leaves earlier blocks + untouched. Block allocation reconstructs the in-use map by traversing the + volume, as littlefs itself does. Wired into `create -t littlefs`, the TOML + spec (`type = "littlefs"`, with `block_size` / `block_count` / `prog_size` / + `version` / `name_max` / `inline_max` options), `repack --fs-type littlefs`, + `info`, `add`/`rm`, the in-memory authoring surface, and the browser build. + littlefs user attributes surface as `user.littlefs.` extended + attributes; the format has no symlinks, device nodes or POSIX metadata, so + those are refused rather than faked. Cross-validated in both directions + against the reference C implementation via `littlefs-python`, including + handing an image back and forth mid-edit. + - *(memedit)* new in-memory authoring surface: `Workspace` formats a blank filesystem or lays out a partitioned disk (MBR/GPT), takes files and directories, and hands back the image bytes at any point. - `creatable_filesystems()` advertises 14 types with their real minimum sizes. + `creatable_filesystems()` advertises 15 types with their real minimum sizes. - *(wasm)* `Workspace` and `creatable_filesystems()` bindings, so the browser build can author images as well as read them. - *(web)* "Create a new image" mode: pick a filesystem and size, or build a diff --git a/Cargo.toml b/Cargo.toml index b03c6ad..007ca6e 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "fstool" -version = "0.4.21" +version = "0.4.22" edition = "2024" # Floor is the edition-2024 minimum (1.85) bumped to 1.88 by the # `purecrypto` dependency (encrypted-DMG crypto), whose MSRV is 1.88 as of diff --git a/README.md b/README.md index d0287a4..603a82c 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ fstool repack base.tar patch.tar flat.tar # OCI-style layer merge with .w fstool also ships as a static, client-side web app. It builds images as well as reads them: format a blank filesystem (ext2/3/4, FAT12/16/32, exFAT, NTFS, -XFS, HFS+, HFS, AFFS, F2FS, GRF), or lay out an MBR/GPT disk with a +XFS, HFS+, HFS, AFFS, F2FS, littlefs, GRF), or lay out an MBR/GPT disk with a filesystem per partition, add files, and download the image — then keep editing and download again. Or upload any archive or disk image, browse what's inside, extract individual files, and convert the whole @@ -74,6 +74,7 @@ See [web/README.md](web/README.md) for the build and local-dev steps. | APFS | ✅ | ✅ | 🚧 | **Read**: multi-level omap + fs-tree, directory listings + file extents, embedded xattrs, snapshots (read-only, single-leaf snap-meta). **Write**: format + `create_dir`/`create_file`/`create_symlink` + `chmod`/`chown`/`set_times`/`rename`/`unlink`/`link` via fresh COW checkpoints (spaceman with IP ring + SFQ free-queues), round-tripped through a real macOS mount. **Gaps**: in-place edits are whole-file overwrite (no partial-extent COW); `UF_COMPRESSED`/decmpfs files read as empty; encryption, sealed-volume integrity, Fusion tiering, and dstream-backed xattrs are refused; not yet `fsck_apfs`-clean | | NTFS | ✅ | ✅ | ✅ | MFT, attributes, $DATA + ADS, indexes; xattr map; multi-class `$Secure` ($SDS/$SDH/$SII); real `$LogFile` LFS records (Path A) | | F2FS | ✅ | ✅ | — | CP / NAT / dnodes / inline data + dentries; writer passes `fsck.f2fs`; **build-once** — the writer serializes the whole FS from memory at flush, so a re-opened image is read-only (reports `Immutable`) | +| littlefs | ✅ | ✅ | ✅ | The embedded-flash filesystem (`lfs2`, disk versions 2.0 + 2.1): metadata pairs with CRC-committed logs, CTZ skip-list files, inline small files, user attributes (surfaced as `user.littlefs.` xattrs). Every mutation is a real littlefs commit, so `create` / `repack` / `add` / `rm` / `open_file_rw` all write images the reference C implementation mounts and keeps writing to — cross-validated both directions against `littlefs-python` (upstream `lfs.c`), including block-for-block agreement on which blocks are live. No symlinks, device nodes or POSIX metadata: the format has none | | SquashFS | ✅ | ✅ | — | gzip / xz / lz4 / zstd / lzo / lzma via Cargo features; writer round-trips via `unsquashfs`; repack-only | | ISO 9660 | ✅ | ✅ | — | PVD + Joliet (UCS-2) + Rock Ridge (PX/NM/SL/TF) + El Torito boot catalog; repack-only | | GRF | ✅ | ✅ | ✅ | Gravity Ragnarok Online archive — v0x102 / v0x103 / v0x200; permutation cipher (`MIXCRYPT` / `DES`); CP949 filenames | @@ -90,7 +91,7 @@ See [web/README.md](web/README.md) for the build and local-dev steps. `🚧` marks writers / mutation paths with known gaps (see Limitations). All writable filesystems — ext2/3/4, FAT12/16/32, exFAT, XFS, HFS+, NTFS, -APFS, F2FS, SquashFS, ISO 9660, GRF — implement a single +APFS, F2FS, littlefs, SquashFS, ISO 9660, GRF — implement a single `Filesystem` trait, so the CLI (`build`, `repack`, `add`, `rm`) and the TOML `[filesystem] type = "…"` spec dispatch through one codepath; pick a target FS by setting `--fs-type` on `repack` or @@ -123,7 +124,7 @@ through xattrs under `user.ntfs.*` and `system.ntfs_security`. | Command | What it does | |---------------|-------------------------------------------------------------------------| -| `create` | Build a bare image of any supported FS (`-t ext4` / `fat12` / `fat16` / `fat32` / `xfs` / `hfs+` / `ntfs` / `f2fs` / `squashfs` / `iso` / `apfs` / `exfat` / `grf` / `zip` / `cpio` / `ar`) from a host directory tree. FS-specific knobs go through `-O key=val,key=val`. | +| `create` | Build a bare image of any supported FS (`-t ext4` / `fat12` / `fat16` / `fat32` / `xfs` / `hfs+` / `ntfs` / `f2fs` / `littlefs` / `squashfs` / `iso` / `apfs` / `exfat` / `grf` / `zip` / `cpio` / `ar`) from a host directory tree. FS-specific knobs go through `-O key=val,key=val`. | | `build` | Build from a TOML spec — bare FS or a partitioned disk image. | | `info` | Print partition table (whole-disk) or FS summary + root listing. | | `ls` | List a directory inside an image; `-R` walks subdirectories recursively. | diff --git a/examples/README.md b/examples/README.md index 77ade63..89a8fd5 100644 --- a/examples/README.md +++ b/examples/README.md @@ -15,6 +15,7 @@ point `source` at your own tree to populate the filesystems. |------|--------| | [`bare-ext4.toml`](bare-ext4.toml) | A single ext4 filesystem, no partition table (the `genext2fs` replacement). | | [`bare-fat32.toml`](bare-fat32.toml) | A single FAT32 filesystem, no partition table. | +| [`bare-littlefs.toml`](bare-littlefs.toml) | A single littlefs image for embedded flash (block / program size tuned to the target part). | | [`raspberry-pi.toml`](raspberry-pi.toml) | Raspberry Pi SD card — MBR with a FAT32 boot partition + ext4 root. | | [`efi-disk.toml`](efi-disk.toml) | UEFI-bootable GPT disk (also a UEFI USB image) — ESP (FAT32) + ext4 root. | | [`bios-legacy-disk.toml`](bios-legacy-disk.toml) | Legacy-BIOS MBR disk — ext2 `/boot` + ext4 root. | diff --git a/examples/bare-littlefs.toml b/examples/bare-littlefs.toml new file mode 100644 index 0000000..89b5c69 --- /dev/null +++ b/examples/bare-littlefs.toml @@ -0,0 +1,42 @@ +# Example fstool spec: a bare littlefs image for embedded flash. +# +# fstool build examples/bare-littlefs.toml -o /tmp/lfs.img +# fstool info /tmp/lfs.img +# +# littlefs is the fail-safe filesystem used on microcontroller NOR/NAND +# flash. Sizing is exact: the image is whatever the metadata pairs and the +# file data need, so match `size` to the flash region you will write it to. + +[filesystem] +type = "littlefs" + +# Total image size. littlefs derives its block count from this and the +# block size below, so keep it a whole multiple of `block_size`. +size = "1MiB" + +# Host directory whose contents become the image's tree. Optional — +# omit it for an empty volume with just the root directory. +# source = "./rootfs" + +# Erase-block size of the target flash. Must be a power of two, at least +# 128 bytes. 4 KiB matches most NOR parts; some use 64 KiB. +block_size = 4096 + +[filesystem.options] +# Program (page) size the target flash writes in. Commits are padded to +# it so a device mounting this image can append in place. +prog_size = 256 + +# On-disk version: "2.1" (default, current) or "2.0" for firmware built +# against a littlefs older than v2.5, whose reader predates forward-CRC +# tags. +version = "2.1" + +# Files at or below this many bytes are stored inside their directory's +# metadata instead of getting blocks of their own. Defaults to an eighth +# of a block. +# inline_max = 512 + +# Longest filename the volume accepts (must match what the firmware's +# LFS_NAME_MAX allows). Default 255. +# name_max = 255 diff --git a/src/analyze.rs b/src/analyze.rs index af04aee..24d6060 100644 --- a/src/analyze.rs +++ b/src/analyze.rs @@ -344,6 +344,19 @@ pub fn plan_size(source: &Source, plan: &mut dyn FsSizePlan) -> Result { Ok(plan.total_size()) } +/// Content-fit sizing for an already-open filesystem — the [`plan_size`] +/// counterpart `repack --shrink` uses, where the source is mounted rather +/// than a path. Metadata only: file bodies are never read. +pub fn plan_size_fs( + fs: &mut AnyFs, + dev: &mut dyn BlockDevice, + plan: &mut dyn FsSizePlan, +) -> Result { + let mut sink = SizingSink { plan }; + walk_anyfs(fs, dev, &mut sink)?; + Ok(plan.total_size()) +} + #[cfg(test)] mod tests { use super::*; diff --git a/src/bin/fstool/main.rs b/src/bin/fstool/main.rs index 7f3f9b0..b238614 100644 --- a/src/bin/fstool/main.rs +++ b/src/bin/fstool/main.rs @@ -1063,6 +1063,15 @@ fn repack_cmd( // Tar output streams to a file; no pre-sized device, so // the destination size is unused. "tar" => 0, + "littlefs" | "lfs" => { + // littlefs reserves nothing up front, so its own size + // plan gives an exact content fit. + let opts = fstool::fs::littlefs::LittleFsFormatOpts::default(); + let mut plan = + ::size_plan(&opts) + .expect("littlefs has a size plan"); + fstool::analyze::plan_size_fs(&mut src_fs, src_dev, plan.as_mut())? + } "iso" | "iso9660" => { // ISO writer needs ~32 MiB headroom for a small tree. // Real sizing happens during flush; we just want enough @@ -1114,6 +1123,12 @@ fn repack_cmd( let bytes = sum_source_file_bytes(src_dev, &mut src_fs).unwrap_or(0); bytes.saturating_mul(2).saturating_add(16 * 1024 * 1024) } + "littlefs" | "lfs" => { + // File data rounded up to whole blocks, plus room for + // metadata pairs and later edits. + let bytes = sum_source_file_bytes(src_dev, &mut src_fs).unwrap_or(0); + bytes.saturating_mul(2).saturating_add(1024 * 1024) + } _ => src_total, }, }; @@ -1241,6 +1256,16 @@ fn repack_cmd( src_dev, false, )?, + // littlefs has neither symlinks nor device nodes, so the sink + // runs lossy: those entries are skipped instead of failing the + // whole repack. + "littlefs" | "lfs" => repack_via_trait::( + dst_dev.as_mut(), + &fstool::fs::littlefs::LittleFsFormatOpts::default(), + &mut src_fs, + src_dev, + true, + )?, "squashfs" => repack_via_trait::( dst_dev.as_mut(), &fstool::fs::squashfs::FormatOpts::default(), @@ -1602,6 +1627,13 @@ fn repack_layered_to_dst( &layers, false, )?, + "littlefs" | "lfs" => repack_layered_via_trait::( + dst_dev.as_mut(), + &fstool::fs::littlefs::LittleFsFormatOpts::default(), + &model, + &layers, + true, + )?, "iso" | "iso9660" => { let opts = fstool::fs::iso9660::FormatOpts { volume_id: "FSTOOL".into(), @@ -1845,6 +1877,14 @@ fn repack_tar_stream_to_fs( codec, false, )?, + // Lossy: littlefs stores neither symlinks nor device nodes. + "littlefs" | "lfs" => repack_stream_via_trait::( + dst_dev.as_mut(), + &fstool::fs::littlefs::LittleFsFormatOpts::default(), + tar_path, + codec, + true, + )?, "squashfs" => repack_stream_via_trait::( dst_dev.as_mut(), &fstool::fs::squashfs::FormatOpts::default(), @@ -2570,6 +2610,46 @@ fn create_cmd(args: CreateArgs<'_>) -> fstool::Result<()> { }, DEFAULT_MIN_SIZE, )?, + "littlefs" | "lfs" => create_via_factory::( + "littlefs", + source.as_ref(), + args.output, + args.size, + opts, + is_device, + qcow2_cluster_size, + fstool::fs::littlefs::LittleFsFormatOpts::default(), + |o: &mut fstool::fs::littlefs::LittleFsFormatOpts, m| { + if let Some(b) = m.take_u32("block_size")? { + o.block_size = b; + } + if let Some(b) = m.take_u32("block_count")? { + o.block_count = Some(b); + } + if let Some(p) = m.take_u32("prog_size")? { + o.prog_size = p; + } + if let Some(v) = m.take_str("version") { + o.disk_version = match v.trim() { + "2.0" => fstool::fs::littlefs::DISK_VERSION_2_0, + "2.1" => fstool::fs::littlefs::DISK_VERSION_2_1, + other => { + return Err(fstool::Error::InvalidArgument(format!( + "littlefs: unknown disk version {other:?} (use 2.0 or 2.1)" + ))); + } + }; + } + if let Some(n) = m.take_u32("name_max")? { + o.name_max = n; + } + if let Some(n) = m.take_u32("inline_max")? { + o.inline_max = Some(n); + } + Ok(()) + }, + DEFAULT_MIN_SIZE, + )?, "ntfs" => create_via_factory::( "ntfs", source.as_ref(), @@ -2695,7 +2775,7 @@ fn create_cmd(args: CreateArgs<'_>) -> fstool::Result<()> { other => { return Err(fstool::Error::InvalidArgument(format!( "create: unknown --type {other:?} (try ext4, fat32, exfat, hfs+, ntfs, \ - f2fs, squashfs, xfs, iso, grf, apfs, zip, cpio, ar)" + f2fs, littlefs, squashfs, xfs, iso, grf, apfs, zip, cpio, ar)" ))); } } @@ -3480,6 +3560,7 @@ fn print_fs_info(dev: &mut dyn fstool::block::BlockDevice, fs: &mut fstool::insp fstool::inspect::AnyFs::HfsPlus(hfs) => print_hfs_plus_info(hfs), fstool::inspect::AnyFs::Hfs(hfs) => print_hfs_info(hfs), fstool::inspect::AnyFs::Affs(affs) => print_affs_info(affs), + fstool::inspect::AnyFs::LittleFs(lfs) => print_littlefs_info(dev, lfs), fstool::inspect::AnyFs::Apfs(apfs) => print_apfs_info(apfs), fstool::inspect::AnyFs::Ntfs(ntfs) => print_ntfs_info(ntfs), fstool::inspect::AnyFs::F2fs(f2) => print_f2fs_info(f2), @@ -3630,6 +3711,22 @@ fn print_affs_info(affs: &fstool::fs::affs::Affs) { ); } +fn print_littlefs_info( + dev: &mut dyn fstool::block::BlockDevice, + lfs: &mut fstool::fs::littlefs::LittleFs, +) { + let (block_size, block_count) = lfs.geometry(); + let (major, minor) = lfs.version(); + println!("disk version: {major}.{minor}"); + println!("block size: {block_size}"); + println!("block count: {block_count}"); + println!("inline max: {}", lfs.inline_max()); + match lfs.used_blocks(dev) { + Ok(used) => println!("blocks used: {used} / {block_count}"), + Err(e) => println!("blocks used: "), + } +} + fn print_apfs_info(apfs: &fstool::fs::apfs::Apfs) { println!("total bytes: {}", apfs.total_bytes()); println!("block size: {}", apfs.block_size()); diff --git a/src/fs/littlefs/alloc.rs b/src/fs/littlefs/alloc.rs new file mode 100644 index 0000000..5f0b4a2 --- /dev/null +++ b/src/fs/littlefs/alloc.rs @@ -0,0 +1,118 @@ +//! Block allocation. +//! +//! littlefs keeps no free list on disk: a block is in use exactly when +//! something reachable from the superblock points at it, and the real +//! implementation rediscovers that by traversing the filesystem into a +//! "lookahead" bitmap whenever it runs dry. +//! +//! We do the same traversal once, when a handle first needs to allocate +//! (see `LittleFs::scan_used`), and then keep the bitmap exact by marking +//! blocks as they are claimed and clearing them as they are released. That +//! costs one pass over the metadata instead of one per allocation, and — +//! because the map is exact rather than a window — "no free block" here +//! really does mean the volume is full. + +use crate::{Error, Result}; + +/// In-use bitmap over the volume's blocks. +#[derive(Debug, Clone)] +pub struct Alloc { + bits: Vec, + count: u32, + /// Rotating cursor, so successive allocations spread over the device + /// instead of hammering the low blocks. + next: u32, +} + +impl Alloc { + /// An empty map for a volume of `count` blocks. + pub fn new(count: u32) -> Self { + Self { + bits: vec![0; (count as usize).div_ceil(64)], + count, + next: 0, + } + } + + /// Mark `block` in use. Out-of-range blocks are ignored — a corrupt + /// pointer shouldn't panic a traversal. + pub fn mark(&mut self, block: u32) { + if block < self.count { + self.bits[block as usize / 64] |= 1u64 << (block % 64); + } + } + + /// Release `block`. littlefs has no hard links, so nothing else can + /// still reference a block once its owner is gone. + pub fn free(&mut self, block: u32) { + if block < self.count { + self.bits[block as usize / 64] &= !(1u64 << (block % 64)); + } + } + + /// Whether `block` is currently claimed. + pub fn is_used(&self, block: u32) -> bool { + block < self.count && self.bits[block as usize / 64] & (1u64 << (block % 64)) != 0 + } + + /// Number of blocks in use. + pub fn used(&self) -> u32 { + self.bits + .iter() + .map(|w| w.count_ones()) + .sum::() + .min(self.count) + } + + /// Claim the next free block. + pub fn take(&mut self) -> Result { + for i in 0..self.count { + let b = (self.next + i) % self.count; + if !self.is_used(b) { + self.mark(b); + self.next = (b + 1) % self.count; + return Ok(b); + } + } + Err(Error::InvalidArgument( + "littlefs: no free blocks left on the volume".into(), + )) + } + + /// Claim a metadata pair — two distinct blocks. + pub fn take_pair(&mut self) -> Result<[u32; 2]> { + let a = self.take()?; + let b = self.take().inspect_err(|_| self.free(a))?; + Ok([a, b]) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn allocations_are_exact_and_reusable() { + let mut a = Alloc::new(8); + let p = a.take_pair().unwrap(); + assert_ne!(p[0], p[1]); + assert_eq!(a.used(), 2); + a.free(p[0]); + assert_eq!(a.used(), 1); + assert!(!a.is_used(p[0])); + } + + #[test] + fn a_full_volume_reports_out_of_space() { + let mut a = Alloc::new(2); + a.take().unwrap(); + a.take().unwrap(); + assert!(a.take().is_err()); + // The failed pair allocation must not leak the half it did claim. + let mut b = Alloc::new(3); + b.take().unwrap(); + b.take().unwrap(); + assert!(b.take_pair().is_err()); + assert_eq!(b.used(), 2); + } +} diff --git a/src/fs/littlefs/ctz.rs b/src/fs/littlefs/ctz.rs new file mode 100644 index 0000000..8c4c861 --- /dev/null +++ b/src/fs/littlefs/ctz.rs @@ -0,0 +1,317 @@ +//! CTZ skip-lists — how littlefs stores files too large to inline. +//! +//! A file's blocks form a reversed skip-list: block *n* starts with +//! `ctz(n)+1` pointers, the *x*-th of which points at block *n*-2ˣ, and the +//! rest of the block is file data. Only the *last* block (the "head") and +//! the file size are recorded in the metadata, which is enough to reach any +//! offset in O(log n) reads and — crucially for a copy-on-write filesystem — +//! means rewriting the file from some offset onward leaves every earlier +//! block untouched and still correctly pointed at. +//! +//! The first eight blocks of a file, with the pointers each one stores: +//! +//! ```text +//! index pointers stored at the start of the block +//! ---------------------------------------------------- +//! 0 (none — the whole block is data) +//! 1 → 0 +//! 2 → 1, 0 +//! 3 → 2 +//! 4 → 3, 2, 0 +//! 5 → 4 +//! 6 → 5, 4 +//! 7 → 6 +//! ``` +//! +//! Reaching index 0 from index 7 is then three hops (7 → 6 → 4 → 0) +//! rather than seven. + +use std::io::Read; + +use crate::block::BlockDevice; +use crate::{Error, Result}; + +use super::alloc::Alloc; +use super::mdir::Geom; + +/// Number of skip pointers stored at the start of block `index`. +pub fn pointers(index: u32) -> u32 { + if index == 0 { + 0 + } else { + index.trailing_zeros() + 1 + } +} + +/// Bytes of file data block `index` can hold. +pub fn payload(geom: &Geom, index: u32) -> u32 { + geom.block_size - 4 * pointers(index) +} + +/// `ceil(log2(a))`, littlefs's `lfs_npw2`. +fn npw2(a: u32) -> u32 { + 32 - a.wrapping_sub(1).leading_zeros() +} + +/// Map a file offset to `(block index, offset within that block)`. The +/// in-block offset includes the skip pointers, so it is the byte position to +/// read from directly. +/// +/// This is `lfs_ctz_index`: the pointer overhead of the preceding blocks is +/// a population count, because block *n* carries `ctz(n)+1` pointers and +/// `Σ ctz(k) = n - popcount(n)`. +pub fn index_of(geom: &Geom, off: u32) -> (u32, u32) { + let b = geom.block_size - 2 * 4; + let i = off / b; + if i == 0 { + return (0, off); + } + let i = off.saturating_sub(4 * ((i - 1).count_ones() + 2)) / b; + let o = off - b * i - 4 * i.count_ones(); + (i, o) +} + +/// Read one skip pointer out of a block. +fn read_pointer(dev: &mut dyn BlockDevice, geom: &Geom, block: u32, slot: u32) -> Result { + if block >= geom.block_count { + return Err(Error::InvalidImage(format!( + "littlefs: file block {block} beyond block count {}", + geom.block_count + ))); + } + let mut b = [0u8; 4]; + dev.read_at(geom.offset(block) + 4 * slot as u64, &mut b)?; + Ok(u32::from_le_bytes(b)) +} + +/// Walk the skip-list to the block holding file offset `pos`, returning it +/// and the byte offset to read from inside it. +pub fn find( + dev: &mut dyn BlockDevice, + geom: &Geom, + head: u32, + size: u32, + pos: u32, +) -> Result<(u32, u32)> { + if size == 0 { + return Err(Error::InvalidArgument( + "littlefs: seek inside an empty file".into(), + )); + } + let (mut current, _) = index_of(geom, size - 1); + let (target, off) = index_of(geom, pos); + let mut head = head; + // Each hop follows the largest pointer that doesn't overshoot, so the + // walk costs O(log n) reads rather than O(n). `current` strictly + // decreases, so a corrupt pointer can't spin here. + while current > target { + let skip = npw2(current - target + 1) + .saturating_sub(1) + .min(current.trailing_zeros()); + head = read_pointer(dev, geom, head, skip)?; + current -= 1 << skip; + } + Ok((head, off)) +} + +/// Call `cb` once for every block of the file, from the head backwards. +pub fn traverse( + dev: &mut dyn BlockDevice, + geom: &Geom, + head: u32, + size: u32, + cb: &mut dyn FnMut(u32), +) -> Result<()> { + if size == 0 { + return Ok(()); + } + let (mut index, _) = index_of(geom, size - 1); + let mut head = head; + loop { + cb(head); + if index == 0 { + return Ok(()); + } + // An odd index has its predecessor as its only "new" pointer; an + // even one lets us pick up two blocks per read. + let count = 2 - (index & 1); + let mut heads = [0u32; 2]; + for (i, h) in heads.iter_mut().enumerate().take(count as usize) { + *h = read_pointer(dev, geom, head, i as u32)?; + } + for h in heads.iter().take(count as usize - 1) { + cb(*h); + } + head = heads[count as usize - 1]; + // `count` is 1 for an odd index and 2 for an even one, so it never + // exceeds `index` here and the walk always terminates at block 0. + index -= count; + } +} + +/// File offset the data in block `index` starts at. +/// +/// The inverse of [`index_of`]: every earlier block contributes a full +/// block minus its own skip pointers, and `Σ ctz(k) = n - popcount(n)` +/// collapses that sum into a population count. +pub fn block_start(geom: &Geom, index: u32) -> u32 { + if index == 0 { + return 0; + } + index * (geom.block_size - 8) + 8 + 4 * (index - 1).count_ones() +} + +/// Where the bytes written into a CTZ block come from. +/// +/// Rewriting part of a file has to interleave data the caller supplies with +/// data still living in the old skip-list, and reading the latter needs the +/// same block device the writer is using — hence `dev` is threaded through +/// rather than captured. +pub trait ChunkSource { + /// Fill `buf` with the file's contents starting at file offset `off`. + fn fill(&mut self, dev: &mut dyn BlockDevice, off: u64, buf: &mut [u8]) -> Result<()>; +} + +/// A source that simply streams from a reader. +pub struct ReaderSource<'r> { + pub body: &'r mut dyn Read, +} + +impl ChunkSource for ReaderSource<'_> { + fn fill(&mut self, _dev: &mut dyn BlockDevice, _off: u64, buf: &mut [u8]) -> Result<()> { + self.body.read_exact(buf).map_err(Error::from) + } +} + +/// Write file data as CTZ blocks. +/// +/// Blocks are emitted starting at `index`, whose predecessor block is +/// `prev` (`None` only when `index` is 0) and whose first byte is at file +/// offset `file_off`. Returns the new head — the last block written — or +/// `prev` when there is nothing to write. +/// +/// Exactly `len` bytes are pulled from `src`; nothing larger than one block +/// is ever held in memory. +#[allow(clippy::too_many_arguments)] +pub fn write_blocks( + dev: &mut dyn BlockDevice, + geom: &Geom, + alloc: &mut Alloc, + mut index: u32, + mut prev: Option, + mut file_off: u64, + src: &mut dyn ChunkSource, + len: u64, +) -> Result> { + let bs = geom.block_size as usize; + let mut remaining = len; + let mut image = vec![0xffu8; bs]; + + while remaining > 0 { + let block = alloc.take()?; + let skips = pointers(index); + image.fill(0xff); + + // Skip pointers: the first is our predecessor, and each subsequent + // one is found by following the previous pointer's own skip list. + if skips > 0 { + let mut p = prev.ok_or_else(|| { + Error::InvalidArgument("littlefs: skip-list continuation without a head".into()) + })?; + for j in 0..skips { + let o = 4 * j as usize; + image[o..o + 4].copy_from_slice(&p.to_le_bytes()); + if j + 1 < skips { + p = read_pointer(dev, geom, p, j)?; + } + } + } + + let cap = payload(geom, index) as u64; + let n = cap.min(remaining) as usize; + let start = 4 * skips as usize; + src.fill(dev, file_off, &mut image[start..start + n])?; + dev.write_at(geom.offset(block), &image)?; + + prev = Some(block); + index += 1; + file_off += n as u64; + remaining -= n as u64; + } + + Ok(prev) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn geom(block_size: u32) -> Geom { + Geom { + block_size, + block_count: 1024, + prog_size: 16, + fcrc: true, + } + } + + #[test] + fn index_math_agrees_with_block_capacities() { + // Walking the file offset by offset must land on exactly the block + // sequence the capacities imply — this is the invariant that keeps + // reads and writes pointing at the same bytes. + let g = geom(256); + let mut off = 0u32; + for index in 0..40u32 { + let cap = payload(&g, index); + for within in 0..cap { + let (i, o) = index_of(&g, off); + assert_eq!(i, index, "offset {off} should be in block {index}"); + assert_eq!(o, 4 * pointers(index) + within); + off += 1; + } + } + } + + #[test] + fn first_block_holds_a_whole_block() { + let g = geom(4096); + assert_eq!(payload(&g, 0), 4096); + assert_eq!(index_of(&g, 0), (0, 0)); + assert_eq!(index_of(&g, 4095), (0, 4095)); + // Block 1 carries one pointer, so its data starts at byte 4. + assert_eq!(index_of(&g, 4096), (1, 4)); + } + + #[test] + fn block_start_inverts_index_of() { + let g = geom(512); + for index in 0..64u32 { + let start = block_start(&g, index); + assert_eq!(index_of(&g, start), (index, 4 * pointers(index))); + if index > 0 { + // The byte before is the last of the previous block. + assert_eq!(index_of(&g, start - 1).0, index - 1); + } + } + } + + #[test] + fn pointer_counts_follow_ctz() { + assert_eq!(pointers(0), 0); + assert_eq!(pointers(1), 1); + assert_eq!(pointers(2), 2); + assert_eq!(pointers(3), 1); + assert_eq!(pointers(4), 3); + assert_eq!(pointers(8), 4); + } + + #[test] + fn npw2_matches_ceil_log2() { + assert_eq!(npw2(1), 0); + assert_eq!(npw2(2), 1); + assert_eq!(npw2(3), 2); + assert_eq!(npw2(4), 2); + assert_eq!(npw2(5), 3); + } +} diff --git a/src/fs/littlefs/mdir.rs b/src/fs/littlefs/mdir.rs new file mode 100644 index 0000000..290a8dc --- /dev/null +++ b/src/fs/littlefs/mdir.rs @@ -0,0 +1,543 @@ +//! Metadata pairs — littlefs's unit of metadata storage. +//! +//! A metadata pair is two blocks. Each block is a 32-bit revision count +//! followed by an append-only log of *commits*; each commit is a run of +//! tag+data entries terminated by a CRC tag. The block of the pair with the +//! newer revision count that still has a valid commit is the live one. +//! +//! [`fetch`] replays a pair's log into the flat [`Mdir`] view the rest of +//! the backend works with: a `Vec` indexed by file id, the pair's +//! tail pointer, and any global-state delta. [`Mdir::commit`] goes the other +//! way, writing the whole state back as a single fresh commit (a +//! *compaction*) into the pair's stale block, then swapping the pair so the +//! newly written block is the live one. That is the same operation littlefs +//! performs whenever a metadata block fills up or isn't in a known-erased +//! state; always taking it keeps the writer simple and every image we +//! produce is one a stock littlefs can mount and keep appending to. + +use crate::block::BlockDevice; +use crate::{Error, Result}; + +use super::tag::{self, Tag}; + +/// Geometry + format flavour of a littlefs volume. Fixed at format time +/// (block size / count are recorded in the superblock) except for +/// `prog_size`, which is a property of the target flash, not the image. +#[derive(Debug, Clone, Copy)] +pub struct Geom { + /// Logical block (erase unit) size. + pub block_size: u32, + /// Number of blocks in the volume. + pub block_count: u32, + /// Program alignment commits are padded to. Also the size of the window + /// an FCRC covers. + pub prog_size: u32, + /// Whether to emit lfs2.1 forward-CRC tags. Off for images pinned to + /// disk version 2.0, whose readers mistake an FCRC for a commit CRC. + pub fcrc: bool, +} + +impl Geom { + /// Byte offset of `block` on the device. + pub fn offset(&self, block: u32) -> u64 { + block as u64 * self.block_size as u64 + } + + /// Largest commit body (everything after the revision count, before the + /// CRC tag) a metadata block can hold. littlefs reserves 40 bytes for + /// the tail, global state, a trailing delete and the CRC itself. + pub fn commit_limit(&self) -> usize { + self.block_size as usize - 40 + } + + /// Size past which a metadata pair is split in two. littlefs caps a + /// compaction at half a block so a pair that is repeatedly appended to + /// doesn't degenerate into one that must compact on every commit. + pub fn split_limit(&self) -> usize { + let half = (self.block_size as usize / 2).next_multiple_of(self.prog_size.max(1) as usize); + half.min(self.commit_limit()) + } +} + +/// The on-disk structure attached to a file id. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Struct { + /// Directory: pointer to the first metadata pair of the directory. + Dir([u32; 2]), + /// Small file stored directly in the metadata block. + Inline(Vec), + /// File stored as a CTZ skip-list rooted at `head`. + Ctz { head: u32, size: u32 }, +} + +/// One file id within a metadata pair. +#[derive(Debug, Clone, Default)] +pub struct Entry { + /// Chunk field of the name tag: [`tag::TYPE_REG`], [`tag::TYPE_DIR`] or + /// [`tag::TYPE_SUPERBLOCK`]. Zero for an id created by a splice whose + /// name tag hasn't been seen (never happens in a compacted block). + pub kind: u8, + /// File name, as stored (littlefs names are byte strings). + pub name: Vec, + /// Struct tag contents, if any. + pub data: Option, + /// User attributes, keyed by the 8-bit attribute type. + pub attrs: Vec<(u8, Vec)>, +} + +impl Entry { + /// Whether this id is a real filesystem entry. The superblock shares the + /// root directory's metadata pair as id 0 but is not a file — littlefs + /// filters it out of directory listings by masking the name tag's type, + /// and so do we. + pub fn is_file(&self) -> bool { + self.kind == tag::TYPE_REG as u8 || self.kind == tag::TYPE_DIR as u8 + } + + /// Bytes this entry occupies in a commit: its name tag, struct tag and + /// every user attribute. + fn commit_size(&self) -> usize { + let mut n = 4 + self.name.len(); + n += match &self.data { + Some(Struct::Dir(_)) | Some(Struct::Ctz { .. }) => 4 + 8, + Some(Struct::Inline(d)) => 4 + d.len(), + None => 0, + }; + for (_, v) in &self.attrs { + n += 4 + v.len(); + } + n + } +} + +/// A metadata pair, replayed into a flat view. +#[derive(Debug, Clone)] +pub struct Mdir { + /// The pair's blocks, live one first. + pub pair: [u32; 2], + /// Revision count of the live block. + pub rev: u32, + /// Files in this block, indexed by id. + pub entries: Vec, + /// Next metadata pair in the threaded list, if any. + pub tail: Option<[u32; 2]>, + /// Whether `tail` is a *hard* tail — the continuation of this same + /// directory — rather than a soft one that merely threads the list. + pub hard: bool, + /// Global-state delta carried by this pair, preserved verbatim. The + /// filesystem's global state is the XOR of every pair's delta, so a + /// rewrite that dropped it would corrupt the sum. + pub gdelta: Option<[u8; 12]>, + /// Window size of the last forward-CRC seen while replaying the log. + /// That is the program size the volume's previous writer used, which is + /// the only place it is recorded — the superblock doesn't carry it. + pub fcrc_size: Option, +} + +impl Mdir { + /// An empty pair, not yet written to disk. + pub fn empty(pair: [u32; 2]) -> Self { + Self { + pair, + rev: 0, + entries: Vec::new(), + tail: None, + hard: false, + gdelta: None, + fcrc_size: None, + } + } + + /// Find an entry by name, returning its id. + pub fn find(&self, name: &[u8]) -> Option { + self.entries + .iter() + .position(|e| e.is_file() && e.name == name) + } + + /// Total commit size of this pair's contents. + fn body_size(&self) -> usize { + let mut n: usize = self.entries.iter().map(Entry::commit_size).sum(); + if self.tail.is_some() { + n += 4 + 8; + } + if self.gdelta.is_some() { + n += 4 + 12; + } + n + } +} + +/// Read a metadata pair and replay its log. +pub fn fetch(dev: &mut dyn BlockDevice, geom: &Geom, pair: [u32; 2]) -> Result { + for b in pair { + if b >= geom.block_count { + return Err(Error::InvalidImage(format!( + "littlefs: metadata pair block {b} beyond block count {}", + geom.block_count + ))); + } + } + + // Try the block with the newer revision count first; fall back to its + // partner when it holds no valid commit (a power cut mid-compaction). + let mut order = pair; + if let (Some(a), Some(b)) = (read_rev(dev, geom, pair[0]), read_rev(dev, geom, pair[1])) + && tag::rev_newer(b, a) + { + order.swap(0, 1); + } + + for i in 0..2 { + let block = order[i]; + if let Some(mut mdir) = parse_block(dev, geom, block)? { + mdir.pair = [block, order[1 - i]]; + return Ok(mdir); + } + } + + Err(Error::InvalidImage(format!( + "littlefs: corrupted metadata pair {{{}, {}}}", + pair[0], pair[1] + ))) +} + +/// Read just the revision count of a block. `None` when the read fails — +/// treated the same as a corrupt block. +pub fn read_rev(dev: &mut dyn BlockDevice, geom: &Geom, block: u32) -> Option { + let mut b = [0u8; 4]; + dev.read_at(geom.offset(block), &mut b).ok()?; + Some(u32::from_le_bytes(b)) +} + +/// Replay one block's commits. Returns `None` if the block holds no valid +/// commit at all, otherwise the state as of its last valid commit. +fn parse_block(dev: &mut dyn BlockDevice, geom: &Geom, block: u32) -> Result> { + let bs = geom.block_size as usize; + let mut buf = vec![0u8; bs]; + if dev.read_at(geom.offset(block), &mut buf).is_err() { + return Ok(None); + } + + let rev = u32::from_le_bytes([buf[0], buf[1], buf[2], buf[3]]); + let mut live: Option = None; + let mut cur = Mdir::empty([block, block]); + cur.rev = rev; + + let mut off = 0usize; + let mut ptag = tag::PTAG_INIT; + let mut crc = tag::crc(tag::PTAG_INIT, &buf[0..4]); + + loop { + // Tags are chained: the next one starts right after the previous + // tag's data. `PTAG_INIT` has an all-ones size field, i.e. "deleted", + // so the very first step skips exactly the 4-byte revision count. + off += Tag(ptag).dsize(); + if off + 4 > bs { + break; + } + crc = tag::crc(crc, &buf[off..off + 4]); + let t = Tag(tag::be32(&buf[off..off + 4]) ^ ptag); + if !t.is_valid() || off + t.dsize() > bs { + // Unwritten (or interrupted) storage — end of the log. + break; + } + ptag = t.0; + + if t.type2() == tag::TYPE_CCRC { + if off + 8 > bs { + break; + } + if crc != tag::le32(&buf[off + 4..off + 8]) { + break; + } + // The CRC tag's low chunk bit flips the valid-bit state the next + // commit's tags are expected to have. + ptag ^= ((t.chunk() & 1) as u32) << 31; + live = Some(cur.clone()); + crc = tag::PTAG_INIT; + continue; + } + + let data = &buf[off + 4..off + t.dsize()]; + crc = tag::crc(crc, data); + apply(&mut cur, t, data); + } + + Ok(live) +} + +/// Fold one tag into the running state. Later tags supersede earlier ones +/// for the same (type, id), which is what makes a metadata block an +/// append-only log of overrides. +fn apply(mdir: &mut Mdir, t: Tag, data: &[u8]) { + let id = t.id() as usize; + match t.type1() { + tag::T1_NAME => { + grow(mdir, id); + if let Some(e) = mdir.entries.get_mut(id) { + e.kind = t.chunk(); + e.name = data.to_vec(); + } + } + tag::T1_STRUCT => { + grow(mdir, id); + let Some(e) = mdir.entries.get_mut(id) else { + return; + }; + // Any struct supersedes any other struct on the same id. + e.data = match t.type3() { + tag::TYPE_DIRSTRUCT if data.len() >= 8 => Some(Struct::Dir([ + tag::le32(&data[0..4]), + tag::le32(&data[4..8]), + ])), + tag::TYPE_CTZSTRUCT if data.len() >= 8 => Some(Struct::Ctz { + head: tag::le32(&data[0..4]), + size: tag::le32(&data[4..8]), + }), + tag::TYPE_INLINESTRUCT => Some(Struct::Inline(data.to_vec())), + _ => e.data.take(), + }; + } + tag::T1_USERATTR => { + grow(mdir, id); + let Some(e) = mdir.entries.get_mut(id) else { + return; + }; + let key = t.chunk(); + e.attrs.retain(|(k, _)| *k != key); + if !t.is_delete() { + e.attrs.push((key, data.to_vec())); + e.attrs.sort_by_key(|(k, _)| *k); + } + } + tag::T1_SPLICE => { + // A create inserts an id (shifting later ids up), a delete + // removes one (shifting them down) — insertion into and removal + // from an imaginary array of files. + match t.type3() { + tag::TYPE_CREATE => { + if id <= mdir.entries.len() { + mdir.entries.insert(id, Entry::default()); + } else { + grow(mdir, id); + } + } + tag::TYPE_DELETE if id < mdir.entries.len() => { + mdir.entries.remove(id); + } + _ => {} + } + } + tag::T1_TAIL => { + if data.len() >= 8 { + mdir.tail = Some([tag::le32(&data[0..4]), tag::le32(&data[4..8])]); + mdir.hard = t.chunk() & 1 != 0; + } + } + tag::T1_GSTATE => { + if data.len() >= 12 { + let mut g = [0u8; 12]; + g.copy_from_slice(&data[..12]); + mdir.gdelta = Some(g); + } + } + // Commit CRCs are consumed by the parser itself; the only CRC tag + // that reaches here is the lfs2.1 forward-CRC, whose size field + // tells us the writer's program alignment. + tag::T1_CRC if t.type3() == tag::TYPE_FCRC && data.len() >= 8 => { + mdir.fcrc_size = Some(tag::le32(&data[0..4])); + } + _ => {} + } +} + +/// Make room for file id `id`, as a name tag for an id past the current +/// count implicitly does in littlefs. +fn grow(mdir: &mut Mdir, id: usize) { + // Guard against a corrupt tag claiming a huge id; ids are 10 bits and + // 0x3ff is reserved for block-level tags, so nothing legitimate is + // anywhere near this bound. + if id >= tag::ID_NONE as usize { + return; + } + while mdir.entries.len() <= id { + mdir.entries.push(Entry::default()); + } +} + +/// Builder for a single commit: tags are appended, then [`Self::finish`] +/// closes it out with the CRC tag and returns the block image to write. +struct CommitBuf { + buf: Vec, + ptag: u32, +} + +impl CommitBuf { + fn new(rev: u32) -> Self { + Self { + buf: rev.to_le_bytes().to_vec(), + ptag: tag::PTAG_INIT, + } + } + + /// Append one tag and its data. + fn push(&mut self, t: Tag, data: &[u8]) { + let stored = (t.0 & 0x7fff_ffff) ^ self.ptag; + self.buf.extend_from_slice(&stored.to_be_bytes()); + if !t.is_delete() { + self.buf.extend_from_slice(data); + } + self.ptag = t.0 & 0x7fff_ffff; + } + + /// Close the commit and render the full block image. + /// + /// Everything after the commit is left in the erased state (`0xff`), the + /// convention littlefs's block devices use, so that a real littlefs can + /// append its next commit here in place. That is also what the optional + /// forward-CRC records: the checksum of the erased window that follows, + /// proving to the next mount that nothing was half-programmed into it. + fn finish(mut self, geom: &Geom) -> Result> { + let bs = geom.block_size as usize; + let prog = geom.prog_size.max(1) as usize; + + // Room for the FCRC (tag + 8) plus the CRC tag (tag + 4), matching + // the 5-word window littlefs reserves. + let reserve = if geom.fcrc { 5 * 4 } else { 2 * 4 }; + let end = (self.buf.len() + reserve).min(bs).next_multiple_of(prog); + if end > bs { + return Err(Error::InvalidArgument( + "littlefs: commit does not fit in a metadata block".into(), + )); + } + + // The erased tail: everything from the end of this commit onwards. + let mut block = vec![0xffu8; bs]; + + if geom.fcrc && end <= bs - prog { + let fcrc_crc = tag::crc(tag::PTAG_INIT, &block[end..end + prog]); + let mut d = [0u8; 8]; + d[0..4].copy_from_slice(&(prog as u32).to_le_bytes()); + d[4..8].copy_from_slice(&fcrc_crc.to_le_bytes()); + self.push(Tag::new(tag::TYPE_FCRC, tag::ID_NONE, 8), &d); + } + + // The CRC tag's size field covers the padding up to `end`, so a + // fetch can skip straight over it. The low chunk bit is chosen so + // that reading the erased byte at `end` yields an *invalid* tag, + // which is how the next mount recognises unwritten storage. + let pad = end - (self.buf.len() + 4); + if pad > tag::MAX_SIZE { + return Err(Error::InvalidArgument( + "littlefs: commit padding exceeds a single CRC tag".into(), + )); + } + let eperturb: u8 = if end < bs { block[end] } else { 0xff }; + let ccrc = Tag::new( + tag::TYPE_CCRC + ((!eperturb) >> 7) as u16, + tag::ID_NONE, + pad as u16, + ); + let stored = (ccrc.0 & 0x7fff_ffff) ^ self.ptag; + self.buf.extend_from_slice(&stored.to_be_bytes()); + + let crc = tag::crc(tag::PTAG_INIT, &self.buf); + self.buf.extend_from_slice(&crc.to_le_bytes()); + + block[..self.buf.len()].copy_from_slice(&self.buf); + Ok(block) + } +} + +/// Render `mdir`'s state as a commit body and write it to `block` with +/// revision count `rev`. +pub fn write_compaction( + dev: &mut dyn BlockDevice, + geom: &Geom, + mdir: &Mdir, + block: u32, + rev: u32, +) -> Result<()> { + let mut c = CommitBuf::new(rev); + for (id, e) in mdir.entries.iter().enumerate() { + let id = id as u16; + // The name tag must come first for an id; everything else hangs + // off it. + c.push( + Tag::new(tag::TYPE_NAME | e.kind as u16, id, e.name.len() as u16), + &e.name, + ); + match &e.data { + Some(Struct::Dir(p)) => { + let mut d = [0u8; 8]; + d[0..4].copy_from_slice(&p[0].to_le_bytes()); + d[4..8].copy_from_slice(&p[1].to_le_bytes()); + c.push(Tag::new(tag::TYPE_DIRSTRUCT, id, 8), &d); + } + Some(Struct::Ctz { head, size }) => { + let mut d = [0u8; 8]; + d[0..4].copy_from_slice(&head.to_le_bytes()); + d[4..8].copy_from_slice(&size.to_le_bytes()); + c.push(Tag::new(tag::TYPE_CTZSTRUCT, id, 8), &d); + } + Some(Struct::Inline(data)) => { + c.push( + Tag::new(tag::TYPE_INLINESTRUCT, id, data.len() as u16), + data, + ); + } + None => {} + } + for (k, v) in &e.attrs { + c.push( + Tag::new(tag::TYPE_USERATTR | *k as u16, id, v.len() as u16), + v, + ); + } + } + if let Some(t) = mdir.tail { + let mut d = [0u8; 8]; + d[0..4].copy_from_slice(&t[0].to_le_bytes()); + d[4..8].copy_from_slice(&t[1].to_le_bytes()); + let ty = if mdir.hard { + tag::TYPE_HARDTAIL + } else { + tag::TYPE_SOFTTAIL + }; + c.push(Tag::new(ty, tag::ID_NONE, 8), &d); + } + if let Some(g) = mdir.gdelta { + c.push(Tag::new(tag::TYPE_MOVESTATE, tag::ID_NONE, 12), &g); + } + + let image = c.finish(geom)?; + dev.write_at(geom.offset(block), &image) +} + +/// Whether `mdir`'s contents still fit one metadata block, or need to be +/// split across two pairs first. +pub fn needs_split(geom: &Geom, mdir: &Mdir) -> bool { + // littlefs also caps a pair at 0xff ids, halving the split point until + // both bounds hold. + mdir.body_size() > geom.split_limit() || mdir.entries.len() >= 0xff +} + +/// Pick how many leading entries stay in the pair when splitting, mirroring +/// littlefs's "halve until it fits" search. Returns `0` when even a single +/// entry is too large for a block — the caller turns that into an error. +pub fn split_point(geom: &Geom, mdir: &Mdir) -> usize { + let end = mdir.entries.len(); + let mut split = 0usize; + while end - split > 1 { + let size: usize = mdir.entries[split..end] + .iter() + .map(Entry::commit_size) + .sum(); + if end - split < 0xff && size <= geom.split_limit() { + break; + } + split += (end - split) / 2; + } + split +} diff --git a/src/fs/littlefs/mod.rs b/src/fs/littlefs/mod.rs new file mode 100644 index 0000000..fce2a30 --- /dev/null +++ b/src/fs/littlefs/mod.rs @@ -0,0 +1,1296 @@ +//! littlefs — read + write support for the little fail-safe filesystem +//! used on microcontroller flash (`lfs2`, disk versions 2.0 and 2.1). +//! +//! ## On-disk format (little-endian, except tags) +//! +//! littlefs has no fixed superblock region, no allocation table and no inode +//! table. Everything is built from two structures: +//! +//! * **Metadata pairs** — two blocks holding a revision count and an +//! append-only log of commits; the block with the newer revision count that +//! ends in a valid CRC is the live one. Each commit is a run of 32-bit +//! *tags* (the `tag` submodule) and their data. A directory is a linked list of +//! metadata pairs; the pair at blocks `{0, 1}` holds the superblock entry +//! (the magic `"littlefs"` at offset 8) and doubles as the root directory. +//! Every pair in the volume is also threaded onto one list through *tail* +//! pointers, which is what makes a full traversal — and therefore block +//! allocation — possible without an on-disk free map. +//! * **CTZ skip-lists** — file data too large to inline in a metadata block +//! (the `ctz` submodule). Files smaller than `inline_max` live directly in their +//! directory's metadata instead. +//! +//! ## What this backend does +//! +//! [`LittleFs::format`] lays down a fresh volume, and [`LittleFs::open`] +//! mounts an existing one; both return a fully mutable handle. Every +//! mutation is written through immediately as a real littlefs commit — +//! there is no build-once mode and no in-memory image, so `create -t +//! littlefs`, `repack`, `add`/`rm` and `open_file_rw` all drive the same +//! code path and a re-opened image keeps working exactly like a fresh one. +//! +//! Each commit rewrites the whole metadata pair (a *compaction*) into its +//! stale block rather than appending to the live one. That is the same +//! operation littlefs performs whenever a block fills up, so the result is +//! always a volume a stock littlefs can mount and keep appending to — at +//! the cost of writing a block per metadata change, which is the right +//! trade for an image tool. +//! +//! ## Metadata mapping +//! +//! littlefs stores no POSIX metadata at all: no mode, owner, timestamps, +//! symlinks or device nodes. Modes are therefore synthesised on read +//! (`0o755` for directories, `0o644` for files) and dropped on write, and +//! [`Filesystem::create_symlink`] / [`Filesystem::create_device`] report +//! [`Error::Unsupported`] so a `repack` sink skips those entries rather +//! than silently mangling them. littlefs *user attributes* are surfaced as +//! extended attributes named `user.littlefs.`, where `` is the +//! attribute's 8-bit type in decimal. + +use std::collections::HashMap; +use std::collections::VecDeque; +use std::io::Read; +use std::path::Path; + +use crate::block::BlockDevice; +use crate::fs::{ + DirEntry, EntryKind, FileAttrs, FileMeta, FileSource, Filesystem, MutationCapability, StatFs, + XattrPair, +}; +use crate::{Error, Result}; + +mod alloc; +mod ctz; +mod mdir; +mod rw; +mod size_plan; +mod tag; +#[cfg(test)] +mod tests; + +pub use size_plan::LittleFsSizePlan; + +use alloc::Alloc; +use mdir::{Entry, Geom, Mdir, Struct}; + +/// Disk version 2.0 — understood by every littlefs v2 release. Images +/// pinned to it carry no forward-CRC tags, which releases older than +/// lfs2.1 would mistake for a commit CRC. +pub const DISK_VERSION_2_0: u32 = 0x0002_0000; +/// Disk version 2.1 — the current on-disk version, with forward-CRC tags. +pub const DISK_VERSION_2_1: u32 = 0x0002_0001; + +/// The metadata pair every littlefs volume is rooted at. +const SUPERBLOCK_PAIR: [u32; 2] = [0, 1]; +/// Magic string carried by the superblock's name tag. +const MAGIC: &[u8; 8] = b"littlefs"; +/// Largest value littlefs allows for `file_max`. +const FILE_MAX: u32 = 0x7fff_ffff; + +/// Prefix of the extended-attribute names littlefs user attributes are +/// surfaced under; the 8-bit attribute type follows in decimal. +const XATTR_PREFIX: &str = "user.littlefs."; + +/// Format-time options for [`LittleFs::format`]. +#[derive(Debug, Clone)] +pub struct LittleFsFormatOpts { + /// Logical block size — the flash erase-block size. littlefs stores it + /// in the superblock; 4 KiB is the common default. + pub block_size: u32, + /// Number of blocks. `None` fills the device. + pub block_count: Option, + /// Program (page) alignment. Commits are padded to it so that a real + /// littlefs can append in place. + pub prog_size: u32, + /// On-disk version to write: [`DISK_VERSION_2_1`] (default) or + /// [`DISK_VERSION_2_0`] for targets running a pre-2.1 littlefs. + pub disk_version: u32, + /// Longest file name the volume accepts. + pub name_max: u32, + /// Largest file kept inline in its directory's metadata instead of + /// being written out as a CTZ skip-list. `None` picks littlefs's own + /// default of an eighth of a block. + pub inline_max: Option, +} + +impl Default for LittleFsFormatOpts { + fn default() -> Self { + Self { + block_size: 4096, + block_count: None, + prog_size: 256, + disk_version: DISK_VERSION_2_1, + name_max: 255, + inline_max: None, + } + } +} + +/// A mounted littlefs volume. +pub struct LittleFs { + geom: Geom, + version: u32, + name_max: u32, + file_max: u32, + attr_max: u32, + inline_max: u32, + root: [u32; 2], + /// In-use bitmap, built on first allocation and maintained exactly from + /// then on. `None` until something needs to allocate. + alloc: Option, + cache: MdirCache, +} + +/// Small LRU over parsed metadata pairs. Directory operations walk the same +/// pairs repeatedly (a lookup, then an insert, then a commit), and every +/// walk would otherwise re-read and re-parse a block. +struct MdirCache { + map: HashMap<[u32; 2], Mdir>, + order: VecDeque<[u32; 2]>, + cap: usize, +} + +impl MdirCache { + fn new(cap: usize) -> Self { + Self { + map: HashMap::new(), + order: VecDeque::new(), + cap, + } + } + + /// Cache key: a pair addresses the same metadata whichever way round it + /// is written, and a commit swaps the two halves. + fn key(pair: [u32; 2]) -> [u32; 2] { + if pair[0] <= pair[1] { + pair + } else { + [pair[1], pair[0]] + } + } + + fn get(&self, pair: [u32; 2]) -> Option<&Mdir> { + self.map.get(&Self::key(pair)) + } + + fn put(&mut self, mdir: Mdir) { + let k = Self::key(mdir.pair); + if self.map.insert(k, mdir).is_none() { + self.order.push_back(k); + while self.order.len() > self.cap { + if let Some(old) = self.order.pop_front() { + self.map.remove(&old); + } + } + } + } + + fn remove(&mut self, pair: [u32; 2]) { + let k = Self::key(pair); + self.map.remove(&k); + self.order.retain(|p| *p != k); + } +} + +/// What a path resolved to. +enum Resolved { + /// The root directory, which has no entry of its own. + Root, + /// An entry at `id` in the metadata pair `mdir`. + Entry { mdir: Mdir, id: usize }, +} + +impl LittleFs { + /// Format a fresh volume on `dev`. + pub fn format(dev: &mut dyn BlockDevice, opts: &LittleFsFormatOpts) -> Result { + let block_size = opts.block_size; + // 128 bytes is the floor at which a CTZ block can still hold its + // skip pointers (the spec's bound is 104); everything else in + // littlefs assumes a power-of-two erase block. + if block_size < 128 || !block_size.is_power_of_two() { + return Err(Error::InvalidArgument(format!( + "littlefs: block_size {block_size} must be a power of two and at least 128" + ))); + } + let prog_size = opts.prog_size.max(1); + if !prog_size.is_power_of_two() || prog_size > block_size { + return Err(Error::InvalidArgument(format!( + "littlefs: prog_size {prog_size} must be a power of two no larger than the block size" + ))); + } + if opts.disk_version != DISK_VERSION_2_0 && opts.disk_version != DISK_VERSION_2_1 { + return Err(Error::InvalidArgument(format!( + "littlefs: unsupported disk version {:#010x} (use 2.0 or 2.1)", + opts.disk_version + ))); + } + + let avail = (dev.total_size() / block_size as u64).min(u32::MAX as u64) as u32; + let block_count = opts.block_count.unwrap_or(avail); + if block_count > avail { + return Err(Error::InvalidArgument(format!( + "littlefs: block_count {block_count} exceeds the {avail} blocks the device holds" + ))); + } + // The superblock pair plus room for a directory pair and some data. + if block_count < 4 { + return Err(Error::InvalidArgument( + "littlefs: a volume needs at least 4 blocks".into(), + )); + } + if opts.name_max == 0 || opts.name_max > tag::MAX_SIZE as u32 { + return Err(Error::InvalidArgument(format!( + "littlefs: name_max {} must be between 1 and {}", + opts.name_max, + tag::MAX_SIZE + ))); + } + + let geom = Geom { + block_size, + block_count, + prog_size, + fcrc: opts.disk_version >= DISK_VERSION_2_1, + }; + let attr_max = tag::MAX_SIZE as u32; + let inline_max = pick_inline_max(&geom, opts.inline_max)?; + + let mut fs = Self { + geom, + version: opts.disk_version, + name_max: opts.name_max, + file_max: FILE_MAX, + attr_max, + inline_max, + root: SUPERBLOCK_PAIR, + alloc: None, + cache: MdirCache::new(32), + }; + + // The root pair is written twice, exactly as `lfs_format` does: the + // second compaction lands in the other block so that *both* halves + // of the pair are valid littlefs commits, leaving nothing of an + // older filesystem behind for a fetch to trip over. + let mut root = Mdir::empty([SUPERBLOCK_PAIR[1], SUPERBLOCK_PAIR[0]]); + root.entries.push(Entry { + kind: tag::TYPE_SUPERBLOCK as u8, + name: MAGIC.to_vec(), + data: Some(Struct::Inline(fs.superblock_bytes())), + attrs: Vec::new(), + }); + fs.commit(dev, &mut root)?; + fs.commit(dev, &mut root)?; + + // Claim the superblock pair up front so nothing else can hand it out. + let mut a = Alloc::new(block_count); + a.mark(SUPERBLOCK_PAIR[0]); + a.mark(SUPERBLOCK_PAIR[1]); + fs.alloc = Some(a); + Ok(fs) + } + + /// Mount an existing volume. + pub fn open(dev: &mut dyn BlockDevice) -> Result { + // The superblock's inline struct always sits at a fixed offset in + // the first commit of block 0 — that's the only way to learn the + // block size, which everything else needs. + let mut head = [0u8; 44]; + let n = head.len().min(dev.total_size() as usize); + dev.read_at(0, &mut head[..n])?; + if &head[8..16] != MAGIC { + return Err(Error::InvalidImage( + "littlefs: no \"littlefs\" magic at offset 8".into(), + )); + } + let version = tag::le32(&head[20..24]); + let block_size = tag::le32(&head[24..28]); + let block_count = tag::le32(&head[28..32]); + if version >> 16 != 2 { + return Err(Error::Unsupported(format!( + "littlefs: on-disk version {}.{} (only v2 is supported)", + version >> 16, + version & 0xffff + ))); + } + if version & 0xffff > 1 { + return Err(Error::Unsupported(format!( + "littlefs: on-disk version 2.{} is newer than 2.1", + version & 0xffff + ))); + } + if !(128..=16 * 1024 * 1024).contains(&block_size) || block_count == 0 { + return Err(Error::InvalidImage(format!( + "littlefs: implausible geometry ({block_size}-byte blocks × {block_count})" + ))); + } + if (block_size as u64).saturating_mul(block_count as u64) > dev.total_size() { + return Err(Error::InvalidImage(format!( + "littlefs: volume claims {block_count} × {block_size}-byte blocks but the device holds {} bytes", + dev.total_size() + ))); + } + + let geom = Geom { + block_size, + block_count, + prog_size: 1, + fcrc: version >= DISK_VERSION_2_1, + }; + let mut fs = Self { + geom, + version, + name_max: tag::le32(&head[32..36]), + file_max: tag::le32(&head[36..40]), + attr_max: tag::le32(&head[40..44]), + inline_max: 0, + root: SUPERBLOCK_PAIR, + alloc: None, + cache: MdirCache::new(32), + }; + if fs.name_max == 0 || fs.name_max > tag::MAX_SIZE as u32 { + fs.name_max = 255; + } + if fs.file_max == 0 { + fs.file_max = FILE_MAX; + } + if fs.attr_max == 0 || fs.attr_max > tag::MAX_SIZE as u32 { + fs.attr_max = tag::MAX_SIZE as u32; + } + + // Walk the superblock chain: the last pair still carrying a + // superblock entry is the root directory. littlefs grows this chain + // as the root is rewritten, to spread erase cycles. + let mut pair = Some(SUPERBLOCK_PAIR); + let mut hops = 0u32; + while let Some(p) = pair { + let m = mdir::fetch(dev, &fs.geom, p)?; + if m.entries + .first() + .is_some_and(|e| e.kind == tag::TYPE_SUPERBLOCK as u8) + { + fs.root = m.pair; + // A commit's forward-CRC records the program size its + // writer used; reusing it keeps our commits aligned the way + // the volume's creator intended. + if let Some(p) = m.fcrc_size + && p.is_power_of_two() + && p <= block_size + { + fs.geom.prog_size = p; + } + } + pair = m.tail; + hops += 1; + if hops > block_count { + return Err(Error::InvalidImage( + "littlefs: cycle in the metadata-pair list".into(), + )); + } + } + if fs.geom.prog_size == 1 { + fs.geom.prog_size = 256.min(block_size / 4).max(1); + } + fs.inline_max = pick_inline_max(&fs.geom, None)?; + fs.cache = MdirCache::new(32); + Ok(fs) + } + + /// Volume geometry: `(block size, block count)`. + pub fn geometry(&self) -> (u32, u32) { + (self.geom.block_size, self.geom.block_count) + } + + /// On-disk version, as `(major, minor)`. + pub fn version(&self) -> (u16, u16) { + ((self.version >> 16) as u16, (self.version & 0xffff) as u16) + } + + /// Largest file kept inline in metadata rather than written as a CTZ + /// skip-list. + pub fn inline_max(&self) -> u32 { + self.inline_max + } + + /// Blocks currently in use. + pub fn used_blocks(&mut self, dev: &mut dyn BlockDevice) -> Result { + Ok(self.allocator(dev)?.used()) + } + + /// The 24-byte superblock configuration record. + fn superblock_bytes(&self) -> Vec { + let mut b = Vec::with_capacity(24); + for v in [ + self.version, + self.geom.block_size, + self.geom.block_count, + self.name_max, + self.file_max, + self.attr_max, + ] { + b.extend_from_slice(&v.to_le_bytes()); + } + b + } + + // ---- metadata pairs ------------------------------------------------- + + /// Fetch a metadata pair, through the cache. + fn fetch(&mut self, dev: &mut dyn BlockDevice, pair: [u32; 2]) -> Result { + if let Some(m) = self.cache.get(pair) { + return Ok(m.clone()); + } + let m = mdir::fetch(dev, &self.geom, pair)?; + self.cache.put(m.clone()); + Ok(m) + } + + /// Write `mdir` back as a fresh compaction, splitting it across further + /// pairs first if its contents no longer fit one metadata block. + fn commit(&mut self, dev: &mut dyn BlockDevice, mdir: &mut Mdir) -> Result<()> { + while mdir::needs_split(&self.geom, mdir) { + let at = mdir::split_point(&self.geom, mdir); + if at == 0 { + return Err(Error::InvalidArgument( + "littlefs: a single entry is too large for a metadata block".into(), + )); + } + let mut tail = self.new_pair(dev)?; + tail.entries = mdir.entries.split_off(at); + tail.tail = mdir.tail; + tail.hard = mdir.hard; + self.commit(dev, &mut tail)?; + // The overflow pair becomes the continuation of this directory, + // which also keeps it threaded on the filesystem-wide list. + mdir.tail = Some(tail.pair); + mdir.hard = true; + } + + mdir.rev = mdir.rev.wrapping_add(1); + let target = mdir.pair[1]; + mdir::write_compaction(dev, &self.geom, mdir, target, mdir.rev)?; + // The block we just wrote is now the live half of the pair. + mdir.pair.swap(0, 1); + self.cache.put(mdir.clone()); + Ok(()) + } + + /// Allocate a metadata pair that isn't on disk yet. + /// + /// The revision count is seeded from the block we will *not* write + /// first, so that our commit always outranks whatever an earlier + /// filesystem left in the other half of the pair. + fn new_pair(&mut self, dev: &mut dyn BlockDevice) -> Result { + let pair = self.allocator(dev)?.take_pair()?; + let mut m = Mdir::empty(pair); + m.rev = mdir::read_rev(dev, &self.geom, pair[0]).unwrap_or(0); + Ok(m) + } + + // ---- allocation ----------------------------------------------------- + + /// The in-use bitmap, built by traversing the volume the first time + /// anything needs to allocate. + fn allocator(&mut self, dev: &mut dyn BlockDevice) -> Result<&mut Alloc> { + if self.alloc.is_none() { + let a = self.scan_used(dev)?; + self.alloc = Some(a); + } + Ok(self.alloc.as_mut().expect("just built")) + } + + /// Walk every metadata pair on the threaded list and every file's + /// skip-list, marking the blocks they occupy. + fn scan_used(&mut self, dev: &mut dyn BlockDevice) -> Result { + let geom = self.geom; + let mut a = Alloc::new(geom.block_count); + let mut next = Some(SUPERBLOCK_PAIR); + let mut hops = 0u32; + while let Some(pair) = next { + let m = self.fetch(dev, pair)?; + a.mark(m.pair[0]); + a.mark(m.pair[1]); + for e in &m.entries { + if let Some(Struct::Ctz { head, size }) = &e.data { + ctz::traverse(dev, &geom, *head, *size, &mut |b| a.mark(b))?; + } + } + next = m.tail; + hops += 1; + if hops > geom.block_count { + return Err(Error::InvalidImage( + "littlefs: cycle in the metadata-pair list".into(), + )); + } + } + Ok(a) + } + + /// Release every block a file's data occupies. + fn free_data(&mut self, dev: &mut dyn BlockDevice, data: &Struct) -> Result<()> { + let Struct::Ctz { head, size } = data else { + return Ok(()); + }; + let geom = self.geom; + let mut blocks = Vec::new(); + ctz::traverse(dev, &geom, *head, *size, &mut |b| blocks.push(b))?; + let a = self.allocator(dev)?; + for b in blocks { + a.free(b); + } + Ok(()) + } + + // ---- path resolution ------------------------------------------------ + + /// Resolve a path, erroring when it doesn't exist. + fn resolve(&mut self, dev: &mut dyn BlockDevice, path: &Path) -> Result { + self.try_resolve(dev, path)?.ok_or_else(|| { + Error::InvalidArgument(format!("littlefs: no such path {:?}", path.display())) + }) + } + + /// Resolve a path, returning `None` when the final component is absent. + fn try_resolve(&mut self, dev: &mut dyn BlockDevice, path: &Path) -> Result> { + let comps = components(path)?; + let mut dir = self.root; + let mut out = Resolved::Root; + for (i, name) in comps.iter().enumerate() { + let Some((mdir, id)) = self.find_in_dir(dev, dir, name.as_bytes())? else { + return Ok(None); + }; + if i + 1 < comps.len() { + dir = match &mdir.entries[id].data { + Some(Struct::Dir(p)) => *p, + _ => { + return Err(Error::InvalidArgument(format!( + "littlefs: {name:?} is not a directory" + ))); + } + }; + } + out = Resolved::Entry { mdir, id }; + } + Ok(Some(out)) + } + + /// The metadata pair a directory's entries start at. + fn dir_head(&self, r: &Resolved) -> Result<[u32; 2]> { + match r { + Resolved::Root => Ok(self.root), + Resolved::Entry { mdir, id } => match &mdir.entries[*id].data { + Some(Struct::Dir(p)) => Ok(*p), + _ => Err(Error::InvalidArgument( + "littlefs: not a directory".to_string(), + )), + }, + } + } + + /// Resolve `path`'s parent directory to the pair its entries start at. + fn parent_head( + &mut self, + dev: &mut dyn BlockDevice, + path: &Path, + ) -> Result<([u32; 2], String)> { + let comps = components(path)?; + let (name, parents) = comps + .split_last() + .ok_or_else(|| Error::InvalidArgument("littlefs: empty path".into()))?; + let mut dir = self.root; + for p in parents { + let Some((mdir, id)) = self.find_in_dir(dev, dir, p.as_bytes())? else { + return Err(Error::InvalidArgument(format!( + "littlefs: no such directory {p:?}" + ))); + }; + dir = match &mdir.entries[id].data { + Some(Struct::Dir(pair)) => *pair, + _ => { + return Err(Error::InvalidArgument(format!( + "littlefs: {p:?} is not a directory" + ))); + } + }; + } + Ok((dir, (*name).to_string())) + } + + /// Find `name` in the directory whose chain starts at `head`. + fn find_in_dir( + &mut self, + dev: &mut dyn BlockDevice, + head: [u32; 2], + name: &[u8], + ) -> Result> { + for m in self.chain(dev, head)? { + if let Some(id) = m.find(name) { + return Ok(Some((m, id))); + } + } + Ok(None) + } + + /// Every metadata pair of one directory, following its hard tails. + fn chain(&mut self, dev: &mut dyn BlockDevice, head: [u32; 2]) -> Result> { + let mut out = Vec::new(); + let mut pair = Some(head); + while let Some(p) = pair { + let m = self.fetch(dev, p)?; + pair = if m.hard { m.tail } else { None }; + out.push(m); + if out.len() as u32 > self.geom.block_count { + return Err(Error::InvalidImage( + "littlefs: cycle in a directory's metadata chain".into(), + )); + } + } + Ok(out) + } + + /// The metadata pair whose tail points at `pair` — its predecessor on + /// the filesystem-wide threaded list. + fn find_pred(&mut self, dev: &mut dyn BlockDevice, pair: [u32; 2]) -> Result { + let key = MdirCache::key(pair); + let mut next = Some(SUPERBLOCK_PAIR); + let mut hops = 0u32; + while let Some(p) = next { + let m = self.fetch(dev, p)?; + if m.tail.map(MdirCache::key) == Some(key) { + return Ok(m); + } + next = m.tail; + hops += 1; + if hops > self.geom.block_count { + break; + } + } + Err(Error::InvalidImage( + "littlefs: metadata pair is not on the threaded list".into(), + )) + } + + // ---- mutation ------------------------------------------------------- + + /// Insert an entry into a directory, keeping the chain in name order + /// (littlefs sorts directory entries by their raw bytes). + fn insert_entry( + &mut self, + dev: &mut dyn BlockDevice, + head: [u32; 2], + entry: Entry, + ) -> Result<()> { + let mut pair = head; + loop { + let mut m = self.fetch(dev, pair)?; + // The superblock shares the root's pair as id 0 and takes no + // part in the ordering, so entries start after it. + let start = m.entries.iter().take_while(|e| !e.is_file()).count(); + let pos = m.entries[start..] + .iter() + .position(|e| e.name.as_slice() > entry.name.as_slice()) + .map(|p| p + start); + match pos { + Some(p) => { + m.entries.insert(p, entry); + return self.commit(dev, &mut m); + } + None => match (m.hard, m.tail) { + (true, Some(t)) => pair = t, + _ => { + m.entries.push(entry); + return self.commit(dev, &mut m); + } + }, + } + } + } + + /// Shared body of `create_file` / `create_file_streaming`. + fn write_file( + &mut self, + dev: &mut dyn BlockDevice, + path: &Path, + body: &mut dyn Read, + len: u64, + ) -> Result<()> { + let (head, name) = self.parent_head(dev, path)?; + self.check_name(&name)?; + if len > self.file_max as u64 { + return Err(Error::InvalidArgument(format!( + "littlefs: {len} bytes exceeds the volume's {}-byte file limit", + self.file_max + ))); + } + + let existing = self.find_in_dir(dev, head, name.as_bytes())?; + if let Some((m, id)) = &existing + && m.entries[*id].kind == tag::TYPE_DIR as u8 + { + return Err(Error::InvalidArgument(format!( + "littlefs: {name:?} already exists as a directory" + ))); + } + + let data = self.write_data(dev, body, len)?; + match existing { + // Replacing a file keeps its id — only the struct changes. + Some((mut m, id)) => { + if let Some(old) = m.entries[id].data.clone() { + self.free_data(dev, &old)?; + } + m.entries[id].data = Some(data); + self.commit(dev, &mut m) + } + None => self.insert_entry( + dev, + head, + Entry { + kind: tag::TYPE_REG as u8, + name: name.into_bytes(), + data: Some(data), + attrs: Vec::new(), + }, + ), + } + } + + /// Stream `len` bytes of file data into the volume, inlining it when it + /// is small enough to live in the directory's metadata. + fn write_data( + &mut self, + dev: &mut dyn BlockDevice, + body: &mut dyn Read, + len: u64, + ) -> Result { + if len <= self.inline_max as u64 { + let mut buf = vec![0u8; len as usize]; + body.read_exact(&mut buf)?; + return Ok(Struct::Inline(buf)); + } + let geom = self.geom; + let mut src = ctz::ReaderSource { body }; + let alloc = self.allocator(dev)?; + let head = + ctz::write_blocks(dev, &geom, alloc, 0, None, 0, &mut src, len)?.ok_or_else(|| { + Error::InvalidArgument("littlefs: empty skip-list for a non-empty file".into()) + })?; + Ok(Struct::Ctz { + head, + size: len as u32, + }) + } + + /// Create a directory: a fresh metadata pair, threaded onto the + /// filesystem-wide list right after its parent's last pair, plus an + /// entry pointing at it. + fn make_dir(&mut self, dev: &mut dyn BlockDevice, path: &Path) -> Result<()> { + let (head, name) = self.parent_head(dev, path)?; + self.check_name(&name)?; + if let Some((m, id)) = self.find_in_dir(dev, head, name.as_bytes())? { + return if m.entries[id].kind == tag::TYPE_DIR as u8 { + Ok(()) + } else { + Err(Error::InvalidArgument(format!( + "littlefs: {name:?} already exists" + ))) + }; + } + + let mut dir = self.new_pair(dev)?; + // Splice the new pair into the threaded list behind the parent's + // last pair, so a traversal still reaches every metadata block. + let pred_pair = self + .chain(dev, head)? + .last() + .expect("a directory always has at least one pair") + .pair; + let mut pred = self.fetch(dev, pred_pair)?; + dir.tail = pred.tail; + dir.hard = false; + self.commit(dev, &mut dir)?; + pred.tail = Some(dir.pair); + pred.hard = false; + self.commit(dev, &mut pred)?; + + self.insert_entry( + dev, + head, + Entry { + kind: tag::TYPE_DIR as u8, + name: name.into_bytes(), + data: Some(Struct::Dir(dir.pair)), + attrs: Vec::new(), + }, + ) + } + + /// Remove a file or an empty directory. + fn remove_path(&mut self, dev: &mut dyn BlockDevice, path: &Path) -> Result<()> { + let Resolved::Entry { mdir, id } = self.resolve(dev, path)? else { + return Err(Error::InvalidArgument( + "littlefs: cannot remove the root directory".into(), + )); + }; + let entry = mdir.entries[id].clone(); + + if entry.kind == tag::TYPE_DIR as u8 { + let head = match &entry.data { + Some(Struct::Dir(p)) => *p, + _ => { + return Err(Error::InvalidImage( + "littlefs: directory entry without a metadata pair".into(), + )); + } + }; + let chain = self.chain(dev, head)?; + if chain.iter().any(|m| m.entries.iter().any(Entry::is_file)) { + return Err(Error::InvalidArgument(format!( + "littlefs: directory {:?} is not empty", + path.display() + ))); + } + + // Drop the entry first; the predecessor may well be the very + // pair we just rewrote, so it has to be re-read afterwards. + let mut parent = mdir; + parent.entries.remove(id); + self.commit(dev, &mut parent)?; + + let last = chain.last().expect("chain is never empty"); + let mut pred = self.find_pred(dev, head)?; + pred.tail = last.tail; + pred.hard = last.hard; + // Global state lives as a per-pair delta whose XOR across the + // volume is the filesystem's state, so a dropped pair's delta + // has to be carried over rather than lost. + for m in &chain { + if let Some(g) = m.gdelta { + let mut acc = pred.gdelta.unwrap_or([0u8; 12]); + for (a, b) in acc.iter_mut().zip(g.iter()) { + *a ^= *b; + } + pred.gdelta = if acc == [0u8; 12] { None } else { Some(acc) }; + } + } + self.commit(dev, &mut pred)?; + + for m in &chain { + self.cache.remove(m.pair); + let a = self.allocator(dev)?; + a.free(m.pair[0]); + a.free(m.pair[1]); + } + return Ok(()); + } + + if let Some(data) = &entry.data { + self.free_data(dev, data)?; + } + let mut parent = mdir; + parent.entries.remove(id); + self.commit(dev, &mut parent) + } + + /// Reject names littlefs can't store. + fn check_name(&self, name: &str) -> Result<()> { + if name.is_empty() { + return Err(Error::InvalidArgument("littlefs: empty name".into())); + } + if name.len() > self.name_max as usize { + return Err(Error::InvalidArgument(format!( + "littlefs: name {name:?} is longer than the volume's {}-byte limit", + self.name_max + ))); + } + Ok(()) + } + + /// Directory listing shared by `list` and the FUSE-facing helpers. + fn list_dir(&mut self, dev: &mut dyn BlockDevice, path: &Path) -> Result> { + let r = self.resolve(dev, path)?; + let head = self.dir_head(&r)?; + let mut out = Vec::new(); + for m in self.chain(dev, head)? { + for (id, e) in m.entries.iter().enumerate().filter(|(_, e)| e.is_file()) { + out.push(DirEntry { + name: String::from_utf8_lossy(&e.name).into_owned(), + inode: synthetic_inode(m.pair, id, e), + kind: entry_kind(e), + size: entry_size(e), + }); + } + } + Ok(out) + } + + /// Locate a file's contents for reading. + fn file_source(&mut self, dev: &mut dyn BlockDevice, path: &Path) -> Result { + let Resolved::Entry { mdir, id } = self.resolve(dev, path)? else { + return Err(Error::InvalidArgument( + "littlefs: the root is not a file".into(), + )); + }; + let e = &mdir.entries[id]; + if e.kind != tag::TYPE_REG as u8 { + return Err(Error::InvalidArgument(format!( + "littlefs: {:?} is not a regular file", + path.display() + ))); + } + Ok(match &e.data { + Some(Struct::Inline(d)) => rw::Source::Inline(d.clone()), + Some(Struct::Ctz { head, size }) => rw::Source::Ctz { + head: *head, + size: *size, + }, + _ => rw::Source::Inline(Vec::new()), + }) + } +} + +/// littlefs's own default: a file is inlined while it fits in an eighth of +/// a metadata block, bounded by what a single tag can carry. +fn pick_inline_max(geom: &Geom, requested: Option) -> Result { + let ceiling = (tag::MAX_SIZE as u32).min(geom.split_limit() as u32 / 2); + let v = requested.unwrap_or_else(|| (geom.block_size / 8).min(ceiling)); + if v > ceiling { + return Err(Error::InvalidArgument(format!( + "littlefs: inline_max {v} exceeds the {ceiling} bytes a {}-byte block can inline", + geom.block_size + ))); + } + Ok(v) +} + +/// Split a path into its components, rejecting anything that would escape +/// the volume root. +fn components(path: &Path) -> Result> { + let s = path + .to_str() + .ok_or_else(|| Error::InvalidArgument("littlefs: non-UTF-8 path".into()))?; + let mut out: Vec<&str> = Vec::new(); + for c in s.split('/') { + match c { + "" | "." => {} + ".." => { + if out.pop().is_none() { + return Err(Error::InvalidArgument( + "littlefs: path escapes the root".into(), + )); + } + } + other => out.push(other), + } + } + Ok(out) +} + +fn entry_kind(e: &Entry) -> EntryKind { + if e.kind == tag::TYPE_DIR as u8 { + EntryKind::Dir + } else { + EntryKind::Regular + } +} + +fn entry_size(e: &Entry) -> u64 { + match &e.data { + Some(Struct::Inline(d)) => d.len() as u64, + Some(Struct::Ctz { size, .. }) => *size as u64, + _ => 0, + } +} + +/// littlefs has no inode numbers, so we synthesise a stable one: a +/// directory is identified by the first block of its own metadata pair, a +/// file by its id within its parent's pair (ids stop at 0xfe, so eight bits +/// are enough). Callers use these only to tell entries apart — FUSE node +/// ids, and cycle detection in tree walks. +fn synthetic_inode(pair: [u32; 2], id: usize, e: &Entry) -> u32 { + match &e.data { + Some(Struct::Dir(p)) => p[0].max(1), + _ => 0x8000_0000 | (pair[0].wrapping_shl(8) & 0x7fff_ff00) | (id as u32 & 0xff), + } +} + +/// Map a littlefs user-attribute type to its extended-attribute name. +fn xattr_name(kind: u8) -> String { + format!("{XATTR_PREFIX}{kind}") +} + +/// Parse an extended-attribute name back into a littlefs attribute type. +fn xattr_type(name: &str) -> Result { + name.strip_prefix(XATTR_PREFIX) + .and_then(|n| n.parse::().ok()) + .ok_or_else(|| { + Error::Unsupported(format!( + "littlefs: only {XATTR_PREFIX}<0-255> attributes can be stored (got {name:?})" + )) + }) +} + +impl Filesystem for LittleFs { + fn streams_immediately(&self) -> bool { + true + } + + fn create_file( + &mut self, + dev: &mut dyn BlockDevice, + path: &Path, + src: FileSource, + _meta: FileMeta, + ) -> Result<()> { + let (mut reader, len) = src.open()?; + self.write_file(dev, path, &mut reader, len) + } + + fn create_file_streaming( + &mut self, + dev: &mut dyn BlockDevice, + path: &Path, + body: &mut dyn Read, + len: u64, + _meta: FileMeta, + ) -> Result<()> { + self.write_file(dev, path, body, len) + } + + fn create_dir( + &mut self, + dev: &mut dyn BlockDevice, + path: &Path, + _meta: FileMeta, + ) -> Result<()> { + if components(path)?.is_empty() { + return Ok(()); // the root always exists + } + self.make_dir(dev, path) + } + + fn create_symlink( + &mut self, + _dev: &mut dyn BlockDevice, + _path: &Path, + _target: &Path, + _meta: FileMeta, + ) -> Result<()> { + Err(Error::Unsupported( + "littlefs: the format has no symbolic links".into(), + )) + } + + fn create_device( + &mut self, + _dev: &mut dyn BlockDevice, + _path: &Path, + _kind: crate::fs::DeviceKind, + _major: u32, + _minor: u32, + _meta: FileMeta, + ) -> Result<()> { + Err(Error::Unsupported( + "littlefs: the format has no device nodes".into(), + )) + } + + fn remove(&mut self, dev: &mut dyn BlockDevice, path: &Path) -> Result<()> { + self.remove_path(dev, path) + } + + fn list(&mut self, dev: &mut dyn BlockDevice, path: &Path) -> Result> { + self.list_dir(dev, path) + } + + fn read_file<'a>( + &'a mut self, + dev: &'a mut dyn BlockDevice, + path: &Path, + ) -> Result> { + let src = self.file_source(dev, path)?; + Ok(Box::new(rw::FileReader::new(dev, self.geom, src))) + } + + fn open_file_ro<'a>( + &'a mut self, + dev: &'a mut dyn BlockDevice, + path: &Path, + ) -> Result> { + let src = self.file_source(dev, path)?; + Ok(Box::new(rw::FileReader::new(dev, self.geom, src))) + } + + fn open_file_rw<'a>( + &'a mut self, + dev: &'a mut dyn BlockDevice, + path: &Path, + flags: crate::fs::OpenFlags, + meta: Option, + ) -> Result> { + rw::open_rw(self, dev, path, flags, meta) + } + + fn truncate(&mut self, dev: &mut dyn BlockDevice, path: &Path, new_size: u64) -> Result<()> { + rw::truncate(self, dev, path, new_size) + } + + fn rename( + &mut self, + dev: &mut dyn BlockDevice, + old_path: &Path, + new_path: &Path, + ) -> Result<()> { + let Resolved::Entry { mdir, id } = self.resolve(dev, old_path)? else { + return Err(Error::InvalidArgument( + "littlefs: cannot rename the root directory".into(), + )); + }; + let entry = mdir.entries[id].clone(); + let (dst_head, name) = self.parent_head(dev, new_path)?; + self.check_name(&name)?; + if self.find_in_dir(dev, dst_head, name.as_bytes())?.is_some() { + return Err(Error::InvalidArgument(format!( + "littlefs: {:?} already exists", + new_path.display() + ))); + } + + // Drop the old entry first so a name moving within one directory + // doesn't briefly exist twice — the pair is rewritten either way. + let mut src = mdir; + src.entries.remove(id); + self.commit(dev, &mut src)?; + self.insert_entry( + dev, + dst_head, + Entry { + kind: entry.kind, + name: name.into_bytes(), + data: entry.data, + attrs: entry.attrs, + }, + ) + } + + fn getattr(&mut self, dev: &mut dyn BlockDevice, path: &Path) -> Result { + let r = self.resolve(dev, path)?; + let (kind, size, inode) = match &r { + Resolved::Root => (EntryKind::Dir, 0, self.root[0].max(1)), + Resolved::Entry { mdir, id } => { + let e = &mdir.entries[*id]; + ( + entry_kind(e), + entry_size(e), + synthetic_inode(mdir.pair, *id, e), + ) + } + }; + // littlefs stores no permissions, owners or timestamps; these are + // the values a littlefs FUSE mount reports too. + Ok(FileAttrs { + kind, + mode: if kind == EntryKind::Dir { 0o755 } else { 0o644 }, + uid: 0, + gid: 0, + size, + blocks: size.div_ceil(512), + nlink: if kind == EntryKind::Dir { 2 } else { 1 }, + atime: 0, + mtime: 0, + ctime: 0, + rdev: 0, + inode, + }) + } + + fn list_xattrs(&mut self, dev: &mut dyn BlockDevice, path: &Path) -> Result> { + let Resolved::Entry { mdir, id } = self.resolve(dev, path)? else { + return Ok(Vec::new()); + }; + Ok(mdir.entries[id] + .attrs + .iter() + .map(|(k, v)| XattrPair { + name: xattr_name(*k), + value: v.clone(), + }) + .collect()) + } + + fn set_xattr( + &mut self, + dev: &mut dyn BlockDevice, + path: &Path, + name: &str, + value: &[u8], + ) -> Result<()> { + let kind = xattr_type(name)?; + if value.len() > self.attr_max as usize { + return Err(Error::InvalidArgument(format!( + "littlefs: attribute value of {} bytes exceeds the volume's {}-byte limit", + value.len(), + self.attr_max + ))); + } + let Resolved::Entry { mut mdir, id } = self.resolve(dev, path)? else { + return Err(Error::InvalidArgument( + "littlefs: the root has no attributes".into(), + )); + }; + let attrs = &mut mdir.entries[id].attrs; + attrs.retain(|(k, _)| *k != kind); + attrs.push((kind, value.to_vec())); + attrs.sort_by_key(|(k, _)| *k); + self.commit(dev, &mut mdir) + } + + fn remove_xattr(&mut self, dev: &mut dyn BlockDevice, path: &Path, name: &str) -> Result<()> { + let kind = xattr_type(name)?; + let Resolved::Entry { mut mdir, id } = self.resolve(dev, path)? else { + return Err(Error::InvalidArgument( + "littlefs: the root has no attributes".into(), + )); + }; + mdir.entries[id].attrs.retain(|(k, _)| *k != kind); + self.commit(dev, &mut mdir) + } + + fn statfs(&mut self, dev: &mut dyn BlockDevice) -> Result { + let used = self.allocator(dev)?.used() as u64; + let total = self.geom.block_count as u64; + Ok(StatFs { + block_size: self.geom.block_size, + blocks: total, + blocks_free: total.saturating_sub(used), + blocks_avail: total.saturating_sub(used), + inodes: 0, + inodes_free: 0, + name_max: self.name_max, + }) + } + + fn flush(&mut self, dev: &mut dyn BlockDevice) -> Result<()> { + // Every mutation is already committed; this just pushes the block + // device's own buffers out. + dev.sync() + } + + fn mutation_capability(&self) -> MutationCapability { + MutationCapability::Mutable + } +} + +impl crate::fs::FilesystemFactory for LittleFs { + type FormatOpts = LittleFsFormatOpts; + + fn format(dev: &mut dyn BlockDevice, opts: &Self::FormatOpts) -> Result { + LittleFs::format(dev, opts) + } + + fn open(dev: &mut dyn BlockDevice) -> Result { + LittleFs::open(dev) + } + + fn size_plan(opts: &Self::FormatOpts) -> Option> { + Some(Box::new(LittleFsSizePlan::new(opts))) + } +} diff --git a/src/fs/littlefs/rw.rs b/src/fs/littlefs/rw.rs new file mode 100644 index 0000000..fd4d555 --- /dev/null +++ b/src/fs/littlefs/rw.rs @@ -0,0 +1,556 @@ +//! File handles — streaming reads, and in-place writes through real +//! littlefs commits. +//! +//! Reads walk the CTZ skip-list (or copy out of the inline data held in the +//! directory's metadata). Writes are gathered into one contiguous dirty +//! region and applied when the handle is synced, dropped, or the caller +//! seeks somewhere the region can't absorb. +//! +//! Applying a write rebuilds the skip-list from the block containing the +//! first changed byte onwards. That is not an optimisation but the shape of +//! the format: every block points *backwards*, so blocks before the change +//! keep their contents and their pointers, while everything after has to be +//! rewritten anyway — which is exactly the copy-on-write littlefs performs +//! itself. + +use std::io::{self, Read, Seek, SeekFrom, Write}; +use std::path::{Path, PathBuf}; + +use crate::block::BlockDevice; +use crate::fs::{FileHandle, FileMeta, FileReadHandle, OpenFlags}; +use crate::{Error, Result}; + +use super::mdir::{Geom, Struct}; +use super::{LittleFs, Resolved, ctz, entry_size, tag}; + +/// Where a file's bytes currently live. +#[derive(Debug, Clone)] +pub enum Source { + /// Small file, stored in its directory's metadata block. + Inline(Vec), + /// Regular file, stored as a CTZ skip-list. + Ctz { head: u32, size: u32 }, +} + +impl Source { + fn len(&self) -> u64 { + match self { + Source::Inline(d) => d.len() as u64, + Source::Ctz { size, .. } => *size as u64, + } + } +} + +/// Read up to `buf.len()` bytes of `src` at file offset `pos`. Returns 0 at +/// end of file; a short read means the next call continues from the next +/// block. +fn read_source( + dev: &mut dyn BlockDevice, + geom: &Geom, + src: &Source, + pos: u64, + buf: &mut [u8], +) -> Result { + if pos >= src.len() || buf.is_empty() { + return Ok(0); + } + match src { + Source::Inline(d) => { + let start = pos as usize; + let n = buf.len().min(d.len() - start); + buf[..n].copy_from_slice(&d[start..start + n]); + Ok(n) + } + Source::Ctz { head, size } => { + let (block, off) = ctz::find(dev, geom, *head, *size, pos as u32)?; + // A read stops at the end of the block it started in — the next + // call resumes in the next one. + let in_block = (geom.block_size - off) as u64; + let n = (buf.len() as u64).min(in_block).min(*size as u64 - pos) as usize; + dev.read_at(geom.offset(block) + off as u64, &mut buf[..n])?; + Ok(n) + } + } +} + +/// Read-only handle over a file. +pub struct FileReader<'a> { + dev: &'a mut dyn BlockDevice, + geom: Geom, + src: Source, + pos: u64, +} + +impl<'a> FileReader<'a> { + pub(super) fn new(dev: &'a mut dyn BlockDevice, geom: Geom, src: Source) -> Self { + Self { + dev, + geom, + src, + pos: 0, + } + } +} + +impl Read for FileReader<'_> { + fn read(&mut self, buf: &mut [u8]) -> io::Result { + let n = read_source(self.dev, &self.geom, &self.src, self.pos, buf) + .map_err(|e| io::Error::other(e.to_string()))?; + self.pos += n as u64; + Ok(n) + } +} + +impl Seek for FileReader<'_> { + fn seek(&mut self, pos: SeekFrom) -> io::Result { + self.pos = resolve_seek(self.pos, self.src.len(), pos)?; + Ok(self.pos) + } +} + +impl FileReadHandle for FileReader<'_> { + fn len(&self) -> u64 { + self.src.len() + } +} + +fn resolve_seek(cur: u64, len: u64, pos: SeekFrom) -> io::Result { + let new = match pos { + SeekFrom::Start(n) => n as i128, + SeekFrom::End(n) => len as i128 + n as i128, + SeekFrom::Current(n) => cur as i128 + n as i128, + }; + if new < 0 { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "littlefs: seek before start of file", + )); + } + Ok(new as u64) +} + +/// Pending contiguous write. +struct Dirty { + start: u64, + buf: Vec, +} + +/// Read + write handle over a file. +pub struct FileWriter<'a> { + fs: &'a mut LittleFs, + dev: &'a mut dyn BlockDevice, + /// The entry is re-resolved by path on every apply: a commit may split + /// a metadata pair and move the entry to a different one. + path: PathBuf, + size: u64, + pos: u64, + dirty: Option, + /// Committed contents, cached between reads and dropped on every write. + src: Option, +} + +impl FileWriter<'_> { + /// The file's committed contents, resolved on demand. + fn source(&mut self) -> Result { + if let Some(s) = &self.src { + return Ok(s.clone()); + } + let s = self.fs.file_source(self.dev, &self.path)?; + self.src = Some(s.clone()); + Ok(s) + } + + /// Write the pending region into the volume and commit the entry. + fn apply(&mut self) -> Result<()> { + let Some(d) = self.dirty.take() else { + return Ok(()); + }; + self.src = None; + let end = d.start + d.buf.len() as u64; + let old = self.fs.file_source(self.dev, &self.path)?; + let old_size = old.len(); + let new_size = old_size.max(end); + if new_size > self.fs.file_max as u64 { + return Err(Error::InvalidArgument(format!( + "littlefs: {new_size} bytes exceeds the volume's {}-byte file limit", + self.fs.file_max + ))); + } + + let data = if new_size <= self.fs.inline_max as u64 && matches!(old, Source::Inline(_)) { + // Still small enough to live in the metadata block. + let Source::Inline(mut v) = old else { + unreachable!("checked above") + }; + v.resize(new_size as usize, 0); + let s = d.start as usize; + v[s..s + d.buf.len()].copy_from_slice(&d.buf); + Struct::Inline(v) + } else { + self.rebuild(&old, old_size, new_size, d.start, &d.buf)? + }; + + self.store(data)?; + self.size = new_size; + Ok(()) + } + + /// Rewrite the skip-list from the first changed block onwards. + fn rebuild( + &mut self, + old: &Source, + old_size: u64, + new_size: u64, + dirty_start: u64, + dirty: &[u8], + ) -> Result { + let geom = self.fs.geom; + let old_ctz = match old { + Source::Ctz { head, size } => Some((*head, *size)), + Source::Inline(_) => None, + }; + + // Blocks before the first changed byte survive untouched — their + // contents and their back-pointers are still correct. + let (from_index, prev) = match old_ctz { + Some((head, size)) => { + let touch = dirty_start.min(old_size) as u32; + let (k, _) = ctz::index_of(&geom, touch); + let prev = if k == 0 { + None + } else { + let last_byte = ctz::block_start(&geom, k) - 1; + Some(ctz::find(self.dev, &geom, head, size, last_byte)?.0) + }; + (k, prev) + } + // An inline file being outlined is written from scratch. + None => (0, None), + }; + let base = ctz::block_start(&geom, from_index) as u64; + + let mut src = RebuildSource { + geom, + old: old.clone(), + old_size, + dirty_start, + dirty, + }; + let head = { + let alloc = self.fs.allocator(self.dev)?; + ctz::write_blocks( + self.dev, + &geom, + alloc, + from_index, + prev, + base, + &mut src, + new_size - base, + )? + } + .ok_or_else(|| Error::InvalidArgument("littlefs: nothing written to the file".into()))?; + + // Release the blocks the rewrite replaced (everything from + // `from_index` on); earlier blocks are still part of the file. + if let Some((old_head, size)) = old_ctz { + self.free_from(old_head, size, from_index)?; + } + + Ok(Struct::Ctz { + head, + size: new_size as u32, + }) + } + + /// Free the blocks of `head`/`size` whose index is at least `from`. + fn free_from(&mut self, head: u32, size: u32, from: u32) -> Result<()> { + let geom = self.fs.geom; + let (mut index, _) = ctz::index_of(&geom, size - 1); + let mut doomed = Vec::new(); + // `traverse` walks the list head-first, i.e. from the highest index + // down, one index per callback. + ctz::traverse(self.dev, &geom, head, size, &mut |b| { + if index >= from { + doomed.push(b); + } + index = index.saturating_sub(1); + })?; + let alloc = self.fs.allocator(self.dev)?; + for b in doomed { + alloc.free(b); + } + Ok(()) + } + + /// Point the file's entry at `data` and commit its metadata pair. + fn store(&mut self, data: Struct) -> Result<()> { + let Resolved::Entry { mut mdir, id } = self.fs.resolve(self.dev, &self.path)? else { + return Err(Error::InvalidArgument( + "littlefs: file entry vanished".into(), + )); + }; + mdir.entries[id].data = Some(data); + self.fs.commit(self.dev, &mut mdir) + } +} + +/// Feeds a rebuild: the caller's new bytes where they apply, the old +/// contents elsewhere, and zeroes for any gap a write past the end left +/// behind. +struct RebuildSource<'d> { + geom: Geom, + old: Source, + old_size: u64, + dirty_start: u64, + dirty: &'d [u8], +} + +impl ctz::ChunkSource for RebuildSource<'_> { + fn fill(&mut self, dev: &mut dyn BlockDevice, off: u64, buf: &mut [u8]) -> Result<()> { + let dirty_end = self.dirty_start + self.dirty.len() as u64; + let mut done = 0usize; + while done < buf.len() { + let o = off + done as u64; + let want = buf.len() - done; + if o >= self.dirty_start && o < dirty_end { + let s = (o - self.dirty_start) as usize; + let n = want.min(self.dirty.len() - s); + buf[done..done + n].copy_from_slice(&self.dirty[s..s + n]); + done += n; + } else if o < self.old_size && (o >= dirty_end || o < self.dirty_start) { + // Old contents, bounded so we never run past the start of + // the new bytes. + let limit = if o < self.dirty_start { + self.dirty_start.min(self.old_size) - o + } else { + self.old_size - o + }; + let cap = want.min(limit as usize); + let n = read_source(dev, &self.geom, &self.old, o, &mut buf[done..done + cap])?; + if n == 0 { + return Err(Error::InvalidImage( + "littlefs: short read while rewriting a file".into(), + )); + } + done += n; + } else { + // A gap left by a write past the end of the file. + let limit = if o < self.dirty_start { + (self.dirty_start - o) as usize + } else { + want + }; + let n = want.min(limit); + buf[done..done + n].fill(0); + done += n; + } + } + Ok(()) + } +} + +impl Read for FileWriter<'_> { + fn read(&mut self, buf: &mut [u8]) -> io::Result { + self.apply().map_err(|e| io::Error::other(e.to_string()))?; + let src = self.source().map_err(|e| io::Error::other(e.to_string()))?; + let n = read_source(self.dev, &self.fs.geom, &src, self.pos, buf) + .map_err(|e| io::Error::other(e.to_string()))?; + self.pos += n as u64; + Ok(n) + } +} + +impl Write for FileWriter<'_> { + fn write(&mut self, data: &[u8]) -> io::Result { + if data.is_empty() { + return Ok(0); + } + self.src = None; + let absorbs = match &self.dirty { + Some(d) => self.pos >= d.start && self.pos <= d.start + d.buf.len() as u64, + None => false, + }; + if !absorbs { + // The write doesn't touch the pending region — land that one + // first so the two don't have to be tracked separately. + self.apply().map_err(|e| io::Error::other(e.to_string()))?; + self.dirty = Some(Dirty { + start: self.pos, + buf: data.to_vec(), + }); + } else if let Some(d) = &mut self.dirty { + let off = (self.pos - d.start) as usize; + if off + data.len() > d.buf.len() { + d.buf.resize(off + data.len(), 0); + } + d.buf[off..off + data.len()].copy_from_slice(data); + } + self.pos += data.len() as u64; + self.size = self.size.max(self.pos); + Ok(data.len()) + } + + fn flush(&mut self) -> io::Result<()> { + self.apply().map_err(|e| io::Error::other(e.to_string())) + } +} + +impl Seek for FileWriter<'_> { + fn seek(&mut self, pos: SeekFrom) -> io::Result { + self.pos = resolve_seek(self.pos, self.size, pos)?; + Ok(self.pos) + } +} + +impl FileHandle for FileWriter<'_> { + fn len(&self) -> u64 { + self.size + } + + fn set_len(&mut self, new_len: u64) -> Result<()> { + self.apply()?; + if new_len == self.size { + return Ok(()); + } + if new_len > self.size { + // Grow by writing the zero fill through the normal path. + let gap = new_len - self.size; + self.dirty = Some(Dirty { + start: self.size, + buf: vec![0u8; gap as usize], + }); + self.pos = new_len; + return self.apply(); + } + + let old = self.source()?; + self.src = None; + let data = match &old { + Source::Inline(v) => Struct::Inline(v[..new_len as usize].to_vec()), + Source::Ctz { head, size } => { + if new_len <= self.fs.inline_max as u64 { + // littlefs reverts a shrunken file to inline storage. + let mut v = vec![0u8; new_len as usize]; + let mut done = 0usize; + while done < v.len() { + let n = read_source( + self.dev, + &self.fs.geom, + &old, + done as u64, + &mut v[done..], + )?; + if n == 0 { + break; + } + done += n; + } + self.free_from(*head, *size, 0)?; + Struct::Inline(v) + } else { + let geom = self.fs.geom; + // The block holding the last surviving byte becomes the + // new head; everything past it is released. + let (new_head, _) = + ctz::find(self.dev, &geom, *head, *size, new_len as u32 - 1)?; + let (keep, _) = ctz::index_of(&geom, new_len as u32 - 1); + self.free_from(*head, *size, keep + 1)?; + Struct::Ctz { + head: new_head, + size: new_len as u32, + } + } + } + }; + self.store(data)?; + self.size = new_len; + self.pos = self.pos.min(new_len); + Ok(()) + } + + fn sync(&mut self) -> Result<()> { + self.apply() + } +} + +impl Drop for FileWriter<'_> { + fn drop(&mut self) { + if self.dirty.is_some() + && let Err(e) = self.apply() + { + log::warn!("littlefs: dropping a file handle lost pending writes: {e}"); + } + } +} + +/// Open a file for reading and writing, creating it when asked to. +pub(super) fn open_rw<'a>( + fs: &'a mut LittleFs, + dev: &'a mut dyn BlockDevice, + path: &Path, + flags: OpenFlags, + meta: Option, +) -> Result> { + let size = match fs.try_resolve(dev, path)? { + Some(Resolved::Entry { mdir, id }) => { + let e = &mdir.entries[id]; + if e.kind != tag::TYPE_REG as u8 { + return Err(Error::InvalidArgument(format!( + "littlefs: {:?} is not a regular file", + path.display() + ))); + } + entry_size(e) + } + Some(Resolved::Root) => { + return Err(Error::InvalidArgument( + "littlefs: the root is not a file".into(), + )); + } + None => { + if !flags.create { + return Err(Error::InvalidArgument(format!( + "littlefs: no such file {:?}", + path.display() + ))); + } + if meta.is_none() { + return Err(Error::InvalidArgument( + "littlefs: open_file_rw with create needs file metadata".into(), + )); + } + fs.write_file(dev, path, &mut io::empty(), 0)?; + 0 + } + }; + + let mut h = FileWriter { + fs, + dev, + path: path.to_path_buf(), + size, + pos: 0, + dirty: None, + src: None, + }; + if flags.truncate { + h.set_len(0)?; + } + if flags.append { + h.pos = h.size; + } + Ok(Box::new(h)) +} + +/// Resize the file at `path`, the path-flavoured [`FileHandle::set_len`]. +pub(super) fn truncate( + fs: &mut LittleFs, + dev: &mut dyn BlockDevice, + path: &Path, + new_size: u64, +) -> Result<()> { + let mut h = open_rw(fs, dev, path, OpenFlags::default(), None)?; + h.set_len(new_size)?; + h.sync() +} diff --git a/src/fs/littlefs/size_plan.rs b/src/fs/littlefs/size_plan.rs new file mode 100644 index 0000000..04f740f --- /dev/null +++ b/src/fs/littlefs/size_plan.rs @@ -0,0 +1,156 @@ +//! Content-fit sizing: how large an image has to be for a given tree. +//! +//! littlefs's footprint is easy to predict because nothing is reserved up +//! front — a volume is exactly the metadata pairs its directories need plus +//! the blocks its out-of-line files occupy. Each directory costs at least +//! one pair (two blocks) and one more pair per metadata block its entries +//! overflow; each file either rides along inline in that metadata or takes +//! a CTZ skip-list of its own. + +use std::collections::HashMap; + +use crate::fs::{FsSizePlan, split_parent_name}; + +use super::mdir::Geom; +use super::{LittleFsFormatOpts, ctz}; + +/// Accumulates the exact block count a tree needs. +pub struct LittleFsSizePlan { + geom: Geom, + inline_max: u32, + /// Metadata bytes per directory, keyed by path. + dirs: HashMap, + /// Blocks taken by files too large to inline. + data_blocks: u64, +} + +impl LittleFsSizePlan { + /// A plan for a volume formatted with `opts`. + pub fn new(opts: &LittleFsFormatOpts) -> Self { + let geom = Geom { + block_size: opts.block_size, + // Only the block size matters for sizing; the count is what we + // are computing. + block_count: u32::MAX, + prog_size: opts.prog_size.max(1), + fcrc: opts.disk_version >= super::DISK_VERSION_2_1, + }; + let inline_max = + super::pick_inline_max(&geom, opts.inline_max).unwrap_or(opts.block_size / 8); + let mut dirs = HashMap::new(); + // The root pair also carries the superblock entry: its name tag + // plus "littlefs", and the inline-struct tag plus 24 bytes of + // configuration. + dirs.insert("/".to_string(), (4 + 8) + (4 + 24)); + Self { + geom, + inline_max, + dirs, + data_blocks: 0, + } + } + + /// Charge `bytes` of metadata to the directory containing `path`, and + /// make sure that directory is on the books. + fn charge(&mut self, path: &str, bytes: usize) { + let (parent, _) = split_parent_name(path); + *self.dirs.entry(parent.to_string()).or_insert(0) += bytes; + } + + /// Blocks a file of `len` bytes occupies as a CTZ skip-list. + fn data_blocks_for(&self, len: u64) -> u64 { + if len == 0 { + return 0; + } + let last = len.min(u32::MAX as u64) as u32 - 1; + ctz::index_of(&self.geom, last).0 as u64 + 1 + } +} + +impl FsSizePlan for LittleFsSizePlan { + fn add_dir(&mut self, path: &str) { + let (_, name) = split_parent_name(path); + // The entry in the parent: a name tag and an 8-byte dir struct. + self.charge(path, (4 + name.len()) + (4 + 8)); + self.dirs.entry(path.to_string()).or_insert(0); + } + + fn add_file(&mut self, path: &str, len: u64) { + let (_, name) = split_parent_name(path); + let mut bytes = 4 + name.len(); + if len <= self.inline_max as u64 { + bytes += 4 + len as usize; + } else { + bytes += 4 + 8; + self.data_blocks += self.data_blocks_for(len); + } + self.charge(path, bytes); + } + + fn add_symlink(&mut self, _path: &str, _target: &str) { + // littlefs has no symbolic links; the writer refuses them and the + // repack sink skips the entry, so it costs nothing. + } + + fn add_device(&mut self, _path: &str) { + // Likewise for device nodes, FIFOs and sockets. + } + + fn total_size(&self) -> u64 { + let limit = self.geom.split_limit().max(1); + let mut blocks: u64 = 0; + for bytes in self.dirs.values() { + // One metadata pair per block the directory's entries fill. + let pairs = (bytes.div_ceil(limit)).max(1) as u64; + blocks += 2 * pairs; + } + blocks += self.data_blocks; + // Two blocks of slack: littlefs grows the superblock chain by a + // pair when the root has been rewritten enough times, and a volume + // with no free block at all cannot be modified afterwards. + blocks += 2; + blocks.max(4) * self.geom.block_size as u64 + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn plan() -> LittleFsSizePlan { + LittleFsSizePlan::new(&LittleFsFormatOpts::default()) + } + + #[test] + fn empty_tree_is_the_superblock_pair_plus_slack() { + // Root pair (2 blocks) + 2 blocks of slack, at 4 KiB each. + assert_eq!(plan().total_size(), 4 * 4096); + } + + #[test] + fn inline_files_need_no_data_blocks() { + let mut p = plan(); + p.add_file("/small.txt", 16); + assert_eq!(p.total_size(), 4 * 4096); + } + + #[test] + fn large_files_are_charged_their_skip_list() { + let mut p = plan(); + // 4 KiB block: block 0 holds 4096 bytes, block 1 holds 4092. + p.add_file("/big.bin", 4097); + assert_eq!(p.data_blocks, 2); + let mut q = plan(); + q.add_file("/big.bin", 4096); + assert_eq!(q.data_blocks, 1); + } + + #[test] + fn each_directory_costs_a_pair() { + let mut p = plan(); + p.add_dir("/etc"); + p.add_dir("/etc/ssl"); + // Root + two directories = 3 pairs, plus slack. + assert_eq!(p.total_size(), (3 * 2 + 2) * 4096); + } +} diff --git a/src/fs/littlefs/tag.rs b/src/fs/littlefs/tag.rs new file mode 100644 index 0000000..c138cba --- /dev/null +++ b/src/fs/littlefs/tag.rs @@ -0,0 +1,246 @@ +//! littlefs metadata tags — the 32-bit words that describe every piece of +//! metadata on disk, plus the CRC-32 variant commits are checksummed with. +//! +//! A tag packs four fields into its 32 bits (upstream's `SPEC.md`, +//! "Metadata tags", is the reference for the layout): +//! +//! ```text +//! bit 31 valid bit — clear on a real tag, set on unwritten storage +//! bits 30..20 type3: a 3-bit abstract type (type1) then an 8-bit chunk +//! bits 19..10 id: the file this tag belongs to (0x3ff = none) +//! bits 9..0 length of the tag's data (0x3ff = deleted, no data) +//! ``` +//! +//! Tags are the only thing littlefs stores big-endian (the valid bit has to +//! be the first bit of a commit), and each stored word is XORed with the +//! previous tag so a metadata block can be walked in either direction. The +//! first tag of a block is XORed with `0xffffffff`. + +/// Value the running "previous tag" starts at, both when parsing and when +/// building a commit. +pub const PTAG_INIT: u32 = 0xffff_ffff; + +/// `id` value used by tags that belong to the metadata block rather than to +/// any single file (tails, global state, commit CRCs). +pub const ID_NONE: u16 = 0x3ff; + +/// `size` value marking a deleted attribute — such a tag carries no data. +pub const SIZE_DELETED: u16 = 0x3ff; + +/// Largest payload a single tag can carry (the 10-bit size field, minus the +/// reserved "deleted" value). +pub const MAX_SIZE: usize = 0x3fe; + +// type3 values. The upper 3 bits are the abstract type (type1), the lower +// 8 the chunk field. +/// Name tag; the chunk field carries the file type (`TYPE_REG` etc.). +pub const TYPE_NAME: u16 = 0x000; +/// Regular file. +pub const TYPE_REG: u16 = 0x001; +/// Directory. +pub const TYPE_DIR: u16 = 0x002; +/// Superblock entry — the name tag whose data is the magic `"littlefs"`. +pub const TYPE_SUPERBLOCK: u16 = 0x0ff; +/// Struct tag: directory (data is the 8-byte metadata pair). +pub const TYPE_DIRSTRUCT: u16 = 0x200; +/// Struct tag: inline data (data is the file contents). +pub const TYPE_INLINESTRUCT: u16 = 0x201; +/// Struct tag: CTZ skip-list (data is head block + file size). +pub const TYPE_CTZSTRUCT: u16 = 0x202; +/// User attribute; the chunk field is the caller-defined attribute type. +pub const TYPE_USERATTR: u16 = 0x300; +/// Splice: create a file id, shifting later ids up. +pub const TYPE_CREATE: u16 = 0x401; +/// Splice: delete a file id, shifting later ids down. +pub const TYPE_DELETE: u16 = 0x4ff; +/// Commit CRC. The low chunk bit selects the valid-bit state the *next* +/// commit's tags must have. +pub const TYPE_CCRC: u16 = 0x500; +/// Forward CRC (lfs2.1): checksum of the erased bytes following this commit. +pub const TYPE_FCRC: u16 = 0x5ff; +/// Soft tail — next metadata pair in the filesystem-wide threaded list. +pub const TYPE_SOFTTAIL: u16 = 0x600; +/// Hard tail — next metadata pair of *this* directory. +pub const TYPE_HARDTAIL: u16 = 0x601; +/// Global-state delta (move state). +pub const TYPE_MOVESTATE: u16 = 0x7ff; + +// type1 values, as produced by [`Tag::type1`]. +/// type1 of name tags. +pub const T1_NAME: u16 = 0x000; +/// type1 of struct tags. +pub const T1_STRUCT: u16 = 0x200; +/// type1 of user-attribute tags. +pub const T1_USERATTR: u16 = 0x300; +/// type1 of splice (create / delete) tags. +pub const T1_SPLICE: u16 = 0x400; +/// type1 of CRC tags (both commit CRCs and FCRCs). +pub const T1_CRC: u16 = 0x500; +/// type1 of tail tags. +pub const T1_TAIL: u16 = 0x600; +/// type1 of global-state tags. +pub const T1_GSTATE: u16 = 0x700; + +/// A decoded metadata tag. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Tag(pub u32); + +impl Tag { + /// Build a tag from its three fields. The valid bit is left clear, which + /// is what both the parser and the commit builder expect. + pub fn new(type3: u16, id: u16, size: u16) -> Self { + Self(((type3 as u32) << 20) | ((id as u32) << 10) | size as u32) + } + + /// A tag is valid when its high bit is clear. An unwritten (or + /// power-cut) region decodes to a tag with the bit set, which is how + /// littlefs finds the end of a metadata log. + pub fn is_valid(self) -> bool { + self.0 & 0x8000_0000 == 0 + } + + /// 3-bit abstract type, shifted into the same position `type3` uses so + /// the two can be compared against the `T1_*` constants directly. + pub fn type1(self) -> u16 { + ((self.0 & 0x7000_0000) >> 20) as u16 + } + + /// type1 plus the top bit of the chunk field. Commit CRCs are matched on + /// this so that FCRC (`0x5ff`) is *not* mistaken for a commit CRC + /// (`0x50x`) — the distinction lfs2.1 relies on. + pub fn type2(self) -> u16 { + ((self.0 & 0x7800_0000) >> 20) as u16 + } + + /// Full 11-bit type. + pub fn type3(self) -> u16 { + ((self.0 & 0x7ff0_0000) >> 20) as u16 + } + + /// 8-bit chunk field (file type for names, attribute type for user + /// attributes, tail flavour for tails, …). + pub fn chunk(self) -> u8 { + ((self.0 & 0x0ff0_0000) >> 20) as u8 + } + + /// File id this tag belongs to, or [`ID_NONE`] for block-level tags. + pub fn id(self) -> u16 { + ((self.0 & 0x000f_fc00) >> 10) as u16 + } + + /// Length of the tag's data in bytes (meaningless when [`Self::is_delete`]). + pub fn size(self) -> u16 { + (self.0 & 0x0000_03ff) as u16 + } + + /// Whether this tag marks the attribute deleted (size field all ones). + pub fn is_delete(self) -> bool { + self.size() == SIZE_DELETED + } + + /// Total on-disk size of the tag: the 4-byte word plus its data. A + /// deleted tag carries no data. + pub fn dsize(self) -> usize { + 4 + if self.is_delete() { + 0 + } else { + self.size() as usize + } + } +} + +/// littlefs's CRC-32: polynomial `0x04c11db7`, initialised with +/// `0xffffffff`, and — unlike the usual zlib flavour — **no final XOR**, so +/// the running state can be fed straight back in for the next chunk. +/// +/// `crc32fast` computes the finalised (XOR-ed) form, so we un-XOR on the way +/// in and re-XOR on the way out. The `crc_matches_reference` test pins this +/// against a commit CRC taken from an image written by the C implementation. +pub fn crc(state: u32, data: &[u8]) -> u32 { + let mut h = crc32fast::Hasher::new_with_initial(state ^ 0xffff_ffff); + h.update(data); + h.finalize() ^ 0xffff_ffff +} + +/// Read a big-endian tag word. +pub fn be32(b: &[u8]) -> u32 { + u32::from_be_bytes([b[0], b[1], b[2], b[3]]) +} + +/// Read a little-endian word (everything in littlefs except tags). +pub fn le32(b: &[u8]) -> u32 { + u32::from_le_bytes([b[0], b[1], b[2], b[3]]) +} + +/// Sequence comparison of two revision counts, tolerant of wraparound: +/// `true` when `a` is newer than `b`. +pub fn rev_newer(a: u32, b: u32) -> bool { + (a.wrapping_sub(b)) as i32 > 0 +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn tag_fields_round_trip() { + let t = Tag::new(TYPE_INLINESTRUCT, 3, 24); + assert_eq!(t.type3(), TYPE_INLINESTRUCT); + assert_eq!(t.type1(), T1_STRUCT); + assert_eq!(t.chunk(), 0x01); + assert_eq!(t.id(), 3); + assert_eq!(t.size(), 24); + assert_eq!(t.dsize(), 28); + assert!(t.is_valid()); + assert!(!t.is_delete()); + } + + #[test] + fn deleted_tag_carries_no_data() { + let t = Tag::new(TYPE_USERATTR | 0x42, 1, SIZE_DELETED); + assert!(t.is_delete()); + assert_eq!(t.dsize(), 4); + } + + #[test] + fn fcrc_is_not_a_commit_crc() { + // lfs2.1 relies on type2 (not type1) to tell an FCRC apart from the + // commit CRC that ends every commit — get this wrong and every + // fetch of a modern image stops at the first FCRC. + assert_eq!(Tag::new(TYPE_CCRC, ID_NONE, 4).type2(), 0x500); + assert_ne!(Tag::new(TYPE_FCRC, ID_NONE, 8).type2(), 0x500); + assert_eq!(Tag::new(TYPE_FCRC, ID_NONE, 8).type1(), T1_CRC); + } + + #[test] + fn crc_matches_reference() { + // The first commit of a littlefs image formatted by the C library: + // revision count, superblock name tag + "littlefs", inline-struct + // tag + config, FCRC tag, and the commit-CRC tag itself. The stored + // CRC that follows those bytes is 0xa52fadb2. + let mut b = Vec::new(); + b.extend_from_slice(&0u32.to_le_bytes()); // rev + b.extend_from_slice(&[0xf0, 0x0f, 0xff, 0xf7]); // superblock name tag + b.extend_from_slice(b"littlefs"); + b.extend_from_slice(&[0x2f, 0xe0, 0x00, 0x10]); // inline-struct tag + b.extend_from_slice(&[ + 0x01, 0x00, 0x02, 0x00, // version 2.1 + 0x00, 0x10, 0x00, 0x00, // block size 4096 + 0x20, 0x00, 0x00, 0x00, // block count 32 + 0xff, 0x00, 0x00, 0x00, // name max + 0xff, 0xff, 0xff, 0x7f, // file max + 0xfe, 0x03, 0x00, 0x00, // attr max + ]); + b.extend_from_slice(&[0x7f, 0xef, 0xfc, 0x10]); // fcrc tag + b.extend_from_slice(&[0x00, 0x01, 0x00, 0x00, 0xde, 0x57, 0x57, 0x01]); + b.extend_from_slice(&[0x0f, 0xf0, 0x00, 0xcc]); // ccrc tag + assert_eq!(crc(PTAG_INIT, &b), 0xa52f_adb2); + } + + #[test] + fn revision_compare_handles_wraparound() { + assert!(rev_newer(2, 1)); + assert!(!rev_newer(1, 2)); + assert!(rev_newer(0, u32::MAX)); + } +} diff --git a/src/fs/littlefs/tests.rs b/src/fs/littlefs/tests.rs new file mode 100644 index 0000000..c6d38b3 --- /dev/null +++ b/src/fs/littlefs/tests.rs @@ -0,0 +1,389 @@ +//! In-tree tests for the littlefs backend: format, mutate, re-open, and +//! read back through the same code paths the CLI drives. + +use std::io::{Read, Seek, SeekFrom, Write}; +use std::path::Path; + +use crate::block::{BlockDevice, MemoryBackend}; +use crate::fs::{EntryKind, FileMeta, FileSource, Filesystem, OpenFlags}; + +use super::*; + +/// A 256 KiB volume with 4 KiB blocks — 64 blocks, the smallest geometry +/// that still exercises directory splits and multi-block files. +fn fresh(size: u64) -> (MemoryBackend, LittleFs) { + let mut dev = MemoryBackend::new(size); + let fs = LittleFs::format(&mut dev, &LittleFsFormatOpts::default()).unwrap(); + (dev, fs) +} + +fn write_file(fs: &mut LittleFs, dev: &mut dyn BlockDevice, path: &str, data: &[u8]) { + fs.create_file( + dev, + Path::new(path), + FileSource::Reader { + reader: Box::new(std::io::Cursor::new(data.to_vec())), + len: data.len() as u64, + }, + FileMeta::default(), + ) + .unwrap(); +} + +fn read_file(fs: &mut LittleFs, dev: &mut dyn BlockDevice, path: &str) -> Vec { + let mut out = Vec::new(); + fs.read_file(dev, Path::new(path)) + .unwrap() + .read_to_end(&mut out) + .unwrap(); + out +} + +fn names(fs: &mut LittleFs, dev: &mut dyn BlockDevice, path: &str) -> Vec { + fs.list(dev, Path::new(path)) + .unwrap() + .into_iter() + .map(|e| e.name) + .collect() +} + +#[test] +fn format_writes_a_superblock_at_the_documented_offset() { + let (mut dev, fs) = fresh(256 * 1024); + let mut head = [0u8; 16]; + dev.read_at(0, &mut head).unwrap(); + assert_eq!(&head[8..16], b"littlefs"); + assert_eq!(fs.geometry(), (4096, 64)); + assert_eq!(fs.version(), (2, 1)); + // Both halves of the root pair are valid commits after a format, as + // lfs_format also guarantees. + let mut second = [0u8; 16]; + dev.read_at(4096, &mut second).unwrap(); + assert_eq!(&second[8..16], b"littlefs"); +} + +#[test] +fn inline_and_outlined_files_round_trip() { + let (mut dev, mut fs) = fresh(512 * 1024); + let small = b"hello littlefs\n".to_vec(); + let big: Vec = (0..40_000u32).map(|i| (i % 251) as u8).collect(); + write_file(&mut fs, &mut dev, "/small.txt", &small); + write_file(&mut fs, &mut dev, "/big.bin", &big); + + assert_eq!(read_file(&mut fs, &mut dev, "/small.txt"), small); + assert_eq!(read_file(&mut fs, &mut dev, "/big.bin"), big); + + // The small one stays in metadata, the large one gets a skip-list. + let entries = fs.list(&mut dev, Path::new("/")).unwrap(); + let big_entry = entries.iter().find(|e| e.name == "big.bin").unwrap(); + assert_eq!(big_entry.size, big.len() as u64); + assert!(small.len() as u32 <= fs.inline_max()); +} + +#[test] +fn reopening_an_image_sees_everything() { + let mut dev = MemoryBackend::new(512 * 1024); + { + let mut fs = LittleFs::format(&mut dev, &LittleFsFormatOpts::default()).unwrap(); + fs.create_dir(&mut dev, Path::new("/etc"), FileMeta::default()) + .unwrap(); + write_file(&mut fs, &mut dev, "/etc/motd", b"be excellent\n"); + let payload: Vec = (0..30_000u32).map(|i| i as u8).collect(); + write_file(&mut fs, &mut dev, "/etc/blob", &payload); + fs.flush(&mut dev).unwrap(); + } + let mut fs = LittleFs::open(&mut dev).unwrap(); + assert_eq!(names(&mut fs, &mut dev, "/"), vec!["etc"]); + assert_eq!(names(&mut fs, &mut dev, "/etc"), vec!["blob", "motd"]); + assert_eq!(read_file(&mut fs, &mut dev, "/etc/motd"), b"be excellent\n"); + assert_eq!(read_file(&mut fs, &mut dev, "/etc/blob").len(), 30_000); + + // And it is still writable after the round trip. + write_file(&mut fs, &mut dev, "/etc/extra", b"more"); + assert_eq!( + names(&mut fs, &mut dev, "/etc"), + vec!["blob", "extra", "motd"] + ); +} + +#[test] +fn directory_entries_are_kept_in_name_order() { + let (mut dev, mut fs) = fresh(512 * 1024); + for name in ["zulu", "alpha", "mike", "bravo"] { + write_file(&mut fs, &mut dev, &format!("/{name}"), name.as_bytes()); + } + assert_eq!( + names(&mut fs, &mut dev, "/"), + vec!["alpha", "bravo", "mike", "zulu"] + ); +} + +#[test] +fn removing_reclaims_blocks() { + let (mut dev, mut fs) = fresh(512 * 1024); + let before = fs.used_blocks(&mut dev).unwrap(); + let payload: Vec = vec![7u8; 60_000]; + write_file(&mut fs, &mut dev, "/blob", &payload); + let during = fs.used_blocks(&mut dev).unwrap(); + assert!(during > before + 10, "{during} vs {before}"); + + fs.remove(&mut dev, Path::new("/blob")).unwrap(); + assert_eq!(fs.used_blocks(&mut dev).unwrap(), before); + assert!(fs.list(&mut dev, Path::new("/")).unwrap().is_empty()); +} + +#[test] +fn directories_must_be_empty_to_be_removed() { + let (mut dev, mut fs) = fresh(512 * 1024); + fs.create_dir(&mut dev, Path::new("/d"), FileMeta::default()) + .unwrap(); + write_file(&mut fs, &mut dev, "/d/f", b"x"); + assert!(fs.remove(&mut dev, Path::new("/d")).is_err()); + fs.remove(&mut dev, Path::new("/d/f")).unwrap(); + let used_before = fs.used_blocks(&mut dev).unwrap(); + fs.remove(&mut dev, Path::new("/d")).unwrap(); + // The directory's own metadata pair comes back to the free pool. + assert_eq!(fs.used_blocks(&mut dev).unwrap(), used_before - 2); + assert!(fs.list(&mut dev, Path::new("/")).unwrap().is_empty()); +} + +#[test] +fn many_entries_split_the_directory_across_pairs() { + let (mut dev, mut fs) = fresh(2 * 1024 * 1024); + for i in 0..200 { + write_file( + &mut fs, + &mut dev, + &format!("/file{i:04}"), + format!("contents of {i}").as_bytes(), + ); + } + let listed = names(&mut fs, &mut dev, "/"); + assert_eq!(listed.len(), 200); + // Still sorted, and still readable, across the split. + let mut sorted = listed.clone(); + sorted.sort(); + assert_eq!(listed, sorted); + assert_eq!( + read_file(&mut fs, &mut dev, "/file0137"), + b"contents of 137" + ); + + // A re-opened image agrees. + fs.flush(&mut dev).unwrap(); + let mut fs = LittleFs::open(&mut dev).unwrap(); + assert_eq!(names(&mut fs, &mut dev, "/").len(), 200); +} + +#[test] +fn in_place_writes_patch_a_file() { + let (mut dev, mut fs) = fresh(1024 * 1024); + let payload: Vec = (0..50_000u32).map(|i| (i % 97) as u8).collect(); + write_file(&mut fs, &mut dev, "/data.bin", &payload); + + { + let mut h = fs + .open_file_rw(&mut dev, Path::new("/data.bin"), OpenFlags::default(), None) + .unwrap(); + h.seek(SeekFrom::Start(10_000)).unwrap(); + h.write_all(b"PATCHED").unwrap(); + h.sync().unwrap(); + assert_eq!(h.len(), 50_000); + } + + let mut expect = payload.clone(); + expect[10_000..10_007].copy_from_slice(b"PATCHED"); + assert_eq!(read_file(&mut fs, &mut dev, "/data.bin"), expect); +} + +#[test] +fn appending_past_the_end_extends_the_file() { + let (mut dev, mut fs) = fresh(1024 * 1024); + write_file(&mut fs, &mut dev, "/log", b"start"); + { + let mut h = fs + .open_file_rw( + &mut dev, + Path::new("/log"), + OpenFlags { + append: true, + ..OpenFlags::default() + }, + None, + ) + .unwrap(); + // Push it well past the inline limit so it has to be outlined. + h.write_all(&vec![b'z'; 9_000]).unwrap(); + h.sync().unwrap(); + } + let got = read_file(&mut fs, &mut dev, "/log"); + assert_eq!(got.len(), 9_005); + assert_eq!(&got[..5], b"start"); + assert!(got[5..].iter().all(|&b| b == b'z')); +} + +#[test] +fn truncate_shrinks_and_grows() { + let (mut dev, mut fs) = fresh(1024 * 1024); + let payload: Vec = (0..30_000u32).map(|i| i as u8).collect(); + write_file(&mut fs, &mut dev, "/f", &payload); + + fs.truncate(&mut dev, Path::new("/f"), 20_000).unwrap(); + let got = read_file(&mut fs, &mut dev, "/f"); + assert_eq!(got, payload[..20_000]); + + fs.truncate(&mut dev, Path::new("/f"), 25_000).unwrap(); + let got = read_file(&mut fs, &mut dev, "/f"); + assert_eq!(got.len(), 25_000); + assert_eq!(&got[..20_000], &payload[..20_000]); + assert!(got[20_000..].iter().all(|&b| b == 0)); + + // Shrinking below the inline limit puts the file back in metadata. + fs.truncate(&mut dev, Path::new("/f"), 8).unwrap(); + assert_eq!(read_file(&mut fs, &mut dev, "/f"), payload[..8]); +} + +#[test] +fn rename_moves_between_directories() { + let (mut dev, mut fs) = fresh(512 * 1024); + fs.create_dir(&mut dev, Path::new("/a"), FileMeta::default()) + .unwrap(); + fs.create_dir(&mut dev, Path::new("/b"), FileMeta::default()) + .unwrap(); + write_file(&mut fs, &mut dev, "/a/one", b"payload"); + fs.rename(&mut dev, Path::new("/a/one"), Path::new("/b/two")) + .unwrap(); + assert!(names(&mut fs, &mut dev, "/a").is_empty()); + assert_eq!(names(&mut fs, &mut dev, "/b"), vec!["two"]); + assert_eq!(read_file(&mut fs, &mut dev, "/b/two"), b"payload"); +} + +#[test] +fn user_attributes_surface_as_xattrs() { + let (mut dev, mut fs) = fresh(512 * 1024); + write_file(&mut fs, &mut dev, "/f", b"x"); + fs.set_xattr(&mut dev, Path::new("/f"), "user.littlefs.42", b"meta") + .unwrap(); + let attrs = fs.list_xattrs(&mut dev, Path::new("/f")).unwrap(); + assert_eq!(attrs.len(), 1); + assert_eq!(attrs[0].name, "user.littlefs.42"); + assert_eq!(attrs[0].value, b"meta"); + + // Attributes survive a re-open, and non-littlefs names are refused. + fs.flush(&mut dev).unwrap(); + let mut fs = LittleFs::open(&mut dev).unwrap(); + assert_eq!(fs.list_xattrs(&mut dev, Path::new("/f")).unwrap().len(), 1); + assert!( + fs.set_xattr(&mut dev, Path::new("/f"), "user.something", b"v") + .is_err() + ); + + fs.remove_xattr(&mut dev, Path::new("/f"), "user.littlefs.42") + .unwrap(); + assert!( + fs.list_xattrs(&mut dev, Path::new("/f")) + .unwrap() + .is_empty() + ); +} + +#[test] +fn nested_directories_and_getattr() { + let (mut dev, mut fs) = fresh(512 * 1024); + for d in ["/usr", "/usr/local", "/usr/local/share"] { + fs.create_dir(&mut dev, Path::new(d), FileMeta::default()) + .unwrap(); + } + write_file(&mut fs, &mut dev, "/usr/local/share/greeting", b"hi"); + let a = fs + .getattr(&mut dev, Path::new("/usr/local/share/greeting")) + .unwrap(); + assert_eq!(a.kind, EntryKind::Regular); + assert_eq!(a.size, 2); + let d = fs.getattr(&mut dev, Path::new("/usr/local")).unwrap(); + assert_eq!(d.kind, EntryKind::Dir); + assert_eq!(d.mode, 0o755); + assert_eq!(fs.total_file_bytes(&mut dev).unwrap(), 2); +} + +#[test] +fn symlinks_and_devices_are_refused_cleanly() { + let (mut dev, mut fs) = fresh(256 * 1024); + let err = fs + .create_symlink( + &mut dev, + Path::new("/link"), + Path::new("/target"), + FileMeta::default(), + ) + .unwrap_err(); + assert!(matches!(err, crate::Error::Unsupported(_))); + let err = fs + .create_device( + &mut dev, + Path::new("/null"), + crate::fs::DeviceKind::Char, + 1, + 3, + FileMeta::default(), + ) + .unwrap_err(); + assert!(matches!(err, crate::Error::Unsupported(_))); +} + +#[test] +fn a_full_volume_reports_out_of_space() { + // 16 blocks of 4 KiB: enough to format, nowhere near enough for 200 KiB. + let mut dev = MemoryBackend::new(64 * 1024); + let mut fs = LittleFs::format(&mut dev, &LittleFsFormatOpts::default()).unwrap(); + let err = fs + .create_file( + &mut dev, + Path::new("/toobig"), + FileSource::Zero(200 * 1024), + FileMeta::default(), + ) + .unwrap_err(); + assert!( + err.to_string().contains("no free blocks"), + "unexpected error: {err}" + ); +} + +#[test] +fn version_2_0_images_omit_forward_crcs() { + let mut dev = MemoryBackend::new(256 * 1024); + let opts = LittleFsFormatOpts { + disk_version: DISK_VERSION_2_0, + ..LittleFsFormatOpts::default() + }; + let mut fs = LittleFs::format(&mut dev, &opts).unwrap(); + write_file(&mut fs, &mut dev, "/f", b"data"); + assert_eq!(fs.version(), (2, 0)); + + // A pre-lfs2.1 reader mistakes an FCRC tag for a commit CRC, so a 2.0 + // image must not contain one anywhere. + let mut block = vec![0u8; 4096]; + dev.read_at(0, &mut block).unwrap(); + let mut ptag = tag::PTAG_INIT; + let mut off = 0usize; + loop { + off += tag::Tag(ptag).dsize(); + if off + 4 > block.len() { + break; + } + let t = tag::Tag(tag::be32(&block[off..off + 4]) ^ ptag); + if !t.is_valid() { + break; + } + assert_ne!(t.type3(), tag::TYPE_FCRC, "2.0 image carries an FCRC tag"); + ptag = t.0; + if t.type2() == tag::TYPE_CCRC { + ptag ^= ((t.chunk() & 1) as u32) << 31; + } + } + + let mut fs = LittleFs::open(&mut dev).unwrap(); + assert_eq!(fs.version(), (2, 0)); + assert_eq!(read_file(&mut fs, &mut dev, "/f"), b"data"); +} diff --git a/src/fs/mod.rs b/src/fs/mod.rs index d693790..7e4a498 100644 --- a/src/fs/mod.rs +++ b/src/fs/mod.rs @@ -31,6 +31,7 @@ pub mod grf; pub mod hfs; pub mod hfs_plus; pub mod iso9660; +pub mod littlefs; pub mod ntfs; pub mod ramfs; pub mod rootdevs; diff --git a/src/inspect.rs b/src/inspect.rs index 1ce0536..c157be0 100644 --- a/src/inspect.rs +++ b/src/inspect.rs @@ -30,6 +30,7 @@ use crate::fs::f2fs::F2fs; use crate::fs::fat::Fat32; use crate::fs::hfs::Hfs; use crate::fs::hfs_plus::HfsPlus; +use crate::fs::littlefs::LittleFs; use crate::fs::ntfs::Ntfs; use crate::fs::ramfs::Ramfs; use crate::fs::squashfs::Squashfs; @@ -65,6 +66,8 @@ pub enum FsKind { Hfs, /// Amiga OFS/FFS (AFFS) — read-only (write lands in later phases). Affs, + /// littlefs — read + write, including in-place edits. + LittleFs, /// APFS — read-only, single-leaf-tree case only. Apfs, /// NTFS — read + write (MFT, attributes, `$DATA` + ADS, indexes, @@ -156,6 +159,12 @@ pub fn detect_fs(dev: &mut dyn BlockDevice) -> Result { return Ok(FsKind::Grf); } + // littlefs: the superblock entry is always the first tag of block 0's + // first commit, which puts the magic string at exactly offset 8. + if &bs[8..16] == b"littlefs" { + return Ok(FsKind::LittleFs); + } + // Amiga OFS/FFS: boot block "DOS" + a flag byte 0..=7 at offset 0. // Specific enough to not shadow MBR/boot sectors (which don't begin // with "DOS"); the flag byte's high bits being zero rules out ASCII. @@ -296,6 +305,8 @@ pub enum AnyFs { Hfs(Box), /// Amiga OFS/FFS (AFFS) — read-only. Affs(Box), + /// littlefs — read + write (metadata pairs + CTZ skip-lists). + LittleFs(Box), /// APFS — read-only; single-leaf trees only. Apfs(Box), /// NTFS — read + write (MFT, attributes, `$DATA` + ADS, indexes). @@ -351,6 +362,7 @@ impl AnyFs { FsKind::HfsPlus => Ok(Self::HfsPlus(Box::new(HfsPlus::open(dev)?))), FsKind::Hfs => Ok(Self::Hfs(Box::new(Hfs::open(dev)?))), FsKind::Affs => Ok(Self::Affs(Box::new(Affs::open(dev)?))), + FsKind::LittleFs => Ok(Self::LittleFs(Box::new(LittleFs::open(dev)?))), FsKind::Apfs => Ok(Self::Apfs(Box::new(Apfs::open(dev)?))), FsKind::Ntfs => Ok(Self::Ntfs(Box::new(Ntfs::open(dev)?))), FsKind::F2fs => Ok(Self::F2fs(Box::new(F2fs::open(dev)?))), @@ -447,6 +459,7 @@ impl AnyFs { Self::HfsPlus(_) => FsKind::HfsPlus, Self::Hfs(_) => FsKind::Hfs, Self::Affs(_) => FsKind::Affs, + Self::LittleFs(_) => FsKind::LittleFs, Self::Apfs(_) => FsKind::Apfs, Self::Ntfs(_) => FsKind::Ntfs, Self::F2fs(_) => FsKind::F2fs, @@ -474,6 +487,10 @@ impl AnyFs { Self::HfsPlus(hfs) => hfs.list_path(dev, path), Self::Hfs(hfs) => hfs.list_path(path), Self::Affs(affs) => affs.list_path(path), + Self::LittleFs(lfs) => { + use crate::fs::Filesystem; + lfs.list(dev, std::path::Path::new(path)) + } Self::Apfs(apfs) => apfs.list_path(dev, path), Self::Ntfs(ntfs) => ntfs.list_path(dev, path), Self::F2fs(f2) => f2.list_path(dev, path), @@ -580,6 +597,11 @@ impl AnyFs { let mut r = affs.open_file_reader(dev, path)?; pump(&mut r, out, &mut buf) } + Self::LittleFs(lfs) => { + use crate::fs::Filesystem; + let mut r = lfs.read_file(dev, std::path::Path::new(path))?; + pump(&mut r, out, &mut buf) + } Self::HfsPlus(hfs) => { let mut r = hfs.open_file_reader(dev, path)?; pump(&mut r, out, &mut buf) @@ -653,6 +675,10 @@ impl AnyFs { Self::HfsPlus(hfs) => Ok(Box::new(hfs.open_file_reader(dev, path)?)), Self::Hfs(hfs) => Ok(Box::new(hfs.open_file_reader(dev, path)?)), Self::Affs(affs) => Ok(Box::new(affs.open_file_reader(dev, path)?)), + Self::LittleFs(lfs) => { + use crate::fs::Filesystem; + lfs.read_file(dev, std::path::Path::new(path)) + } Self::Apfs(apfs) => Ok(Box::new(apfs.open_file_reader(dev, path)?)), Self::Ntfs(ntfs) => Ok(Box::new(ntfs.open_file_reader(dev, path)?)), Self::F2fs(f2) => Ok(Box::new(f2.open_file_reader(dev, path)?)), @@ -793,6 +819,7 @@ impl AnyFs { Self::HfsPlus(f) => crate::fs::Filesystem::access_mode(f.as_ref()), Self::Hfs(f) => crate::fs::Filesystem::access_mode(f.as_ref()), Self::Affs(f) => crate::fs::Filesystem::access_mode(f.as_ref()), + Self::LittleFs(l) => crate::fs::Filesystem::access_mode(l.as_ref()), Self::Ntfs(f) => crate::fs::Filesystem::access_mode(f.as_ref()), Self::F2fs(f) => crate::fs::Filesystem::access_mode(f.as_ref()), Self::Squashfs(f) => crate::fs::Filesystem::access_mode(f.as_ref()), @@ -814,6 +841,7 @@ impl AnyFs { Self::HfsPlus(h) => crate::fs::Filesystem::mutation_capability(h.as_ref()), Self::Hfs(h) => crate::fs::Filesystem::mutation_capability(h.as_ref()), Self::Affs(a) => crate::fs::Filesystem::mutation_capability(a.as_ref()), + Self::LittleFs(l) => crate::fs::Filesystem::mutation_capability(l.as_ref()), Self::Ntfs(n) => crate::fs::Filesystem::mutation_capability(n.as_ref()), Self::F2fs(fs2) => crate::fs::Filesystem::mutation_capability(fs2.as_ref()), Self::Squashfs(sq) => crate::fs::Filesystem::mutation_capability(sq.as_ref()), @@ -842,6 +870,7 @@ impl AnyFs { Self::HfsPlus(h) => crate::fs::Filesystem::clone_capability(h.as_ref()), Self::Hfs(h) => crate::fs::Filesystem::clone_capability(h.as_ref()), Self::Affs(a) => crate::fs::Filesystem::clone_capability(a.as_ref()), + Self::LittleFs(l) => crate::fs::Filesystem::clone_capability(l.as_ref()), Self::Ntfs(n) => crate::fs::Filesystem::clone_capability(n.as_ref()), Self::F2fs(fs2) => crate::fs::Filesystem::clone_capability(fs2.as_ref()), Self::Squashfs(sq) => crate::fs::Filesystem::clone_capability(sq.as_ref()), @@ -924,6 +953,7 @@ impl AnyFs { Self::HfsPlus(h) => f(h.as_mut()), Self::Hfs(h) => f(h.as_mut()), Self::Affs(a) => f(a.as_mut()), + Self::LittleFs(l) => f(l.as_mut()), Self::Ntfs(n) => f(n.as_mut()), Self::F2fs(fs2) => f(fs2.as_mut()), Self::Squashfs(sq) => f(sq.as_mut()), @@ -1009,6 +1039,7 @@ impl AnyFs { Self::HfsPlus(_) => "hfs+", Self::Hfs(_) => "hfs", Self::Affs(_) => "affs", + Self::LittleFs(_) => "littlefs", Self::Apfs(_) => "apfs", Self::Ntfs(_) => "ntfs", Self::F2fs(_) => "f2fs", @@ -1105,6 +1136,7 @@ impl AnyFs { Self::HfsPlus(b) => b, Self::Hfs(b) => b, Self::Affs(b) => b, + Self::LittleFs(b) => b, Self::Apfs(b) => b, Self::Ntfs(b) => b, Self::F2fs(b) => b, diff --git a/src/memconv.rs b/src/memconv.rs index 62e5c65..8e7dd02 100644 --- a/src/memconv.rs +++ b/src/memconv.rs @@ -337,6 +337,16 @@ impl MemImage { sz, ) } + "littlefs" | "lfs" => { + // littlefs reserves nothing up front, so a content-fit size + // from its own size plan is enough; keep a floor so tiny + // sources still leave room to edit the result afterwards. + let sz = self.geometry_size()?; + self.build_generic::( + &crate::fs::littlefs::LittleFsFormatOpts::default(), + sz, + ) + } "apfs" => { let sz = self.geometry_size()?; self.build_generic::( @@ -655,6 +665,12 @@ pub fn supported_targets() -> Vec { ext: "adf", streaming: false, }, + TargetInfo { + id: "littlefs", + label: "littlefs", + ext: "img", + streaming: false, + }, TargetInfo { id: "squashfs", label: "SquashFS", diff --git a/src/memedit.rs b/src/memedit.rs index 4bfb287..065f422 100644 --- a/src/memedit.rs +++ b/src/memedit.rs @@ -180,6 +180,14 @@ pub fn creatable_filesystems() -> Vec { editable: true, options: "fstype,intl,volume_name", }, + FsTypeInfo { + id: "littlefs", + label: "littlefs", + min_size: 16 << 10, + default_size: 4 << 20, + editable: true, + options: "block_size,block_count,prog_size,version,name_max,inline_max", + }, FsTypeInfo { id: "f2fs", label: "F2FS", @@ -216,6 +224,7 @@ fn canonical_fs_id(fs_type: &str) -> String { "vfat" => "fat32".to_string(), "hfsplus" => "hfs+".to_string(), "ofs" | "ffs" => "affs".to_string(), + "lfs" => "littlefs".to_string(), other => other.to_string(), } } @@ -359,6 +368,31 @@ fn format_blank(fs_type: &str, dev: &mut MemoryBackend, options: &str) -> Result bag.check_empty(&id)?; AnyFs::Affs(Box::new(Affs::format(dev, &opts)?)) } + "littlefs" => { + use crate::fs::littlefs::{LittleFs, LittleFsFormatOpts}; + // Same knobs `spec::littlefs_format_opts` accepts. + let mut opts = LittleFsFormatOpts::default(); + if let Some(b) = bag.take_u32("block_size")? { + opts.block_size = b; + } + if let Some(b) = bag.take_u32("block_count")? { + opts.block_count = Some(b); + } + if let Some(p) = bag.take_u32("prog_size")? { + opts.prog_size = p; + } + if let Some(v) = bag.take_str("version") { + opts.disk_version = crate::spec::parse_littlefs_version(&v)?; + } + if let Some(n) = bag.take_u32("name_max")? { + opts.name_max = n; + } + if let Some(n) = bag.take_u32("inline_max")? { + opts.inline_max = Some(n); + } + bag.check_empty(&id)?; + AnyFs::LittleFs(Box::new(LittleFs::format(dev, &opts)?)) + } "f2fs" => { use crate::fs::f2fs::{F2fs, FormatOpts}; let opts = FormatOpts::default(); diff --git a/src/spec.rs b/src/spec.rs index 92927bc..22fc3df 100644 --- a/src/spec.rs +++ b/src/spec.rs @@ -206,6 +206,11 @@ fn build_bare_fs(fs: &FilesystemSpec, output: &Path) -> Result<()> { "affs" | "ffs" | "ofs" => { build_bare_via_trait::(fs, output, affs_format_opts(fs)?) } + "littlefs" | "lfs" => build_bare_via_trait::( + fs, + output, + littlefs_format_opts(fs)?, + ), "ntfs" => build_bare_via_trait::(fs, output, ntfs_format_opts(fs)?), "f2fs" => build_bare_via_trait::(fs, output, f2fs_format_opts(fs)?), "squashfs" => build_bare_via_trait::( @@ -417,6 +422,48 @@ fn affs_format_opts(fs: &FilesystemSpec) -> Result Result { + let mut bag = options_bag_for(fs)?; + let mut opts = crate::fs::littlefs::LittleFsFormatOpts::default(); + // The generic `block_size` key doubles as littlefs's erase-block size. + if let Some(bs) = fs.block_size { + opts.block_size = bs; + } + if let Some(bs) = bag.take_u32("block_size")? { + opts.block_size = bs; + } + if let Some(bc) = bag.take_u32("block_count")? { + opts.block_count = Some(bc); + } + if let Some(p) = bag.take_u32("prog_size")? { + opts.prog_size = p; + } + if let Some(v) = bag.take_str("version") { + opts.disk_version = parse_littlefs_version(&v)?; + } + if let Some(n) = bag.take_u32("name_max")? { + opts.name_max = n; + } + if let Some(n) = bag.take_u32("inline_max")? { + opts.inline_max = Some(n); + } + bag.check_empty("littlefs")?; + Ok(opts) +} + +/// Parse a littlefs on-disk version: `2.0` / `2.1` (or the raw `0x00020001`). +pub(crate) fn parse_littlefs_version(v: &str) -> Result { + match v.trim() { + "2.0" | "20000" | "0x00020000" => Ok(crate::fs::littlefs::DISK_VERSION_2_0), + "2.1" | "20001" | "0x00020001" => Ok(crate::fs::littlefs::DISK_VERSION_2_1), + other => Err(crate::Error::InvalidArgument(format!( + "littlefs: unknown disk version {other:?} (use 2.0 or 2.1)" + ))), + } +} + fn hfs_plus_format_opts(fs: &FilesystemSpec) -> Result { let mut bag = options_bag_for(fs)?; let mut opts = crate::fs::hfs_plus::FormatOpts { @@ -809,6 +856,13 @@ fn build_partitioned(image: &ImageSpec, partitions: &[PartitionSpec], output: &P affs_format_opts(fs)?, )?; } + "littlefs" | "lfs" => { + format_in_partition_via_trait::( + &mut slice, + fs, + littlefs_format_opts(fs)?, + )?; + } "ntfs" => { format_in_partition_via_trait::( &mut slice, diff --git a/tests/fs_fuzz.rs b/tests/fs_fuzz.rs index 29caeb0..b5227b5 100644 --- a/tests/fs_fuzz.rs +++ b/tests/fs_fuzz.rs @@ -820,6 +820,27 @@ fn fuzz_f2fs() { ); } +#[test] +fn fuzz_littlefs() { + use fstool::fs::littlefs::{LittleFs, LittleFsFormatOpts}; + // 4 MiB of 4 KiB blocks. Every mutation rewrites a metadata pair + // and (for outlined files) the tail of a CTZ skip-list, so this run + // leans hard on the block allocator: a single leaked or + // double-allocated block shows up as corruption within a few + // iterations. + const SIZE: u64 = 4 * 1024 * 1024; + let mut dev = MemoryBackend::new(SIZE); + let opts = LittleFsFormatOpts::default(); + let mut fs = LittleFs::format(&mut dev, &opts).expect("format littlefs"); + fuzz_filesystem( + &mut fs, + &mut dev, + 0x11F5_2222_C0DE, + FUZZ_ITERS, + &Caps::mutable_small(), + ); +} + #[test] fn fuzz_xfs() { use fstool::fs::xfs::{self, FormatOpts}; diff --git a/tests/littlefs_external.rs b/tests/littlefs_external.rs new file mode 100644 index 0000000..841f357 --- /dev/null +++ b/tests/littlefs_external.rs @@ -0,0 +1,513 @@ +#![cfg(unix)] +//! littlefs conformance against the reference C implementation. +//! +//! The checks here drive `littlefs-python`, which wraps the upstream C +//! `lfs.c` — so "the reference implementation mounts it" means exactly +//! that. Every test skips (with a note) when no Python with the module +//! installed is available, so the suite stays green on hosts without it: +//! +//! ```sh +//! pip install littlefs-python +//! # or point the tests at a specific interpreter: +//! FSTOOL_LITTLEFS_PYTHON=/path/to/venv/bin/python cargo test --test littlefs_external +//! ``` +//! +//! Four directions are covered: images we write must mount and read +//! correctly under littlefs; images littlefs writes must read correctly +//! under fstool; littlefs must be able to *keep writing* to an image we +//! produced (which exercises the erased-state and forward-CRC rules that +//! decide whether a metadata block can be appended to); and we must be able +//! to keep writing to an image it produced. + +use std::io::{Read, Seek, Write}; +use std::path::{Path, PathBuf}; +use std::process::Command; + +use fstool::block::{BlockDevice, FileBackend}; +use fstool::fs::littlefs::{DISK_VERSION_2_0, LittleFs, LittleFsFormatOpts}; +use fstool::fs::{EntryKind, FileMeta, FileSource, Filesystem, OpenFlags}; +use tempfile::TempDir; + +/// Find an interpreter that can `import littlefs`. +fn python() -> Option { + let mut candidates: Vec = Vec::new(); + if let Ok(p) = std::env::var("FSTOOL_LITTLEFS_PYTHON") { + candidates.push(p.into()); + } + candidates.push("python3".into()); + candidates.push("python".into()); + candidates.into_iter().find(|p| { + Command::new(p) + .args(["-c", "import littlefs"]) + .output() + .is_ok_and(|o| o.status.success()) + }) +} + +/// The helper script every test drives. Geometry is read out of the image's +/// own superblock so the harness never has to be told it. +const SCRIPT: &str = r#" +import sys, struct +from littlefs import LittleFS +from littlefs.context import UserContext + +def fnv(data): + h = 0xcbf29ce484222325 + for b in data: + h = ((h ^ b) * 0x100000001b3) & 0xffffffffffffffff + return h + +def load(path): + data = bytearray(open(path, 'rb').read()) + assert data[8:16] == b'littlefs', 'no littlefs magic' + bs, bc = struct.unpack(' u64 { + let mut h: u64 = 0xcbf2_9ce4_8422_2325; + for b in data { + h = (h ^ *b as u64).wrapping_mul(0x100_0000_01b3); + } + h +} + +struct Harness { + python: PathBuf, + dir: TempDir, + script: PathBuf, +} + +impl Harness { + /// `None` when littlefs-python isn't installed — the caller skips. + fn new() -> Option { + let python = python()?; + let dir = TempDir::new().ok()?; + let script = dir.path().join("lfs_ref.py"); + std::fs::write(&script, SCRIPT).ok()?; + Some(Self { + python, + dir, + script, + }) + } + + fn image(&self, name: &str) -> PathBuf { + self.dir.path().join(name) + } + + /// Run the helper script, returning its stdout. Panics with the script's + /// stderr on failure — a mount error there is the finding, not noise. + fn run(&self, cmd: &str, image: &Path) -> String { + let out = Command::new(&self.python) + .arg(&self.script) + .arg(cmd) + .arg(image) + .output() + .expect("failed to run the littlefs reference helper"); + assert!( + out.status.success(), + "littlefs reference `{cmd}` failed:\n{}", + String::from_utf8_lossy(&out.stderr) + ); + String::from_utf8_lossy(&out.stdout).trim().to_string() + } +} + +/// Deterministic payload of `len` bytes. +fn pattern(len: usize) -> Vec { + (0..len).map(|i| ((i * 7) % 251) as u8).collect() +} + +fn create_image(path: &Path, bytes: u64, opts: &LittleFsFormatOpts) -> (FileBackend, LittleFs) { + let mut dev = FileBackend::create(path, bytes).unwrap(); + let fs = LittleFs::format(&mut dev, opts).unwrap(); + (dev, fs) +} + +fn put(fs: &mut LittleFs, dev: &mut dyn BlockDevice, path: &str, body: &[u8]) { + fs.create_file( + dev, + Path::new(path), + FileSource::Reader { + reader: Box::new(std::io::Cursor::new(body.to_vec())), + len: body.len() as u64, + }, + FileMeta::default(), + ) + .unwrap(); +} + +fn slurp(fs: &mut LittleFs, dev: &mut dyn BlockDevice, path: &str) -> Vec { + let mut out = Vec::new(); + fs.read_file(dev, Path::new(path)) + .unwrap() + .read_to_end(&mut out) + .unwrap(); + out +} + +/// Walk an fstool-mounted volume into the same manifest the helper script +/// prints, so the two implementations can be compared line for line. +fn manifest(fs: &mut LittleFs, dev: &mut dyn BlockDevice) -> Vec { + fn walk(fs: &mut LittleFs, dev: &mut dyn BlockDevice, dir: &str, out: &mut Vec) { + let path = if dir.is_empty() { "/" } else { dir }; + let mut entries = fs.list(dev, Path::new(path)).unwrap(); + entries.sort_by(|a, b| a.name.cmp(&b.name)); + for e in &entries { + let child = format!("{dir}/{}", e.name); + match e.kind { + EntryKind::Dir => { + out.push(format!("d {child}")); + walk(fs, dev, &child, out); + } + _ => { + let body = slurp(fs, dev, &child); + out.push(format!("f {child} {} {}", body.len(), fnv(&body))); + } + } + } + } + let mut out = Vec::new(); + walk(fs, dev, "", &mut out); + out.sort(); + out +} + +/// Build the tree both directions of the round trip use. +fn build_reference_tree(fs: &mut LittleFs, dev: &mut dyn BlockDevice) { + put(fs, dev, "/README", b"written by fstool\n"); + fs.create_dir(dev, Path::new("/etc"), FileMeta::default()) + .unwrap(); + put(fs, dev, "/etc/motd", b"stay curious\n"); + fs.create_dir(dev, Path::new("/etc/deep"), FileMeta::default()) + .unwrap(); + fs.create_dir(dev, Path::new("/etc/deep/nested"), FileMeta::default()) + .unwrap(); + put(fs, dev, "/etc/deep/nested/leaf", &pattern(3)); + // Well past the inline limit: a multi-block CTZ skip-list. + put(fs, dev, "/big.bin", &pattern(40_000)); + // Enough entries to split the directory across metadata pairs. + fs.create_dir(dev, Path::new("/many"), FileMeta::default()) + .unwrap(); + for i in 0..80 { + put( + fs, + dev, + &format!("/many/file{i:03}"), + format!("entry number {i}").as_bytes(), + ); + } + fs.set_xattr(dev, Path::new("/README"), "user.littlefs.7", b"attrvalue") + .unwrap(); + fs.flush(dev).unwrap(); +} + +#[test] +fn our_images_mount_in_the_reference_implementation() { + let Some(h) = Harness::new() else { + eprintln!("skipping: no python with littlefs-python installed"); + return; + }; + let img = h.image("written-by-fstool.img"); + let (mut dev, mut fs) = create_image(&img, 4 * 1024 * 1024, &LittleFsFormatOpts::default()); + build_reference_tree(&mut fs, &mut dev); + let ours = manifest(&mut fs, &mut dev); + let our_blocks = fs.used_blocks(&mut dev).unwrap(); + drop(dev); + + let out = h.run("check", &img); + let mut lines = out.lines(); + let their_blocks: u32 = lines + .next() + .and_then(|l| l.strip_prefix("used ")) + .and_then(|n| n.parse().ok()) + .expect("check prints the block count first"); + let theirs: Vec = lines.map(|s| s.to_string()).collect(); + assert_eq!( + theirs, ours, + "the reference implementation and fstool disagree about the image" + ); + // Both sides traverse the volume to decide which blocks are live; if + // they disagree, one of them is leaking or about to reuse a live block. + assert_eq!( + their_blocks, our_blocks, + "block accounting differs between the implementations" + ); + // Spot-check the shape rather than trusting agreement alone. + assert!(theirs.iter().any(|l| l.starts_with("f /big.bin 40000 "))); + assert!(theirs.contains(&"d /etc/deep/nested".to_string())); + assert_eq!( + theirs.iter().filter(|l| l.starts_with("f /many/")).count(), + 80 + ); +} + +#[test] +fn version_2_0_images_mount_in_the_reference_implementation() { + let Some(h) = Harness::new() else { + eprintln!("skipping: no python with littlefs-python installed"); + return; + }; + let img = h.image("v2_0.img"); + let opts = LittleFsFormatOpts { + disk_version: DISK_VERSION_2_0, + block_size: 512, + prog_size: 128, + ..LittleFsFormatOpts::default() + }; + let (mut dev, mut fs) = create_image(&img, 1024 * 1024, &opts); + put(&mut fs, &mut dev, "/a.txt", b"small"); + put(&mut fs, &mut dev, "/b.bin", &pattern(20_000)); + fs.flush(&mut dev).unwrap(); + let ours = manifest(&mut fs, &mut dev); + drop(dev); + + let theirs: Vec = h + .run("manifest", &img) + .lines() + .map(|s| s.to_string()) + .collect(); + assert_eq!(theirs, ours); +} + +#[test] +fn reference_images_read_back_identically() { + let Some(h) = Harness::new() else { + eprintln!("skipping: no python with littlefs-python installed"); + return; + }; + let img = h.image("written-by-littlefs.img"); + h.run("create", &img); + let theirs: Vec = h + .run("manifest", &img) + .lines() + .map(|s| s.to_string()) + .collect(); + + let mut dev = FileBackend::open(&img).unwrap(); + let mut fs = LittleFs::open(&mut dev).unwrap(); + assert_eq!(fs.version(), (2, 1)); + assert_eq!(manifest(&mut fs, &mut dev), theirs); + + // Contents, not just checksums, and the user attribute it set. + assert_eq!( + slurp(&mut fs, &mut dev, "/greeting.txt"), + b"written by littlefs\n" + ); + assert_eq!( + slurp(&mut fs, &mut dev, "/data/blob.bin"), + pattern(9000).as_slice() + ); + let attrs = fs + .list_xattrs(&mut dev, Path::new("/greeting.txt")) + .unwrap(); + assert_eq!(attrs.len(), 1); + assert_eq!(attrs[0].name, "user.littlefs.7"); + assert_eq!(attrs[0].value, b"attrvalue"); +} + +#[test] +fn the_reference_implementation_can_keep_writing_to_our_images() { + let Some(h) = Harness::new() else { + eprintln!("skipping: no python with littlefs-python installed"); + return; + }; + let img = h.image("handed-over.img"); + let (mut dev, mut fs) = create_image(&img, 4 * 1024 * 1024, &LittleFsFormatOpts::default()); + build_reference_tree(&mut fs, &mut dev); + drop(dev); + + // littlefs adds, removes and creates a directory in our image. + h.run("mutate", &img); + + let mut dev = FileBackend::open(&img).unwrap(); + let mut fs = LittleFs::open(&mut dev).unwrap(); + assert_eq!( + slurp(&mut fs, &mut dev, "/added-by-lfs.txt"), + b"appended by the reference implementation\n" + ); + let payload = slurp(&mut fs, &mut dev, "/lfsdir/payload.bin"); + assert_eq!(payload.len(), 256 * 40); + assert_eq!(&payload[..4], &[0, 1, 2, 3]); + // Its removal is visible to us, and everything else survived. + assert!( + fs.list(&mut dev, Path::new("/")) + .unwrap() + .iter() + .all(|e| e.name != "README") + ); + assert_eq!(slurp(&mut fs, &mut dev, "/big.bin"), pattern(40_000)); + assert_eq!(fs.list(&mut dev, Path::new("/many")).unwrap().len(), 80); +} + +#[test] +fn in_place_edits_of_large_files_are_readable_by_the_reference() { + let Some(h) = Harness::new() else { + eprintln!("skipping: no python with littlefs-python installed"); + return; + }; + let img = h.image("patched.img"); + let (mut dev, mut fs) = create_image(&img, 4 * 1024 * 1024, &LittleFsFormatOpts::default()); + // Long enough to need several levels of skip pointers (index 20+ at a + // 4 KiB block size), so a rewrite has to rebuild them correctly. + let original = pattern(120_000); + put(&mut fs, &mut dev, "/big.bin", &original); + put(&mut fs, &mut dev, "/shrink.bin", &pattern(60_000)); + + let mut expected = original.clone(); + { + // A patch in the middle: blocks before it must survive untouched + // while everything after is rewritten. + let mut handle = fs + .open_file_rw(&mut dev, Path::new("/big.bin"), OpenFlags::default(), None) + .unwrap(); + handle.seek(std::io::SeekFrom::Start(50_000)).unwrap(); + handle.write_all(b"PATCHED-IN-PLACE").unwrap(); + handle.sync().unwrap(); + } + expected[50_000..50_016].copy_from_slice(b"PATCHED-IN-PLACE"); + + // An append past the end, and a truncation of the other file. + { + let mut handle = fs + .open_file_rw( + &mut dev, + Path::new("/big.bin"), + OpenFlags { + append: true, + ..OpenFlags::default() + }, + None, + ) + .unwrap(); + handle.write_all(&pattern(5_000)).unwrap(); + handle.sync().unwrap(); + } + expected.extend_from_slice(&pattern(5_000)); + fs.truncate(&mut dev, Path::new("/shrink.bin"), 21_000) + .unwrap(); + fs.flush(&mut dev).unwrap(); + assert_eq!(slurp(&mut fs, &mut dev, "/big.bin"), expected); + drop(dev); + + let out = h.run("check", &img); + assert!( + out.contains(&format!("f /big.bin {} {}", expected.len(), fnv(&expected))), + "reference read of the patched file differs:\n{out}" + ); + let shrunk = pattern(60_000)[..21_000].to_vec(); + assert!( + out.contains(&format!("f /shrink.bin 21000 {}", fnv(&shrunk))), + "reference read of the truncated file differs:\n{out}" + ); +} + +#[test] +fn we_can_keep_writing_to_reference_images() { + let Some(h) = Harness::new() else { + eprintln!("skipping: no python with littlefs-python installed"); + return; + }; + let img = h.image("taken-over.img"); + h.run("create", &img); + + { + let mut dev = FileBackend::open(&img).unwrap(); + let mut fs = LittleFs::open(&mut dev).unwrap(); + put(&mut fs, &mut dev, "/added-by-fstool.txt", b"our turn\n"); + put(&mut fs, &mut dev, "/data/second.bin", &pattern(12_345)); + fs.create_dir(&mut dev, Path::new("/data/more"), FileMeta::default()) + .unwrap(); + put(&mut fs, &mut dev, "/data/more/deep", b"deep"); + fs.remove(&mut dev, Path::new("/data/inner/tiny")).unwrap(); + fs.remove(&mut dev, Path::new("/data/inner")).unwrap(); + fs.remove(&mut dev, Path::new("/entry007")).unwrap(); + fs.flush(&mut dev).unwrap(); + } + + let out = h.run("check", &img); + assert!(out.contains("f /added-by-fstool.txt 9 "), "{out}"); + assert!(out.contains("f /data/more/deep 4 "), "{out}"); + assert!( + out.contains(&format!( + "f /data/second.bin 12345 {}", + fnv(&pattern(12_345)) + )), + "{out}" + ); + assert!(!out.contains("/data/inner"), "{out}"); + assert!(!out.contains("/entry007"), "{out}"); + // The files it wrote are still intact under its own reader. + assert!( + out.contains(&format!("f /data/blob.bin 9000 {}", fnv(&pattern(9000)))), + "{out}" + ); +} diff --git a/tests/reopen_mutate.rs b/tests/reopen_mutate.rs index 1e7c74b..24a2c74 100644 --- a/tests/reopen_mutate.rs +++ b/tests/reopen_mutate.rs @@ -305,6 +305,34 @@ fn exfat_reopen_mutate() { } } +// ---------------------------------------------------------------------- +// littlefs (every mutation is a metadata-pair commit; a reopened handle +// is no different from a freshly formatted one) +// ---------------------------------------------------------------------- +#[test] +fn littlefs_reopen_mutate() { + use fstool::fs::littlefs::{LittleFs, LittleFsFormatOpts}; + let tmp = NamedTempFile::new().unwrap(); + { + let mut dev = FileBackend::create(tmp.path(), 4 * 1024 * 1024).unwrap(); + let opts = LittleFsFormatOpts::default(); + let mut fs: Box = Box::new(LittleFs::format(&mut dev, &opts).unwrap()); + fs.create_file( + &mut dev, + Path::new("/seed.txt"), + src(SEED), + FileMeta::default(), + ) + .unwrap(); + fs.flush(&mut dev).unwrap(); + dev.sync().unwrap(); + } + reopen_add_then_verify(tmp.path()); + anyfs_put_flush_survives(tmp.path()); + // No `fsck.littlefs` exists; `tests/littlefs_external.rs` cross-checks + // against the reference C implementation instead. +} + // ---------------------------------------------------------------------- // NTFS (DirBatch directory writes; AnyFs::flush must drain them) // ---------------------------------------------------------------------- diff --git a/web/README.md b/web/README.md index d5dc85f..0c04912 100644 --- a/web/README.md +++ b/web/README.md @@ -6,7 +6,7 @@ locally in WebAssembly memory: - **Inspect** — drop in an archive or disk image, browse what's inside, extract individual files, convert the whole thing to another format. -- **Create** — format a blank filesystem (14 types) or lay out a partitioned +- **Create** — format a blank filesystem (15 types) or lay out a partitioned disk (MBR/GPT, one filesystem per partition), add files and folders, and download the image at any point. You can keep editing and download again.