diff --git a/CHANGELOG.md b/CHANGELOG.md index 68b8e0d..313cd25 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -70,6 +70,30 @@ because it turns other people's test suites red. ### Added +- **An archive can hold its files in directories.** `--set depth=3` puts every + file three levels down, and `--set directory_entries=true` also makes the + archive list the directories themselves. Both work on `zip` and on `targz`. + + Two settings rather than one, because they are two questions. Depth is about + the paths inside. Directory entries are about whether the archive names the + directories at all - and extractors differ there: some create a directory + when they meet a path that needs one, and some create only what the archive + names. An archive is the one format where you can test both. + + The default is flat, which is what archives from this tool have always been, + so **no existing file changes by a byte**. Asking for `directory_entries` + without a depth is refused rather than quietly ignored: a flat archive has no + directories to name, and the message says so and names both settings. + + Depth goes up to 50. The limit is measured rather than picked: a `.tar.gz` + writes USTAR headers, which carry a path in a 155 byte prefix and a 100 byte + name split on a slash, and past a certain length no split works. Directories + cost 512 bytes each in a `.tar.gz` and about 76 plus the path in a `.zip`. + The size you order is still the size you get, to the byte. + + The padding entry stays at the top of the archive rather than moving into the + directories, so you can always tell it apart from the files you asked for. + - **A zip can be locked with ZipCrypto, the old scheme.** `--set encryption=zipcrypto`. It is here for what it does to a reader rather than for what it protects. Measured: .NET's own `ZipFile` opens one of these, reports the entry at its true length, hands back a stream and fills it with the ENCRYPTED bytes - and never says the entry was encrypted at all. An application built on that library processes noise and calls it data. AES fails loudly in the same library, which is the safer defect and the less interesting one. diff --git a/internal/format/archive/archive.go b/internal/format/archive/archive.go index 9ae8101..9d50343 100644 --- a/internal/format/archive/archive.go +++ b/internal/format/archive/archive.go @@ -113,6 +113,19 @@ func mustSize(s string) int64 { // somebody kept them so by hand, and the comment saying so was the whole // mechanism. var axes = map[string]format.Property{ + Depth: { + Name: Depth, Kind: format.PropertyInt, + Min: 0, Max: maxDepth, + Default: strconv.Itoa(defaultDepth), + Detail: "How many directories deep the files inside sit. 0 puts them all at the top.", + }, + DirectoryEntries: { + Name: DirectoryEntries, Kind: format.PropertyBool, + Default: "false", + Detail: "Whether the archive also lists the directories themselves. " + + "Most file browsers show the same folders either way, so the difference is in the entry list rather than on screen. " + + "It matters for readers that only create a directory the archive names.", + }, Entries: { Name: Entries, Kind: format.PropertyInt, Min: 0, Max: maxEntries, diff --git a/internal/format/archive/layout.go b/internal/format/archive/layout.go new file mode 100644 index 0000000..c2023cc --- /dev/null +++ b/internal/format/archive/layout.go @@ -0,0 +1,186 @@ +package archive + +import ( + "fmt" + "strconv" + "strings" + + "github.com/donislawdev/TestingFilesGenerator/internal/format" +) + +// Where the files inside an archive sit, and whether the archive lists the +// directories themselves. +// +// Both containers read this one description rather than each growing its own, +// for the reason the package exists: zip and tar held six constants with +// identical values and two identical readers before it, and the only thing +// keeping them equal was a comment. +// +// Two settings rather than one, and the pairing is deliberate. A tester asking +// "does the tool under test cope with nested paths" wants depth. A tester +// asking "does it cope with an archive that does NOT list its directories" +// wants the other, because extractors differ: some create a directory when +// they meet a path that needs one, and some only create what the archive +// names. An archive is the one format where those are separate questions. +const ( + Depth = "depth" + DirectoryEntries = "directory_entries" +) + +const ( + // defaultDepth is flat, and it has to stay flat. Every archive this tool + // has written so far holds its files at the top, so any other default + // would move the bytes of all of them - untouchable rule 3, and the reason + // this change needs no version bump at all. + defaultDepth = 0 + + // maxDepth is measured rather than chosen, and the measurement is about + // tar rather than about taste. + // + // targz pins tar.FormatUSTAR (size.go), which carries a path in two + // fields: a 155 byte prefix and a 100 byte name, split ON A SLASH. So a + // path is writable when some slash leaves at most 155 before it and at + // most 100 after it, which is a rule about where the slashes fall and not + // about length. With the segments below, slashes sit every 4 bytes, the + // last usable one is at 155, and the path therefore has to come to 256 + // bytes or fewer: 4*depth + len(entry name) <= 256. + // + // Measured 2026-09-01 against Go's own archive/tar with USTAR pinned: + // depth 61 with a 12 byte name is taken at 256 bytes and depth 62 is + // REFUSED at 260. The size is flat the whole way - 1536 B at every depth + // up to the refusal, no hidden step - so the tar arithmetic needs no + // length term at all. + // + // 61 is therefore the ceiling for a 12 byte name and NOT the ceiling to + // declare, because the entry name is not always 12 bytes. The longest one + // this build can produce is targz_0001.tar.gz at 17, which lands the limit + // at 59. Fifty leaves room for a name of 56 bytes, which is far past + // anything the registry holds, and a guard proves it for every registered + // format rather than trusting this paragraph. + maxDepth = 50 + + // dirSegment numbers the levels so a path reads as what it is. Two digits + // because maxDepth is two digits, and a fixed width so every segment is + // the same size and the arithmetic above stays a multiplication. + dirSegment = "d%02d/" + + // dirSegmentBytes is what one segment comes to once rendered - "d00/" is + // four bytes where the format string above is six. Written out rather than + // taken as len(dirSegment), which is the bug the depth guard caught the + // first time it ran: the arithmetic said every path was 2 bytes per level + // longer than it is, which would have understated the ceiling rather than + // overstating it, so nothing would have failed until somebody widened the + // segment. A guard compares this against a really rendered path. + dirSegmentBytes = 4 +) + +// Layout is what the two settings come to once read. +type Layout struct { + // Depth is how many directories deep the files sit. Zero is flat. + Depth int + // DirEntries says whether the archive also names the directories. + DirEntries bool +} + +// Path is where an entry called name sits under this layout. +// +// The empty name gives the directory chain itself with its trailing slash, +// which is what both containers want a directory entry to be called. +func (l Layout) Path(name string) string { + if l.Depth <= 0 { + return name + } + var b strings.Builder + b.Grow(l.Depth*len(dirSegment) + len(name)) + for i := 0; i < l.Depth; i++ { + fmt.Fprintf(&b, dirSegment, i) + } + b.WriteString(name) + return b.String() +} + +// Directories is every directory this layout creates, outermost first. +// +// Outermost first because that is the order an extractor wants to meet them +// in: a reader that creates directories as it goes cannot make d00/d01 before +// it has made d00. It is empty when nothing was asked for, so a caller can +// range over it without asking whether the setting is on. +func (l Layout) Directories() []string { + if !l.DirEntries || l.Depth <= 0 { + return nil + } + out := make([]string, 0, l.Depth) + for i := 1; i <= l.Depth; i++ { + out = append(out, Layout{Depth: i}.Path("")) + } + return out +} + +// LongestPath is the longest path this layout can produce for an entry name of +// the given length. It exists for the guard that proves maxDepth is safe. +func LongestPath(depth, nameLen int) int { + return depth*dirSegmentBytes + nameLen +} + +// MaxDepth is the deepest nesting this build offers, for the guard that checks +// the declaration against what tar will actually take. +func MaxDepth() int { return maxDepth } + +// ReadLayout works out where the files go, and refuses a pair that cannot mean +// anything. +// +// directory_entries with a flat archive is the pair, and it is a refusal +// rather than a setting quietly doing nothing. There are no directories in a +// flat archive, so the answer would be the same whichever way it was set - and +// rule 6 forbids exactly that silence. The message names BOTH halves, because +// a reader who set one of them cannot tell from "directory_entries is not +// allowed" which one to change. +// +// It is reachable from the window as well as from a recipe, which is why it +// has to be a good message rather than an internal check: the control is a +// checkbox, a checkbox always sends its value, and somebody can tick it while +// depth is still nought. +func ReadLayout(id string, r format.Request) (Layout, error) { + depth, err := intProperty(id, r.Properties, Depth, defaultDepth, 0, maxDepth) + if err != nil { + return Layout{}, err + } + dirs, err := boolProperty(id, r.Properties, DirectoryEntries, false) + if err != nil { + return Layout{}, err + } + if dirs && depth == 0 { + return Layout{}, &format.PropertyValueError{ + Format: id, + Key: DirectoryEntries, + Value: "true", + Reason: "a flat archive has no directories to list, and " + Depth + " is 0", + Remedy: "Ask for " + Depth + " of 1 or more, or leave " + DirectoryEntries + " off.", + } + } + return Layout{Depth: depth, DirEntries: dirs}, nil +} + +// boolProperty reads a true or false setting. +// +// The registry has already refused anything that is not true or false by the +// time a generator runs, since the declaration says the kind. This repeats the +// check for the same reason intProperty repeats its range: a caller reaching +// the generator directly is not going through the registry. +func boolProperty(id string, props map[string]string, key string, fallback bool) (bool, error) { + raw, ok := props[key] + if !ok || raw == "" { + return fallback, nil + } + v, err := strconv.ParseBool(strings.ToLower(raw)) + if err != nil { + return false, &format.PropertyValueError{ + Format: id, + Key: key, + Value: raw, + Reason: "it takes true or false", + Remedy: "Write " + key + ": true or " + key + ": false.", + } + } + return v, nil +} diff --git a/internal/format/targz/size.go b/internal/format/targz/size.go index 61c5d65..0c9a0ea 100644 --- a/internal/format/targz/size.go +++ b/internal/format/targz/size.go @@ -59,6 +59,16 @@ func roundUpBlock(n int64) int64 { // tarLength is the length of the tar stream before it reaches gzip. func tarLength(m memo) int64 { total := int64(2 * tarBlock) // the end of archive marker + // A directory is a header and nothing else, so it costs exactly one block. + // Measured 2026-09-01 against archive/tar rather than read off the format: + // a tar holding one directory and nothing else comes to 1536 B, of which + // 1024 is the end of archive marker. A guard holds that number. + // + // The path length does NOT appear here, and that is measured too. USTAR + // splits a path across a 155 byte prefix and a 100 byte name, so a header + // is the same 512 bytes at every depth it accepts - flat all the way to + // the refusal, with no step. maxDepth is what keeps it on the near side. + total += int64(len(m.layout.Directories())) * tarBlock for _, c := range m.children { total += tarBlock + roundUpBlock(c.plan.Bytes) } @@ -279,6 +289,20 @@ func build(ctx context.Context, w io.Writer, m memo) error { } tw := tar.NewWriter(zw) + // Directories first, outermost first, and only when asked for. They are + // not in m.children on purpose: a child's seed is FileSeed(seed, index) + // over a running index, so a directory in that list would shift the seed + // of every file after it and rewrite its contents. That is untouchable + // rule 2 - an edit in one place moving the bytes in another. + for _, dir := range m.layout.Directories() { + if err := ctx.Err(); err != nil { + return err + } + if err := writeDirectory(tw, dir, m.own); err != nil { + return fmt.Errorf("targz: the directory %q could not be named: %w", dir, err) + } + } + for _, c := range m.children { if err := ctx.Err(); err != nil { return err @@ -334,6 +358,34 @@ func writeEntry(ctx context.Context, tw *tar.Writer, e tarEntry, body func(io.Wr return body(tw) } +// writeDirectory names one directory in the tar. +// +// The mode is 0755 rather than own.Mode, and that is a decision rather than an +// oversight. entry_mode is declared as "the permissions recorded for each file +// inside", and a directory is not a file - recording 644 on one would produce +// an archive that extracts into directories nothing can be written into, which +// is a surprise nobody asked this setting for. The owner fields DO follow +// entry_owner, so an archive that says everything belongs to root says it about +// the directories too. +// +// It costs exactly one block and no more, which is the whole of what the +// arithmetic above had to learn about it. Measured rather than read off the +// format, and a guard holds the number. +func writeDirectory(tw *tar.Writer, name string, own archive.Ownership) error { + return tw.WriteHeader(&tar.Header{ + Name: name, + Size: 0, + Mode: 0o755, + Uid: own.Uid, + Gid: own.Gid, + Uname: own.Uname, + Gname: own.Gname, + ModTime: fixedTime, + Typeflag: tar.TypeDir, + Format: tar.FormatUSTAR, + }) +} + // tarEntry is one entry's header, as both the measuring pass and the // writing pass describe it. // diff --git a/internal/format/targz/targz.go b/internal/format/targz/targz.go index 133ae70..0e5aa4b 100644 --- a/internal/format/targz/targz.go +++ b/internal/format/targz/targz.go @@ -116,6 +116,7 @@ func init() { // package. Listed rather than received whole, so a format takes only // the axes it can actually carry. Properties: archive.Axes(archive.Entries, archive.EntryFormat, archive.EntrySize, + archive.Depth, archive.DirectoryEntries, archive.EntryMode, archive.EntryOwner), // Neither half of this format has anywhere to put a password, and @@ -164,6 +165,11 @@ type memo struct { // The zero value is not the default - ReadOwnership fills it, because // the mode this format has always written is 644 rather than 0. own archive.Ownership + // layout is where the files inside sit. It has to be here rather than an + // argument because tarLength counts from this struct and build writes + // from it: a layout the two disagreed about would promise one size and + // write another. + layout archive.Layout // withExtra says whether the header carries a gzip extra field at all, // and extraLen says how many bytes it holds. Two fields rather than one // with a sentinel, matching withFiller beside them, because an EMPTY @@ -185,8 +191,13 @@ func (generator) Plan(r format.Request) (format.Plan, error) { return format.Plan{}, err } - m := memo{seed: r.Seed, own: own} - if m.children, err = planChildren(r, groups); err != nil { + layout, err := archive.ReadLayout("targz", r) + if err != nil { + return format.Plan{}, err + } + + m := memo{seed: r.Seed, own: own, layout: layout} + if m.children, err = planChildren(r, groups, layout); err != nil { return format.Plan{}, err } @@ -209,7 +220,7 @@ func (generator) Plan(r format.Request) (format.Plan, error) { // Members are numbered across the whole archive rather than per group, so the // seed of a member does not move when a group above it changes count. That is // untouchable rule 2 applied one level down. -func planChildren(r format.Request, groups []format.Content) ([]child, error) { +func planChildren(r format.Request, groups []format.Content, layout archive.Layout) ([]child, error) { var out []child index := 0 // Numbering runs per format rather than per group, so two groups of the @@ -231,7 +242,7 @@ func planChildren(r format.Request, groups []format.Content) ([]child, error) { } numbered[g.Format]++ out = append(out, child{ - name: fmt.Sprintf("%s_%04d%s", g.Format, numbered[g.Format], desc.Extension), + name: layout.Path(fmt.Sprintf("%s_%04d%s", g.Format, numbered[g.Format], desc.Extension)), desc: desc, plan: cp, }) @@ -290,7 +301,12 @@ func describe(target int64, label string, m memo, groups []format.Content) forma // Stored rather than deflated, which is what makes the size exact // in one pass. Stated here so a test can assert on it rather than // infer it from how well the file compresses. - "compression": "none", + "compression": "none", + // Where the files sit, and whether the directories are named - + // written every time rather than only when nested, so a harness + // never has to read a missing key as flat. + archive.Depth: m.layout.Depth, + archive.DirectoryEntries: m.layout.DirEntries, format.PropertyLabelEmbedded: label != "", }, } diff --git a/internal/format/zip/directories.go b/internal/format/zip/directories.go new file mode 100644 index 0000000..646e14a --- /dev/null +++ b/internal/format/zip/directories.go @@ -0,0 +1,56 @@ +package zip + +import ( + stdzip "archive/zip" + "context" + "fmt" + "io/fs" + + "github.com/donislawdev/TestingFilesGenerator/internal/format/archive" +) + +// writeDirectories names the directories the archive was asked to list. +// +// Outermost first, because that is the order a reader can act on: one that +// creates a directory when it meets it cannot make d00/d01 before it has made +// d00. Layout.Directories gives them that way and gives nothing at all when +// the archive was not asked to name them, so there is no flag to read here. +// +// These are NOT entries in m.children, and that is the load bearing part. +// A child's seed is core.FileSeed(seed, index) over a running index, so a +// directory sitting in that list would shift the seed of every file after it +// and rewrite the contents of all of them - untouchable rule 2, where an edit +// in one place moves the bytes in another. Nothing here consumes an index. +// +// archiveSize still counts them, because it counts by running build against a +// counting writer rather than by arithmetic. So the size stays exact without +// anybody adding a term for this. +func writeDirectories(ctx context.Context, zw *stdzip.Writer, layout archive.Layout) error { + for _, dir := range layout.Directories() { + select { + case <-ctx.Done(): + return ctx.Err() + default: + } + if err := writeDirectory(zw, dir); err != nil { + return err + } + } + return nil +} + +// writeDirectory names one directory in the archive. +// +// It does not go through openEntry, and that is not a shortcut. openEntry is +// where locking happens, and locking is about content: a directory has none, +// so running it through the encrypting path would ask for a checksum and a +// header over nothing. Every reader takes a stored, empty, slash terminated +// entry as a directory. +func writeDirectory(zw *stdzip.Writer, name string) error { + h := &stdzip.FileHeader{Name: name, Method: stdzip.Store} + h.SetMode(fs.ModeDir | 0o755) + if _, err := zw.CreateHeader(h); err != nil { + return fmt.Errorf("zip: the directory %q could not be named: %w", name, err) + } + return nil +} diff --git a/internal/format/zip/filler.go b/internal/format/zip/filler.go new file mode 100644 index 0000000..8f48788 --- /dev/null +++ b/internal/format/zip/filler.go @@ -0,0 +1,47 @@ +package zip + +import ( + stdzip "archive/zip" + "context" + "io" +) + +// writeFillerEntry puts the padding entry into the archive. +// +// It lives beside build rather than inside it because build had grown past the +// length this project caps functions at, and this is the piece that comes out +// whole: everything here is about one entry that is not one of the files +// anybody ordered. +// +// The filler stays at the top of the archive even when the files inside are +// nested. Its path is then a constant, so the size arithmetic does not move +// with the depth - and it is not one of the ordered files, so a reader can +// always tell it apart from them. +func writeFillerEntry(ctx context.Context, zw *stdzip.Writer, m memo, withContents bool) error { + if !m.withFiller { + return nil + } + // The filler is locked with everything else. An archive where one entry + // opens without the password and the rest do not is a file nobody + // asked for, and the arithmetic is the same either way. + crc, err := plaintextCRC(ctx, m, withContents, func(w io.Writer) error { + return writeFiller(ctx, w, m.seed, m.fillerSize) + }) + if err != nil { + return err + } + entry, shut, err := openEntry(zw, m, entryPlan{ + name: fillerName, plain: m.fillerSize, index: len(m.children), + withContents: withContents, crc: crc, + }) + if err != nil { + return err + } + if !withContents { + return nil + } + if err := writeFiller(ctx, entry, m.seed, m.fillerSize); err != nil { + return err + } + return shut() +} diff --git a/internal/format/zip/zip.go b/internal/format/zip/zip.go index 08f939a..6a7d844 100644 --- a/internal/format/zip/zip.go +++ b/internal/format/zip/zip.go @@ -11,6 +11,7 @@ import ( "fmt" "hash/crc32" "io" + "strings" "time" @@ -75,6 +76,7 @@ func init() { // package. Listed rather than received whole, so a format takes only // the axes it can actually carry. Properties: archive.Axes(archive.Entries, archive.EntryFormat, archive.EntrySize, + archive.Depth, archive.DirectoryEntries, archive.Password, archive.Encryption), Container: true, GeneratorVersion: generatorVersion, @@ -101,6 +103,11 @@ type memo struct { // through an argument because the counting pass and the writing pass // have to agree about it exactly, and they share this. lock archive.Lock + // layout is where the files inside sit, and it travels the same way and + // for the same reason: archiveSize counts by running build against a + // counter, so a layout the two passes disagreed about would produce an + // archive whose size had been promised for a different shape. + layout archive.Layout } func (generator) Plan(r format.Request) (format.Plan, error) { @@ -114,8 +121,13 @@ func (generator) Plan(r format.Request) (format.Plan, error) { return format.Plan{}, err } - m := memo{seed: r.Seed, lock: lock} - if m.children, err = planChildren(r, groups); err != nil { + layout, err := archive.ReadLayout("zip", r) + if err != nil { + return format.Plan{}, err + } + + m := memo{seed: r.Seed, lock: lock, layout: layout} + if m.children, err = planChildren(r, groups, layout); err != nil { return format.Plan{}, err } @@ -204,7 +216,7 @@ func withinZip32(total int64) error { // Members are numbered across the whole archive rather than per group, so the // seed of a member does not move when a group above it changes count. That is // untouchable rule 2 applied one level down. -func planChildren(r format.Request, groups []format.Content) ([]child, error) { +func planChildren(r format.Request, groups []format.Content, layout archive.Layout) ([]child, error) { var out []child index := 0 // Numbering runs per format rather than per group, so two groups of the @@ -227,7 +239,7 @@ func planChildren(r format.Request, groups []format.Content) ([]child, error) { } numbered[g.Format]++ out = append(out, child{ - name: fmt.Sprintf("%s_%04d%s", g.Format, numbered[g.Format], desc.Extension), + name: layout.Path(fmt.Sprintf("%s_%04d%s", g.Format, numbered[g.Format], desc.Extension)), desc: desc, plan: cp, }) @@ -296,9 +308,18 @@ func describe(target int64, label string, m memo, groups []format.Content) forma Exact: true, Determinism: format.DeterminismByte, Properties: map[string]any{ - "entries": len(m.children), - "contains": contentSummary(groups), - "method": "store", + "entries": len(m.children), + "contains": contentSummary(groups), + "method": "store", + // Where the files sit, and whether the directories are named. + // Written every time rather than only when nested, like method + // and entries beside them: a harness reading this should not have + // to know that a missing key means flat. Rule 6 - what the run + // produced has to be visible in the manifest, and "the archive is + // three levels deep" is exactly the kind of thing a test asserts + // against. + archive.Depth: m.layout.Depth, + archive.DirectoryEntries: m.layout.DirEntries, format.PropertyLabelEmbedded: label != "", }, } @@ -523,6 +544,12 @@ func build(ctx context.Context, w io.Writer, m memo, withContents bool) error { return fmt.Errorf("zip: the archive comment was refused: %w", err) } + // The directories the archive names come before anything that sits in + // them. See writeDirectories for why they are not children. + if err := writeDirectories(ctx, zw, m.layout); err != nil { + return err + } + for i, c := range m.children { select { case <-ctx.Done(): @@ -552,31 +579,8 @@ func build(ctx context.Context, w io.Writer, m memo, withContents bool) error { } } - if m.withFiller { - // The filler is locked with everything else. An archive where one entry - // opens without the password and the rest do not is a file nobody - // asked for, and the arithmetic is the same either way. - crc, err := plaintextCRC(ctx, m, withContents, func(w io.Writer) error { - return writeFiller(ctx, w, m.seed, m.fillerSize) - }) - if err != nil { - return err - } - entry, shut, err := openEntry(zw, m, entryPlan{ - name: fillerName, plain: m.fillerSize, index: len(m.children), - withContents: withContents, crc: crc, - }) - if err != nil { - return err - } - if withContents { - if err := writeFiller(ctx, entry, m.seed, m.fillerSize); err != nil { - return err - } - if err := shut(); err != nil { - return err - } - } + if err := writeFillerEntry(ctx, zw, m, withContents); err != nil { + return err } return zw.Close() diff --git a/internal/guard/archivedepth_test.go b/internal/guard/archivedepth_test.go new file mode 100644 index 0000000..73e7aa1 --- /dev/null +++ b/internal/guard/archivedepth_test.go @@ -0,0 +1,245 @@ +package guard + +import ( + "archive/tar" + "bytes" + "errors" + "fmt" + "io" + "strings" + "testing" + + "github.com/donislawdev/TestingFilesGenerator/internal/format" + _ "github.com/donislawdev/TestingFilesGenerator/internal/format/all" + "github.com/donislawdev/TestingFilesGenerator/internal/format/archive" +) + +// The deepest path this build can produce still has to fit a USTAR header. +// +// This guard was written BEFORE depth existed, because the behaviour it holds +// is the one the change could break silently. targz pins tar.FormatUSTAR, and +// USTAR carries a path in a 155 byte prefix and a 100 byte name split ON A +// SLASH - so whether a path fits is a question about where its slashes fall, +// not about its length. Go answers it by refusing to write the header, and a +// refusal at write time is the worst place to find out: the size has already +// been planned and promised. +// +// Measured 2026-09-01: with four byte segments the last usable slash sits at +// 155, so a path has to come to 256 bytes or fewer. Depth 61 with a 12 byte +// name is taken at 256 and depth 62 is refused at 260. That makes the ceiling +// depend on the ENTRY NAME, which is not a constant - the longest this build +// can make is targz_0001.tar.gz at 17 bytes, which puts the real limit at 59 +// rather than 61. +// +// So the declaration cannot be checked against a number somebody wrote down. +// It is checked against every registered format, at the deepest nesting the +// build offers, by asking tar itself. A format added tomorrow with a longer id +// or extension reddens this without a line changing here. +func TestTheDeepestPathEveryFormatCanMakeStillFitsAUstarHeader(t *testing.T) { + depth := archive.MaxDepth() + layout := archive.Layout{Depth: depth} + + checked := 0 + for _, d := range format.All() { + // 9999 rather than 0001: the counter is four digits wide, so the + // longest name a format can produce is the one with the largest + // number in it. They are the same width today and would stop being so + // the day the counter grows. + name := fmt.Sprintf("%s_%04d%s", d.ID, 9999, d.Extension) + path := layout.Path(name) + + if got, want := len(path), archive.LongestPath(depth, len(name)); got != want { + t.Errorf("%s: the path is %d bytes and LongestPath says %d - "+ + "the guard and the thing it guards disagree about the arithmetic", + d.ID, got, want) + } + + if err := ustarAccepts(path); err != nil { + t.Errorf("%s: the deepest path this build can make cannot be written:\n"+ + " path %q (%d bytes)\n"+ + " tar %v\n"+ + " Lower maxDepth in internal/format/archive/layout.go, or shorten the segment.", + d.ID, path, len(path), err) + } + checked++ + } + + if checked == 0 { + t.Fatal("no formats were checked, so this proved nothing about any of them") + } +} + +// And the one below it has to be refused, or the ceiling is decoration. +// +// A ceiling nobody can reach on purpose is a ceiling nobody has watched work, +// and this project has thrown away several pieces of defensive code for that +// reason. If tar took every depth, maxDepth would be holding back nothing and +// the guard above would pass whatever it said. +func TestAPathOneStepPastTheCeilingIsRefusedByTar(t *testing.T) { + // The longest name the registry holds, so this asks about the tightest + // format rather than a comfortable one. + longest := "" + for _, d := range format.All() { + if name := fmt.Sprintf("%s_%04d%s", d.ID, 9999, d.Extension); len(name) > len(longest) { + longest = name + } + } + + // Walk out until tar says no. It has to say no somewhere, and it has to + // say no ABOVE the depth we offer rather than at it. + refusedAt := 0 + for depth := archive.MaxDepth(); depth <= archive.MaxDepth()+200; depth++ { + if err := ustarAccepts((archive.Layout{Depth: depth}).Path(longest)); err != nil { + refusedAt = depth + break + } + } + + if refusedAt == 0 { + t.Fatalf("tar took every depth from %d to %d for %q, so nothing here is a ceiling", + archive.MaxDepth(), archive.MaxDepth()+200, longest) + } + if refusedAt <= archive.MaxDepth() { + t.Fatalf("tar refuses depth %d for %q and this build offers %d - "+ + "the declared ceiling is already past what tar takes", + refusedAt, longest, archive.MaxDepth()) + } +} + +// Asking for directory entries in a flat archive is a refusal naming both +// halves, not a setting that quietly does nothing. +// +// Rule 6 forbids the silence: with depth 0 there are no directories, so the +// archive would come out identical whichever way the box was ticked, and the +// person who ticked it would never learn that. The message has to name both +// keys because from "directory_entries is not allowed" a reader cannot tell +// which of the two to change - and the window makes this pair easy to reach, +// since a checkbox always sends its value. +func TestAskingForDirectoryEntriesInAFlatArchiveNamesBothSettings(t *testing.T) { + _, err := archive.ReadLayout("zip", format.Request{ + Properties: map[string]string{archive.DirectoryEntries: "true"}, + }) + if err == nil { + t.Fatal("directory_entries with depth 0 was accepted, so the setting did nothing and said nothing") + } + + var refusal *format.PropertyValueError + if !errors.As(err, &refusal) { + t.Fatalf("the refusal is %T, so it does not carry the four things a refusal owes a reader", err) + } + said := refusal.Reason + " " + refusal.Remedy + for _, half := range []string{archive.Depth, archive.DirectoryEntries} { + if !strings.Contains(said, half) { + t.Errorf("the refusal never names %q, so a reader cannot tell which half to change:\n %s", + half, said) + } + } +} + +// Flat stays flat, and that is what keeps every existing hash where it is. +// +// The default has to be the layout every archive this tool has written so far +// already had. Anything else moves the bytes of all of them, which is +// untouchable rule 3 - and it would do it without a single recipe changing. +func TestAnArchiveNobodyAskedToNestIsStillFlat(t *testing.T) { + l, err := archive.ReadLayout("zip", format.Request{}) + if err != nil { + t.Fatalf("an archive with nothing asked for was refused: %v", err) + } + if l.Depth != 0 { + t.Errorf("depth defaults to %d, so every archive already written moves", l.Depth) + } + if got := l.Path("txt_0001.txt"); got != "txt_0001.txt" { + t.Errorf("a flat archive puts its entry at %q rather than at the top", got) + } + if dirs := l.Directories(); len(dirs) != 0 { + t.Errorf("a flat archive names %d directories: %v", len(dirs), dirs) + } +} + +// The directories come outermost first, because that is the only order a +// reader can act on. +// +// A reader that creates directories as it meets them cannot make d00/d01 +// before it has made d00. Sorting them the other way produces an archive that +// is correct on paper and fails in a real extractor, which is the kind of +// defect no size check would ever notice. +func TestTheDirectoriesAnArchiveNamesComeOutermostFirst(t *testing.T) { + l, err := archive.ReadLayout("zip", format.Request{Properties: map[string]string{ + archive.Depth: "3", archive.DirectoryEntries: "true", + }}) + if err != nil { + t.Fatalf("a nested archive with directory entries was refused: %v", err) + } + + dirs := l.Directories() + if len(dirs) != 3 { + t.Fatalf("depth 3 names %d directories, want 3: %v", len(dirs), dirs) + } + for i, dir := range dirs { + if !strings.HasSuffix(dir, "/") { + t.Errorf("%q does not end in a slash, so neither container reads it as a directory", dir) + } + if i > 0 && !strings.HasPrefix(dir, dirs[i-1]) { + t.Errorf("%q does not sit inside %q, so the chain is not a chain", dir, dirs[i-1]) + } + } + if want := l.Path(""); dirs[len(dirs)-1] != want { + t.Errorf("the innermost directory is %q and the files live in %q", dirs[len(dirs)-1], want) + } +} + +// ustarAccepts asks tar itself rather than reimplementing the split rule. +// +// The rule is "some slash with at most 155 bytes before it and at most 100 +// after", and writing that out here would be a second implementation to keep +// in step with the standard library - the exact shape O163 punished, where a +// channel measured against five readers said nothing about what Go does. +func ustarAccepts(path string) error { + w := tar.NewWriter(io.Discard) + err := w.WriteHeader(&tar.Header{ + Name: path, Size: 0, Mode: 0o644, Format: tar.FormatUSTAR, + }) + if err != nil { + return err + } + return w.Close() +} + +// A directory entry costs one block and nothing else, which is the whole of +// what the tar arithmetic has to learn. +// +// size.go computes the tar length as 1024 plus, per entry, 512 plus the +// content rounded up to a block. A directory has no content, so it should add +// exactly one block - measured here rather than assumed, because the whole +// exact-size promise rests on that number being right. +func TestADirectoryEntryCostsExactlyOneTarBlock(t *testing.T) { + measure := func(h *tar.Header) int { + var buf bytes.Buffer + w := tar.NewWriter(&buf) + if err := w.WriteHeader(h); err != nil { + t.Fatalf("writing %q: %v", h.Name, err) + } + if err := w.Close(); err != nil { + t.Fatalf("closing after %q: %v", h.Name, err) + } + return buf.Len() + } + + empty := func() int { + var buf bytes.Buffer + w := tar.NewWriter(&buf) + if err := w.Close(); err != nil { + t.Fatalf("closing an empty tar: %v", err) + } + return buf.Len() + }() + + withDir := measure(&tar.Header{ + Name: "d00/", Mode: 0o755, Typeflag: tar.TypeDir, Format: tar.FormatUSTAR, + }) + + if got, want := withDir-empty, 512; got != want { + t.Errorf("a directory entry adds %d bytes, and the size arithmetic assumes %d", got, want) + } +} diff --git a/internal/guard/booleansetting_test.go b/internal/guard/booleansetting_test.go new file mode 100644 index 0000000..84da34d --- /dev/null +++ b/internal/guard/booleansetting_test.go @@ -0,0 +1,92 @@ +package guard + +import ( + "archive/zip" + "path/filepath" + "testing" + + _ "github.com/donislawdev/TestingFilesGenerator/internal/format/all" + "github.com/donislawdev/TestingFilesGenerator/internal/gui/text" + "github.com/donislawdev/TestingFilesGenerator/internal/gui/window" +) + +// A switch on the window has to reach the file, not just the screen. +// +// This is the first true or false setting any format declares. Every other +// declared setting is a box somebody types in or a menu, and both of those +// answer with text - so the path from a SWITCH to the engine had never carried +// anything, and a guard that only compared the screen against the registry +// would have been satisfied by a control that was drawn and then ignored. +// +// It presses Generate rather than Preview, and it reads the archive off the +// disk rather than the plan. A preview writes nothing, and the plan is the +// thing under test saying what it intended - neither could tell a switch that +// works from one that is drawn and dropped. +// +// The two runs differ ONLY in the switch. Same seed, same size, same depth, so +// anything that comes back different is the switch and nothing else. +func TestASwitchOnTheWindowReachesTheFileItDescribes(t *testing.T) { + entriesFor := func(t *testing.T, ticked bool) []string { + t.Helper() + dir := t.TempDir() + + host := newFakeHost(t) + screen := window.NewGenerate(host) + content := screen.Object() + t.Cleanup(func() { join(host) }) + + fields := screen.Fields() + chooserIn(t, fields, "format").SetSelected("zip") + setBox(t, fields, "size", "64kb") + setBox(t, fields, "depth", "2") + setBox(t, fields, "entries", "2") + toggleIn(t, fields, "directory_entries").SetChecked(ticked) + + entryUnder(t, content, text.FieldOutputDir()).SetText(dir) + press(t, content, text.ButtonGenerate()) + join(host) + + made, err := filepath.Glob(filepath.Join(dir, "*.zip")) + if err != nil || len(made) != 1 { + t.Fatalf("the window wrote %v (err %v) with the switch %v, and this guard needs exactly one archive", + made, err, ticked) + } + r, err := zip.OpenReader(made[0]) + if err != nil { + t.Fatalf("the archive the window wrote cannot be opened: %v", err) + } + defer r.Close() + + var names []string + for _, f := range r.File { + names = append(names, f.Name) + } + return names + } + + off := entriesFor(t, false) + on := entriesFor(t, true) + + countDirs := func(names []string) int { + n := 0 + for _, name := range names { + if len(name) > 0 && name[len(name)-1] == '/' { + n++ + } + } + return n + } + + if countDirs(off) != 0 { + t.Errorf("the switch was off and the archive still names %d directory entr(ies): %v", + countDirs(off), off) + } + if got := countDirs(on); got != 2 { + t.Errorf("the switch was ON and the archive names %d directory entries, want 2.\n"+ + " with the switch off: %v\n"+ + " with the switch on: %v\n"+ + "Reason: a switch is the one control kind no format used before this setting, so the\n"+ + "path from it to the engine is the one nothing had ever carried a value along.", + got, off, on) + } +} diff --git a/internal/guard/mutationcoverage_test.go b/internal/guard/mutationcoverage_test.go index a9a44b7..3e20737 100644 --- a/internal/guard/mutationcoverage_test.go +++ b/internal/guard/mutationcoverage_test.go @@ -32,6 +32,15 @@ import ( // means saying out loud that a guard is unproven. The list should only ever get // shorter. var notProvenByMutation = map[string]bool{ + // A directory entry costs one tar block and nothing more. + // + // There is no line of ours under this one to break. It asserts what + // archive/tar does with a directory header, and that number is what the + // tar size arithmetic is built on - so the guard is here to catch the + // standard library moving, which is the one thing a substitution in this + // repository cannot simulate. The five guards that came with it on + // 2026-09-01 are proven by mutation instead. + "TestADirectoryEntryCostsExactlyOneTarBlock": true, // The property belongs to the USTAR header rather than to anything here: // every field of it is fixed width, so the mode and the owner are written // into space already paid for whatever they say. There is no line of ours diff --git a/internal/guard/parity_test.go b/internal/guard/parity_test.go index cdf63c1..ad37903 100644 --- a/internal/guard/parity_test.go +++ b/internal/guard/parity_test.go @@ -124,6 +124,8 @@ var reachableFromTheWindow = []string{ "property:pdf.pages", "property:png.height", "property:png.width", + "property:targz.depth", + "property:targz.directory_entries", "property:targz.entries", "property:targz.entry_mode", "property:targz.entry_owner", @@ -143,6 +145,8 @@ var reachableFromTheWindow = []string{ "property:wav.channels", "property:wav.content", "property:wav.sample_rate", + "property:zip.depth", + "property:zip.directory_entries", "property:zip.encryption", "property:zip.entries", "property:zip.entry_format", diff --git a/web/public/formats/index.html b/web/public/formats/index.html index c73d80d..5c28e47 100644 --- a/web/public/formats/index.html +++ b/web/public/formats/index.html @@ -480,6 +480,16 @@

Settings each format accepts

entry_size a size such as 2mb + + + depth + 0 - 50 + + + + directory_entries + true or false + entry_mode @@ -555,6 +565,16 @@

Settings each format accepts

entry_size a size such as 2mb + + + depth + 0 - 50 + + + + directory_entries + true or false + password diff --git a/web/public/pl/formaty/index.html b/web/public/pl/formaty/index.html index a9da2ec..ebeb413 100644 --- a/web/public/pl/formaty/index.html +++ b/web/public/pl/formaty/index.html @@ -480,6 +480,16 @@

Ustawienia, które przyjmuje każdy format

entry_size rozmiar, na przykład 2mb + + + depth + 0 - 50 + + + + directory_entries + prawda albo fałsz + entry_mode @@ -555,6 +565,16 @@

Ustawienia, które przyjmuje każdy format

entry_size rozmiar, na przykład 2mb + + + depth + 0 - 50 + + + + directory_entries + prawda albo fałsz + password