feat(engine): start segments at 8 MiB and double them as the store grows

docs/architecture/overview.md

@@ -252,7 +252,7 @@ semantic/state                           53 B  persisted HNSW index state (datab
 | `LOCK` | `StorageEngine.open` | An exclusive OS file lock. A second process gets `database ... is opened by another process`. |
 | `catalog/generations/%016x.cat` | `CatalogStore.publish` | `int magic 0x54414348`, `int crc32c(payload)`, `long length`, then the `CatalogImage` payload. Written to `*.tmp`, fsynced, atomically renamed, and the directory fsynced. The three newest files are kept. |
 | `catalog/manifest` | `CatalogStore` | The file name of the latest `.cat`, written durably the same way. |
-| `data/segments/%08x.seg` | `PageStore` / `SegmentFile` | Append-only extents of node images packed at 64-byte alignment, up to 256 MiB per segment by default (1 GiB maximum); see [pages.md](../storage/pages.md). |
+| `data/segments/%08x.seg` | `PageStore` / `SegmentFile` | Append-only extents of node images packed at 64-byte alignment, from 8 MiB per segment, doubling as the store grows, up to 256 MiB by default (1 GiB maximum); see [pages.md](../storage/pages.md). |
 | `data/segments/pages.dir` | `PageDirectory` | 64-byte header, then one `i64` address (segment, offset, units) per page number, memory-mapped in 8 MiB chunks; see [pages.md](../storage/pages.md#page-numbers-and-the-page-directory). |
 | `data/segments/pages.ref`, `pages.ref.journal` | `ReferenceCounts`, `ReferenceJournal` | One `i64` reference count per page number in the same layout, changed only at checkpoints through the journal; see [pages.md](../storage/pages.md#reference-counts). |
 | `wal/segments/%016x.wal` | `WriteAheadLog` | Each segment is named by its starting LSN; see [wal-and-recovery.md](../transactions/wal-and-recovery.md). |

docs/benchmarks.md

@@ -116,9 +116,9 @@ then forces a checkpoint so the cleaned files are deleted. The latency table com
 *Resources* table adds the compaction time and the space reclaimed. Disk is measured at a reopen just before the
 window and again at a reopen just after it, so segments that HStore only deletes once older readers are gone are
 counted too. That figure can be negative when compaction frees less than the writer adds. The two windows run at a
-fixed rate, so they are left out of the main table, where their ratio would always be about 1. HStore's default segments are 256 MiB, so
-at scale 1 most data shares one or two segments. `--pages-per-segment` (`PAGES_PER_SEGMENT`) and
-`--compaction-live-ratio` (`COMPACTION_LIVE_RATIO`) change HStore's settings for a run.
+fixed rate, so they are left out of the main table, where their ratio would always be about 1. HStore's segments start at 8 MiB and grow
+to 256 MiB as the store grows. `--pages-per-segment` (`PAGES_PER_SEGMENT`) sets the largest segment for a run, and
+`--compaction-live-ratio` (`COMPACTION_LIVE_RATIO`) the compaction threshold.
 
 **Crash recovery.** `reopen` closes the store cleanly first. `recover` doesn't:
 1. The run starts a second JVM (`Comparison crash`) with the same flags and classpath. It ingests nodes into a

docs/operations/studio.md

@@ -34,7 +34,7 @@ flowchart LR
 | **Console** | HQL editor with syntax highlighting, line numbers, completion of keywords, type, property and role names, and persistent history. Results appear as a stream of frames, newest first. Each frame offers *Graph* (when the result contains atoms), *Table*, *JSON* and *Plan* (when the statement produced a trace) tabs. Multi-statement scripts produce one frame per statement. A failed statement gets a red frame with the error code and stops the script. |
 | **Explore** | Full-screen hypergraph canvas. You can load a sample of every hyperedge type, search (`42`, `@42`, `Type:key`, `Type:'key'` or a type name), expand atoms, hide types from the legend, switch render modes and toggle labels. An inspector panel shows the selected atom. A generation slider re-renders the same atoms at any retained generation. |
 | **Schema** | Cards for every node and hyperedge type: kind, visible atom count, properties with `indexed`/`required` badges and value type, JSON indexes, and roles. *Explore* and *Query* buttons on each card, plus a branch table. |
-| **Dashboard** | ADMIN only. Polls `/api/stats` every 2 s. Metric tiles show generation, commits, conflicts, group-commit batch size, cache hit rate, data written, WAL and uptime. Live charts show commits/s, pages written/s, pages read/s and WAL throughput. A segment map is coloured by state (active, sealed, compacting, retired) and filled by bytes used relative to the segment size. Hover a segment for its bytes, node images and live images. Configuration and recovery details are listed. |
+| **Dashboard** | ADMIN only. Polls `/api/stats` every 2 s. Metric tiles show generation, commits, conflicts, group-commit batch size, cache hit rate, data written, WAL and uptime. Live charts show commits/s, pages written/s, pages read/s and WAL throughput. A segment map is coloured by state (active, sealed, compacting, retired) and filled by bytes used relative to the largest segment size, so the bars also show how segments grew. Hover a segment for its bytes, node images and live images. Configuration and recovery details are listed. |
 | **Access** | ADMIN only. Tenants (usage and quotas) and users, forms to create users and tenants, and *Drop* for users. Every action runs ordinary HQL (`CREATE USER …`, `CREATE TENANT … QUOTA (…)`, `DROP USER …`) through the session. |
 
 The top bar shows the current branch, user, tenant, role and the latest durable generation (polled every
@@ -141,7 +141,7 @@ Errors use one JSON shape:
 | `GET /api/graph` | yes | Hypergraph neighbourhood for visualisation; see below. |
 | `GET /api/schema` | yes | Types visible to the session: `id`, `name`, `kind`, `version`, `count`, `roles`, `properties[{name,type,indexed,required}]`, `jsonIndexes[{path,type}]`, plus `branches`. |
 | `GET /api/history` | yes | `{"current": G, "generations": [{"id","wallTime","txnId"}]}` for the retained generations. |
-| `GET /api/stats` | ADMIN | Engine counters, segments (`id`, `state`, `pages` = node images, `bytes`, `live`), recovery outcome, configuration (`pageSize`, `segmentBytes`, `durability`, `walMode`, `nodeCacheBytes`, `historyLimit`, `checkpointWalBytes`), semantic index generation, Studio session count, branches. |
+| `GET /api/stats` | ADMIN | Engine counters, segments (`id`, `state`, `pages` = node images, `bytes`, `live`), recovery outcome, configuration (`pageSize`, `segmentBytes` (the largest segment), `durability`, `walMode`, `nodeCacheBytes`, `historyLimit`, `checkpointWalBytes`), semantic index generation, Studio session count, branches. |
 
 Example `GET /api/info`:
 

docs/storage/maintenance.md

@@ -315,7 +315,7 @@ A failure names the page, for example `HStoreException.corrupt(pageId, "checksum
 | `compaction_live_ratio` | 0.5 | A sealed segment is compacted when fewer than this fraction of its images are live |
 | `page_size` | 16384 | Maximum node image size; fixed at `init` |
 
-The segment capacity is `EngineOptions.pagesPerSegment × pageSize`, by default 16 384 × 16 KiB = 256 MiB, capped at 1 GiB by the 24-bit unit offset.
+New segments start at 8 MiB and double as the store grows, up to `EngineOptions.pagesPerSegment × pageSize`, by default 16 384 × 16 KiB = 256 MiB and never more than 1 GiB because of the 24-bit unit offset ([pages.md](pages.md#segment-files)). Small stores therefore have many small segments that compaction can empty one at a time.
 
 ```sql
 CHECKPOINT;

docs/storage/pages.md

@@ -244,7 +244,7 @@ xxd -s $((64 + 1234 * 8)) -l 8 -g 8 -e data/segments/pages.ref
 |---|---|
 | Location | `<data>/data/segments/%08x.seg`, e.g. `00000001.seg` |
 | Contents | Node images packed back to back. Each image starts on a 64-byte boundary and occupies ⌈length / 64⌉ units. The bytes after an image up to the next boundary are never written (file holes or zeros). |
-| Capacity | `segmentUnits = min(2²⁴, pagesPerSegment × pageSize / 64)`. With the defaults (16 KiB × 16 384) that is 4 194 304 units = 256 MiB. |
+| Capacity | Grows with the store. A new segment gets 8 MiB, doubled for as long as the result is at most an eighth of the bytes in the store's other segments (retired ones not counted) and at most `maximumUnits = min(2²⁴, pagesPerSegment × pageSize / 64)`. With the defaults (16 KiB × 16 384) the maximum is 256 MiB, so segments are 8 MiB until the store holds 128 MiB, 16 MiB from there, and 256 MiB from 2 GiB on. A store that shrinks after compaction gets small segments again. |
 | Mutability | Append-only. A byte range, once written, is never rewritten. Old images become garbage when no retained generation references them, and are removed by deleting whole segments ([maintenance.md](maintenance.md)). |
 | Id assignment | `segments.lastKey() + 1` |
 
@@ -270,7 +270,7 @@ sequenceDiagram
     M->>NC: encode(frozen, PageId.NONE, epoch, pageSize)
     NC-->>M: image (80 + payloadLength bytes, sealed with pageId 0)
     M->>PS: allocate(image.byteSize())
-    PS->>PS: units = ⌈len / 64⌉; roll the segment if nextUnit + units > segmentUnits
+    PS->>PS: units = ⌈len / 64⌉; roll the segment if nextUnit + units > activeUnits
     PS->>PS: index = next index; directory[index] = (activeSegment, nextUnit, units)
     PS-->>M: pack(index, 0); nextUnit += units
     M->>H: assign(image, pageId): write pageId at offset 8, recompute the crc
@@ -279,7 +279,8 @@ sequenceDiagram
 
 * `allocate` rejects `length > pageSize`.
 * Allocation is serialized by a `ReentrantLock`, and only the counter and directory updates run under it.
-* Rolling a segment marks the previous one `SEALED`, creates the next id, forces the `data/segments` directory so the new file's entry is durable before any page in it is, and resets `nextUnit` to 0.
+* Rolling a segment marks the previous one `SEALED`, creates the next id, sets `activeUnits` to the new segment's capacity, forces the `data/segments` directory so the new file's entry is durable before any page in it is, and resets `nextUnit` to 0.
+* A segment's capacity is not recorded anywhere. Page addresses are a segment id and a unit offset, so nothing depends on how big a segment was meant to be. On open, the active segment's capacity is worked out again from the other segments. Segments of different sizes sit side by side, and a store written with fixed 256 MiB segments opens as it is.
 * `PageStore.write(pageId, image)` looks the number up in the directory and performs a positional `FileChannel.write` at `offset × 64`, then marks the file dirty. It refuses an image longer than the units allocated for it. A checkpoint writes each image as soon as it has allocated it. For the pages a commit writes, the write happens later, inside `TransactionManager.append`, after the WAL records have been appended (see [wal-and-recovery.md](../transactions/wal-and-recovery.md)).
 * `PageStore.sync()` calls `FileChannel.force(false)` on every segment written since the last sync, and leaves every other segment alone, open or not. The group committer calls it once per batch in `PAGE_REFERENCES` WAL mode, before the WAL sync, if a commit in the batch wrote pages. `PageStore.checkpoint()` does the same and then forces the page directory; only `Checkpointer` calls it.
 

engine/src/main/java/io/hstore/engine/page/PageStore.java

@@ -28,10 +28,13 @@ public record Scan(long[] dead, long[] unreferenced, long unplaced) {
     }
 
     private static final long PASS_BYTES = 4L << 20;
+    private static final long FIRST_SEGMENT_BYTES = 8L << 20;
+    private static final long GROWTH_FRACTION = 8;
 
     private final Path directory;
     private final int pageSize;
-    private final int segmentUnits;
+    private final int firstUnits;
+    private final int maximumUnits;
     private final FilePool pool;
     private final Map<Integer, SegmentFile> files = new ConcurrentHashMap<>();
     private final ConcurrentSkipListMap<Integer, SegmentInfo> segments = new ConcurrentSkipListMap<>();
@@ -48,11 +51,13 @@ public record Scan(long[] dead, long[] unreferenced, long unplaced) {
     private int releasedCount;
     private int activeSegment;
     private int nextUnit;
+    private int activeUnits;
 
-    private PageStore(Path directory, int pageSize, int pagesPerSegment, int openFiles) {
+    private PageStore(Path directory, int pageSize, long firstSegmentBytes, int pagesPerSegment, int openFiles) {
         this.directory = directory;
         this.pageSize = pageSize;
-        this.segmentUnits = (int) Math.min(PageAddress.MAX_OFFSET + 1L, (long) pagesPerSegment * pageSize / PageId.UNIT_BYTES);
+        this.maximumUnits = (int) Math.min(PageAddress.MAX_OFFSET + 1L, (long) pagesPerSegment * pageSize / PageId.UNIT_BYTES);
+        this.firstUnits = (int) Math.min(firstSegmentBytes / PageId.UNIT_BYTES, maximumUnits);
         this.pool = new FilePool(openFiles);
     }
 
@@ -62,7 +67,12 @@ public static PageStore open(Path directory, int pageSize, int pagesPerSegment,
 
     public static PageStore open(Path directory, int pageSize, int pagesPerSegment, int openFiles, Collection<SegmentInfo> known,
                                  long checkpointLsn) {
-        PageStore store = new PageStore(directory, pageSize, pagesPerSegment, openFiles);
+        return open(directory, pageSize, FIRST_SEGMENT_BYTES, pagesPerSegment, openFiles, known, checkpointLsn);
+    }
+
+    static PageStore open(Path directory, int pageSize, long firstSegmentBytes, int pagesPerSegment, int openFiles,
+                          Collection<SegmentInfo> known, long checkpointLsn) {
+        PageStore store = new PageStore(directory, pageSize, firstSegmentBytes, pagesPerSegment, openFiles);
         store.restore(known, checkpointLsn);
         return store;
     }
@@ -102,6 +112,7 @@ private void restore(Collection<SegmentInfo> known, long checkpointLsn) {
                 .ifPresent(info -> {
                     activeSegment = info.id();
                     nextUnit = Math.toIntExact(info.units());
+                    activeUnits = unitsForNextSegment(info.id());
                 });
         files.forEach((id, file) -> {
             if (id != activeSegment) {
@@ -236,7 +247,7 @@ private void repoint(long[] moved, int count) {
     }
 
     private long claim(int units) {
-        if (activeSegment == 0 || nextUnit + units > segmentUnits) {
+        if (activeSegment == 0 || nextUnit + units > activeUnits) {
             rollSegment();
         }
         int offset = nextUnit;
@@ -254,12 +265,25 @@ private void rollSegment() {
             }
         }
         int id = segments.isEmpty() ? 1 : segments.lastKey() + 1;
+        activeUnits = unitsForNextSegment(id);
         files.put(id, SegmentFile.create(id, SegmentFile.pathFor(directory, id), pool));
         segments.put(id, new SegmentInfo(id, SegmentState.ACTIVE, 0, 0, 0));
         activeSegment = id;
         nextUnit = 0;
     }
 
+    private int unitsForNextSegment(int id) {
+        long stored = segments.values().stream()
+                .filter(info -> info.id() != id && info.state() != SegmentState.RETIRED)
+                .mapToLong(SegmentInfo::units)
+                .sum();
+        long units = firstUnits;
+        while (units * 2 <= maximumUnits && units * 2 * GROWTH_FRACTION <= stored) {
+            units *= 2;
+        }
+        return (int) units;
+    }
+
     private void adjustLive(long address, int sign) {
         live.computeIfAbsent(PageAddress.segmentOf(address), _ -> new AtomicLong())
                 .addAndGet(sign * (long) PageAddress.unitsOf(address) * PageId.UNIT_BYTES);
@@ -465,7 +489,12 @@ public List<SegmentInfo> segments() {
     }
 
     public long segmentBytes() {
-        return (long) segmentUnits * PageId.UNIT_BYTES;
+        allocation.lock();
+        try {
+            return (long) activeUnits * PageId.UNIT_BYTES;
+        } finally {
+            allocation.unlock();
+        }
     }
 
     public void sealActive(int segment) {