diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index d689920..6df4852 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -252,7 +252,7 @@ semantic/state 53 B persisted HNSW index state (datab | `LOCK` | `StorageEngine.open` | An exclusive OS file lock. A second process gets `database ... is opened by another process`. | | `catalog/generations/%016x.cat` | `CatalogStore.publish` | `int magic 0x54414348`, `int crc32c(payload)`, `long length`, then the `CatalogImage` payload. Written to `*.tmp`, fsynced, atomically renamed, and the directory fsynced. The three newest files are kept. | | `catalog/manifest` | `CatalogStore` | The file name of the latest `.cat`, written durably the same way. | -| `data/segments/%08x.seg` | `PageStore` / `SegmentFile` | Append-only extents of node images packed at 64-byte alignment, up to 256 MiB per segment by default (1 GiB maximum); see [pages.md](../storage/pages.md). | +| `data/segments/%08x.seg` | `PageStore` / `SegmentFile` | Append-only extents of node images packed at 64-byte alignment, from 8 MiB per segment, doubling as the store grows, up to 256 MiB by default (1 GiB maximum); see [pages.md](../storage/pages.md). | | `data/segments/pages.dir` | `PageDirectory` | 64-byte header, then one `i64` address (segment, offset, units) per page number, memory-mapped in 8 MiB chunks; see [pages.md](../storage/pages.md#page-numbers-and-the-page-directory). | | `data/segments/pages.ref`, `pages.ref.journal` | `ReferenceCounts`, `ReferenceJournal` | One `i64` reference count per page number in the same layout, changed only at checkpoints through the journal; see [pages.md](../storage/pages.md#reference-counts). | | `wal/segments/%016x.wal` | `WriteAheadLog` | Each segment is named by its starting LSN; see [wal-and-recovery.md](../transactions/wal-and-recovery.md). | diff --git a/docs/benchmarks.md b/docs/benchmarks.md index e95645a..706401f 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -116,9 +116,9 @@ then forces a checkpoint so the cleaned files are deleted. The latency table com *Resources* table adds the compaction time and the space reclaimed. Disk is measured at a reopen just before the window and again at a reopen just after it, so segments that HStore only deletes once older readers are gone are counted too. That figure can be negative when compaction frees less than the writer adds. The two windows run at a -fixed rate, so they are left out of the main table, where their ratio would always be about 1. HStore's default segments are 256 MiB, so -at scale 1 most data shares one or two segments. `--pages-per-segment` (`PAGES_PER_SEGMENT`) and -`--compaction-live-ratio` (`COMPACTION_LIVE_RATIO`) change HStore's settings for a run. +fixed rate, so they are left out of the main table, where their ratio would always be about 1. HStore's segments start at 8 MiB and grow +to 256 MiB as the store grows. `--pages-per-segment` (`PAGES_PER_SEGMENT`) sets the largest segment for a run, and +`--compaction-live-ratio` (`COMPACTION_LIVE_RATIO`) the compaction threshold. **Crash recovery.** `reopen` closes the store cleanly first. `recover` doesn't: 1. The run starts a second JVM (`Comparison crash`) with the same flags and classpath. It ingests nodes into a diff --git a/docs/operations/studio.md b/docs/operations/studio.md index a0da61c..9f84e15 100644 --- a/docs/operations/studio.md +++ b/docs/operations/studio.md @@ -34,7 +34,7 @@ flowchart LR | **Console** | HQL editor with syntax highlighting, line numbers, completion of keywords, type, property and role names, and persistent history. Results appear as a stream of frames, newest first. Each frame offers *Graph* (when the result contains atoms), *Table*, *JSON* and *Plan* (when the statement produced a trace) tabs. Multi-statement scripts produce one frame per statement. A failed statement gets a red frame with the error code and stops the script. | | **Explore** | Full-screen hypergraph canvas. You can load a sample of every hyperedge type, search (`42`, `@42`, `Type:key`, `Type:'key'` or a type name), expand atoms, hide types from the legend, switch render modes and toggle labels. An inspector panel shows the selected atom. A generation slider re-renders the same atoms at any retained generation. | | **Schema** | Cards for every node and hyperedge type: kind, visible atom count, properties with `indexed`/`required` badges and value type, JSON indexes, and roles. *Explore* and *Query* buttons on each card, plus a branch table. | -| **Dashboard** | ADMIN only. Polls `/api/stats` every 2 s. Metric tiles show generation, commits, conflicts, group-commit batch size, cache hit rate, data written, WAL and uptime. Live charts show commits/s, pages written/s, pages read/s and WAL throughput. A segment map is coloured by state (active, sealed, compacting, retired) and filled by bytes used relative to the segment size. Hover a segment for its bytes, node images and live images. Configuration and recovery details are listed. | +| **Dashboard** | ADMIN only. Polls `/api/stats` every 2 s. Metric tiles show generation, commits, conflicts, group-commit batch size, cache hit rate, data written, WAL and uptime. Live charts show commits/s, pages written/s, pages read/s and WAL throughput. A segment map is coloured by state (active, sealed, compacting, retired) and filled by bytes used relative to the largest segment size, so the bars also show how segments grew. Hover a segment for its bytes, node images and live images. Configuration and recovery details are listed. | | **Access** | ADMIN only. Tenants (usage and quotas) and users, forms to create users and tenants, and *Drop* for users. Every action runs ordinary HQL (`CREATE USER …`, `CREATE TENANT … QUOTA (…)`, `DROP USER …`) through the session. | The top bar shows the current branch, user, tenant, role and the latest durable generation (polled every @@ -141,7 +141,7 @@ Errors use one JSON shape: | `GET /api/graph` | yes | Hypergraph neighbourhood for visualisation; see below. | | `GET /api/schema` | yes | Types visible to the session: `id`, `name`, `kind`, `version`, `count`, `roles`, `properties[{name,type,indexed,required}]`, `jsonIndexes[{path,type}]`, plus `branches`. | | `GET /api/history` | yes | `{"current": G, "generations": [{"id","wallTime","txnId"}]}` for the retained generations. | -| `GET /api/stats` | ADMIN | Engine counters, segments (`id`, `state`, `pages` = node images, `bytes`, `live`), recovery outcome, configuration (`pageSize`, `segmentBytes`, `durability`, `walMode`, `nodeCacheBytes`, `historyLimit`, `checkpointWalBytes`), semantic index generation, Studio session count, branches. | +| `GET /api/stats` | ADMIN | Engine counters, segments (`id`, `state`, `pages` = node images, `bytes`, `live`), recovery outcome, configuration (`pageSize`, `segmentBytes` (the largest segment), `durability`, `walMode`, `nodeCacheBytes`, `historyLimit`, `checkpointWalBytes`), semantic index generation, Studio session count, branches. | Example `GET /api/info`: diff --git a/docs/storage/maintenance.md b/docs/storage/maintenance.md index 9b2ec34..f59e733 100644 --- a/docs/storage/maintenance.md +++ b/docs/storage/maintenance.md @@ -315,7 +315,7 @@ A failure names the page, for example `HStoreException.corrupt(pageId, "checksum | `compaction_live_ratio` | 0.5 | A sealed segment is compacted when fewer than this fraction of its images are live | | `page_size` | 16384 | Maximum node image size; fixed at `init` | -The segment capacity is `EngineOptions.pagesPerSegment × pageSize`, by default 16 384 × 16 KiB = 256 MiB, capped at 1 GiB by the 24-bit unit offset. +New segments start at 8 MiB and double as the store grows, up to `EngineOptions.pagesPerSegment × pageSize`, by default 16 384 × 16 KiB = 256 MiB and never more than 1 GiB because of the 24-bit unit offset ([pages.md](pages.md#segment-files)). Small stores therefore have many small segments that compaction can empty one at a time. ```sql CHECKPOINT; diff --git a/docs/storage/pages.md b/docs/storage/pages.md index 8095db5..97242ab 100644 --- a/docs/storage/pages.md +++ b/docs/storage/pages.md @@ -244,7 +244,7 @@ xxd -s $((64 + 1234 * 8)) -l 8 -g 8 -e data/segments/pages.ref |---|---| | Location | `/data/segments/%08x.seg`, e.g. `00000001.seg` | | Contents | Node images packed back to back. Each image starts on a 64-byte boundary and occupies ⌈length / 64⌉ units. The bytes after an image up to the next boundary are never written (file holes or zeros). | -| Capacity | `segmentUnits = min(2²⁴, pagesPerSegment × pageSize / 64)`. With the defaults (16 KiB × 16 384) that is 4 194 304 units = 256 MiB. | +| Capacity | Grows with the store. A new segment gets 8 MiB, doubled for as long as the result is at most an eighth of the bytes in the store's other segments (retired ones not counted) and at most `maximumUnits = min(2²⁴, pagesPerSegment × pageSize / 64)`. With the defaults (16 KiB × 16 384) the maximum is 256 MiB, so segments are 8 MiB until the store holds 128 MiB, 16 MiB from there, and 256 MiB from 2 GiB on. A store that shrinks after compaction gets small segments again. | | Mutability | Append-only. A byte range, once written, is never rewritten. Old images become garbage when no retained generation references them, and are removed by deleting whole segments ([maintenance.md](maintenance.md)). | | Id assignment | `segments.lastKey() + 1` | @@ -270,7 +270,7 @@ sequenceDiagram M->>NC: encode(frozen, PageId.NONE, epoch, pageSize) NC-->>M: image (80 + payloadLength bytes, sealed with pageId 0) M->>PS: allocate(image.byteSize()) - PS->>PS: units = ⌈len / 64⌉; roll the segment if nextUnit + units > segmentUnits + PS->>PS: units = ⌈len / 64⌉; roll the segment if nextUnit + units > activeUnits PS->>PS: index = next index; directory[index] = (activeSegment, nextUnit, units) PS-->>M: pack(index, 0); nextUnit += units M->>H: assign(image, pageId): write pageId at offset 8, recompute the crc @@ -279,7 +279,8 @@ sequenceDiagram * `allocate` rejects `length > pageSize`. * Allocation is serialized by a `ReentrantLock`, and only the counter and directory updates run under it. -* Rolling a segment marks the previous one `SEALED`, creates the next id, forces the `data/segments` directory so the new file's entry is durable before any page in it is, and resets `nextUnit` to 0. +* Rolling a segment marks the previous one `SEALED`, creates the next id, sets `activeUnits` to the new segment's capacity, forces the `data/segments` directory so the new file's entry is durable before any page in it is, and resets `nextUnit` to 0. +* A segment's capacity is not recorded anywhere. Page addresses are a segment id and a unit offset, so nothing depends on how big a segment was meant to be. On open, the active segment's capacity is worked out again from the other segments. Segments of different sizes sit side by side, and a store written with fixed 256 MiB segments opens as it is. * `PageStore.write(pageId, image)` looks the number up in the directory and performs a positional `FileChannel.write` at `offset × 64`, then marks the file dirty. It refuses an image longer than the units allocated for it. A checkpoint writes each image as soon as it has allocated it. For the pages a commit writes, the write happens later, inside `TransactionManager.append`, after the WAL records have been appended (see [wal-and-recovery.md](../transactions/wal-and-recovery.md)). * `PageStore.sync()` calls `FileChannel.force(false)` on every segment written since the last sync, and leaves every other segment alone, open or not. The group committer calls it once per batch in `PAGE_REFERENCES` WAL mode, before the WAL sync, if a commit in the batch wrote pages. `PageStore.checkpoint()` does the same and then forces the page directory; only `Checkpointer` calls it. diff --git a/engine/src/main/java/io/hstore/engine/page/PageStore.java b/engine/src/main/java/io/hstore/engine/page/PageStore.java index a1526d3..2dc3d9b 100644 --- a/engine/src/main/java/io/hstore/engine/page/PageStore.java +++ b/engine/src/main/java/io/hstore/engine/page/PageStore.java @@ -28,10 +28,13 @@ public record Scan(long[] dead, long[] unreferenced, long unplaced) { } private static final long PASS_BYTES = 4L << 20; + private static final long FIRST_SEGMENT_BYTES = 8L << 20; + private static final long GROWTH_FRACTION = 8; private final Path directory; private final int pageSize; - private final int segmentUnits; + private final int firstUnits; + private final int maximumUnits; private final FilePool pool; private final Map files = new ConcurrentHashMap<>(); private final ConcurrentSkipListMap segments = new ConcurrentSkipListMap<>(); @@ -48,11 +51,13 @@ public record Scan(long[] dead, long[] unreferenced, long unplaced) { private int releasedCount; private int activeSegment; private int nextUnit; + private int activeUnits; - private PageStore(Path directory, int pageSize, int pagesPerSegment, int openFiles) { + private PageStore(Path directory, int pageSize, long firstSegmentBytes, int pagesPerSegment, int openFiles) { this.directory = directory; this.pageSize = pageSize; - this.segmentUnits = (int) Math.min(PageAddress.MAX_OFFSET + 1L, (long) pagesPerSegment * pageSize / PageId.UNIT_BYTES); + this.maximumUnits = (int) Math.min(PageAddress.MAX_OFFSET + 1L, (long) pagesPerSegment * pageSize / PageId.UNIT_BYTES); + this.firstUnits = (int) Math.min(firstSegmentBytes / PageId.UNIT_BYTES, maximumUnits); this.pool = new FilePool(openFiles); } @@ -62,7 +67,12 @@ public static PageStore open(Path directory, int pageSize, int pagesPerSegment, public static PageStore open(Path directory, int pageSize, int pagesPerSegment, int openFiles, Collection known, long checkpointLsn) { - PageStore store = new PageStore(directory, pageSize, pagesPerSegment, openFiles); + return open(directory, pageSize, FIRST_SEGMENT_BYTES, pagesPerSegment, openFiles, known, checkpointLsn); + } + + static PageStore open(Path directory, int pageSize, long firstSegmentBytes, int pagesPerSegment, int openFiles, + Collection known, long checkpointLsn) { + PageStore store = new PageStore(directory, pageSize, firstSegmentBytes, pagesPerSegment, openFiles); store.restore(known, checkpointLsn); return store; } @@ -102,6 +112,7 @@ private void restore(Collection known, long checkpointLsn) { .ifPresent(info -> { activeSegment = info.id(); nextUnit = Math.toIntExact(info.units()); + activeUnits = unitsForNextSegment(info.id()); }); files.forEach((id, file) -> { if (id != activeSegment) { @@ -236,7 +247,7 @@ private void repoint(long[] moved, int count) { } private long claim(int units) { - if (activeSegment == 0 || nextUnit + units > segmentUnits) { + if (activeSegment == 0 || nextUnit + units > activeUnits) { rollSegment(); } int offset = nextUnit; @@ -254,12 +265,25 @@ private void rollSegment() { } } int id = segments.isEmpty() ? 1 : segments.lastKey() + 1; + activeUnits = unitsForNextSegment(id); files.put(id, SegmentFile.create(id, SegmentFile.pathFor(directory, id), pool)); segments.put(id, new SegmentInfo(id, SegmentState.ACTIVE, 0, 0, 0)); activeSegment = id; nextUnit = 0; } + private int unitsForNextSegment(int id) { + long stored = segments.values().stream() + .filter(info -> info.id() != id && info.state() != SegmentState.RETIRED) + .mapToLong(SegmentInfo::units) + .sum(); + long units = firstUnits; + while (units * 2 <= maximumUnits && units * 2 * GROWTH_FRACTION <= stored) { + units *= 2; + } + return (int) units; + } + private void adjustLive(long address, int sign) { live.computeIfAbsent(PageAddress.segmentOf(address), _ -> new AtomicLong()) .addAndGet(sign * (long) PageAddress.unitsOf(address) * PageId.UNIT_BYTES); @@ -465,7 +489,12 @@ public List segments() { } public long segmentBytes() { - return (long) segmentUnits * PageId.UNIT_BYTES; + allocation.lock(); + try { + return (long) activeUnits * PageId.UNIT_BYTES; + } finally { + allocation.unlock(); + } } public void sealActive(int segment) { diff --git a/engine/src/test/java/io/hstore/engine/page/PageStoreTest.java b/engine/src/test/java/io/hstore/engine/page/PageStoreTest.java index 36131c3..611afc1 100644 --- a/engine/src/test/java/io/hstore/engine/page/PageStoreTest.java +++ b/engine/src/test/java/io/hstore/engine/page/PageStoreTest.java @@ -361,6 +361,94 @@ void readersAndWritersShareASmallPoolWithoutSeeingAClosedFile() throws Exception } } + @Test + void newSegmentsStartSmallAndDoubleAsTheStoreGrowsUpToTheMaximum() { + long first = 64 << 10; + int maximumPages = 256; + Map written = new LinkedHashMap<>(); + List segments; + try (PageStore store = PageStore.open(directory, PAGE_SIZE, first, maximumPages, OPEN_FILES, List.of(), -1)) { + Random random = new Random(13); + for (int i = 0; i < 6_000; i++) { + int before = store.activeSegment(); + byte[] image = image(random, PAGE_SIZE); + long pageId = store.allocate(image.length); + store.write(pageId, MemorySegment.ofArray(image)); + written.put(pageId, image); + int active = store.activeSegment(); + if (active != before) { + long stored = store.segments().stream().filter(info -> info.id() != active).mapToLong(SegmentInfo::bytes).sum(); + long expected = first; + while (expected * 2 <= (long) maximumPages * PAGE_SIZE && expected * 2 * 8 <= stored) { + expected *= 2; + } + assertEquals(expected, store.segmentBytes(), "segment " + active + " after " + stored + " bytes"); + } + } + segments = store.segments(); + } + List sealed = segments.stream().limit(segments.size() - 1).map(SegmentInfo::bytes).toList(); + assertEquals(first, sealed.getFirst()); + assertEquals((long) maximumPages * PAGE_SIZE, sealed.getLast()); + for (long size = first; size <= (long) maximumPages * PAGE_SIZE; size *= 2) { + assertTrue(sealed.contains(size), "no segment of " + size + " bytes in " + sealed); + } + for (int i = 1; i < sealed.size(); i++) { + assertTrue(sealed.get(i) >= sealed.get(i - 1), "segments shrank in " + sealed); + } + try (PageStore reopened = PageStore.open(directory, PAGE_SIZE, first, maximumPages, OPEN_FILES, segments, -1)) { + assertEquals((long) maximumPages * PAGE_SIZE, reopened.segmentBytes()); + written.forEach((pageId, image) -> assertArrayEquals(image, bytes(reopened.read(pageId)))); + } + } + + @Test + void segmentsGetSmallerAgainOnceTheStoreShrinks() { + long first = 64 << 10; + try (PageStore store = PageStore.open(directory, PAGE_SIZE, first, 256, OPEN_FILES, List.of(), -1)) { + while (store.segmentBytes() < 256 * PAGE_SIZE) { + write(store, PAGE_SIZE); + } + int active = store.activeSegment(); + store.sealActive(active); + store.segments().forEach(info -> { + store.transition(info.id(), SegmentState.COMPACTING, 0); + store.transition(info.id(), SegmentState.RETIRED, 1); + store.delete(info.id()); + }); + long pageId = write(store, PAGE_SIZE); + assertEquals(1, store.segments().size()); + assertEquals(first, store.segmentBytes()); + assertArrayEquals(image(new Random(PAGE_SIZE), PAGE_SIZE), bytes(store.read(pageId))); + } + } + + @Test + void aStoreWrittenWithFixedSegmentsKeepsWorkingWithGrowingOnes() { + Map written = new LinkedHashMap<>(); + List segments; + Random random = new Random(17); + try (PageStore store = PageStore.open(directory, PAGE_SIZE, 256L * PAGE_SIZE, 256, OPEN_FILES, List.of(), -1)) { + for (int i = 0; i < 600; i++) { + byte[] image = image(random, PAGE_SIZE); + long pageId = store.allocate(image.length); + store.write(pageId, MemorySegment.ofArray(image)); + written.put(pageId, image); + } + segments = store.segments(); + } + try (PageStore store = PageStore.open(directory, PAGE_SIZE, 16L * PAGE_SIZE, 256, OPEN_FILES, segments, -1)) { + for (int i = 0; i < 600; i++) { + byte[] image = image(random, PAGE_SIZE); + long pageId = store.allocate(image.length); + store.write(pageId, MemorySegment.ofArray(image)); + written.put(pageId, image); + } + written.forEach((pageId, image) -> assertArrayEquals(image, bytes(store.read(pageId)))); + assertTrue(store.segments().stream().map(SegmentInfo::bytes).distinct().count() > 1); + } + } + private static long openDescriptors() throws IOException { try (Stream listing = Files.list(DESCRIPTORS)) { return listing.count(); diff --git a/server/src/main/resources/studio/js/views/dashboard.js b/server/src/main/resources/studio/js/views/dashboard.js index 32f27c9..29a9d80 100644 --- a/server/src/main/resources/studio/js/views/dashboard.js +++ b/server/src/main/resources/studio/js/views/dashboard.js @@ -100,7 +100,7 @@ export class DashboardPage { h('h2', { class: 'section-title' }, 'Storage'), h('div', { class: 'split' }, h('div', { class: 'card' }, - h('div', { class: 'card-head' }, h('h3', {}, 'Data segments'), h('span', { class: 'muted' }, `${count(segments.length)} segments · ${bytes(configuration.segmentBytes)} each`)), + h('div', { class: 'card-head' }, h('h3', {}, 'Data segments'), h('span', { class: 'muted' }, `${count(segments.length)} segments · up to ${bytes(configuration.segmentBytes)} each`)), h('div', { class: 'segments' }, segments.map(segment => { const fill = h('span'); fill.style.height = `${Math.round(Math.min(1, segment.bytes / configuration.segmentBytes) * 100)}%`;