Repository navigation
feat(engine): start segments at 8 MiB and double them as the store grows - #95
Merged
Merged
Conversation
Every segment was pagesPerSegment x pageSize, 256 MiB by default, so a store of a few hundred megabytes sat in one or two segments and compaction had nothing it could empty: the active segment is never a victim. A new segment now gets 8 MiB, doubled for as long as that stays at most an eighth of the bytes in the store's other segments, and never more than pagesPerSegment x pageSize. With the defaults segments are 8 MiB up to 128 MiB of data, 16 MiB from there and 256 MiB from 2 GiB on. A store that compaction shrank gets small segments again. Capacity is not stored. Page addresses are a segment id and a unit offset, so segments of different sizes can sit side by side, and on open the active segment's capacity is worked out again from the others. A store written with fixed 256 MiB segments opens unchanged, and the storage format stays at 8.
venkat1701
force-pushed
the
feat/growing-segments
branch
from
October 14, 2026 13:28
a5663b5 to
1b84fd7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #52
Stacked on #94 (
fix/segment-file-pool). Merge that one first, then retarget this to main.Every segment used to be
pagesPerSegment × pageSize, 256 MiB by default. A store of a few hundred megabytes sat in one or two segments, and the active one is never a compaction victim, so background compaction had little it could empty.As decided in #52, new segments now start at 8 MiB and double as the store grows, up to 256 MiB:
pagesPerSegment × pageSize.pagesPerSegmentkeeps its meaning as the cap. The benchmark's--pages-per-segmentnow sets the largest segment.No format change. A segment's capacity is not stored anywhere: page addresses are a segment id and a unit offset, and nothing reads a segment's position from its size. On open, the active segment's capacity is worked out again from the other segments. Segments of different sizes sit side by side, a store written with fixed 256 MiB segments opens and keeps working, and the format stays at 8.
PageStore.segmentBytes()now returns the active segment's capacity, so background compaction seals a mostly dead active segment once it is a quarter of its own size, not a quarter of 256 MiB. Studio labels the configured size as "up to N each".Disk size at scale 1 (#47)
Comparison run --store hstore --scale 1 --durability async --threads 8 --history 64, main (0c847f7) against this branch, two runs each, bytes in the database directory at a clean shutdown:disk)disk.churned)disk.compacted)The churn is 20,000 of 100,000 hyperedges deleted plus 20,000 member removals. After it, the branch is 19 to 46 MB smaller. I haven't traced it, but the likely reason is that background compaction can empty small sealed segments while the churn runs, where main has one big active segment it won't touch until it is mostly dead. After a full compaction both sides end at the same 96 MB.
That 96 MB is now mostly not segments. The scale-1 data is much smaller than when #47 was filed, and a full compaction on main already gets back to it because the mostly dead active segment is sealed first. A separate probe that only loads, churns and compacts (no other workloads) shows the split:
After compaction 46 MB of the 83 MB is the active WAL segment, 12 MB is the change feed and 16 MB is
pages.dirpluspages.ref. So what remains of #47 at scale 1 is WAL and feed retention, not segment size.I also tried letting each segment double on every roll (8, 8, 16, 32, ... MiB, so the active segment is as big as everything before it). One run gave 93,076,874 after load, 161,498,939 after the churn and 97,927,274 after compaction, so after the churn it was no better than main. That is why the new size is held to an eighth of the store.
Tests
PageStoreTestgains three tests: segment sizes follow the rule from the first segment up to the cap and all pages read back after a reopen; a store whose segments were all compacted away starts again at the first size; a store written with fixed segments keeps working when reopened with growing ones. With fixed-size segments (first size set to the cap) the first two fail.