Skip to content

feat(engine): start segments at 8 MiB and double them as the store grows - #95

Merged
venkat1701 merged 3 commits into
mainfrom
feat/growing-segments
Oct 14, 2026
Merged

venkat1701 merged 3 commits into
mainfrom
feat/growing-segments

Conversation

@venkat1701

Copy link
Copy Markdown
Collaborator

Closes #52

Stacked on #94 (fix/segment-file-pool). Merge that one first, then retarget this to main.

Every segment used to be pagesPerSegment × pageSize, 256 MiB by default. A store of a few hundred megabytes sat in one or two segments, and the active one is never a compaction victim, so background compaction had little it could empty.

As decided in #52, new segments now start at 8 MiB and double as the store grows, up to 256 MiB:

  • A new segment gets 8 MiB, doubled for as long as the result is at most an eighth of the bytes in the store's other segments (retired segments don't count), and never more than pagesPerSegment × pageSize.
  • With the defaults that means 8 MiB segments up to 128 MiB of data, 16 MiB from 128 MiB, and 256 MiB from 2 GiB on. A 100 GB store still ends up with about 400 segments of 256 MiB plus a few dozen small ones, and Every segment file stays open for the life of the engine #68's pool keeps the open file count bounded either way.
  • After a compaction shrinks the store, new segments get small again.
  • pagesPerSegment keeps its meaning as the cap. The benchmark's --pages-per-segment now sets the largest segment.

No format change. A segment's capacity is not stored anywhere: page addresses are a segment id and a unit offset, and nothing reads a segment's position from its size. On open, the active segment's capacity is worked out again from the other segments. Segments of different sizes sit side by side, a store written with fixed 256 MiB segments opens and keeps working, and the format stays at 8.

PageStore.segmentBytes() now returns the active segment's capacity, so background compaction seals a mostly dead active segment once it is a quarter of its own size, not a quarter of 256 MiB. Studio labels the configured size as "up to N each".

Disk size at scale 1 (#47)

Comparison run --store hstore --scale 1 --durability async --threads 8 --history 64, main (0c847f7) against this branch, two runs each, bytes in the database directory at a clean shutdown:

main run 1 main run 2 branch run 1 branch run 2
after load (disk) 96,041,466 96,254,645 88,299,197 94,461,945
after the churn (disk.churned) 182,787,741 161,597,738 136,919,314 142,941,262
after a full compaction (disk.compacted) 96,073,756 96,170,380 96,040,767 96,368,561

The churn is 20,000 of 100,000 hyperedges deleted plus 20,000 member removals. After it, the branch is 19 to 46 MB smaller. I haven't traced it, but the likely reason is that background compaction can empty small sealed segments while the churn runs, where main has one big active segment it won't touch until it is mostly dead. After a full compaction both sides end at the same 96 MB.

That 96 MB is now mostly not segments. The scale-1 data is much smaller than when #47 was filed, and a full compaction on main already gets back to it because the mostly dead active segment is sealed first. A separate probe that only loads, churns and compacts (no other workloads) shows the split:

main branch
after load: total / segments 80,409,072 / 14,541,628 (1 file) 78,617,088 / 12,749,658 (2 files)
after churn: total / segments 101,150,099 / 26,260,300 (1 file) 103,677,625 / 28,787,769 (4 files)
after compaction: total / segments 82,700,047 / 7,810,124 (1 file) 82,699,401 / 7,809,445 (1 file)

After compaction 46 MB of the 83 MB is the active WAL segment, 12 MB is the change feed and 16 MB is pages.dir plus pages.ref. So what remains of #47 at scale 1 is WAL and feed retention, not segment size.

I also tried letting each segment double on every roll (8, 8, 16, 32, ... MiB, so the active segment is as big as everything before it). One run gave 93,076,874 after load, 161,498,939 after the churn and 97,927,274 after compaction, so after the churn it was no better than main. That is why the new size is held to an eighth of the store.

Tests

PageStoreTest gains three tests: segment sizes follow the rule from the first segment up to the cap and all pages read back after a reopen; a store whose segments were all compacted away starts again at the first size; a store written with fixed segments keeps working when reopened with growing ones. With fixed-size segments (first size set to the cap) the first two fail.

Every segment was pagesPerSegment x pageSize, 256 MiB by default, so a
store of a few hundred megabytes sat in one or two segments and
compaction had nothing it could empty: the active segment is never a
victim.

A new segment now gets 8 MiB, doubled for as long as that stays at most
an eighth of the bytes in the store's other segments, and never more
than pagesPerSegment x pageSize. With the defaults segments are 8 MiB up
to 128 MiB of data, 16 MiB from there and 256 MiB from 2 GiB on. A store
that compaction shrank gets small segments again.

Capacity is not stored. Page addresses are a segment id and a unit
offset, so segments of different sizes can sit side by side, and on open
the active segment's capacity is worked out again from the others. A
store written with fixed 256 MiB segments opens unchanged, and the
storage format stays at 8.
@venkat1701
venkat1701 changed the base branch from fix/segment-file-pool to main October 14, 2026 13:27
@venkat1701
venkat1701 force-pushed the feat/growing-segments branch from a5663b5 to 1b84fd7 Compare October 14, 2026 13:28
@venkat1701
venkat1701 merged commit 64b9666 into main Oct 14, 2026
@venkat1701
venkat1701 deleted the feat/growing-segments branch October 14, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default segment size is too big for small databases

1 participant