Repository navigation
fix(engine): keep at most a fixed number of segment files open - #94
Merged
Merged
Conversation
This was referenced Oct 14, 2026
PageStore opened every segment file at startup and kept its FileChannel until the engine closed, so the number of open files grew with the database. Segment files now go through a FilePool that holds at most EngineOptions.maxOpenSegmentFiles handles (256 by default). A file is opened when a write, pread, map or sync needs it, and each of those calls borrows the handle for its own length. When the pool is full the least recently used handle that no call is borrowing is closed, after forcing it if it has unsynced writes. Writes add their file to an unsynced set and sync() forces only the files in it. Sealed segments are still read through their memory map, which does not need the file to stay open, and opening the store reads segment sizes with stat instead of opening every file.
It sets EngineOptions.maxOpenSegmentFiles from hstore.conf, HSTORE_MAX_OPEN_SEGMENT_FILES or --max-open-segment-files. Values below 1 or above Integer.MAX_VALUE fail at startup with a message that names the setting.
venkat1701
force-pushed
the
fix/segment-file-pool
branch
from
October 14, 2026 13:27
7218b7b to
b01b196
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #68
PageStoreopened every segment file at startup and kept aFileChannelfor each one until the engine closed. The open file count grew with the database, and smaller segments (#52) would have made that much worse.Segment files now go through a
FilePoolthat keeps at mostmax_open_segment_filesof them open (256 by default).How it works:
stat. A file is opened the first time a write,pread, map or sync needs it.FileHandlefor its own length. A handle that a call is using is never closed. When the pool is full, the handle unused for the longest time is closed to make room, and if every handle is in use the caller waits for one to come back.sync()forces only the files in that set. Clean files are not touched, open or not.Sealed segments are still read through the read-only memory maps from #77. A map stays valid after its file is closed, so a mapped segment needs no handle for reads, and its handle is closed like any other once it is the least recently used. The pool builds on
FileHandlefrom #87, so interrupted callers still can't close a shared channel.The setting is
EngineOptions.maxOpenSegmentFiles, and the server reads it fromhstore.conf,HSTORE_MAX_OPEN_SEGMENT_FILESor--max-open-segment-files. Values below 1 or aboveInteger.MAX_VALUEfail at startup with a message that names the setting.Open file count
Same probe on both sides (
PageStorewith 4 KiB pages and 16-page segments, 16,000 pages written, so 1,000 segments; descriptors counted from/dev/fdrelative to before the store was opened):The +2 is
pages.dirandpages.ref.Tests
FilePoolTest: least recently used order, a dirty file is forced before it is closed to make room,sync()forces only files written since the last sync, eight threads on a pool of 3 across 16 files never see their channel closed mid-call and never see more than 3 open, and a closed entry is not reopened or recreated.PageStoreTest: more than 100 segments with a pool of 4 stay within the pool (checked with the pool counter and with/dev/fd) before and after a reopen; a segment closed to make room is forced first and the next sync skips it; four readers and a writer share a pool of 2 without seeing a closed file.ServerTest: the setting from the file, the environment and a flag, and the range check.I checked that the tests catch the bug by breaking the code three ways: an unbounded pool fails 3
PageStoreTesttests, closing without forcing fails 2 tests, and letting the pool close a handle that is in use fails the contention test with aClosedChannelException.