Repository navigation
Stop allocating per frame when the WAL is opened - #67
Merged
Merged
Conversation
Opening the log scans the last segment to find the end of the last valid frame. The reader allocated a new header and body buffer for every frame and decoded every record, copying page images it then threw away. It now reuses its buffers and, for that scan, only checks frames.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #58
I profiled an open of a clean scale-1 database with JFR. Most of what isn't JVM startup came from
WriteAheadLog.restore. It scans the last segment frame by frame to find where valid data ends, and for every frame it:ByteBufferpositioned.next()Now each reader reuses one header buffer and one body buffer (growing the body only for a bigger frame), and the restore scan only checks headers and CRCs without decoding. Normal reads (
read(from), used by recovery) still decode everything. Buffer reuse is safe becauseWalRecord.readPayloadcopies everything it keeps (getBytescopies; the rest are primitives).Measured on the same data:
reopen(in-process close and open, same harness, 2 runs each)Open time didn't change (about 0.4 to 0.55 s); it's dominated by reading and checksumming. Neither version paused for GC in these reopen runs, so the 0.86 s pause in the issue didn't reproduce here.
The change-feed restore still allocates a payload array per frame (about 16 MiB here). Its frames hand the array to consumers, so reusing it isn't safe without a separate validate-only path. I left that out.
New
WriteAheadLogTest, which passes on both the old and the new code:CrashRecoveryTest(16 cases) passed three runs in a row, and the full./mvnw installpasses (95 tests).