Repository navigation
fix(engine): keep shared files open when a caller's thread is interrupted - #87
Merged
Merged
Conversation
A FileChannel closes itself for every thread when one thread is interrupted during I/O on it. FileHandle wraps a channel that several threads share: it clears the caller's interrupt flag for the call and sets it again afterwards, and if the channel was closed anyway, by an interrupt in the middle of the call or on another thread, it reopens the file and runs the call again. Calls are expected to be positional, so running one twice reads or writes the same bytes.
…d callers A commit or read on an interrupted thread closed the segment file it touched, and every thread using that segment failed with ClosedChannelException from then on, including background compaction and the checkpoint at shutdown. Growing pages.dir or pages.ref on an interrupted thread did the same to those files. Do their I/O through FileHandle.
…mmitters Committing on an interrupted thread closed the active write-ahead log segment, so that commit and every later one failed with "write-ahead log append failed" until the process restarted. The change feed's segments had the same problem for commits and for subscribers reading them. Do their I/O through FileHandle.
…read A checkpoint writes short-lived files, the catalog and the reference journal, that an interrupt would fail. Shutdown waited for the maintenance thread with join, which returns at once on an interrupted thread, so files could be closed while maintenance was still using them. Run both with the caller's interrupt flag cleared and set it again at the end. The new test commits, reads, replays the feed and checkpoints from an interrupted thread, keeps committing from another one, closes from an interrupted thread and checks everything is there after reopening.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #86.
Why
The JDK closes a
FileChannelfor every thread when one thread is interrupted during I/O on it. The engine shares its channels between threads, so one interrupted commit closed the write-ahead log. After that, every commit and the shutdown checkpoint failed until the process restarted. Segment files, the change feed,pages.dirandpages.refhad the same problem.What changes
page/FileHandlewraps a channel that several threads share:SegmentFileandMappedLongs(behindpages.dirandpages.ref) use it.joinon the maintenance thread returned straight away on an interrupted thread, so shutdown could close files while maintenance was still using them.docs/storage/pages.mdexplains all this.The WAL reader that recovery uses keeps its own channel per reader. An interrupt there only fails that reader, which is what an interrupt should do.
Testing
FileHandleTest:PageStoreTest: an interrupted thread writes, reads and syncs pages in the growing segment, then another thread keeps using the same segment.PageDirectoryTest: an interrupted thread grows the directory past its first 8 MiB chunk.InterruptTest:SegmentFile,MappedLongs,WriteAheadLog,ChangeFeedandCheckpointer. Every time, at least one of the new tests fails.closestill succeeds in the test, because the maintenance thread happens to be idle.examples/clinical-claims.hql,hstore checkverifies 105 pages and the reference counts match;COMPACT;runs andcheckpasses afterwards.