fix: rollover durability β no malformed archives or lost logs on abrupt app death#427
Draft
tony19 wants to merge 2 commits into
Draft
fix: rollover durability β no malformed archives or lost logs on abrupt app death#427tony19 wants to merge 2 commits into
tony19 wants to merge 2 commits into
Conversation
An app killed during rollover compression left a truncated archive at the final .gz/.zip name: Compressor streamed straight into the target file, so the malformed archive looked complete, was never repaired (retries abort with 'exist already'), and with the file property set the original data was stranded in a nanotime-suffixed .tmp file that nothing ever cleaned. Worse, a compression that failed with an exception still deleted the source file, silently destroying that period's logs. Compression now streams into '<target>.tmp' and atomically renames to the final name only after a fully successful write and close, so a malformed archive can never appear under the final name. The source file is deleted only after the rename succeeds; on any failure the partial temp file is removed and the source is kept. Stale '*.gz.tmp' / '*.zip.tmp' partials from previous kills are swept when a later compression runs in the same directory β only when old enough that they cannot belong to an in-flight compression, and never matching the rename-stage temp files that hold original log data. Fixes #369 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JD28c26fMeRWhJNWyNWk5Q
An ordinary flush hands bytes to the OS page cache, which survives a process kill but not an abrupt device power-off β the kernel may not have written the tail of the log back to flash yet, which is why users lose the last stretch of logs when a device is powered off. Add an opt-in syncOnFlush property to FileAppender (and the schema): when enabled, every flush also syncs the file descriptor to the storage device, so with the default immediateFlush each event is durable the moment the append returns, at the cost of a slower write per event. The stream is also flushed (and synced) before close. Refs #371 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JD28c26fMeRWhJNWyNWk5Q
This was referenced Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third round of issue triage, tackling the rollover durability reports.
No malformed archives or lost logs on interrupted rollover (fixes #369)
The reporter's fleet kills the app right at midnight β mid-rollover β and ends up with malformed
.gzarchives. Three defects inCompressorconspired:.gz/.zipname. A kill mid-compression left a truncated archive that looks complete, is never repaired, and blocks any retry ("exist already. Aborting").fileproperty set, rollover renames the active log to<name>.gz<nanotime>.tmpbefore compressing; a kill leaves that data invisible tomaxHistory/totalSizeCapcleanup forever.The fix:
<target>.tmpand atomically renames to the final name only after a fully successful write and close (close moved in-try, so a failed close counts as a failed compression). A malformed archive can no longer exist under the final name β after a kill you get the intact uncompressed source instead of a corrupt.gz.*.gz.tmp/*.zip.tmppartials from previous kills are swept when a later compression runs in the same directory β only when untouched for 30+ minutes (an in-flight compression writes continuously, so it can't be swept), and the filter can't match the rename-stage<name>.gz<digits>.tmpfiles, which hold original log data.New
CompressorDurabilityTestcovers: success leaves no temp file; failure preserves the source (both GZ and ZIP β this is the regression test for defect 2); an interrupted compression can be retried over its stale temp file; and the sweep removes old partials while sparing fresh temps and rename-stage data files.Opt-in
syncOnFlushfor power-off durability (refs #371)The #371 reporter loses the tail of the log when the device is powered off. That's physics, not a bug: an ordinary flush hands bytes to the OS page cache, which survives a process kill but not a power-off β the kernel may not have written the tail back to flash yet. The only fix is
fsync, which costs real latency, so it's opt-in:When enabled, every flush also syncs the file descriptor to the storage device, so with the default
immediateFlusheach event is durable the moment the append returns. The stream is also flushed (and synced) on close. Added toFileAppender,ResilientFileOutputStream, and the XML schema; a sync failure flows into the existing resilient-recovery path. Documented with the suggestion to pair it withAsyncAppenderwhen write latency matters.Verification
Compiled all main sources and ran the tests standalone against Robolectric's
android-alljar (Gradle/AGP can't run in this sandbox βdl.google.comblocked): the 6 new durability tests, the pre-existingCompressTest(3), the end-to-endTimeBasedRollingTest(10, exercises real GZ/ZIP rollovers through the new code), and the newsyncOnFlushtest all pass; the updated schema still validates configs usingsyncOnFlush. CI runs the full suite.π€ Generated with Claude Code
https://claude.ai/code/session_01JD28c26fMeRWhJNWyNWk5Q
Generated by Claude Code