mouth: sweep orphaned espeak temp dirs at startup - #8
Open
BigpapaWarren wants to merge 1 commit into
Open
Conversation
phonemizer copies the espeak shared library into a fresh mkdtemp() for
every backend it builds, because espeak-ng keeps its state in globals and
dlopen will not load the same file twice. Kokoro builds several backends,
so one warm() leaves several of these directories behind.
On POSIX that cleanup rides on weakref.finalize and happens promptly. On
Windows phonemizer can only register it with atexit, and says so itself:
But... weakref implementation does not work on windows so we register
the cleanup with atexit. This means that, on Windows, all the
temporary directories created by EspeakAPI instances will remain on
disk until the Python process exit.
-- phonemizer/backend/espeak/api.py
atexit does not run when a process is killed rather than exited. Anything
that launches backtalk from a wrapper and stops it by terminating the
process - a launcher script, a service wrapper, a supervisor - therefore
leaks every directory it ever created. On the machine this was found on,
sixty had accumulated; the count grows by one per backend per start and
never falls.
Patching site-packages is not a fix: an installer that runs `uv sync` or
`pip install -U` on launch overwrites it. Sweeping at our own startup
bounds the total at one run's worth instead.
Safety does not rest on heuristics:
1. only a directory whose sole entry is an espeak shared library is
removed - phonemizer's scratch dirs look exactly like that, and it is
a shape nothing else in the temp directory shares;
2. Windows refuses to delete a loaded DLL, so a concurrently running
instance's directory fails the rmtree and is skipped. The OS enforces
that; this code does not have to detect it.
Verified against a live instance with four in-use directories present:
three synthetic orphans removed, three decoy directories (multi-file,
non-espeak, empty) untouched, and all four in-use directories left alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sweep orphaned espeak temp dirs at startup
The problem
phonemizercopies the espeak shared library into a freshmkdtemp()for everybackend it builds — espeak-ng keeps its state in globals, and
dlopenwon't loadthe same file twice. Kokoro builds several backends, so one
warm()leavesseveral of these directories behind.
Cleanup is registered two different ways, and only one of them works reliably.
From
phonemizer/backend/espeak/api.py:atexitdoes not run when a process is killed rather than exited. Anythingthat launches backtalk from a wrapper and stops it by terminating the process — a
launcher script, a service wrapper, a supervisor, a stop button that calls
taskkill— leaks every directory it ever created.This is not hypothetical: 60 orphaned directories had accumulated on the
machine where I found it. The count grows by one per backend per start and never
falls. Each holds a ~410 KB copy of
espeak-ng.dll.Why not fix it in phonemizer
Two reasons:
can guarantee, and
atexitgenuinely cannot coverSIGKILL-equivalenttermination.
uv syncorpip install -Uon launch overwritessite-packages.Sweeping at backtalk's own startup bounds the total at one run's worth,
regardless of how the previous run ended.
The change
One function, called from
warm()immediately before Kokoro creates this run'sdirectories. 78 lines added, nothing modified or removed.
Why it's safe to point at the shared temp directory
Two independent guarantees, neither of them a heuristic:
library is removed. phonemizer's scratch dirs look exactly like that, and
nothing else in a temp directory does.
concurrently running instance's directory fails the
rmtreeand is skipped.The code doesn't have to detect concurrent instances; it cannot win that race
because the filesystem won't let it.
shutil.rmtreefailures are caught and ignored: a directory still in use is leftfor the next start, which is the correct outcome.
Testing
Run against a live instance with four in-use directories present:
The last line is the one that matters: the sweep ran with four in-use
directories sitting in the same temp folder and left every one alone.
Confirmed in production across three restarts — the log line appears, the count
returns to one run's worth each time instead of climbing:
Notes
harmless on POSIX, where
weakref.finalizealready keeps the count near zero.shutilandtempfileare stdlib.