Skip to content

mouth: sweep orphaned espeak temp dirs at startup - #8

Open
BigpapaWarren wants to merge 1 commit into
jaredrhod:mainfrom
BigpapaWarren:fix/espeak-tempdir-leak-on-kill
Open

mouth: sweep orphaned espeak temp dirs at startup#8
BigpapaWarren wants to merge 1 commit into
jaredrhod:mainfrom
BigpapaWarren:fix/espeak-tempdir-leak-on-kill

Conversation

@BigpapaWarren

Copy link
Copy Markdown

Sweep orphaned espeak temp dirs at startup

The problem

phonemizer copies the espeak shared library into a fresh mkdtemp() for every
backend it builds — espeak-ng keeps its state in globals, and dlopen won't load
the same file twice. Kokoro builds several backends, so one warm() leaves
several of these directories behind.

Cleanup is registered two different ways, and only one of them works reliably.
From phonemizer/backend/espeak/api.py:

# But... weakref implementation does not work on windows so we register
# the cleanup with atexit. This means that, on Windows, all the
# temporary directories created by EspeakAPI instances will remain on
# disk until the Python process exit.
if sys.platform == 'win32':
    atexit.register(self._delete_win32)
else:
    weakref.finalize(self, self._delete, self._library, self._tempdir)

atexit does not run when a process is killed rather than exited. Anything
that launches backtalk from a wrapper and stops it by terminating the process — a
launcher script, a service wrapper, a supervisor, a stop button that calls
taskkill — leaks every directory it ever created.

This is not hypothetical: 60 orphaned directories had accumulated on the
machine where I found it. The count grows by one per backend per start and never
falls. Each holds a ~410 KB copy of espeak-ng.dll.

Why not fix it in phonemizer

Two reasons:

  1. It's arguably not phonemizer's bug — its comment is accurate about what it
    can guarantee, and atexit genuinely cannot cover SIGKILL-equivalent
    termination.
  2. Even as a local workaround it doesn't survive: any install step that runs
    uv sync or pip install -U on launch overwrites site-packages.

Sweeping at backtalk's own startup bounds the total at one run's worth,
regardless of how the previous run ended.

The change

One function, called from warm() immediately before Kokoro creates this run's
directories. 78 lines added, nothing modified or removed.

Why it's safe to point at the shared temp directory

Two independent guarantees, neither of them a heuristic:

  1. Shape check — only a directory whose sole entry is an espeak shared
    library is removed. phonemizer's scratch dirs look exactly like that, and
    nothing else in a temp directory does.
  2. The OS enforces the rest — Windows refuses to delete a loaded DLL, so a
    concurrently running instance's directory fails the rmtree and is skipped.
    The code doesn't have to detect concurrent instances; it cannot win that race
    because the filesystem won't let it.

shutil.rmtree failures are caught and ignored: a directory still in use is left
for the next start, which is the correct outcome.

Testing

Run against a live instance with four in-use directories present:

synthetic orphans removed : 3/3     <- want 3
decoy (multi-file) kept   : True    <- want True
decoy (non-espeak) kept   : True    <- want True
decoy (empty dir) kept    : True    <- want True
LIVE voice-line dirs kept : 4/4     <- want all (DLL is loaded)

The last line is the one that matters: the sweep ran with four in-use
directories sitting in the same temp folder and left every one alone.

Confirmed in production across three restarts — the log line appears, the count
returns to one run's worth each time instead of climbing:

11:45:11 [mouth] swept 4 orphaned espeak temp dir(s)
12:00:20 [mouth] swept 4 orphaned espeak temp dir(s)

Notes

  • Windows is where the leak is unbounded, but the sweep is platform-neutral and
    harmless on POSIX, where weakref.finalize already keeps the count near zero.
  • It logs only when it actually removes something, so a clean system stays quiet.
  • No new dependencies; shutil and tempfile are stdlib.

phonemizer copies the espeak shared library into a fresh mkdtemp() for
every backend it builds, because espeak-ng keeps its state in globals and
dlopen will not load the same file twice. Kokoro builds several backends,
so one warm() leaves several of these directories behind.

On POSIX that cleanup rides on weakref.finalize and happens promptly. On
Windows phonemizer can only register it with atexit, and says so itself:

    But... weakref implementation does not work on windows so we register
    the cleanup with atexit. This means that, on Windows, all the
    temporary directories created by EspeakAPI instances will remain on
    disk until the Python process exit.
    -- phonemizer/backend/espeak/api.py

atexit does not run when a process is killed rather than exited. Anything
that launches backtalk from a wrapper and stops it by terminating the
process - a launcher script, a service wrapper, a supervisor - therefore
leaks every directory it ever created. On the machine this was found on,
sixty had accumulated; the count grows by one per backend per start and
never falls.

Patching site-packages is not a fix: an installer that runs `uv sync` or
`pip install -U` on launch overwrites it. Sweeping at our own startup
bounds the total at one run's worth instead.

Safety does not rest on heuristics:

  1. only a directory whose sole entry is an espeak shared library is
     removed - phonemizer's scratch dirs look exactly like that, and it is
     a shape nothing else in the temp directory shares;
  2. Windows refuses to delete a loaded DLL, so a concurrently running
     instance's directory fails the rmtree and is skipped. The OS enforces
     that; this code does not have to detect it.

Verified against a live instance with four in-use directories present:
three synthetic orphans removed, three decoy directories (multi-file,
non-espeak, empty) untouched, and all four in-use directories left alone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant