metal : wind down leftover residency sets at teardown instead of aborting

[AlexCherrypi]

What

On macOS 15+ (Apple Silicon), ggml_metal_rsets_free() does GGML_ASSERT([rsets->data count] == 0), which calls abort() when the Metal device is torn down while residency sets are still registered. The device is freed from a C++ static destructor at process exit (ggml_metal_device_get's function-local static vector), so any app embedding the Metal backend that exits without freeing every Metal buffer first crashes on every quit.

Why it happens

A residency set is added in ggml_metal_buffer_init_* and removed from the collection in exactly one place — ggml_metal_buffer_free(). An application that lets the OS reclaim its model/weights on exit (a common, historically fine pattern) never calls ggml_backend_buffer_free for those buffers, so the collection is non-empty when the device's static destructor runs ggml_metal_rsets_free(), and the assert fires.

The device does not own those buffers and cannot free them from its destructor, so the assert can't be made to legitimately hold from within ggml_metal_rsets_free().

Fix

Make teardown defensive instead of aborting:

1. stop the keep-alive heartbeat (existing d_stop + dispatch_group_wait),
2. wind down residency on any leftover sets — endResidency + removeAllAllocations, mirroring ggml_metal_buffer_rset_free() but without -release (each set is still owned by its not-yet-freed buffer, so releasing here would over-release),
3. then release the collection as before.

The backing buffers are reclaimed by the OS as the process exits. No behavior change when all buffers were freed — the array is empty and the loop is a no-op. Guarded by the existing GGML_METAL_HAS_RESIDENCY_SETS + @available(macOS 15.0, …).

Notes

• The assert was added in metal : add residency sets keep-alive heartbeat #17766 ("metal : add residency sets keep-alive heartbeat"). It's a teardown sanity check, not a correctness/security guard: nothing reads rsets->data after this point, so leftover entries cause no UB — only the abort.
• Same fix submitted to whisper.cpp (which syncs this file from here): metal : wind down leftover residency sets at teardown instead of aborting whisper.cpp#3870.
• Repro: drive the Metal backend on macOS 15+ Apple Silicon and quit the process without explicitly freeing the backend buffers → abort on quit. Surfaced in a real macOS app (live dictation).

Happy to adjust — feedback welcome.

[ggml-gh-bot]

Hi @AlexCherrypi, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.