Skip to content

[client] LogRecordReadContext.close() can leak Arrow memory after a failed close#4351

Description

@EchoLee5

Search before asking

  • I searched in the issues and found nothing similar.

Fluss version

main (development), follow-up to #4345 / #4350

Please describe the bug 馃悶

LogRecordReadContext.close() in #4350 always clears vectorSchemaRootMap and nulls unshadedBufferAllocator in a finally block, even when IOUtils.closeAll() failed.

Two consequences:

  1. Failed close is not retryable. The previous implementation only nulled unshadedBufferAllocator after a successful close. If close throws because callers still hold Arrow batches (pollRecordBatch()), a later close() no longer has a reference to the unshaded allocator and cannot release it. The same applies to vector schema roots that failed to close.

  2. Unshaded allocator is closed outside unshadedArrowResourceLock. The close path snapshots the allocator under the lock, then closes it after releasing the lock. getOrCreateUnshadedBufferAllocator() can still observe the old reference and hand it out while close() is closing it. LogRecordReadContext is annotated @ThreadSafe.

synchronized (unshadedArrowResourceLock) {
    if (unshadedBufferAllocator != null) {
        final AutoCloseable unshaded = unshadedBufferAllocator;
        closeables.add(() -> unshaded.close());
    }
}
try {
    IOUtils.closeAll(closeables); // closes unshaded without holding the lock
} finally {
    vectorSchemaRootMap.clear();
    synchronized (unshadedArrowResourceLock) {
        unshadedBufferAllocator = null; // dropped even if close failed
    }
}

Expected

  • Keep allocator/root references when close fails, so a later close() can still release leftover native memory.
  • Close the unshaded allocator while holding unshadedArrowResourceLock, matching getOrCreateUnshadedBufferAllocator().

Are you willing to submit a PR?

  • I am willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions