Skip to content

register_record_batches panics ("index out of bounds") on an empty partition #1626

Description

@fangchenli

Describe the bug
SessionContext.register_record_batches raises an unhandled Rust PanicException instead of a normal Python error when given a partition that contains no record batches — for example, the batches of an empty pyarrow.Table, whose to_batches() returns [].

The schema is read from the first batch with unchecked indexing (crates/core/src/context.rs):

let schema = partitions.0[0][0].schema(); // panics when partitions.0[0] is empty

With empty input, partitions.0 is [[]] — one partition holding zero batches — so [0][0] indexes out of bounds and panics across the Python/Rust boundary.

This is the unfixed sibling of #575 / #613, which hit the identical panic on from_arrow_table. That fix added an optional schema to create_dataframe and threads the pyarrow object's schema through, but register_record_batches takes no schema and was left unguarded.

To Reproduce

import datafusion as d, pyarrow as pa
from datafusion import SessionContext

empty = pa.table({"k": pa.array([], pa.string()), "v": pa.array([], pa.float64())})
print(empty.to_batches())  # -> []  (no batches)

ctx = SessionContext()
ctx.register_record_batches("t", [empty.to_batches()])

pyo3_runtime.PanicException: index out of bounds: the len is 0 but the index is 0

Observed on datafusion 54.0.0; the offending line is still present on main.

Expected behavior
A catchable Python error, not a panic. An empty pyarrow.Table loses its schema through to_batches(), so the schema genuinely cannot be recovered at this point — a clear error is the right outcome (matching the graceful handling added for from_arrow_table in #613).

Additional context
The same unchecked partitions.0[0][0] also remains in create_dataframe's no-schema else branch, reachable when it is called directly without a schema. (Out of scope here; noted for completeness.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions