Support writing Arrow RecordBatchReader or Scanner to Iceberg tables

[chitralverma]

Feature Request / Improvement

Summary

Please consider adding support in pyiceberg for writing data to Iceberg tables using streamable Arrow-native types such as

• pyarrow.RecordBatchReader
• Iterator[pyarrow.RecordBatch]
• pyarrow.RecordBatch
• pyarrow.dataset.Scanner
• pyarrow.Table (existing or fallback)

Operations could include:

table.append(record_batch_reader)
table.overwrite(scanner)
table.upsert(scanner, primary_keys=["id"])

---

Motivation

Currently, writing data into Iceberg via Python requires materializing data entirely in memory (e.g., via pyarrow.Table) and converting it to Parquet manually. This limits scalability and performance, especially for:

• Large datasets that exceed memory
• Incremental / streaming ingestion
• Lazy pipelines using DuckDB, ADBC, or Scanner.from_batches(...)

RecordBatchReader and Scanner are both streamable abstractions ideal for these use cases.

Benefits

• Enables lazy and streaming by avoiding unnecessary disk I/O and memory pressure
• Avoids intermediate Parquet files, temp storage and big data pipelines without requiring full materialization
• Enables clean integration with Arrow-native tools (e.g., ADBC, DuckDB, pyarrow ecosystem)
• Avoids unnecessary disk I/O and memory pressure

Related Context

• delta-rs supports Arrow-native ingestion via ArrowStreamExportable, ArrowArrayExportable, and sequences of arrays

---

This feature would unlock efficient Python-native data ingestion workflows for Iceberg and align pyiceberg more closely with the rest of the Arrow ecosystem.

[chitralverma]

another ref: #402

[jayceslesar]

You can currently achieve this by something like...

for batch in my_batch_iterator:
    # would need to check schema here
    table.append(pa.Table.from_batches([batch]))

Not saying that this isnt a feature that would be nice but the ability to make it happen is already possible

[chitralverma]

You can currently achieve this by something like...

for batch in my_batch_iterator:
    # would need to check schema here
    table.append(pa.Table.from_batches([batch]))

Not saying that this isnt a feature that would be nice but the ability to make it happen is already possible

this is what I'm doing at the moment

[djouallah]

honestly it make pyiceberg not usable in any relatively big dataset
#1004

this does not works with overwrite

for batch in my_batch_iterator:
    # would need to check schema here
    table.append(pa.Table.from_batches([batch]))

[jayceslesar]

I think this is really hard to do without fully materializing the table and would take a very long time to complete if you are writing a large stream of record batches.

We would have to process the entire stream of batches here

iceberg-python/pyiceberg/io/pyarrow.py

Line 2711 in dc43940

def _determine_partitions(spec: PartitionSpec, schema: Schema, arrow_table: pa.Table) -> List[_TablePartition]:

at least, and then re-sink everything to files per partition, then process all of those temporary files that now need to be cleaned up as a stream.

You are correct that it does not work with overwrite in its current state. I would really suggest just using a bigger compute runtime if you have to use pyiceberg for an operation like this otherwise use spark

[corleyma]

if you have to use pyiceberg for an operation like this otherwise use spark

or daft if you want a distributed dataframe option that doesn't require the JVM and supports append and overwrite to Iceberg tables (but not upsert currently). see e.g. https://docs.daft.ai/en/stable/api/dataframe/#daft.DataFrame.write_iceberg

[kevinjqliu]

I think this is a good idea. I also want to continue the discussion from #1004 :)

For context, heres something iceberg-go has implemented apache/iceberg-go#369
This materializes arrow stream as parquet files and then registers those files back to the iceberg table. I think its a very neat trick and can speed up certain use cases.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[maslennikov]

You can currently achieve this by something like...

for batch in my_batch_iterator:
# would need to check schema here
table.append(pa.Table.from_batches([batch]))

Not saying that this isnt a feature that would be nice but the ability to make it happen is already possible

Still would be nice to have a proper streaming / batch reader support out of the box with added benefit of correct transactionality and overwrite semantics.

with my_table.transaction() as tx:
    # imitate overwrite logic by truncating the table first
    tx.delete(delete_filter=AlwaysTrue())
    for batch in batch_reader:
        tx.append(pa.Table.from_batches([batch]))

[djouallah]

any update, iceberg adoption seems to be accelerating and pyiceberg is not keeping up :)