Feature Request: Apache Arrow support

[charlie430]

I'm very interested in Apache Arrow being supported for the in-memory scenario.

Is there any information you can provide on when that might be supported?

Thanks!

[Giorgi]

It's not implemented at the moment but feel free to send a PR for the API: Arrow Interface

[nazar554]

Would it be ok to use Apache.Arrow package as a dependency?
It has data types for some objects, like CArrowSchema for duckdb_arrow_schema

[Giorgi]

Would that be added to the Binding project or Data project?

[nazar554]

Probably Binding project, but the package might be too heavy for it.
I guess for now I can create a DuckDBArrowSchema with private fields that has compatible struct layout.
So consumers can just cast the pointer to CArrowSchema*.

[Giorgi]

Honestly, I haven't looked much into Arrow and can't tell now for sure. Feel free to join DuckDB Discord, we can discuss it in more detail in the dotnet channel.

[CurtHagenlocher]

An alternative for getting data as Arrow could be to use the C# ADBC implementation with the generic driver importer. This code is not very mature yet, but you can run queries and get the result back as Arrow. Example:

            using AdbcDriver duckdb = CAdbcDriverImporter.Load("D:\\testdata\\duckdb.dll", "duckdb_adbc_init");
            using AdbcDatabase db = duckdb.Open(new Dictionary<string, string> { { "path", "d:/testdata/ddbt.db"} });
            using AdbcConnection cn = db.Connect(null);
            using AdbcStatement stmt = cn.CreateStatement();
            stmt.SqlQuery = "CREATE TABLE integers(foo INTEGER, bar INTEGER);";
            stmt.ExecuteUpdate();

            stmt.SqlQuery = "INSERT INTO integers VALUES (3, 4), (5, 6), (7, 8);";
            stmt.ExecuteUpdate();

            stmt.SqlQuery = "SELECT * from integers";
            var results = stmt.ExecuteQuery();

            // results.Stream is an IArrowArrayStream, which lets you get the schema
            // and a set of record batches

NOTE that this code is not super mature and we haven't yet reached a 1.0 release.

[Giorgi]

Nice! I think it would be great if there was a way to go from DuckDBConnection (provided by this library) to AdbcConnection. I can expose the underlying pointer to the database (obtained by duckdb_open and duckdb_connect) but looks like there is no way to convert such pointer to an AdbcConnection object.

[CurtHagenlocher]

I know next to nothing about DuckDB internals, so I have no idea how plausible something like this is. For ADBC, we need an array of function pointers that defines the ADBC driver API -- this is what duckdb_adbc_init is initializing -- and the connection is then roughly an indirected opaque pointer that gets passed to some of these function pointers.

[pantonis]

Is this something that was implemented. Is Apache Arrow supported by this library?

[Giorgi]

No, I haven't implemented it.

[pantonis]

Thanks for your really quick prompt. Any plans to do it?

[luisquintanilla]

Thanks for keeping this issue open. I know it has been around for a while, and I saw there was an earlier draft PR. I wanted to suggest a small path that avoids the legacy Arrow API and keeps the first version low-risk.

I put together a small standalone repo while looking into this, which includes the analysis and a couple of runnable .NET 10 samples that get Arrow batches out of DuckDB today:

• Repo: https://github.com/luisquintanilla/arrow-dotnet-compute-kernels
• DuckDB + Arrow analysis (what the current C Data Interface offers vs the deprecated API): ANALYSIS.md section 2
• DuckDB through ADBC, Arrow-native result batches (works today as a workaround): demo/samples/03-duckdb-adbc-arrow.cs
• Arrow-at-rest path: demo/samples/02-duckdb-and-arrow-ipc.cs

Proposal: add an Arrow surface to DuckDB.NET.Data built on DuckDB's current Arrow C Data Interface, with Apache.Arrow doing the managed import work.

A possible public shape:

using DuckDB.NET.Data;
using Apache.Arrow;

await using var connection = new DuckDBConnection("Data Source=:memory:");
connection.Open();

await using var command = connection.CreateCommand();
command.CommandText = "select 42 as answer, 'duckdb' as name";

await foreach (RecordBatch batch in command.ExecuteArrowBatchesAsync(cancellationToken))
{
    // consume Apache.Arrow RecordBatch values
}

Or, for callers that want the stream abstraction directly:

IArrowArrayStream stream = command.ExecuteArrowStream();

Implementation sketch:

• Add the current Arrow C Data Interface entry points to the Bindings project:
  • duckdb_result_get_arrow_options
  • duckdb_to_arrow_schema
  • duckdb_data_chunk_to_arrow
  • duckdb_destroy_arrow_options
• Keep the Apache.Arrow package dependency in DuckDB.NET.Data, not in Bindings.
• Add a small DuckDBArrowOptions safe handle around the native options pointer.
• Build a managed IArrowArrayStream over the existing result/chunk flow:
  • execute the command and keep the native result alive
  • build the ArrowSchema once from the result column types/names via duckdb_to_arrow_schema
  • loop duckdb_fetch_chunk
  • for each duckdb_data_chunk, call duckdb_data_chunk_to_arrow into an ArrowArray
  • hand the filled ArrowSchema / ArrowArray pointers to Apache.Arrow.C.CArrowSchemaImporter.ImportSchema and Apache.Arrow.C.CArrowArrayImporter.ImportRecordBatch

The main reason I would avoid the older duckdb_query_arrow / duckdb_execute_prepared_arrow / duckdb_arrow_scan family is that those APIs are deprecated and scheduled for removal. The current C Data Interface is a little more explicit, but maps cleanly to .NET because Apache.Arrow already has the C import side.

ADBC is also worth mentioning: users can get Arrow batches from DuckDB today with Apache.Arrow.Adbc by loading DuckDB's bundled ADBC driver (see sample 03 above). That is a good workaround for anyone blocked right now. The gap a first-class DuckDB.NET API would fill is reusing the existing DuckDBConnection / DuckDBCommand model instead of starting from a separate ADBC connection.

For background, the Apache.Arrow.Compute aggregation kernels these samples use are also proposed upstream in apache/arrow-dotnet#375 (PR: apache/arrow-dotnet#379), but that is independent of this DuckDB Arrow surface.

If this direction sounds reasonable, I would be happy to send a PR for a query-result-only first pass. The main thing I would want to confirm first is where you would prefer the Apache.Arrow dependency to sit - my assumption is DuckDB.NET.Data, while the Bindings project only exposes the native structs/functions.

cc: @Giorgi