Skip to content

[C++][Python] PyArrow Parquet writer fails writing [null] for null structs, but Datafusion works. Need to add [null] to current working [""] for NULLs. #50470

Closed
Closed
[C++][Python] PyArrow Parquet writer fails writing [null] for null structs, but Datafusion works. Need to add [null] to current working [""] for NULLs.#50470
Assignees
fenfeng9
Labels
Component: C++Component: PythonType: bug

Description

@davlee1972
davlee1972
opened on Jul 10, 2026

Describe the bug, including details regarding any error messages, version, and platform.

I'm getting the following error trying to write a parquet file.

pyarrow.lib.ArrowInvalid: Column '_Id' is declare non-nullable, but contains nulls.

I think this is happening because there might be actual nulls in the column for the purpose of making a struct array even between the outer struct and inner struct. The image below is showing is_valid: [false] which looks odd to begin with..

The entire ShareClassType field is nullable and it contains a NULL, but the writer is complaining that a non-nullable field in belonging to non-nullable inner struct contains NULLs.

Here is the schema for this item:

Image

and the data in it:

Image

Component(s)

C++, Python

Activity

  1. davlee1972
    added
    Type: bug
    on Jul 10, 2026
  2. github-actions
    changed the title [-]PyArrow Parquet writer failing on writing inner structs which are not null, but outer struct is nullable[/-] [+][C++][Python] PyArrow Parquet writer failing on writing inner structs which are not null, but outer struct is nullable[/+] on Jul 10, 2026
  3. github-actions
    added
    Component: C++
    Component: Python
    on Jul 10, 2026

  4. davlee1972 commented on Jul 10, 2026

    @davlee1972
    davlee1972
    on Jul 10, 2026
    Author

    As a temporary work around.. I was able to write the pyarrow table using Datafusion..

    Read it back using pyarrow and the rewrite it back as a rowgroup..

    Image

    Datafusion changed the underlying data to [""].

    Schema:

    Image

    Data:

    Image

  5. davlee1972 commented on Jul 10, 2026

    @davlee1972
    davlee1972
    on Jul 10, 2026
    Author

    In PyArrow, a NULL column inside a StructArray is often initialized with a default value like "" (empty string) or 0. This happens because Arrow does not allocate null buffers for nested struct fields; instead, it puts a placeholder in the null slot.To fix this, use PyArrow compute functions to replace the empty strings with proper null values.

    I guess using pa.null() or [None] to even out nested struct arrays won't work? I should be adding empty strings instead even though an empty string technically isn't a struct data type?

    This could get complex since the outer struct has both scalar types and struct types..

    I think this should be fixed in the parquet writer.. The [null] value appear to be perfectly valid in the pyarrow table..

  6. davlee1972
    changed the title [-][C++][Python] PyArrow Parquet writer failing on writing inner structs which are not null, but outer struct is nullable[/-] [+][C++][Python] PyArrow Parquet writer fails writing [None] for null structs, but Datafusion works. Need to add [None] to current working [""] and [0] for NULLs.[/+] on Jul 10, 2026
  7. davlee1972
    changed the title [-][C++][Python] PyArrow Parquet writer fails writing [None] for null structs, but Datafusion works. Need to add [None] to current working [""] and [0] for NULLs.[/-] [+][C++][Python] PyArrow Parquet writer fails writing [null] for null structs, but Datafusion works. Need to add [null] to current working [""] and [0] for NULLs.[/+] on Jul 10, 2026
  8. davlee1972
    changed the title [-][C++][Python] PyArrow Parquet writer fails writing [null] for null structs, but Datafusion works. Need to add [null] to current working [""] and [0] for NULLs.[/-] [+][C++][Python] PyArrow Parquet writer fails writing [null] for null structs, but Datafusion works. Need to add [null] to current working [""] for NULLs.[/+] on Jul 10, 2026

  9. fenfeng9 commented on Jul 12, 2026

    @fenfeng9
    fenfeng9
    on Jul 12, 2026
    Contributor

    A null parent struct masks its child slots. Therefore, an underlying Arrow child-array slot may be null when the parent struct is null, even if the child field is non-nullable. This is valid because the child is logically absent. The Parquet writer incorrectly rejects this case.

    Reproduce

    import pyarrow as pa
import pyarrow.parquet as pq


value_field = pa.field("value", pa.int32(), nullable=False)

# The physical child slot is null.
value_column = pa.array([None], type=pa.int32())

# True means that the corresponding parent struct is null.
parent_is_null = pa.array([True])

array = pa.StructArray.from_arrays(
    arrays=[value_column],
    fields=[value_field],
    mask=parent_is_null,
)

print("logical array:", array.to_pylist())
print("physical child:", array.field("value").to_pylist())

pq.write_table(pa.table({"data": array}), "output.parquet")

    Result

    logical array: [None]
physical child: [None]
Traceback (most recent call last):
  File "/path/to/test.py", line 22, in <module>
    pq.write_table(pa.table({"data": array}), "output.parquet")
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/site-packages/pyarrow/parquet/core.py", line 2015, in write_table
    writer.write_table(table, row_group_size=row_group_size)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/site-packages/pyarrow/parquet/core.py", line 1180, in write_table
    self.writer.write_table(table, row_group_size=row_group_size)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_parquet.pyx", line 2397, in pyarrow._parquet.ParquetWriter.write_table
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 'value' is declared non-nullable but contains nulls

    Analysis

    This appears to be a regression introduced by #44921.

    The check uses leaf_array.null_count() without accounting for null parent structs, so it incorrectly rejects child nulls masked by a null parent.

    arrow/cpp/src/parquet/column_writer.cc

    Lines 1432 to 1442 in 5815dc0

    if (!leaf_field_nullable && leaf_array.null_count() != 0) {
    return Status::Invalid("Column '", descr_->name(),
    "' is declared non-nullable but contains nulls");
    }
    bool maybe_parent_nulls = level_info_.HasNullableValues() && !single_nullable_element;
    if (maybe_parent_nulls) {
    ARROW_ASSIGN_OR_RAISE(
    bits_buffer_,
    ::arrow::AllocateResizableBuffer(
    bit_util::BytesForBits(properties_->write_batch_size()), ctx->memory_pool));
    bits_buffer_->ZeroPadding();

  10. fenfeng9 commented on Jul 12, 2026

    @fenfeng9
    fenfeng9
    on Jul 12, 2026
    Contributor

    take

  11. github-actions
    assigned
    fenfeng9
    on Jul 12, 2026

  12. davlee1972 commented on Oct 26, 2026

    @davlee1972
    davlee1972
    on Oct 26, 2026
    Author

    #55136

  13. HuaHuaY commented on Oct 27, 2026

    @HuaHuaY
    HuaHuaY
    on Oct 27, 2026
    Contributor

    #55136

    Oh, I didn't notice that an issue already existed.

  14. fenfeng9 commented on Oct 27, 2026

    @fenfeng9
    fenfeng9
    on Oct 27, 2026
    Contributor

    13. > #55136

     Oh, I didn't notice that an issue already existed.

    Thanks for handling this, I’ve been busy lately and totally forgot about this issue — appreciate you picking it up.

  15. HuaHuaY commented on Oct 27, 2026

    @HuaHuaY
    HuaHuaY
    on Oct 27, 2026
    Contributor

    #55136 has been merged. I’ve verified that the issue is resolved by building PyArrow locally, so I close this issue.

  16. HuaHuaY
    closed this as completedon Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Component: C++Component: PythonType: bug

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions