[C++][Python][Dataset][ORC] Dataset scans return nulls when a dotted field name collides with a nested path #55326

Describe the bug, including details regarding any error messages, version, and platform.

An ORC Dataset scan can silently replace stored values with nulls when the file contains both a literal top-level field named a.b and a struct field a containing b.

This occurs during a full scan without a columns selector. Filtering the affected field can subsequently discard matching rows. Reading the same file through ORCFile.read() without a columns selector preserves both fields correctly.

The issue was found while investigating ORC reads used by Ray Data. The reproducer requires only PyArrow.

Reproducer

import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.orc as orc

table = pa.table({
    "a.b": [1, 2],
    "a": [{"b": 10}, {"b": 20}],
})
sink = pa.BufferOutputStream()
orc.write_table(table, sink)
data = sink.getvalue()

def scan(**kwargs):
    fragment = ds.OrcFileFormat().make_fragment(pa.BufferReader(data))
    return fragment.to_table(**kwargs).to_pydict()

reader = orc.ORCFile(pa.BufferReader(data))
print("ORCFile:", reader.read().to_pydict())
print("Dataset:", scan())
print("Dataset filter:", scan(
    columns=["a.b"],
    filter=ds.field("a.b") > 1,
))

Actual output

ORCFile: {'a.b': [1, 2], 'a': [{'b': 10}, {'b': 20}]}
Dataset: {'a.b': [None, None], 'a': [{'b': 10}, {'b': 20}]}
Dataset filter: {'a.b': []}

No exception is raised. The Dataset result also passes validate(full=True). The file itself retains the correct values; the incorrect nulls are produced during reading.

Expected behavior

The full Dataset scan should preserve both fields and their stored values. The filtered result should be {'a.b': [2]}, matching an in-memory filter of the correctly read table.

This report concerns full-scan correctness and does not require redefining precedence between ambiguous string selectors.

Additional observations

• Reversing the top-level field order to ["a", "a.b"] makes these scans return correct results.
• The corresponding cases without a field-name collision work.
• The issue also reproduces with an on-disk ORC file, so it is not specific to BufferReader.
• For this reproducer, ORCFile.read() without a columns selector, followed by in-memory selection or filtering, is a workaround, but requires whole-file materialization.

Likely cause / Source analysis

The source suggests the following chain:

• liborc’s ColumnSelector maps both field paths to the same dotted string, with the later assignment replacing the earlier one.
• The Arrow ORC Dataset reader passes field names to this selection path.
• MakeExecBatch() supplies nulls for an expected field missing from the returned batch.

This source-based explanation is consistent with the observed behavior, but has not been confirmed through runtime debugging.

Environment

• macOS arm64, Python 3.10.18, released PyArrow 23.0.1.
• Additional independent reproduction: Linux x86_64, Python 3.12.14, with PyArrow 23.0.1 and 17.0.0, including the on-disk file and control cases above.

Related work

#52997 / #53010 discuss related dotted-name selection behavior in Parquet. They do not cover this ORC Dataset full-scan correctness issue. Searches found no existing ORC Dataset report or merged fix covering this behavior.

Investigation, local reproduction and report drafting used AI assistance.

Component(s)

C++, Python