Skip to content

Wrong result: grouped MAX(NULL) row is dropped with ORDER BY aggregate NULLS FIRST LIMIT #23440

Description

@Fly-a-Kite

Describe the bug

DataFusion 54.0.0 returns an empty result for a grouped aggregate query where the aggregate value is NULL, followed by ORDER BY ... NULLS FIRST LIMIT.

The grouped aggregate should still produce one output row. The final ORDER BY / LIMIT should not remove it.

This looks related to #22190, but I can still reproduce it on the latest datafusion==54.0.0 with a smaller reproducer.

To Reproduce

  import datafusion
  import pyarrow as pa
  from datafusion import SessionContext

  print("datafusion", datafusion.__version__)

  batch = pa.record_batch(
      [
          pa.array(["gamma"], type=pa.string()),
          pa.array([None], type=pa.float64()),
      ],
      names=["s", "y"],
  )

  ctx = SessionContext()
  ctx.register_record_batches("t0", [[batch]])

  query = """
  SELECT max_y
  FROM (
    SELECT s, MAX(y) AS max_y
    FROM t0
    GROUP BY s
  )
  ORDER BY max_y DESC NULLS FIRST
  LIMIT 3
  """

  result = ctx.sql(query).to_pandas()
  print(result)
  print("row count:", len(result))

### Expected behavior

  ### Expected behavior

  The query should return one row:

     max_y
  0   NULL

  Reason: the input contains one group, s = 'gamma'. MAX(y) over a group where all y values are NULL should produce one grouped output row with max_y = NULL.

  ### Actual behavior

  DataFusion 54.0.0 returns zero rows:

  Empty DataFrame
  Columns: [max_y]
  Index: []
  row count: 0

### Additional context

 ### Environment

  - datafusion==54.0.0
  - pyarrow==24.0.0
  - Python 3.12.3
  - Linux x86_64

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions