Skip to content

test: add sqllogictest coverage for DISTINCT / GROUP BY / aggregation on map columns#23406

Open
PG1204 wants to merge 4 commits into
apache:mainfrom
PG1204:fix-15428-map-distinct-tests
Open

test: add sqllogictest coverage for DISTINCT / GROUP BY / aggregation on map columns#23406
PG1204 wants to merge 4 commits into
apache:mainfrom
PG1204:fix-15428-map-distinct-tests

Conversation

@PG1204

@PG1204 PG1204 commented Jul 9, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

SELECT DISTINCT on map columns used to fail with ArrowError(NotYetImplemented) because arrow-rs's RowConverter did not support Map types. That support has since landed upstream (apache/arrow-rs#7879) and is available on DataFusion main via arrow 59.1, so these queries now work. As suggested in #15428 (comment) (maintainer comment), this PR adds sqllogictest coverage to lock in the behavior and close the issue.

What changes are included in this PR?

Adds a new test section to datafusion/sqllogictest/test_files/map.slt covering:

  • SELECT DISTINCT on a map column, including the exact reproducer from the issue (DISTINCT ... LIMIT), collapsing of duplicate maps and duplicate NULLs, and DISTINCT over map + scalar columns together
  • GROUP BY on a map column, GROUP BY map + scalar keys, and HAVING with a map grouping key
  • Aggregations with map inputs: COUNT(map), COUNT(DISTINCT map), SUM grouped by map, and array_agg of map values
  • Edge cases: empty maps under DISTINCT (equal to each other, distinct from NULL), NULL maps, and unsorted maps with the same entries in different order being treated as distinct values
  • UNION (distinct) on map columns
  • DISTINCT / GROUP BY over the existing 209-row parquet_map.parquet file for coverage on real Parquet-backed map data

Note: a bare MAP {'key': NULL} literal infers a Null value type and fails type coercion across a VALUES list against Map(Utf8, Int64) rows, so the test uses CAST(NULL AS BIGINT) for the null map value. That coercion gap may be worth a separate follow-up issue.

Are these changes tested?

Yes, this PR is test-only. All tests pass locally via cargo test -p datafusion-sqllogictest --test sqllogictests -- map.slt.

Are there any user-facing changes?

No. Test-only change; no API or behavior changes.

@github-actions github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

select distinct on map types yields ArrowError NotYetImplemented

2 participants