[GH-3131] Align glob and directory-glob path handling in the raster metadata data sources

[jiayuasu]

Did you read the Contributor Guide?

• Yes, I have read the Contributor Rules and Contributor Development Guide

Is this PR related to a ticket?

• Yes, and the PR name follows the format [GH-XXX] my subject. Closes Align glob and directory-glob path handling in the raster metadata data sources #3131

What changes were proposed in this PR?

geotiff.metadata and netcdf.metadata rewrote a single file-glob path such as /dir/a*.nc into path /dir plus pathGlobFilter="a*.nc". Because pathGlobFilter matches leaf file names at any depth, combining the glob with an explicit recursiveFileLookup=true widened the match into nested files such as /dir/sub/a1.nc — unlike the native Spark glob, which matches direct children only. Directory globs such as /data/region* got neither of the directory-scan defaults: the getFileStatus probe fails on a glob string, so the expanded directories were listed non-recursively and unfiltered, inconsistent with loading the same directories by explicit path.

This PR takes the second direction proposed in #3131:

• The glob-to-filter rewrite is removed. File paths and file globs reach Spark untouched and keep native listing semantics in every option combination — the explicit-pathGlobFilter case has taken this path since [GH-3128] Fix geotiff.metadata partition values, scan identity, and option handling #3129 and passes CI across the Spark matrix, so the rewrite no longer earned its divergence. As in every other Spark file source, a file glob that matches nothing is now a PATH_NOT_FOUND error rather than a silently empty result.
• Directory detection goes through FileSystem.globStatus, which resolves plain paths and glob patterns alike: a path stands for directories when it matches at least one status and all matches are directories. Directory globs therefore receive the same recursive-scan and extension-filter defaults as explicit directory paths. The defaults still apply only when every root resolves to directories, preserving the mixed-load safeguard from [GH-3128] Fix geotiff.metadata partition values, scan identity, and option handling #3129.

Both data sources are changed identically and stay aligned.

How was this patch tested?

Six new regression tests, three per data source, mirrored across geotiffMetadataTest and netcdfMetadataTest:

• a file glob with explicit recursiveFileLookup=true matches direct children only (nested a2.nc/a2.tiff is not pulled in),
• a directory glob receives the directory-scan defaults (a nested file is found via recursion; a non-raster decoy is excluded by the extension filter),
• a file glob matching nothing raises AnalysisException (PATH_NOT_FOUND).

Full suites pass: 86/86 across both (-Dspark=3.5 -Dscala=2.12), including the pre-existing glob, directory, multi-directory, partition-discovery, and option-precedence tests.

Did this PR include necessary documentation updates?

• Yes, I have updated the documentation. Both data source pages now document the native glob semantics (direct children only, missing-path error for empty globs) and that directory globs receive the directory-scan defaults.

[Copilot]

Pull request overview

This PR aligns path/glob handling between geotiff.metadata and netcdf.metadata by removing the internal “glob-to-pathGlobFilter rewrite” and switching directory-root detection to Hadoop FileSystem.globStatus, so both data sources preserve native Spark glob semantics while still applying directory-scan defaults (recursive lookup + extension filtering) when the load roots resolve to directories.

Changes:

• Stop rewriting file globs (e.g., /dir/a*.nc, /dir/a*.tif) into (dir, pathGlobFilter) and pass file paths/globs through to Spark unchanged.
• Make directory-root detection glob-aware via FileSystem.globStatus, so “directory globs” receive the same defaults as explicit directory paths.
• Add regression tests (3 per data source) and update both docs pages to document the native glob semantics and directory-glob defaults.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/netcdfmetadata/NetCdfMetadataDataSource.scala	Removes glob-rewrite logic and uses glob-aware directory-root detection to apply directory defaults consistently.
spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/geotiffmetadata/GeoTiffMetadataDataSource.scala	Mirrors the NetCDF changes to keep GeoTIFF metadata source behavior aligned.
spark/common/src/test/scala/org/apache/sedona/sql/netcdfMetadataTest.scala	Adds regression coverage for file-glob recursion semantics, directory-glob defaults, and empty-glob error behavior.
spark/common/src/test/scala/org/apache/sedona/sql/geotiffMetadataTest.scala	Adds the same regression coverage for the GeoTIFF metadata data source.
docs/tutorial/files/netcdfmetadata-sedona-spark.md	Documents native Spark glob semantics and clarifies that directory globs receive directory-scan defaults.
docs/tutorial/files/geotiffmetadata-sedona-spark.md	Documents native Spark glob semantics and clarifies that directory globs receive directory-scan defaults.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.