[GH-3128] Fix geotiff.metadata partition values, scan identity, and option handling

[jiayuasu]

Did you read the Contributor Guide?

• Yes, I have read the Contributor Rules and Contributor Development Guide

Is this PR related to a ticket?

• Yes, and the PR name follows the format [GH-XXX] my subject. Closes Fix geotiff.metadata defects found during the netcdf.metadata reviews #3128

What changes were proposed in this PR?

Apply to geotiff.metadata the fixes that hardened the same code patterns in netcdf.metadata during the #2829 reviews. netcdf.metadata was originally modeled on geotiff.metadata, so these defects exist identically in the GeoTIFF package on master.

Correctness (silent wrong results)

1. Per-file partition values. A bin-packed partition wrapped one reader in a single PartitionReaderWithPartitionValues using partition.files.head.partitionValues, so files from different Hive partitions (e.g. year=2020/, year=2021/) all reported the first file's partition values. Each file's reader is now wrapped with its own values (GeoTiffMetadataConcatPartitionReader; sub-readers open lazily, at most one file open at a time).

2. Scan equality and hashing include pushdown state. FileScan's concrete equals/hashCode suppress the case-class-synthesized versions, so pushedLimit and options did not participate in equality. Since the scan alone enforces the pushed limit (isPartiallyPushed = false removes the Limit operator), exchange/subquery reuse could serve a limited scan's result for an unlimited one. equals now includes both fields and hashCode is derived from every equality field.

Crash

3. fallbackFileFormat no longer returns null. CREATE TABLE ... USING geotiff.metadata threw a NullPointerException. The fallback is now a stub FileFormat that fails on instantiation with a descriptive UnsupportedOperationException, which also rejects CREATE TABLE with an explicit schema (that path skips inferSchema).

Behavior

4. Directory defaults never override explicit options. Single-directory loads forced recursiveFileLookup=true, overriding an explicit false and silently disabling Hive-style partition discovery; multi-directory loads bypassed the GeoTIFF extension filter entirely. Both are now defaults only, applied for any directory root.

5. Option presence checks are case-insensitive. Defaults were guarded with containsKey on a copied HashMap, so an explicit mixed-case option (e.g. RecursiveFileLookup=false) could nondeterministically lose to the lowercase default.

6. A glob path combined with an explicit pathGlobFilter keeps both constraints. .load("/dir/a*.tif") with pathGlobFilter="b*.tif" previously rewrote the path and kept only one filter; the rewrite is now skipped when the user supplied a filter, so Spark applies both natively.

Every fix mirrors its review-hardened counterpart in spark/common/.../io/netcdfmetadata/, adjusted only for naming and the .tif/.tiff extensions. Documentation gains the directory-load defaults and the unsupported-catalog-operations note, mirroring the NetCDF page.

How was this patch tested?

7 new tests in geotiffMetadataTest.scala (19 total), mirroring the netcdf.metadata regression coverage:

• Hive-style partition discovery with an explicit mixed-case RecursiveFileLookup=false, asserting every row carries its own file's partition value even when bin-packed into one Spark partition
• Scan hash/equality consistency across equal, limit-differing, and options-differing copies
• Glob path + explicit pathGlobFilter returning the native empty intersection (with a glob-only sanity check)
• Multi-directory loads filtering non-GeoTIFF files
• CREATE TABLE ... USING geotiff.metadata failing with a clear error instead of an NPE, both without and with an explicit schema

Full geotiffMetadataTest, netcdfMetadataTest, and rasteralgebraTest suites pass on Spark 3.4/Scala 2.12 (240 tests); sources cross-compile on Spark 4.0/Scala 2.13.

Did this PR include necessary documentation updates?

• Yes, I have updated the documentation.