What is the problem the feature request solves?
To my knowledge, as of Spark 4.2 the GEOMETRY type will be turned on and the limited set of spatial functions will be present: ST_AsBinary, ST_GeogFromWKB, ST_GeomFromWKB, ST_SRID
and ST_SetSRID.
As per #4455 I don't think more advanced functions are in scope for Comet, but implementing those few I think force a few things with respect to the ability to capture Spark's type metadata (the SRID can be either type level or per-row, thus requires some FieldRefs where there are currently DataTypes).
Describe the potential solution
The LakeSail implementation of geometry and geography types is possibly the closest to the scope of Comet's support here: lakehq/sail#1325 , which I believe uses GeoArrow as the Arrow storage but I haven't followed how it handles SRID mappings (I think in Spark there's a mapper from integer to authority:code / string to handle Spark's SRID-based system with Parquet / Icebergs string-based system.
Additional context
Happy to help!
What is the problem the feature request solves?
To my knowledge, as of Spark 4.2 the GEOMETRY type will be turned on and the limited set of spatial functions will be present: ST_AsBinary, ST_GeogFromWKB, ST_GeomFromWKB, ST_SRID
and ST_SetSRID.
As per #4455 I don't think more advanced functions are in scope for Comet, but implementing those few I think force a few things with respect to the ability to capture Spark's type metadata (the SRID can be either type level or per-row, thus requires some FieldRefs where there are currently DataTypes).
Describe the potential solution
The LakeSail implementation of geometry and geography types is possibly the closest to the scope of Comet's support here: lakehq/sail#1325 , which I believe uses GeoArrow as the Arrow storage but I haven't followed how it handles SRID mappings (I think in Spark there's a mapper from integer to authority:code / string to handle Spark's SRID-based system with Parquet / Icebergs string-based system.
Additional context
Happy to help!