Add more spatial extras (length, area, distance_to_filter and intersection_area) - #187
LionelZoubritzky-IGN wants to merge 37 commits into
Conversation
e1f709c to
f791f43
Compare
1c76f02 to
67960b8
Compare
67960b8 to
9fd69e0
Compare
9fd69e0 to
4bfe90d
Compare
|
Better to isolate refactors in single PRs than to mix it with a thematic PR. See the new |
| case "travel_time": { | ||
| const spatialFilterGeometry = spatialFilterToGeometry(spatialFilter, resolvedGeometryRef) | ||
| const filterPolygons = geometryToPolygons(spatialFilterGeometry); | ||
| const inter = filterPolygons ? intersect(featureCollection([feature(filterPolygons), feature(geo)])) : null; |
There was a problem hiding this comment.
intersect() is re-run against the full reference geometry for every returned feature, and spatialFilterToGeometry + geometryToPolygons (lines 126-127) are recomputed on every iteration.
Measured with a real reference (département 35, a MultiPolygon with 60,697 vertices) against real communes: 431 ms per feature, i.e. 43 s of blocked event loop at limit=100 — the default — and ~36 min at limit=5000. That is exactly the scenario of the integration test this branch adds (communes × département). Nothing bounds it upstream: the CQL goes out in a POST body, so no URL length limit caps the size of the reference geometry.
Under http-stream transport this freezes every session, and the AbortControllers armed in helpers/http.ts:157 during the freeze all fire at once when it lifts → cascading 504s for unrelated callers.
Measured fix (×217, areas bit-identical — bboxClip, bbox and dropEmptyRings are already available in this file):
case "intersects_feature":
case "travel_time": {
const spatialFilterGeometry = spatialFilterToGeometry(spatialFilter, resolvedGeometryRef);
const filterPolygons = geometryToPolygons(spatialFilterGeometry);
if (!filterPolygons) return null;
// Whatever lies outside the feature's bbox cannot intersect it, so clipping the
// reference first leaves the result unchanged while polyclip only ever processes
// the neighbouring vertices.
const clippedFilter = dropEmptyRings(bboxClip(filterPolygons, bbox(geo)).geometry);
if (!clippedFilter) return null;
const inter = intersect(featureCollection([feature(clippedFilter), feature(geo)]));
return inter === null ? null : inter.geometry;
}area(geo)" — is wrong: a département is concave.
There was a problem hiding this comment.
|
|
||
| if (requires_distance_to_filter) { | ||
| try { | ||
| const filterCentroid = spatialFilterToCentroid(spatialFilter, resolvedGeometryRef); |
There was a problem hiding this comment.
=> The filter geometry and its centroid are recomputed for every feature.
spatialFilterToCentroid / spatialFilterToGeometry depend only on input and resolvedGeometryRef, both constant across the whole collection — yet they are called from the per-feature loop in transformFeatureCollectionResponse (response.ts:94-100).
Measured: centroid(département 35) costs 2.79 ms per call, i.e. ~93% of the 3.0 ms per feature spent on distance_to_filter → 15 s of pure waste at limit=5000 for an identical result. Same pattern for the circle(64) of dwithin_point and the bboxPolygon of bbox_filter.
Suggestion: a prepareSpatialContext(input, resolvedGeometryRef) computed once in transformFeatureCollectionResponse (filter geometry, its polygons, its centroid) and passed down to deriveFromGeometry. It is also the natural home for the intersection_area performance fix.
There was a problem hiding this comment.
| return { type: "Point", coordinates: [spatialFilter.lon, spatialFilter.lat] }; | ||
| } | ||
| case "intersects_feature": | ||
| case "travel_time": |
There was a problem hiding this comment.
case "travel_time" falls into the centroid(spatialFilterToGeometry(...)) branch, so it measures from the centroid of the isochrone even though travelTimeFilterSchema already carries lon/lat, described as "Longitude / Latitude du point de départ" (schema.ts:95-96). That is exact, free, and it is what a user means by "distance to the travel-time filter".
Suggestion : move case "travel_time" into the dwithin_point / intersects_point branch and return { type: "Point", coordinates: [spatialFilter.lon, spatialFilter.lat] }. Then align the description at schema.ts:180, which currently says "le centroïde du filtre" without distinguishing the cases.
There was a problem hiding this comment.
8f8f5c5 (I forgot to commit it separately)
| if (requires_intersection_area) { | ||
| try { | ||
| const intersection = intersectionAreaWithSpatialFilter(geo, spatialFilter, resolvedGeometryRef); | ||
| ret.intersection_area = intersection ? area(intersection) : 0; |
There was a problem hiding this comment.
=< intersection_area returns 0 where the published contract promises null.
intersectionAreaWithSpatialFilter returns null for three distinct situations — "nothing areal"
(l. 113-115), "point filter" (l. 119) and "empty intersection" and this line collapses all three
into 0.
The entry point is real: getGeometryTypeDimension("GeometryCollection") returns "?" (properties.ts:81), which short-circuits the validation loop (properties.ts:153), so the 29 geometry-any collections in the catalog (wfs_du, wfs_sup, wfs_scot) pass with no dimension check at all.
Suggestion: have intersectionAreaWithSpatialFilter return number | null, so the distinction is carried by the callee instead of being reconstructed by the caller:
// `null` = not computable (nothing areal in the geometry)
// `0` = computed, no overlap (including a point filter: a point has an empty interior)The intersects_point → 0 semantics are preserved: that value is geometrically correct.
There was a problem hiding this comment.
I'm not completely clear on what we actually want in the end. What makes most sense to me is the following:
- If, for any reason, we cannot compute what is asked (for example, the centroid of a feature whose WFS did not return a geometry), return
nullfor that specificspatial_extra. - if the LLM already has the information at query-time that should let it know that the query itself is asking for something trivial or absurd (for example, the intersection area with a point filter), error immediately with an explicit message stating why you should not ask for that.
In the case of area intersection here, I see the following possibilities:
- The filter is not 2D and the LLM should know that: throw an error. This should the behavior for
intersects_point_filteronly. It's not the case currently. - The filter is not 2D but it is not trivial to see (the LLM does not necessarily have this information): return
intersection_area=0: simple and correct.returnsee next commentintersection_area=null. This should be the behavior for a non-2Dintersects_feature_filter. - The filter is 2D and the intersection with the geometry is empty: return
intersection_area=0. This can happen withbbox_filterbecause of Document howbbox_filterworks #196 for example.
@esgn what do you think?
There was a problem hiding this comment.
Actually, after writing #187 (comment), I think it's just easier to put intersection_area to 0 in the second case, instead of intersection_area=null. We should reserve null for error cases (for instance, there was an internal error when computing the intersection).
There was a problem hiding this comment.
First in schema.ts it is written
"Si l'élément à calculer est trivial (bbox d'un point, aire d'une géométrie linéaire), une erreur indiquera comment corriger la requête.\n"+
"Si une valeur n'est pas calculable pour une autre raison, elle sera remplacée par `null` dans la réponse.
So :
- case 1 : the mismatch is knowable before the query, from the catalog => error, the whole request fails
- case 2 : the mismatch only shows up on that one feature =>
null - case 3 : the computation ran => a number,
0included
Examining the current version of the code :
| Extra | Returns | null should mean |
State |
|---|---|---|---|
centroid |
{lon, lat} |
geometry missing or unreadable | ok |
bbox |
4-value array | geometry missing, unreadable or empty | off: an empty geometry yields [null,null,null,null], a third shape that is neither a bbox nor null |
length |
a number | non-linear geometry, so 0 only ever means a line of zero length |
off: returns null for any GeometryCollection, even one made only of LineStrings, where the length is computable and area does walk collections |
area |
a number | non-areal geometry, so 0 only ever means a polygon of zero area |
off: an empty Polygon yields 0 |
distance_to_filter |
a number | geometry or filter unreadable, so 0 only ever means the feature covers the reference point |
ok |
intersection_area |
a number | feature or filter not areal, so 0 only ever means genuinely no overlap |
off: five cases out of six yield 0 |
| .describe("Liste ordonnée des critères de tri."), | ||
| })) | ||
| .merge(gpfGeometryExtraInputSchema) | ||
| .merge(gpfGetFeaturesGeometryExtraInputSchema.required()) // must never be optional to ensure queryIsGetFeaturesInput correctness |
There was a problem hiding this comment.
.required() is a no-op here: it only unwraps ZodOptional, and this is a ZodEffects(ZodDefault(ZodArray)). Verified on the published schema required === ["typename"].
So the comment credits the wrong construct: .default([]) alone guarantees the field.
Drop both .required() calls (here and l. 405), move the comment, and pin the invariant with expect(gpfGetFeaturesInputSchema.parse({ typename: "X" })).toHaveProperty("spatial_extras").
There was a problem hiding this comment.
| return `Éléments calculés depuis la géométrie à renvoyer pour ${target}. Peut inclure ${allowedExtrasDocNames}, aucun par défaut.\n`+ | ||
| `${SPATIAL_EXTRAS_BASE_DESCRIPTION_LINES.join("\n")}\n`+ | ||
| optionalFilterLine+ | ||
| "Si une valeur n'est pas calculable, elle sera remplacée par `null` dans la réponse."; |
There was a problem hiding this comment.
This promises "…elle sera remplacée par null dans la réponse", but validateSpatialExtras (properties.ts:134-160) now fails the whole request instead. And the bbox-on-geometry-point rejection (70 collections) is documented nowhere, unlike length/area at l. 154-155.
Restrict the null sentence to what is not computable per feature, and state that a dimension mismatch is a hard error.
There was a problem hiding this comment.
It's like for #187 (comment): I think we need to distinguish trivial queries (like bbox on a geometry-point, or a dimension mismatch), which should error, and failures to compute, which should be marked with a null in the answer, so the LLM knows that something went wrong.
The edge-case is when we detect a "trivial" case only after running a WFS query, typically for a intersects_feature_filter where the feature ends up not having the correct dimension. I think null is the correct answer here.
If you agree with this, I can document it.
|
I'm starting a review commit by commit to try and not miss anything. Regarding bbc29e7 In
|
|
Now commit 2 : 026c561
const gpfGeometryExtraInputSchema = z.object({
spatial_extras: z
.array(z.enum(GPF_GET_FEATURES_SPATIAL_EXTRAS))
.default([]) <= in here |
| export class FeatureTypeNotFoundError extends Error { | ||
| constructor(name: string) { | ||
| super(`Le type '${name}' est introuvable`); | ||
| super(`Le type '${name}' n'existe pas. Utilise le tool gpf_search_types pour trouver les types existants et pertinents pour ta recherche.`); |
There was a problem hiding this comment.
The wrapper is redundant now (and being a catch-all, it also staples the hint onto network failures and overwrites error.name) => suggest dropping it.
Nit: these two new messages are the only Utilise in src/ (2 vs 15 Utiliser).
There was a problem hiding this comment.
I don't see what wrapper you mean to drop here?
|
Commit 3 : bdeadda Done in previous comments. |
e89dd84 to
0e49535
Compare
| travel_time_filter: "travel_time", | ||
| } as const satisfies Record<(typeof GPF_GET_FEATURES_SPATIAL_FILTER_KEYS)[number], string>; | ||
| const FILTER_KEY_TO_OPERATOR = Object.fromEntries( | ||
| GPF_GET_FEATURES_SPATIAL_FILTER_KEYS.map((key) => [key, key.replace(/_filter$/, "")]), |
There was a problem hiding this comment.
Nice cleanup => I checked the derived SpatialFilter against the hand-written union it replaces
(discriminated field access, literal operator, never exhaustiveness) and it is strictly equivalent,
with key order unchanged.
One thing it makes load-bearing: key.replace(/_filter$/, "") assumes every filter key ends in _filter. If one ever doesn't, the operator silently becomes the whole key, SpatialFilterEntry yields never so that member drops out of the union, and the default-less switch in queryPreparation.ts:198 returns cqlFilter: undefined => a query with no spatial filter at all, and no error.
Cheap guard: default: { const _: never = spatialFilter; } in that switch.
There was a problem hiding this comment.
I think it's a bit redundant with the identical guard already present in wfs/spatialExtras.ts, but it doesn't hurt either.
|
Commit 4 : 0892147 Done in previous comment |
| * @param value Unknown feature geometry value. | ||
| * @returns `true` when the value looks like a GeoJSON geometry object. | ||
| */ | ||
| export function isGeometryLike(value: unknown): value is Geometry { |
There was a problem hiding this comment.
Good deduplication (the two copies were byte-identical).
value is Exclude<Geometry, GeometryCollection> is exactly what the body checks.
But the return type is new, and value is Geometry is not what the body checks. It requires coordinates, so it rejects a valid GeometryCollection (which carries geometries instead), and it never looks at what coordinates holds, so { type: "CustomType", coordinates: null } passes and gets narrowed to Geometry. value is Exclude<Geometry, GeometryCollection> would describe the body exactly.
➡️ Related: test/wfs/geometry.test.ts drops "should throw for a completely unknown type" because "CustomType" no longer typechecks, but that is the very case this guard still lets through from remote data. Worth keeping with a cast.
There was a problem hiding this comment.
I narrowed the return type to Exclude<Geometry, GeometryCollection>, which has no downstream effect since callers widen it back to Geometry, and I don't see the point in threading the precise GeometryCollection exclusion everywhere. But the test is relevant, so I added it back in b56ddaf
|
Commit 5 : fba3ca3 Done in previous comments |
| @@ -52,9 +54,9 @@ export function compileDwithinSpatialFilter(geometryName: string, spatialFilter: | |||
| * Compiles an `intersects_feature` spatial filter once the reference geometry is already serialized. | |||
There was a problem hiding this comment.
The summary line is now wrong: this function is what serializes the geometry, so it is no longer "already serialized" when it arrives. The @param below was updated, this line was not.
* Compiles an `intersects_feature` spatial filter from the resolved reference geometry.There was a problem hiding this comment.
|
Commit 6 : 8b1d2d5 Done in previous comments |
Co-authored-by: Emmanuel S. <5435148+esgn@users.noreply.github.com>
Co-authored-by: Emmanuel S. <5435148+esgn@users.noreply.github.com>
Co-authored-by: Emmanuel S. <5435148+esgn@users.noreply.github.com>
Co-authored-by: Emmanuel S. <5435148+esgn@users.noreply.github.com>
Co-authored-by: Emmanuel S. <5435148+esgn@users.noreply.github.com>
Co-authored-by: Emmanuel S. <5435148+esgn@users.noreply.github.com>
4b2d84d to
51ecdc0
Compare
…geometry Co-authored-by: Emmanuel S. <5435148+esgn@users.noreply.github.com>
2894470 to
4bb7fbe
Compare
Co-authored-by: Emmanuel S. <5435148+esgn@users.noreply.github.com>
…erlap - intersection_area: null when the feature or the filter is not areal, or when the filter geometry could not be prepared; 0 only when both are areal and do not overlap - every extra is null on a missing or empty geometry - length sums the linear parts of a GeometryCollection; area of an empty Polygon is null - measures sum the parts of a GeometryCollection, overlaps included (JTS) - state the contract in deriveFromGeometry and in the LLM-facing descriptions - regenerate docs/mcp-tools.md
…hema - move "requires a spatial filter" from compileQueryParts to the zod refinement: invalid-tool-params instead of execution-error, raised before any catalog or network call - reject distance_to_filter with intersects_point_filter (always 0) - list only the compatible filters in the error message - state the rule in the spatial_extras description, regenerate docs/mcp-tools.md - move the related tests from compileQueryParts to the tool
buildPropertyNameWithGeometry always passes an empty spatial_extras list, and the `!spatial_extras` guard let it through: the by-id layer tool and the proxy by-id resolve then read the geometry format and threw on an unlisted or missing one. Return early on an empty list, so already minted by-id layer URLs do not depend on the catalog format values.
…turned features - tool and spatial_extras descriptions: extras are computed after the query, on the returned features only, and cannot be used in where/order_by; a ranking or a sum needs numberReturned == numberMatched - "property does not exist" error: explain when the name is a spatial extra (message only, a real catalog column with that name is still accepted)
…center for clarity
- centroid: may fall outside a concave geometry, so an intersects_point_filter on it may return neither the feature nor what contains it - distance_to_filter_center: distance from the filter center to the closest point of the feature, 0 when the feature contains the center; list the center of each filter - intersection_area: null when the feature or the intersects_feature_filter reference has no areal part, 0 when they do not overlap; name the areal filters (box, disc, isochrone, reference) - rankings and sums: raise limit or restrict the spatial filter; for the N closest, use dwithin_point_filter with distance_to_filter_center, sort, and widen distance_m when fewer than N come back - catalog check: report every incompatible extra in a single error, in the order of the request, instead of one per call - bbox on a Point layer: fix the double period and the missing comma in the error message - regenerate docs/mcp-tools.md
…al extra hint The hint is shared by every tool that validates property names. From gpf_get_feature_by_id, which has its own spatial_extras, it pointed to another tool instead of the same call's spatial_extras; from gpf_count_features and the layer tools, it suggested a tool switch that cannot help. Say only that the name is computed via spatial_extras and is not usable in select, where or order_by.
- bbox: [ouest, sud, est, nord] in WGS84 lon/lat, matching the west, south, east and north fields of bbox_filter - regenerate docs/mcp-tools.md
…atures gpf_get_features returns several features, so "La géométrie de l'objet" read as if a single one came back. Say "La géométrie renvoyée", as the other catalog errors of the same check already do.
…erences rule out - intersection_area needs an areal intersects_feature reference: a linear, MultiPoint or Point reference is now an error instead of null for every feature - distance_to_filter_center is always 0 from a reference reduced to a single position (a Point, or a MultiPoint whose points are all the same): now an error - check the reference against its catalog format before fetching it, then against its fetched geometry, which alone settles a geometry-any type or a single-position MultiPoint - run the queried layer check before any network call (reference or isochrone fetch), so its errors now come before select, where and order_by errors - intersection_area description: the reference must be areal; regenerate docs/mcp-tools.md
This PR introduces a number of small refactors (each in their own commit) and finally adds four new
spatial_extrastargets:length,area,distance_to_filterandintersection_area. The latter two can only be used when there is a spatial filter.Requests error when they are absurd, for example asking for
areaon a geometry which is stored asformat: "geometry-point"in the schema. When they are not absurd but still impossible in practice (the same but with"geometry-any"that actually returns a Point for example), the required field is set tonull.Close #136, although we may want to add even more
spatial_extrain the future. For the time being, these newspatial_extrasallow answering new questions like:length) -> 515,08 kmarea) -> La Cousinerie (24ha), Sainte-Radegonde (16ha), ...distance_to_filter) -> Piscine du Sivom des Trois Vallées de Mondeville (1.37 km)intersection_area) -> 1 039,3 ha