SparqlItemSelector pages through a SELECT with LIMIT/OFFSET, advancing the offset by the rows each page fetched. SPARQL leaves solution order unspecified in the absence of ORDER BY, so LIMIT/OFFSET only walks a stable window when the endpoint happens to answer the same query in the same order every time. Nothing in the selector, and nothing in selectByClass, asks for that guarantee:
SELECT ?root WHERE { ?root a <class> FILTER(!isBlank(?root)) }When the order does shift between pages, a root can be skipped or repeated. A repeat is wasted work – the writes are idempotent upserts. A skip is a missing document, and under a rebuild it is worse than missing: deletion is implicit, so a document a run does not write is one the run says is gone. The run reports success either way.
Why it has not bitten
In practice the shipped path is deterministic. selectByClass is a single triple-pattern scan, and the deployment’s own endpoint is QLever, which answers such a scan from its index in a stable order; the same query on the same immutable import returns the same rows in the same sequence, so consecutive pages line up. That is a property of the engine and the query shape, not a guarantee the code asks for or the standard provides.
So this is filed as latent rather than observed: no known run has lost a root to it. What makes it worth recording is that both halves of that accident are about to stop holding for supplied selectors.
What changes the exposure
#754 lets a deployment supply its own ItemSelector, and the motivating one is not a plain scan. Selecting an entity by the identity a profile says consumers must compare on – the sameAs target rather than the publisher’s local node – means OPTIONAL plus a BIND, and a DISTINCT to collapse several local nodes onto one canonical IRI:
SELECT DISTINCT ?root WHERE {
?place a <http://schema.org/Place> .
OPTIONAL { ?place <http://schema.org/sameAs> ?canonical . FILTER(STRSTARTS(STR(?canonical), "https://sws.geonames.org/")) }
BIND(COALESCE(?canonical, ?place) AS ?root)
}A DISTINCT over a join is typically answered from a hash table, whose iteration order an engine is free to vary between evaluations, and which no index order underwrites. Every deployment-supplied selector inherits the paging contract without inheriting the accident that currently makes it safe.
SparqlItemSelectorOptions.maxResults rides on the same pagination, so an unstable order also makes a sample unreproducible – relevant to @lde/pipeline-shacl-sampler, though a shifting sample is a much smaller harm than a dropped root.
A deployment can immunise its own selector today
An explicit ORDER BY ?root on a supplied query makes its page boundaries deterministic without waiting on anything here – the motivating selector in #754 has already been changed to carry one. That is the difference between latent and urgent: nobody writing a selector before this lands is broken, provided they know to order it. Knowing to is the problem, since nothing in ItemSelector, SparqlItemSelector or selectByClass says so, and the one selector the library ships as a convenience does not do it.
Options
- Document the requirement. State on
ItemSelector/SparqlItemSelector that a paginated query needs a total order over its projected variables, and give selectByClass an ORDER BY ?root. Cheapest, and it is what a supplied selector needs in order to inherit the guarantee rather than rediscover it – but it leaves every existing hand-written selector query to be audited.
- Inject an
ORDER BY when the query has none. The selector already parses and rewrites the query to page it, so adding a deterministic order over the projected variables is mechanical. It makes the guarantee unconditional, at the cost of an engine-side sort the current query shape does not pay for – worth measuring on QLever against a large class before assuming it is free.
- Keyset pagination.
ORDER BY ?root plus FILTER(?root > <last seen>) instead of OFFSET, which is both stable and cheaper than deep offsets on a large class. Bigger change, only applies to single-variable projections, and it interacts with the dropped-row accounting the current pager does (fetched rows drive pagination, yielded rows only maxResults).
Suggested: (1) now, so #754’s supplied selectors have a contract to meet, with (2) measured before committing – a blanket sort could cost more than the bug it prevents on exactly the class sizes where the bug matters.
SparqlItemSelectorpages through a SELECT withLIMIT/OFFSET, advancing the offset by the rows each page fetched. SPARQL leaves solution order unspecified in the absence ofORDER BY, soLIMIT/OFFSETonly walks a stable window when the endpoint happens to answer the same query in the same order every time. Nothing in the selector, and nothing inselectByClass, asks for that guarantee:When the order does shift between pages, a root can be skipped or repeated. A repeat is wasted work – the writes are idempotent upserts. A skip is a missing document, and under a rebuild it is worse than missing: deletion is implicit, so a document a run does not write is one the run says is gone. The run reports success either way.
Why it has not bitten
In practice the shipped path is deterministic.
selectByClassis a single triple-pattern scan, and the deployment’s own endpoint is QLever, which answers such a scan from its index in a stable order; the same query on the same immutable import returns the same rows in the same sequence, so consecutive pages line up. That is a property of the engine and the query shape, not a guarantee the code asks for or the standard provides.So this is filed as latent rather than observed: no known run has lost a root to it. What makes it worth recording is that both halves of that accident are about to stop holding for supplied selectors.
What changes the exposure
#754 lets a deployment supply its own
ItemSelector, and the motivating one is not a plain scan. Selecting an entity by the identity a profile says consumers must compare on – thesameAstarget rather than the publisher’s local node – meansOPTIONALplus aBIND, and aDISTINCTto collapse several local nodes onto one canonical IRI:A
DISTINCTover a join is typically answered from a hash table, whose iteration order an engine is free to vary between evaluations, and which no index order underwrites. Every deployment-supplied selector inherits the paging contract without inheriting the accident that currently makes it safe.SparqlItemSelectorOptions.maxResultsrides on the same pagination, so an unstable order also makes a sample unreproducible – relevant to@lde/pipeline-shacl-sampler, though a shifting sample is a much smaller harm than a dropped root.A deployment can immunise its own selector today
An explicit
ORDER BY ?rooton a supplied query makes its page boundaries deterministic without waiting on anything here – the motivating selector in #754 has already been changed to carry one. That is the difference between latent and urgent: nobody writing a selector before this lands is broken, provided they know to order it. Knowing to is the problem, since nothing inItemSelector,SparqlItemSelectororselectByClasssays so, and the one selector the library ships as a convenience does not do it.Options
ItemSelector/SparqlItemSelectorthat a paginated query needs a total order over its projected variables, and giveselectByClassanORDER BY ?root. Cheapest, and it is what a supplied selector needs in order to inherit the guarantee rather than rediscover it – but it leaves every existing hand-written selector query to be audited.ORDER BYwhen the query has none. The selector already parses and rewrites the query to page it, so adding a deterministic order over the projected variables is mechanical. It makes the guarantee unconditional, at the cost of an engine-side sort the current query shape does not pay for – worth measuring on QLever against a large class before assuming it is free.ORDER BY ?rootplusFILTER(?root > <last seen>)instead ofOFFSET, which is both stable and cheaper than deep offsets on a large class. Bigger change, only applies to single-variable projections, and it interacts with the dropped-row accounting the current pager does (fetched rows drive pagination, yielded rows onlymaxResults).Suggested: (1) now, so #754’s supplied selectors have a contract to meet, with (2) measured before committing – a blanket sort could cost more than the bug it prevents on exactly the class sizes where the bug matters.