fix(index): sort JSON-path values before btree training (#7485)

[pjdurden]

What

Fixes #7485. A JSON-path scalar index (target_index_type=btree) over a float path returns wrong results once the path contains negative values: range predicates and boundary equality return empty even though the rows exist.

rows = [{"v": 10.5}, {"v": 40.1}, {"v": -3.2}]
# ... write + create json/btree index on path "v" ...
ds.to_table(filter="json_get_float(data,'v') > 0")   # -> []   (expected [1, 2])
ds.to_table(filter="json_get_float(data,'v') = 40.1")# -> []   (expected [2])

Diagnosis (differs from the issue)

The issue describes this as a non-exact-float problem. It isn't. After isolating the variables, the trigger is negative values, not float precision. Same value set, only the sign changes:

data	predicate > 0	result
[0.5, 1.5, 2.5] (positive, exact)	[1,2,3]	correct
[10.5, 40.1, 3.2] (positive, non-exact)	[1,2,3]	correct
[0.5, 1.5, -2.5] (negative, exact)	[]	wrong
[10.5, 40.1, -3.2] (negative, non-exact)	[]	wrong

A non-exact but all-positive path indexes fine; an all-exact path with one negative is broken. The issue's "exact" control case happened to be all-positive, which hid the real variable.

Two more facts that pin it down:

• A plain Float64 btree index (no JSON) over the exact same values [0.5, 1.5, -2.5] returns correct results. So the btree itself is fine; the problem is specific to the JSON training path.
• With [0.5, 1.5, -2.5], interior equality (= 0.5) works, but the page min (= -2.5) and page max (= 1.5) both miss, and wide ranges (< 100) still return everything. That is the signature of corrupted page min/max stats, not bad values.

Root cause

train_btree_index assumes its input stream is globally sorted by value: analyze_batch records each page's stats as min = values[0], max = values[values.len()-1] (the first and last element of the page). The btree plugin's train_index does not sort; the caller is expected to pass pre-sorted data, and the normal scalar-index path sorts the column upstream.

JsonIndexPlugin::train_index extracts the typed values from the JSON column at training time and passes that stream straight to the target index trainer without sorting. The extracted stream is in storage order, not value order. When storage order matches numeric order (all-positive data) the first/last elements still bracket the page, so it happens to work. Introduce a negative and the page's recorded max can end up smaller than real values the page contains, so range and boundary lookups skip the page and return nothing.

Fix

Sort the extracted (value, row_id) stream ascending by value (nulls first) before handing it to the target index trainer, mirroring what the normal column path does upstream. Implemented as sort_stream_by_value and called between value conversion and target_plugin.train_index.

Tests

Added test_json_index_negative_floats in python/tests/test_scalar_index.py. It builds a json/btree float index over data with negatives and asserts the indexed result equals the un-indexed scan for >, <, >=, = (including equality on the negative values and on the page min/max). Fails on main, passes with this change.

Note

cargo fmt and cargo clippy -p lance-index --lib -D warnings are clean; the new regression test is Python-side, since cargo test -p lance-index currently fails to compile on main for an unrelated reason (FailNewFileStore mock in fmindex.rs missing IndexStore::with_io_priority).

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[wjones127]

Closing in favor of #7835, which avoids the double-sort.