fix(index): sort JSON-path values once after extraction, not the raw column

[wjones127]

Problem

A JSON-path scalar index (target_index_type=btree) returns wrong results when the
indexed path holds float64 values that are not exactly representable (e.g. 40.1,
-3.2): ranges come back empty and equality misses rows.

Fixes #7485.

Root cause

The btree trainer requires its input sorted by value (page min/max are taken from the
first/last row of each page). scan_training_data sorts by the raw JSON column when
the target requests TrainingOrdering::Values, but JsonIndexPlugin extracts a
different value (the value at path) from that column downstream, so the extracted
stream is not actually sorted by value. Silently corrupted page stats then cause range
and equality lookups to skip pages that hold matching rows.

Relationship to #7493, #7771, #7819

All three open PRs diagnose this correctly and fix it by adding a second SortExec
after JSON extraction, on top of the raw-column sort the scanner already performs. That
raw-column sort is wasted work — it sorts a key nothing downstream uses.

This PR instead has JsonTrainingRequest::criteria() override the target's
TrainingOrdering::Values to TrainingOrdering::None when delegating to the scanner
(the scanner's sort can't help the extracted value anyway), and sorts the extracted
(value, row_id) stream once, in train_index, right before handing it to the target
trainer. Same correctness fix, one sort instead of two.

Testing

• test_json_float_btree_index_unsorted_input (rust/lance-index/src/scalar/json.rs):
  builds a real JSON/btree index through the plugin trainer/registry with the issue's
  repro values fed in raw storage order, and asserts range/equality results across 5
  cases. Also asserts JsonTrainingRequest::criteria().ordering == TrainingOrdering::None.
• test_json_index_non_exact_floats (python/python/tests/test_scalar_index.py): the
  issue's repro end to end via create_scalar_index + json_get_float filters.

cargo fmt --all, cargo clippy -p lance-index --tests -- -D warnings, and
uv run make lint are clean.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 9a97c11c-8755-4a7a-bfde-fc67610ac40f

📥 Commits

Reviewing files that changed from the base of the PR and between 6aa0d95 and 363444a.

📒 Files selected for processing (1)

• rust/lance-index/src/scalar/json.rs

---

📝 Walkthrough

Walkthrough

JSON scalar index training now sorts extracted JSON-path values for value-ordered targets such as btrees. Tests cover unsorted storage input, null pairing, non-Values targets, and non-exact float predicates.

Changes

JSON scalar index ordering

Layer / File(s)	Summary
Training criteria and scanner ordering rust/lance-index/src/scalar/json.rs	JsonTrainingRequest requests unordered scanner input while retaining row identifier requirements and exposes the rewritten criteria.
Extracted-value sorting and regression validation rust/lance-index/src/scalar/json.rs, python/python/tests/test_scalar_index.py	Extracted JSON values are sorted before value-ordered training, while other targets retain extraction order; tests validate float btree searches, null pairing, planner usage, and predicate results.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Scanner
  participant JsonIndexPlugin
  participant SortExec
  participant BtreeTrainer
  participant ScalarIndexQuery
  Scanner->>JsonIndexPlugin: return JSON rows in scanner order
  JsonIndexPlugin->>JsonIndexPlugin: extract JSON values and row ids
  JsonIndexPlugin->>SortExec: sort extracted values when ordering is Values
  SortExec-->>JsonIndexPlugin: return value-sorted stream
  JsonIndexPlugin->>BtreeTrainer: train target index
  ScalarIndexQuery->>BtreeTrainer: search float predicates
  BtreeTrainer-->>ScalarIndexQuery: return matching row ids

Loading

Possibly related PRs

• lance-format/lance#7771: Adds overlapping JSON training criteria, extracted-value sorting, and non-exact float regression coverage.
• lance-format/lance#7819: Re-sorts JSON scalar index values and tests float equality and range queries.

Suggested reviewers: westonpace, xuanwo

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main fix: sorting JSON-path values after extraction instead of the raw column.
Description check	✅ Passed	The description matches the change set and explains the float-index bug and fix clearly.
Linked Issues check	✅ Passed	The implementation and tests address #7485 by sorting extracted JSON values once and covering the non-exact float regression.
Out of Scope Changes check	✅ Passed	The added tests and supporting trainer changes stay focused on the JSON-path index fix and related regressions.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 99.35484% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/json.rs	99.35%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[coderabbitai]

🧹 Nitpick comments (1)

python/python/tests/test_scalar_index.py (1)

2493-2503: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use @pytest.mark.parametrize for the per-filter cases.

These iterations differ only by the filter input, so they should be parametrized rather than looped. This gives each case an independent pass/fail and clearer test IDs.

♻️ Suggested parametrization

`@pytest.mark.parametrize`(
    "filter",
    [
        "json_get_float(data, 'latitude') > 0",
        "json_get_float(data, 'latitude') >= 10.5",
        "json_get_float(data, 'latitude') = 40.1",
        "json_get_float(data, 'latitude') = 10.5",
        "json_get_float(data, 'latitude') < 100",
    ],
)
def test_json_index_non_exact_floats(filter):
    ...
    assert "ScalarIndexQuery" in ds.scanner(filter=filter).explain_plan()
    assert ds.to_table(filter=filter) == ds.to_table(
        filter=filter, use_scalar_index=False
    ), filter

As per coding guidelines: "Use @pytest.mark.parametrize for Python tests that differ only by inputs."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/python/tests/test_scalar_index.py` around lines 2493 - 2503, Replace
the filter loop in test_json_index_non_exact_floats with a
pytest.mark.parametrize decorator over the existing filter expressions, and move
the shared assertions into the test body so each filter runs as an independent
case with its value available through the filter parameter.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@python/python/tests/test_scalar_index.py`:
- Around line 2493-2503: Replace the filter loop in
test_json_index_non_exact_floats with a pytest.mark.parametrize decorator over
the existing filter expressions, and move the shared assertions into the test
body so each filter runs as an independent case with its value available through
the filter parameter.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 02cdab06-8d53-4379-a3cf-5fe87d5793b9

📥 Commits

Reviewing files that changed from the base of the PR and between da5ef90 and 6aa0d95.

📒 Files selected for processing (2)

• python/python/tests/test_scalar_index.py
• rust/lance-index/src/scalar/json.rs

[wjones127]

Added test coverage for two gaps found in self-review:

• test_json_btree_index_null_at_path: a null-at-path row (missing key) keeps its row_id paired with its null value through sort_stream_by_value, and IsNull/range/equality queries still behave correctly with nulls mixed in.
• test_json_ngram_index_skips_value_sort: a JSON index over an ngram target (TrainingOrdering::None) skips the value sort and still returns correct results — covers the previously-untested else branch in train_index.

(Noted in the null test: an explicit JSON null literal at path, as opposed to a missing key, currently fails in convert_stream_by_type — pre-existing, unrelated to this fix, out of scope here.)