fix(fts): restore format version environment fallback

[Xuanwo]

Context

New FTS / inverted index creation defaults to format v4, but the rollout also stopped honoring LANCE_FTS_FORMAT_VERSION when format_version is omitted. Deployments that rely on this process-wide compatibility switch could therefore create indexes unreadable by older readers despite retaining the rollout setting.

This restores the established precedence: an explicit creation parameter wins, otherwise the environment variable is used, and v4 remains the default when neither is present. Invalid environment values fail clearly. The fallback is limited to new-index training, so append, optimize, and other maintenance operations continue preserving each index's existing format.

Regression coverage exercises the precedence and error cases in isolated subprocesses, and the migration and binding documentation now describe the compatibility behavior.

[coderabbitai]

📝 Walkthrough

Walkthrough

FTS index creation now prioritizes explicit format_version, then LANCE_FTS_FORMAT_VERSION, and finally v4. Invalid environment values fail creation. Rust resolution, Python tests, and Java, Python, and migration documentation reflect the updated behavior.

Changes

FTS format resolution

Layer / File(s)	Summary
Format resolution core rust/lance-index/src/scalar/inverted/tokenizer.rs	Adds creation-time format resolution using explicit options, the environment variable, or the v4 default, with errors for invalid values.
Training JSON creation integration rust/lance-index/src/scalar/inverted/tokenizer.rs	Applies format resolution during training JSON ingestion before compatibility validation.
Cross-language documentation and validation python/python/tests/test_scalar_index.py, docs/src/guide/migration.md, java/src/main/java/org/lance/index/scalar/InvertedIndexParams.java, python/python/lance/dataset.py	Tests environment precedence, explicit overrides, defaults, and failures; updates corresponding documentation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant IndexCreation
  participant from_training_json
  participant resolve_creation_format_version
  participant Environment
  IndexCreation->>from_training_json: create FTS index
  from_training_json->>resolve_creation_format_version: pass explicit format_version
  resolve_creation_format_version->>Environment: read LANCE_FTS_FORMAT_VERSION when unset
  Environment-->>resolve_creation_format_version: version or invalid value
  resolve_creation_format_version-->>from_training_json: resolved version or error
  from_training_json-->>IndexCreation: create index or fail

Loading

Suggested labels: A-format

Suggested reviewers: bubblecal

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the main change: restoring the FTS format-version environment fallback.
Description check	✅ Passed	The description accurately describes the precedence, error handling, scope, and docs/tests in this PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch xuanwo/fix-fts-format-rollout-env

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)

rust/lance-index/src/scalar/inverted/tokenizer.rs-162-164 (1)

162-164: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Document invalid environment handling consistently.

The migration guide states that invalid LANCE_FTS_FORMAT_VERSION values fail creation and that an explicit format_version wins even when the environment is invalid, but these public API docs omit those effects.

• rust/lance-index/src/scalar/inverted/tokenizer.rs#L162-L164: document invalid-value failure and explicit precedence.
• rust/lance-index/src/scalar/inverted/tokenizer.rs#L822-L825: document invalid-value failure and explicit precedence.
• java/src/main/java/org/lance/index/scalar/InvertedIndexParams.java#L267-L268: document invalid-value failure and explicit precedence.
• python/python/lance/dataset.py#L3374-L3375: document invalid-value failure and explicit precedence.

As per coding guidelines, “Add doc comments to public API elements that convey semantic meaning, valid values, and effects.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-index/src/scalar/inverted/tokenizer.rs` around lines 162 - 164,
Update the public API documentation for the format-version settings at
rust/lance-index/src/scalar/inverted/tokenizer.rs:162-164 and :822-825,
java/src/main/java/org/lance/index/scalar/InvertedIndexParams.java:267-268, and
python/python/lance/dataset.py:3374-3375 to state that invalid
LANCE_FTS_FORMAT_VERSION values fail index creation, while an explicitly
supplied format_version takes precedence even when the environment variable is
invalid.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Other comments:
In `@rust/lance-index/src/scalar/inverted/tokenizer.rs`:
- Around line 162-164: Update the public API documentation for the
format-version settings at
rust/lance-index/src/scalar/inverted/tokenizer.rs:162-164 and :822-825,
java/src/main/java/org/lance/index/scalar/InvertedIndexParams.java:267-268, and
python/python/lance/dataset.py:3374-3375 to state that invalid
LANCE_FTS_FORMAT_VERSION values fail index creation, while an explicitly
supplied format_version takes precedence even when the environment variable is
invalid.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: c6931127-2653-44d2-872c-8bff5ac63ba0

📥 Commits

Reviewing files that changed from the base of the PR and between 654f0d4 and 2d9d6eb.

📒 Files selected for processing (5)

• docs/src/guide/migration.md
• java/src/main/java/org/lance/index/scalar/InvertedIndexParams.java
• python/python/lance/dataset.py
• python/python/tests/test_scalar_index.py
• rust/lance-index/src/scalar/inverted/tokenizer.rs

[codecov]

Codecov Report

❌ Patch coverage is 58.82353% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/inverted/tokenizer.rs	58.82%	6 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!