chore(lance): upgrade to 9.0.0-beta.21

[aaltshuler]

Summary

• advance every Lance dependency from 9.0.0-beta.15 to 9.0.0-beta.21 and refresh the lockfile
• realign force-delete, DirectoryNamespace physical-version visibility, and deterministic FTS/RRF behavior with beta.21
• document the beta.15 to beta.21 audit and correct the stale full-table vector staging blocker while leaving the EnsureIndices adapter migration for its own slice

Validation

• scripts/check-agents-md.sh
• git diff --check
• cargo test --workspace --locked
• cargo test -p omnigraph-engine --features failpoints --test failpoints --locked

The feature-gated failpoint suite passes all 111 tests.

---

Greptile Summary

This PR upgrades OmniGraph's Lance dependency to 9.0.0-beta.21 and realigns the engine with the new surface. The main changes are:

• Lance workspace dependencies and lockfile refreshed to the beta.21 revision.
• Branch delete behavior updated around beta.21 force-delete idempotency.
• DirectoryNamespace tests updated for physical-version visibility.
• Search ranking expectations refreshed for beta.21 FTS behavior.
• Developer docs updated for the Lance audit and vector-index staging status.

Confidence Score: 4/5

This is close, but the branch recovery cleanup path should be fixed before merging.

• The main wrapper now handles the missing-branch outcomes introduced by raced cleanup.
• One recovery discard path can still call Lance directly and fail on the same already-deleted branch state.
• The Lance revision and test updates otherwise look consistent with the changed behavior.

crates/omnigraph/src/branch_control.rs and the branch recovery discard path

Important Files Changed

Filename	Overview
crates/omnigraph/src/branch_control.rs	The wrapper now treats both missing-ref and missing-tree outcomes as successful branch cleanup.
Cargo.toml	The Lance workspace dependencies now point at the beta.21 revision.
Cargo.lock	The lockfile was refreshed for the Lance beta.21 dependency graph.
crates/omnigraph/src/db/manifest/tests.rs	The DirectoryNamespace guard now checks beta.21 physical-version visibility while keeping OmniGraph manifest authority unchanged.
crates/omnigraph/tests/lance_surface_guards.rs	The Lance surface guard now pins force-delete success for fully absent branches.
crates/omnigraph/tests/staged_writes.rs	The staged-write guard still documents vector index creation as an inline residual.
docs/dev/lance.md	The Lance audit was updated for beta.21 behavior and the corrected vector-index staging status.

Comments Outside Diff (1)

1. crates/omnigraph/src/branch_control.rs, line 55 (link)

   Recovery Bypasses Wrapper

   This normalization now covers the TableStore::force_delete_branch cleanup paths, but the same already-reclaimed branch case is still reachable from recovery. The no-effect fork discard path calls Lance's dataset.force_delete_branch(target_branch) directly and maps any Lance error to OmniError::Lance, so a fork that another cleanup already removed can still turn RefNotFound or NotFound into a recovery failure instead of converging as already deleted. Routing that recovery discard through this idempotent helper keeps branch cleanup behavior consistent.

   

_{Reviews (2): Last reviewed commit: "fix(branches): preserve idempotent NotFo..." | Re-trigger Greptile}

Context used:

• Context used - AGENTS.md (source)
• Context used - CLAUDE.md (source)

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.