feat: support distributed merge_into

[raygao25]

Summary

Adds lance_ray.merge_into for large-scale, daily merge ingestions: match a source batch to a Lance table on a join key, update existing rows, and insert new ones, committed as a single atomic version.

It is the distributed counterpart of pylance’s LanceDataset.merge_insert(on).when_matched_update_all().when_not_matched_insert_all(), for Ray jobs where the source and target are too large to process on a single machine. Unlike lance-spark’s MERGE INTO, which shuffle-joins the source against the target, this path probes a scalar index (BTREE is preferred) to map each source key to its target fragment, then applies updates.

How it works

1. Dedupe — sort the source on the join key across Ray and keep one row per key.
2. Plan — each worker maps its keys to target fragment ids with batched index lookups (key IN (...), served by a BTREE/scalar index when present), then shuffles rows to the worker that owns that fragment.
3. Apply — matched rows are masked with per-fragment deletion files (data files are not rewritten); replacements and inserts are appended as new fragments.
4. Commit — one LanceOperation.Update on the driver. Concurrent appends rebase; a concurrent rewrite of a touched fragment fails rather than silently dropping the other write.

import lance_ray as lr

dataset = lr.merge_into(daily_batch, "s3://bucket/users.lance", on="user_id", num_workers=128)

A scalar index on the join key is strongly recommended for large targets.

Performance

Measured merging 1M source rows into a 2.5B-row table with a BTREE index on the join key, using 64 Ray workers (2 cpu each):

Phase	Time
Dedupe (repartition + sort + drop dups)	11.0 s
Plan (index probe)	36.1 s
Apply + commit	2.7 s
Total	49.8 s

Peak memory ~5.5 GB.

Test plan

• uv run pytest tests/test_merge_into.py
• uv run ruff check lance_ray tests
• uv run ruff format --check lance_ray tests