Support atomic conditional overwrite writes (INSERT OVERWRITE ... WHERE)

[FANNG1]

Problem

Pipelines that rebuild one slice of a table — a day's partition, a tenant, a re-scored batch — have no atomic way to do it in daft-lance today. write_lance offers create, append, and a whole-table overwrite, so replacing a slice means LanceDataset.delete(predicate) followed by an append: two commits, with a window in between where readers see the old rows gone and the new ones not yet there. If the second commit never lands, the slice stays missing.

This is the operation Spark spells INSERT OVERWRITE ... WHERE and Delta spells replaceWhere.

Lance already has the physical primitive: LanceFragment.delete() produces updated fragment metadata without rewriting data files, and LanceOperation.Update carries updated_fragments, removed_fragment_ids and new_fragments in a single transaction. No Lance-side change is needed.

Proposed API

daft_lance.write_lance(
    df,
    uri,
    mode="overwrite_where",
    predicate="dt = DATE '2026-08-25'",
).collect()

One commit deletes the existing rows matching predicate and adds the DataFrame's rows, so readers see either the whole replacement or none of it. predicate is required by, and only valid with, this mode. Namespace-addressed tables should work the same way.

Execution and transaction semantics

• Pin the dataset version on the driver at the start of the write; that version is the commit's read version.
• Workers write the input into new, unreferenced fragments exactly as append does.
• At finalize, apply predicate to the fragments of the pinned snapshot and collect the fragments that gained a deletion file and the ones the predicate emptied.
• Commit one LanceOperation.Update(updated_fragments, removed_fragment_ids, new_fragments, fields_modified=[]). Deletions do not change field values, so indexes built on the surviving fragments stay valid; the new fragments are simply unindexed.
• Skip the commit entirely when nothing matched and nothing was written, rather than creating an empty version.
• Prune the fragments to visit with a row-address lookup when a scalar index answers the predicate; fall back to visiting every fragment when none does.

Input semantics

The delete range is decided by predicate alone and the input is appended as-is, so a row outside the predicate is never replaced by a re-run of the same write — it accumulates. The write should therefore check by default that every input row satisfies predicate, with an opt-out for callers who mean it.

Initial scope and limitations

• Existing tables only; copy-on-write path only. use_mem_wal=True is rejected.
• Schema, Blob-column and data-storage-version compatibility follow the append rules.
• No automatic retry on commit conflict; the caller re-runs the whole write.
• Exposed on daft_lance.write_lance only, not on Daft's built-in DataFrame.write_lance.

Concurrency caveat

Lance does not treat a concurrent Append as conflicting with this Update: the commit succeeds and rows another writer added during the overwrite survive it, even when they match the predicate. This is not detected today, so the API must document that the table needs a single writer for the duration of a conditional overwrite. Confirmed against lance 8.0.0 — a stale-read-version commit with a concurrent append in between succeeded even with max_retries=0.

Acceptance criteria

• Replacing a predicate's rows across several fragments, including a fragment that mixes matched and unmatched rows, produces exactly one new version.
• Re-running the same write is idempotent, including for the rows the previous run appended.
• Empty input deletes the matched rows; a predicate matching nothing only appends.
• A fragment whose rows all match is removed rather than left empty.
• Input rows outside the predicate are rejected by default and appended under the opt-out.
• Missing predicate, predicate on another mode, missing table, incompatible schema, storage-version conflict and use_mem_wal=True all fail with clear errors.
• Scalar and vector indexes still return correct results after an overwrite: deleted rows invisible, new rows found.
• The index-pruning path is exercised end to end, including a second overwrite whose target rows live in a fragment the index does not cover.
• Namespace-addressed tables work.