Skip to content

feat: add update_columns_from API - #5281

Open
hfutatzhanghb wants to merge 3 commits into
lance-format:mainfrom
hfutatzhanghb:codex/update-columns-from
Open

feat: add update_columns_from API#5281
hfutatzhanghb wants to merge 3 commits into
lance-format:mainfrom
hfutatzhanghb:codex/update-columns-from

Conversation

@hfutatzhanghb

@hfutatzhanghb hfutatzhanghb commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add lance_ray.update_columns_from() for fragment-local updates of existing columns using Lance row addresses.
  • Validate source metadata and schemas before grouping updates by fragment with Ray Data.
  • Commit updates once against the source snapshot so conflicts cannot retry stale fragment metadata.

Background

Lance Spark exposes UPDATE COLUMNS FROM, which updates existing columns by matching row metadata. Unlike MERGE INTO, it uses Lance's fragment update path and avoids a global hash join and full-row rewrites.

Lance-Ray currently has additive data-evolution APIs (add_columns, add_columns_from, and merge_columns_from), but no equivalent fragment-level API for updating existing columns. This PR adds that primitive.

API

import lance_ray as lr

source = lr.read_lance("my_dataset.lance", with_metadata=True)
source = source.map_batches(modify_status, batch_format="pandas")

lr.update_columns_from(
    "my_dataset.lance",
    source,
    columns=["status"],
)

The source Ray Dataset must contain _rowaddr and every requested update column. _fragid is optional: it is derived from _rowaddr when absent and validated against the encoded fragment ID when supplied.

Unmatched source rows are ignored. This API only updates existing columns; it does not insert, delete, or upsert rows.

Changes

  • lance_ray/io.py
    • add update_columns_from()
    • validate URI/namespace input, batch size, column names and types, row-address uniqueness, and fragment routing metadata
    • normalize integer _rowaddr / _fragid values to uint64
    • group source rows by _fragid and call fragment.update_columns(..., left_on="_rowaddr", right_on="_rowaddr")
    • preserve the Lance source version through normal Ray Data transforms when available
    • commit LanceOperation.Update once against the resolved source version and surface conflicts without retrying stale work
    • forward Ray remote arguments to map_groups()
  • lance_ray/__init__.py: export update_columns_from.
  • docs/src/data-evolution.md: document the API contract and conflict behavior.
  • tests/test_update_columns_from.py: cover partial and filtered updates, metadata preservation, schema and row-address validation, source-version conflicts, remote arguments, batch sizing, and namespace mode.

Validation

  • python -m ruff check lance_ray/io.py lance_ray/__init__.py tests/test_update_columns_from.py
  • python -m ruff format --check lance_ray/io.py lance_ray/__init__.py tests/test_update_columns_from.py
  • uv run mypy
  • pytest tests/test_update_columns_from.py -q

Local pytest execution is unavailable on macOS x86_64 because the pinned PyLance wheel does not support that platform; integration tests run in CI on Linux.

@github-actions github-actions Bot added the enhancement New feature or request label Aug 19, 2026
@hfutatzhanghb
hfutatzhanghb force-pushed the codex/update-columns-from branch 2 times, most recently from a2a5858 to b066daf Compare August 27, 2026 13:21
@hfutatzhanghb

Copy link
Copy Markdown
Contributor Author

Hi, @yanghua . Could you please help review this PR when have free time? Thanks very much !

@hfutatzhanghb
hfutatzhanghb force-pushed the codex/update-columns-from branch from 4bf8254 to 70c57fa Compare September 1, 2026 13:25
@hfutatzhanghb

Copy link
Copy Markdown
Contributor Author

@claude cc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant