feat: add update_columns_from API - #5281
Open
hfutatzhanghb wants to merge 3 commits into
Open
Conversation
hfutatzhanghb
force-pushed
the
codex/update-columns-from
branch
2 times, most recently
from
August 27, 2026 13:21
a2a5858 to
b066daf
Compare
Contributor
Author
|
Hi, @yanghua . Could you please help review this PR when have free time? Thanks very much ! |
hfutatzhanghb
force-pushed
the
codex/update-columns-from
branch
from
September 1, 2026 13:25
4bf8254 to
70c57fa
Compare
Contributor
Author
|
@claude cc. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lance_ray.update_columns_from()for fragment-local updates of existing columns using Lance row addresses.Background
Lance Spark exposes
UPDATE COLUMNS FROM, which updates existing columns by matching row metadata. UnlikeMERGE INTO, it uses Lance's fragment update path and avoids a global hash join and full-row rewrites.Lance-Ray currently has additive data-evolution APIs (
add_columns,add_columns_from, andmerge_columns_from), but no equivalent fragment-level API for updating existing columns. This PR adds that primitive.API
The source Ray Dataset must contain
_rowaddrand every requested update column._fragidis optional: it is derived from_rowaddrwhen absent and validated against the encoded fragment ID when supplied.Unmatched source rows are ignored. This API only updates existing columns; it does not insert, delete, or upsert rows.
Changes
lance_ray/io.pyupdate_columns_from()_rowaddr/_fragidvalues touint64_fragidand callfragment.update_columns(..., left_on="_rowaddr", right_on="_rowaddr")LanceOperation.Updateonce against the resolved source version and surface conflicts without retrying stale workmap_groups()lance_ray/__init__.py: exportupdate_columns_from.docs/src/data-evolution.md: document the API contract and conflict behavior.tests/test_update_columns_from.py: cover partial and filtered updates, metadata preservation, schema and row-address validation, source-version conflicts, remote arguments, batch sizing, and namespace mode.Validation
python -m ruff check lance_ray/io.py lance_ray/__init__.py tests/test_update_columns_from.pypython -m ruff format --check lance_ray/io.py lance_ray/__init__.py tests/test_update_columns_from.pyuv run mypypytest tests/test_update_columns_from.py -qLocal pytest execution is unavailable on macOS x86_64 because the pinned PyLance wheel does not support that platform; integration tests run in CI on Linux.