Repository navigation
Conversation
Merge insert only accepted a single column as the match key, so composite-key
upserts were not expressible through the namespace API even though Lance core's
`MergeInsertBuilder` already takes a list of join columns.
Changes `on` to an array of Lance field paths (`minItems: 1`) at both sites:
the `/v1/table/{id}/merge_insert` query parameter and
`MergeInsertIntoTableRequest.on`. Regenerates the Java, Python, and Rust
clients, the Spring Boot server, and the model docs.
The query parameter uses the OpenAPI default `style: form, explode: true`, so a
one-element array serializes to `?on=id` — byte-identical to what clients send
today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
enhancement
wjones127
marked this pull request as ready for review
August 31, 2026 20:21
This was referenced Aug 31, 2026
Merged
jackye1995
approved these changes
Sep 1, 2026
wjones127
added a commit
to lancedb/lancedb
that referenced
this pull request
Sep 1, 2026
…4102) Merge insert has always taken a list of columns to match on, and local tables have always joined on all of them. Remote tables did not: any list longer than one was rejected with `MergeInsertBuilder only supports a single 'on' column`, so a composite-key upsert was impossible against LanceDB Cloud and Enterprise from Rust, Python or TypeScript. The remote request now carries `on` as a list and sends it as one repeated query parameter per column — `?on=shard_key&on=id`. That is how the lance-namespace spec encodes an array-valued `on`, so the server receives a composite key in the shape it expects. A single column still serializes to `?on=id`, exactly what clients sent before, so existing callers are unaffected. A column repeated within `on` is now rejected client-side rather than sent for the server to reject with a 400. No binding changes were needed: `Table.merge_insert` in Python and `Table.mergeInsert` in TypeScript already accepted a list, it just could not reach a remote table. Both gain a test for composite keys, and the doc comments now say what passing several columns means. Part of [ENT-2084](https://linear.app/lancedb/issue/ENT-2084/mergeinsertintotablerequest-support-multiple-columns-for-the). ## Example ```python table.merge_insert(["shard_key", "id"]) \ .when_matched_update_all() \ .when_not_matched_insert_all() \ .execute(new_data) ``` A row whose `id` matches an existing row but whose `shard_key` differs is an insert, not an update. ## Not included Java. Java callers reach merge insert through `org.lance.namespace.LanceNamespace`, whose `MergeInsertIntoTableRequest.on` is a single string until lance-namespace 0.12 ([lance-namespace#363](lance-format/lance-namespace#363), [lance#8915](lance-format/lance#8915)). There is nothing in this repo's Java SDK to change until the `lance-core` pin can move. Sending more than one column requires a server that accepts the repeated parameter ([sophon#7571](lancedb/sophon#7571)); an older server returns a 400 rather than silently merging on one column. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
wjones127
added a commit
to lance-format/lance
that referenced
this pull request
Sep 2, 2026
…#8915) Merge insert (upsert) through the namespace API only accepted a single column as the match key, so there was no way to upsert on a composite key — even though Lance core has always supported it (`MergeInsertBuilder::try_new` takes a list of join columns). The limitation was purely in the request and transport layer. `MergeInsertIntoTableRequest.on` is now a list of Lance field paths, and both implementations pass it straight through to the builder. An empty list or a repeated column is rejected with `InvalidInput`. This is the implementation half of lance-format/lance-namespace#363, which made the same change in the spec and the generated clients. ## Example ``` POST /v1/table/orders/merge_insert?on=customer_id&on=order_date&when_matched_update_all=true ``` ## Breaking changes `MergeInsertIntoTableRequest.on` changes from `Option<String>` to `Option<Vec<String>>`. Rust callers passing a single column need to wrap it in a list. The HTTP wire format is unchanged for single-column callers: the query parameter uses `style: form, explode: true`, so a one-element list still serializes to `?on=id`. An older client keeps working against a server built from this change. Moving from one match column to several also changes how NULL keys behave, because core switches NULL join semantics on the arity of the key: a single-column key treats NULL as equal to NULL, while a composite key uses standard SQL equality, under which a NULL key matches nothing — not even a byte-identical NULL. That is pre-existing core behavior, but composite keys are reachable through the namespace API for the first time here, so it is newly visible. `test_merge_insert_composite_key_never_matches_a_null_key_column` pins it down. Java and Python SDK requests reach the Rust implementations as JSON across JNI and pyo3, so a jar or wheel predating this change sends `"on": "id"` where the model now expects `"on": ["id"]`. Those four bridge sites deserialize through `LenientMergeInsertIntoTableRequest`, which promotes a scalar to a one-element list, so mismatched SDK builds keep working. This is inbound only — a Java namespace implementation called *from* Rust still needs a matching jar. ## Not included The Java and Python `lance-namespace` pins stay on 0.11, so their generated models still send `on` as a bare string and rely on the promotion described above. Moving those pins to 0.12 and adding binding-level merge-insert coverage is a follow-up. The LanceDB Enterprise namespace server needs to accept the repeated `on` query parameter separately; that change is backward compatible on its own and does not depend on this one. Part of ENT-2084. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge insert (upsert) only accepted a single column as the match key, so there was no way to upsert on a composite key through the namespace API. Lance core already supports this —
MergeInsertBuilder::try_newtakes a list of join columns — so the limitation was purely in the request/transport layer.This PR changes
onfrom a single string to an array of Lance field paths (minItems: 1) at both places it appears in the spec: the/v1/table/{id}/merge_insertquery parameter and theMergeInsertIntoTableRequestmodel. The regenerated Java, Python, and Rust clients, the Spring Boot server interface, and the model docs are included in the same commit.This is the spec-and-codegen half of the change. Wiring the list through the directory and REST implementations, and adding end-to-end coverage, happens in follow-up PRs in the
lancerepo once a release of this crate/package is cut.Example
Match on a composite key by repeating the parameter:
Breaking changes
The generated model field changes type —
String→List<String>in Java,str→List[str]in Python,Option<String>→Option<Vec<String>>in Rust — as does theonargument of the generatedmerge_insert_into_tableclient method. Callers passing a single column need to wrap it in a list.The HTTP wire format is unchanged for single-column callers. The query parameter uses the OpenAPI default
style: form, explode: true, so a one-element array serializes to?on=id, exactly what clients send today. An existing client therefore keeps working against a server built from the new spec with no change.A client sending multiple columns to a server that has not been updated fails loudly rather than silently merging on one column: Rust/axum servers backed by
serde_urlencodedreject the repeated key withduplicate field 'on'and return a 400.