[python][ray] Incrementally commit update_by_row_id file groups

docs/docs/pypaimon/ray-data.md

@@ -528,6 +528,37 @@ metrics = update_by_row_id(
 print(metrics)   # {"num_updated": 50}
 ```
 
+The default is one atomic commit. To explicitly opt into non-atomic,
+incremental commits:
+
+```python
+metrics = update_by_row_id(
+    target="database_name.table_name",
+    source=ray_dataset,
+    catalog_options={"warehouse": "/path/to/warehouse"},
+    update_cols=["feature"],
+    commit_mode="incremental",
+    max_groups_per_commit=64,
+    operation_id="feature-backfill-2026-07",
+)
+```
+
+If the call fails, fix the failing input or infrastructure issue and invoke it
+again with the same `operation_id`. The checkpoint is stored with the table, so
+the new driver skips target file groups committed by the previous invocation:
+
+```python
+metrics = update_by_row_id(
+    target="database_name.table_name",
+    source=ray_dataset,
+    catalog_options={"warehouse": "/path/to/warehouse"},
+    update_cols=["feature"],
+    commit_mode="incremental",
+    max_groups_per_commit=64,
+    operation_id="feature-backfill-2026-07",  # resume
+)
+```
+
 **Parameters:**
 - `source`: a `ray.data.Dataset`, `pyarrow.Table`, or `pandas.DataFrame` carrying the
   target `_ROW_ID` and every column in `update_cols`; extra columns are ignored, and
@@ -537,6 +568,14 @@ print(metrics)   # {"num_updated": 50}
 - `num_partitions`: parallelism for grouping the update rows by target file;
   defaults to `max(1, cluster_cpus * 2)`.
 - `ray_remote_args`: Ray remote options applied to the update tasks.
+- `commit_mode`: `"atomic"` (default) commits all groups together.
+  `"incremental"` explicitly opts into non-atomic, windowed commits.
+- `max_groups_per_commit`: required in incremental mode; positive number of
+  completed target file groups per commit. It is rejected in atomic mode.
+- `operation_id`: required in incremental mode. It identifies one immutable
+  update input and its durable progress across driver restarts. Do not run two
+  callers concurrently with the same id or reuse an id for different values or
+  `update_cols`.
 
 **Returns:** `{"num_updated": <rows>}`.
 
@@ -548,6 +587,38 @@ print(metrics)   # {"num_updated": 50}
 - Partition columns cannot be updated (in-place rewrite can't move a row across partitions).
 - Deletion-vectors-enabled tables are not supported yet: a DV-deleted row still lives
   in its data file, so it can't be told apart from a live row without reading the target.
+- Incremental commits are not atomic across the whole operation. Commits completed
+  before a later failure remain visible.
+- A duplicate `_ROW_ID` within a target file group is a deterministic validation
+  error. In incremental mode it is reported as a result so successful groups can
+  finish and be committed before the error is raised.
+- Other application and I/O exceptions propagate to Ray and remain subject to the
+  retry options in `ray_remote_args`. If retries are exhausted, earlier incremental
+  commits remain visible. Worker loss may also discard successful results still
+  buffered in the failed Ray task.
+- Each incremental snapshot stores compressed cumulative row-id ranges, and
+  internal Paimon tags retain the latest checkpoint across normal snapshot
+  expiration. Larger commit windows reduce checkpoint metadata at the cost of
+  redoing more groups after a failure. A retry with the same `operation_id`
+  filters completed ranges before rewrite and returns cumulative `num_updated`.
+  If a process stops between snapshot commit and tag advancement, recovery may
+  redo at most the latest commit window.
+- The checkpoint avoids repeating Paimon's target file read/rewrite work. It
+  cannot checkpoint arbitrary transformations upstream of `update_by_row_id`.
+  Persist a multi-day or nondeterministic source first (for example to Parquet or
+  a staging Paimon table), then pass the persisted `(_ROW_ID, values)` dataset to
+  every attempt.
+- Keep the checkpoint until a successful result has been acknowledged. Afterwards
+  it can be removed explicitly:
+  ```python
+  from pypaimon.ray import delete_update_by_row_id_checkpoint
+
+  delete_update_by_row_id_checkpoint(
+      "database_name.table_name",
+      {"warehouse": "/path/to/warehouse"},
+      "feature-backfill-2026-07",
+  )
+  ```
 
 ## Read By Row Id
 

paimon-python/pypaimon/ray/__init__.py

@@ -27,7 +27,10 @@
     target_col,
     lit,
 )
-from pypaimon.ray.update_by_row_id import update_by_row_id
+from pypaimon.ray.update_by_row_id import (
+    delete_update_by_row_id_checkpoint,
+    update_by_row_id,
+)
 from pypaimon.ray.read_by_row_id import read_by_row_id
 
 __all__ = [
@@ -37,6 +40,7 @@
     "bucket_join",
     "merge_into",
     "update_by_row_id",
+    "delete_update_by_row_id_checkpoint",
     "read_by_row_id",
     "WhenMatched",
     "WhenNotMatched",

paimon-python/pypaimon/ray/data_evolution_merge_into.py

@@ -281,6 +281,8 @@ def _build_datasets(
     # snapshot the caller observed; otherwise concurrent commits in between
     # would mix data from different snapshots.
     base_snapshot_id = base_snapshot.id if base_snapshot is not None else None
+    target_empty = (
+        base_snapshot is None or base_snapshot.total_record_count == 0)
 
     update_ds = None
     delete_ds = None
@@ -317,7 +319,7 @@ def _build_datasets(
     # Mirror Spark: matched/not-matched run as two independent joins
     # (inner / left_anti). One unified left_outer join would force
     # joined.materialize() to feed both branches, which can OOM on large merges.
-    if matched_specs and base_snapshot is not None:
+    if matched_specs and not target_empty:
         update_cols_union = _union_update_cols(matched_specs)
         if update_cols_union:
             update_ds = build_matched_update_ds(
@@ -362,7 +364,7 @@ def _build_datasets(
             catalog_options=ctx.catalog_options,
             num_partitions=num_partitions,
             snapshot_id=base_snapshot_id,
-            target_empty=base_snapshot is None,
+            target_empty=target_empty,
             ray_remote_args=ray_remote_args,
         )