[python][ray] Incrementally commit update_by_row_id file groups

[XiaoHongbo-Hope]

Purpose

For large update_by_row_id jobs, a single final commit makes late failures expensive because all uncommitted file groups must be rewritten.

Add optional max_groups_per_commit windows while keeping the default single atomic commit. In incremental mode, ordinary group exceptions are returned as results so successful groups can be committed before the error is raised. Worker loss and other task-level failures which bypass Python exception handling may still lose results buffered by that task.

Tests

• Relevant Python regressions: 143 passed, 10 subtests.
• Python 3.6 compatibility: 92 passed.
• Flake8, py_compile, and diff checks passed.

[Zouxxyy]

Two findings from this review:

[JingsongLi]

Just looking at the changes alone, it's too complicated; we need a simpler, more straightforward approach.

[XiaoHongbo-Hope]

Just looking at the changes alone, it's too complicated; we need a simpler, more straightforward approach.

Got your point, just left core change now.

[leaves12138]

I found one blocking cleanup issue. The focused Ray update/merge/read test suites pass, but an incremental driver-side commit failure can abandon already-materialized group outputs.

[JingsongLi]

Incremental commits are not atomic across the whole operation. Ordinary exceptions
raised while applying a group are reported as results so other groups can finish;
completed groups are flushed before the error is raised.
Ray task retry options in ray_remote_args do not retry group exceptions reported
as results, so a transient group failure can leave a partial update. Worker loss and
other task-level failures which bypass Python exception handling may still lose
results buffered by that task.

If this is indeed the case, as stated in your documentation, then the implications are significant—far beyond what a simple description like “max_groups_per_commit” can capture. This is a major semantic change that has been overlooked.

[XiaoHongbo-Hope]

Incremental commits are not atomic across the whole operation. Ordinary exceptions
raised while applying a group are reported as results so other groups can finish;
completed groups are flushed before the error is raised.
Ray task retry options in ray_remote_args do not retry group exceptions reported
as results, so a transient group failure can leave a partial update. Worker loss and
other task-level failures which bypass Python exception handling may still lose
results buffered by that task.

If this is indeed the case, as stated in your documentation, then the implications are significant—far beyond what a simple description like “max_groups_per_commit” can capture. This is a major semantic change that has been overlooked.

Updated, can you help check whether this looks good.