Skip to content

[spark] Clean up orphan files left by Spark speculative execution - #8927

Open
hbgstc123 wants to merge 3 commits into
apache:masterfrom
hbgstc123:spark-speculative-writer-cleanup
Open

[spark] Clean up orphan files left by Spark speculative execution#8927
hbgstc123 wants to merge 3 commits into
apache:masterfrom
hbgstc123:spark-speculative-writer-cleanup

Conversation

@hbgstc123

Copy link
Copy Markdown

Purpose

Problem
When Spark speculative execution is enabled, Paimon Spark writers use useCommitCoordinator=false. Multiple task attempts can all finish writing and call prepareCommit(), but Spark only commits the winner's WriterCommitMessage. Loser attempts leave data/index files on disk that are never referenced by any snapshot — orphan files.
Why it happens
Spark kills the loser attempt by interrupting the task thread. Paimon writers do not abort prepared files on interruption today, and core abort/delete paths can fail with InterruptedIOException while the thread is interrupted, so loser files are often left behind until orphan cleanup runs (if ever).
What this PR does

  • Add task-side SparkAttemptCleanup to detect interruption and abort unprepared/prepared files before the attempt returns successfully.
  • Wire cleanup into Spark V1/V2 writers, data-evolution writes, format-table writes, and compact procedure.
  • Make core abort paths interrupt-tolerant (FileIO.deleteQuietlyIgnoringInterrupt, FileStoreCommitImpl.abort index cleanup, writer abort executors).
  • Register prepared files incrementally during prepareCommit so a mid-attempt kill can still abort already-finished files.

Tests

  • SparkAttemptCleanupTest, DataEvolutionTableDataWriteTest
  • FileStoreCommitAbortTest, FileIOTest, DeletionVectorsIndexFileTest
  • SparkSpeculativeWriteITCase (DSv2/V1 insert with speculation; loser cleanup and orphan-clean fallback)

@JingsongLi

Copy link
Copy Markdown
Contributor

cc @Zouxxyy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants