fix(clustered): correctly reconcile JobSet phase during restarts

[AdilFayyaz]

Why are the changes needed?

The clustered (JobSet) plugin was prematurely surfacing task failures during JobSet restarts: a single worker failure
within the restart budget could fast-fail the task, and rank-0 pod lookups used an unsuffixed name that never matched the real, randomly-suffixed pod. This caused spurious failures and missed diagnostics while a JobSet was legitimately restarting.

What changes were proposed in this pull request?

• Gate maybeFastFailWorker0 on restart budget: worker failures only surface as RetryableFailure once Status.Restarts >= MaxRestarts; otherwise the task stays Running. Pending-pod diagnostics (e.g. ImagePullBackOff) still fast-fail regardless of budget.
• Handle the RestartingJobSet condition and the no-condition-yet case via a new hasJobSetStarted check (restarts, worker readiness/activity, or prior plugin state), reporting Running with a restart-attempt reason instead of falling back to Initializing/failure.
• Replace unsuffixed rank0PodName matching with findRank0Pod, a suffix-tolerant lookup that deterministically prefer

How was this patch tested?

• go test ./flyteplugins/go/tasks/plugins/k8s/clustered/...
• New unit tests cover: suffixed/deterministic rank-0 selection, failure-with-budget-remaining → Running, budget-exhausted → RetryableFailure, pending ImagePullBackOff → fast-fail regardless of budget, RestartingJobSet condition → Running (attempt N), restarts with no true condition → Running, and plugin-state read error → fallback to status.
• Tested a failing example from the sdk on dogfood-1

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Stack

If you do use git town to manage PR Stacks, the stack relevant to this PR
will show below. Otherwise, you can ignore this section.

Docs link

[Copilot]

Pull request overview

This PR improves the clustered (JobSet) plugin’s phase reconciliation during JobSet restarts to avoid prematurely surfacing worker failures and to correctly identify rank-0 pods whose real names include controller-assigned suffixes.

Changes:

• Add suffix-tolerant rank-0 pod identification/selection and reuse it across fast-fail and maintenance-retry paths.
• Gate worker-failure fast-fail behavior on restart budget exhaustion, while still fast-failing pending-pod diagnostics.
• Add/extend unit tests covering restart-condition handling, budget gating behavior, and deterministic rank-0 selection.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
flyteplugins/go/tasks/plugins/k8s/clustered/util.go	Adds rank-0 pod matching and findRank0Pod selection logic used by phase/log inspection.
flyteplugins/go/tasks/plugins/k8s/clustered/phase.go	Updates phase reconciliation to account for restarts, restart budget, and restarting conditions.
flyteplugins/go/tasks/plugins/k8s/clustered/logs.go	Uses rank-0 prefix matching helper for primary pod selection in log context.
flyteplugins/go/tasks/plugins/k8s/clustered/clustered_test.go	Adds tests for new restart/budget logic and suffixed rank-0 pod handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.