Repository navigation
Add job dependency chaining (depends_on) - #7
Merged
Merged
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live integration run currently blocked: the cluster rejects the local test config's '#BSUB -P scicompsoft' project (pre-existing; all integration tests fail the same way until tests/cluster_config.yaml gets a valid -P project). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
depends_onoption tosubmit()/submit_array()so a job starts only after all its dependency jobs succeed, and is auto-cancelled if any dependency fails. Motivated by the org-wide LSF survey (tensorswitch's hand-rolledbsub -K/bwaitcoordinator scripts; twocof's raw-w 'ended(...)').depends_on: Sequence[JobRecord | str] | Noneon both submit methods; resolved ids stored onJobRecord.depends_on. Fan-in supported via the list form. Fully backward compatible (optional kwarg + defaulted field).bsub -w "done(a) && done(b)" -ti— chains live in the scheduler, survive client restarts, and LSF terminates orphaned dependents itself (no jobs stuck PEND forever).waiting-Nid and spawn when deps finish; killed withmetadata["dependency_failed"]if a dep fails.metadata["dependency_failed"]soon_failurecallbacks can distinguish upstream failure from the job's own.Design spec:
docs/superpowers/specs/2026-07-24-job-dependency-chaining-design.md(in base PR #6's branch)Test plan
pixi run check— 175 passed, lint clean-w/-tiargs (single, fan-in, array, default-absent)@StephanPreibisch @dchen116