Repository navigation
LSF: key array elements by JOBINDEX so parent status aggregates correctly - #9
Merged
Merged
Conversation
…ctly bjobs -json reports the bare parent ID in JOBID for every array element and puts the index only in JOBINDEX, which was not requested. All element rows collapsed into one dict entry (last row wins), and poll() then set the parent's status directly from that row instead of aggregating via compute_array_status(). Because bjobs lists DONE elements last, the parent flipped to DONE as soon as any single element finished, and array_elements was never populated. Request jobindex and re-key element rows as "parent[idx]" so the existing element-handling paths in poll() and reconnect() work. Fixes #8. Also reported as JaneliaSciComp/fileglancer#470. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
LSF reports the bare parent ID in JOBID and the index in JOBINDEX; the fixtures previously used a "12345[1]" JOBID that LSF never emits. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The previous commit swept in a local WIP change to lsf.py via a mistaken `git stash pop`. Restore lsf.py to the fix commit's version. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
JobMonitor.wait_for()resolved for an LSF array job while 3 of 4 elements were still running (#8, also JaneliaSciComp/fileglancer#470).Root cause
Real
bjobs -jsonoutput reports the bare parent ID inJOBIDfor every array element. The index lives only inJOBINDEX, which_BJOBS_FIELDSdid not request. Verified on login1:{"JOBID":"154791288", "JOBINDEX":"195", "STAT":"RUN", "JOB_NAME":"r1_bc1_38JD_a100[195]"}Consequences:
_parse_job_statuseskeyed every element row by the sameJOBID, so all rows collapsed into one dict entry. Last row wins.poll()only recognizes elements by a12345[1]key. The bare ID fell into the single-job branch and set the parent'sstatusdirectly, bypassingcompute_array_status().bjobs -alists elements grouped RUN, then PEND, then DONE last (confirmed across live arrays). So the first poll after any one element finished saw a DONE row last and marked the parent DONE.array_elementsstayed empty.The unit fixtures used
"JOBID": "12345[1]", a format LSF never emits, which is why this wasn't caught.Fix
jobindexto_BJOBS_FIELDS._parse_job_statuses, re-key rows with a nonzeroJOBINDEXasparent[idx]. The existing element paths inpoll()andreconnect()then work as designed.Tests
test_real_lsf_format_uses_jobindex: parser re-keys real-format rows and leavesJOBINDEX: "0"singles alone.test_poll_real_lsf_format_one_element_done: reproduces the issue timeline (DONE row listed last, bare IDs). Parent stays RUNNING with all 4 elements tracked.Both failed before the fix.
pixi run checkpasses (177 tests).@StephanPreibisch @JaneliaSciComp/fileglancer