Repository navigation
LSF: poll tracked jobs by explicit ID, not name filter - #3
Merged
Merged
Conversation
With job_name_prefix configured, _build_status_args() queried by name filter (-J prefix-*) and never passed the tracked job IDs. bjobs name filters are scoped to the calling user unless -u all is given, so a process polling other users' jobs on their behalf silently got no information about them — their statuses appeared frozen at the seeded value. Explicit job IDs are visible across users, so query tracked IDs directly and only fall back to the prefix filter when nothing is tracked. Combining -J with IDs would intersect (hiding any tracked job whose name doesn't match the prefix), so the filter is dropped when IDs are present. reconnect() keeps its own name-based discovery query. Also regenerates pixi.lock (format v6 -> v7) for the version bump and raises the CI pixi accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 8, 2026
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When
job_name_prefixis configured,LSFExecutor._build_status_args()builds the status query asbjobs -a -J "<prefix>-*" -o ... -jsonand never passes the tracked job IDs. bjobs name filters only return the calling user's jobs unless-u allis given, so a service that polls many users' jobs through a single account (e.g. Fileglancer dispatching the poll through one user's worker) silently receives no information about anyone else's jobs — their statuses stay frozen at the seeded value indefinitely.This bit us in production testing today: jobs from four users sat in PENDING in the Fileglancer database while actually RUNNING on the cluster, and users cancelled (bkilled) their own healthy jobs because they looked stuck.
Fix
Query tracked jobs by explicit ID — LSF resolves explicit job IDs across user boundaries without
-u all. The-J <prefix>-*filter now applies only when nothing is tracked yet (discovery-style call). The two are never combined, since bjobs treats name filter + IDs as an intersection, which would hide any tracked job whose name doesn't match the prefix.reconnect()is unchanged and keeps its own name-based discovery query, which genuinely needs the prefix.Verified end-to-end on the Janelia cluster: an executor with
job_name_prefix='fg', tracking four other users' job IDs seeded as PENDING and polled from an unprivileged account, resolved all four with correct statuses and exec hosts in one cycle.Also in this PR
pixi.lockregenerated for the manifest change; pixi upgraded the lock format v6 → v7, so the CI pixi is raised to v0.70.2 to match.Tests
Two new cases in
tests/test_lsf.py: prefix + tracked IDs produces an ID query with no-J(terminal jobs excluded, IDs after all options), andpoll_all_users=Truestill emits-u allalongside IDs. Fullpixi run check: 158 passed.@StephanPreibisch @JaneliaSciComp/fileglancer