[krokicki]

What this adds

A read-only curses dashboard for debugging the job system, runnable as fileglancer debug (works via uvx fileglancer debug once released) or pixi run debug-launch. Single file (fileglancer/debug.py), stdlib-only — no new dependencies in the wheel.

Panes

• Jobs (all users): statuses, cluster IDs, age, time-in-state; a toggles active-only vs all. Enter opens a detail overlay (DB fields, work-dir file probe, stderr tail) pinned to the job it was opened on.
• Processes (this host): server processes and per-user worker subprocesses (owner, age, CPU, RSS, current activity), with the poll-lock holder marked ● POLLER.
• Scheduler activity (this host): live bsub/bjobs/bkill commands as they spawn under workers, caught by /proc sampling.
• Events: status transitions from DB diffing, worker spawn/exit, poll-lock changes, observed scheduler commands. --log FILE appends the feed to a file for post-mortem analysis.
• Log viewer (o/e/s): full-screen live tail of a job's stdout/stderr/script with follow mode, scrolling, and panning. Tails the last 256 KB so multi-GB HPC logs are safe.

How it observes

Each data source is sampled by its own background thread so a slow source (NFS stat, bjobs) never blocks the UI: the DB (jobs table snapshots, diffed into events), /proc (processes + the flock holder of the poll lock file, whose mtime doubles as a poll heartbeat), job work directories (phase/service_url/job.pid/log sizes), and optionally the scheduler itself — the b key enables a bjobs cross-check that flags rows where LSF disagrees with the DB.

The cross-check found a real production bug on its first day: the DB stuck at PENDING while bjobs said RUN for other users' jobs, diagnosed as the name-filtered polling visibility issue fixed in JaneliaSciComp/py-cluster-api#3.

Behavior notes

• Process/lock/heartbeat data is host-local: when watching a remote server's database, those panes describe the local machine, and the header shows POLLER/heartbeat as n/a instead of raising false "NO POLL ACTIVITY" alarms. Work-dir probes and the log viewer work cross-host over shared mounts; the bjobs cross-check queries the cluster so it is valid from anywhere.
• DB credentials are redacted (postgresql://***@host/db) in the header, event feed, and --log output.
• Strictly read-only: never modifies the DB, never signals processes; the only scheduler command it runs itself is the opt-in bjobs cross-check.

Testing

Driven end-to-end in a pty with a seeded database: dashboard rendering, poll-lock holder detection (flock held by a child process, matched via /proc/locks), live status-transition events, detail-overlay pinning while the table re-sorts, log viewer follow mode picking up live appends, file switching, credential redaction (screen + log file), and remote-DB mode simulated in an unprivileged PID namespace (unshare --mount-proc) to verify the n/a header path.

@StephanPreibisch @JaneliaSciComp/fileglancer