Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
be88d26
Updated installation scripts to report issues more clearly
Ladme Apr 27, 2026
3ce50ae
gmx-eta accepts more arguments
Kikoun18 May 1, 2026
91ac6ad
Added --all option
Kikoun18 May 1, 2026
85841b8
Clean-up and some refactoring
Ladme May 4, 2026
d30c1d1
Updated to python 3.13; updated ty; resolved type checker errors
Ladme May 4, 2026
3c674df
Fixed type errors in qq scripts
Ladme May 4, 2026
ea5ca7b
Changed CI workflow to run on push to any branch
Ladme May 4, 2026
8586761
Updated license year
Ladme May 4, 2026
7f4ff33
BatchMeta refactor
Ladme May 4, 2026
3c492b2
Batch systems registration
Ladme May 4, 2026
777fd46
Documentation for the batch interface module
Ladme May 4, 2026
d53b346
Updated python banner
Ladme May 4, 2026
c71bc65
Removed useless check
Ladme May 5, 2026
bf3f978
Respawner prototype
Ladme May 6, 2026
5561fcd
Submitting from remote machine + making submit thread-safe
Ladme May 7, 2026
8804761
Switched env var priority when resolving input dir
Ladme May 7, 2026
bd68715
Refactored resubmitting
Ladme May 7, 2026
f2a4221
Handling exceptions at config reading
Ladme May 7, 2026
27cb305
Resubmit from option
Ladme May 7, 2026
914c78e
Fixed a bug where resubmission from local storage was never possible
Ladme May 7, 2026
8cc8148
Updated changelog
Ladme May 7, 2026
c35afe0
Handling config file reading failures
Ladme May 7, 2026
1923daa
Fixed CI error
Ladme May 7, 2026
0e44586
Making CFG into a frozen dataclass
Ladme May 7, 2026
f416036
Default archive dir and archive format are now configurable
Ladme May 7, 2026
fba1c33
Updated readme
Ladme May 7, 2026
3cbfb8e
Interpreters now support arguments
Ladme May 7, 2026
71f3785
Updated changelog
Ladme May 7, 2026
07b19ab
Doc string formatting
Ladme May 7, 2026
939cfc6
Respawner refactoring, docs, and tests
Ladme May 9, 2026
f3d9e03
Docs for respawn and resubmit
Ladme May 9, 2026
10c6ef2
Refactored qq go, info, kill, respawn, sync, wipe
Ladme May 9, 2026
d9a9e0f
Updated changelog
Ladme May 9, 2026
be8c5b7
Removed unused method
Ladme May 9, 2026
5d58bda
Wording
Ladme May 9, 2026
ed4db98
Changed the way resubmission hosts are propagated
Ladme May 10, 2026
4da2547
Restructured tests
Ladme May 10, 2026
1500305
Moved Interpreter into properties module
Ladme May 10, 2026
b24e428
Updated help for --interpreter option
Ladme May 10, 2026
9ed8855
Exporting new classes
Ladme May 10, 2026
a9030af
qq clear --dir option
Ladme May 10, 2026
43122bb
Build action now runs via a self-hosted runner
Ladme May 12, 2026
8c1716b
Updated changelog
Ladme May 12, 2026
ff3be2a
Merge branch 'main' into v0.11
Ladme May 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 21 additions & 23 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,40 +2,38 @@ name: CI

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
test:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v5

- name: Install uv
uses: astral-sh/setup-uv@v7
with:
enable-cache: true
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
enable-cache: true

- name: Install dependencies
run: uv sync --all-groups
- name: Install dependencies
run: uv sync --all-groups

- name: Run ruff check
run: uv run ruff check .
- name: Run ruff check
run: uv run ruff check .

- name: Run ruff format check
run: uv run ruff format --check .
- name: Run ruff format check
run: uv run ruff format --check .

- name: Run ty check
run: uv run ty check .
- name: Run ty check
run: uv run ty check .

- name: Run pytest
run: uv run pytest --cov --cov-report=xml
- name: Run pytest
run: uv run pytest --cov --cov-report=xml

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
file: ./coverage.xml
fail_ci_if_error: false
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
file: ./coverage.xml
fail_ci_if_error: false
98 changes: 49 additions & 49 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,56 +9,56 @@ permissions:

jobs:
build-and-upload:
runs-on: ubuntu-22.04
runs-on: [self-hosted, Linux, X64]

steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- uses: actions/checkout@v5
with:
fetch-depth: 0

- name: Install uv
uses: astral-sh/setup-uv@v7
with:
enable-cache: true

- name: Install dependencies and pyinstaller
run: |
uv sync --all-groups

- name: Get version from tag
id: version
run: |
TAG="${{ github.event.release.tag_name }}"
VERSION="${TAG#v}"
echo "version=$VERSION" >> "$GITHUB_OUTPUT"

- name: Build with PyInstaller
run: |
uv run pyinstaller qq.spec

- name: Add VERSION file to package directory
run: |
echo "${{ steps.version.outputs.version }}" > dist/qq/VERSION

- name: Update version in installation scripts
run: |
VERSION="${{ steps.version.outputs.version }}"
for script in scripts/installation_scripts/*; do
sed -i "s/__VERSION__/$VERSION/g" "$script"
done

- name: Install uv
uses: astral-sh/setup-uv@v7
with:
enable-cache: true

- name: Install dependencies and pyinstaller
run: |
uv sync --all-groups

- name: Get version from tag
id: version
run: |
TAG="${{ github.event.release.tag_name }}"
VERSION="${TAG#v}"
echo "version=$VERSION" >> "$GITHUB_OUTPUT"

- name: Build with PyInstaller
run: |
uv run pyinstaller qq.spec

- name: Add VERSION file to package directory
run: |
echo "${{ steps.version.outputs.version }}" > dist/qq/VERSION
- name: Create tarball
run: |
cd dist
tar -czf ../qq-release.tar.gz qq/
cd ..

- name: Update version in installation scripts
run: |
VERSION="${{ steps.version.outputs.version }}"
for script in scripts/installation_scripts/*; do
sed -i "s/__VERSION__/$VERSION/g" "$script"
done
- name: Upload release assets
run: |
# upload the package
gh release upload ${{ github.event.release.tag_name }} qq-release.tar.gz

- name: Create tarball
run: |
cd dist
tar -czf ../qq-release.tar.gz qq/
cd ..

- name: Upload release assets
run: |
# upload the package
gh release upload ${{ github.event.release.tag_name }} qq-release.tar.gz

# upload installation scripts
gh release upload ${{ github.event.release.tag_name }} scripts/installation_scripts/*
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# upload installation scripts
gh release upload ${{ github.event.release.tag_name }} scripts/installation_scripts/*
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
2 changes: 1 addition & 1 deletion .python-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.12
3.13
35 changes: 34 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,36 @@
## Version 0.11
### qq respawn
- Failed or killed jobs can now be easily "respawned" using `qq respawn`. When respawning a job, qq will remove the working directory of the failed job, clear all runtime files, and resubmit the job with the same parameters as before.

### Specifying multiple job IDs
- `qq info`, `qq kill`, `qq sync`, `qq respawn`, `qq wipe`, and `qq go` now accept multiple job IDs as arguments.
- All of these commands now also internally resolve the jobs in parallel, making them much faster when dealing with a large number of jobs.

### Clearing runtime files in a specified directory
- `qq clear` now supports clearing runtime files in a directory other than the current one via the `-d`/`--dir` flag.

### Better support for non-bash interpreters
- Interpreters now support additional command-line arguments.

### Resubmitting with fallback hosts
- Resubmission of loop and continuous jobs now supports multiple fallback hosts. Previously, jobs were resubmitted from a single machine (the input machine or the working node, depending on the batch system). A list of hosts can now be specified via `--resubmit-from` or in the config file; they are tried in order until one succeeds, making resubmission more resilient to individual machine failures. See [the manual](https://vachalab.github.io/qq-manual/resubmit_hosts.html) for more information.

### Other changes
- Updated the installation scripts to more clearly report issues that occurred during the install.

### Internal changes
- qq now uses Python 3.13 for better generics support.
- Fixed type errors in qq scripts.
- Refactored BatchMeta.
- Jobs can now be submitted from a remote machine (only via Python API).
- Submitting jobs using `Submitter.submit` is now thread-safe.
- When reading a configuration file fails, an exception is no longer raised; instead, an error is reported and a default configuration is used.
- `CFG` is now a frozen dataclass.
- Default archive directory and archive format are now configurable.
- Changed the internal representation of the interpreter specified to execute the script.

***

## Version 0.10.1
- The minimal width of the job status info panel was slightly increased to better accomodate long node names.
- **Bug fix:** None is no longer displayed in the subtitle of the panel when using `qq nodes`.
Expand Down Expand Up @@ -166,4 +199,4 @@

### Internal changes
- Renamed PBSJobInfo to PBSJob.
- Set up GitHub Actions to take care of releases.
- Set up GitHub Actions to take care of releases.
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2025 Ladislav Bartos and Robert Vacha Lab
Copyright (c) 2025-2026 Ladislav Bartos and Robert Vacha Lab

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# <img src="assets/qq_logo.png" alt="qq logo" width="40" style="vertical-align: middle;"/> : A friendly interface to batch processing
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/downloads/)
[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
[![CI](https://github.com/ladme/qq/actions/workflows/ci.yml/badge.svg)](https://github.com/ladme/qq/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/ladme/qq/branch/main/graph/badge.svg)](https://codecov.io/gh/ladme/qq)
Expand All @@ -9,4 +9,4 @@

Read the [qq manual](https://vachalab.github.io/qq-manual) for more information.

**Designed for the specific needs of the [RoVa Research Group](https://vacha.ceitec.cz/).**
**Designed for the specific needs of the [RoVa Research Group](https://vacha.ceitec.cz/).**
5 changes: 2 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
name = "qq"
description = "A friendly interface to batch processing"
readme = "README.md"
requires-python = ">=3.12"
requires-python = ">=3.13"
license = { text = "MIT" }
authors = [{ name = "Ladislav Bartos", email = "ladmeb@gmail.com" }]
keywords = ["batch", "processing", "cli", "automation"]
Expand All @@ -16,7 +16,6 @@ classifiers = [
"License :: OSI Approved :: MIT License",
"Operating System :: POSIX :: Linux",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Topic :: Scientific/Engineering",
"Topic :: System :: Distributed Computing",
Expand Down Expand Up @@ -51,7 +50,7 @@ dev = [
"pytest-cov>=7.0.0",
"ruff>=0.13.0",
"snakeviz>=2.2.2",
"ty>=0.0.18",
"ty>=0.0.34",
]


Expand Down
20 changes: 17 additions & 3 deletions scripts/installation_scripts/qq-karolina-install.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash
# Installs qq on IT4Innovations' Karolina.
# Script version: 0.3.0
# Script version: 0.4.0

set -euo pipefail

Expand Down Expand Up @@ -32,19 +32,33 @@ chmod +x "$TMP_INSTALLER"

echo "INFO [qq karolina installer] Installing qq ${QQ_VERSION} from ${RELEASE_URL}"

MISSING_HOMES=()

for HOME_DIR in "${TARGET_HOMES[@]}"; do
echo "--------------------------------------------"
echo "INFO [qq karolina installer] Installing qq into $HOME_DIR ..."
if [ -d "$HOME_DIR" ]; then
"$TMP_INSTALLER" "$HOME_DIR" "$RELEASE_URL"
else
echo "WARN [qq karolina installer] Skipping $HOME_DIR (directory not found)"
MISSING_HOMES+=("$HOME_DIR")
fi
done

echo "--------------------------------------------"
echo "INFO [qq karolina installer] qq installation completed for all target home directories."
echo "INFO [qq karolina installer] Run 'source ${HOME}/.bashrc' to make qq available on the current machine."

if [ ${#MISSING_HOMES[@]} -gt 0 ]; then
echo "WARN [qq karolina installer] qq installation failed for the following home directories:"
for MISSING in "${MISSING_HOMES[@]}"; do
echo "WARN [qq karolina installer] - $MISSING"
done
else
echo "INFO [qq karolina installer] qq installation completed for all target home directories."
fi

if [[ ! " ${MISSING_HOMES[*]} " == *" ${HOME} "* ]]; then
echo "INFO [qq karolina installer] Run 'source ${HOME}/.bashrc' to make qq available on the current machine."
fi

# Cleanup
rm -f "$TMP_INSTALLER"
20 changes: 17 additions & 3 deletions scripts/installation_scripts/qq-lumi-install.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash
# Installs qq on the Lumi supercomputer.
# Script version: 0.2.0
# Script version: 0.3.0

set -euo pipefail

Expand Down Expand Up @@ -32,19 +32,33 @@ chmod +x "$TMP_INSTALLER"

echo "INFO [qq lumi installer] Installing qq ${QQ_VERSION} from ${RELEASE_URL}"

MISSING_HOMES=()

for HOME_DIR in "${TARGET_HOMES[@]}"; do
echo "--------------------------------------------"
echo "INFO [qq lumi installer] Installing qq into $HOME_DIR ..."
if [ -d "$HOME_DIR" ]; then
"$TMP_INSTALLER" "$HOME_DIR" "$RELEASE_URL"
else
echo "WARN [qq lumi installer] Skipping $HOME_DIR (directory not found)"
MISSING_HOMES+=("$HOME_DIR")
fi
done

echo "--------------------------------------------"
echo "INFO [qq lumi installer] qq installation completed for all target home directories."
echo "INFO [qq lumi installer] Run 'source ${HOME}/.bashrc' to make qq available on the current machine."

if [ ${#MISSING_HOMES[@]} -gt 0 ]; then
echo "WARN [qq lumi installer] qq installation failed for the following home directories:"
for MISSING in "${MISSING_HOMES[@]}"; do
echo "WARN [qq lumi installer] - $MISSING"
done
else
echo "INFO [qq lumi installer] qq installation completed for all target home directories."
fi

if [[ ! " ${MISSING_HOMES[*]} " == *" ${HOME} "* ]]; then
echo "INFO [qq lumi installer] Run 'source ${HOME}/.bashrc' to make qq available on the current machine."
fi

# Cleanup
rm -f "$TMP_INSTALLER"
20 changes: 17 additions & 3 deletions scripts/installation_scripts/qq-metacentrum-install.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash
# Installs qq on your current desktop and on the computing nodes of all Metacentrum clusters.
# Script version: 0.5.0
# Script version: 0.6.0

set -euo pipefail

Expand Down Expand Up @@ -78,13 +78,16 @@ echo "INFO [qq metacentrum installer] Downloading qq setup from ${SETUP_SCRIP
curl -fsSL -o "$TMP_SETUP" "$SETUP_SCRIPT_URL"
chmod +x "$TMP_SETUP"

MISSING_HOMES=()

for HOME_DIR in "${TARGET_HOMES[@]}"; do
echo "--------------------------------------------"
echo "INFO [qq metacentrum installer] Linking qq to ${HOME_DIR}..."
if [ -d "$HOME_DIR" ]; then
"$TMP_SETUP" "$HOME_DIR" "${MAIN_HOME}/qq"
else
echo "WARN [qq metacentrum installer] Skipping ${HOME_DIR} (directory not found)"
MISSING_HOMES+=("$HOME_DIR")
fi
done

Expand All @@ -101,8 +104,19 @@ for HOST in "${LOCAL_HOME_HOSTS[@]}"; do
done

echo "--------------------------------------------"
echo "INFO [qq metacentrum installer] qq installation completed for all target home directories."
echo "INFO [qq metacentrum installer] Run 'source ${HOME}/.bashrc' to make qq available on the current machine."

if [ ${#MISSING_HOMES[@]} -gt 0 ]; then
echo "WARN [qq metacentrum installer] qq installation failed for the following home directories:"
for MISSING in "${MISSING_HOMES[@]}"; do
echo "WARN [qq metacentrum installer] - $MISSING"
done
else
echo "INFO [qq metacentrum installer] qq installation completed for all target home directories."
fi

if [[ ! " ${MISSING_HOMES[*]} " == *" ${HOME} "* ]]; then
echo "INFO [qq metacentrum installer] Run 'source ${HOME}/.bashrc' to make qq available on the current machine."
fi

# Cleanup
rm -f "$TMP_INSTALLER"
Loading
Loading