Skip to content

add muse2 main branch install to ci #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Aurashk wants to merge 9 commits into
base: main
Choose a base branch
from
+193 −112
Open

add muse2 main branch install to ci #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
267aeec
add muse2 main branch install to ci
Aurashk Apr 24, 2026
e76effd
correct cargo install command
Aurashk Apr 24, 2026
9cfb02c
clean up CI naming
Aurashk Apr 24, 2026
baab108
add check the the file at muse2 path is executable
Aurashk Apr 27, 2026
e335c0c
remove string check
Aurashk Apr 27, 2026
ab0b170
make codecov token required instead of checking it's there
Aurashk Apr 27, 2026
e2fc632
add locked flag to cargo install
Aurashk Apr 27, 2026
351e279
on windows must be exe
Aurashk Apr 27, 2026
dd6004c
swap curl for gh cli for downloading releases
Aurashk Apr 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/actions/setup-muse2-main/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: Setup MUSE2 from main
description: Build and configure MUSE2_PATH from the MUSE2 main branch via cargo install
inputs:
muse2_exe:
description: MUSE2 executable name for the current OS
required: true
runs:
using: composite
steps:
- name: Install Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1

- name: Build and install MUSE2 from main (cargo)
shell: bash
run: |
cargo install \
--git https://github.com/EnergySystemsModellingLab/MUSE2 \
--branch main \
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cargo install from a git repo should generally be run with --locked to ensure the build uses the repo's Cargo.lock (and fails if it's out of date) rather than implicitly updating dependencies, which can make CI flaky and less reproducible.

Suggested change
--branch main \
--branch main \
--locked \

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

@Aurashk Aurashk Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems sensible ?

--root "$GITHUB_WORKSPACE/muse2_cargo_root" \
Copy link
Copy Markdown
Member

@alexdewar alexdewar Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the --root option needed?

Copy link
Copy Markdown
Contributor Author

@Aurashk Aurashk Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the alternative is to point MUSE2_PATH at wherever the default root is instead ($HOME/.cargo/?), I don't have too much of a preference if you think that's better

Copy link
Copy Markdown
Member

@alexdewar alexdewar Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see! Well, I think cargo will install muse2 so that it's on the PATH and then the code will be able to find it directly, without you having to set MUSE2_PATH. See:

muse2_data_analysis/muse2_data_analysis/muse2.py

Line 25 in c5e3a08

path = os.getenv("MUSE2_PATH") or shutil.which("muse2")

I think that would be slightly cleaner.

--locked

- name: Set MUSE2_PATH (Linux / macOS)
if: runner.os == 'Linux' || runner.os == 'macOS'
shell: bash
run: |
echo "MUSE2_PATH=${{ github.workspace }}/muse2_cargo_root/bin/${{ inputs.muse2_exe }}" >> "$GITHUB_ENV"

- name: Set MUSE2_PATH (Windows)
if: runner.os == 'Windows'
shell: pwsh
run: |
$exePath = "${{ github.workspace }}\muse2_cargo_root\bin\${{ inputs.muse2_exe }}"
echo "MUSE2_PATH=$exePath" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
Copy link
Copy Markdown
Member

@alexdewar alexdewar Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use the bash shell on Windows too btw. I'm not sure about passing the exe file name to this Action though. I feel like it should "know" that it needs a .exe extension for Windows. Same for the other Action.

Copy link
Copy Markdown
Contributor Author

@Aurashk Aurashk Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess my resistance to this is that you would have to update the exe name in multiple places if we ever changed it for some reason... Not that that would be too cumbersome though, do you prefer just having the actual names if the os branches rather than inputs.muse2_exe?

Copy link
Copy Markdown
Member

@alexdewar alexdewar Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you take my suggestion above, I don't think you'll need the name of the muse2 exe at all.

41 changes: 41 additions & 0 deletions .github/actions/setup-muse2-release/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Setup MUSE2 from latest release
description: Download latest MUSE2 release asset for current OS and configure MUSE2_PATH
inputs:
github_token:
description: GitHub token for API access
required: true
muse2_exe:
description: MUSE2 executable name for the current OS
required: true
runs:
using: composite
steps:
- name: Download MUSE2 release asset
shell: bash
env:
GITHUB_TOKEN: ${{ inputs.github_token }}
run: |
gh release download \
--repo EnergySystemsModellingLab/MUSE2 \
--pattern "${{ runner.os == 'Windows' && 'muse2_windows.zip' || runner.os == 'macOS' && 'muse2_macos_arm.tar.gz' || 'muse2_linux.tar.gz' }}" \
--output "muse2_asset_download" \
--clobber

- name: Extract and set MUSE2_PATH (Linux / macOS)
if: runner.os == 'Linux' || runner.os == 'macOS'
shell: bash
run: |
mkdir -p muse2_bin
tar -xf muse2_asset_download -C muse2_bin
chmod +x "muse2_bin/${{ inputs.muse2_exe }}"
echo "MUSE2_PATH=${{ github.workspace }}/muse2_bin/${{ inputs.muse2_exe }}" >> "$GITHUB_ENV"

- name: Extract and set MUSE2_PATH (Windows)
if: runner.os == 'Windows'
shell: pwsh
run: |
New-Item -ItemType Directory -Force -Path muse2_bin | Out-Null
Rename-Item -Path "muse2_asset_download" -NewName "muse2_asset_download.zip"
Expand-Archive -Path "muse2_asset_download.zip" -DestinationPath muse2_bin -Force
$exePath = "${{ github.workspace }}\muse2_bin\${{ inputs.muse2_exe }}"
echo "MUSE2_PATH=$exePath" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
34 changes: 34 additions & 0 deletions .github/actions/verify-muse2/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Verify MUSE2 installation
description: Validate that MUSE2_PATH points to an executable file for the current OS
Comment thread
Aurashk marked this conversation as resolved.
runs:
using: composite
steps:
- name: Verify MUSE2_PATH (Linux / macOS)
if: runner.os == 'Linux' || runner.os == 'macOS'
shell: bash
run: |
if [[ ! -f "$MUSE2_PATH" ]]; then
echo "::error::MUSE2 executable not found at: $MUSE2_PATH"
Comment thread
Aurashk marked this conversation as resolved.
exit 1
fi
if [[ ! -x "$MUSE2_PATH" ]]; then
echo "::error::MUSE2 file at $MUSE2_PATH is not executable"
exit 1
fi
echo "MUSE2_PATH is set to: $MUSE2_PATH"

- name: Verify MUSE2_PATH (Windows)
if: runner.os == 'Windows'
shell: pwsh
run: |
if (-not (Test-Path -Path $env:MUSE2_PATH -PathType Leaf)) {
Write-Error "MUSE2 executable not found at: $env:MUSE2_PATH"
exit 1
}
$executableExtensions = @('.exe')
$extension = [System.IO.Path]::GetExtension($env:MUSE2_PATH).ToLower()
if ($executableExtensions -notcontains $extension) {
Write-Error "MUSE2 file at $env:MUSE2_PATH does not have an executable extension"
exit 1
}
Write-Host "MUSE2_PATH is set to: $env:MUSE2_PATH"
126 changes: 14 additions & 112 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Test and build
name: Test against MUSE2
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR/issue description mentions running the MUSE2-main build periodically, but ci.yml doesn't add a schedule trigger (it only runs on push/PR/manual). Either add a cron schedule for the main-branch job or adjust the description/issue expectations.

Copilot uses AI. Check for mistakes.
on:
push:
branches: [main]
Expand All @@ -7,114 +7,16 @@ on:
workflow_call:

jobs:
test:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
python-version: '3.14'
muse2_asset: muse2_linux.tar.gz
muse2_exe: muse2
- os: windows-latest
python-version: '3.14'
muse2_asset: muse2_windows.zip
muse2_exe: muse2.exe
- os: macos-latest
python-version: '3.14'
muse2_asset: muse2_macos_arm.tar.gz
muse2_exe: muse2

steps:
- uses: actions/checkout@v6

- uses: astral-sh/setup-uv@v7
with:
enable-cache: true
prune-cache: false
activate-environment: true
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: uv sync

# Query the GitHub API for the latest MUSE2 release tag (e.g. "v2.0.0").
# The tag is written to GITHUB_OUTPUT so subsequent steps can reference it.
- name: Get latest MUSE2 release tag
id: muse2_release
shell: bash
run: |
TAG=$(curl -sSf \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${{ github.token }}" \
https://api.github.com/repos/EnergySystemsModellingLab/MUSE2/releases/latest \
| jq -r '.tag_name')

if [[ -z "$TAG" || "$TAG" == "null" ]]; then
echo "::error::Failed to retrieve latest MUSE2 release tag."
exit 1
fi

echo "Resolved latest MUSE2 release: $TAG"
echo "tag=$TAG" >> "$GITHUB_OUTPUT"

# Download the platform-appropriate MUSE2 release asset.
# 'shell: bash' is used on all platforms
- name: Download MUSE2 release asset
shell: bash
run: |
curl -sSfL \
"https://github.com/EnergySystemsModellingLab/MUSE2/releases/download/${{ steps.muse2_release.outputs.tag }}/${{ matrix.muse2_asset }}" \
--output "muse2_asset_download"

- name: Extract MUSE2 and set environment variable (Linux / macOS)
if: runner.os == 'Linux' || runner.os == 'macOS'
shell: bash
run: |
mkdir -p muse2_bin
tar -xf muse2_asset_download -C muse2_bin
chmod +x "muse2_bin/${{ matrix.muse2_exe }}"
echo "MUSE2_PATH=${{ github.workspace }}/muse2_bin/${{ matrix.muse2_exe }}" >> "$GITHUB_ENV"

- name: Extract MUSE2 and set environment variable (Windows)
if: runner.os == 'Windows'
shell: pwsh
run: |
New-Item -ItemType Directory -Force -Path muse2_bin | Out-Null
Rename-Item -Path "muse2_asset_download" -NewName "muse2_asset_download.zip"
Expand-Archive -Path "muse2_asset_download.zip" -DestinationPath muse2_bin -Force
$exePath = "${{ github.workspace }}\muse2_bin\${{ matrix.muse2_exe }}"
echo "MUSE2_PATH=$exePath" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append

# Confirm the muse2 executable is present and the environment variable is set correctly.
- name: Verify MUSE2 installation (Linux / macOS)
if: runner.os == 'Linux' || runner.os == 'macOS'
shell: bash
run: |
if [[ ! -f "$MUSE2_PATH" ]]; then
echo "::error::MUSE2 executable not found at: $MUSE2_PATH"
exit 1
fi
echo "MUSE2_PATH is set to: $MUSE2_PATH"

- name: Verify MUSE2 installation (Windows)
if: runner.os == 'Windows'
shell: pwsh
run: |
if (-not (Test-Path -Path $env:MUSE2_PATH -PathType Leaf)) {
Write-Error "MUSE2 executable not found at: $env:MUSE2_PATH"
exit 1
}
Write-Host "MUSE2_PATH is set to: $env:MUSE2_PATH"

- name: Run tests
run: pytest

- name: Upload coverage to Codecov
if: runner.os == 'Linux'
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./coverage.xml
fail_ci_if_error: true
test_release:
name: Latest MUSE2 release
uses: ./.github/workflows/test-with-muse2.yml
with:
muse2_source: release
secrets: inherit

test_main:
name: Current MUSE2 main branch
uses: ./.github/workflows/test-with-muse2.yml
with:
muse2_source: main
secrets: inherit
Comment on lines +17 to +22
Copy link

Copilot AI Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds a full “build MUSE2 from main” test job to the standard CI on every push/PR, but the PR description/issue calls out running the main-branch build periodically. If the intent is periodic coverage, consider gating test_main behind a schedule/manual trigger (or making it conditional) to avoid significantly increasing CI duration and external network/build load on every PR.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

@Aurashk Aurashk Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you probably need this in PR's too though?

Copy link
Copy Markdown
Member

@alexdewar alexdewar Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we def want it for PRs, but it would be good to also run this every week or so. We have an issue for this (#43) and I've taken the liberty of assigning you to it just now 😉.

You just need to add a cron trigger for the workflow.

Copy link
Copy Markdown
Member

@alexdewar alexdewar Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you do this in this PR while you're at it? It's just a small change to this file.

71 changes: 71 additions & 0 deletions .github/workflows/test-with-muse2.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
name: Test with MUSE2

on:
workflow_call:
inputs:
muse2_source:
description: Which MUSE2 source to test against (release or main)
required: true
type: string
secrets:
CODECOV_TOKEN:
required: true

Comment on lines +10 to +13
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CODECOV_TOKEN is marked as a required secret for the reusable workflow. Because required workflow_call secrets are enforced at invocation time (even when the Codecov step is skipped), this makes all callers (including the muse2_source: main run) require the token and will cause CI to fail for PRs from forks where secrets aren’t available. Consider making CODECOV_TOKEN optional and additionally gating the Codecov upload step on the token being present (or moving coverage upload to a separate workflow that’s only invoked for release).

Copilot uses AI. Check for mistakes.
jobs:
test:
name: Run Pytest
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
python-version: '3.14'
muse2_exe: muse2
- os: windows-latest
python-version: '3.14'
muse2_exe: muse2.exe
- os: macos-latest
python-version: '3.14'
muse2_exe: muse2

steps:
- uses: actions/checkout@v6

- uses: astral-sh/setup-uv@v7
with:
enable-cache: true
prune-cache: false
activate-environment: true
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: uv sync

- name: Setup MUSE2 from main
if: inputs.muse2_source == 'main'
uses: ./.github/actions/setup-muse2-main
with:
muse2_exe: ${{ matrix.muse2_exe }}
Comment on lines +45 to +49
Copy link

Copilot AI Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because MUSE2 setup is entirely conditional on inputs.muse2_source, any unexpected value will skip both setup steps and then fail later in verify-muse2 with a misleading “executable not found” error. Consider adding an explicit validation step early in the job (and a clear error message) to enforce allowed values (e.g. main/release).

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

@Aurashk Aurashk Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unlikely to come up but I added a check anyway


- name: Setup MUSE2 from latest release
if: inputs.muse2_source == 'release'
uses: ./.github/actions/setup-muse2-release
Comment on lines +45 to +53
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

muse2_source controls which setup step runs, but the workflow never validates the input. Any unexpected value will skip both setup steps and then fail in verify-muse2 with a misleading “executable not found” error. Add an explicit validation step early in the job to enforce allowed values (e.g. main/release) and emit a clear error message before proceeding.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

@Aurashk Aurashk Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there doesn't seem to be a nice way to do this within actions scripts, and I don't think it's super likely to cause an issue

Copy link
Copy Markdown
Member

@alexdewar alexdewar Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is fine.

with:
github_token: ${{ github.token }}
muse2_exe: ${{ matrix.muse2_exe }}

- name: Verify MUSE2 installation
uses: ./.github/actions/verify-muse2

- name: Run tests
run: pytest

- name: Upload coverage to Codecov
# Latest linux release only - they should all be the same
if: runner.os == 'Linux' && inputs.muse2_source == 'release'
Comment thread
Aurashk marked this conversation as resolved.
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./coverage.xml
fail_ci_if_error: true
Comment thread
Aurashk marked this conversation as resolved.
Loading