Repository navigation
MNT Rework workflows: split concerns #16
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| name: Nightly wheels | ||
|
|
||
| on: | ||
| schedule: | ||
| # 01:01 UTC every day | ||
| - cron: "1 1 * * *" | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| build: | ||
| name: Build sdist and wheels | ||
| if: github.repository == 'scikit-learn/scikit-learn-release' | ||
| uses: ./.github/workflows/build-wheels.yml | ||
| with: | ||
| ref: main | ||
|
|
||
| nightly_upload: | ||
| name: Nightly upload | ||
| runs-on: ubuntu-latest | ||
| environment: upload_anaconda | ||
| needs: [build] | ||
|
|
||
| steps: | ||
| - name: Download sdist and wheels | ||
| uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | ||
| with: | ||
| merge-multiple: true | ||
|
|
||
| - name: Install micromamba | ||
| uses: mamba-org/setup-micromamba@d7c9bd84e824b79d2af72a2d4196c7f4300d3476 # v3.0.0 | ||
| with: | ||
| # For installation of anaconda-client, required for upload to anaconda.org | ||
| init-shell: bash | ||
| environment-name: upload-env | ||
| create-args: >- | ||
| anaconda-client | ||
|
|
||
| - name: Upload to anaconda.org | ||
| shell: bash -el {0} | ||
| env: | ||
| # anaconda-client uses this env var as the default token. | ||
| ANACONDA_API_TOKEN: ${{ secrets.SCIKIT_LEARN_NIGHTLY_UPLOAD_TOKEN }} | ||
| run: | | ||
| anaconda -q upload --force -u scientific-python-nightly-wheels ./dist/* | ||
|
|
||
| # Open/update a tracking issue on scikit-learn when this nightly fails. | ||
| # Uncomment when: | ||
| # - BOT_GITHUB_TOKEN is set on this repo (issues: write on scikit-learn) | ||
| # - scikit-learn/.github/workflows/update_tracking_issue.yml accepts | ||
| # issue_repo, always checkouts scikit-learn/scikit-learn for the script, | ||
| # and no longer requires github.repository == 'scikit-learn/scikit-learn' | ||
| # | ||
| # update-tracker: | ||
| # needs: [build] | ||
| # if: github.repository == 'scikit-learn/scikit-learn-release' | ||
| # uses: scikit-learn/scikit-learn/.github/workflows/update_tracking_issue.yml@main | ||
| # with: | ||
| # job_status: ${{ needs.build.result }} | ||
| # issue_repo: scikit-learn/scikit-learn | ||
| # secrets: | ||
| # BOT_GITHUB_TOKEN: ${{ secrets.BOT_GITHUB_TOKEN }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| name: Publish to TestPyPI or PyPI | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| version: | ||
| description: scikit-learn version to publish | ||
| required: true | ||
| type: string | ||
| target: | ||
| description: Package index to upload to | ||
| required: true | ||
| type: choice | ||
| options: ["testpypi", "pypi"] | ||
|
|
||
| run-name: Publish ${{ inputs.version }} to ${{ inputs.target }} | ||
|
|
||
| jobs: | ||
| publish: | ||
| name: Upload to ${{ inputs.target }} | ||
| if: github.repository == 'scikit-learn/scikit-learn-release' | ||
| runs-on: ubuntu-latest | ||
| environment: | ||
| name: ${{ inputs.target }} | ||
| url: ${{ inputs.target == 'pypi' && 'https://pypi.org/p/scikit-learn' || 'https://test.pypi.org/p/scikit-learn' }} | ||
| permissions: | ||
| actions: read # list runs and download artifacts | ||
| id-token: write # mandatory for trusted publishing | ||
| steps: | ||
| # find the last successful build-wheels.yml runs, manually dispatched from `main`, | ||
| # for which the sdist name matches the requested version. | ||
| - name: Find build run | ||
| id: find | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| GH_REPO: ${{ github.repository }} | ||
| SKLEARN_VERSION: ${{ inputs.version }} | ||
| run: | | ||
| set -euo pipefail | ||
| sdist="scikit_learn-${SKLEARN_VERSION}.tar.gz" | ||
| run_ids=$( | ||
| gh run list --workflow build-wheels.yml --status success --limit 10 \ | ||
| --event workflow_dispatch --branch main \ | ||
| --json databaseId --jq '.[].databaseId' | ||
| ) | ||
| for id in $run_ids; do | ||
| rm -rf sdist-check | ||
| if gh run download "$id" -n cibw-sdist -D sdist-check && find sdist-check -name "$sdist" | grep -q .; then | ||
| echo "Using https://github.com/${GH_REPO}/actions/runs/${id}" | ||
| echo "run_id=$id" >> "$GITHUB_OUTPUT" | ||
| exit 0 | ||
| fi | ||
| done | ||
| echo "No successful build-wheels.yml run with $sdist in the last 10 runs." | ||
| exit 1 | ||
|
|
||
| - name: Download sdist and wheels | ||
| uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | ||
| with: | ||
| run-id: ${{ steps.find.outputs.run_id }} | ||
| github-token: ${{ github.token }} | ||
| merge-multiple: true | ||
|
|
||
| - name: View artifacts | ||
| run: ls -l dist | ||
|
|
||
| # Make sure a tag exists and matches the built commit before uploading to PyPI. | ||
| - name: Check tag matches the built commit | ||
| if: inputs.target == 'pypi' | ||
| env: | ||
| SKLEARN_VERSION: ${{ inputs.version }} | ||
| run: | | ||
| set -euo pipefail | ||
| built=$(cat sklearn-sha.txt) | ||
| echo "built sklearn commit: $built" | ||
| git clone --depth 1 --branch "$SKLEARN_VERSION" https://github.com/scikit-learn/scikit-learn.git sklearn-tag | ||
| tagged=$(git -C sklearn-tag rev-parse HEAD) | ||
| echo "tag $SKLEARN_VERSION: $tagged" | ||
| if [[ "$built" != "$tagged" ]]; then | ||
| echo "tag ${SKLEARN_VERSION} must point at the commit that produced these artifacts." | ||
| exit 1 | ||
| fi | ||
|
Comment on lines
+67
to
+82
Member
Author
There was a problem hiding this comment. I added a check to verify that the wheels we're about to upload have been built from the same commit that was tagged. I find it's a nice safeguard. I wonder if it's too strict. For instance if we merge the release PR, build the wheels, merge a small backport - of something not impacting the wheels like for the website - in the maintenance branch, then tag it would break the check. Maybe we can start with it and relax if it proves too painful. |
||
|
|
||
| - name: Publish | ||
| uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0 | ||
| with: | ||
| repository-url: ${{ inputs.target == 'testpypi' && 'https://test.pypi.org/legacy/' || '' }} | ||
| skip-existing: ${{ inputs.target == 'testpypi' }} | ||
| print-hash: true | ||
| attestations: true | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's probably the most controversial part of this rework. To be able in
publishto reuse the artifacts frombuild-wheels, it looks at the 10 last build-wheels runs for the last one with wheels for the requested sklearn version.It mimics the current scheme in scikit-learn: we build wheels and upload them the anaconda staging area. Then the publish_pypi workflow looks in the staging area for the wheels matching the requested sklearn version.
There's a very simple alternative if we don't want that: require a
run_idas input for the publish workflow. That way you provide the run_id of the exact build-wheels workflow you want and you know exactly the wheels you'll get.It didn't use run_id so far because I felt more convenient to not have to look for the run_id, and have it been done automatically. But I can live without if we find this script to brittle :)