diff --git a/.github/workflows/gradle.yml b/.github/workflows/gradle.yml index f2354737c75..abad73b8034 100644 --- a/.github/workflows/gradle.yml +++ b/.github/workflows/gradle.yml @@ -31,9 +31,15 @@ on: # results would be for a superseded commit, and the run that replaces it publishes the newer one. The publish # jobs (publishGradle, publish, publishForge, verifyWrapper, docs) run on always() with explicit result checks # instead, so a publish that has started completes: an interrupted upload would leave the snapshot repository -# with modules from two commits. Nothing of the cancelled commit is published when the cancel arrives before -# publish starts, because a cancelled test job fails the explicit checks. The replacing run is pending until -# the cancelled run has finished every job, so two publishes of the same branch never overlap. +# with modules from two commits. publishGradle waits for every build job (buildGradle, buildForge and +# validateDependencies), so nothing of the cancelled commit is published when the cancel arrives before the +# builds have finished. A later cancel lets the publish jobs whose needs have already succeeded or been skipped +# run: on a run without tests that is the whole chain, and on a run whose tests are still running only +# publishGradle, because a cancelled test job fails the explicit checks of publish and of everything after it. +# A cancel after every test has passed lets the whole chain run as well. The replacing run is pending until the +# cancelled run has finished every job, so two publishes of the same branch never overlap, and it then publishes +# the newer commit over whatever the cancelled run published. A normal cancel does not stop a running always() +# job either; the CI policy section of AGENTS.md says how to force one. concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true @@ -536,14 +542,18 @@ jobs: -PonlyHibernate5Tests -PskipCodeStyle publishGradle: - # always() with an explicit result check, see the concurrency comment at the top. + # always() with explicit result checks, see the concurrency comment at the top. It waits for every build job, + # not only buildGradle, so that a run cancelled before its builds have finished publishes nothing, and a + # commit whose Forge build or dependency validation failed is not published. if: >- ${{ always() && github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && - needs.buildGradle.result == 'success' + needs.buildGradle.result == 'success' && + needs.buildForge.result == 'success' && + needs.validateDependencies.result == 'success' }} - needs: [ buildGradle ] + needs: [ buildGradle, buildForge, validateDependencies ] runs-on: ubuntu-24.04 steps: - name: "Output Agent IP" # in the event RAO blocks this agent, this can be used to debug it @@ -588,17 +598,19 @@ jobs: name: grails-gradle-artifacts.txt path: grails-gradle/build/grails-gradle-artifacts.txt publish: - needs: [ publishGradle, build, functional, security, securityConfig, redis, hibernate5Functional, mongodbFunctional ] + needs: [ publishGradle, build, buildRerunTasks, functional, security, securityConfig, redis, hibernate5Functional, mongodbFunctional ] # always() with explicit result checks, see the concurrency comment at the top. The results are checked # explicitly because the implicit success() check also covers the skipped test jobs upstream, which would - # skip this job on a build-only run, and because a cancelled test job must not be published: nothing of a - # commit whose run was cancelled before this job started reaches the snapshot repository. + # skip this job on a build-only run, and because a cancelled test job must not be published: the core of a + # commit whose tests were cancelled never reaches the snapshot repository, although its Gradle plugins may. + # Every test job of this workflow belongs in these needs. if: >- ${{ always() && github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && needs.publishGradle.result == 'success' && (needs.build.result == 'success' || needs.build.result == 'skipped') && + (needs.buildRerunTasks.result == 'success' || needs.buildRerunTasks.result == 'skipped') && (needs.functional.result == 'success' || needs.functional.result == 'skipped') && (needs.security.result == 'success' || needs.security.result == 'skipped') && (needs.securityConfig.result == 'success' || needs.securityConfig.result == 'skipped') && diff --git a/AGENTS.md b/AGENTS.md index cf9e3aba24c..0365e66c47e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -255,9 +255,9 @@ GitHub-hosted runner time is shared with every other Apache project, so the work - Merging a pull request, or labelling it `no ci`, cancels its runs in flight: `cancel-unneeded-pr-runs.yml` cancels the queued, pending and in-progress `pull_request` runs of every open pull request labelled `no ci`, and on a push to a release branch also those of each pull request the pushed commit merged that were created before the merge. A run is kept while an open pull request not labelled `no ci` has the same head commit, and the runs of a merged pull request labelled `run ci` are kept too: on a release branch below the default, the push after the merge builds without the tests, so the tests the label asked for on the pull request are the only ones the change gets there, and a late failure still calls for a follow-up fix. The workflow runs on every push to a release branch and when a pull request is labelled `no ci`, but a `pull_request` run from a fork gets a read-only token that cannot cancel runs, so labelling a fork's pull request skips the job, and that pull request's runs are cancelled by the next push to a release branch. For the same reason merges are found from the push rather than from the pull request being closed. It lists the unfinished `pull_request` runs of the repository once per sweep, not once per pull request, to spare the `GITHUB_TOKEN` rate limit. - Every workflow that runs tests calls `.github/workflows/ci-policy.yml` as its first job, `policy`, and gates its test jobs (and the test steps of its build jobs) on `needs.policy.outputs.run-tests`. Never test `github.event.head_commit.message` in a workflow: a `pull_request` payload has no such field, so that test silently ran the tests on every pull request. The policy reads the head commit of the push, of the pull request, or of the dispatched branch, and matches the marker against the start of the message's first line. The read is retried with a growing pause; when it still fails, the tests run, as they did before the policy existed, and a warning annotation and the step summary say so, so an API outage costs runner time rather than a test result. - The tests run by default only on the default branch and the release branches above it, and there a commit message starting with `[skip tests]` skips them. Everywhere else, that is on a release branch below the default (7.0.x, 7.1.x and 7.2.x while 8.0.x is the default) and on a pull request whose base is a side branch rather than a release branch, the tests run only on request: when the commit message starts with `[run tests]`, when the pull request carries the `run ci` label, or when a manual run (`workflow_dispatch`) in `apache/grails-core` has its `run-tests` input ticked. Otherwise the run builds, and on a push publishes snapshots, without tests, exactly like a `[skip tests]` commit; a `[skip tests]` commit also wins over the label and the input. The label is the way to test a backport pull request without rewriting its head commit, and the input is how the commit a release is cut from gets tested (see RELEASE.md); both are read when the run starts. A marker anywhere but at the start of the first line is ignored. The rule follows the default branch: when a new one is promoted, the branches below it stop running tests by default without an edit. The older release branches are kept quiet on purpose: a backport lands as one push to the oldest branch and a merge-up to each branch above it, and its tests belong on the pull request, labelled `run ci`, not on every push after it. Side branches are unreviewed development work (see the review process in CONTRIBUTING.md); their tests run when the branch is proposed to a release branch. In a fork, where the runner time is the fork owner's and not the ASF's, a manual run runs the tests unless the commit message starts with `[skip tests]`. -- The jobs that run no tests (dependency validation, the build-without-tests steps, code style, build health, and the snapshot publish) still run on every push to a release branch and on every non-draft pull request; RAT and CodeQL run on every push to a release branch and on every non-draft pull request into one, since their `pull_request` triggers list only release-branch bases. Only the `no ci` label, or a draft without `run ci`, stops them. -- The publish jobs in `gradle.yml` (`publishGradle`, `publish`, `publishForge`, `verifyWrapper`, `docs`) run on `always()` and check the result of each job they need explicitly. The explicit checks are needed because the implicit `success()` check covers every job upstream, so a skipped test job would skip them although their own needs succeeded, which is what silently skipped the Forge and docs publish on every `[skip tests]` push. `always()` rather than `!cancelled()` because a newer push to the same branch cancels the run in flight: GitHub stops only the jobs whose `if` turns false, so the build and test jobs stop, a cancelled test job's result fails the explicit checks and nothing of that commit is published, but a publish that has started completes, since an interrupted upload leaves the snapshot repository with modules from two commits. The replacing run is pending until the cancelled run has finished every job, so two publishes of the same branch never overlap. A job added after the tests follows the same pattern. -- A workflow that gains a test job gets `needs: [ policy ]` and `if: ${{ needs.policy.outputs.run-tests == 'true' }}` on that job; a workflow that gains a non-test job copies the label-and-draft `if` from its sibling jobs. +- The jobs that run no tests (dependency validation, the build-without-tests steps, code style and build health) still run on every push to a release branch and on every non-draft pull request; RAT and CodeQL run on every push to a release branch and on every non-draft pull request into one, since their `pull_request` triggers list only release-branch bases. Only the `no ci` label, or a draft without `run ci`, stops them. The snapshot publish runs on every push to a release branch and on a manual run (the docs publish only on a push), never on a pull request. +- The publish jobs in `gradle.yml` (`publishGradle`, `publish`, `publishForge`, `verifyWrapper`, `docs`) run on `always()` and check the result of each job they need explicitly. The explicit checks are needed because the implicit `success()` check covers every job upstream, so a skipped test job would skip them although their own needs succeeded, which is what silently skipped the Forge and docs publish on every `[skip tests]` push. `always()` rather than `!cancelled()` because a newer push to the same branch cancels the run in flight: GitHub stops only the jobs whose `if` turns false, so the build and test jobs stop but a publish that has started completes, since an interrupted upload leaves the snapshot repository with modules from two commits. `publishGradle` waits for every build job (`buildGradle`, `buildForge` and `validateDependencies`), so a cancel that arrives before the builds have finished publishes nothing of that commit. A later cancel lets the publish jobs whose needs have already succeeded or been skipped run: on a run without tests that is the whole chain, and on a run whose tests are still running only `publishGradle`, because a cancelled test job fails the explicit checks of `publish` and of everything after it; a cancel after every test has passed lets the whole chain run as well. The replacing run is pending until the cancelled run has finished every job, so two publishes of the same branch never overlap, and it then publishes the newer commit over whatever the cancelled run published. A normal cancel (the Cancel button or `gh run cancel`) does not stop a running `always()` job either, so a publish stuck on a slow or failing upload, which the retry step around it lets run for hours, holds the run and the branch's concurrency group until it ends; force-cancel the run to stop it: `gh api -X POST repos/apache/grails-core/actions/runs//force-cancel`. A job added after the tests follows the same pattern. +- A workflow that gains a test job gets `needs: [ policy ]` and `if: ${{ needs.policy.outputs.run-tests == 'true' }}` on that job; a workflow that gains a non-test job copies the label-and-draft `if` from its sibling jobs. In `gradle.yml`, a new test job also goes into the `needs` of `publish`, with a `success` or `skipped` check, and a new job that builds or validates without tests into those of `publishGradle`, with a `success` check; a job left out is not waited for, so a commit whose job failed or was cancelled is published anyway. ### Gradle caching diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 961e5047dbd..e7f35ef0c10 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -280,7 +280,7 @@ project, so two controls limit what runs: * A draft pull request runs no workflow unless it carries the `run ci` label; marking it ready for review starts CI. A pull request labelled `no ci` runs no workflow at all, draft or not. Labels are read when a push arrives, so add `run ci` (or remove `no ci`) and then push, or mark the draft ready, to get a run. Pushing to a draft still cancels - the run in flight for it, as every push to a pull request or a branch does. + the run in flight for it, as every push to a pull request does. * Adding `no ci` to a pull request cancels the runs it has in flight, and merging a pull request into a release branch cancels the runs started before the merge, unless the pull request is labelled `run ci`: on an older release branch the push after the merge does not run the tests, so the tests the label asked for are left to finish. A pull request diff --git a/RELEASE.md b/RELEASE.md index be4b8307aa9..872ec2f7645 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -35,6 +35,8 @@ gh workflow run gradle.yml --repo apache/grails-core --ref 7.0.x -f run-tests=tr The command line works on any branch whose `gradle.yml` has the input; the manual-run form in the GitHub UI offers it once the default branch has it. +The manual run shares its concurrency group with the pushes to that branch, so a merge to the branch while it is in progress cancels its build and test jobs, and the run started by that push does not run the tests. If the manual run ends up cancelled, start it again once nothing else is being merged to the branch. + ## 1. Staging During the staging step, we must create a source distribution & stage any binary artifacts that end users will consume.