From c8e1feac8302e8db6efc572f133e46d124daf8c5 Mon Sep 17 00:00:00 2001 From: JingsongLi Date: Thu, 30 Jul 2026 22:21:51 +0800 Subject: [PATCH 1/8] [docs] Add release and verification guides --- docs/docs/project/creating-a-release.md | 367 ++++++++++++++++++ .../project/verifying-a-release-candidate.md | 342 ++++++++++++++++ docs/sidebars.js | 2 + 3 files changed, 711 insertions(+) create mode 100644 docs/docs/project/creating-a-release.md create mode 100644 docs/docs/project/verifying-a-release-candidate.md diff --git a/docs/docs/project/creating-a-release.md b/docs/docs/project/creating-a-release.md new file mode 100644 index 000000000000..28e6f7689b1d --- /dev/null +++ b/docs/docs/project/creating-a-release.md @@ -0,0 +1,367 @@ +--- +title: "Creating a Release" +sidebar_position: 2 +--- + + + +# Creating a Release + +This guide is for the Release Manager (RM) of Apache Paimon and PyPaimon. It +follows the [ASF Release Policy](https://www.apache.org/legal/release-policy.html) +and the [ASF Release Distribution Policy](https://infra.apache.org/release-distribution). + +:::warning + +The signed source archives are the Apache releases. Maven and PyPI packages are +convenience artifacts. The final release must promote the source files and Java +staging repositories approved by the community; do not rebuild or replace them +after the vote. + +::: + +## Release model + +Paimon contains two independently versioned deliverables. Do not assume that +their version numbers are equal. + +| Deliverable | Candidate | Published location | +| --- | --- | --- | +| Paimon source | `apache-paimon-PAIMON_VERSION-src.tgz`, `.asc`, `.sha512` | ASF distribution | +| Java convenience artifacts | Maven artifacts built in the JDK 8, 11, and 17 lanes | Apache Nexus staging, then Maven Central | +| PyPaimon source | `pypaimon-PYPAIMON_VERSION.tar.gz`, `.asc`, `.sha512` | ASF distribution | +| Python convenience package | `pypaimon==PYPAIMON_VERSIONrcRC_NUMBER` for an RC | TestPyPI, then `pypaimon==PYPAIMON_VERSION` on PyPI | + +A combined release vote covers both signed source candidates. If PyPaimon is +released separately, use the same Python steps and hold a separate vote. + +### Java build matrix + +The three Java lanes are different release targets, not interchangeable build +JDKs: + +| JDK | Maven profiles and scope | Main artifacts | +| --- | --- | --- | +| 8 | `spark3,flink1` and the default reactor | Paimon core, Flink 1.x, Spark 3.x, Hive, filesystems, bundles, and other Java 8 artifacts | +| 11 | `flink2` plus `paimon-iceberg` | Flink 2.x, `paimon-flink2-common`, and Iceberg integration | +| 17 | `spark4` | Spark 4.x and its Scala 2.13 common artifacts | + +Each lane must use the matching JDK. Building everything on JDK 17 with a lower +compiler target is not a substitute for running the JDK 8 and JDK 11 lanes. + +## GitHub Actions release workflow + +The release process uses a **Release** workflow in +[GitHub Actions](https://github.com/apache/paimon/actions) to package every +candidate from its signed RC tag. Before cutting an RC, confirm that this +workflow is present and enabled on the release branch. It must record the tag, +commit SHA, inputs, tool versions, artifact checksums, and Nexus staging +repository IDs. + +The workflow has the following contract: + +| Job | Required behavior | +| --- | --- | +| Source | Run `tools/releasing/create_source_release.sh`, and upload the Paimon archive, signature, and checksum without changing them | +| Java 8 | Use Temurin 8 and `tools/releasing/deploy_staging_jars.sh` | +| Java 11 | Use Temurin 11 and `tools/releasing/deploy_staging_jars_for_jdk11.sh` | +| Java 17 | Use Temurin 17 and `tools/releasing/deploy_staging_jars_for_jdk17.sh` | +| Python | Build the PyPaimon source distribution, sign and checksum it, run the supported Python test matrix, and publish the RC package to TestPyPI | +| Manifest | Collect the SHA-512 digest of every workflow artifact and all Nexus staging repository IDs in one release manifest | + +The Java jobs deploy signed artifacts to Nexus staging but must not close or +release those repositories. The Python RC job publishes +`PYPAIMON_VERSIONrcRC_NUMBER` to TestPyPI. Stable PyPI publication and all +promotion steps remain disabled until the vote passes. + +Configure the protected release environment with the Nexus credentials already +used by the snapshot workflows (`NEXUS_USER` and `NEXUS_PW`), the release GPG +private key and passphrase, and TestPyPI/PyPI credentials. Require an RM approval +for jobs that use publishing credentials. + +## One-time RM setup + +Before managing the first release: + +1. Create a GPG key associated with your `@apache.org` identity and publish it + to a public key server. +2. Append the public key to the Paimon + [KEYS](https://downloads.apache.org/paimon/KEYS) file. Never remove keys + required to verify an older release. +3. Configure Git to sign tags with the same key. +4. Confirm access to the ASF distribution SVN repository, Apache Nexus, GitHub + release environments, TestPyPI, and PyPI. +5. Verify that all required release secrets are configured without printing + their values in an Actions log. + +```shell +gpg --list-secret-keys --keyid-format LONG +git config user.signingkey +svn --version +``` + +## Prepare the release + +### Agree on the release + +Discuss the release on `dev@paimon.apache.org`, select an RM, resolve release +blockers, review incompatible changes and upgrade notes, and prepare release +notes. Ensure CI is green on the commit from which the candidate will be cut. + +### Set the release variables + +The following example deliberately uses different Java and Python versions: + +```shell +PAIMON_VERSION="2.0.0" +PYPAIMON_VERSION="1.5.0" +RC_NUMBER="1" + +RC_REF="release-${PAIMON_VERSION}-rc${RC_NUMBER}" +RELEASE_TAG="release-${PAIMON_VERSION}" +``` + +Use these exact values in the branch, tag, workflow inputs, SVN directories, +vote email, and release manifest. + +### Work from a clean clone + +```shell +git clone https://github.com/apache/paimon.git paimon-release +cd paimon-release +git checkout master +git pull --ff-only origin master +git status --short +``` + +The last command must produce no output. + +### Create the RC branch and set versions + +Create the RC branch with the existing helper: + +```shell +RELEASE_VERSION="${PAIMON_VERSION}" \ +RELEASE_CANDIDATE="${RC_NUMBER}" \ + ./tools/releasing/create_release_branch.sh +``` + +This creates `release-PAIMON_VERSION-rcRC_NUMBER`. + +Change all Maven modules from `PAIMON_VERSION-SNAPSHOT` to +`PAIMON_VERSION`. The helper commits the Maven version change: + +```shell +NEW_VERSION="${PAIMON_VERSION}" \ + ./tools/releasing/update_branch_version.sh +``` + +Set `VERSION` in `paimon-python/setup.py` to the final +`PYPAIMON_VERSION`, without `.dev` or an RC suffix. The release workflow +derives the TestPyPI version by appending `rcRC_NUMBER`; the source candidate +keeps the final version. + +Review and commit the Python version, dependency declarations, release notes, +`LICENSE`, `NOTICE`, and any generated legal files. Then confirm that the +candidate contains no snapshot or development version: + +```shell +mvn -q -DforceStdout help:evaluate -Dexpression=project.version +python3 paimon-python/setup.py --version +rg --glob 'pom.xml' "${PAIMON_VERSION}-SNAPSHOT" +rg '^VERSION = ".*\.dev' paimon-python/setup.py +git status --short +``` + +The first two commands must print the requested release versions, the `rg` +commands must find no release-version marker, and the worktree must be clean. + +### Sign and push the RC tag + +The historical Paimon convention uses the same name for the RC branch and tag. +Use explicit refs when pushing so Git cannot select the wrong one: + +```shell +git tag -s "${RC_REF}" \ + -m "Apache Paimon ${PAIMON_VERSION} and PyPaimon ${PYPAIMON_VERSION} RC${RC_NUMBER}" +git tag -v "${RC_REF}" + +git push origin \ + "refs/heads/${RC_REF}:refs/heads/${RC_REF}" +git push origin \ + "refs/tags/${RC_REF}:refs/tags/${RC_REF}" +``` + +Pushing the signed tag starts the Release workflow. Wait for every source, +Java, Python, and manifest job to succeed. Record: + +- the workflow run URL and `head_sha`; +- the Paimon and PyPaimon source artifact SHA-512 digests; +- all JDK 8, 11, and 17 Nexus staging repository IDs; +- the TestPyPI project/version URL. + +Do not start the vote when a required lane is missing or has been rerun from a +different commit. + +## Stage the source candidates + +Download the source artifacts from the successful workflow run and verify their +manifest before uploading them to ASF dist dev. + +```shell +svn checkout --depth=immediates \ + https://dist.apache.org/repos/dist/dev/paimon/ paimon-dist-dev + +mkdir "paimon-dist-dev/paimon-${PAIMON_VERSION}-rc${RC_NUMBER}" +cp "apache-paimon-${PAIMON_VERSION}-src.tgz"* \ + "paimon-dist-dev/paimon-${PAIMON_VERSION}-rc${RC_NUMBER}/" + +mkdir "paimon-dist-dev/pypaimon-${PYPAIMON_VERSION}-rc${RC_NUMBER}" +cp "pypaimon-${PYPAIMON_VERSION}.tar.gz"* \ + "paimon-dist-dev/pypaimon-${PYPAIMON_VERSION}-rc${RC_NUMBER}/" + +svn add \ + "paimon-dist-dev/paimon-${PAIMON_VERSION}-rc${RC_NUMBER}" \ + "paimon-dist-dev/pypaimon-${PYPAIMON_VERSION}-rc${RC_NUMBER}" +svn commit -m \ + "Stage Paimon ${PAIMON_VERSION} and PyPaimon ${PYPAIMON_VERSION} RC${RC_NUMBER}" \ + paimon-dist-dev +``` + +Never overwrite an existing RC directory. If any byte changes, create a new RC. + +## Call the vote + +Send a plain-text vote to `dev@paimon.apache.org`. Keep it open for at least +72 hours. The vote must receive at least three binding `+1` votes and more +binding `+1` than binding `-1` votes. + +```text +Subject: [VOTE] Release Apache Paimon ${PAIMON_VERSION} and PyPaimon ${PYPAIMON_VERSION} (RC${RC_NUMBER}) + +Hi everyone, + +Please review and vote on Apache Paimon ${PAIMON_VERSION} and +PyPaimon ${PYPAIMON_VERSION}, release candidate ${RC_NUMBER}. + +[ ] +1 Approve +[ ] 0 No opinion +[ ] -1 Do not approve (please explain) + +Paimon source candidate: +https://dist.apache.org/repos/dist/dev/paimon/paimon-${PAIMON_VERSION}-rc${RC_NUMBER}/ + +PyPaimon source candidate: +https://dist.apache.org/repos/dist/dev/paimon/pypaimon-${PYPAIMON_VERSION}-rc${RC_NUMBER}/ + +Signed Git tag and commit: +https://github.com/apache/paimon/releases/tag/release-${PAIMON_VERSION}-rc${RC_NUMBER} + + +GitHub Actions release run: + + +KEYS: +https://downloads.apache.org/paimon/KEYS + +Java staging repositories: + + + + +PyPaimon RC: +https://test.pypi.org/project/pypaimon/${PYPAIMON_VERSION}rc${RC_NUMBER}/ + +Verification guide: +https://github.com/apache/paimon/blob/release-${PAIMON_VERSION}-rc${RC_NUMBER}/docs/docs/project/verifying-a-release-candidate.md + +The vote will remain open for at least 72 hours. +``` + +After the deadline, tally binding and non-binding votes separately and send +`[RESULT][VOTE]` to the same thread. + +## Replace a failed candidate + +If the vote finds a problem: + +1. Fix it through the normal review process. +2. Drop every Nexus staging repository belonging to the failed RC. +3. Remove the superseded dist-dev directories, or retain them temporarily when + useful to the vote discussion. Never replace their contents. +4. Increment `RC_NUMBER`. +5. Create a new RC branch and signed tag, run every workflow lane again, stage + new source candidates, and start a new 72-hour vote. + +## Finalize an approved release + +### Create the final signed tag + +The final tag must point to exactly the approved RC commit: + +```shell +git tag -s "${RELEASE_TAG}" "${RC_REF}^{commit}" \ + -m "Release Apache Paimon ${PAIMON_VERSION}" + +test "$(git rev-parse "${RC_REF}^{commit}")" = \ + "$(git rev-parse "${RELEASE_TAG}^{commit}")" +git tag -v "${RELEASE_TAG}" +git push origin "refs/tags/${RELEASE_TAG}:refs/tags/${RELEASE_TAG}" +``` + +### Promote the source releases + +Move, rather than copy or rebuild, both approved candidate directories: + +```shell +svn mv -m "Release Apache Paimon ${PAIMON_VERSION}" \ + "https://dist.apache.org/repos/dist/dev/paimon/paimon-${PAIMON_VERSION}-rc${RC_NUMBER}" \ + "https://dist.apache.org/repos/dist/release/paimon/paimon-${PAIMON_VERSION}" + +svn mv -m "Release PyPaimon ${PYPAIMON_VERSION}" \ + "https://dist.apache.org/repos/dist/dev/paimon/pypaimon-${PYPAIMON_VERSION}-rc${RC_NUMBER}" \ + "https://dist.apache.org/repos/dist/release/paimon/pypaimon-${PYPAIMON_VERSION}" +``` + +### Promote convenience artifacts + +1. In Nexus, close every recorded JDK 8, 11, and 17 staging repository. Inspect + all rule failures and artifact trees before continuing. +2. Release those exact repositories to Maven Central. Do not run Maven deploy + again. +3. Approve the protected PyPI promotion job for the final tag. It must build + `pypaimon==PYPAIMON_VERSION` from the approved tag commit and must not change + project source. +4. Verify Maven Central and PyPI before announcing the release. + +### Publish and announce + +Create a GitHub release from `release-PAIMON_VERSION`, review the generated +notes, and link both source releases. Update the Paimon download page and +versioned documentation. After ASF mirrors, Maven Central, and PyPI are all +available, announce the release to `dev@paimon.apache.org` and +`announce@apache.org` from an `@apache.org` address. + +Remove superseded releases from the live ASF distribution area when required; +they remain available from the +[Apache archive](https://archive.apache.org/dist/paimon/). + +See [Verifying a Release Candidate](./verifying-a-release-candidate.md) for the +voter checklist. diff --git a/docs/docs/project/verifying-a-release-candidate.md b/docs/docs/project/verifying-a-release-candidate.md new file mode 100644 index 000000000000..80fce3ccdcd8 --- /dev/null +++ b/docs/docs/project/verifying-a-release-candidate.md @@ -0,0 +1,342 @@ +--- +title: "Verifying a Release Candidate" +sidebar_position: 3 +--- + + + +# Verifying a Release Candidate + +Verify a release candidate independently before voting. The signed Paimon and +PyPaimon source archives are the release. Maven, TestPyPI, and GitHub Actions +checks supplement source verification but do not replace it. + +Report only the checks, platforms, and tool versions that you actually used. + +## Collect the candidate inputs + +Take all values and URLs from the vote email: + +```shell +PAIMON_VERSION="2.0.0" +PYPAIMON_VERSION="1.5.0" +RC_NUMBER="1" +RC_TAG="release-${PAIMON_VERSION}-rc${RC_NUMBER}" + +PAIMON_RC_DIR="paimon-${PAIMON_VERSION}-rc${RC_NUMBER}" +PAIMON_ARCHIVE="apache-paimon-${PAIMON_VERSION}-src.tgz" +PYPAIMON_RC_DIR="pypaimon-${PYPAIMON_VERSION}-rc${RC_NUMBER}" +PYPAIMON_ARCHIVE="pypaimon-${PYPAIMON_VERSION}.tar.gz" +``` + +Download the candidates from ASF dist dev, not from a third-party mirror: + +```shell +curl -O "https://dist.apache.org/repos/dist/dev/paimon/${PAIMON_RC_DIR}/${PAIMON_ARCHIVE}" +curl -O "https://dist.apache.org/repos/dist/dev/paimon/${PAIMON_RC_DIR}/${PAIMON_ARCHIVE}.asc" +curl -O "https://dist.apache.org/repos/dist/dev/paimon/${PAIMON_RC_DIR}/${PAIMON_ARCHIVE}.sha512" + +curl -O "https://dist.apache.org/repos/dist/dev/paimon/${PYPAIMON_RC_DIR}/${PYPAIMON_ARCHIVE}" +curl -O "https://dist.apache.org/repos/dist/dev/paimon/${PYPAIMON_RC_DIR}/${PYPAIMON_ARCHIVE}.asc" +curl -O "https://dist.apache.org/repos/dist/dev/paimon/${PYPAIMON_RC_DIR}/${PYPAIMON_ARCHIVE}.sha512" + +curl -O https://downloads.apache.org/paimon/KEYS +``` + +Keep the workflow run URL, announced commit SHA, three Nexus staging URLs, and +TestPyPI URL beside these files. All of them must identify this RC. + +## Verify signatures and checksums + +```shell +gpg --import KEYS +gpg --verify "${PAIMON_ARCHIVE}.asc" "${PAIMON_ARCHIVE}" +gpg --verify "${PYPAIMON_ARCHIVE}.asc" "${PYPAIMON_ARCHIVE}" +``` + +Confirm that both signatures are good and that the full signing-key fingerprint +belongs to the RM and appears in the downloaded `KEYS` file. + +On Linux: + +```shell +sha512sum -c "${PAIMON_ARCHIVE}.sha512" +sha512sum -c "${PYPAIMON_ARCHIVE}.sha512" +``` + +On macOS: + +```shell +shasum -a 512 -c "${PAIMON_ARCHIVE}.sha512" +shasum -a 512 -c "${PYPAIMON_ARCHIVE}.sha512" +``` + +Also compare the two archive digests with the release manifest attached to the +GitHub Actions run. + +## Verify Git provenance + +```shell +git clone https://github.com/apache/paimon.git paimon-candidate-git +git -C paimon-candidate-git fetch --tags +git -C paimon-candidate-git tag -v "${RC_TAG}" +git -C paimon-candidate-git rev-parse "${RC_TAG}^{commit}" +``` + +The resolved commit must equal both the SHA in the vote email and the +`head_sha` of the successful Release workflow run. Inspect the commits since the +previous release and confirm that the RC tag is attached to the intended branch. + +## Inspect the source archives + +List the archives before extracting them: + +```shell +tar tzf "${PAIMON_ARCHIVE}" | sed -n '1,100p' +tar tzf "${PYPAIMON_ARCHIVE}" | sed -n '1,100p' +``` + +Check at least the following: + +- The Paimon archive has exactly one top-level directory named + `paimon-PAIMON_VERSION`. +- The PyPaimon archive has exactly one top-level directory named + `pypaimon-PYPAIMON_VERSION`. +- `LICENSE`, `NOTICE`, README files, build files, dependency declarations, and + required source files are present and correct. +- No Git metadata, IDE state, credentials, Maven `target` directories, Python + `dist` or `__pycache__` directories, compiled classes, generated JARs, or + unrelated binary files are included. +- All bundled third-party material is compatible with the Apache License 2.0 + and is recorded in `LICENSE` or `NOTICE` where required. +- Maven POMs use `PAIMON_VERSION` without `-SNAPSHOT`. +- `paimon-python/setup.py` and PyPaimon package metadata use + `PYPAIMON_VERSION` without `.dev`. + +Extract the candidates: + +```shell +tar xzf "${PAIMON_ARCHIVE}" +tar xzf "${PYPAIMON_ARCHIVE}" +``` + +Run the repository licensing checks from the extracted sources: + +```shell +cd "paimon-${PAIMON_VERSION}" +mvn -ntp -DskipTests apache-rat:check +cd .. + +SOURCE_PACKAGE="${PYPAIMON_ARCHIVE}" \ + "./paimon-${PAIMON_VERSION}/paimon-python/dev/check-licensing.sh" +``` + +Review the generated reports rather than relying only on the exit codes. + +## Build Java from the source archive + +Use three clean extracted copies or three clean containers so that class files +and Maven state from one JDK cannot leak into another lane. Record `java +-version` and `mvn -version` for each build. + +### JDK 8 lane + +This lane covers the default reactor, Flink 1.x, and Spark 3.x: + +```shell +mvn -ntp clean verify -Pspark3,flink1 +``` + +If a full verification is not practical on your machine, at minimum build the +same scope used for staging and state that tests were skipped: + +```shell +mvn -ntp clean install -DskipTests \ + -Pdocs-and-source,spark3,flink1 +``` + +### JDK 11 lane + +This lane covers Flink 2.x and Iceberg: + +```shell +mvn -ntp clean verify -Pflink2 \ + -pl org.apache.paimon:paimon-flink-2.0,org.apache.paimon:paimon-flink-2.1,org.apache.paimon:paimon-flink-2.2,org.apache.paimon:paimon-iceberg \ + -am +``` + +The packaging-equivalent build is: + +```shell +mvn -ntp clean install -DskipTests -Pdocs-and-source,flink2 \ + -pl org.apache.paimon:paimon-flink-2.0,org.apache.paimon:paimon-flink-2.1,org.apache.paimon:paimon-flink-2.2,org.apache.paimon:paimon-iceberg \ + -am +``` + +### JDK 17 lane + +This lane covers Spark 4.x: + +```shell +mvn -ntp clean verify -Pspark4 \ + -pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \ + -am +``` + +The packaging-equivalent build is: + +```shell +mvn -ntp clean install -DskipTests -Pdocs-and-source,spark4 \ + -pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \ + -am +``` + +Investigate warnings and skipped modules. A successful compiler exit does not, +by itself, establish that the candidate is suitable for release. + +## Verify Java staging repositories + +Use all three exact `orgapachepaimon-XXXX` staging URLs from the vote email. +Do not resolve artifacts from Maven Central or a local cache when testing the +candidate. + +For every staging repository: + +- Confirm that the version is `PAIMON_VERSION`, never a snapshot. +- Confirm that each published module has its POM, main artifact, source JAR, + Javadoc JAR, detached signatures, and checksums where applicable. +- Inspect shaded and bundled JARs for the required `META-INF/LICENSE` and + `META-INF/NOTICE` content and for unexpected duplicate or bundled + dependencies. +- Resolve representative artifacts in a clean Maven repository and run a small + consumer smoke test. + +Confirm the lane boundaries: + +| Lane | Must contain | +| --- | --- | +| JDK 8 | Default Java 8 artifacts, Flink 1.x, and Spark 3.x | +| JDK 11 | Flink 2.x, `paimon-flink2-common`, and `paimon-iceberg` | +| JDK 17 | Spark 4.x and its Scala 2.13 common artifacts | + +Check representative class-file targets. Java class major versions are 52 for +Java 8, 55 for Java 11, and 61 for Java 17: + +```shell +JAR_FILE="/path/to/a/representative.jar" +CLASS_NAME="$(jar tf "${JAR_FILE}" | grep '\.class$' | grep -v 'module-info' | head -1 | sed 's#/#.#g;s#\.class$##')" +javap -verbose -classpath "${JAR_FILE}" "${CLASS_NAME}" | grep 'major version' +``` + +Run this against at least one representative JAR from each lane. Also check +that Java 8 artifacts can be loaded by a JDK 8 runtime; class-file inspection +alone does not detect every use of a newer JDK API. + +## Build and test PyPaimon from source + +First build the official PyPaimon source candidate in an isolated environment: + +```shell +python3 -m venv pypaimon-rc-venv +. pypaimon-rc-venv/bin/activate +python -m pip install --upgrade pip setuptools wheel + +cd "pypaimon-${PYPAIMON_VERSION}" +python setup.py sdist bdist_wheel +python -m pip install "dist/pypaimon-${PYPAIMON_VERSION}"*.whl +python -c "import pypaimon; print('PyPaimon import OK')" +cd .. +``` + +Inspect the generated sdist and wheel. They must contain the expected Python +sources, `LICENSE`, `NOTICE`, README, and package metadata, with no tests, +caches, credentials, or unrelated build output. + +The complete Python test suite lives in the Paimon source archive. Run it from +`paimon-PAIMON_VERSION/paimon-python` on as many supported Python versions as +your environment allows. The project CI selects Python 3.6, 3.7, 3.10, and +3.11 for its main compatibility lanes: + +```shell +python -m pip install -r dev/requirements.txt +python -m pip install -r dev/requirements-dev.txt +python -m pytest pypaimon/tests -v +``` + +Some optional integration tests need additional services or packages. Record +which tests ran, skipped, or failed. + +## Verify the TestPyPI candidate + +Install the exact RC version in a new environment: + +```shell +python3 -m venv pypaimon-testpypi-venv +. pypaimon-testpypi-venv/bin/activate +python -m pip install \ + --index-url https://test.pypi.org/simple/ \ + --extra-index-url https://pypi.org/simple/ \ + "pypaimon==${PYPAIMON_VERSION}rc${RC_NUMBER}" +python -c "import pypaimon; print('TestPyPI package OK')" +``` + +Check the available files, package metadata, Python requirement, dependencies, +license files, and a representative read/write operation. The RC suffix is the +only intended version change from the final PyPaimon source candidate; any +source-code difference is a release blocker. + +## Verify GitHub Actions evidence + +Open the workflow run linked in the vote email and confirm: + +- it was triggered from the signed `RC_TAG`; +- `head_sha` equals the announced commit; +- source, JDK 8, JDK 11, JDK 17, Python, and manifest jobs all succeeded; +- the logs show the expected JDK and Python versions; +- the source archive hashes equal the files in ASF dist dev; +- the Nexus IDs and TestPyPI version equal those in the vote email; +- no later rerun silently replaced a failed lane with artifacts from another + commit. + +GitHub Actions evidence is useful for reviewing the complete platform matrix, +but voters should still perform independent source, signature, build, and smoke +checks. + +## Report your vote + +Reply to the vote thread with `+1`, `0`, or `-1`, state whether your vote is +binding, and list the checks you completed. + +```text ++1 (binding/non-binding) + +Verified: +- signed RC tag and announced commit SHA +- Paimon and PyPaimon GPG signatures and SHA-512 checksums +- LICENSE, NOTICE, source-only archive contents, and release versions +- Paimon source build on , JDK <8/11/17>, Maven +- Java staging repositories and representative class-file targets +- PyPaimon source build/tests on Python +- TestPyPI installation and smoke test +- GitHub Actions run provenance and artifact manifest +``` + +For a `-1`, describe the failure precisely enough for the RM to reproduce it. +See [Creating a Release](./creating-a-release.md) for the RM workflow. diff --git a/docs/sidebars.js b/docs/sidebars.js index ed0db484e7cb..56a9c3e21107 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -318,6 +318,8 @@ const sidebars = { }, "items": [ "project/download", + "project/creating-a-release", + "project/verifying-a-release-candidate", "project/contributing", "project/committer", "project/security" From 7f98044c46e16b5c721088cdf58170722709998c Mon Sep 17 00:00:00 2001 From: JingsongLi Date: Thu, 30 Jul 2026 22:39:07 +0800 Subject: [PATCH 2/8] [release] Add automated release workflows --- .github/workflows/release-java.yml | 192 ++++++++++++++++++ .github/workflows/release-python-publish.yml | 88 ++++++++ .github/workflows/release-python.yml | 129 ++++++++++++ .github/workflows/release.yml | 58 ++++++ docs/docs/project/creating-a-release.md | 73 ++++--- .../project/verifying-a-release-candidate.md | 12 +- 6 files changed, 522 insertions(+), 30 deletions(-) create mode 100644 .github/workflows/release-java.yml create mode 100644 .github/workflows/release-python-publish.yml create mode 100644 .github/workflows/release-python.yml create mode 100644 .github/workflows/release.yml diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml new file mode 100644 index 000000000000..a05bb5bd7527 --- /dev/null +++ b/.github/workflows/release-java.yml @@ -0,0 +1,192 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +name: Release Java + +on: + workflow_call: + workflow_dispatch: + +concurrency: + group: release-java-${{ github.ref }} + cancel-in-progress: false + +permissions: + contents: read + +jobs: + deploy-staging: + name: Stage ${{ matrix.lane }} + if: >- + github.repository == 'apache/paimon' && + startsWith(github.ref, 'refs/tags/') && + contains(github.ref_name, '-rc') + environment: release + runs-on: ubuntu-latest + strategy: + fail-fast: false + max-parallel: 1 + matrix: + include: + - lane: jdk8 + java_version: 8 + script: tools/releasing/deploy_staging_jars.sh + - lane: jdk11 + java_version: 11 + script: tools/releasing/deploy_staging_jars_for_jdk11.sh + - lane: jdk17 + java_version: 17 + script: tools/releasing/deploy_staging_jars_for_jdk17.sh + steps: + - uses: actions/checkout@v6 + + - name: Set up JDK ${{ matrix.java_version }} + uses: actions/setup-java@v5 + with: + java-version: ${{ matrix.java_version }} + distribution: temurin + cache: maven + server-id: apache.releases.https + server-username: MAVEN_USERNAME + server-password: MAVEN_PASSWORD + gpg-private-key: ${{ secrets.GPG_SECRET_KEY }} + gpg-passphrase: MAVEN_GPG_PASSPHRASE + + - name: Set up Maven + uses: stCarolas/setup-maven@v5 + with: + maven-version: 3.8.8 + + - name: Validate release tag and version + id: release + shell: bash + run: | + set -euo pipefail + + release_version="$( + mvn -q -DforceStdout help:evaluate \ + -Dexpression=project.version + )" + rc_number="${GITHUB_REF_NAME##*-rc}" + expected_tag="release-${release_version}-rc${rc_number}" + + if [[ "${GITHUB_REF_NAME}" != "${expected_tag}" ]]; then + echo "Tag ${GITHUB_REF_NAME} does not match Maven version ${release_version}" >&2 + exit 1 + fi + if [[ "${release_version}" == *-SNAPSHOT ]]; then + echo "Release version must not contain -SNAPSHOT" >&2 + exit 1 + fi + + description="apache-paimon-${release_version}-rc${rc_number}-${{ matrix.lane }}-run-${GITHUB_RUN_ID}" + echo "version=${release_version}" >> "${GITHUB_OUTPUT}" + echo "rc_number=${rc_number}" >> "${GITHUB_OUTPUT}" + echo "description=${description}" >> "${GITHUB_OUTPUT}" + + java -version + mvn -version + + - name: Open Apache Nexus staging repository + id: staging + shell: bash + env: + MAVEN_PASSWORD: ${{ secrets.NEXUS_PW }} + MAVEN_USERNAME: ${{ secrets.NEXUS_USER }} + STAGING_DESCRIPTION: ${{ steps.release.outputs.description }} + run: | + set -euo pipefail + + mkdir -p release-staging + profiles="${RUNNER_TEMP}/nexus-profiles.json" + curl --fail --silent --show-error \ + --retry 3 \ + --user "${MAVEN_USERNAME}:${MAVEN_PASSWORD}" \ + --header "Accept: application/json" \ + https://repository.apache.org/service/local/staging/profiles \ + > "${profiles}" + + mapfile -t profile_ids < <( + jq -r \ + '.data[] + | select(.name | ascii_downcase | contains("paimon")) + | .id' \ + "${profiles}" + ) + if [[ "${#profile_ids[@]}" -ne 1 ]]; then + echo "Expected one Apache Paimon staging profile, found ${#profile_ids[@]}" >&2 + exit 1 + fi + + request="${RUNNER_TEMP}/nexus-open-request.json" + response="${RUNNER_TEMP}/nexus-open-response.json" + jq -n \ + --arg description "${STAGING_DESCRIPTION}" \ + '{data: {description: $description}}' \ + > "${request}" + curl --fail --silent --show-error \ + --user "${MAVEN_USERNAME}:${MAVEN_PASSWORD}" \ + --header "Accept: application/json" \ + --header "Content-Type: application/json" \ + --request POST \ + --data "@${request}" \ + "https://repository.apache.org/service/local/staging/profiles/${profile_ids[0]}/start" \ + > "${response}" + + repository_id="$(jq -r '.data.stagedRepositoryId // empty' "${response}")" + if [[ ! "${repository_id}" =~ ^orgapachepaimon- ]]; then + echo "Nexus returned an unexpected staging repository ID" >&2 + exit 1 + fi + echo "repository_id=${repository_id}" >> "${GITHUB_OUTPUT}" + + manifest="release-staging/${{ matrix.lane }}-manifest.txt" + { + echo "lane=${{ matrix.lane }}" + echo "jdk=${{ matrix.java_version }}" + echo "version=${{ steps.release.outputs.version }}" + echo "rc=${{ steps.release.outputs.rc_number }}" + echo "commit=${GITHUB_SHA}" + echo "repository=${repository_id}" + echo "description=${STAGING_DESCRIPTION}" + } > "${manifest}" + cat "${manifest}" + + - name: Deploy signed artifacts to Apache Nexus staging + shell: bash + env: + CUSTOM_OPTIONS: >- + -B + -DaltDeploymentRepository=apache.releases.https::default::https://repository.apache.org/service/local/staging/deployByRepositoryId/${{ steps.staging.outputs.repository_id }} + MAVEN_GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }} + MAVEN_PASSWORD: ${{ secrets.NEXUS_PW }} + MAVEN_USERNAME: ${{ secrets.NEXUS_USER }} + MAVEN_OPTS: -Xmx4096m + run: | + set -euo pipefail + mkdir -p release-staging + bash "${{ matrix.script }}" 2>&1 \ + | tee "release-staging/${{ matrix.lane }}.log" + + - name: Upload staging evidence + if: always() + uses: actions/upload-artifact@v5 + with: + name: java-staging-${{ matrix.lane }} + path: release-staging/ + if-no-files-found: warn diff --git a/.github/workflows/release-python-publish.yml b/.github/workflows/release-python-publish.yml new file mode 100644 index 000000000000..ed921c8a6b46 --- /dev/null +++ b/.github/workflows/release-python-publish.yml @@ -0,0 +1,88 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +name: Release PyPaimon Publish + +on: + workflow_call: + +concurrency: + group: release-python-publish-${{ github.ref }} + cancel-in-progress: false + +permissions: + actions: read + contents: read + +jobs: + publish: + name: Publish PyPaimon + if: >- + github.repository == 'apache/paimon' && + startsWith(github.ref, 'refs/tags/') + environment: release + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + + - uses: actions/download-artifact@v5 + with: + name: pypaimon-distributions + path: dist + + - name: Verify package versions + shell: bash + run: | + set -euo pipefail + + base_version="$( + sed -n 's/^VERSION = "\(.*\)"/\1/p' \ + paimon-python/setup.py + )" + expected_version="${base_version}" + if [[ "${GITHUB_REF_NAME}" == *-rc* ]]; then + expected_version="${base_version}rc${GITHUB_REF_NAME##*-rc}" + fi + + test -f "dist/pypaimon-${expected_version}.tar.gz" + compgen -G "dist/pypaimon-${expected_version}-*.whl" > /dev/null + + shopt -s nullglob + distributions=(dist/*) + if [[ "${#distributions[@]}" -ne 2 ]]; then + echo "Expected one sdist and one wheel, found ${#distributions[@]} files" >&2 + printf '%s\n' "${distributions[@]}" >&2 + exit 1 + fi + + - name: Publish RC to TestPyPI + if: contains(github.ref_name, '-rc') + uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e + with: + repository-url: https://test.pypi.org/legacy/ + skip-existing: true + packages-dir: dist + password: ${{ secrets.TEST_PYPI_API_TOKEN }} + + - name: Publish final release to PyPI + if: ${{ !contains(github.ref_name, '-rc') }} + uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e + with: + skip-existing: true + packages-dir: dist + password: ${{ secrets.PYPI_API_TOKEN }} diff --git a/.github/workflows/release-python.yml b/.github/workflows/release-python.yml new file mode 100644 index 000000000000..5f264aa1377a --- /dev/null +++ b/.github/workflows/release-python.yml @@ -0,0 +1,129 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +# Build the PyPaimon source distribution and universal wheel. Publishing is +# handled by release-python-publish.yml after all required release jobs pass. + +name: Release PyPaimon Package + +on: + workflow_call: + workflow_dispatch: + +concurrency: + group: release-python-${{ github.ref }} + cancel-in-progress: false + +permissions: + contents: read + +jobs: + package: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + cache: pip + cache-dependency-path: paimon-python/dev/requirements.txt + + - name: Set package version + id: package_version + shell: bash + run: | + set -euo pipefail + + base_version="$( + sed -n 's/^VERSION = "\(.*\)"/\1/p' \ + paimon-python/setup.py + )" + if [[ -z "${base_version}" || "${base_version}" == *dev* || "${base_version}" == *rc* ]]; then + echo "PyPaimon release base version is invalid: ${base_version:-missing}" >&2 + exit 1 + fi + + package_version="${base_version}" + if [[ "${GITHUB_REF}" == refs/tags/*-rc* ]]; then + rc_number="${GITHUB_REF_NAME##*-rc}" + package_version="${base_version}rc${rc_number}" + sed -i \ + "s/^VERSION = \".*\"/VERSION = \"${package_version}\"/" \ + paimon-python/setup.py + fi + + echo "version=${package_version}" >> "${GITHUB_OUTPUT}" + echo "Building PyPaimon ${package_version}" + + - name: Build distributions + working-directory: paimon-python + run: | + python -m pip install --upgrade pip setuptools wheel twine + python setup.py sdist bdist_wheel + python -m twine check dist/* + + - name: Verify distribution contents + working-directory: paimon-python + env: + EXPECTED_VERSION: ${{ steps.package_version.outputs.version }} + run: | + python - <<'PY' + import glob + import os + import tarfile + import zipfile + + expected = os.environ["EXPECTED_VERSION"] + sdist = "dist/pypaimon-{}.tar.gz".format(expected) + wheels = glob.glob("dist/pypaimon-{}-*.whl".format(expected)) + if not os.path.isfile(sdist): + raise SystemExit("Missing expected sdist: {}".format(sdist)) + if len(wheels) != 1: + raise SystemExit("Expected one wheel, found: {}".format(wheels)) + + with tarfile.open(sdist, "r:gz") as archive: + names = archive.getnames() + prefix = "pypaimon-{}/".format(expected) + for required in ("LICENSE", "NOTICE", "README.md", "setup.py"): + if prefix + required not in names: + raise SystemExit("Missing {} from sdist".format(required)) + + with zipfile.ZipFile(wheels[0]) as archive: + names = archive.namelist() + license_files = [ + name for name in names + if ".dist-info/" in name and name.endswith("/LICENSE") + ] + notice_files = [ + name for name in names + if ".dist-info/" in name and name.endswith("/NOTICE") + ] + if not license_files or not notice_files: + raise SystemExit("Wheel is missing LICENSE or NOTICE") + PY + + - name: Upload PyPaimon distributions + uses: actions/upload-artifact@v5 + with: + name: pypaimon-distributions + path: | + paimon-python/dist/*.tar.gz + paimon-python/dist/*.whl + if-no-files-found: error diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 000000000000..b22b70c4b751 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,58 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +# Orchestrate Java staging and PyPaimon packaging. The signed ASF source +# archives remain a Release Manager responsibility and are not rebuilt here. + +name: Release + +on: + push: + tags: + - "release-[0-9]+.[0-9]+.[0-9]+" + - "release-[0-9]+.[0-9]+.[0-9]+-rc[0-9]+" + workflow_dispatch: + +concurrency: + group: release-${{ github.ref }} + cancel-in-progress: false + +permissions: + actions: read + contents: read + +jobs: + java: + name: Java staging + uses: ./.github/workflows/release-java.yml + secrets: inherit + + python-package: + name: PyPaimon package + uses: ./.github/workflows/release-python.yml + + python-publish: + name: PyPaimon publish + needs: [java, python-package] + if: >- + always() && + startsWith(github.ref, 'refs/tags/') && + needs.python-package.result == 'success' && + (needs.java.result == 'success' || !contains(github.ref_name, '-rc')) + uses: ./.github/workflows/release-python-publish.yml + secrets: inherit diff --git a/docs/docs/project/creating-a-release.md b/docs/docs/project/creating-a-release.md index 28e6f7689b1d..859ea776ac1a 100644 --- a/docs/docs/project/creating-a-release.md +++ b/docs/docs/project/creating-a-release.md @@ -68,23 +68,21 @@ compiler target is not a substitute for running the JDK 8 and JDK 11 lanes. ## GitHub Actions release workflow -The release process uses a **Release** workflow in -[GitHub Actions](https://github.com/apache/paimon/actions) to package every -candidate from its signed RC tag. Before cutting an RC, confirm that this -workflow is present and enabled on the release branch. It must record the tag, -commit SHA, inputs, tool versions, artifact checksums, and Nexus staging -repository IDs. +The release process uses the +[Release workflow](https://github.com/apache/paimon/actions/workflows/release.yml) +to stage Java and package PyPaimon from every signed RC tag. The RM creates and +signs the two ASF source archives locally from the same tag; they are not +rebuilt by GitHub Actions. The workflow has the following contract: | Job | Required behavior | | --- | --- | -| Source | Run `tools/releasing/create_source_release.sh`, and upload the Paimon archive, signature, and checksum without changing them | -| Java 8 | Use Temurin 8 and `tools/releasing/deploy_staging_jars.sh` | -| Java 11 | Use Temurin 11 and `tools/releasing/deploy_staging_jars_for_jdk11.sh` | -| Java 17 | Use Temurin 17 and `tools/releasing/deploy_staging_jars_for_jdk17.sh` | -| Python | Build the PyPaimon source distribution, sign and checksum it, run the supported Python test matrix, and publish the RC package to TestPyPI | -| Manifest | Collect the SHA-512 digest of every workflow artifact and all Nexus staging repository IDs in one release manifest | +| Java 8 | Use Temurin 8 and `tools/releasing/deploy_staging_jars.sh`, then upload the staging manifest and log | +| Java 11 | Use Temurin 11 and `tools/releasing/deploy_staging_jars_for_jdk11.sh`, then upload the staging manifest and log | +| Java 17 | Use Temurin 17 and `tools/releasing/deploy_staging_jars_for_jdk17.sh`, then upload the staging manifest and log | +| Python package | Build and validate the PyPaimon source distribution and universal wheel, then upload them as workflow artifacts | +| Python publish | Publish an RC to TestPyPI only after all Java and Python packaging jobs pass; publish a final tag to PyPI | The Java jobs deploy signed artifacts to Nexus staging but must not close or release those repositories. The Python RC job publishes @@ -93,8 +91,9 @@ promotion steps remain disabled until the vote passes. Configure the protected release environment with the Nexus credentials already used by the snapshot workflows (`NEXUS_USER` and `NEXUS_PW`), the release GPG -private key and passphrase, and TestPyPI/PyPI credentials. Require an RM approval -for jobs that use publishing credentials. +private key and passphrase (`GPG_SECRET_KEY` and `GPG_PASSPHRASE`), and the +Python repository tokens (`TEST_PYPI_API_TOKEN` and `PYPI_API_TOKEN`). Require +an RM approval for jobs that use publishing credentials. ## One-time RM setup @@ -139,7 +138,7 @@ RELEASE_TAG="release-${PAIMON_VERSION}" ``` Use these exact values in the branch, tag, workflow inputs, SVN directories, -vote email, and release manifest. +vote email, and Java staging manifests. ### Work from a clean clone @@ -209,12 +208,12 @@ git push origin \ "refs/tags/${RC_REF}:refs/tags/${RC_REF}" ``` -Pushing the signed tag starts the Release workflow. Wait for every source, -Java, Python, and manifest job to succeed. Record: +Pushing the signed tag starts the Release workflow. Wait for every Java and +Python job to succeed. Record: - the workflow run URL and `head_sha`; -- the Paimon and PyPaimon source artifact SHA-512 digests; -- all JDK 8, 11, and 17 Nexus staging repository IDs; +- all JDK 8, 11, and 17 Nexus staging repository IDs from the uploaded + manifests; - the TestPyPI project/version URL. Do not start the vote when a required lane is missing or has been rerun from a @@ -222,19 +221,47 @@ different commit. ## Stage the source candidates -Download the source artifacts from the successful workflow run and verify their -manifest before uploading them to ASF dist dev. +Create both source candidates locally from the exact signed tag in a fresh +clone. The Paimon helper creates, signs, and checksums the main source archive. +Build the PyPaimon source distribution separately and sign it with the same RM +key: + +```shell +git checkout --detach "refs/tags/${RC_REF}" + +RELEASE_VERSION="${PAIMON_VERSION}" \ + ./tools/releasing/create_source_release.sh + +cd paimon-python +python3 setup.py sdist +gpg --armor --detach-sig \ + "dist/pypaimon-${PYPAIMON_VERSION}.tar.gz" + +if command -v sha512sum >/dev/null 2>&1; then + sha512sum "dist/pypaimon-${PYPAIMON_VERSION}.tar.gz" \ + > "dist/pypaimon-${PYPAIMON_VERSION}.tar.gz.sha512" +else + shasum -a 512 "dist/pypaimon-${PYPAIMON_VERSION}.tar.gz" \ + > "dist/pypaimon-${PYPAIMON_VERSION}.tar.gz.sha512" +fi + +cp "dist/pypaimon-${PYPAIMON_VERSION}.tar.gz"* ../release/ +cd .. +``` + +Verify both signatures and checksum files locally before uploading them to ASF +dist dev. ```shell svn checkout --depth=immediates \ https://dist.apache.org/repos/dist/dev/paimon/ paimon-dist-dev mkdir "paimon-dist-dev/paimon-${PAIMON_VERSION}-rc${RC_NUMBER}" -cp "apache-paimon-${PAIMON_VERSION}-src.tgz"* \ +cp "release/apache-paimon-${PAIMON_VERSION}-src.tgz"* \ "paimon-dist-dev/paimon-${PAIMON_VERSION}-rc${RC_NUMBER}/" mkdir "paimon-dist-dev/pypaimon-${PYPAIMON_VERSION}-rc${RC_NUMBER}" -cp "pypaimon-${PYPAIMON_VERSION}.tar.gz"* \ +cp "release/pypaimon-${PYPAIMON_VERSION}.tar.gz"* \ "paimon-dist-dev/pypaimon-${PYPAIMON_VERSION}-rc${RC_NUMBER}/" svn add \ diff --git a/docs/docs/project/verifying-a-release-candidate.md b/docs/docs/project/verifying-a-release-candidate.md index 80fce3ccdcd8..5328bfa08456 100644 --- a/docs/docs/project/verifying-a-release-candidate.md +++ b/docs/docs/project/verifying-a-release-candidate.md @@ -88,9 +88,6 @@ shasum -a 512 -c "${PAIMON_ARCHIVE}.sha512" shasum -a 512 -c "${PYPAIMON_ARCHIVE}.sha512" ``` -Also compare the two archive digests with the release manifest attached to the -GitHub Actions run. - ## Verify Git provenance ```shell @@ -308,10 +305,11 @@ Open the workflow run linked in the vote email and confirm: - it was triggered from the signed `RC_TAG`; - `head_sha` equals the announced commit; -- source, JDK 8, JDK 11, JDK 17, Python, and manifest jobs all succeeded; +- JDK 8, JDK 11, JDK 17, Python packaging, and Python publishing jobs all + succeeded; - the logs show the expected JDK and Python versions; -- the source archive hashes equal the files in ASF dist dev; -- the Nexus IDs and TestPyPI version equal those in the vote email; +- the Nexus IDs in the three staging manifests and the TestPyPI version equal + those in the vote email; - no later rerun silently replaced a failed lane with artifacts from another commit. @@ -335,7 +333,7 @@ Verified: - Java staging repositories and representative class-file targets - PyPaimon source build/tests on Python - TestPyPI installation and smoke test -- GitHub Actions run provenance and artifact manifest +- GitHub Actions run provenance, Java staging manifests, and Python artifacts ``` For a `-1`, describe the failure precisely enough for the RM to reproduce it. From dc9c66c03383a292d0536d35cccd802a221ed3b8 Mon Sep 17 00:00:00 2001 From: JingsongLi Date: Thu, 30 Jul 2026 22:52:05 +0800 Subject: [PATCH 3/8] [release] Keep Java signing on RM machine --- .github/workflows/release-java.yml | 175 +++++++++--------- .github/workflows/release.yml | 7 +- docs/docs/project/creating-a-release.md | 89 ++++++--- .../project/verifying-a-release-candidate.md | 7 +- 4 files changed, 165 insertions(+), 113 deletions(-) diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml index a05bb5bd7527..30e3b6c1e9fa 100644 --- a/.github/workflows/release-java.yml +++ b/.github/workflows/release-java.yml @@ -30,28 +30,23 @@ permissions: contents: read jobs: - deploy-staging: - name: Stage ${{ matrix.lane }} + package: + name: Package ${{ matrix.lane }} if: >- github.repository == 'apache/paimon' && startsWith(github.ref, 'refs/tags/') && contains(github.ref_name, '-rc') - environment: release runs-on: ubuntu-latest strategy: fail-fast: false - max-parallel: 1 matrix: include: - lane: jdk8 java_version: 8 - script: tools/releasing/deploy_staging_jars.sh - lane: jdk11 java_version: 11 - script: tools/releasing/deploy_staging_jars_for_jdk11.sh - lane: jdk17 java_version: 17 - script: tools/releasing/deploy_staging_jars_for_jdk17.sh steps: - uses: actions/checkout@v6 @@ -61,11 +56,6 @@ jobs: java-version: ${{ matrix.java_version }} distribution: temurin cache: maven - server-id: apache.releases.https - server-username: MAVEN_USERNAME - server-password: MAVEN_PASSWORD - gpg-private-key: ${{ secrets.GPG_SECRET_KEY }} - gpg-passphrase: MAVEN_GPG_PASSPHRASE - name: Set up Maven uses: stCarolas/setup-maven@v5 @@ -94,99 +84,114 @@ jobs: exit 1 fi - description="apache-paimon-${release_version}-rc${rc_number}-${{ matrix.lane }}-run-${GITHUB_RUN_ID}" echo "version=${release_version}" >> "${GITHUB_OUTPUT}" echo "rc_number=${rc_number}" >> "${GITHUB_OUTPUT}" - echo "description=${description}" >> "${GITHUB_OUTPUT}" java -version mvn -version - - name: Open Apache Nexus staging repository - id: staging + - name: Package Java release artifacts shell: bash env: - MAVEN_PASSWORD: ${{ secrets.NEXUS_PW }} - MAVEN_USERNAME: ${{ secrets.NEXUS_USER }} - STAGING_DESCRIPTION: ${{ steps.release.outputs.description }} + LANE: ${{ matrix.lane }} + MAVEN_OPTS: -Xmx4096m run: | set -euo pipefail - mkdir -p release-staging - profiles="${RUNNER_TEMP}/nexus-profiles.json" - curl --fail --silent --show-error \ - --retry 3 \ - --user "${MAVEN_USERNAME}:${MAVEN_PASSWORD}" \ - --header "Accept: application/json" \ - https://repository.apache.org/service/local/staging/profiles \ - > "${profiles}" - - mapfile -t profile_ids < <( - jq -r \ - '.data[] - | select(.name | ascii_downcase | contains("paimon")) - | .id' \ - "${profiles}" - ) - if [[ "${#profile_ids[@]}" -ne 1 ]]; then - echo "Expected one Apache Paimon staging profile, found ${#profile_ids[@]}" >&2 - exit 1 + mkdir -p release-java + log="release-java/${LANE}.log" + artifact_dirs=() + case "${LANE}" in + jdk8) + mvn clean install -ntp -B \ + -Papache-release,docs-and-source,spark3,flink1 \ + -DskipTests -Dgpg.skip=true \ + 2>&1 | tee "${log}" + ;; + jdk11) + mvn clean install -ntp -B \ + -Pdocs-and-source,flink2 \ + -DskipTests \ + -pl org.apache.paimon:paimon-flink-2.0,org.apache.paimon:paimon-flink-2.1,org.apache.paimon:paimon-flink-2.2,org.apache.paimon:paimon-iceberg \ + -am \ + 2>&1 | tee "${log}" + mvn package -ntp -B \ + -Papache-release,docs-and-source,flink2 \ + -DskipTests -Dgpg.skip=true \ + -pl org.apache.paimon:paimon-flink-2.0,org.apache.paimon:paimon-flink-2.1,org.apache.paimon:paimon-flink-2.2,org.apache.paimon:paimon-flink2-common,org.apache.paimon:paimon-iceberg \ + 2>&1 | tee -a "${log}" + artifact_dirs=( + paimon-flink/paimon-flink-2.0 + paimon-flink/paimon-flink-2.1 + paimon-flink/paimon-flink-2.2 + paimon-flink/paimon-flink2-common + paimon-iceberg + ) + ;; + jdk17) + mvn clean install -ntp -B \ + -Pdocs-and-source,spark4 \ + -DskipTests \ + -pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \ + -am \ + 2>&1 | tee "${log}" + mvn package -ntp -B \ + -Papache-release,docs-and-source,spark4 \ + -DskipTests -Dgpg.skip=true \ + -pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13 \ + 2>&1 | tee -a "${log}" + artifact_dirs=( + paimon-spark/paimon-spark-common + paimon-spark/paimon-spark4-common + paimon-spark/paimon-spark-4.0 + paimon-spark/paimon-spark-4.1 + ) + ;; + *) + echo "Unknown Java release lane: ${LANE}" >&2 + exit 1 + ;; + esac + + files="release-java/${LANE}-files.txt" + if [[ "${#artifact_dirs[@]}" -eq 0 ]]; then + find . -type f -path '*/target/*.jar' -print \ + | LC_ALL=C sort \ + > "${files}" + else + for artifact_dir in "${artifact_dirs[@]}"; do + find "${artifact_dir}/target" \ + -maxdepth 1 -type f -name '*.jar' -print + done | LC_ALL=C sort > "${files}" fi - - request="${RUNNER_TEMP}/nexus-open-request.json" - response="${RUNNER_TEMP}/nexus-open-response.json" - jq -n \ - --arg description "${STAGING_DESCRIPTION}" \ - '{data: {description: $description}}' \ - > "${request}" - curl --fail --silent --show-error \ - --user "${MAVEN_USERNAME}:${MAVEN_PASSWORD}" \ - --header "Accept: application/json" \ - --header "Content-Type: application/json" \ - --request POST \ - --data "@${request}" \ - "https://repository.apache.org/service/local/staging/profiles/${profile_ids[0]}/start" \ - > "${response}" - - repository_id="$(jq -r '.data.stagedRepositoryId // empty' "${response}")" - if [[ ! "${repository_id}" =~ ^orgapachepaimon- ]]; then - echo "Nexus returned an unexpected staging repository ID" >&2 + if [[ ! -s "${files}" ]]; then + echo "No Java release artifacts were produced for ${LANE}" >&2 exit 1 fi - echo "repository_id=${repository_id}" >> "${GITHUB_OUTPUT}" - manifest="release-staging/${{ matrix.lane }}-manifest.txt" + while IFS= read -r artifact; do + sha512sum "${artifact}" + done < "${files}" \ + > "release-java/${LANE}-sha512.txt" + + tar -czf "release-java/${LANE}-packages.tar.gz" \ + -T "${files}" + sha512sum "release-java/${LANE}-packages.tar.gz" \ + > "release-java/${LANE}-packages.tar.gz.sha512" + { - echo "lane=${{ matrix.lane }}" + echo "lane=${LANE}" echo "jdk=${{ matrix.java_version }}" echo "version=${{ steps.release.outputs.version }}" echo "rc=${{ steps.release.outputs.rc_number }}" echo "commit=${GITHUB_SHA}" - echo "repository=${repository_id}" - echo "description=${STAGING_DESCRIPTION}" - } > "${manifest}" - cat "${manifest}" - - - name: Deploy signed artifacts to Apache Nexus staging - shell: bash - env: - CUSTOM_OPTIONS: >- - -B - -DaltDeploymentRepository=apache.releases.https::default::https://repository.apache.org/service/local/staging/deployByRepositoryId/${{ steps.staging.outputs.repository_id }} - MAVEN_GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }} - MAVEN_PASSWORD: ${{ secrets.NEXUS_PW }} - MAVEN_USERNAME: ${{ secrets.NEXUS_USER }} - MAVEN_OPTS: -Xmx4096m - run: | - set -euo pipefail - mkdir -p release-staging - bash "${{ matrix.script }}" 2>&1 \ - | tee "release-staging/${{ matrix.lane }}.log" + echo "signed=false" + echo "staged=false" + } > "release-java/${LANE}-manifest.txt" - - name: Upload staging evidence - if: always() + - name: Upload Java packages uses: actions/upload-artifact@v5 with: - name: java-staging-${{ matrix.lane }} - path: release-staging/ - if-no-files-found: warn + name: java-package-${{ matrix.lane }} + path: release-java/ + if-no-files-found: error diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b22b70c4b751..e675a60f4cd2 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -16,8 +16,8 @@ # limitations under the License. ################################################################################ -# Orchestrate Java staging and PyPaimon packaging. The signed ASF source -# archives remain a Release Manager responsibility and are not rebuilt here. +# Orchestrate Java and PyPaimon packaging. Java signing and Nexus staging, plus +# the signed ASF source archives, remain Release Manager responsibilities. name: Release @@ -38,9 +38,8 @@ permissions: jobs: java: - name: Java staging + name: Java packages uses: ./.github/workflows/release-java.yml - secrets: inherit python-package: name: PyPaimon package diff --git a/docs/docs/project/creating-a-release.md b/docs/docs/project/creating-a-release.md index 859ea776ac1a..05a6b59a2c9a 100644 --- a/docs/docs/project/creating-a-release.md +++ b/docs/docs/project/creating-a-release.md @@ -70,30 +70,28 @@ compiler target is not a substitute for running the JDK 8 and JDK 11 lanes. The release process uses the [Release workflow](https://github.com/apache/paimon/actions/workflows/release.yml) -to stage Java and package PyPaimon from every signed RC tag. The RM creates and -signs the two ASF source archives locally from the same tag; they are not -rebuilt by GitHub Actions. +to package Java and PyPaimon from every signed RC tag. The RM signs and stages +Java locally, and creates and signs the two ASF source archives locally from +the same tag. The RM's GPG private key is never stored in GitHub Actions. The workflow has the following contract: | Job | Required behavior | | --- | --- | -| Java 8 | Use Temurin 8 and `tools/releasing/deploy_staging_jars.sh`, then upload the staging manifest and log | -| Java 11 | Use Temurin 11 and `tools/releasing/deploy_staging_jars_for_jdk11.sh`, then upload the staging manifest and log | -| Java 17 | Use Temurin 17 and `tools/releasing/deploy_staging_jars_for_jdk17.sh`, then upload the staging manifest and log | +| Java 8 | Use Temurin 8 to package the default reactor with Spark 3 and Flink 1, then upload the package, checksums, manifest, and log | +| Java 11 | Use Temurin 11 to package Flink 2 and Iceberg, then upload the package, checksums, manifest, and log | +| Java 17 | Use Temurin 17 to package Spark 4, then upload the package, checksums, manifest, and log | | Python package | Build and validate the PyPaimon source distribution and universal wheel, then upload them as workflow artifacts | | Python publish | Publish an RC to TestPyPI only after all Java and Python packaging jobs pass; publish a final tag to PyPI | -The Java jobs deploy signed artifacts to Nexus staging but must not close or -release those repositories. The Python RC job publishes -`PYPAIMON_VERSIONrcRC_NUMBER` to TestPyPI. Stable PyPI publication and all -promotion steps remain disabled until the vote passes. +The Java jobs use `-Dgpg.skip=true` and never receive Nexus credentials or a +GPG private key. Their artifacts are build evidence, not the Maven staging +repositories used for the vote. The Python RC job publishes +`PYPAIMON_VERSIONrcRC_NUMBER` to TestPyPI. -Configure the protected release environment with the Nexus credentials already -used by the snapshot workflows (`NEXUS_USER` and `NEXUS_PW`), the release GPG -private key and passphrase (`GPG_SECRET_KEY` and `GPG_PASSPHRASE`), and the -Python repository tokens (`TEST_PYPI_API_TOKEN` and `PYPI_API_TOKEN`). Require -an RM approval for jobs that use publishing credentials. +Configure the protected release environment with only the Python repository +tokens (`TEST_PYPI_API_TOKEN` and `PYPI_API_TOKEN`). Require an RM approval for +jobs that use publishing credentials. ## One-time RM setup @@ -104,10 +102,14 @@ Before managing the first release: 2. Append the public key to the Paimon [KEYS](https://downloads.apache.org/paimon/KEYS) file. Never remove keys required to verify an older release. -3. Configure Git to sign tags with the same key. -4. Confirm access to the ASF distribution SVN repository, Apache Nexus, GitHub +3. Configure Git and local GPG to sign tags and Maven artifacts with the same + key. +4. Configure the local Maven server ID `apache.releases.https` with the RM's + Apache Nexus credentials. Do not copy the GPG key or Nexus credentials into + GitHub Actions. +5. Confirm access to the ASF distribution SVN repository, Apache Nexus, GitHub release environments, TestPyPI, and PyPI. -5. Verify that all required release secrets are configured without printing +6. Verify that the Python release secrets are configured without printing their values in an Actions log. ```shell @@ -138,7 +140,7 @@ RELEASE_TAG="release-${PAIMON_VERSION}" ``` Use these exact values in the branch, tag, workflow inputs, SVN directories, -vote email, and Java staging manifests. +vote email, and Java package manifests. ### Work from a clean clone @@ -212,13 +214,58 @@ Pushing the signed tag starts the Release workflow. Wait for every Java and Python job to succeed. Record: - the workflow run URL and `head_sha`; -- all JDK 8, 11, and 17 Nexus staging repository IDs from the uploaded - manifests; +- the JDK 8, 11, and 17 package artifact names, manifests, and SHA-512 + checksums; - the TestPyPI project/version URL. Do not start the vote when a required lane is missing or has been rerun from a different commit. +## Stage the Java convenience artifacts locally + +Check out the exact signed RC tag in a clean clone. Use the RM machine's local +GPG key and Maven credentials; do not download a private key into a CI runner. +Use Maven 3.8.8, matching the Release workflow. Export `GPG_TTY` when the local +GPG agent needs terminal access: + +```shell +git checkout --detach "refs/tags/${RC_REF}" +export GPG_TTY="$(tty)" + +gpg --list-secret-keys --keyid-format LONG +mvn -q -DforceStdout help:evaluate -Dexpression=project.version +``` + +Download the three Java package artifacts from the recorded workflow run and +verify each `*-packages.tar.gz.sha512` file. Keep the downloaded +`jdk8-sha512.txt`, `jdk11-sha512.txt`, and `jdk17-sha512.txt` files for the +reproducibility check after each local build. + +Run each existing staging script under its required JDK. Confirm the output of +`java -version` and `mvn -version` before every command: + +```shell +# JDK 8: default reactor, Flink 1.x, and Spark 3.x +./tools/releasing/deploy_staging_jars.sh +sha512sum -c /path/to/java-package-jdk8/jdk8-sha512.txt + +# JDK 11: Flink 2.x and Iceberg +./tools/releasing/deploy_staging_jars_for_jdk11.sh +sha512sum -c /path/to/java-package-jdk11/jdk11-sha512.txt + +# JDK 17: Spark 4.x +./tools/releasing/deploy_staging_jars_for_jdk17.sh +sha512sum -c /path/to/java-package-jdk17/jdk17-sha512.txt +``` + +On macOS, replace `sha512sum -c` with `shasum -a 512 -c`. A checksum mismatch +between the CI package and the locally signed build is a release blocker. +Maven signs the release artifacts with the RM's local GPG key and deploys them +using the local `apache.releases.https` server credentials. After each command, +record the `orgapachepaimon-XXXX` repository ID, confirm that it contains only +the intended lane, and leave it staged for the vote. Do not close or release a +repository before the vote passes. + ## Stage the source candidates Create both source candidates locally from the exact signed tag in a fresh diff --git a/docs/docs/project/verifying-a-release-candidate.md b/docs/docs/project/verifying-a-release-candidate.md index 5328bfa08456..2e43df934d5f 100644 --- a/docs/docs/project/verifying-a-release-candidate.md +++ b/docs/docs/project/verifying-a-release-candidate.md @@ -308,8 +308,9 @@ Open the workflow run linked in the vote email and confirm: - JDK 8, JDK 11, JDK 17, Python packaging, and Python publishing jobs all succeeded; - the logs show the expected JDK and Python versions; -- the Nexus IDs in the three staging manifests and the TestPyPI version equal - those in the vote email; +- each Java package artifact contains the expected manifest and SHA-512 + checksums for its lane; +- the TestPyPI version equals the version in the vote email; - no later rerun silently replaced a failed lane with artifacts from another commit. @@ -333,7 +334,7 @@ Verified: - Java staging repositories and representative class-file targets - PyPaimon source build/tests on Python - TestPyPI installation and smoke test -- GitHub Actions run provenance, Java staging manifests, and Python artifacts +- GitHub Actions run provenance, Java package manifests, and Python artifacts ``` For a `-1`, describe the failure precisely enough for the RM to reproduce it. From f7a06af2b790d8aae47a262252a1cc69d14718af Mon Sep 17 00:00:00 2001 From: JingsongLi Date: Thu, 30 Jul 2026 23:26:05 +0800 Subject: [PATCH 4/8] [release] Fix release workflow compliance gaps --- .github/workflows/release-java.yml | 67 +++++++++---------- .github/workflows/release-python-publish.yml | 4 +- .github/workflows/release.yml | 1 - docs/docs/project/creating-a-release.md | 57 ++++++++++------ .../project/verifying-a-release-candidate.md | 29 ++++++-- 5 files changed, 90 insertions(+), 68 deletions(-) diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml index 30e3b6c1e9fa..d72cc7c1d323 100644 --- a/.github/workflows/release-java.yml +++ b/.github/workflows/release-java.yml @@ -58,7 +58,7 @@ jobs: cache: maven - name: Set up Maven - uses: stCarolas/setup-maven@v5 + uses: stCarolas/setup-maven@d6af6abeda15e98926a57b5aa970a96bb37f97d1 # v5 with: maven-version: 3.8.8 @@ -100,52 +100,39 @@ jobs: mkdir -p release-java log="release-java/${LANE}.log" - artifact_dirs=() + install_log="release-java/${LANE}-install.log" case "${LANE}" in jdk8) mvn clean install -ntp -B \ -Papache-release,docs-and-source,spark3,flink1 \ - -DskipTests -Dgpg.skip=true \ - 2>&1 | tee "${log}" + -DskipTests -Dgpg.skip=true -Dstyle.color=never \ + 2>&1 | tee "${log}" "${install_log}" ;; jdk11) mvn clean install -ntp -B \ -Pdocs-and-source,flink2 \ - -DskipTests \ + -DskipTests -Dstyle.color=never \ -pl org.apache.paimon:paimon-flink-2.0,org.apache.paimon:paimon-flink-2.1,org.apache.paimon:paimon-flink-2.2,org.apache.paimon:paimon-iceberg \ -am \ 2>&1 | tee "${log}" - mvn package -ntp -B \ + mvn install -ntp -B \ -Papache-release,docs-and-source,flink2 \ - -DskipTests -Dgpg.skip=true \ + -DskipTests -Dgpg.skip=true -Dstyle.color=never \ -pl org.apache.paimon:paimon-flink-2.0,org.apache.paimon:paimon-flink-2.1,org.apache.paimon:paimon-flink-2.2,org.apache.paimon:paimon-flink2-common,org.apache.paimon:paimon-iceberg \ - 2>&1 | tee -a "${log}" - artifact_dirs=( - paimon-flink/paimon-flink-2.0 - paimon-flink/paimon-flink-2.1 - paimon-flink/paimon-flink-2.2 - paimon-flink/paimon-flink2-common - paimon-iceberg - ) + 2>&1 | tee -a "${log}" "${install_log}" ;; jdk17) mvn clean install -ntp -B \ -Pdocs-and-source,spark4 \ - -DskipTests \ + -DskipTests -Dstyle.color=never \ -pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \ -am \ 2>&1 | tee "${log}" - mvn package -ntp -B \ + mvn install -ntp -B \ -Papache-release,docs-and-source,spark4 \ - -DskipTests -Dgpg.skip=true \ + -DskipTests -Dgpg.skip=true -Dstyle.color=never \ -pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13 \ - 2>&1 | tee -a "${log}" - artifact_dirs=( - paimon-spark/paimon-spark-common - paimon-spark/paimon-spark4-common - paimon-spark/paimon-spark-4.0 - paimon-spark/paimon-spark-4.1 - ) + 2>&1 | tee -a "${log}" "${install_log}" ;; *) echo "Unknown Java release lane: ${LANE}" >&2 @@ -154,18 +141,25 @@ jobs: esac files="release-java/${LANE}-files.txt" - if [[ "${#artifact_dirs[@]}" -eq 0 ]]; then - find . -type f -path '*/target/*.jar' -print \ - | LC_ALL=C sort \ - > "${files}" - else - for artifact_dir in "${artifact_dirs[@]}"; do - find "${artifact_dir}/target" \ - -maxdepth 1 -type f -name '*.jar' -print - done | LC_ALL=C sort > "${files}" - fi + while IFS= read -r artifact; do + if [[ "${artifact}" == "${GITHUB_WORKSPACE}/"* ]]; then + artifact="./${artifact#"${GITHUB_WORKSPACE}/"}" + elif [[ "${artifact}" != ./* ]]; then + echo "Unexpected installed artifact path: ${artifact}" >&2 + exit 1 + fi + if [[ ! -f "${artifact}" ]]; then + echo "Installed artifact does not exist: ${artifact}" >&2 + exit 1 + fi + printf '%s\n' "${artifact}" + done < <( + sed -n \ + 's/^\[INFO\] Installing \(.*\.jar\) to .*$/\1/p' \ + "${install_log}" + ) | LC_ALL=C sort -u > "${files}" if [[ ! -s "${files}" ]]; then - echo "No Java release artifacts were produced for ${LANE}" >&2 + echo "No installed Java release artifacts were found for ${LANE}" >&2 exit 1 fi @@ -187,6 +181,7 @@ jobs: echo "commit=${GITHUB_SHA}" echo "signed=false" echo "staged=false" + echo "artifact_source=maven-install-log" } > "release-java/${LANE}-manifest.txt" - name: Upload Java packages diff --git a/.github/workflows/release-python-publish.yml b/.github/workflows/release-python-publish.yml index ed921c8a6b46..8ff75eff9ad6 100644 --- a/.github/workflows/release-python-publish.yml +++ b/.github/workflows/release-python-publish.yml @@ -35,7 +35,8 @@ jobs: if: >- github.repository == 'apache/paimon' && startsWith(github.ref, 'refs/tags/') - environment: release + environment: + name: ${{ contains(github.ref_name, '-rc') && 'testpypi' || 'pypi' }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 @@ -83,6 +84,5 @@ jobs: if: ${{ !contains(github.ref_name, '-rc') }} uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e with: - skip-existing: true packages-dir: dist password: ${{ secrets.PYPI_API_TOKEN }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index e675a60f4cd2..b80ad4af78b1 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -54,4 +54,3 @@ jobs: needs.python-package.result == 'success' && (needs.java.result == 'success' || !contains(github.ref_name, '-rc')) uses: ./.github/workflows/release-python-publish.yml - secrets: inherit diff --git a/docs/docs/project/creating-a-release.md b/docs/docs/project/creating-a-release.md index 05a6b59a2c9a..caa6b6aa7992 100644 --- a/docs/docs/project/creating-a-release.md +++ b/docs/docs/project/creating-a-release.md @@ -49,8 +49,11 @@ their version numbers are equal. | PyPaimon source | `pypaimon-PYPAIMON_VERSION.tar.gz`, `.asc`, `.sha512` | ASF distribution | | Python convenience package | `pypaimon==PYPAIMON_VERSIONrcRC_NUMBER` for an RC | TestPyPI, then `pypaimon==PYPAIMON_VERSION` on PyPI | -A combined release vote covers both signed source candidates. If PyPaimon is -released separately, use the same Python steps and hold a separate vote. +A combined release vote covers both signed source candidates. This guide does +not define an independent PyPaimon release. Before releasing PyPaimon +separately, the PMC must define a Python-only tag and workflow which do not +depend on the Maven version or Java jobs, and must provide a signed source +package which is independently sufficient to build and test the release. ### Java build matrix @@ -86,12 +89,15 @@ The workflow has the following contract: The Java jobs use `-Dgpg.skip=true` and never receive Nexus credentials or a GPG private key. Their artifacts are build evidence, not the Maven staging -repositories used for the vote. The Python RC job publishes -`PYPAIMON_VERSIONrcRC_NUMBER` to TestPyPI. +repositories used for the vote. The Python RC job uses the protected +`testpypi` environment to publish `PYPAIMON_VERSIONrcRC_NUMBER` to TestPyPI. +The final job uses the separate protected `pypi` environment. -Configure the protected release environment with only the Python repository -tokens (`TEST_PYPI_API_TOKEN` and `PYPI_API_TOKEN`). Require an RM approval for -jobs that use publishing credentials. +Configure both environments with an RM approval and a deployment policy which +allows only the corresponding signed release tags. Store +`TEST_PYPI_API_TOKEN` only in `testpypi` and `PYPI_API_TOKEN` only in `pypi`. +Do not store either token as a repository or organization secret, and do not +pass unrelated secrets to the reusable publishing workflow. ## One-time RM setup @@ -107,9 +113,13 @@ Before managing the first release: 4. Configure the local Maven server ID `apache.releases.https` with the RM's Apache Nexus credentials. Do not copy the GPG key or Nexus credentials into GitHub Actions. -5. Confirm access to the ASF distribution SVN repository, Apache Nexus, GitHub - release environments, TestPyPI, and PyPI. -6. Verify that the Python release secrets are configured without printing +5. Confirm access to the ASF distribution SVN repository, Apache Nexus, + TestPyPI, and PyPI. +6. Create the protected GitHub environments `testpypi` and `pypi`. Configure + required RM reviewers and release-tag deployment policies, store only the + matching token in each environment, and remove any repository- or + organization-level copies of those tokens. +7. Verify that the Python release secrets are configured without printing their values in an Actions log. ```shell @@ -262,9 +272,10 @@ On macOS, replace `sha512sum -c` with `shasum -a 512 -c`. A checksum mismatch between the CI package and the locally signed build is a release blocker. Maven signs the release artifacts with the RM's local GPG key and deploys them using the local `apache.releases.https` server credentials. After each command, -record the `orgapachepaimon-XXXX` repository ID, confirm that it contains only -the intended lane, and leave it staged for the vote. Do not close or release a -repository before the vote passes. +record the `orgapachepaimon-XXXX` repository ID and confirm that it contains +only the intended lane. Close each repository and resolve every close-time rule +failure before starting the vote. Closing freezes the candidate that voters +inspect. Do not release a repository before the vote passes. ## Stage the source candidates @@ -345,9 +356,11 @@ https://dist.apache.org/repos/dist/dev/paimon/paimon-${PAIMON_VERSION}-rc${RC_NU PyPaimon source candidate: https://dist.apache.org/repos/dist/dev/paimon/pypaimon-${PYPAIMON_VERSION}-rc${RC_NUMBER}/ -Signed Git tag and commit: -https://github.com/apache/paimon/releases/tag/release-${PAIMON_VERSION}-rc${RC_NUMBER} - +Signed Git tag: +release-${PAIMON_VERSION}-rc${RC_NUMBER} + +Commit: +https://github.com/apache/paimon/commit/ GitHub Actions release run: @@ -355,7 +368,7 @@ GitHub Actions release run: KEYS: https://downloads.apache.org/paimon/KEYS -Java staging repositories: +Closed Java staging repositories: @@ -416,11 +429,11 @@ svn mv -m "Release PyPaimon ${PYPAIMON_VERSION}" \ ### Promote convenience artifacts -1. In Nexus, close every recorded JDK 8, 11, and 17 staging repository. Inspect - all rule failures and artifact trees before continuing. -2. Release those exact repositories to Maven Central. Do not run Maven deploy - again. -3. Approve the protected PyPI promotion job for the final tag. It must build +1. In Nexus, confirm that every recorded JDK 8, 11, and 17 staging repository + is still closed and has the exact artifact tree approved by the vote. +2. Release those exact closed repositories to Maven Central. Do not run Maven + deploy again. +3. Approve the protected `pypi` promotion job for the final tag. It must build `pypaimon==PYPAIMON_VERSION` from the approved tag commit and must not change project source. 4. Verify Maven Central and PyPI before announcing the release. diff --git a/docs/docs/project/verifying-a-release-candidate.md b/docs/docs/project/verifying-a-release-candidate.md index 2e43df934d5f..de0eaab6ae9e 100644 --- a/docs/docs/project/verifying-a-release-candidate.md +++ b/docs/docs/project/verifying-a-release-candidate.md @@ -127,14 +127,14 @@ Check at least the following: - `paimon-python/setup.py` and PyPaimon package metadata use `PYPAIMON_VERSION` without `.dev`. -Extract the candidates: +Extract the Paimon candidate: ```shell tar xzf "${PAIMON_ARCHIVE}" -tar xzf "${PYPAIMON_ARCHIVE}" ``` -Run the repository licensing checks from the extracted sources: +Run the repository licensing checks. The PyPaimon checker temporarily extracts +its source package and removes that temporary directory when it finishes: ```shell cd "paimon-${PAIMON_VERSION}" @@ -146,6 +146,12 @@ SOURCE_PACKAGE="${PYPAIMON_ARCHIVE}" \ ``` Review the generated reports rather than relying only on the exit codes. +After the licensing check, extract a fresh PyPaimon source tree for the build +and test steps below: + +```shell +tar xzf "${PYPAIMON_ARCHIVE}" +``` ## Build Java from the source archive @@ -161,8 +167,8 @@ This lane covers the default reactor, Flink 1.x, and Spark 3.x: mvn -ntp clean verify -Pspark3,flink1 ``` -If a full verification is not practical on your machine, at minimum build the -same scope used for staging and state that tests were skipped: +For supplementary or non-binding verification, you may build the same scope +used for staging and state explicitly that tests were skipped: ```shell mvn -ntp clean install -DskipTests \ @@ -206,7 +212,11 @@ mvn -ntp clean install -DskipTests -Pdocs-and-source,spark4 \ ``` Investigate warnings and skipped modules. A successful compiler exit does not, -by itself, establish that the candidate is suitable for release. +by itself, establish that the candidate is suitable for release. The +`-DskipTests` commands above do not by themselves satisfy the requirements for +a binding `+1`. A binding voter must compile the signed source package and test +the result on their own platform; run an appropriate test or smoke test against +the artifacts produced by that source build and report the exact scope. ## Verify Java staging repositories @@ -216,6 +226,8 @@ candidate. For every staging repository: +- Confirm that its status is closed, so the candidate cannot change during the + vote. - Confirm that the version is `PAIMON_VERSION`, never a snapshot. - Confirm that each published module has its POM, main artifact, source JAR, Javadoc JAR, detached signatures, and checksums where applicable. @@ -266,7 +278,10 @@ Inspect the generated sdist and wheel. They must contain the expected Python sources, `LICENSE`, `NOTICE`, README, and package metadata, with no tests, caches, credentials, or unrelated build output. -The complete Python test suite lives in the Paimon source archive. Run it from +This guide covers a combined Paimon and PyPaimon release. The complete Python +test suite for the PyPaimon candidate is supplied in the signed Paimon source +archive; the PyPaimon source distribution must not be used by itself for an +independent release. Run the tests from `paimon-PAIMON_VERSION/paimon-python` on as many supported Python versions as your environment allows. The project CI selects Python 3.6, 3.7, 3.10, and 3.11 for its main compatibility lanes: From 43b4af77ee355e5d8269c9d567d4c6cecac36ae6 Mon Sep 17 00:00:00 2001 From: JingsongLi Date: Fri, 31 Jul 2026 08:15:41 +0800 Subject: [PATCH 5/8] [release] Reject snapshot dependencies in release builds --- .github/workflows/release-java.yml | 35 +++++++++++++++++++++++++ docs/docs/project/creating-a-release.md | 6 +++++ 2 files changed, 41 insertions(+) diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml index d72cc7c1d323..ade3ea7e3bbd 100644 --- a/.github/workflows/release-java.yml +++ b/.github/workflows/release-java.yml @@ -90,6 +90,41 @@ jobs: java -version mvn -version + - name: Reject SNAPSHOT project versions and dependencies + shell: bash + env: + LANE: ${{ matrix.lane }} + run: | + set -euo pipefail + + enforcer_goal="org.apache.maven.plugins:maven-enforcer-plugin:3.6.3:enforce" + enforcer_rules="requireReleaseVersion,requireReleaseDeps" + case "${LANE}" in + jdk8) + mvn -ntp -B "${enforcer_goal}" \ + -Denforcer.rules="${enforcer_rules}" \ + -Papache-release,docs-and-source,spark3,flink1 + ;; + jdk11) + mvn -ntp -B "${enforcer_goal}" \ + -Denforcer.rules="${enforcer_rules}" \ + -Papache-release,docs-and-source,flink2 \ + -pl org.apache.paimon:paimon-flink-2.0,org.apache.paimon:paimon-flink-2.1,org.apache.paimon:paimon-flink-2.2,org.apache.paimon:paimon-flink2-common,org.apache.paimon:paimon-iceberg \ + -am + ;; + jdk17) + mvn -ntp -B "${enforcer_goal}" \ + -Denforcer.rules="${enforcer_rules}" \ + -Papache-release,docs-and-source,spark4 \ + -pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13 \ + -am + ;; + *) + echo "Unknown Java release lane: ${LANE}" >&2 + exit 1 + ;; + esac + - name: Package Java release artifacts shell: bash env: diff --git a/docs/docs/project/creating-a-release.md b/docs/docs/project/creating-a-release.md index caa6b6aa7992..361f43c83453 100644 --- a/docs/docs/project/creating-a-release.md +++ b/docs/docs/project/creating-a-release.md @@ -87,6 +87,12 @@ The workflow has the following contract: | Python package | Build and validate the PyPaimon source distribution and universal wheel, then upload them as workflow artifacts | | Python publish | Publish an RC to TestPyPI only after all Java and Python packaging jobs pass; publish a final tag to PyPI | +Before packaging, every Java lane runs Maven Enforcer's +`requireReleaseVersion` and `requireReleaseDeps` rules over its complete reactor +scope. The latter includes transitive dependencies. Any remaining +`-SNAPSHOT` project, parent, direct dependency, or transitive dependency is a +release blocker. + The Java jobs use `-Dgpg.skip=true` and never receive Nexus credentials or a GPG private key. Their artifacts are build evidence, not the Maven staging repositories used for the vote. The Python RC job uses the protected From fd7d3f03c054c5905d24ec289219c639d5f53eb8 Mon Sep 17 00:00:00 2001 From: JingsongLi Date: Fri, 31 Jul 2026 08:18:47 +0800 Subject: [PATCH 6/8] [release] Add common release tag validation --- .github/workflows/release.yml | 77 ++++++++++++++++++- docs/docs/project/creating-a-release.md | 6 +- .../project/verifying-a-release-candidate.md | 3 + 3 files changed, 82 insertions(+), 4 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b80ad4af78b1..e9f14bf48397 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -37,20 +37,93 @@ permissions: contents: read jobs: + validation: + name: Validate release tag + runs-on: ubuntu-latest + outputs: + release_version: ${{ steps.release.outputs.release_version }} + release_kind: ${{ steps.release.outputs.release_kind }} + rc_number: ${{ steps.release.outputs.rc_number }} + steps: + - uses: actions/checkout@v6 + + - name: Set up JDK 8 + uses: actions/setup-java@v5 + with: + java-version: 8 + distribution: temurin + cache: maven + + - name: Set up Maven + uses: stCarolas/setup-maven@d6af6abeda15e98926a57b5aa970a96bb37f97d1 # v5 + with: + maven-version: 3.8.8 + + - name: Validate tag against project version + id: release + shell: bash + run: | + set -euo pipefail + + if [[ "${GITHUB_REF}" != refs/tags/* ]]; then + echo "Release workflow must run from a tag, got ${GITHUB_REF}" >&2 + exit 1 + fi + + release_version="$( + mvn -q -DforceStdout help:evaluate \ + -Dexpression=project.version + )" + if [[ -z "${release_version}" || "${release_version}" == *-SNAPSHOT ]]; then + echo "Project release version is invalid: ${release_version:-missing}" >&2 + exit 1 + fi + + final_tag="release-${release_version}" + rc_prefix="${final_tag}-rc" + release_kind="" + rc_number="" + if [[ "${GITHUB_REF_NAME}" == "${final_tag}" ]]; then + release_kind="final" + elif [[ "${GITHUB_REF_NAME}" == "${rc_prefix}"* ]]; then + rc_number="${GITHUB_REF_NAME#"${rc_prefix}"}" + if [[ ! "${rc_number}" =~ ^[0-9]+$ ]]; then + echo "RC tag must be ${rc_prefix}N, got ${GITHUB_REF_NAME}" >&2 + exit 1 + fi + release_kind="rc" + else + echo "Tag must be ${final_tag} or ${rc_prefix}N, got ${GITHUB_REF_NAME}" >&2 + exit 1 + fi + + echo "release_version=${release_version}" >> "${GITHUB_OUTPUT}" + echo "release_kind=${release_kind}" >> "${GITHUB_OUTPUT}" + echo "rc_number=${rc_number}" >> "${GITHUB_OUTPUT}" + + echo "Validated ${release_kind} tag ${GITHUB_REF_NAME}" + java -version + mvn -version + java: name: Java packages + needs: validation + if: needs.validation.outputs.release_kind == 'rc' uses: ./.github/workflows/release-java.yml python-package: name: PyPaimon package + needs: validation uses: ./.github/workflows/release-python.yml python-publish: name: PyPaimon publish - needs: [java, python-package] + needs: [validation, java, python-package] if: >- always() && + needs.validation.result == 'success' && startsWith(github.ref, 'refs/tags/') && needs.python-package.result == 'success' && - (needs.java.result == 'success' || !contains(github.ref_name, '-rc')) + (needs.java.result == 'success' || + needs.validation.outputs.release_kind == 'final') uses: ./.github/workflows/release-python-publish.yml diff --git a/docs/docs/project/creating-a-release.md b/docs/docs/project/creating-a-release.md index 361f43c83453..e0e4480f8d15 100644 --- a/docs/docs/project/creating-a-release.md +++ b/docs/docs/project/creating-a-release.md @@ -81,6 +81,7 @@ The workflow has the following contract: | Job | Required behavior | | --- | --- | +| Validation | Require an RC tag named `release-PAIMON_VERSION-rcN` or a final tag named `release-PAIMON_VERSION`, where `PAIMON_VERSION` exactly equals the root Maven `project.version` | | Java 8 | Use Temurin 8 to package the default reactor with Spark 3 and Flink 1, then upload the package, checksums, manifest, and log | | Java 11 | Use Temurin 11 to package Flink 2 and Iceberg, then upload the package, checksums, manifest, and log | | Java 17 | Use Temurin 17 to package Spark 4, then upload the package, checksums, manifest, and log | @@ -226,8 +227,9 @@ git push origin \ "refs/tags/${RC_REF}:refs/tags/${RC_REF}" ``` -Pushing the signed tag starts the Release workflow. Wait for every Java and -Python job to succeed. Record: +Pushing the signed tag starts the Release workflow. The common validation job +must succeed before any Java or Python packaging or publishing job can run. +Wait for every required job to succeed. Record: - the workflow run URL and `head_sha`; - the JDK 8, 11, and 17 package artifact names, manifests, and SHA-512 diff --git a/docs/docs/project/verifying-a-release-candidate.md b/docs/docs/project/verifying-a-release-candidate.md index de0eaab6ae9e..a36c3c594cb5 100644 --- a/docs/docs/project/verifying-a-release-candidate.md +++ b/docs/docs/project/verifying-a-release-candidate.md @@ -320,6 +320,9 @@ Open the workflow run linked in the vote email and confirm: - it was triggered from the signed `RC_TAG`; - `head_sha` equals the announced commit; +- the common validation job confirmed that `RC_TAG` is exactly + `release-PAIMON_VERSION-rcRC_NUMBER` and that `PAIMON_VERSION` equals the + root Maven `project.version`; - JDK 8, JDK 11, JDK 17, Python packaging, and Python publishing jobs all succeeded; - the logs show the expected JDK and Python versions; From 7a872cda39abc094632be8844dae8be26d6dd3e9 Mon Sep 17 00:00:00 2001 From: JingsongLi Date: Fri, 31 Jul 2026 08:25:26 +0800 Subject: [PATCH 7/8] [release] Reject reused TestPyPI RC versions --- .github/workflows/release-python-publish.yml | 32 +++++++++++++++++++- docs/docs/project/creating-a-release.md | 8 +++-- 2 files changed, 37 insertions(+), 3 deletions(-) diff --git a/.github/workflows/release-python-publish.yml b/.github/workflows/release-python-publish.yml index 8ff75eff9ad6..eaa2af9eb1fc 100644 --- a/.github/workflows/release-python-publish.yml +++ b/.github/workflows/release-python-publish.yml @@ -47,6 +47,7 @@ jobs: path: dist - name: Verify package versions + id: package_version shell: bash run: | set -euo pipefail @@ -71,12 +72,41 @@ jobs: exit 1 fi + echo "version=${expected_version}" >> "${GITHUB_OUTPUT}" + + - name: Reject existing TestPyPI RC version + if: contains(github.ref_name, '-rc') + shell: bash + env: + EXPECTED_VERSION: ${{ steps.package_version.outputs.version }} + run: | + set -euo pipefail + + status="$( + curl --silent --show-error --output /dev/null \ + --write-out '%{http_code}' \ + "https://test.pypi.org/pypi/pypaimon/${EXPECTED_VERSION}/json" + )" + + case "${status}" in + 404) + ;; + 200) + echo "pypaimon ${EXPECTED_VERSION} already exists on TestPyPI; create a new RC" >&2 + exit 1 + ;; + *) + echo "TestPyPI version check returned HTTP ${status}; refusing to publish" >&2 + exit 1 + ;; + esac + - name: Publish RC to TestPyPI if: contains(github.ref_name, '-rc') uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e with: repository-url: https://test.pypi.org/legacy/ - skip-existing: true + skip-existing: false packages-dir: dist password: ${{ secrets.TEST_PYPI_API_TOKEN }} diff --git a/docs/docs/project/creating-a-release.md b/docs/docs/project/creating-a-release.md index e0e4480f8d15..ef3c779fd97f 100644 --- a/docs/docs/project/creating-a-release.md +++ b/docs/docs/project/creating-a-release.md @@ -338,7 +338,11 @@ svn commit -m \ paimon-dist-dev ``` -Never overwrite an existing RC directory. If any byte changes, create a new RC. +Never overwrite an existing RC directory. TestPyPI versions are immutable for +this release process as well: the workflow checks for an existing version before +upload and fails instead of skipping files. The uploader also rejects duplicate +files to cover races after the check. If any byte changes, create a new RC with +a new RC number. ## Call the vote @@ -401,7 +405,7 @@ If the vote finds a problem: 2. Drop every Nexus staging repository belonging to the failed RC. 3. Remove the superseded dist-dev directories, or retain them temporarily when useful to the vote discussion. Never replace their contents. -4. Increment `RC_NUMBER`. +4. Increment `RC_NUMBER`; never reuse the failed candidate's TestPyPI version. 5. Create a new RC branch and signed tag, run every workflow lane again, stage new source candidates, and start a new 72-hour vote. From 790223b141e985054670e94f0f52f20f23de4ae9 Mon Sep 17 00:00:00 2001 From: JingsongLi Date: Fri, 31 Jul 2026 08:29:00 +0800 Subject: [PATCH 8/8] [release] Fix verification command paths and refs --- docs/docs/project/creating-a-release.md | 6 +- .../project/verifying-a-release-candidate.md | 59 +++++++++++++------ 2 files changed, 43 insertions(+), 22 deletions(-) diff --git a/docs/docs/project/creating-a-release.md b/docs/docs/project/creating-a-release.md index ef3c779fd97f..bfe20952e6c1 100644 --- a/docs/docs/project/creating-a-release.md +++ b/docs/docs/project/creating-a-release.md @@ -416,11 +416,11 @@ If the vote finds a problem: The final tag must point to exactly the approved RC commit: ```shell -git tag -s "${RELEASE_TAG}" "${RC_REF}^{commit}" \ +git tag -s "${RELEASE_TAG}" "refs/tags/${RC_REF}^{commit}" \ -m "Release Apache Paimon ${PAIMON_VERSION}" -test "$(git rev-parse "${RC_REF}^{commit}")" = \ - "$(git rev-parse "${RELEASE_TAG}^{commit}")" +test "$(git rev-parse "refs/tags/${RC_REF}^{commit}")" = \ + "$(git rev-parse "refs/tags/${RELEASE_TAG}^{commit}")" git tag -v "${RELEASE_TAG}" git push origin "refs/tags/${RELEASE_TAG}:refs/tags/${RELEASE_TAG}" ``` diff --git a/docs/docs/project/verifying-a-release-candidate.md b/docs/docs/project/verifying-a-release-candidate.md index a36c3c594cb5..6910c909f442 100644 --- a/docs/docs/project/verifying-a-release-candidate.md +++ b/docs/docs/project/verifying-a-release-candidate.md @@ -94,7 +94,7 @@ shasum -a 512 -c "${PYPAIMON_ARCHIVE}.sha512" git clone https://github.com/apache/paimon.git paimon-candidate-git git -C paimon-candidate-git fetch --tags git -C paimon-candidate-git tag -v "${RC_TAG}" -git -C paimon-candidate-git rev-parse "${RC_TAG}^{commit}" +git -C paimon-candidate-git rev-parse "refs/tags/${RC_TAG}^{commit}" ``` The resolved commit must equal both the SHA in the vote email and the @@ -164,15 +164,21 @@ and Maven state from one JDK cannot leak into another lane. Record `java This lane covers the default reactor, Flink 1.x, and Spark 3.x: ```shell -mvn -ntp clean verify -Pspark3,flink1 +( + cd "paimon-${PAIMON_VERSION}" + mvn -ntp clean verify -Pspark3,flink1 +) ``` For supplementary or non-binding verification, you may build the same scope used for staging and state explicitly that tests were skipped: ```shell -mvn -ntp clean install -DskipTests \ - -Pdocs-and-source,spark3,flink1 +( + cd "paimon-${PAIMON_VERSION}" + mvn -ntp clean install -DskipTests \ + -Pdocs-and-source,spark3,flink1 +) ``` ### JDK 11 lane @@ -180,17 +186,23 @@ mvn -ntp clean install -DskipTests \ This lane covers Flink 2.x and Iceberg: ```shell -mvn -ntp clean verify -Pflink2 \ - -pl org.apache.paimon:paimon-flink-2.0,org.apache.paimon:paimon-flink-2.1,org.apache.paimon:paimon-flink-2.2,org.apache.paimon:paimon-iceberg \ - -am +( + cd "paimon-${PAIMON_VERSION}" + mvn -ntp clean verify -Pflink2 \ + -pl org.apache.paimon:paimon-flink-2.0,org.apache.paimon:paimon-flink-2.1,org.apache.paimon:paimon-flink-2.2,org.apache.paimon:paimon-iceberg \ + -am +) ``` The packaging-equivalent build is: ```shell -mvn -ntp clean install -DskipTests -Pdocs-and-source,flink2 \ - -pl org.apache.paimon:paimon-flink-2.0,org.apache.paimon:paimon-flink-2.1,org.apache.paimon:paimon-flink-2.2,org.apache.paimon:paimon-iceberg \ - -am +( + cd "paimon-${PAIMON_VERSION}" + mvn -ntp clean install -DskipTests -Pdocs-and-source,flink2 \ + -pl org.apache.paimon:paimon-flink-2.0,org.apache.paimon:paimon-flink-2.1,org.apache.paimon:paimon-flink-2.2,org.apache.paimon:paimon-iceberg \ + -am +) ``` ### JDK 17 lane @@ -198,17 +210,23 @@ mvn -ntp clean install -DskipTests -Pdocs-and-source,flink2 \ This lane covers Spark 4.x: ```shell -mvn -ntp clean verify -Pspark4 \ - -pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \ - -am +( + cd "paimon-${PAIMON_VERSION}" + mvn -ntp clean verify -Pspark4 \ + -pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \ + -am +) ``` The packaging-equivalent build is: ```shell -mvn -ntp clean install -DskipTests -Pdocs-and-source,spark4 \ - -pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \ - -am +( + cd "paimon-${PAIMON_VERSION}" + mvn -ntp clean install -DskipTests -Pdocs-and-source,spark4 \ + -pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \ + -am +) ``` Investigate warnings and skipped modules. A successful compiler exit does not, @@ -287,9 +305,12 @@ your environment allows. The project CI selects Python 3.6, 3.7, 3.10, and 3.11 for its main compatibility lanes: ```shell -python -m pip install -r dev/requirements.txt -python -m pip install -r dev/requirements-dev.txt -python -m pytest pypaimon/tests -v +( + cd "paimon-${PAIMON_VERSION}/paimon-python" + python -m pip install -r dev/requirements.txt + python -m pip install -r dev/requirements-dev.txt + python -m pytest pypaimon/tests -v +) ``` Some optional integration tests need additional services or packages. Record