From 31a62e01c48514604ab1d6972967f4e8485430f3 Mon Sep 17 00:00:00 2001 From: Vinish Reddy Date: Thu, 30 Jul 2026 16:44:16 -0700 Subject: [PATCH 1/2] [701] Keep exactly one LICENSE and NOTICE in every published jar Every jar carried its licensing information twice. META-INF/LICENSE-bundled and META-INF/NOTICE-bundled are inputs for the shade plugin, which writes their content out as META-INF/LICENSE and META-INF/NOTICE, but they were also packaged verbatim because they live in src/main/resources. In the non-shaded jars that second copy was not just redundant, it described dependencies that are not in the jar at all. On top of that, DontIncludeResourceTransformer only drops the exact resource names LICENSE, NOTICE and NOTICE.txt, so the shaded jars additionally carried each bundled dependency's own license file wherever that dependency happened to put it. xtable-hive-metastore-bundled shipped 19 such files across LICENSE.txt, about_files/, license/, META-INF/license/, META-INF/ASM_LICENSE.txt and META-INF/CLI-LICENSE.txt, which made the licensing of the bundle very hard to review. maven-jar-plugin now excludes the two -bundled files from the jar. They stay in target/classes, which is where IncludeResourceTransformer reads them from, so the shaded jars still get the bundled listing as META-INF/LICENSE while the non-shaded jars keep the plain Apache-2.0 text. The shade filters drop the non-canonical dependency license files. The source files under src/main/resources/META-INF are untouched, so validate_shaded_license_coverage.sh keeps working against them unchanged. Adds validate_jar_license_layout.sh, wired into the License Check workflow, to assert the invariant against the built jars: exactly one META-INF/LICENSE, a META-INF/NOTICE, and third party texts only under META-INF/licenses/. Run against the 0.4.0-incubating-rc1 artifacts it reports all 21 offending entries, which is how this went unnoticed for several releases. xtable-utilities is skipped, matching validate_shaded_license_coverage.sh: its deploy is skipped and create_source_release.sh excludes it, so it is not part of the published set. Its shaded jar has no META-INF/LICENSE at all, which is worth fixing separately. --- .github/workflows/mvn-license-check.yml | 6 + pom.xml | 42 +++++ .../scripts/validate_jar_license_layout.sh | 162 ++++++++++++++++++ 3 files changed, 210 insertions(+) create mode 100755 release/scripts/validate_jar_license_layout.sh diff --git a/.github/workflows/mvn-license-check.yml b/.github/workflows/mvn-license-check.yml index 193bc286d..04d37021a 100644 --- a/.github/workflows/mvn-license-check.yml +++ b/.github/workflows/mvn-license-check.yml @@ -50,3 +50,9 @@ jobs: - name: Validate Shaded Dependency License Metadata run: release/scripts/validate_shaded_license_coverage.sh + + # Builds the shaded modules itself if their jars are not present, so it also + # covers the shape of the jars we actually publish rather than only the + # metadata in the source tree. + - name: Validate Jar License Layout + run: release/scripts/validate_jar_license_layout.sh diff --git a/pom.xml b/pom.xml index f33e5b01c..1a962ee04 100644 --- a/pom.xml +++ b/pom.xml @@ -650,6 +650,26 @@ + + org.apache.maven.plugins + maven-jar-plugin + + + + META-INF/LICENSE-bundled + META-INF/NOTICE-bundled + + + org.apache.maven.plugins maven-shade-plugin @@ -669,6 +689,28 @@ META-INF/*.DSA META-INF/*.RSA **/Log4j2Plugins.dat + + about_files/** + license/** + licenses/** + META-INF/license/** + META-INF/*LICENSE*.txt + META-INF/*NOTICE*.txt + META-INF/LICENSE.md + META-INF/NOTICE.md + META-INF/NOTICE.markdown + LICENSE.txt + LICENSE.md + NOTICE.txt + NOTICE.md diff --git a/release/scripts/validate_jar_license_layout.sh b/release/scripts/validate_jar_license_layout.sh new file mode 100755 index 000000000..6398ea5b3 --- /dev/null +++ b/release/scripts/validate_jar_license_layout.sh @@ -0,0 +1,162 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +set -o errexit +set -o nounset +set -o pipefail + +# Asserts that every jar we publish carries its licensing information in exactly +# one canonical place: META-INF/LICENSE, META-INF/NOTICE and, for the shaded +# bundles, the third party texts under META-INF/licenses/. +# +# This exists because the layout regressed silently across several releases +# (XTABLE-701): every jar shipped a second, differently named copy of the +# licensing information as META-INF/LICENSE-bundled and META-INF/NOTICE-bundled, +# and the shaded jars additionally carried each bundled dependency's own license +# file in whatever location that dependency happened to use, which made the +# licensing of a bundle very hard to review. + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)" + +cd "${ROOT_DIR}" + +# Entries a jar is allowed to carry. Anything else that looks like licensing +# information is a finding: it means the information is duplicated, or filed +# somewhere a reviewer will not think to look. +is_allowed_entry() { + case "$1" in + META-INF/LICENSE|META-INF/NOTICE|META-INF/DISCLAIMER|META-INF/DISCLAIMER-WIP|META-INF/DEPENDENCIES) + return 0 + ;; + META-INF/licenses/*) + return 0 + ;; + esac + return 1 +} + +# xtable-utilities is not a release artifact: maven-deploy-plugin is skipped for +# it, and create_source_release.sh leaves it out of the source tarball. Its shaded +# jar has no META-INF/LICENSE at all, which is worth fixing on its own but is not +# part of the published set that XTABLE-701 is about. validate_shaded_license_ +# coverage.sh skips it for the same reason. +SKIPPED_MODULES=( + "xtable-utilities" +) + +discover_jars() { + local skip_pattern + skip_pattern="$(printf '%s\n' "${SKIPPED_MODULES[@]}" | paste -sd'|' -)" + find . -type f -name '*.jar' 2>/dev/null \ + | grep -E '/target/[^/]+\.jar$' \ + | grep -vE '\-(sources|javadoc)\.jar$' \ + | grep -vE '/target/original-' \ + | grep -vE "^\./(${skip_pattern})/" \ + | sort +} + +# Only the shaded modules, never the whole reactor: xtable-service needs a newer +# JDK than the one this runs under in CI, and it produces no shaded jar anyway. +shade_modules() { + grep -rl 'maven-shade-plugin' --include=pom.xml . \ + | xargs -n1 dirname \ + | sed 's|^\./||' \ + | grep -v '^\.$' \ + | sort -u \ + | paste -sd, - +} + +jars=("$@") + +if [[ ${#jars[@]} -eq 0 ]]; then + while IFS= read -r jar; do + jars+=("${jar}") + done < <(discover_jars) +fi + +if [[ ${#jars[@]} -eq 0 ]]; then + modules="$(shade_modules)" + echo "No jars found under */target, building the shaded modules first: ${modules}" + ./mvnw -q package -B -DskipTests -Dmaven.build.cache.enabled=false -pl "${modules}" -am + while IFS= read -r jar; do + jars+=("${jar}") + done < <(discover_jars) +fi + +if [[ ${#jars[@]} -eq 0 ]]; then + echo "No jars found under */target even after building. Pass jar paths as arguments." + exit 1 +fi + +overall_status=0 + +for jar in "${jars[@]}"; do + if [[ ! -f "${jar}" ]]; then + echo "FAIL ${jar}: not a file." + overall_status=1 + continue + fi + + # original-*.jar is the pre-shade copy the shade plugin leaves behind, it is + # never published. + case "$(basename "${jar}")" in + original-*) continue ;; + esac + + entries="$(unzip -Z1 "${jar}" | grep -v '/$' || true)" + + # Only consider entries that look like licensing information. Class files that + # merely contain the word "license" in their name are not interesting here. + # Deliberately matches the words anywhere in the path, not just at a path + # boundary: dependencies ship things like META-INF/ASM_LICENSE.txt and + # META-INF/CLI-LICENSE.txt, which a boundary-anchored pattern would miss. + candidates="$(printf '%s\n' "${entries}" \ + | grep -vE '\.class$' \ + | grep -iE '(licen[cs]e|notice|disclaimer|copying)|(^|/)about_files/' \ + || true)" + + findings=() + while IFS= read -r entry; do + [[ -z "${entry}" ]] && continue + if ! is_allowed_entry "${entry}"; then + findings+=("${entry}") + fi + done < <(printf '%s\n' "${candidates}") + + # Match in the shell rather than piping into "grep -q": grep exits on the first + # match, the writer then takes SIGPIPE, and under "set -o pipefail" the whole + # pipeline reports failure. That only shows up on the big shaded jars, where the + # writer has not finished yet, so it would fail intermittently and only for the + # bundles. + if [[ $'\n'${entries}$'\n' != *$'\n'"META-INF/LICENSE"$'\n'* ]]; then + echo "FAIL $(basename "${jar}"): META-INF/LICENSE is missing." + overall_status=1 + continue + fi + + if [[ ${#findings[@]} -gt 0 ]]; then + echo "FAIL $(basename "${jar}"): licensing information outside the canonical locations:" + printf ' - %s\n' "${findings[@]}" + overall_status=1 + else + texts="$(printf '%s\n' "${entries}" | grep -c '^META-INF/licenses/' || true)" + echo "OK $(basename "${jar}"): META-INF/LICENSE, META-INF/NOTICE, ${texts} bundled license text(s)." + fi +done + +exit "${overall_status}" From 4321fdc9ab42288d49182f8769e308db9e4f9db8 Mon Sep 17 00:00:00 2001 From: Vinish Reddy Date: Thu, 30 Jul 2026 16:49:10 -0700 Subject: [PATCH 2/2] [701] Broaden the shade license excludes to be dependency-tree independent Verifying the previous commit on main surfaced LICENSE-junit.txt in the root of xtable-hive-metastore-bundled, which branch-0.4 does not produce: the two branches resolve different junit versions, and only one of them ships that file. Enumerating the suffixes seen on one branch is therefore not enough, so the excludes now cover LICENSE*, NOTICE* and COPYING* at the jar root and META-INF/*LICENSE* / META-INF/*NOTICE* rather than a fixed list of extensions. Our own files are unaffected: META-INF/LICENSE and META-INF/NOTICE are written by IncludeResourceTransformer after filtering, and META-INF/licenses/ sits in a path segment of its own so none of these patterns can match it. Confirmed by rebuilding every shaded module from clean and checking that the bundled jars still carry the dependency listing (56 coordinates for aws, 183 for hive-metastore, 13 for hudi-support-extensions, 8 for spark-runtime) while the non-shaded jars still carry the plain 202 line Apache-2.0 text. --- pom.xml | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/pom.xml b/pom.xml index 1a962ee04..6924c9c2e 100644 --- a/pom.xml +++ b/pom.xml @@ -697,20 +697,26 @@ the shaded jars also carried each dependency's own copy in whatever location it happened to ship it, which made the licensing of the bundle very hard to review. + + These are deliberately broad rather than an enumeration of + the suffixes seen so far: dependencies use names like + LICENSE.txt, LICENSE-junit.txt, META-INF/ASM_LICENSE.txt and + META-INF/NOTICE.markdown, and which of them show up changes + with the dependency tree, so branch-0.4 and main did not even + produce the same set. Our own META-INF/LICENSE and + META-INF/NOTICE are written by IncludeResourceTransformer, and + META-INF/licenses/ has a path segment of its own, so none of + these patterns can match what we ship. --> about_files/** license/** licenses/** META-INF/license/** - META-INF/*LICENSE*.txt - META-INF/*NOTICE*.txt - META-INF/LICENSE.md - META-INF/NOTICE.md - META-INF/NOTICE.markdown - LICENSE.txt - LICENSE.md - NOTICE.txt - NOTICE.md + META-INF/*LICENSE* + META-INF/*NOTICE* + LICENSE* + NOTICE* + COPYING*