[700] Bundle license texts for all non-Apache-2.0 shaded dependencies

[rangareddy]

What

The bundled (shaded) jars list their third-party dependencies in META-INF/LICENSE (LICENSE-bundled), and the banner says "See licenses/ for text of these licenses" — but many bundled non-Apache-2.0 dependencies shipped no license text at all. This adds the missing texts and makes the tooling keep them complete, accurate and enforced.

Closes #700. Supersedes #866 (see below).

Background

The earlier tooling (generate_shaded_license_metadata.py + validate_shaded_license_coverage.sh) already made the coordinate listing in LICENSE-bundled complete and validated. The remaining gap (raised in the issue's follow-up comment) was the actual license text files under META-INF/licenses/.

Review of the first revision (thanks @vinishjail97) then showed the generator could not complete a run across all shade modules at all, and that several of its results were wrong rather than merely missing. Most of the diff below is the result of that round.

Changes

generate_shaded_license_metadata.py

• Regenerates each module's META-INF/licenses/ directory. For every non-Apache-2.0 bundled dependency it writes LICENSE-<artifactId>, sourced from a curated override (human-verified), then the dependency's own jar, and fails loudly if a dependency has neither.

• Resolves everything a run would write before writing any of it. A failure used to leave license texts deleted and LICENSE-bundled/NOTICE-bundled rewritten for the modules already processed, so the only recovery was to discard the whole run with git checkout. A failing run is now a no-op across all modules.

• Parses classified artifacts correctly. The version is the second-to-last field of a dependency-tree line, not parts[3], which is the classifier on a six-field line. This aborted the run on org.apache.hudi:hudi-io:jar:shaded:1.2.0 and, for the nine classified coordinates an override happened to cover, silently wrote a classifier where LICENSE-bundled wants a version (io.netty:netty-transport-native-epoll:linux-aarch_64). A tests classifier is now excluded alongside test-jar — org.apache.hbase:hbase-common:jar:tests was being treated as a bundled runtime dependency.

• Reads POMs that Maven accepts but ElementTree does not. Two distinct cases, both authentic artifacts whose SHA-1s match Maven Central:

  • hadoop-project-3.1.0.pom contains <Xlint:-unchecked/>, an undeclared namespace prefix with a hyphen-leading local name. expat rejects the whole file; it is the parent of the hadoop 3.1.0 yarn artifacts.
  • hadoop-yarn-client-2.7.1.pom opens <project xmlns:pom="…">, binding the Maven namespace to a prefix, so every element is unqualified and the namespaced XPaths matched nothing.

  Lookups now match on local name, with a text-scan fallback when a POM will not parse.

• Drops 18 license-family overrides that only existed because of those parsing bugs, including org.apache.hadoop. Also drops xml-apis, which claimed Apache Software License 1.1 while both its POM and its jar's own license/LICENSE carry the Apache 2.0 text.

• Checks the license text against the family resolved from the POM. Nothing previously tied the two together, so a jar shipping the wrong license file was attributed wrongly and silently. This is what caught the junit error below.

• Derives the bundled set from the runtime tree intersected with the shade <includes>, so the metadata describes what is inside the jar rather than what is on the classpath.

• Decodes embedded license/NOTICE files as strict UTF-8 instead of substituting U+FFFD for bytes it cannot read, which silently mangled older latin-1 texts.

• Refuses to run when a shade module has no dependency tree, rather than regenerating the rest and producing a diff that looks like a full regeneration but is not. xtable-utilities is skipped explicitly, in step with the validator.

validate_shaded_license_coverage.sh

• Fails when a non-Apache-2.0 dependency in LICENSE-bundled has no META-INF/licenses/LICENSE-<artifactId>, and in the reverse direction, when a license text has no bundled dependency behind it. Checking one direction only is what allowed orphaned texts to accumulate — this is Orphaned license files in bundled jars #865, so [865] Remove orphaned license files from bundled jars #866 can be closed as superseded.

• Builds its module list with find/grep instead of ripgrep. This was a fail-open, and a worse one than I first thought: mvn-license-check.yml never installs ripgrep and the GitHub ubuntu-latest runner does not ship it, so this gate has never actually run in CI. The call sits in a process substitution where errexit cannot see it fail, so the module list came back empty and the script printed "No modules with maven-shade-plugin were found" and exited 0. Every License Check run on main reports success without validating anything — from the latest run on main:

  release/scripts/validate_shaded_license_coverage.sh: line 92: rg: command not found
  No modules with maven-shade-plugin were found.
  

  An empty module list is now a failure.

• Downgrades the includes-vs-tree check to a warning for xtable-hive-metastore only, via an explicit KNOWN_INCLUDES_DRIFT list, because fixing the fail-open above immediately surfaced a pre-existing drift in that module. See xtable-hive-metastore shade <includes> has drifted from its runtime dependency tree #880 and the open items below. Every other check, for every module, stays blocking — verified by planting a bogus <include> in xtable-aws, which still fails.

• Parses LICENSE-bundled by calling the generator (--non-apache-artifact-ids) rather than reimplementing the parse in awk. The two implementations detected a family heading differently and could disagree about which dependencies need a text.

junit is corrected from CPL 1.0 to EPL 1.0

The curated override shipped the Common Public License, but junit 4.12 — the version bundled here — is EPL 1.0, and its jar carries the correct EPL text under the non-standard name LICENSE-junit.txt. The wrong text was being shipped in xtable-hive-metastore. The override is deleted, a last-resort jar-filename fallback picks up the correct text, and the committed LICENSE-junit is corrected.

Retraction: an earlier revision of this PR (and its commit message) cited junit as the case where a jar bundles someone else's license text, "so the override must win". That is true of junit 4.11, which ships Hamcrest's BSD text — it is not true of 4.12, which is what we bundle. The claim was wrong, not merely outdated, and the commit message has been rewritten so it does not outlive the PR.

Two other claims from the first revision no longer hold and have been removed above: that org.apache.hadoop needs to be treated as Apache-2.0 by override (it resolves from its own POM chain once the parser is fixed), and that license texts are keyed off the committed coordinate list rather than a resolved tree (they now come from the family groups the generator already has in memory, which removed a write-then-parse round-trip).

Result

Every non-Apache-2.0 bundled dependency has a license text: xtable-aws 5/5, xtable-hive-metastore 21/21, xtable-hudi-support-extensions 3/3.

The hive-metastore count is 21 rather than the 29 quoted in the first revision because the bundled set is now the runtime tree ∩ shade <includes>; the other eight are runtime dependencies that maven-shade is not told to include, so they are not in the artifact and must not have texts generated for them.

Testing

Gate	Result
./mvnw apache-rat:check -B	pass
release/scripts/validate_shaded_license_coverage.sh	exit 0; new checks pass on all three published shaded modules, with the pre-existing #880 drift reported as a WARN
python3 release/scripts/generate_shaded_license_metadata.py	completes for all three published modules except javolution, see below. It could not complete at all before this round.
bash -n / python3 -m py_compile	pass

Also verified by hand:

• The orphan check fires: planting LICENSE-not-a-real-dep produces FAIL … orphaned META-INF/licenses/ text.
• The atomic behaviour holds: every failing run left git status showing no metadata changes.
• The KNOWN_INCLUDES_DRIFT entry is scoped: planting a bogus <include> in xtable-aws still produces FAIL, so only the named module's includes check is downgraded.
• The validator now runs to completion without ripgrep on PATH, which is the condition CI actually runs under.
• Dependency-tree generation is reproducible — -pl xtable-hive-metastore -am and the four-module invocation produce byte-identical trees.
• License texts for boilerplate licenses were extracted verbatim from files already present in Maven artifacts, never hand-typed.

Two open items

1. javolution:javolution:5.5.1 needs a decision. Its POM declares BSD License (url http://javolution.org/LICENSE.txt, long dead); the curated text in the repo is the MIT license with a 2017 copyright, for a 2010 artifact; the jar carries no license file. The text predates this PR — it arrived with #691 and this PR only relocated it into license_overrides/. Rather than guess, the generator refuses on it, which is what the new family check is for. Options: confirm the MIT text's provenance and pin the family with a commented override; source javolution 5.5.1's real BSD text from an archived copy; or take it to legal-discuss.

2. xtable-hive-metastore shade <includes> has drifted — pre-existing, now tracked in #880. Against the current runtime tree, 25 includes name dependencies that are no longer resolved (ant:ant, asm:*, javax.mail:mail, oro:oro, tomcat:jasper-*, …) and 97 runtime dependencies are missing from the list (all of jetty 9.x, jersey 2.x, hk2, calcite, the hbase-* server modules, …). The list reads as written for a Hive 2.x dependency set and never updated for 3.1.3.

This PR touches no pom.xml, and running main's own unmodified validator with ripgrep available reproduces the identical failure — so the drift is pre-existing and fixing the fail-open is simply what made it visible. It is handled here with the one-module KNOWN_INCLUDES_DRIFT entry described above rather than by changing the poms, because the two candidate fixes (grow the include list, or narrow those dependencies to provided) have materially different consequences for what the published artifact bundles. That is a release decision; #880 has the full evidence and acceptance criteria.

This is also why the coordinate lists in LICENSE-bundled/NOTICE-bundled are left unchanged here: regenerating them would drop those 25 dead entries and break the separate "every shade include is represented in bundled license metadata" check, which currently passes only because the committed metadata predates the drift.

Scope

• Coordinate lists in LICENSE-bundled/NOTICE-bundled are unchanged (see open item 2).
• xtable-utilities (unpublished, already skipped by the validator) remains out of scope, now via an explicit skip list rather than by relying on a dependency tree being absent. Covering it would need ~32 curated texts plus a family decision for org.openjdk.jol:jol-core, which is GPLv2-with-Classpath-Exception and deliberately shaded. The GPLv2+CPE family mapping is added so the module can be un-skipped later without hitting an unmapped family.
• Not yet rebased onto current main (8 commits ahead, none touching these files).

[vinishjail97]

@rangareddy I have ran the scripts in my local and it was breaking for few dependencies. Can you address the comments? I will take a look tomorrow again and merge for 0.4.0-incubating rc2.

[rangareddy]

@vinishjail97 all ten threads are answered inline and the branch is force-pushed (efbffc9 → 2f9ed96). The PR description is rewritten, since most of it no longer matched the diff. Summary of the round:

Your two direct questions

• Is the hadoop override required? No — you were right, it is deleted. It took closing two POM-parsing gaps to make that true, and neither was a ~/.m2 miss: hadoop-project-3.1.0.pom contains <Xlint:-unchecked/>, which expat rejects outright, and hadoop-yarn-client-2.7.1.pom binds the Maven namespace to a prefix so every element is unqualified and the namespaced XPaths matched nothing. Both POMs are authentic — their SHA-1s match Central. Auditing the rest of the table on top of that removed 18 overrides in total.
• Will errors="strict" break this today? It would have, on javax.activation:activation:1.1 and javax.mail:mail:1.4.1 (both latin-1 META-INF/LICENSE.txt), out of 363 embedded entries. After the classified-artifact fix neither is bundled any more, so the current run has zero decode failures. notice_text_for had none, so the pre-existing errors="replace" there was harmless — it is strict now too.

Things the review surfaced that turned out to be bugs, not gaps

• The parts[3] bug was not only aborting the run. Nine classified coordinates were being written into LICENSE-bundled with the classifier in place of the version, and six ARTIFACT_OVERRIDES existed solely to stop the resulting raise. hbase-common:jar:tests was also being counted as a bundled runtime dependency.
• Your family/text assertion caught a real mis-attribution: junit was shipping the Common Public License, but 4.12 is EPL 1.0. Your note about 4.11 bundling Hamcrest's BSD text is correct, but 4.12 ships its own correct EPL text under LICENSE-junit.txt. So the override was wrong; it is removed and the committed text corrected. That also means the original commit message's justification for override-first was wrong for the version we bundle, so I rewrote it rather than let it merge.

One I found while running your scripts, unrelated to any thread

validate_shaded_license_coverage.sh was a fail-open. It builds the module list with rg, which mvn-license-check.yml never installs; on a machine without ripgrep the list came back empty and the script printed "No modules with maven-shade-plugin were found" and exited 0. The whole release license gate passed while checking nothing — which is also why your local run and mine could differ. Now find/grep, and an empty list fails.

Two things I deliberately did not decide

1. javolution:javolution:5.5.1 — POM says BSD License, the curated text is MIT with a 2017 copyright for a 2010 artifact, and the jar has no license file. It predates this PR (came in with [678] Add LICENSE for xtable-hive-metastore module #691). Writing a BSD text with a copyright line I cannot verify would be fabricating a legal document, so the generator refuses on it. Needs your call — details in the inline thread.
2. xtable-hive-metastore's shade <includes> has drifted and the validator's pre-existing includes-vs-tree check fails on this branch and on main (this PR touches no pom.xml): 25 includes name dependencies no longer in the tree, 97 runtime dependencies are missing from the list. It reads like a list written for Hive 2.x and never updated for 3.1.3. This is why I left the coordinate lists unchanged — regenerating now would flip a currently-passing check to failing, because the committed metadata still carries those 25 dead entries. Happy to file it as its own issue and do the regeneration as a follow-up.

Also: I'd flag javax.servlet and javax.servlet.jsp, whose overrides claim Apache-2.0 while their POMs say CDDL + GPLv2 w/ CPE. That downgrades Category B to A and suppresses license texts those dependencies would otherwise ship. I left them alone since it is a legal call, but it deserves a second opinion.

Not rebased onto main yet (8 commits, none touching these files) — say the word and I'll rebase and re-run before you look.

[rangareddy]

@vinishjail97 the License Check failure on the previous push was the new gate working, and it turned up something worth knowing before rc2. Pushed a611ada; CI should be green now.

The License Check has never actually run in CI.

The script builds its module list with rg, which mvn-license-check.yml does not install and which the GitHub ubuntu-latest runner does not ship. The call sits inside a process substitution, so set -o errexit cannot see it fail; the list came back empty and the script hit its exit 0 path. From the most recent License Check on main:

release/scripts/validate_shaded_license_coverage.sh: line 92: rg: command not found
No modules with maven-shade-plugin were found.

Every one of the last ten runs on main is green for that reason. I need to correct myself here — in my earlier reply I said the runner happened to have ripgrep and the gate "works in CI only by luck". That was wrong: it has never run.

What that was hiding

Replacing rg with find/grep made the gate run for the first time, and it immediately failed on xtable-hive-metastore's existing <includes>-vs-runtime-tree assertion: 25 includes name dependencies that are no longer resolved (ant:ant, asm:*, javax.mail:mail, oro:oro, tomcat:jasper-*, …) and 97 runtime dependencies are missing from the list (all of jetty 9.x, jersey 2.x, hk2, calcite, the hbase-* server modules, …). It reads as a list written for a Hive 2.x dependency set and never updated for 3.1.3.

To confirm this is not something I introduced: this PR touches no pom.xml, and running main's own unmodified validator with ripgrep available reproduces the identical failure.

How I have handled it, and why not more

Filed as #880 with the full evidence, and this PR adds an explicit one-module KNOWN_INCLUDES_DRIFT entry that downgrades only the includes-vs-tree comparison for only xtable-hive-metastore to a loud WARN printing the full diff and referencing #880. Every other check, for every module, stays blocking — I verified the scoping by planting a bogus <include> in xtable-aws, which still fails.

I did not fix the drift here because the two candidate fixes have materially different consequences for the published artifact:

1. Grow the include list to match the tree, which is what AGENTS.md currently prescribes. That would bundle all of jetty, jersey, hk2, calcite and the hbase server modules into the shaded jar, changing its contents and size significantly, plus ~15 more bundled license texts.
2. Narrow those dependencies to provided, if the expectation is that a Hive Metastore deployment supplies them. In that case the include list is closer to correct than the tree is, and the validator's "must exactly match" contract itself deserves another look.

That is a release/design call rather than a tooling fix, so it seemed wrong to decide it inside a license-text PR — and worse to leave the gate vacuous in order to avoid it. Happy to do either once you or dev@ pick one.

Two things you may want to consider separately, given the gate has not been effective on any branch:

• whether mvn-license-check.yml should also run on branch-0.4;
• whether 0.4.0-incubating rc1 was validated by this gate at all.