fix(deps): bump org.apache.hadoop:hadoop-common from 3.4.3 to 3.5.0#6227
fix(deps): bump org.apache.hadoop:hadoop-common from 3.4.3 to 3.5.0#6227dependabot[bot] wants to merge 5 commits into
Conversation
Bumps org.apache.hadoop:hadoop-common from 3.4.3 to 3.5.0. --- updated-dependencies: - dependency-name: org.apache.hadoop:hadoop-common dependency-version: 3.5.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
dependabot
Bot
added
dependencies
Automated Reviewer SuggestionsBased on the
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 401 | 0.245 | 23,581/35,447/35,447 us | 🔴 +7.7% / 🔴 +134.5% |
| 🔴 | bs=100 sw=10 sl=64 | 810 | 0.494 | 121,767/149,399/149,399 us | 🔴 +9.9% / 🔴 +39.5% |
| ⚪ | bs=1000 sw=10 sl=64 | 928 | 0.566 | 1,077,049/1,119,457/1,119,457 us | ⚪ within ±5% / 🔴 -9.1% |
Baseline details
Latest main 27f145a from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 401 tuples/sec | 420 tuples/sec | 779.42 tuples/sec | -4.5% | -48.6% |
| bs=10 sw=10 sl=64 | MB/s | 0.245 MB/s | 0.256 MB/s | 0.476 MB/s | -4.3% | -48.5% |
| bs=10 sw=10 sl=64 | p50 | 23,581 us | 21,889 us | 12,550 us | +7.7% | +87.9% |
| bs=10 sw=10 sl=64 | p95 | 35,447 us | 33,650 us | 15,115 us | +5.3% | +134.5% |
| bs=10 sw=10 sl=64 | p99 | 35,447 us | 33,650 us | 18,652 us | +5.3% | +90.0% |
| bs=100 sw=10 sl=64 | throughput | 810 tuples/sec | 864 tuples/sec | 1,000 tuples/sec | -6.3% | -19.0% |
| bs=100 sw=10 sl=64 | MB/s | 0.494 MB/s | 0.527 MB/s | 0.61 MB/s | -6.3% | -19.1% |
| bs=100 sw=10 sl=64 | p50 | 121,767 us | 116,287 us | 99,872 us | +4.7% | +21.9% |
| bs=100 sw=10 sl=64 | p95 | 149,399 us | 135,903 us | 107,073 us | +9.9% | +39.5% |
| bs=100 sw=10 sl=64 | p99 | 149,399 us | 135,903 us | 114,377 us | +9.9% | +30.6% |
| bs=1000 sw=10 sl=64 | throughput | 928 tuples/sec | 929 tuples/sec | 1,020 tuples/sec | -0.1% | -9.0% |
| bs=1000 sw=10 sl=64 | MB/s | 0.566 MB/s | 0.567 MB/s | 0.622 MB/s | -0.2% | -9.1% |
| bs=1000 sw=10 sl=64 | p50 | 1,077,049 us | 1,078,201 us | 987,499 us | -0.1% | +9.1% |
| bs=1000 sw=10 sl=64 | p95 | 1,119,457 us | 1,112,744 us | 1,028,784 us | +0.6% | +8.8% |
| bs=1000 sw=10 sl=64 | p99 | 1,119,457 us | 1,112,744 us | 1,057,136 us | +0.6% | +5.9% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,498.61,200,128000,401,0.245,23581.06,35447.13,35447.13
1,100,10,64,20,2470.44,2000,1280000,810,0.494,121767.35,149398.60,149398.60
2,1000,10,64,20,21557.40,20000,12800000,928,0.566,1077049.29,1119456.66,1119456.66
There was a problem hiding this comment.
Pull request overview
This PR is a Dependabot-driven dependency update intended to bump Hadoop’s hadoop-common version used by the Amber (workflow execution) module.
Changes:
- Update
val hadoopVersioninamber/build.sbtfrom3.4.3to3.5.0for the Amber module’shadoop-commondependency.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ///////////////////////////////////////////////////////////////////////////// | ||
| // Hadoop related | ||
| val hadoopVersion = "3.4.3" | ||
| val hadoopVersion = "3.5.0" |
There was a problem hiding this comment.
Good catch — done. I bumped hadoop-common and hadoop-mapreduce-client-core in common/workflow-core/build.sbt to 3.5.0 as well, so every module now resolves a single Hadoop version and there's no cross-module eviction. hdfs-client / yarn-* follow transitively at 3.5.0, and the per-service LICENSE-binary/NOTICE files are re-synced.
…ommon 3.5.0 Hadoop 3.5.0 switched its web stack from Jersey 1.x (com.sun.jersey) to Jersey 2.46 (org.glassfish.jersey.*), which drops AbstractValueFactoryProvider. SBT's max-version eviction then replaced Dropwizard 1.3.23's Jersey 2.25.1 across the amber classpath (pulled transitively through the direct hadoop-common, workflow-core, and commons-vfs2 -> hadoop-hdfs-client), breaking dropwizard-auth's AuthValueFactoryProvider.Binder in ComputingUnitMaster. Pin the shared Jersey core/containers artifacts back to 2.25.1 in WorkflowExecutionService and drop Hadoop's 2.46-only jersey-hk2 module (jersey-inject did not exist in 2.25.1) so the classpath matches Dropwizard's expectations. Sync amber/LICENSE-binary-java and regenerate amber/NOTICE-binary for the bundled hadoop 3.5.0 jars and the glassfish JAXB runtime hadoop now pulls (jettison is no longer bundled).
|
Pushed a follow-up commit to get CI green — hadoop-common 3.5.0 swaps hadoop's web stack from Jersey 1.x ( The 2.46 leaked in through three transitive paths (direct Also synced |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6227 +/- ##
============================================
- Coverage 60.74% 60.48% -0.27%
+ Complexity 3368 3361 -7
============================================
Files 1135 1134 -1
Lines 44188 43928 -260
Branches 4830 4774 -56
============================================
- Hits 26844 26571 -273
- Misses 15889 15894 +5
- Partials 1455 1463 +8
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| of the Eclipse Distribution License v. 1.0 which is available at | ||
| http://www.eclipse.org/org/documents/edl-v10.php. | ||
|
|
||
| SPDX-License-Identifier: BSD-3-Clause |
There was a problem hiding this comment.
is this license good to use?
There was a problem hiding this comment.
This block came from Hadoop 3.5.0's new Jersey 2.x / glassfish JAXB web stack, which Texera doesn't use — we only pull hadoop as a filesystem client. I've excluded that whole stack, so the glassfish JAXB runtime + istack-commons jars (and this NOTICE block) are no longer bundled. For the record it's EDL 1.0 (== BSD-3-Clause, ASF Category A), so it would have been fine either way, but dropping it keeps the dist minimal.
| * License: SAX-PD | ||
| * Project: http://www.megginson.com/downloads/SAX/ | ||
| * Source: http://sourceforge.net/project/showfiles.php?group_id=29449 |
There was a problem hiding this comment.
Same as above — this was the JAXB RI's own upstream third-party list (Ant, testng, etc.), not jars we actually bundle. It's gone now that the glassfish JAXB runtime is excluded.
|
|
||
| Scala/Java jars: | ||
| - com.sun.activation.jakarta.activation-2.0.0.jar | ||
| - com.sun.istack.istack-commons-runtime-3.0.12.jar |
There was a problem hiding this comment.
I want to make sure this has a compatible license. presumably it is transitive from apache Hadoop, but good to check.
it has Eclipse Public License 2.0, is that good?
There was a problem hiding this comment.
This was jakarta.servlet.jsp-api (EPL 2.0), also part of Hadoop's unused web stack — now excluded, so it's no longer bundled. (EPL 2.0 is ASF Category B, fine for binary distribution, so it would have been acceptable, but we don't need it here.)
…de its web stack
Resolve the amber/LICENSE-binary-java conflict against main's jOOQ 3.19.36 /
r2dbc 1.0 changes by regenerating the license/notice files from a fresh dist.
Address review feedback:
- Copilot: bump hadoop-common + hadoop-mapreduce-client-core in common/workflow-core
to 3.5.0 so every module resolves a single Hadoop version (no cross-module eviction).
- Yicong-Huang: Hadoop 3.5.0's web stack (Jersey 2.x, glassfish JAXB runtime,
istack-commons, Jakarta JSP API) is unused by Texera, which only uses hadoop as a
filesystem client. Exclude it at every hadoop-bearing declaration (workflow-core +
amber hadoop-common/commons-vfs2), so those jars never enter any dist and Dropwizard
1.3.23's Jersey 2.25.1 is no longer evicted. This removes the need for the earlier
jersey dependencyOverride in WorkflowExecutionService (reverted).
Also fix the pre-existing no-op glassfish-jersey exclusion (organization
"org.glassfish.jersey" never matched the real org.glassfish.jersey.{core,containers,
inject} artifacts).
Re-sync LICENSE-binary + NOTICE-binary for every service that bundles hadoop
(amber, computing-unit-managing-service, file-service, workflow-compiling-service):
hadoop 3.4.3 -> 3.5.0, drop the dropped Jersey-1.x jersey-guice / servlet-api, add
failsafe / jettison / jakarta.servlet-api that hadoop 3.5.0 now pulls.
| // Texera uses hadoop only as a filesystem client, so none of this servlet/JAX-RS/JAXB | ||
| // web stack is needed; excluding it keeps Dropwizard's Jersey 2.25.1 from being evicted | ||
| // and keeps these jars out of every service dist. |
There was a problem hiding this comment.
Reworded here too. workflow-core is a plain library with no Dropwizard of its own, so I dropped the hardcoded versions and reframed it: the exclusion keeps the unused web-stack jars out of every downstream dist, and for the one downstream service still on a Jersey 2.x server (amber) it avoids evicting the Jersey its auth wiring needs. The org-string exactness note stays since that's version-independent.
| // Hadoop 3.5.0 moved its web stack to Jersey 2.x (org.glassfish.jersey.{core,containers, | ||
| // inject}) plus the glassfish JAXB runtime, istack-commons, and the Jakarta JSP API. Left | ||
| // in, Jersey 2.46 would evict Dropwizard 1.3.23's Jersey 2.25.1 (dropping the | ||
| // AbstractValueFactoryProvider that dropwizard-auth needs). Texera uses hadoop only as a | ||
| // filesystem client, so exclude the whole servlet/JAX-RS/JAXB web stack. |
There was a problem hiding this comment.
Reworded — but a clarification: this service (amber / WorkflowExecutionService) is actually still on Dropwizard 1.3.23 (dropwizardVersion a few lines up), which bundles Jersey 2.25.1. That's the whole reason the break is real here — Hadoop 3.5.0's Jersey 2.x evicts 2.25.1 and drops AbstractValueFactoryProvider. The 4.x / Jersey 3.x (jakarta) stack lives in the other services. I kept the version-specific detail since it's the actionable bit for future debugging, but scoped it explicitly to "this service" so it's clear it's not a repo-wide claim.
Reword the amber and workflow-core exclusion comments so the Dropwizard/Jersey rationale is module-scoped: amber runs the old Dropwizard 1.3.23 (Jersey 2.25.1) server that needs AbstractValueFactoryProvider, while workflow-core is a library and only references that downstream consumer. No functional change.
Bumps org.apache.hadoop:hadoop-common from 3.4.3 to 3.5.0.
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)