apache · ctrlaltdilj · Sep 7, 2026
diff --git a/docs/content.zh/docs/deployment/filesystems/s3.md b/docs/content.zh/docs/deployment/filesystems/s3.md
@@ -220,6 +220,33 @@ cp ./opt/flink-s3-fs-hadoop-{{< version >}}.jar ./plugins/s3-fs-hadoop/
 
 The [common configuration](#common-configuration) options apply. In addition, [Hadoop's s3a configuration keys](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A) are supported. Hadoop configuration keys are automatically translated — for example, `fs.s3a.connection.maximum` becomes `s3.connection.maximum`.
 
+**Input stream type:** hadoop-aws 3.4.3 changes its default input stream to the [S3 Analytics Accelerator](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/stream_architecture.html) (`fs.s3a.input.stream.type: analytics`), which alters read patterns, per-stream memory usage, and the volume of S3 GET requests. To keep upgrades behavior-preserving, Flink pins the classic input stream by default. To opt in to the analytics stream, set `s3.input.stream.type: analytics`; setting the legacy `s3.prefetch.enabled: true` toggle is also respected and selects Hadoop's prefetching stream.
+
+#### Changed defaults from the Hadoop 3.4 upgrade
+
+The Hadoop S3 FileSystem passes Hadoop's s3a defaults through unchanged. Hadoop 3.4 retuned several of them as part of its migration to the AWS SDK v2; Flink deliberately keeps the new values (the input stream type above is the only pinned exception), so that the client behaves the way the Hadoop community tunes and tests it. The table below lists the changed defaults and why upstream changed them. Any key can be restored through the `s3.` passthrough shown underneath.
+
+| Key | Hadoop 3.3 default | Hadoop 3.4 default | Why it changed |
+|-----|--------------------|--------------------|----------------|
+| `fs.s3a.connection.maximum` | 96 | 500 | HTTP connection pool resized for the AWS SDK v2 client and the larger thread pool, avoiding pool starvation under parallel uploads. |
+| `fs.s3a.threads.max` | 10 | 96 | Sized to match the larger connection pool for parallel upload and copy operations. |
+| `fs.s3a.attempts.maximum` | 10 | 5 | Retries now use the SDK v2 adaptive retry strategy (client-side throttling-aware rate limiting), so fewer, smarter attempts replace many blind ones. |
+| `fs.s3a.connection.establish.timeout` | 50s | 30s | Fail over to healthy endpoints sooner. |
+| `fs.s3a.connection.acquisition.timeout` | n/a | 60s | New in 3.4: bounds the wait for a pooled connection under load instead of blocking indefinitely. |
+| `fs.s3a.connection.ttl` | n/a | 5m | New in 3.4: recycles pooled connections so load-balancer and DNS changes are picked up. |
+| `fs.s3a.directory.marker.retention` | `delete` | `keep` | Keeping zero-byte directory markers saves one DELETE request per object created under a path and avoids delete tombstones on versioned buckets. Markers are invisible through the filesystem API; restore `delete` only if the bucket is shared with Hadoop clients older than 3.1 or with tools that list raw objects. |
+| `fs.s3a.checksum.generation` | n/a | `false` | New in 3.4: disabled by default to stay compatible with S3-compatible stores that reject the SDK v2 default integrity checksums. |
+| `fs.s3a.input.stream.type` | `classic` | `analytics` upstream, **Flink pins `classic`** | See the input stream type note above. |
+
+To restore the previous value of any of these keys (normally unnecessary), configure the corresponding `s3.` key, for example:
+
+```yaml
+s3.connection.maximum: 96
+s3.threads.max: 10
+s3.attempts.maximum: 10
+s3.directory.marker.retention: delete
+```
+
 ---
 
 ## Using Multiple S3 Implementations

diff --git a/docs/content/docs/deployment/filesystems/s3.md b/docs/content/docs/deployment/filesystems/s3.md
@@ -228,6 +228,33 @@ cp ./opt/flink-s3-fs-hadoop-{{< version >}}.jar ./plugins/s3-fs-hadoop/
 
 The [common configuration](#common-configuration) options apply. In addition, [Hadoop's s3a configuration keys](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A) are supported. Hadoop configuration keys are automatically translated — for example, `fs.s3a.connection.maximum` becomes `s3.connection.maximum`.
 
+**Input stream type:** hadoop-aws 3.4.3 changes its default input stream to the [S3 Analytics Accelerator](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/stream_architecture.html) (`fs.s3a.input.stream.type: analytics`), which alters read patterns, per-stream memory usage, and the volume of S3 GET requests. To keep upgrades behavior-preserving, Flink pins the classic input stream by default. To opt in to the analytics stream, set `s3.input.stream.type: analytics`; setting the legacy `s3.prefetch.enabled: true` toggle is also respected and selects Hadoop's prefetching stream.
+
+#### Changed defaults from the Hadoop 3.4 upgrade
+
+The Hadoop S3 FileSystem passes Hadoop's s3a defaults through unchanged. Hadoop 3.4 retuned several of them as part of its migration to the AWS SDK v2; Flink deliberately keeps the new values (the input stream type above is the only pinned exception), so that the client behaves the way the Hadoop community tunes and tests it. The table below lists the changed defaults and why upstream changed them. Any key can be restored through the `s3.` passthrough shown underneath.
+
+| Key | Hadoop 3.3 default | Hadoop 3.4 default | Why it changed |
+|-----|--------------------|--------------------|----------------|
+| `fs.s3a.connection.maximum` | 96 | 500 | HTTP connection pool resized for the AWS SDK v2 client and the larger thread pool, avoiding pool starvation under parallel uploads. |
+| `fs.s3a.threads.max` | 10 | 96 | Sized to match the larger connection pool for parallel upload and copy operations. |
+| `fs.s3a.attempts.maximum` | 10 | 5 | Retries now use the SDK v2 adaptive retry strategy (client-side throttling-aware rate limiting), so fewer, smarter attempts replace many blind ones. |
+| `fs.s3a.connection.establish.timeout` | 50s | 30s | Fail over to healthy endpoints sooner. |
+| `fs.s3a.connection.acquisition.timeout` | n/a | 60s | New in 3.4: bounds the wait for a pooled connection under load instead of blocking indefinitely. |
+| `fs.s3a.connection.ttl` | n/a | 5m | New in 3.4: recycles pooled connections so load-balancer and DNS changes are picked up. |
+| `fs.s3a.directory.marker.retention` | `delete` | `keep` | Keeping zero-byte directory markers saves one DELETE request per object created under a path and avoids delete tombstones on versioned buckets. Markers are invisible through the filesystem API; restore `delete` only if the bucket is shared with Hadoop clients older than 3.1 or with tools that list raw objects. |
+| `fs.s3a.checksum.generation` | n/a | `false` | New in 3.4: disabled by default to stay compatible with S3-compatible stores that reject the SDK v2 default integrity checksums. |
+| `fs.s3a.input.stream.type` | `classic` | `analytics` upstream, **Flink pins `classic`** | See the input stream type note above. |
+
+To restore the previous value of any of these keys (normally unnecessary), configure the corresponding `s3.` key, for example:
+
+```yaml
+s3.connection.maximum: 96
+s3.threads.max: 10
+s3.attempts.maximum: 10
+s3.directory.marker.retention: delete
+```
+
 ---
 
 ## Using Multiple S3 Implementations

diff --git a/flink-filesystems/flink-azure-fs-hadoop/src/main/resources/META-INF/NOTICE b/flink-filesystems/flink-azure-fs-hadoop/src/main/resources/META-INF/NOTICE
@@ -6,19 +6,15 @@ The Apache Software Foundation (http://www.apache.org/).
 
 This project bundles the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
 
-- com.fasterxml.jackson.core:jackson-core:2.22.2
-- com.google.guava:guava:20.0
 - commons-codec:commons-codec:1.15
 - commons-logging:commons-logging:1.1.3
-- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
-- org.apache.hadoop:hadoop-azure:3.3.4
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.5.0
+- org.apache.hadoop:hadoop-azure:3.4.3
 - org.apache.httpcomponents:httpclient:4.5.13
 - org.apache.httpcomponents:httpcore:4.4.14
-- org.codehaus.jackson:jackson-core-asl:1.9.13
-- org.codehaus.jackson:jackson-mapper-asl:1.9.14.jdk17-redhat-00001
 - org.eclipse.jetty:jetty-util-ajax:9.3.24.v20180605
 - org.eclipse.jetty:jetty-util:9.3.24.v20180605
-- org.wildfly.openssl:wildfly-openssl:1.0.7.Final
+- org.wildfly.openssl:wildfly-openssl:2.2.5.Final
 
 This project bundles the following dependencies under the MIT (https://opensource.org/licenses/MIT)
 
@@ -29,4 +25,4 @@ The bundled Apache Hadoop Relocated (Shaded) Third-party Miscellaneous Libs
 org.apache.hadoop.thirdparty:hadoop-shaded-guava dependency bundles the following dependencies under
 the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
 
-- com.google.guava:guava:30.1.1-jre
+- com.google.guava:guava:33.4.8-jre
diff --git a/flink-filesystems/flink-fs-hadoop-shaded/pom.xml b/flink-filesystems/flink-fs-hadoop-shaded/pom.xml
@@ -43,6 +43,12 @@ under the License.
 				<artifactId>commons-beanutils</artifactId>
 				<version>1.9.4</version>
 			</dependency>
+			<dependency>
+				<!-- Align with the metrics-core version bundled by the S3 filesystems -->
+				<groupId>io.dropwizard.metrics</groupId>
+				<artifactId>metrics-core</artifactId>
+				<version>3.2.5</version>
+			</dependency>
 		</dependencies>
 	</dependencyManagement>
 
@@ -78,6 +84,19 @@ under the License.
 					<groupId>com.sun.jersey</groupId>
 					<artifactId>jersey-server</artifactId>
 				</exclusion>
+				<!-- Pulled in by hadoop-common 3.4.x via com.github.pjfanning:jersey-json,
+				     ships com/sun/jersey/json/** classes. -->
+				<exclusion>
+					<groupId>com.github.pjfanning</groupId>
+					<artifactId>jersey-json</artifactId>
+				</exclusion>
+				<!-- Transitive of jersey-json above; ships com/sun/xml/bind/** classes
+				     (CDDL+GPLv2-with-classpath-exception). The JDK provides JAXB RI; we
+				     only ship the API as a Java 11+ multi-release resource. -->
+				<exclusion>
+					<groupId>com.sun.xml.bind</groupId>
+					<artifactId>jaxb-impl</artifactId>
+				</exclusion>
 				<exclusion>
 					<groupId>org.apache.avro</groupId>
 					<artifactId>avro</artifactId>
@@ -178,6 +197,21 @@ under the License.
 					<groupId>org.slf4j</groupId>
 					<artifactId>slf4j-reload4j</artifactId>
 				</exclusion>
+				<!-- Hadoop 3.4.x declares Netty as a compile-time dep of hadoop-common,
+				     but hadoop-common's compiled bytecode contains zero io.netty.*
+				     references (verified empirically). The dep exists for sibling
+				     Hadoop modules (hadoop-hdfs-client, hadoop-kms-server) that this
+				     shaded jar does NOT bundle. Exclude both transitive roots
+				     (netty-handler and netty-transport-native-epoll) to drop the
+				     entire 7-artifact Netty tree. -->
+				<exclusion>
+					<groupId>io.netty</groupId>
+					<artifactId>netty-handler</artifactId>
+				</exclusion>
+				<exclusion>
+					<groupId>io.netty</groupId>
+					<artifactId>netty-transport-native-epoll</artifactId>
+				</exclusion>
 			</exclusions>
 		</dependency>
 	</dependencies>
@@ -296,13 +330,6 @@ under the License.
 									</excludes>
 								</filter>
 							</filters>
-							<transformers combine.children="append">
-								<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
-									<manifestEntries>
-										<Multi-Release>true</Multi-Release>
-									</manifestEntries>
-								</transformer>
-							</transformers>
 						</configuration>
 					</execution>
 				</executions>

diff --git a/flink-filesystems/flink-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE b/flink-filesystems/flink-fs-hadoop-shaded/src/main/resources/META-INF/NOTICE
@@ -9,39 +9,46 @@ This project bundles the following dependencies under the Apache Software Licens
 - com.fasterxml.jackson.core:jackson-annotations:2.22
 - com.fasterxml.jackson.core:jackson-core:2.22.2
 - com.fasterxml.jackson.core:jackson-databind:2.22.2
-- com.fasterxml.woodstox:woodstox-core:5.3.0
-- com.google.guava:failureaccess:1.0
-- com.google.guava:guava:27.0-jre
+- com.fasterxml.woodstox:woodstox-core:5.4.0
+- com.google.guava:failureaccess:1.0.1
+- com.google.guava:guava:32.0.1-jre
 - com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
-- com.google.j2objc:j2objc-annotations:1.1
-- commons-beanutils:commons-beanutils:1.9.4
-- commons-collections:commons-collections:3.2.2
+- com.google.j2objc:j2objc-annotations:2.8
 - commons-io:commons-io:2.15.1
 - commons-logging:commons-logging:1.1.3
+- io.dropwizard.metrics:metrics-core:3.2.5
+- org.apache.commons:commons-collections4:4.4
 - org.apache.commons:commons-compress:1.26.0
-- org.apache.commons:commons-configuration2:2.1.1
+- org.apache.commons:commons-configuration2:2.10.1
 - org.apache.commons:commons-lang3:3.18.0
 - org.apache.commons:commons-text:1.10.0
-- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.1.1
-- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:1.1.1
-- org.apache.hadoop:hadoop-annotations:3.3.4
-- org.apache.hadoop:hadoop-auth:3.3.4
-- org.apache.hadoop:hadoop-common:3.3.4
-- org.apache.kerby:kerb-core:1.0.1
-- org.apache.kerby:kerby-asn1:1.0.1
-- org.apache.kerby:kerby-pkix:1.0.1
-- org.apache.kerby:kerby-util:1.0.1
+- org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.5.0
+- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.5.0
+- org.apache.hadoop:hadoop-annotations:3.4.3
+- org.apache.hadoop:hadoop-auth:3.4.3
+- org.apache.hadoop:hadoop-common:3.4.3
+- org.apache.kerby:kerb-core:2.0.3
+- org.apache.kerby:kerb-crypto:2.0.3
+- org.apache.kerby:kerb-util:2.0.3
+- org.apache.kerby:kerby-asn1:2.0.3
+- org.apache.kerby:kerby-config:2.0.3
+- org.apache.kerby:kerby-pkix:2.0.3
+- org.apache.kerby:kerby-util:2.0.3
+- org.codehaus.jettison:jettison:1.5.4
 - org.xerial.snappy:snappy-java:1.1.10.7
 
 This project bundles the following dependencies under the MIT (https://opensource.org/licenses/MIT)
 
-- org.checkerframework:checker-qual:2.5.2
-- org.codehaus.mojo:animal-sniffer-annotations:1.17
+- org.checkerframework:checker-qual:3.33.0
 
 This project bundles the following dependencies under BSD-2 License (https://opensource.org/licenses/BSD-2-Clause).
 See bundled license files for details.
 
-- dnsjava:dnsjava:2.1.7
+- dnsjava:dnsjava:3.6.1
+
+This project bundles the following dependencies under the Bouncy Castle License (https://www.bouncycastle.org/licence.html)
+
+- org.bouncycastle:bcprov-jdk18on:1.82
 
 This project bundles the following dependencies under the Go License (https://golang.org/LICENSE).
 See bundled license files for details.
@@ -52,3 +59,15 @@ This project bundles the following dependencies under BSD License (https://opens
 See bundled license files for details.
 
 - org.codehaus.woodstox:stax2-api:4.2.1 (https://github.com/FasterXML/stax2-api/tree/stax2-api-4.2.1)
+
+The bundled Apache Hadoop Relocated (Shaded) Third-party Miscellaneous Libs
+org.apache.hadoop.thirdparty:hadoop-shaded-guava dependency bundles the following dependencies under
+the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
+
+- com.google.guava:guava:33.4.8-jre
+
+The bundled Apache Hadoop Relocated (Shaded) Third-party Miscellaneous Libs
+org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25 dependency bundles the following dependencies under
+the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
+
+- com.google.protobuf:protobuf-java:3.25.5
diff --git a/flink-filesystems/flink-oss-fs-hadoop/src/main/resources/META-INF/NOTICE b/flink-filesystems/flink-oss-fs-hadoop/src/main/resources/META-INF/NOTICE
@@ -16,7 +16,7 @@ This project bundles the following dependencies under the Apache Software Licens
 - io.opentracing:opentracing-api:0.33.0
 - io.opentracing:opentracing-noop:0.33.0
 - io.opentracing:opentracing-util:0.33.0
-- org.apache.hadoop:hadoop-aliyun:3.3.4
+- org.apache.hadoop:hadoop-aliyun:3.4.3
 - org.apache.httpcomponents:httpclient:4.5.13
 - org.apache.httpcomponents:httpcore:4.4.14
 - org.codehaus.jettison:jettison:1.5.4

diff --git a/flink-filesystems/flink-s3-fs-base/pom.xml b/flink-filesystems/flink-s3-fs-base/pom.xml
@@ -32,6 +32,10 @@ under the License.
 
 	<properties>
 		<fs.s3.aws.version>1.12.779</fs.s3.aws.version>
+		<!-- Held above Hadoop 3.4.3's bundled aws-java-sdk-v2 because hadoop-aws 3.4.3's
+		     ChecksumSupport.<clinit> references CRC64_NVME, which was added to
+		     software.amazon.awssdk:checksums-spi in 2.30.x. Aligned with flink-s3-fs-native. -->
+		<fs.s3.aws.sdk2.version>2.44.4</fs.s3.aws.sdk2.version>
 		<japicmp.skip>true</japicmp.skip>
 		<surefire.module.config> <!--
 			S5CmdOnSeaweedFsITCase uses ArraysAsListSerializer indirectly
@@ -87,6 +91,19 @@ under the License.
 					<groupId>com.sun.jersey</groupId>
 					<artifactId>jersey-server</artifactId>
 				</exclusion>
+				<!-- Pulled in by hadoop-common 3.4.x via com.github.pjfanning:jersey-json,
+				     ships com/sun/jersey/json/** classes. -->
+				<exclusion>
+					<groupId>com.github.pjfanning</groupId>
+					<artifactId>jersey-json</artifactId>
+				</exclusion>
+				<!-- Transitive of jersey-json above; ships com/sun/xml/bind/** classes
+				     (CDDL+GPLv2-with-classpath-exception). The JDK provides JAXB RI; we
+				     only ship the API as a Java 11+ multi-release resource. -->
+				<exclusion>
+					<groupId>com.sun.xml.bind</groupId>
+					<artifactId>jaxb-impl</artifactId>
+				</exclusion>
 				<exclusion>
 					<groupId>org.apache.avro</groupId>
 					<artifactId>avro</artifactId>
@@ -234,6 +251,11 @@ under the License.
 					<groupId>com.amazonaws</groupId>
 					<artifactId>aws-java-sdk-bundle</artifactId>
 				</exclusion>
+				<!-- Exclude SDK V2 bundle - use individual modules instead to reduce JAR size -->
+				<exclusion>
+					<groupId>software.amazon.awssdk</groupId>
+					<artifactId>bundle</artifactId>
+				</exclusion>
 				<exclusion>
 					<groupId>ch.qos.reload4j</groupId>
 					<artifactId>reload4j</artifactId>
@@ -245,6 +267,38 @@ under the License.
 			</exclusions>
 		</dependency>
 
+		<!-- AWS SDK V2 individual modules (instead of bundle to reduce JAR size from 611MB to ~13MB) -->
+		<dependency>
+			<groupId>software.amazon.awssdk</groupId>
+			<artifactId>s3</artifactId>
+			<version>${fs.s3.aws.sdk2.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>software.amazon.awssdk</groupId>
+			<artifactId>sts</artifactId>
+			<version>${fs.s3.aws.sdk2.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>software.amazon.awssdk</groupId>
+			<artifactId>kms</artifactId>
+			<version>${fs.s3.aws.sdk2.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>software.amazon.awssdk</groupId>
+			<artifactId>sso</artifactId>
+			<version>${fs.s3.aws.sdk2.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>software.amazon.awssdk</groupId>
+			<artifactId>auth</artifactId>
+			<version>${fs.s3.aws.sdk2.version}</version>
+		</dependency>
+		<dependency>
+			<groupId>software.amazon.awssdk</groupId>
+			<artifactId>s3-transfer-manager</artifactId>
+			<version>${fs.s3.aws.sdk2.version}</version>
+		</dependency>
+
 		<!-- flink-runtime and curator dependencies are needed for the HAJobRunOnSeaweedFsS3StoreITCase -->
 		<dependency>
 			<groupId>org.apache.flink</groupId>

diff --git a/...k-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/AbstractS3FileSystemFactory.java b/...k-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/AbstractS3FileSystemFactory.java
@@ -186,7 +186,8 @@ public FileSystem create(URI fsUri) throws IOException {
             // create the Hadoop FileSystem
             org.apache.hadoop.conf.Configuration hadoopConfig =
                     hadoopConfigLoader.getOrLoadHadoopConfig();
-            AbstractS3DelegationTokenReceiver.updateHadoopConfig(hadoopConfig);
+            applyHadoopConfigDefaults(hadoopConfig);
+            updateDelegationTokenConfig(hadoopConfig);
             org.apache.hadoop.fs.FileSystem fs = createHadoopFileSystem();
             fs.initialize(getInitURI(fsUri, hadoopConfig), hadoopConfig);
 
@@ -258,4 +259,21 @@ protected FileSystem createFlinkFileSystem(
 
     @Nullable
     protected abstract S3AccessHelper getS3AccessHelper(org.apache.hadoop.fs.FileSystem fs);
+
+    /**
+     * Applies Flink-side defaults to the loaded Hadoop configuration before the file system is
+     * initialized. Subclasses override this to pin Hadoop defaults that would otherwise change
+     * behavior across Hadoop upgrades. Implementations must only set keys the user has not
+     * configured and must stay idempotent, because the loaded Hadoop configuration is cached across
+     * {@link #create(URI)} calls.
+     */
+    protected void applyHadoopConfigDefaults(org.apache.hadoop.conf.Configuration hadoopConfig) {}
+
+    /**
+     * Updates the Hadoop configuration with delegation token credentials provider. Subclasses can
+     * override this method to use a different credentials provider (e.g., SDK v2 only).
+     */
+    protected void updateDelegationTokenConfig(org.apache.hadoop.conf.Configuration hadoopConfig) {
+        AbstractS3DelegationTokenReceiver.updateHadoopConfig(hadoopConfig);
+    }
 }