[CVE][filesystem] Align Hadoop dependencies to 3.4.3

[litiliu]

Purpose

Linked issue: close #3682

Fluss currently uses inconsistent Hadoop versions (fluss.hadoop.version=3.4.0, fs.hadoopshaded.version=3.3.4). This PR aligns all filesystem plugins to Hadoop 3.4.3 and migrates fluss-fs-s3 from the end-of-life AWS SDK for Java v1 to the modular AWS SDK v2, while preserving authentication, STS, KMS and S3 transfer-manager support.

Brief change log

• Bump fluss.hadoop.version 3.4.0 → 3.4.3 and fs.hadoopshaded.version 3.3.4 → 3.4.3.
• Migrate fluss-fs-s3 to AWS SDK v2 (software.amazon.awssdk): STS / KMS / auth / s3 / s3-transfer-manager as individual modules; exclude software.amazon.awssdk:bundle; drop the unused DynamoDB dependency.
• Rewrite S3DelegationTokenProvider and DynamicTemporaryAWSCredentialsProvider against SDK v2 APIs; remove the copied XmlResponsesSaxParser and its checkstyle/coverage suppressions.
• Exclude netty / dropwizard-metrics / jettison that are not used by hadoop-common; refresh NOTICE/LICENSE for all filesystem modules (including COS); add the reactive-streams license.

Artifact size review

Artifact	Before	After	Change
Hadoop shaded	22.20 MB	24.13 MB	+1.94 MB (+8.72%)
HDFS	34.66 MB	35.29 MB	+0.63 MB (+1.82%)
S3	31.39 MB	44.52 MB	+13.14 MB (+41.86%)
OSS	38.35 MB	28.40 MB	−9.95 MB (−25.95%)
COS	41.63 MB	39.84 MB	−1.79 MB (−4.30%)
GS	65.78 MB	67.71 MB	+1.94 MB (+2.94%)
Azure	38.35 MB	30.80 MB	−7.54 MB (−19.68%)
Total	272.35 MB	270.70 MB	−1.65 MB (−0.61%)

The S3 increase is primarily the AWS SDK V2 S3 model and its asynchronous Netty transport. The s3-transfer-manager classes themselves account for only about 0.20 MB compressed, but Hadoop 3.4.3 S3A invokes S3TransferManager.copy() on copy/rename paths, so its module and transport cannot be removed safely. The resulting S3 artifact contains STS, KMS, Transfer Manager, and the async transport, and contains neither DynamoDB nor the complete AWS SDK bundle.

Tests

• Updated S3DelegationTokenProviderTest for the SDK v2 credential APIs.
• Existing filesystem IT cases (S3 / OSS / COS / GS / Azure / HDFS).
• mvn clean verify, spotless and RAT.

API and Format

No public API or storage-format change. Note: s3.aws.credentials.provider (fs.s3a.aws.credentials.provider) now requires an AWS SDK v2 software.amazon.awssdk.auth.credentials.AwsCredentialsProvider implementation instead of an SDK v1 provider (documented).

Documentation

Updated the S3 and HDFS filesystem docs for AWS SDK v2 / Hadoop 3.4.3.

Generative AI disclosure

• Yes (please specify the tool below)

Generated-by: OpenAI Codex following the guidelines (https://github.com/apache/fluss/blob/main/AGENTS.md)

[litiliu]

@luoyuxia @fresh-borzoni PTAL

[litiliu]

@leekeiabstraction Could you please take a look?

[leekeiabstraction]

TY for the PR. Did a quick eyeball, left a question. Will follow up with more thorough review once addressed.

[leekeiabstraction]

TY for the PR, left further comments. PTAL 🙏

[litiliu]

@leekeiabstraction Nice catches. I have addressed the review comments in 445fc7d and resolved the corresponding threads. Please continue the review when you have time. Thanks!

[leekeiabstraction]

Thank you @litiliu for the updates. Can you confirm if the steps in the glue doc has been tested?

@qzyu999 Can you review the glue doc changes here as you have context? Much appreciated.

[litiliu]

Thank you @litiliu for the updates. Can you confirm if the steps in the glue doc has been tested?

@qzyu999 Can you review the glue doc changes here as you have context? Much appreciated.

@leekeiabstraction Thanks for checking. To be precise, I have not re-run the complete Glue walkthrough end to end on ECS, EKS, or EC2. I verified that the documented AWS SDK v2 provider classes exist and implement the interface expected by Hadoop S3A, but I did not test live credential retrieval against the AWS runtime metadata endpoints.

After another look, I think the guide should avoid maintaining separate provider lists for ECS, EKS, and EC2 altogether. AWS SDK v2 already provides DefaultCredentialsProvider, which owns the environment-detection logic for environment variables, EKS web identity, ECS task roles, EC2 instance profiles, and shared AWS profiles.
I therefore propose replacing the environment-specific table with:
fs.s3a.aws.credentials.provider: software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider
This keeps the AWS SDK as the single source of truth and prevents the documentation from drifting when its provider chain changes. Users can still configure a specific provider when they intentionally need to restrict the credential source.
What do you think, @qzyu999 ?

[qzyu999]

Hi @litiliu and @leekeiabstraction, the SDK v2 class name swap for the existing providers LGTM, just a mechanical migration.

The change for DefaultCredentialsProvider I haven't tested, but IIUC it seems that it would be set in the server.yaml which ideally is sufficient for majority of use-cases. It would also still provide room for users to do custom overrides in case they have some outlier setup that has issues with DefaultCredentialsProvider, or they want to maybe avoid some slower cold starts with it as I believe it will try multiple methods with timeout until one works.

SGTM as long as the user can still choose their own path in case DefaultCredentialsProvider isn't suitable.

However, until it has been tested E2E it's hard to say for sure what should go in the docs regarding this issue. Perhaps we can go that route while adding some note that it's not tested in all possible use-cases? That might not be necessary either.

[litiliu]

@qzyu999 Thanks for confirming the AWS SDK v2 class-name swap. I agree that switching the guide to DefaultCredentialsProvider would require additional end-to-end validation. To keep this PR focused on the mechanical v1-to-v2 migration, I won’t include that additional change in this PR and will retain the current provider mappings. @leekeiabstraction

[leekeiabstraction]

Sounds good to me @litiliu, can either you or @qzyu999 raise an issue to follow up with the DefaultCredentialsProvider + manual e2e test/verification? Thank you.

[litiliu]

@leekeiabstraction, Created #3916 to track the DefaultCredentialsProvider evaluation and manual E2E verification. If you have no further concerns, would you mind approving this PR when you have time? Thanks!

[leekeiabstraction]

Why do we set disabledWithoutDocker? Doing so may cause the test be skipped instead of failed if CI / local build does not have docker running

[leekeiabstraction]

Approved, TY for your contribution!