Skip to content

Eliminate trust policy race by migrating emr_eks test to Pod Identity - #71207

Open
ramitkataria wants to merge 1 commit into
apache:mainfrom
aws-mwaa:ramitkataria/eks-pod-identity
Open

Eliminate trust policy race by migrating emr_eks test to Pod Identity#71207
ramitkataria wants to merge 1 commit into
apache:mainfrom
aws-mwaa:ramitkataria/eks-pod-identity

Conversation

@ramitkataria

Copy link
Copy Markdown
Contributor

The test shares one EMR job execution role across runs while each run creates its own EKS cluster. With IRSA, every run had to rewrite the shared role's trust policy to federate its own cluster's OIDC provider, and pruning old statements to stay under the trust policy size limit meant concurrent runs deleted each other's trust entries, failing jobs with AssumeRoleWithWebIdentity AccessDenied. IAM offers no compare-and-swap on trust policies, so this race cannot be fixed while runs mutate shared IAM state.

EKS Pod Identity (supported by EMR on EKS since emr-7.3.0) removes the shared mutable state: the role keeps one static trust policy and each run creates Pod Identity associations scoped to its own cluster, also avoiding the 4096-char trust policy cap that limits IRSA to roughly a dozen clusters per role.

Pod Identity is also AWS's recommended mechanism for pod credentials:

Unless you have specific usecases for IRSA, we recommend you use
EKS Pod Identities when using EKS.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@ramitkataria
ramitkataria requested a review from o-nikolas as a code owner August 5, 2026 22:03
@boring-cyborg boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Aug 5, 2026
The test shares one EMR job execution role across runs while each run
creates its own EKS cluster. With IRSA, every run had to rewrite the
shared role's trust policy to federate its own cluster's OIDC provider,
and pruning old statements to stay under the trust policy size limit
meant concurrent runs deleted each other's trust entries, failing jobs
with AssumeRoleWithWebIdentity AccessDenied. IAM offers no
compare-and-swap on trust policies, so this race cannot be fixed while
runs mutate shared IAM state.

EKS Pod Identity (supported by EMR on EKS since emr-7.3.0) removes the
shared mutable state: the role keeps one static trust policy and each
run creates Pod Identity associations scoped to its own cluster, also
avoiding the 4096-char trust policy cap that limits IRSA to roughly a
dozen clusters per role.

Pod Identity is also AWS's recommended mechanism for pod credentials:

> Unless you have specific usecases for IRSA, we recommend you use
> EKS Pod Identities when using EKS.

https://docs.aws.amazon.com/eks/latest/best-practices/identity-and-access-management.html#_eks_pod_identities_compared_to_irsa
@ramitkataria
ramitkataria force-pushed the ramitkataria/eks-pod-identity branch from 39c4d48 to b17b493 Compare August 5, 2026 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant