Multi Team: `/dags` screen is very slow to load with multiple teams

[stephen-bracken]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.2.1

What happened and how to reproduce it?

We have onboarded approximately 10 teams onto a multi-team airflow instance, and have dags on the order of hundreds, and now the /dags screen takes approximately 10 seconds to load. Our test Airflow has only one team and the /dags screen loads almost instantly.

What you think should happen instead?

Since the Keycloak Auth Manager already uses caching, perhaps we need the api server to send out multiple calls simultaneously to evaluate dag permissions?

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Other Docker-based deployment

Apache Airflow Provider(s)

keycloak

Versions of Apache Airflow Providers

apache-airflow-providers-keycloak==0.7.1

Official Helm Chart version

Not Applicable

Kubernetes Version

1.33.5

Helm Chart configuration

No response

Docker Image customizations

Some additional pip and apt packages

Anything else?

Multi team deployment using the Keycloak Auth Manager

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Andrushika]

I took a quick look at the Keycloak auth manager path and I could reproduce the lower-level behavior behind this.

KeycloakAuthManager.filter_authorized_dag_ids() wraps the result with cache/single-flight, but on cache miss it falls back to the base auth manager implementation. That base implementation checks each Dag id individually, so Keycloak ends up sending one authorization HTTP request per Dag id in that filter call.

I verified this locally with a small unit test: 25 Dag ids + mocked http_session.post resulted in 25 POST calls, each with a different resource_id claim.

I have not reproduced the full 10s /dags page load, but this per-Dag request pattern seems consistent with the reported slowdown when Keycloak calls have non-trivial latency.

AWS auth manager seems to avoid this by batching authorization checks in filter_authorized_dag_ids(). Would a similar batch path make sense for Keycloak?

BTW, this may also need area:providers / provider:keycloak labels.