-
Notifications
You must be signed in to change notification settings - Fork 16.6k
Description
Official Helm Chart version
1.16.0 (latest released)
Apache Airflow version
2.10.5
Kubernetes Version
1.30.10
Helm Chart configuration
Hello!
When using multiple executors, tasks are routed to the appropriate executor using the executor attribute.
However, the default KEDA query in the chart uses the queue column in the database, which was relevant for the hybrid CeleryKubernetesExecutor.
KEDA Query:
SELECT ceil(COUNT(*)::decimal / 4) FROM task_instance WHERE (state='running' OR state='queued') AND queue != 'kubernetes'
Docker Image customizations
not relevant
What happened
Celery workers scale up even when the task is configured to use KubernetesExecutor, since its queue attribute is default, while its executor attribute is KubernetesExecutor
What you think should happen instead
When using the hybrid executor, the query should use the queue.
Otherwise, the query should use executor.
How to reproduce
Deploy Airflow with keda enabled, using multiple executors, with celery being the default:
executor: CeleryExecutor,KubernetesExecutor
workers:
keda:
enabled: true
Create a DAG to launch multiple tasks using KubernetesExecutor with default queue.
The KEDA query will cause scale up of celery workers.
Anything else
Thank you and please let me know if any additional details are needed, or if I missed something to address this issue :)
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct