-
Notifications
You must be signed in to change notification settings - Fork 16.6k
dag_bundle_config_list Configuration in Airflow Helm Chart#60630Description
Description
Currently, the dag_bundle_config_list configuration in the Airflow Helm chart(1.18.0) requires users to provide a JSON string directly in the values file. This approach has several drawbacks:
-
Poor Readability: Long JSON strings in YAML files are difficult to read and maintain, especially when configuring multiple DAG bundles.
-
Error-Prone: Manual JSON string construction is prone to syntax errors (missing quotes, commas, brackets, etc.).
-
Limited IDE Support: IDEs cannot provide proper syntax highlighting, validation, or autocomplete for JSON strings embedded in YAML.
-
Maintenance Burden: Adding, removing, or modifying bundle configurations requires careful string manipulation.
Current Approach
dag_processor:
dag_bundle_config_list: '[{"name": "bundle1", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"git_conn_id": "GITHUB__repo1", "subdir": "dags", "tracking_ref": "main", "refresh_interval": 60}}, {"name": "bundle2", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"git_conn_id": "GITHUB__repo2", "subdir": "dags", "tracking_ref": "develop", "refresh_interval": 120}}]'
As you can see, this becomes unwieldy even with just two bundles, and it gets significantly worse with more configurations.
Proposed Solution
Allow users to define DAG bundle configurations in a structured YAML format, and use a Helm template helper function to automatically convert it to the required JSON string format.
Proposed Structure
- name: bundle1
classpath: "airflow.providers.git.bundles.git.GitDagBundle"
kwargs:
git_conn_id: "GITHUB__repo1"
subdir: "dags"
tracking_ref: "main"
refresh_interval: 60
- name: bundle2
classpath: "airflow.providers.git.bundles.git.GitDagBundle"
kwargs:
git_conn_id: "GITHUB__repo2"
subdir: "dags"
tracking_ref: "develop"
refresh_interval: 120
- name: localbundle
classpath: "airflow.dag_processing.bundles.local.LocalDagBundle"
config:
dag_processor:
dag_bundle_config_list: '{{ include "dag_bundles" }}'
Use case/motivation
No response
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct