Dark Mode

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Improve dag_bundle_config_list Configuration in Airflow Helm Chart#60645

Merged
jscheffl merged 5 commits intoapache:mainfrom
uplsh580:chart/improve-dag_bundle_config_list-configuration
Jan 18, 2026
Merged

Improve dag_bundle_config_list Configuration in Airflow Helm Chart#60645
jscheffl merged 5 commits intoapache:mainfrom
uplsh580:chart/improve-dag_bundle_config_list-configuration

Conversation

Copy link
Contributor

uplsh580 commented Jan 16, 2026 *
edited
Loading

Overview

Related Issue: #60630
Improve dag_bundle_config_list Configuration in Airflow Helm Chart

This PR improves the dag_bundle_config_list configuration in the Airflow Helm Chart by allowing users to define DAG bundle configurations in a structured YAML format instead of requiring a JSON string directly in the values file.

Changes

  • Added a new dag_bundles section under dagProcessor in values.yaml that allows users to define DAG bundles in a structured YAML format
  • Created a Helm template helper function dag_bundles that automatically converts the YAML structure to the required JSON string format
  • Set default value for dagProcessor.dag_bundles with a local DAG bundle configuration
  • Updated config.dag_processor.dag_bundle_config_list to use the helper function
  • Moved dag_bundles configuration under dagProcessor section for better logical grouping

Benefits

  1. Better Readability: Long JSON strings in YAML files are difficult to read and maintain. The new structured format is much more readable.
  2. Less Error-Prone: Manual JSON string construction is prone to syntax errors. The YAML format reduces the chance of errors.
  3. Better IDE Support: IDEs can provide proper syntax highlighting, validation, and autocomplete for YAML structures.
  4. Easier Maintenance: Adding, removing, or modifying bundle configurations is now straightforward.
  5. Better Organization: DAG bundle configuration is now logically grouped under the dagProcessor section.

Usage

Users can now define DAG bundles like this:

dagProcessor:
dag_bundles:
- name: dags-folder
classpath: "airflow.dag_processing.bundles.local.LocalDagBundle"
kwargs: {}
- name: bundle1
classpath: "airflow.providers.git.bundles.git.GitDagBundle"
kwargs:
git_conn_id: "GITHUB__repo1"
subdir: "dags"
tracking_ref: "main"
refresh_interval: 60
- name: bundle2
classpath: "airflow.providers.git.bundles.git.GitDagBundle"
kwargs:
git_conn_id: "GITHUB__repo2"
subdir: "dags"
tracking_ref: "develop"
refresh_interval: 120

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
  • Cursor

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

uplsh580 requested review from hussein-awala, jedcunningham and jscheffl as code owners January 16, 2026 15:45
boring-cyborg bot added the area:helm-chart Airflow Helm Chart label Jan 16, 2026
Copy link
Contributor Author

uplsh580 commented Jan 16, 2026 *
edited
Loading

Validation

Limitation

Values are transmitted in a single-line format.

uplsh580 commented Jan 16, 2026
uplsh580 force-pushed the chart/improve-dag_bundle_config_list-configuration branch 3 times, most recently from 3c59689 to 61b69a2 Compare January 17, 2026 19:33
uplsh580 force-pushed the chart/improve-dag_bundle_config_list-configuration branch from 61b69a2 to 0641f54 Compare January 17, 2026 19:35
jscheffl reviewed Jan 17, 2026
Copy link
Contributor

jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some tests are missing - can you add some please?

uplsh580 reacted with thumbs up emoji
jscheffl reviewed Jan 17, 2026
jscheffl reviewed Jan 17, 2026
uplsh580 requested a review from dstandish as a code owner January 18, 2026 01:57
Copy link
Contributor Author

uplsh580 commented Jan 18, 2026

Some tests are missing - can you add some please?

Thank you for the feedback. I've updated the PR with the following changes:

  • Added unit tests for dagProcessor.dagBundleConfigList.
  • Marked the setting as deprecated in the comments to avoid overwhelming users with multiple parallel options, as suggested.
  • Standardized variable names to align with the style of other values in the Helm chart.

uplsh580 force-pushed the chart/improve-dag_bundle_config_list-configuration branch from ecf3957 to eb4dacb Compare January 18, 2026 02:21
choo121600 reviewed Jan 18, 2026
Copy link
Member

choo121600 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small nit suggestion:
we've agreed to refer to DAG as Dag going forward.
It would be great if you could update the occurrences accordingly :)

[VOTE] results as of https://lists.apache.org/thread/0651j4vdgzmlhgndmotvznlg97kyh328

uplsh580 reacted with thumbs up emoji
Copy link
Contributor Author

uplsh580 commented Jan 18, 2026

One small nit suggestion:
we've agreed to refer to DAG as Dag going forward.
It would be great if you could update the occurrences accordingly :)

[VOTE] results as of https://lists.apache.org/thread/0651j4vdgzmlhgndmotvznlg97kyh328

Great catch, thank you for sharing the discussion link. I've gone through the PR and adjusted DAG to Dag to stay consistent with the community's decision. Thanks for your guidance!

choo121600 reacted with thumbs up emoji

jscheffl approved these changes Jan 18, 2026
Copy link
Contributor

jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good for me.

uplsh580 reacted with heart emoji
jscheffl added this to the Airflow Helm Chart 1.19.0 milestone Jan 18, 2026
jscheffl merged commit 8c42617 into apache:main Jan 18, 2026
94 checks passed
jason810496 pushed a commit to jason810496/airflow that referenced this pull request Jan 22, 2026
...pache#60645)

* Improve dag_bundle_config_list Configuration in Airflow Helm Chart

* Apply review of @jscheffi - Mark config.dag_processor.dag_bundle_config_list as deprecated

* Apply review of @jscheffi - Add test for dagProcessor.dag_bundles

* Rename dag_bundles to dagBundleConfigList for consistency

* Apply review of @choo121600 - Update term DAG -> Dag
shreyas-dev pushed a commit to shreyas-dev/airflow that referenced this pull request Jan 29, 2026
...pache#60645)

* Improve dag_bundle_config_list Configuration in Airflow Helm Chart

* Apply review of @jscheffi - Mark config.dag_processor.dag_bundle_config_list as deprecated

* Apply review of @jscheffi - Add test for dagProcessor.dag_bundles

* Rename dag_bundles to dagBundleConfigList for consistency

* Apply review of @choo121600 - Update term DAG -> Dag
jedcunningham mentioned this pull request Jan 30, 2026
98 tasks
jhgoebbert pushed a commit to jhgoebbert/airflow_Owen-CH-Leung that referenced this pull request Feb 8, 2026
...pache#60645)

* Improve dag_bundle_config_list Configuration in Airflow Helm Chart

* Apply review of @jscheffi - Mark config.dag_processor.dag_bundle_config_list as deprecated

* Apply review of @jscheffi - Add test for dagProcessor.dag_bundles

* Rename dag_bundles to dagBundleConfigList for consistency

* Apply review of @choo121600 - Update term DAG -> Dag
choo121600 pushed a commit to choo121600/airflow that referenced this pull request Feb 22, 2026
...pache#60645)

* Improve dag_bundle_config_list Configuration in Airflow Helm Chart

* Apply review of @jscheffi - Mark config.dag_processor.dag_bundle_config_list as deprecated

* Apply review of @jscheffi - Add test for dagProcessor.dag_bundles

* Rename dag_bundles to dagBundleConfigList for consistency

* Apply review of @choo121600 - Update term DAG -> Dag
Subham-KRLX pushed a commit to Subham-KRLX/airflow that referenced this pull request Mar 4, 2026
...pache#60645)

* Improve dag_bundle_config_list Configuration in Airflow Helm Chart

* Apply review of @jscheffi - Mark config.dag_processor.dag_bundle_config_list as deprecated

* Apply review of @jscheffi - Add test for dagProcessor.dag_bundles

* Rename dag_bundles to dagBundleConfigList for consistency

* Apply review of @choo121600 - Update term DAG -> Dag
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

choo121600 choo121600 left review comments

jscheffl jscheffl approved these changes

jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

hussein-awala Awaiting requested review from hussein-awala hussein-awala is a code owner

dstandish Awaiting requested review from dstandish dstandish is a code owner

Assignees

No one assigned

Labels

area:helm-chart Airflow Helm Chart

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants