-
Notifications
You must be signed in to change notification settings - Fork 16.6k
Fix migration when xcom has NaN values#53812
Conversation
closes: #51178
Testing:
- Ran a DAG with 2.11.0, which generates nan value in db
[{"day": "2024-06-07", "ArticleCountMetric": NaN}]
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
def push_nan_to_xcom(**kwargs):
# This dict contains a NaN, which is not valid JSON
value = [{"day": "2024-06-07", "ArticleCountMetric": float("nan")}]
kwargs["ti"].xcom_push(key="bad_json", value=value)
with DAG(
dag_id="xcom_nan_example",
start_date=datetime(2024, 1, 1),
schedule_interval=None,
catchup=False,
) as dag:
push_nan = PythonOperator(
task_id="push_nan",
python_callable=push_nan_to_xcom,
provide_context=True,
)
- Migrated to main now verified migration worked fine with all databases
- Verified in db nan is getting converted to string
[{"day": "2024-06-07", "ArticleCountMetric": "nan"}]
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
|
I wonder if it's better to delete those rows instead. XCom is not designed as a long term storage. Also, what happens if a new ti tries to push invalid values in 3.x? |
I prefer this too |
As per the discussion with @uranusjr, we can serialize nan to string |
amoghrajesh
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@vatsrahul1001 thanks, I think the fix would work as it achieves json compliancy and doesn't change the type too.
Backport failed to create: v3-0-test. View the failure log Run details
You can attempt to backport this manually by running: cherry_picker dee3a61 v3-0-test
This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continue
|