-
Notifications
You must be signed in to change notification settings - Fork 472
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
how to upgrade the airflow cluster while keep my airflow jobs running with no downtime #839
Comments
@zeddit Because changing the airflow image will always require a restart of all worker pods, your only options are something like:
|
@thesuperzapper really appreciate for your reply, and I learned a lot. could I understand that as, if I use what if I need to upgrade scheduler and other components, will that break the correctness of jobs if I followed the dag principle stated above? great thanks |
Checks
User-Community Airflow Helm Chart
.Motivation
It's a common case for upgrading the airflow, e.g. installing a new 3rd party provider's package and upgrade the airflow major version.
All these processes above will involve an upgrade to the airflow image running on k8s. which needs to shutdown the related pods and start a new one with the new version.
And this kind of shutdowns will make the jobs running on top of airflow stopped and may be not recovered because it will stop at a middle place in DAG and marked as success.
How to make airflow helm upgrade do not influence the jobs running and keep the result of job executions correct?
Implementation
No response
Are you willing & able to help?
The text was updated successfully, but these errors were encountered: