💨 Define Airflow EKS Cluster migration activities

[Gary-H9]

Note

A significant number of DAGs run in both environments and these are used daily by the Data Engineering community so any activity needs to take this into account.

Airflow (a.k.a. MWAA / Amazon Managed Workflows for Apache Airflow) exists on two EKS clusters in the analytical-platform-data-production account, the clusters are called airflow-dev and airflow-prod. (The sandpit has been deleted).

Historically these were created in Pulumi and they have been partially migrated to Terraform in the analytical-platform repository.

The purpose of this ticket is to identify the steps required to migrate away from the previously mentioned EKS clusters to clusters that are more actively maintained.

Definition of Done

• Should airflow be migrated to the analytical-platform-compute cluster?
• Understand overall approach. Do we build a new cluster, define a time to sunset the cluster, declare this to the community and ask them to migrate DAGs across?
• Capture requirements around IRSA and document this
• Capture requirements around TGW / networking and document this
• Capture requirements arounnd permissions and document this
• Ensure that Airflow is defined as IaC - aws_mwaa_environment
• Monitor the airflow version automatically so it doesn't fall out of date
• Ensure that the ownership of Airflow and associated components is clearly documented publicly
• Raise any further tickets

Miscellaneous

#4470