⛰ Move Airflow Compute to APC cluster

[jhpyke]

Brief summary of what this Epic is

This Epic captures the work required to move Airflow jobs from the existing Airflow Cluster(s) to APC. This work was triggered by the failure of Kube2IAM, but represents a long term plan to improve the stability of the cluster

List of issues in this epic

• 🚚 Convert Kube2IAM-based Airflow pod to using IRSA, and capture process #4319
• 🚚 Migrate Airflow workloads to APC #4490
• Comms to users on how to re-run their image pipelines to make their images available using new compute
• Liaise with users with specific networking requirements to test DAGs on new cluster

Other information

Overall plan is:

• Create a new release of mojap-airflow-tools to be available within dev. This will have conditional logic that triggers the new cluster/IRSA on our basic pods ONLY.
• Release it, and update requirements file for airflow-dev to reflect the new release. This should take effect within 20 mins - if not check scheduler logs (requirements-ip-install-<date/time>.log)
• Run a test pod using the BasicPodOperator to ensure it can accept the new service-account info correctly.
• Assuming testing is smooth, release comms to users on how to update their images to be accessible from new cluster (new image release pipeline already made by @jacobwoffenden)
• Work with users with specific networking requirements to ensure these are met by new cluster
• Transition new user workloads to APC.