Metaflow x Armada POC

[Sharpz7]

Entry point

The easiest entry-point for this system would be as a new compute provider alongside AWS Batch and Kubernetes.

There are no docs for setting up local Kubernetes (Netflix/metaflow#1640) but it seems for other providers that you use terraform to install metaflow+the infra required.

Getting a POC that lets you run something through Armada should not be so bad, but adding all the other "features" that metaflow expects might be more challenging.

• Debugging
• Maintenance
• First Class Monitoring

The Plan

• Create a "Empty" Provider plugin: Essentially, mock everything we need, see if we can get it to work. This ensures that we don't miss anything before integrating Armada in.
• Integrate Armada