[argo] Add k8s labels to tempalte.metadata instead of spec.podMetadata

[jhasm]

Summary

Argo does not propagate the spec.podLabels to the pods. Only the sections under templates from the WorkflowTemplate are propagated to the Workflow and the pods. Their docs are misleading.

Here are related argo issues,

1. ClusterWorkflowTemplate podMetadata are ignored with workflowTemplateRef  argoproj/argo-workflows#7091
2. Pods created by Argo Workflows don't have the desired labels defined by the podMetadata field argoproj/argo-workflows#7013

Particularly, this comment is helpful in demystifying some confusion of the argo terminology

Testing

I tested against my argo installation. I did not see my custom labels before the change and I saw the labels applied after the change. I used METAFLOW_KUBERNETES_LABELS property in the ~/.metaflowconfig/config.json file.

[saikonen]

Took this for a quick spin but I have a few questions just so I understand the situation correctly.
Tried with something along the lines of

METAFLOW_KUBERNETES_LABELS="customlabelA=testA,customlabelB=123" python test.py argo-workflows create
python test.py argo-workflows trigger

then grabbing the Kubernetes pod name for f.ex. the start task and running kubectl describe pod -n metaflow-ns start-task-pod-name. For my case the pod has both of the custom labels. Is this the case that was not working for you?

Which version of Argo Workflows are you running on? I noticed that there was mention of this behavior being a bug in v3.1.3

[jhasm]

Yeah, I did not see the custom labels in my case. Our argo version is 3.0.7. I know this is kind of old, but adding the labels under template metadata makes it work for all argo version.

[jhasm]

@saikonen any thoughts?