Support full podtemplate spec for affinity assistant pod

[piyush-garg]

Feature request

Right now, we can provide the complete set of field for taskrun pods and also assinity assistant pods using the configmap here default-pod-template and default-affinity-assistant-pod-template

From the documentation provided here it gives a perception that you can provide all the fields in config-defaults configmap for both pod-template and affinity-assistant-pod-template and can also override three fields (tolerations, nodeSelector, and imagePullSecrets) mentioned in documentation from taskrun and pipelinerun podtemplate which you provide during taskrun and pipeinerun creation.

However, when I tried this feature, it looks like only the three fields (tolerations, nodeSelector, and imagePullSecrets) from the default-affinity-assitant-pod-template are getting populated in the pod. I tried to figure out the code and see only these three field are getting read by the controller from default-affinity-assistant-pod-template https://github.com/tektoncd/pipeline/blob/main/pkg/apis/config/default.go#L156 and https://github.com/tektoncd/pipeline/blob/main/pkg/apis/pipeline/pod/affinity_assitant_template.go#L29

We should support all field for the default value provided in config-defaults configmap and make documentation more clear.

Use case

Use case is to provide the same podtemplate for both taskrun pods and assifnity assistant pod, so fields like priorityClassName can be provided and all pods get same priority value to schedule.

[piyush-garg]

@chenbh Any thoughts on this? Any specific reason we did not expose all the fields? cc @vdemeester