Connect to GCS and KMS using GKE workload identity federation

[abhinavg6]

In dedicated, we currently access customer-managed "external" buckets (BYOB) and KMS using a W&B deployment IAM service account created per deployment. Customer adds that W&B deployment IAM service account to their bucket & key policies and allows the required actions. It works but it's not the most secure form of cross-project / tenant access. Standard is service account impersonation across projects / tenants - https://cloud.google.com/iam/docs/service-account-impersonation using https://cloud.google.com/iam/docs/create-short-lived-credentials-direct.

We would like to support service account impersonation based access such that a W&B deployment IAM service account can impersonate an intermediary cross-project/tenant service account in customer's project to perform the necessary GCS and KMS actions. Customer on their part would configure the cross-project service account to trust the W&B deployment service account and allow required actions to GCS and KMS. And we would like to do this using the GKE workload identity federation - https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.

Would be good to have a migration plan in place such that for older customers we can continue to support current direct service account based mechanism until they're ready to make changes to use the service account impersonation based mechanism.