Requirement to specifyIAP_BACKEND_SERVICE_ID creates a circular dependency

[aebrahim]

I do not see a good way to create the cloud run job using an infrastructure as code solution like terraform due to the circular nature of the requirement to specify the IAP_BACKEND_SERVICE_ID in the cloud run service itself, which needs to be created before the backend service can be created.

This is sample failing terraform configuration:

resource "google_service_account" "jit_access" {
  account_id   = "jit-access"
  project      = var.project_id
  display_name = "Grants just-in-time access to IAM roles"
}

resource "google_project_iam_member" "jit_access_iam_admin" {
  project = var.project_id
  role    = "roles/resourcemanager.projectIamAdmin"
  member  = google_service_account.jit_access.member
}

resource "google_project_iam_member" "jit_access_cloudasset_viewer" {
  project = var.project_id
  role    = "roles/cloudasset.viewer"
  member  = google_service_account.jit_access.member
}

resource "google_project_iam_member" "jit_access_token_creator" {
  project = var.project_id
  member  = google_service_account.jit_access.member
  role    = "roles/iam.serviceAccountTokenCreator"
}

resource "google_cloud_run_v2_service" "jit_access" {
  name     = "jit-access"
  location = var.region
  project  = var.project_id
  ingress  = "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER"
  traffic {
    type    = "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"
    percent = 100
  }
  template {
    containers {
      image = "${var.region}-docker.pkg.dev/${var.project}/containers/jit-access:latest"
      name  = "jit-access"
      env {
        name  = "RESOURCE_SCOPE"
        value = "projects/${var.project_id}"
      }
      env {
        name  = "RESOURCE_CUSTOMER_ID"
        value = var.customer_id
      }
      env {
        name  = "IAP_BACKEND_SERVICE_ID"
        value = google_compute_backend_service.jit_access.generated_id
      }
      resources {
        startup_cpu_boost = true
        cpu_idle          = true
      }
    }

    service_account       = google_service_account.jit_access.email
    execution_environment = "EXECUTION_ENVIRONMENT_GEN2"
  }
}


data "google_iam_policy" "jit_access_iap" {
  binding {
    role = "roles/iap.httpsResourceAccessor"
    members = ["group:[email protected]"]
  }
}

resource "google_iap_web_backend_service_iam_policy" "jit_access" {
  project             = var.project_id
  web_backend_service = google_compute_backend_service.jit_access.name
  policy_data         = data.google_iam_policy.jit_access_iap.policy_data
}

resource "google_compute_region_network_endpoint_group" "jit_access" {
  name                  = "jit-access"
  network_endpoint_type = "SERVERLESS"
  region                = var.region

  cloud_run {
    service = google_cloud_run_v2_service.jit_access.name
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "google_compute_backend_service" "jit_access" {
  name = "jit-access"

  backend {
    group = google_compute_region_network_endpoint_group.jit_access.id
  }
  iap {
    oauth2_client_id     = google_iap_client.my_iap_client.client_id
    oauth2_client_secret = google_iap_client.my_iap_client.secret
  }

  custom_response_headers = local.security_headers
  log_config {
    enable = true
  }
}

Can the required environmental variable IAP_BACKEND_SERVICE_ID be removed from the cloud run job? From the docs, I am not clear on why the cloud run job needs to be aware of the IAP environment feeding traffic to it.

[jpassing]

JIT Access completely relies on IAP for authentication and authorization. To verify that (a) IAP is enabled at all and (b) that a request has indeed being vetted by IAP, JIT Access verifies the x-goog-iap-jwt-assertion header. For that, it needs to know what audience to expect in the header.

On AppEngine, the expected audience can be derived from the project ID. On Cloud Run, there's no way for the application to determine the audience automatically, hence the need for the IAP_BACKEND_SERVICE_ID variable.

To break the cyclic dependency between the load balancer and Cloud Run, you can deploy things in the following order:

1. Load balancer backend
2. Cloud Run
3. Other load balancer components

That's the sequence followed by the manual setup instructions. Maybe you can get Terraform to follow this sequence by using depends_on?

[aebrahim]

Thanks @jpassing for the response!

Unfortunately, to the best of my knowledge, depends_on doesn't resolve circular dependencies, it only pulls an unconnected dependency into the dependency graph for the current node.

Is there a way we can rely on the terraform configuration to ensure that all traffic flowing in to the jit-access service has already been authenticated and authorized by IAP, and use a new flag to skip re-validation in jit-access? If this is possible, IMHO, the requirement to break infrastructure as code and require a superuser to perform manual steps negates any security improvement from a re-validation of an already validated token.

[jpassing]

What should work is to...

1. Manually create the backend:

gcloud compute backend-services create jitaccess-backend \
  --load-balancing-scheme=EXTERNAL \
  --global

2. Let the TF module use the existing backend (either by using a data source or passing the backend ID as parameter)

I know it's not great, but (1) is a one-time thing, so any subsequent deployments could be fully automatic.