From 87aa2b9a6f59aa9364f17b2087fde8df6a51acbe Mon Sep 17 00:00:00 2001 From: Aman Pruthi Date: Tue, 18 Jun 2024 23:17:15 +0530 Subject: [PATCH 1/2] feat: Added service account name in stackdriver conf (#136) * feat: added support for stackdriver and otel metrics * fixed checks * fixed tmp service account issue * fixed lint * fixed sa issue * added namespace var * added service account var * terraform-docs: automated action * fixed naming * terraform-docs: automated action * Added node pool metadata * feat: Enabled workload identity access for kms and gcs (#113) * enabled workload identity access for kms and gcs * Fixed wandb-app service account issue * added permissions to sa * fixed permission issue * terraform-docs: automated action * Added service name in otel conf --------- Co-authored-by: amanpruthi Co-authored-by: github-actions[bot] --- README.md | 8 ++- main.tf | 32 ++++++---- modules/app_gke/main.tf | 18 ++++-- modules/service_accounts/main.tf | 87 ++++++++++++++++++++++++--- modules/service_accounts/outputs.tf | 9 ++- modules/service_accounts/variables.tf | 15 +++-- outputs.tf | 6 +- variables.tf | 21 ++++--- 8 files changed, 149 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index 3e2c602..f3979f6 100644 --- a/README.md +++ b/README.md @@ -101,16 +101,18 @@ resources that lack official modules. | [app\_wandb\_env](#input\_app\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no | | [bucket\_name](#input\_bucket\_name) | Use an existing bucket. | `string` | `""` | no | | [create\_redis](#input\_create\_redis) | Boolean indicating whether to provision an redis instance (true) or not (false). | `bool` | `false` | no | +| [create\_workload\_identity](#input\_create\_workload\_identity) | Flag to indicate whether to create a workload identity for the service account. | `bool` | `true` | no | | [database\_machine\_type](#input\_database\_machine\_type) | Specifies the machine type to be allocated for the database | `string` | `"db-n1-standard-2"` | no | | [database\_sort\_buffer\_size](#input\_database\_sort\_buffer\_size) | Specifies the sort\_buffer\_size value to set for the database | `number` | `67108864` | no | | [database\_version](#input\_database\_version) | Version for MySQL | `string` | `"MYSQL_8_0_31"` | no | | [deletion\_protection](#input\_deletion\_protection) | If the instance should have deletion protection enabled. The database / Bucket can't be deleted when this value is set to `true`. | `bool` | `true` | no | | [disable\_code\_saving](#input\_disable\_code\_saving) | Boolean indicating if code saving is disabled | `bool` | `false` | no | | [domain\_name](#input\_domain\_name) | Domain for accessing the Weights & Biases UI. | `string` | `null` | no | -| [enable\_stackdriver](#input\_enable\_stackdriver) | n/a | `bool` | `false` | no | +| [enable\_stackdriver](#input\_enable\_stackdriver) | n/a | `bool` | `true` | no | | [force\_ssl](#input\_force\_ssl) | Enforce SSL through the usage of the Cloud SQL Proxy (cloudsql://) in the DB connection string | `bool` | `false` | no | | [gke\_machine\_type](#input\_gke\_machine\_type) | Specifies the machine type to be allocated for the database | `string` | `"n1-standard-4"` | no | | [gke\_node\_count](#input\_gke\_node\_count) | n/a | `number` | `2` | no | +| [kms\_gcs\_sa\_name](#input\_kms\_gcs\_sa\_name) | n/a | `string` | `"wandb-app"` | no | | [labels](#input\_labels) | Labels to apply to resources | `map(string)` | `{}` | no | | [license](#input\_license) | Your wandb/local license | `string` | n/a | yes | | [local\_restore](#input\_local\_restore) | Restores W&B to a stable state if needed | `bool` | `false` | no | @@ -126,16 +128,15 @@ resources that lack official modules. | [redis\_tier](#input\_redis\_tier) | Specifies the tier for this Redis instance | `string` | `"STANDARD_HA"` | no | | [resource\_limits](#input\_resource\_limits) | Specifies the resource limits for the wandb deployment | `map(string)` | 
{
  "cpu": null,
  "memory": null
}
| no | | [resource\_requests](#input\_resource\_requests) | Specifies the resource requests for the wandb deployment | `map(string)` | 
{
  "cpu": "2000m",
  "memory": "2G"
}
| no | -| [service\_account\_name](#input\_service\_account\_name) | n/a | `string` | `"stackdriver"` | no | | [size](#input\_size) | Deployment size for the instance | `string` | `null` | no | | [ssl](#input\_ssl) | Enable SSL certificate | `bool` | `true` | no | +| [stackdriver\_sa\_name](#input\_stackdriver\_sa\_name) | n/a | `string` | `"wandb-stackdriver"` | no | | [subdomain](#input\_subdomain) | Subdomain for accessing the Weights & Biases UI. Default creates record at Route53 Route. | `string` | `null` | no | | [subnetwork](#input\_subnetwork) | Pre-existing subnetwork self link | `string` | `null` | no | | [use\_internal\_queue](#input\_use\_internal\_queue) | Uses an internal redis queue instead of using google pubsub. | `bool` | `false` | no | | [wandb\_image](#input\_wandb\_image) | Docker repository of to pull the wandb image from. | `string` | `"wandb/local"` | no | | [wandb\_version](#input\_wandb\_version) | The version of Weights & Biases local to deploy. | `string` | `"latest"` | no | | [weave\_wandb\_env](#input\_weave\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no | -| [workload\_account\_id](#input\_workload\_account\_id) | n/a | `string` | `"stackdriver"` | no | ## Outputs @@ -157,6 +158,7 @@ resources that lack official modules. | [fqdn](#output\_fqdn) | The FQDN to the W&B application | | [gke\_node\_count](#output\_gke\_node\_count) | n/a | | [gke\_node\_instance\_type](#output\_gke\_node\_instance\_type) | n/a | +| [sa\_account\_email](#output\_sa\_account\_email) | This output provides the email address of the service account created for workload identity, if workload identity is enabled. Otherwise, it returns null | | [service\_account](#output\_service\_account) | Weights & Biases service account used to manage resources. | | [standardized\_size](#output\_standardized\_size) | n/a | | [url](#output\_url) | The URL to the W&B application | diff --git a/main.tf b/main.tf index dbbd542..b1fcd25 100644 --- a/main.tf +++ b/main.tf @@ -29,13 +29,14 @@ locals { } module "service_accounts" { - source = "./modules/service_accounts" - namespace = var.namespace - bucket_name = var.bucket_name - account_id = var.workload_account_id - service_account_name = var.service_account_name - enable_stackdriver = var.enable_stackdriver - depends_on = [module.project_factory_project_services] + source = "./modules/service_accounts" + namespace = var.namespace + bucket_name = var.bucket_name + kms_gcs_sa_name = var.kms_gcs_sa_name + create_workload_identity = var.create_workload_identity + stackdriver_sa_name = var.stackdriver_sa_name + enable_stackdriver = var.enable_stackdriver + depends_on = [module.project_factory_project_services] } module "kms" { @@ -87,7 +88,7 @@ module "app_gke" { network = local.network subnetwork = local.subnetwork service_account = module.service_accounts.service_account - create_workload_identity = var.enable_stackdriver + create_workload_identity = var.create_workload_identity depends_on = [module.project_factory_project_services] } @@ -151,7 +152,6 @@ module "gke_app" { database_connection_string = module.database.connection_string redis_connection_string = local.redis_connection_string redis_ca_cert = local.redis_certificate - oidc_client_id = var.oidc_client_id oidc_issuer = var.oidc_issuer oidc_auth_method = var.oidc_auth_method @@ -247,6 +247,13 @@ module "wandb" { app = { extraEnvs = var.app_wandb_env + serviceAccount = var.create_workload_identity ? { + name = var.kms_gcs_sa_name + annotations = { "iam.gke.io/gcp-service-account" = module.service_accounts.sa_account_email } + } : { + name = "" + annotations = {} + } } ingress = { @@ -261,9 +268,10 @@ module "wandb" { stackdriver = var.enable_stackdriver ? { install = true stackdriver = { - projectId = data.google_client_config.current.project + projectId = data.google_client_config.current.project + serviceAccountName = var.stackdriver_sa_name } - serviceAccount = { annotations = { "iam.gke.io/gcp-service-account" = module.service_accounts.monitoring_role } } + serviceAccount = { annotations = { "iam.gke.io/gcp-service-account" = module.service_accounts.stackdriver_email } } } : { install = false stackdriver = {} @@ -281,7 +289,7 @@ module "wandb" { scheme = "http" metrics_path = "/metrics" dns_sd_configs = [ - { names = ["stackdriver"] + { names = ["wandb-stackdriver"] type = "A" port = 9255 } diff --git a/modules/app_gke/main.tf b/modules/app_gke/main.tf index 5027a22..965168f 100644 --- a/modules/app_gke/main.tf +++ b/modules/app_gke/main.tf @@ -7,12 +7,12 @@ locals { resource "google_container_cluster" "default" { name = "${var.namespace}-cluster" - network = var.network.self_link - subnetwork = var.subnetwork.self_link - networking_mode = "VPC_NATIVE" - + network = var.network.self_link + subnetwork = var.subnetwork.self_link + networking_mode = "VPC_NATIVE" enable_intranode_visibility = true + binary_authorization { evaluation_mode = "PROJECT_SINGLETON_POLICY_ENFORCE" } @@ -24,7 +24,7 @@ resource "google_container_cluster" "default" { workload_pool = "${local.project_id}.svc.id.goog" } } - + ip_allocation_policy { cluster_ipv4_cidr_block = "/14" services_ipv4_cidr_block = "/19" @@ -77,6 +77,14 @@ resource "google_container_node_pool" "default" { "https://www.googleapis.com/auth/trace.append", "https://www.googleapis.com/auth/sqlservice.admin", ] + + dynamic "workload_metadata_config" { + for_each = var.create_workload_identity == true ? [1] : [] + content { + mode = "GKE_METADATA" + } + } + shielded_instance_config { enable_secure_boot = true } diff --git a/modules/service_accounts/main.tf b/modules/service_accounts/main.tf index 3dc0337..eddd168 100644 --- a/modules/service_accounts/main.tf +++ b/modules/service_accounts/main.tf @@ -62,30 +62,97 @@ resource "google_project_iam_member" "secretmanager_admin" { role = "roles/secretmanager.admin" } +####### service account for kms and gcs cross project access +resource "google_service_account" "kms_gcs_sa" { + count = var.create_workload_identity == true ? 1 : 0 + account_id = var.kms_gcs_sa_name + display_name = "Service Account For Workload Identity" +} + +resource "google_project_iam_member" "cloudsql_client_gcs" { + count = var.create_workload_identity == true ? 1 : 0 + project = local.project_id + role = "roles/cloudsql.client" + member = "serviceAccount:${google_service_account.kms_gcs_sa[0].email}" +} + +resource "google_project_iam_member" "secretmanager_admin_gcs" { + count = var.create_workload_identity == true ? 1 : 0 + project = local.project_id + member = "serviceAccount:${google_service_account.kms_gcs_sa[0].email}" + role = "roles/secretmanager.admin" +} + +# For some reason we need this permission otherwise backend is throwing an error +# hopfully this is a short term fix. +resource "google_project_iam_member" "log_writer_gcs" { + count = var.create_workload_identity == true ? 1 : 0 + project = local.project_id + role = "roles/logging.logWriter" + member = "serviceAccount:${google_service_account.kms_gcs_sa[0].email}" +} + +resource "google_project_iam_member" "storage" { + count = var.create_workload_identity == true ? 1 : 0 + project = local.project_id + role = "roles/storage.admin" + member = "serviceAccount:${google_service_account.kms_gcs_sa[0].email}" +} + +resource "google_storage_bucket_iam_member" "gcs_admin" { + count = var.bucket_name != "" ? 1 : 0 + bucket = var.bucket_name + member = google_service_account.kms_gcs_sa[0].email + role = "roles/storage.objectAdmin" +} + + +resource "google_project_iam_member" "kms" { + count = var.create_workload_identity == true ? 1 : 0 + project = local.project_id + role = "roles/cloudkms.admin" + member = "serviceAccount:${google_service_account.kms_gcs_sa[0].email}" +} + +resource "google_service_account_iam_member" "token_creator_binding" { + count = var.create_workload_identity == true ? 1 : 0 + service_account_id = google_service_account.kms_gcs_sa[0].id + role = "roles/iam.serviceAccountTokenCreator" + member = "serviceAccount:${google_service_account.kms_gcs_sa[0].email}" +} + +resource "google_service_account_iam_member" "workload_binding" { + count = var.create_workload_identity == true ? 1 : 0 + service_account_id = google_service_account.kms_gcs_sa[0].id + role = "roles/iam.workloadIdentityUser" + member = "serviceAccount:${local.project_id}.svc.id.goog[default/${var.kms_gcs_sa_name}]" +} + -resource "google_service_account" "workload-identity-user-sa" { +### service account for stackdriver +resource "google_service_account" "stackdriver" { count = var.enable_stackdriver == true ? 1 : 0 - account_id = "stackdriver" + account_id = var.stackdriver_sa_name display_name = "Service Account For Workload Identity" } -resource "google_project_iam_member" "monitoring-role" { +resource "google_project_iam_member" "monitoring" { count = var.enable_stackdriver == true ? 1 : 0 project = local.project_id role = "roles/monitoring.viewer" - member = "serviceAccount:${google_service_account.workload-identity-user-sa[count.index].email}" + member = "serviceAccount:${google_service_account.stackdriver[0].email}" } -resource "google_service_account_iam_member" "monitoring-role" { +resource "google_service_account_iam_member" "stackdriver_token_creator" { count = var.enable_stackdriver == true ? 1 : 0 - service_account_id = google_service_account.workload-identity-user-sa[count.index].id + service_account_id = google_service_account.stackdriver[0].id role = "roles/iam.serviceAccountTokenCreator" - member = "serviceAccount:${google_service_account.workload-identity-user-sa[count.index].email}" + member = "serviceAccount:${google_service_account.stackdriver[0].email}" } -resource "google_service_account_iam_member" "workload_identity-role" { +resource "google_service_account_iam_member" "stackdriver_binding" { count = var.enable_stackdriver == true ? 1 : 0 - service_account_id = google_service_account.workload-identity-user-sa[count.index].id + service_account_id = google_service_account.stackdriver[0].id role = "roles/iam.workloadIdentityUser" - member = "serviceAccount:${local.project_id}.svc.id.goog[default/${var.service_account_name}]" + member = "serviceAccount:${local.project_id}.svc.id.goog[default/${var.stackdriver_sa_name}]" } \ No newline at end of file diff --git a/modules/service_accounts/outputs.tf b/modules/service_accounts/outputs.tf index ba84de5..e485235 100644 --- a/modules/service_accounts/outputs.tf +++ b/modules/service_accounts/outputs.tf @@ -1,9 +1,12 @@ output "service_account" { value = google_service_account.main - description = "The service account." } -output "monitoring_role" { - value = var.enable_stackdriver == true ? google_service_account.workload-identity-user-sa[0].email : null +output "sa_account_email" { + value = var.create_workload_identity == true ? google_service_account.kms_gcs_sa[0].email : null +} + +output "stackdriver_email" { + value = var.enable_stackdriver == true ? google_service_account.stackdriver[0].email : null } \ No newline at end of file diff --git a/modules/service_accounts/variables.tf b/modules/service_accounts/variables.tf index 6cc7675..2da1963 100644 --- a/modules/service_accounts/variables.tf +++ b/modules/service_accounts/variables.tf @@ -9,12 +9,17 @@ variable "bucket_name" { default = "" } -variable "account_id" { - description = "The ID of the Google Cloud Platform (GCP) account." - type = string + +variable "create_workload_identity" { + description = "Flag to indicate whether to create a workload identity for the service account." + type = bool } -variable "service_account_name" { +variable "kms_gcs_sa_name" { + type = string +} + +variable "stackdriver_sa_name" { description = "The name of the service account." type = string } @@ -22,4 +27,4 @@ variable "service_account_name" { variable "enable_stackdriver" { description = "Flag to indicate whether to enable workload identity for the service account." type = bool -} \ No newline at end of file +} diff --git a/outputs.tf b/outputs.tf index 0a9cbe2..1a600d3 100644 --- a/outputs.tf +++ b/outputs.tf @@ -85,5 +85,7 @@ output "database_instance_type" { value = coalesce(try(local.deployment_size[var.size].db, null), var.database_machine_type) } - - +output "sa_account_email" { + description = "This output provides the email address of the service account created for workload identity, if workload identity is enabled. Otherwise, it returns null" + value = var.create_workload_identity == true ? module.service_accounts.sa_account_email : null +} \ No newline at end of file diff --git a/variables.tf b/variables.tf index 57aa658..510d6f1 100644 --- a/variables.tf +++ b/variables.tf @@ -254,17 +254,24 @@ variable "parquet_wandb_env" { default = {} } +variable "create_workload_identity" { + description = "Flag to indicate whether to create a workload identity for the service account." + type = bool + default = true +} + +variable "kms_gcs_sa_name" { + type = string + default = "wandb-app" +} + variable "enable_stackdriver" { type = bool - default = false + default = true } -variable "workload_account_id" { +variable "stackdriver_sa_name" { type = string - default = "stackdriver" + default = "wandb-stackdriver" } -variable "service_account_name" { - type = string - default = "stackdriver" -} \ No newline at end of file From 444ae0260e0ff83961ab327e527594f437621646 Mon Sep 17 00:00:00 2001 From: semantic-release-bot Date: Tue, 18 Jun 2024 17:47:45 +0000 Subject: [PATCH 2/2] chore(release): version 3.3.0 [skip ci] ## [3.3.0](https://github.com/wandb/terraform-google-wandb/compare/v3.2.0...v3.3.0) (2024-06-18) ### Features * Added service account name in stackdriver conf ([#136](https://github.com/wandb/terraform-google-wandb/issues/136)) ([87aa2b9](https://github.com/wandb/terraform-google-wandb/commit/87aa2b9a6f59aa9364f17b2087fde8df6a51acbe)), closes [#113](https://github.com/wandb/terraform-google-wandb/issues/113) --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f35a7f2..2a3afb1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ All notable changes to this project will be documented in this file. +## [3.3.0](https://github.com/wandb/terraform-google-wandb/compare/v3.2.0...v3.3.0) (2024-06-18) + + +### Features + +* Added service account name in stackdriver conf ([#136](https://github.com/wandb/terraform-google-wandb/issues/136)) ([87aa2b9](https://github.com/wandb/terraform-google-wandb/commit/87aa2b9a6f59aa9364f17b2087fde8df6a51acbe)), closes [#113](https://github.com/wandb/terraform-google-wandb/issues/113) + ## [3.2.0](https://github.com/wandb/terraform-google-wandb/compare/v3.1.1...v3.2.0) (2024-06-07)