Cloud-Native API Deployment with CI/CD on Google Cloud Platform (GCP)

This repository encompasses a web application backend utilizing Python 3.8 and MySQL, performing a series of tasks: health check (GET - /healthz), user creation (POST - /v1/user) with a subsequent verification email sent to the user, retrieval of user details (GET - /v1/user/self), and user details update (PUT - /v1/user/self). Upon user creation, a message is dispatched to GCP Pub/Sub, triggering a serverless function responsible for emailing the user with a unique verification link. Terraform is utilized to configure all necessary resources for the app setup.

Several GitHub workflows are established for Continuous Integration and Continuous Deployment. These workflows activate upon pull request creation, executing functional tests on the webapp backend to detect failures and examining the packer code formatting. Post successful workflow execution and code merging, packer is employed to generate a golden image of the application, configuring all essential libraries/dependencies. Subsequently, the existing instances in the GCP project are replaced with the updated packer image.

Install Google Cloud CLI

Follow this URL to find the installer steps for your machine https://cloud.google.com/sdk/docs/install

Note: Add the bin directory of gcloud-sdk to PATH variable, else you would have to run the commands with "<path-to-bin of gcloud sdk>/" prefix

Setup Google Cloud CLI

$ gcloud auth login
$ gcloud auth application-default login

Setup default project

$ gcloud config set project <project_id>

To revoke access to Google Cloud CLI

$ gcloud auth revoke
$ gcloud auth application-default revoke

Enable APIs in Google Cloud Platform

1. Navigate to google cloud dashboard: https://console.cloud.google.com/welcome/new
2. From the Navigation Menu > APIs and services > Library
3. Enable the following APIs:
• Compute Engine API
• Cloud SQL Admin API
• Service Networking API
• Cloud Source Repositories API
• Identity and Access Management (IAM) API
• Cloud Monitoring API
• Cloud Logging API
• Serverless VPC Access API
• Eventarc API
• Cloud Deployment Manager V2 API
• Cloud DNS API
• Cloud Functions API
• Artifact Registry API
• Cloud Pub/Sub API
• Cloud Build API
• Service Usage API
• Secret Manager API
• Certificate Manager API
• Cloud Key Management Service (KMS) API
4. After enabling the APIs it may take about 10-15 mins to be activated

Create Service account and give permissions

1. Navigate to google cloud dashboard: https://console.cloud.google.com/welcome/new
2. From the Navigation Menu > IAM and admin > Service accounts
3. Create a new / modify the permissions of existing service account with the following permissions

Cloud SQL Editor
Compute Instance Admin (v1)
Compute Network Admin
Compute Security Admin
IAP-secured Tunnel User
OSPolicyAssignment Editor
Pub/Sub Publisher
Secret Manager Secret Accessor
Service Account Token Creator
Service Account User
Storage Object Viewer

Add Service Account key to the environment vairable

Create JSON key for the service account

1. Navigate to google cloud dashboard: https://console.cloud.google.com/welcome/new
2. From the Navigation Menu > IAM and admin > Service accounts
3. Click on the service account to be used
4. Under "KEYS" tab, click on "ADD KEY" dropdown and select "Create new key"

# After downloading the JSON key file, run the following
$ export GOOGLE_APPLICATION_CREDENTIALS="<path to key>/<file_name>.json"

Install packer

Follow this URL to find the installer steps for your machine https://developer.hashicorp.com/packer/install

Choosing the OS family

https://cloud.google.com/compute/docs/images/os-details

Create a input variable file

1. Create a file named *.pkrvars.hcl
2. Add the data in the format shown below (expected variables are mentioned in the variables.tf file):

project_id          = "<project_id>"
zone                = "<project_zone>"
machine_type        = "<machine_type>"
ssh_username        = "<name_of_the_service_account>"
use_os_login        = false or true
source_image_family = "<os_family>"
webapp_version      = <webapp_version>
mysql_root_password = <mysql_password to be created for 'root' user>

Note: If the variable file is named *.auto.pkrvars.hcl it will automatically be picked up by packer. The --var-file="*.pkrvars.hcl" is not required to be passed to validate or build the image

Formatting packer code

Formatting the packer code

$ packer fmt <file_name or .>

Validate packer config

Check and validate the packer config to be created created

$ packer validate -var-file="*.pkrvars.hcl" <file_name or .>

Building image by packer

Building the image using packer

$ packer build -var-file="*.pkrvars.hcl" <file_name or .>

Starting the application

Create and activate python3 environment using the following command

# Create environment
$ python -m venv <env_name>

# Activating environment
## For mac and linux os users
$ source <env_name>/bin/activate

## For windows users
$ <env_name>/Scripts/activate

Install required packages from the requirements file

$ pip install -r requirements.txt

Create .env file in with the following key value pairs

Note: Update the DEV_HOST, DEV_PORT, PROD_HOST, PROD_PORT based on the requirement

SQLALCHEMY_DATABASE_URI_DEV = "mysql://<USERNAME>:<PASSWORD>@<HOST>:<PORT>/<DBNAME>"
# The following flag can toggle on/off the tracking of inserts, updates, and deletes for models
SQLALCHEMY_TRACK_MODIFICATIONS = False
DEV_HOST = "0.0.0.0"
DEV_PORT = 8080
PROD_HOST = "0.0.0.0"
PROD_PORT = 8080
PYTHON_ENV = "development"
GOOGLE_PROJECT_ID = "<gcp_project_id>"
GOOGLE_TOPIC_NAME = "<pub/sub_topic_name>"

Run the create_databse.py to create the database if it does not exists

$ python create_database.py

Run the following scripts to create table schema in the database

$ flask db init
$ flask db migrate
$ flask db upgrade

To run the tests

# Run the following command to run the tests
$ python -m pytest

To run the app

# Run the following command to start the server
$ python ./app.py

To run GitHub Workflows

Add the secrets to the Repository secrets

1. Go to repository settings
2. Under "Secrets and variables", select "Actions"
3. Create "New repository secrets" for the following:

# To run integration tests workflow
DB_PASSWORD: <password of the database> ('root' works in most cases)
DB_USER: <user of the database> ('root' works in most cases)
ENV_FILE: Add the contents of the webapp .env file with the above user and password for databse URI

# To run packer workflows
GCP_CREDENTIALS_JSON: Add the contents of the JSON key file created for the service account
PACKER_CONFIG: Add the content of the *.pkrvars.hcl or *.auto.pkrvars.hcl file