Python, Pandas, Lambda, Terraform, S3, Parquet, Redshift

This example demonstrates creating a Lambda function in Python language and deploy it using Terraform. For examples of Pandas, S3, Parquet & Redshift. See: https://github.com/datahealer/jupyter-s3-parquet-redshift

Requirements

• AWS Account : To create and use AWS services, you need to create an AWS account.

• AWS CLI : The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services. With terminal commands, you can manage your AWS services. To install aws-cli, see installing or updating the latest version of the AWS CLI.

• AWS Credentials : In order to manage your services from command line tools, you need an aws_access_key_id and an aws_secret_access_key. Creating a new user for IaC purposes is recommended. For this, you can create a new user with IAM. For this;

  1. Go to IAM
  2. Under Users click 'Add User'
  3. Give a. username (like terraform_user)
  4. For credential type select 'Access key - Programmatic access' and click next
  5. Click 'Create Group', specify a group name and select 'AdministratorAccess' policy.
  6. Click Review and create user. This user has a programmatic access and admin permissions.

  After you create the user, go to Users and select the user you have created. Go to 'Security Credentials' and click 'Create access key'. This will give an access key id and a secret access key. Save and dont share these credentials. You can not see access key again after you close this window.

  Once you have your credentials, open terminal and type:

  aws credentials
  

  Paste yur access key id, secret access key id. You can select a default region either.

• Terraform : For this project, you need to have Hashicorp Terraform, see Download Terraform.On Mac, you can download with:

  brew tap hashicorp/tap
  brew install hashicorp/tap/terraform
  

Project

• After you deploy this project, a Lambda function and an IAM role will be created in us-east-1 region. You can change AWS_REGION variable under variables.tf file. Under provider.tf file, you can see the AWS as provider.
• In iam.tf, 's3_role_new' and 'lambda_policy' for Lambda function will be created. This is like creating a user and attach a policy with management console. s3_role_new has full access for S3 and CloudWatch. You may wanna change this permissions under Statement for security purposes. You can check out this site to create AWS policies.
• lambda.tf file will create firstly a zip for .py file. Then it will create a Lambda function. Lambda function will have:
  • IAM role which is created from iam.tf
  • Project name which is defined in variables.tf
  • Script zip which is created lambda.tf
  • Runtime of Python 3.8
• After you set up credentials, go to terminal and start terraform with:

  terraform init
  

  This will install hashicorp/aws v3.74.1 and create a file called .terraform. You need to have ''Terraform has been successfully initialized!'' message. Before deployment, you can view the services which will be created with:

  terraform plan
  

  If everyting seems OK, start deployment. This will take apprx. 20 seconds.

  terraform apply
  

  Check the Lambda function from AWS management console.