Kafka.QuickStart.Cloud.Templates

Docker-compose templates for Cloud providers Microsoft Azure and AWS to deploy micro-services application leveraging Kafka producer-consumer pattern.

1. Folder organization

| -- /AWS/
     | -- Dockerrun.aws.json

| -- /AZURE/
     | -- docker-compose-extended-azure.yml
     | -- docker-compose-light-azure.yml      -> light version

2. Prerequisites

• You need to have an Microsoft Azure account or AWS account.
• You need to create Twitter API keys.

3. High Level Architecture

4. Instructions

4.1. Docker images

These are docker-compose templates for deployment under Azure or AWS.

It uses docker images hosted in Docker Hub Registry.

They are compatible with for Linux OS.

Docker images	Docker Repository	Description
zookeeper:3.4.9	link	Zookeeper
confluentinc/cp-kafka:5.4.0	link	Kafka
danmgs/kafkaquickstart-backend-console-producer:1.8	link	Kafka C# Console Producer
danmgs/kafkaquickstart-backend-console-consumer:1.8	link	Kafka C# Console Consumer
danmgs/kafkaquickstart-middleware-webapi:1.7	link	C# WebAPI
danmgs/kafkaquickstart-frontend-webapp:1.7	link	Angular WebApp

4.2. Configuration

Before using these templates, you need to configure the Twitter config options:

Options	Instructions	Mandatory/Optional
TWITTER_OAUTH_CONSUMER_KEY	to replace with your twitter API key	Mandatory
TWITTER_OAUTH_CONSUMER_SECRET	to replace with your twitter API secret	Mandatory
TWITTER_SEARCH_INTERVAL_MS	checks frequency in ms of tweets (*)	Optional
TWITTER_SEARCH_KEYWORD	keyword for tweet search (musique by default)	Optional

(*) Note Twitter has a throttling limit with "Too Many Requests" 429 response code. Choose milliseconds value accordingly (not too small)

4.3.1 Deployment under Azure

We will create a web app in App Service.

1. Select Publish: Docker container and Operating System: Linux

Click to expand details

2. Select Docker Options: Docker compose and Image Source: Docker Hub.

3. Upload configuration file docker-compose-extended-azure.yml

Click to expand details

4.3.2 Deployment under AWS

We will create a web app in ElasticBeanstalk.

1. Create an environment by selecting Web Environment tiers
2. Select Platform: Multi-container Docker
3. Upload configuration file Dockerrun.aws.json
4. Select Configure More Option, in the Capacity, you should select EC2 instance type: t3 medium at least (minimum requirement).

Click to expand details