Skip to content

Demo application on how to create a serverless realtime analytics application using Kinesis Data Streams, Kinesis Firehose, DynamoDB and Lambda, using Cognito to secure the App.

License

Notifications You must be signed in to change notification settings

aws-samples/serverless-realtime-analytics

Serverless Realtime Analytics

This application demonstrates how to create a realtime analytics serverless application using Amazon Kinesis Data Streams, Amazon Kinesis Firehose, Amazon DynamoDB, AWS Lambda, Amazon API Gateway, Amazon Cognito, Amazon Simple Storage Service, Amazon Cloudfront, AWS Amplify and AWS Cloud Development Kit.

  • Amazon Kinesis Data Streams - We will use this service as it provides a serverless mechanism for realtime data ingestion, also, provides the flexibility to add a Lambda as a trigger and attach a Kinesis Firehose for data storage to S3.
  • Amazon Kinesis Firehose - We will use this service to store the data ingested by the Data Stream for later analytics.
  • Amazon DynamoDB - Using DynamoDB we will have a Serverless and low latency database.
  • AWS Lambda - Here we will deploy the update and query lambda functions.
  • Amazon Simple Storage Service - We will use a bucket to store all the data ingested for future analysis.
  • Amazon Cognito - We will create a user and identity pool in Cognito to secure our analytics application.
  • Amazon API Gateway - API Gateway will be used as the Facade for our query lambda function and will secure the function using a Cognito Authenticator.
  • Amazon Cloudfront - We will create a distribution for our static content.
  • AWS Cloud Development Kit - Using AWS CDK we will create all the infrastructure needed for the project.
  • AWS Amplify - Using Amplify in our web application we will integrate it with Cognito and API Gateway.

Pre requisites

To deploy this demo you need:

NOTES:

Architecture

This is an overview of all the components in the application architecture.

Architecture

Deployment

This application consists of two CDK stacks:

  • RealTimeAnalyticsPocStack - This stack contains the most of the infrastructure and application. To deploy use the following commands:
    • $ npm run build
    • $ cdk deploy RealTimeAnalyticsPocStack

Sample output Output

  • RealTimeAnalyticsWebStack - This stack contains only the static web application, you will need to update the web/dashboard/src/main.js file with the outputs of the previous stack. Once updated:
    • $ cdk deploy RealTimeAnalyticsWebStack

Testing

To test the application we can use the project Kinesis Data Generator. Follow the instructions in the project page to deploy the tool.

You can use the following template for testing: session={{date.now('YYYYMMDD')}}|sequence={{date.now('x')}}|reception={{date.now('x')}}|instrument={{random.number(9)}}|l={{random.number(20)}}|price_0={{random.number({"min":10000, "max":30000})}}|price_1={{random.number({"min":10000, "max":30000})}}|price_2={{random.number({"min":10000, "max":30000})}}|price_3={{random.number({"min":10000, "max":30000})}}|price_4={{random.number({"min":10000, "max":30000})}}|price_5={{random.number({"min":10000, "max":30000})}}|price_6={{random.number({"min":10000, "max":30000})}}|price_7={{random.number({"min":10000, "max":30000})}}|price_8={{random.number({"min":10000, "max":30000})}}|

Clean Up

To clean up your account delete all files in the bucket created and run delete the CloudFormation stacks:

  • $ cdk destroy RealTimeAnalyticsWebStack
  • $ cdk destroy RealTimeAnalyticsPocStack

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

About

Demo application on how to create a serverless realtime analytics application using Kinesis Data Streams, Kinesis Firehose, DynamoDB and Lambda, using Cognito to secure the App.

Topics

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published