Skip to content

Monitor specific log pattern/aggregations by Elasticsearch API. Leverage AWS Lambda+DynamoDB to achieve serverless log monitoring solution.

License

Notifications You must be signed in to change notification settings

charles-hsiao/aws-lambda-elasticsearch-log-monitor

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 

Repository files navigation

AWS Lambda Elasticsearch LogMonitor

Monitor specific log pattern/aggregations by Elasticsearch API.
Leverage AWS Lambda+DynamoDB to achieve serverless log monitoring solution.

Introduction

ElasticSearch provide Search API, Aggregations API to query logs query/aggregations easily.
By leverage these API, we can create a monitoring method by aggragate specific log pattern volume/aggregations. And send notification.
In this project, we use Python to create AWS Lambda+DynamoDB to achieve serverless and flexible configuration log monitoring solution. (Lambda function create by python-lambda)

Get Started

Create Monitor Item

  1. Using HTTP request tool like Postman to valid your log filter patterns, the post body will be parameters we use later
  2. git clone
  3. Create monitor item json content (Please see paragraph "Monitor Item Parameter" or reference "/tools/sample.json")
  4. Use /tools/validation.py ${json_file} to valid the monitor item json content that you just created
  5. If validation passed, put this json content to DynamoDB table "lambda.elk.monitor"

Monitor Item Parameter

Parameters Description Optional Value Example
monitor_id The monitor name, will use for notification - Cyclops-Beta-SignIn_FailedRate
enable Whether enable this monitor or not 1. true
2. false
true
elk_env The elasticsearch environment to connect 1. staging
2. production
staging
formula The formula for calculation, parameter:
1. ${}:Log count (${([A-Za-z0-9_.]+)})
2. %{}:Json path with "." split (%{([A-Za-z0-9_.]+)})
- 1. ${SignInFailed_Count}/${SignInTotal_Count}
2. %{aggregations.upstream_time.avg_value.value}
parameters The filter json content on Create Monitor Item - step1 - -
alarms The alarm criteria, will trigger by calculation result of formula 1. alert
2. warning
{"alert": ">=0.1","warning": ">=0.05"}
notify The notification method 1. slack {"slack": "${SLACK_WebHook_URL}"}