EFK Kubernetes Deployment

A set of yaml for Kubernetes deployment of EFK stack (Elasticsearch, Fluentd and Kibana).

Successfully tested on Azure Kubernetes Services.

Quickstart

Step 0: Please make sure you have Kubernetes installed, and you have the cluster admin permission.

Step 1: Deploy ECK (Elastic Cloud on Kubernetes), following the latest official guide, or just input the following.

kubectl apply -f https://download.elastic.co/downloads/eck/1.7.0/crds.yaml -f https://download.elastic.co/downloads/eck/1.7.0/operator.yaml

Step 2: Create logging namespace and deploy Elasticsearch and Kibana.

kubectl create namespace logging
kubectl apply -f elasticsearch-kibana_default.yaml    # with default storage

Step 3: Replace actual elasticsearch password in fluentd.yaml and deploy Fluentd.

# Get elasticsearch password
kubectl get secret efk-es-elastic-user -n logging -o=jsonpath='{.data.elastic}' | base64 --decode; echo
kubectl apply -f fluentd.yaml

Visualization

To visit Kibana or Elasticsearch from local, Kubernetes services need to be forwarded.

# Forward Kibana
kubectl port-forward service/efk-kb-http -n logging 5601
# Forward Elasticsearch
kubectl port-forward service/efk-es-http -n logging 9200

Customized Usage

Changing location for Elasticsearch data storage

In Step 2, Elasticsearch data is default stored by Kubernetes default storage class. If you wish to store elsewhere, please follow the ECK instructions here. And there is also provided an example for using Azure Storage Account, using the following command instead.

kubectl apply -f elasticsearch-kibana_azurestorage.yaml    # with Azure Storage Account

Changing rules for Fluentd

In Step 3, Fluentd is set to collect node-level logs on each node. The logs contained stdout of all pods, see explanation. You can also customize the Fluentd config following official documentation.