Skip to content

Collect and filter docker journald using fluentd, in a kubernetes cluster.

License

Notifications You must be signed in to change notification settings

aliasmee/fluentd-journald-elasticsearch

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Fluentd-journald-elasticsearch

Collect and filter docker journald using fluentd, in a kubernetes cluster.

Getting Started

These instructions will cover usage information and for the docker container

Prerequisities

In order to run this container you'll need docker installed.

kubernetes cluster

  • Docker Logging Driver on Node: journald
  • Reliable elasticsearch cluster

Usage

Container Parameters

Run the container and mount the host directory /run/log/journal to the container's corresponding directory (This example: container's dir is /run/log/journal)

docker run -it --rm -v /run/log/journal:/run/log/journal fluentd-journald-elasticsearch

Note: The default log is output to host elasticsearch_logging: 9200

You can modify the default environment variable to specify the elasticsearch host when running the container. e.g:

docker run -it --rm -v /run/log/journal:/run/log/journal -e "FLUENT_ELASTICSEARCH_HOST=es-cluster-ip" fluentd-journald-elasticsearch

One of the log examples:

{
"_index": "kubernetes-2018.01.28",
"_type": "fluentd",
"_id": "HMnVO2EBv8K-t54ZDxjW",
"_version": 1,
"_score": 1,
"_source": {
"Node": "dce-k8s-worker3",
"PRIORITY": "6",
"TRANSPORT": "journal",
"PID": "7630",
"COMM": "dockerd",
"CAP_EFFECTIVE": "3fffffffff",
"SYSTEMD_UNIT": "docker.service",
"BOOT_ID": "9c6a0630abe645f99ca1c4c88c196573",
"MACHINE_ID": "617f06359788ed0e86d064455995028d",
"CONTAINER_ID": "ad0f351daf0f",
"CONTAINER_ID_FULL": "ad0f351daf0f0c2429bbf53c5a980a27909c426fbeab50774ea9e5a14c08a27d",
"CONTAINER_NAME": "k8s_calico-node_calico-node-4vz7c_kube-system_bd7f55e3-0277-11e8-8327-0242ac130006_2",
"CONTAINER_TAG": "ad0f351daf0f",
"MESSAGE": "2018-01-28 03:30:55.221 [INFO][159] int_dataplane.go 705: Finished applying updates to dataplane. msecToApply=1.847487",
"SOURCE_REALTIME_TIMESTAMP": "1517110255221523",
"docker": {
"container_id": "ad0f351daf0f0c2429bbf53c5a980a27909c426fbeab50774ea9e5a14c08a27d"
},
"kubernetes": {
"container_name": "calico-node",
"namespace_name": "kube-system",
"pod_name": "calico-node-4vz7c"
},
"@timestamp": "2018-01-28T03:30:55.221677000+00:00"
}
}

Environment Variables

The following is the current dynamic environment variables

  • FLUENT_ELASTICSEARCH_HOST - elasticsearch cluster ip address (the default value is elasticsearch_logging)
  • FLUENT_ELASTICSEARCH_PORT - elasticsearch cluster port (default value is 9200 port)
  • FLUENT_ELASTICSEARCH_USER - elasticsearch certified user [Optional]
  • FLUENT_ELASTICSEARCH_PASSWORD - elasticsearch certified user's password [Optional]

Volumes

  • /run/log/journal - OS systemd journald location, Or in the /var/log/journal directory,according to actual condition.

Useful File Locations

  • /bin/entrypoint.sh - You also modify this script to add dynamic environment variables.

Built With

Find Us

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Authors

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Acknowledgments

About

Collect and filter docker journald using fluentd, in a kubernetes cluster.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages