Skip to content

engapa/kafka-k8s-openshift

Repository files navigation

Kafka Docker Image

Build status Docker Pulls Docker Layering Docker Version OSS

This project is meant to create an optimised docker image to run kafka containers as 'statefulset' into kubernetes/openshift.

Obviously, the docker image can be used locally for testing or development purposes.

Build and publish a kafka docker image

To get a docker image ready with default values type:

$ make clean-all docker-build docker-test docker-push 

To get your own image:

$ export KAFKA_HOME="/opt/kafka"
$ export SCALA_VERSION="2.13"
$ export KAFKA_VERSION="2.5.0"
$ docker build --build-arg SCALA_VERSION=$SCALA_VERSION --build-arg KAFKA_VERSION=$KAFKA_VERSION --build-arg KAFKA_HOME=$KAFKA_HOME \
-t your-org/kafka:${SCALA_VERSION}-${KAFKA_VERSION} .

NOTE: build-args are optional arguments if you want different values from default ones in the Dockerfile

The built docker image will contain a kafka distribution (${SCALA_VERSION}-${KAFKA_VERSION}) under the directory $KAFKA_HOME.

The provided scripts are:

  • kafka_download.sh : This script is used to download the suitable release.
  • kafka_env.sh : It purpose is load the default environments variables.
  • kafka_setup.sh : Configure kafka and zookeeper dynamically , based on utils-docker project
  • kafka_server.sh : A central script to manage kafka and optional zookeeper processes.
  • kafka_server_status.sh : Checks kafka server status.

Public docker images are available HERE

Getting started with a single docker container locally

The example bellow shows you how to run an all-in-one docker kafka container (with zookeeper as internal sidecar):

$ docker run -it -p 9092:9092 -p 2181:2181 \
  -e "SETUP_DEBUG=true" \
  -e "SERVER_advertised_listeners=PLAINTEXT://localhost:9092" \
  -e "SERVER_listener_security_protocol_map=PLAINTEXT:PLAINTEXT" \
  -e "SERVER_listeners=PLAINTEXT://0.0.0.0:9092" \
  -h kafka engapa/kafka:${SCALA_VERSION}-${KAFKA_VERSION}

Writing environment variables to file :

PREFIX           : SERVER_
DEST_FILE        : /opt/kafka/config/server.properties
EXCLUSIONS       :
CREATE_FILE      : true
OVERRIDE         : true
FROM_SEPARATOR   : _
TO_SEPARATOR     : .
LOWER            : true
.......................................

[DEBUG] [2017-01-31_20:17:26] -  [OVERRIDE] : SERVER_log_dirs --> log.dirs=/opt/kafka/logs
[DEBUG] [2017-01-31_20:17:26] -  [OVERRIDE] : SERVER_zookeeper_connect --> zookeeper.connect=localhost:2181
[DEBUG] [2017-01-31_20:17:26] -  [OVERRIDE] : SERVER_broker_id --> broker.id=-1
.......................................

Writing environment variables to file :

PREFIX           : ZK_
DEST_FILE        : /opt/kafka/config/zookeeper.properties
EXCLUSIONS       :
CREATE_FILE      : true
OVERRIDE         : true
FROM_SEPARATOR   : _
TO_SEPARATOR     : .
LOWER            : false
.......................................

[DEBUG] [2017-01-31_20:17:26] -  [OVERRIDE] : ZK_dataDir --> dataDir=/opt/kafka/zookeeper/data
[DEBUG] [2017-01-31_20:17:26] -  [OVERRIDE] : ZK_clientPort --> clientPort=2181
[DEBUG] [2017-01-31_20:17:26] -  [  ADD   ] : ZK_dataLogDir --> dataLogDir=/opt/kafka/zookeeper/data-log
...
[2017-01-31 20:17:28,150] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2017-01-31 20:17:28,308] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x159f62cc8c00000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
...
[2017-01-31 20:17:29,646] INFO Kafka version : 2.3.0 (org.apache.kafka.common.utils.AppInfoParser)
[2017-01-31 20:17:29,646] INFO Kafka commitId : f10ef2720b03b247 (org.apache.kafka.common.utils.AppInfoParser)
[2017-01-31 20:17:29,647] INFO [Kafka Server 1001], started (kafka.server.KafkaServer)

NOTE: We've passed a SETUP_DEBUG environment variable (SETUP_DEBUG=true) to view the setup process details.

Setting up

Users can provide parameters to config files just adding environment variables with specific name patterns.

This table collects the patterns of variable names which will are written in each file:

PREFIX FILE (${KAFKA_HOME}/config) Example
SERVER_ server.properties SERVER_broker_id=1 --> broker.id=1
LOG4J_ log4j.properties LOG4J_log4j_rootLogger=INFO, stdout--> log4j.rootLogger=INFO, stdout
CONSUMER_ consumer.properties CONSUMER_zookeeper_connect=127.0.0.1:2181 --> zookeeper.connect=127.0.0.1:2181
PRODUCER_ producer.properties PRODUCER_compression_type=none --> compression.type=none
ZK_ zookeeper.properties ZK_maxClientCnxns=0 --> maxClientCnxns=0
CONN_CONSOLE_SINK_ connect-console-sink.properties CONN_CONSOLE_SINK_tasks_max=1 --> tasks.max=1
CONN_CONSOLE_SOURCE_ connect-console-source.properties CONN_CONSOLE_SOURCE_topic=connect-test --> topic=connect-test
CONN_DISTRIB_ connect-distributed.properties CONN_DISTRIB_group_id=connect-cluster --> group.id=connect-cluster
CONN_FILE_SINK_ connect-file-sink.properties CONN_FILE_SINK_connector_class=FileStreamSink --> connector.class=FileStreamSink
CONN_FILE_SOURCE_ connect-file-source.properties CONN_FILE_SOURCE_tasks_max=1 --> tasks.max=1
CONN_LOG4J_ connect-log4j.properties CONN_LOG4J_log4j_rootLogger=INFO, stdout --> log4j.rootLogger=INFO, stdout
CONN_STANDALONE_ connect-standalone.properties CONN_STANDALONE_bootstrap_servers=localhost:9092 --> bootstrap.servers=localhost:9092
TOOLS_LOG4J_ tools-log4j.properties TOOLS_LOG4J_log4j_appender_stderr_Target=System.err --> log4j.appender.stderr.Target=System.err

So we can configure our kafka server via environment variables directly:

$ docker run -it -d -e "LOG4J_log4j_rootLogger=DEBUG, stdout" -e "SERVER_log_retention_hours=24"\
engapa/kafka:${SCALA_VERSION}-${KAFKA_VERSION}

Also you may use --env-file option to load these variables from a file.

And, of course, you could provide your own property files directly by option -v with the suitable properties files.

The override option of kafka server is preserved and anybody can use it on this way:

$ docker run -it \
  -e "SETUP_DEBUG=true" \
  -h kafka engapa/kafka:${SCALA_VERSION}-${KAFKA_VERSION} \
 /bin/bash -c "kafka_server.sh start --override advertised.host.name=kafka"
 [2017-02-04 19:06:10,504] INFO KafkaConfig values:
	advertised.host.name = kafka
...
[2017-02-04 19:06:11,693] INFO [Kafka Server 1001], started (kafka.server.KafkaServer)

Run local zookeeper

By default a zookeeper process is started too, as we said previously. This behaviour is managed by the env variable KAFKA_ZK_LOCAL (defaults to "true").

External zookeeper

If you want to deploy a kafka server without a local zookeeper then you should provide these env values:

  • KAFKA_ZK_LOCAL=false
  • SERVER_zookeeper_connect=<zookeeper_host:zookeeper_port>[,<zookeeper_host:zookeeper_port>]

For instance:

$ docker run -it \
  -e "KAFKA_ZK_LOCAL=false" \
  -e "SERVER_zookeeper_connect=zookeeperserver1:2181,zookeeperserver2:2181,zookeeperserver3:2181" \
  engapa/kafka:${SCALA_VERSION}-${KAFKA_VERSION}

Kubernetes

In k8s directory there are some examples and utilities for Kubernetes

Openshift

In openshift directory there are some resources for Openshift.

Extra Dockers

Another great kafka docker images can be found at:

Author

Enrique Garcia [email protected]