Docker Apache NiFi

Warning

These images are no longer maintained as of October 6th, 2017. It is recommended to use the offical Docker images from the Apache NiFi team.

This is an unofficial Apache NiFi Docker image. The images are built by Travis CI and are available on Docker Hub at mkobit/nifi.

Table of Contents

• Images
• Example
• Building Images
  • Requirements
  • Building
  • Templates
  • Travis
• Building Extensions
• Volumes
• Official Apache NiFi Documentation and Guides
• ListenHTTP Processor
• Usage
• Issues
• Contributing to this repository
• Contributing to NiFi

Images

Image	Image Size (by imagelayers.io)
mkobit/nifi:latest
mkobit/nifi:1.4.0-openjdk-8
mkobit/nifi:1.3.0-openjdk-8

Example

To try out NiFi on Docker:

1. Pull the latest image

   docker pull mkobit/nifi

2. Run image and expose the default ports to the host.

   # --rm : remove container on exit
   # -i : interactive
   # -t : allocate TTY
   # -p : publish each container port to host port. format: ip:hostPort:containerPort | ip::containerPort | hostPort:containerPort | containerPort
   docker run -it --rm -p 8080-8081:8080-8081 mkobit/nifi

3. Go to default local NiFi endpoint in browser - http://localhost:8080/nifi/

Building Images

Requirements

• Docker

• Python 3.5

Building

The main entrypoint for building is the make_docker.py file. The Python script makes use of subprocess, string.Template, and argparse.

The build is essentially divided into 3 phases, which are all mapped to different subcommands:

1. generate - generate the Dockerfile to be used

2. build - run docker build and tags images

3. push - Push up images to Docker Hub

For help, simple run the script with the --help argument:

make_docker.py Help

./make_docker.py --help

To get subcommand help, you can specify --help for that option:

make_docker.py Subcommand Help

./make_docker.py generate --help

You can see the main script uses from the _.travis.yml file.

Build example

Here is an example that would be executed on the CI system:

make_docker.py build

./make_docker.py build \
    --template-substitutions "base_image=java:8-jre,nifi_version=0.7.0" \
    --template-file templates/Dockerfile-template \
    --destination-file 0.7.0-openjdk-8.dockerfile \
    --repository mkobit/nifi \
    --tags 0.7.0-openjdk-8,latest

Templates

The Dockerfile templates can be found in the templates directory.

Travis

The Travis build entrypoint is the _travis_build.sh script. This script determines if it is a pull request, merge into master, or any other branch.

Pull requests are built, while commits to master trigger a build and publish to Docker Hub.

Travis utilities

There is a Travis-CI utility Docker image that installs the travis RubyGem. There are a couple utility scripts to build and run that image:

• build-travis-docker.sh - build the Dockerfile-travis image

• exec-travis-docker.sh - runs the Dockerfile-travis image, with default arguments to lint this repository’s .travis.yml file

Building Extensions

This can easily be used as a base image to create NiFi applications. One way would be to build a NAR with extensions built upon the framework and including it in your own image. I recommend reading the Developer’s Guide if you want to head down this route.

Volumes

These are the default locations as specified by the Apache NiFi properties. You can find more information about each of these repositories on the System Administration Guide.

• $NIFI_HOME/database_repository - user access and flow controller history

• $NIFI_HOME/flowfile_repository - FlowFile attributes and current state in the system

• $NIFI_HOME/content_repository - content for all the FlowFiles in the system

• $NIFI_HOME/provenance_repository - information related to Data Provenance

Official Apache NiFi Documentation and Guides

• Overview

• User Guide

• Expression Language

• Development Quickstart

• Developer’s Guide

• System Administrator

ListenHTTP Processor

The standard library has a built-in processor for an HTTP endpoint listener. That processor is named ListenHTTP. You should set the Listening Port of the instantiated processor to 8081 if you follow the instructions from above.

Usage

This image can either be used as a base image for building on top of NiFi or just to experiment with. I personally have not attempted to use this in a production use case.

Issues

If you have any problems, comments, or questions with this image, feel free to reach out at mkobit/docker-nifi. If you have Apache NiFi specific questions or concerns you can reach out on one of the community mailing lists.

Contributing to this repository

Contributing changes to this repository is extremely welcome. If it is a larger change, it is usually best to discuss your plans first. Please see the issues to see if a similar issue already exists.

Contributing to NiFi

The Apache NiFi source code can be found on Github at apache/nifi. You can browse issues related to the project on the Apache NiFi Jira.