Skip to content

wizart-tech/docker-airflow

Repository files navigation

Dockerized Apache Airflow with Postgres Backend.

This Docker image contains Airflow 2.0 with minimal dependencies which makes the image lighter and more customizable.

All the docker-compose*.yml examples were developed taking into account all the best practices that are used for deploying docker containers in our company - Wizart Tech.

Prerequisites

Supported Executors

  • LocalExecutor
  • CeleryExecutor

Security

  1. In order to secure Airflow Connections and Variables, use fernet_key encryption. It can be done using Python:

    >> from cryptography.fernet import Fernet
    >> Fernet.generate_key().decode()
    '=69ksvvORDpeoBrz2N38El18kOxJFPU2peg22So66k7U=' # here is your fernet key
  2. Specify the secret_key setting under the [webserver] config. Change this value to a new, per-environment, randomly generated string.

    For example using this command openssl rand -hex 30

Store the generated keys in env_file and name It airflow.env, just like It shown below:

AIRFLOW__CORE__FERNET_KEY=69ksvvORDpeoBrz2N38El18kOxJFPU2peg22So66k7U=
AIRFLOW__WEBSERVER__SECRET_KEY=1ca384d704f852756df25a7560c3338cb3a65cccf2fd734440f94deb5d32

Note: You can use airflow.env to define any container-level configurations for Airflow.

Additional information about securing connections can be found here.

Usage

For the first run, build containers using predefined make shortcuts:

make up

to list all available make shortcuts, type

make help

Create Users

docker-compose run --rm webserver bash  # or `make shell-root`
airflow users create \
    --role Admin \
    --username admin \
    --firstname FIRST_NAME \
    --lastname LAST_NAME \
    --email [email protected]

Executing airflow commands

If you want to run any of airflow commands, you can do the following: docker-compose run --rm webserver [some command]

  • docker-compose run --rm webserver airflow dags list - List dags
  • docker-compose run --rm webserver airflow tasks test [DAG_ID] [TASK_ID] [EXECUTION_DATE] - Test specific task
  • docker-compose run --rm webserver python /usr/local/airflow/dags/[PYTHON-FILE].py - Test custom python script