Skip to content

Kubernetes operator for managing the lifecycle of Apache Hadoop Yarn Tasks on Kubernetes.

Notifications You must be signed in to change notification settings

chriskery/hadoop-operator

Repository files navigation

Hadoop Operator

Go Report Build Status Coverage Status Version

Overview

Hadoop is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, providing a highly-available service on top of a cluster of computers, each of which may be prone to failures

The hadoop Operator manages hadoop clusters deployed to Kubernetes, and provides a cloud-native way to submit Hadoop Yarn jobs.

Architecture Diagram

Install the Operator

Cert Manager

hadoop operator relies on cert-manager, you must have cert-manager installed and configured in your Kubernetes cluster. cert-manager is a Kubernetes add-on that automates the management of certificates and is a requirement for this Operator to function correctly.

Installing cert-manager:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.3/cert-manager.yaml

📖 Read more about installing cert-manager using kubectl apply and static manifests.

Master Branch

kubectl apply -k github.com/chriskery/hadoop-operator/manifests/default

Quick Start

Please refer to the quick-start.md for more information.

Features

Contributing and Community

We thrive to build a welcoming and open community for anyone who wants to use the operator or contribute to it.