Skip to content

Clustered Elixir OTP application on Kubernetes with Horde and LibCluster

License

Notifications You must be signed in to change notification settings

haighis/ex_cluster

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ExCluster

ExCluster is a demo application built with Elixir and OTP to show how to integrate Distributed Elixir on Kubernetes with Horde, LibCluster, and Distillery. The application mimics a service in charge of processing orders, a GenServer models a order and stores in it's state a list of integers as order contents. All state is kept in-memory, and using Distributed Elixir and Horde when a Node goes down gracefully, processes are restarted throughout the remaining Cluster Nodes (with process state transferred). This is integrated on Kubernetes via a HeadlessService setup with LibCluster.

Running Locally - Single Node

You can run the application locally simply via iex -S mix, this will run a single Node without any clustering.

Running Locally - MultiNode

To run a Cluster locally, run the application multiple times with unique Node names and a consistent cookie specified via ERL_FLAGS, for example this will run a 3 Node cluster:

# Terminal 1
$ ERL_FLAGS="-name [email protected] -setcookie cookie" NODES="[email protected],[email protected]" iex -S mix

# Terminal 2
$ ERL_FLAGS="-name [email protected] -setcookie cookie" NODES="[email protected],[email protected]" iex -S mix

# Terminal 3
$ ERL_FLAGS="-name [email protected] -setcookie cookie" NODES="[email protected],[email protected]" iex -S mix

Running on minikube

To run on minikube, start it up and then deploy the charts:

$ docker build -t ex_cluster:local .
$ eval $(minikube docker-env)
$ kubectl create -f k8s/service-headless.yaml
$ kubectl create -f k8s/deployment.yaml
$ kubectl get pods

Exec onto one, attach to the console, and create a few orders:
$ kubectl exec -it ex-cluster-5b45c8577b-jltm9 /bin/bash
[in the pod shell]$ _build/prod/rel/ex_cluster/bin/ex_cluster remote_console
[should now be attached to iex on the Pod]
iex([email protected])1> Horde.Supervisor.start_child(ExCluster.OrderSupervisor, { ExCluster.Order, "John" })
iex([email protected])3> ExCluster.Order.add("John", [1,2,3])
:ok
iex([email protected])4> ExCluster.Order.contents("John")
[1, 2, 3]

Terminate the Pod where the order is running:
$ kubectl delete pod ex-cluster-5b45c8577b-jltm9

A new Pod should have been automatically spun up, with the terminated Pod now going down. Exec onto one of the two running Pods, and check on the order:
$ kubectl get pods
NAME                          READY     STATUS        RESTARTS   AGE
ex-cluster-5b45c8577b-8cg9c   1/1       Running       0          3m
ex-cluster-5b45c8577b-d4cm8   1/1       Running       0          5s
ex-cluster-5b45c8577b-fkfj2   1/1       Terminating   0          3m
$ kubectl exec -it ex-cluster-5b45c8577b-d4cm8 /bin/bash
[in the pod shell]$ _build/prod/rel/ex_cluster/bin/ex_cluster remote_console
[should now be attached to iex on the Pod]
iex([email protected])1> ExCluster.Order.contents("John")
[1, 2, 3]
There you go, the order was preserved through Pod tear down!

You can now get Pods, exec onto one and create orders, kill the Pod, and view the restarted order process with state still alive in the Cluster.

About

Clustered Elixir OTP application on Kubernetes with Horde and LibCluster

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Elixir 96.7%
  • Dockerfile 3.3%