Push

Push is a Real Time Reactive Push Messaging API highly inspired by Netflix 's Zuul Push, but with some differences. It was written in Kotlin with the Spring framework and it uses Redis as a message broker. Push provides WebSocket and Server-Sent Events endpoints that can be used to subscribe to real-time messages in a web environment. It also provides an HTTP endpoint to publish messages.

Architecture

Push was designed with the goal of being scalable and possibly used in a distributed environment, it uses Redis as message broker.

The following diagram demonstrates the architecture behind Push in a Kubernetes environment, where we can create a cluster for Redis with master and slave nodes in order to have horizontal scalability and resiliency and also multiple instances of the application itself for the same purpose.

Push Architecture

This section describes the architecture of Push through a component diagram.

Deployment Architecture

Push takes advantage of the "Deployment" workload API from Kubernetes. This section describes how it works under the hood. A Deployment provides declarative updates for Pods and ReplicaSets and is used to tell Kubernetes how to create or modify instances of the pods that hold a containerized application. Deployments can scale the number of replica pods, enable rollout of updated code in a controlled manner, or roll back to an earlier deployment version if necessary.

StatefulSet Architecture

Push makes use of the "StatefulSet" workload API to manage the Redis cluster. This section describes how a kubernetes StatefulSet works under the hood. StatefulSet is the workload API object used to manage stateful applications. It manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods. Similar to a Deployment, a StatefulSet manages Pods that are based on an identical container spec.

Getting Started

Required software:

1. docker
2. minikube
3. kubectl

If you also want to compile the code directly on your machine you'll also need:

1. java
2. gradle

Building from Source

To compile the code simply run

$ gradle clean build -x test -x compileAotMainJava

Running Unit Tests

Locally

$ gradle clean build -x compileAotMainJava

With Docker

$ docker-compose -f docker-compose.test.yml build unit-tests && docker-compose -f docker-compose.test.yml run unit-tests

Running Integration Tests

1. Starting the needed dependencies:

$ docker-compose -f docker-compose.test.yml up -d redis

Locally

$ gradle clean build integrationTest -x compileAotMainJava

With Docker

$ docker-compose -f docker-compose.test.yml build integration-tests && docker-compose -f docker-compose.test.yml run integration-tests

Running the Application

Locally

1. Starting the needed dependencies:

$ docker-compose up -d redis

2. Booting the application

$ gradle clean build bootRun -x compileAotMainJava

With Docker

1. Booting

$ docker-compose up --build -d push

2. Terminating

$ docker-compose down

With Kubernetes

1. Booting

$ ./k8s/init-cluster.sh && kubectl port-forward svc/push 8080:8080 8000:8000

2. Terminating

$ kubectl delete -f ./k8s/cluster.yml

Testing

1. Publishing messages to channel qwerty

$ while true; do curl -H 'Content-Type: application/json' --request POST --data '{"channel":"qwerty","message":"a message to channel 'qwerty'"}' http://localhost:8080/messages ; done
# {"subscribers":0}

2. Subscribing to channel qwerty and channel xyz by SSE (Server-Sent Events)

$ curl 'http://localhost:8080/sse/messages?channels=xyz,qwerty'
# event:heartbeat
# data:{}

# event:message
# data:{"channel":"qwerty","message":"a message to channel qwerty"}

3. Subscribing to channel qwerty and channel xyz by WS (WebSockets)

$ websocat -v 'ws://localhost:8080/ws/messages?channels=qwerty,xyz'
# [INFO  websocat::lints] Auto-inserting the line mode
# [INFO  websocat::stdio_threaded_peer] get_stdio_peer (threaded)
# [INFO  websocat::ws_client_peer] get_ws_client_peer
# [INFO  websocat::ws_client_peer] Connected to ws
# {"event":"ping","data":{}}
# {"event":"message","data":{"channel":"qwerty","message":"a message to channel 'qwerty'"}}

Environment Variables

Name	Description	Default value
push.reconnect.dither.min.duration	Minimum value for a randomization window for each client's max connection lifetime. Helps in spreading subsequent client reconnects across time.	120s (120 seconds / 2 minutes)
push.reconnect.dither.max.duration	Maximum value for a randomization window for each client's max connection lifetime. Helps in spreading subsequent client reconnects across time.	180s (180 seconds / 3 minutes)
push.client.close.grace.period.duration	Time the server will wait for the client to close the connection or respond to a "ping" before it closes it forcefully from its side	4s (4 seconds)
push.heartbeat.interval.duration	Interval for when the server will emit heartbeats or pings to the client	30s (Every 30 seconds)

How Push Works

Both Server-Sent Events and WebSockets require persistent connections, so there are additional challenges that need to be addressed in order to provide scalability to the application.

// TODO complete

To Do List

• Add unit and integration tests
• Write documentation
• Dockerize application
• Add observability and monitoring
• Add heartbeats, timeouts and reconnection events to the SSE endpoint
• Add ping-pongs, timeouts and reconnection events to the WS endpoint
• Create Kubernetes cluster with redundancy

Improvements

• Automatically configure cluster for Redis
  • Configure horizontal pod autoscaler for Redis
• Configure horizontal pod autoscaler for Push based on custom metrics (e.g. throughput, number of concurrent connections, etc)
• Add Sharded Pub/Sub capacity
• Use Terraform to deploy to Kubernetes
• Add "acks" to websockets messages
• Configure native compilation

Built with

References

• https://github.com/Netflix/zuul/wiki/Push-Messaging
• https://www.youtube.com/watch?v=IdR6N9B-S1E
• https://github.com/rustudorcalin/deploying-redis-cluster
• https://www.vmware.com/topics/glossary/content/kubernetes-deployment.html
• https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
• https://congdonglinux.com/rolling-updates-and-rollbacks-in-kubernetes/
• https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
• https://jayendrapatil.com/kubernetes-components/