Skip to content

Kubernetes application example. Does not drop connections when terminating. Implemented in Go.

Notifications You must be signed in to change notification settings

integrii/go-k8s-graceful-termination

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

25 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Overview

This is an example Go appliation and self-paced tutorial for implementing the graceful shutdown of Kubernetes pods. This Go application does not drop connections when terminating due to proper shutdown signal trapping. If your app does not capture shutdown signals properly today, your service probably drops connections for users when your pods are re-deployed or removed for any reason.

image

This follows the best practices for Kubernetes web services by implementing shutdown signal capturing that keeps the pod alive while connections are still ariving to it. Once a pod goes into the Terminating state, the pod is removed from the Kubernetes endpoints list. When an endpoint is removed, the Kubernetes cluster will reconfigure to remove it from all load balancing. Only after that process completes can your pod be removed gracefully. You can find detailed documentation from Kubernetes about this.

Check out the go code here

This should be combined with a pod disruption budget to restrict how many pods can be unavailable at one time as well as a pod anti-affinity policy to stripe your pods across nodes for best production resiliency.

Order of operations

This is how graceful removal of pods from load balancers should look:

  • The Kubernetes API is sent a delete command for a pod and changes the pod to the Terminating state
  • Endpoints for the pod are removed from the Kubernetes API
  • The kubelet responsible for this pod instructs the CRI to stop the containers in this pod
  • The CRI sends a shutdown signal to the containerized processes
  • The containerized process catches this signal gracefully
    • The containerized application continues serving requests that find their way to it for 10 seconds
      • The duration of time of this delay must be less than the terminationGracePeriodSeconds of its pod spec
  • Meanwhile, the Kubernetes cluster finishes reconfiguration that removes the pod from the flow of service traffic
  • The pod exits gracefully

Example Spec

This spec is in this repo as kubernetes.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: graceful-shutdown-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: graceful-shutdown-app
  template:
    metadata:
      labels:
        app: graceful-shutdown-app
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: graceful-shutdown-app
        image: integrii/go-k8s-graceful-termination:latest
        livenessProbe:
          httpGet:
            path: /alive
            port: 8080
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: 128Mi
            cpu: 500m
          limits:
            cpu: 1
            memory: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: graceful-shutdown-app
spec:
  ports:
  - name: "8080"
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: graceful-shutdown-app
  type: NodePort

Try it yourself

You can test this graceful shutdown yourself. Clone this repo and try the following:

kubectl create ns graceful-termination
kubectl -n graceful-termination apply -f https://raw.githubusercontent.com/integrii/go-k8s-graceful-termination/main/kubernetes.yaml
<wait for service to come online>
kubectl -n graceful-termination port-forward service/graceful-shutdown-app 8080 (in another terminal)
kubectl -n graceful-termination logs -f -l app=graceful-shutdown-app (in another terminal)
for i in `seq 1 100000`; do 
   curl -v http://localhost:8080 
done (in another terminal)
kubectl -n graceful-termination set env deployment/graceful-shutdown-app TEST=`date +%s` (this will cause a rolling update to the deployment)
watch kubectl -n graceful-termination get pods
<observe terminal doing curl tests>
kubectl delete namespace graceful-termination (when you're done with everything)

You should not see dropped connections during the rolling update, even though there is only one pod!

Some Closing Notes

It is too common that I have seen applications not take care when being removed from the flow of traffic, resulting in connection failures. Hopefully this clears things up. This process has always existed, even with traditional load balncers, and in those situations it remains a regular procedure to remove backends from the load balancer before bringing down those applications.

You also could alternatively do a graceful shutdown integration using preStop hooks, which can be configured to send a web request to your application before it is sent a termination signal - but that approach wasn't covered here.

About

Kubernetes application example. Does not drop connections when terminating. Implemented in Go.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published