Skip to content

Applying the lessons learned from "Kubernetes The Hard Way" with cilium as CNI

License

Notifications You must be signed in to change notification settings

missingcharacter/kubernetes-the-hard-way-the-easy-way

Repository files navigation

Kubernetes The Hard Way The Easy Way

Creates a Single Controller Kubernetes cluster using multipass and cilium

All scripts are available to learn how it is built.

Specs

  • Ubuntu 22.04
  • Kubernetes 1.28.4
  • etcd 3.5.10
  • containerd 1.7.10
  • cni plugins 1.4.0
  • cilium 1.14.4 (via helm chart)
  • coredns 1.28.2 (via helm chart)

Requirements

  • helm 3+
    • linux: here
    • mac: brew install helm
  • curl
    • linux: sudo apt install curl
    • mac: brew install curl
  • ipcalc
    • linux: sudo apt install ipcalc
    • mac: brew install ipcalc
  • multipass
    • linux: sudo snap install multipass --classic
    • mac: brew cask install multipass
  • cfssl and cfssljson
    • linux:

      wget -q --show-progress --https-only --timestamping \
        https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/linux/cfssl \
        https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/linux/cfssljson
      chmod +x cfssl cfssljson
      sudo mv cfssl cfssljson /usr/local/bin/
    • mac: brew install cfssl

  • kubectl
    • linux:

      wget https://storage.googleapis.com/kubernetes-release/release/v1.18.5/bin/linux/amd64/kubectl
      chmod +x kubectl
      sudo mv kubectl /usr/local/bin/
    • mac:

      curl -o kubectl https://storage.googleapis.com/kubernetes-release/release/v1.18.5/bin/darwin/amd64/kubectl
      chmod +x kubectl
      sudo mv kubectl /usr/local/bin/

Procedure

  1. Create your machines:

    ./setup.sh
  2. Wait a couple of minutes for cilium and coredns to start working

    $  kubectl get all --all-namespaces
    NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
    kube-system   pod/cilium-25lnk                       1/1     Running   0          73s
    kube-system   pod/cilium-d8p59                       1/1     Running   0          74s
    kube-system   pod/cilium-operator-67f9b484d8-ksvzj   1/1     Running   0          74s
    kube-system   pod/cilium-operator-67f9b484d8-mqvsq   1/1     Running   0          74s
    kube-system   pod/coredns-coredns-67c645769c-klwlk   1/1     Running   0          73s
    kube-system   pod/hubble-relay-85664d47c4-ftd5n      1/1     Running   0          74s
    kube-system   pod/hubble-ui-7c6789876c-7b9mh         3/3     Running   0          74s
    
    NAMESPACE     NAME                      TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
    default       service/kubernetes        ClusterIP   172.17.0.1    <none>        443/TCP         2m31s
    kube-system   service/coredns-coredns   ClusterIP   172.17.0.10   <none>        53/UDP,53/TCP   73s
    kube-system   service/hubble-relay      ClusterIP   172.17.0.83   <none>        80/TCP          74s
    kube-system   service/hubble-ui         ClusterIP   172.17.0.35   <none>        80/TCP          74s
    
    NAMESPACE     NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    kube-system   daemonset.apps/cilium   2         2         2       2            2           <none>          74s
    
    NAMESPACE     NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
    kube-system   deployment.apps/cilium-operator   2/2     2            2           74s
    kube-system   deployment.apps/coredns-coredns   1/1     1            1           73s
    kube-system   deployment.apps/hubble-relay      1/1     1            1           74s
    kube-system   deployment.apps/hubble-ui         1/1     1            1           74s
    
    NAMESPACE     NAME                                         DESIRED   CURRENT   READY   AGE
    kube-system   replicaset.apps/cilium-operator-67f9b484d8   2         2         2       74s
    kube-system   replicaset.apps/coredns-coredns-67c645769c   1         1         1       73s
    kube-system   replicaset.apps/hubble-relay-85664d47c4      1         1         1       74s
    kube-system   replicaset.apps/hubble-ui-7c6789876c         1         1         1       74s
  3. Your cluster is ready, lets verify data encryption works

    $ kubectl create secret \
      generic kubernetes-the-hard-way --from-literal="mykey=mydata"
    $ multipass exec controller-k8s -- sudo ETCDCTL_API=3 etcdctl get \
      --endpoints=https://127.0.0.1:2379 --cacert=/etc/etcd/ca.pem \
      --cert=/etc/etcd/kubernetes.pem --key=/etc/etcd/kubernetes-key.pem \
      /registry/secrets/default/kubernetes-the-hard-way | hexdump -C
    00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
    00000010  73 2f 64 65 66 61 75 6c  74 2f 6b 75 62 65 72 6e  |s/default/kubern|
    00000020  65 74 65 73 2d 74 68 65  2d 68 61 72 64 2d 77 61  |etes-the-hard-wa|
    00000030  79 0a 6b 38 73 3a 65 6e  63 3a 61 65 73 63 62 63  |y.k8s:enc:aescbc|
    00000040  3a 76 31 3a 6b 65 79 31  3a 61 bb c0 45 f2 df 88  |:v1:key1:a..E...|
    00000050  36 46 05 df c1 df 26 e1  e0 59 18 9f 7d 51 7a d9  |6F....&..Y..}Qz.|
    00000060  28 0d 03 4e c3 14 55 01  51 d6 aa cc 50 21 a5 09  |(..N..U.Q...P!..|
    00000070  86 92 89 9b 33 82 43 09  7d 5b fe bb 68 45 43 48  |....3.C.}[..hECH|
    00000080  96 9a 1e a8 88 30 82 a8  2c d8 26 ea 12 19 58 da  |.....0..,.&...X.|
    00000090  3a 25 ed 6b 47 1f e2 e9  31 91 e6 cf 64 bb 19 41  |:%.kG...1...d..A|
    000000a0  fe 2b 7a 86 a8 be e4 c0  b6 98 2e dc 96 92 58 92  |.+z...........X.|
    000000b0  c4 6b c1 85 a9 d0 ec d6  03 2d c7 2c 14 f5 da 03  |.k.......-.,....|
    000000c0  ef c6 c9 2b bc 26 9c 36  ab 0c da 08 f2 8b 79 c7  |...+.&.6......y.|
    000000d0  12 98 55 5f 4c 56 f7 fd  e1 71 45 16 a3 59 01 76  |..U_LV...qE..Y.v|
    000000e0  97 5b d1 cc 91 92 c5 d9  05 0a                    |.[........|
    000000ea

    The etcd key should be prefixed with k8s:enc:aescbc:v1:key1, which indicates the aescbc provider was used to encrypt the data with the key1 encryption key.

  4. Deployments as they are described on Kubernetes the hard way will work

    $ kubectl create deployment nginx --image=nginx
    deployment.apps/nginx created

    List the pod created

    $ kubectl get pods -l app=nginx
    NAME                    READY   STATUS    RESTARTS   AGE
    nginx-8f458dc5b-86vs4   1/1     Running   0          36s
  5. NodePort service will work in the following way (Depends on nginx deployment described in step 4)

    $ kubectl expose deployment nginx --port 80 --type NodePort
    $ NODE_PORT=$(kubectl get svc nginx \
      --output=jsonpath='{range .spec.ports[0]}{.nodePort}')
    $ WORKER_IP=$(multipass info 'worker-1-k8s' | grep 'IPv4' | \
      awk '{ print $2 }')
    $ curl -I "http://${WORKER_IP}:${NODE_PORT}"
    HTTP/1.1 200 OK
    Server: nginx/1.25.3
    Date: Thu, 07 Dec 2023 05:28:35 GMT
    Content-Type: text/html
    Content-Length: 615
    Last-Modified: Tue, 24 Oct 2023 13:46:47 GMT
    Connection: keep-alive
    ETag: "6537cac7-267"
    Accept-Ranges: bytes
  6. You've done a kubernetes!

How to see hubble-ui

  1. Forward hubble-ui port

    POD_NAME=$(kubectl get pods -n kube-system -l k8s-app=hubble-ui \
      -o jsonpath="{.items[0].metadata.name}")
    kubectl port-forward -n kube-system ${POD_NAME} 8080:8081
  2. In your browser go to http://localhost:8080 -> pick a namespace with pods. Example below:

    cilium-hubble-ui

Troubleshooting

All nodes should be able to reach each other via hostname

01-config-files/distribute-config-files.sh generates multipass-hosts and later the bootstrap scripts append it to /etc/hosts on the controllers and workers

Related links

About

Applying the lessons learned from "Kubernetes The Hard Way" with cilium as CNI

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages