Error in run: file node.yaml not found for version cis-1.23

[Algoss]

Overview
I have config file in /etc/kubernetes-kubelet/kubelet_config.yaml path. I added /etc/kubernetes-kubelet/kubelet_config.yaml to config.yaml and mounted them via configmap to pod in the path /opt/kube-bench/cfg.

While starting kube-bench pod in k8s cluster, it gives error Error in run: file node.yaml not found for version cis-1.23

How did you run kube-bench?

I ran kube-bench pod in k8s cluster.
kubectl apply -f kube-bench-pod.yaml

What happened?

I1118 14:22:54.676795 15498 util.go:486] Checking for oc
I1118 14:22:54.676867 15498 util.go:515] Can't find oc command: exec: "oc": executable file not found in $PATH
I1118 14:22:54.676879 15498 kubernetes_version.go:36] Try to get version from Rest API
I1118 14:22:54.676931 15498 kubernetes_version.go:161] Loading CA certificate
I1118 14:22:54.678044 15498 kubernetes_version.go:115] getWebData srvURL: https://kubernetes.default.svc/version
I1118 14:22:54.683608 15498 kubernetes_version.go:100] vd: {
"major": "1",
"minor": "19",
"gitVersion": "v1.19.16",
"gitCommit": "e37e4ab4cc8dcda84f1344dda47a97bb1927d074",
"gitTreeState": "clean",
"buildDate": "2021-10-27T16:20:18Z",
"goVersion": "go1.15.15",
"compiler": "gc",
"platform": "linux/amd64"
}
I1118 14:22:54.683663 15498 kubernetes_version.go:105] vrObj: &cmd.VersionResponse{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:20:18Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
I1118 14:22:54.683676 15498 util.go:293] Kubernetes REST API Reported version: &{1 19 v1.19.16}
I1118 14:22:54.683708 15498 kubernetes_version.go:36] Try to get version from Rest API
I1118 14:22:54.683753 15498 kubernetes_version.go:161] Loading CA certificate
I1118 14:22:54.683768 15498 kubernetes_version.go:115] getWebData srvURL: https://kubernetes.default.svc/version
I1118 14:22:54.688118 15498 kubernetes_version.go:100] vd: {
"major": "1",
"minor": "19",
"gitVersion": "v1.19.16",
"gitCommit": "e37e4ab4cc8dcda84f1344dda47a97bb1927d074",
"gitTreeState": "clean",
"buildDate": "2021-10-27T16:20:18Z",
"goVersion": "go1.15.15",
"compiler": "gc",
"platform": "linux/amd64"
}
I1118 14:22:54.688159 15498 kubernetes_version.go:105] vrObj: &cmd.VersionResponse{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:20:18Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
I1118 14:22:54.688172 15498 util.go:293] Kubernetes REST API Reported version: &{1 19 v1.19.16}
I1118 14:22:54.688212 15498 common.go:281] mapToBenchmarkVersion for k8sVersion: "1.19" cisVersion: "cis-1.23" found: true
I1118 14:22:54.688223 15498 common.go:347] Mapped Kubernetes version: 1.19 to Benchmark version: cis-1.23
I1118 14:22:54.688231 15498 common.go:350] Kubernetes version: "1.19" to Benchmark version: "cis-1.23"
I1118 14:22:54.688238 15498 run.go:40] Checking targets [node] for cis-1.23
I1118 14:22:54.688341 15498 common.go:267] No version-specific config.yaml file in cfg/cis-1.23
I1118 14:22:54.688350 15498 common.go:273] Using config file: cfg/cis-1.23/config.yaml

What did you expect to happen:

I expected kube-bench pod to start successfully and scan the k8s cluster with correct config file /etc/kubernetes-kubelet/kubelet_config.yaml

Environment

Kube-bench: 0.6.8
Kuberneted: 1.19.16

Running processes
I am running kube-bench pod in k8s cluster.

ConfigMap: config.yaml

`---
master:
components:
- apiserver
- scheduler
- controllermanager
- etcd
- flanneld
# kubernetes is a component to cover the config file /etc/kubernetes/config that is referred to in the benchmark
- kubernetes
- kubelet

kubernetes:
defaultconf: /etc/kubernetes/config

apiserver:
bins:
- "kube-apiserver"
- "hyperkube apiserver"
- "hyperkube kube-apiserver"
- "apiserver"
- "openshift start master api"
- "hypershift openshift-kube-apiserver"
confs:
- /etc/kubernetes/manifests/kube-apiserver.yaml
- /etc/kubernetes/manifests/kube-apiserver.yml
- /etc/kubernetes/manifests/kube-apiserver.manifest
- /var/snap/kube-apiserver/current/args
- /var/snap/microk8s/current/args/kube-apiserver
- /etc/origin/master/master-config.yaml
- /etc/kubernetes/manifests/talos-kube-apiserver.yaml
defaultconf: /etc/kubernetes/manifests/kube-apiserver.yaml

scheduler:
bins:
- "kube-scheduler"
- "hyperkube scheduler"
- "hyperkube kube-scheduler"
- "scheduler"
- "openshift start master controllers"
confs:
- /etc/kubernetes/manifests/kube-scheduler.yaml
- /etc/kubernetes/manifests/kube-scheduler.yml
- /etc/kubernetes/manifests/kube-scheduler.manifest
- /var/snap/kube-scheduler/current/args
- /var/snap/microk8s/current/args/kube-scheduler
- /etc/origin/master/scheduler.json
- /etc/kubernetes/manifests/talos-kube-scheduler.yaml
defaultconf: /etc/kubernetes/manifests/kube-scheduler.yaml
kubeconfig:
- /etc/kubernetes/scheduler.conf
- /var/lib/kube-scheduler/kubeconfig
- /var/lib/kube-scheduler/config.yaml
- /system/secrets/kubernetes/kube-scheduler/kubeconfig
defaultkubeconfig: /etc/kubernetes/scheduler.conf

controllermanager:
bins:
- "kube-controller-manager"
- "kube-controller"
- "hyperkube controller-manager"
- "hyperkube kube-controller-manager"
- "controller-manager"
- "openshift start master controllers"
- "hypershift openshift-controller-manager"
confs:
- /etc/kubernetes/manifests/kube-controller-manager.yaml
- /etc/kubernetes/manifests/kube-controller-manager.yml
- /etc/kubernetes/manifests/kube-controller-manager.manifest
- /var/snap/kube-controller-manager/current/args
- /var/snap/microk8s/current/args/kube-controller-manager
- /etc/kubernetes/manifests/talos-kube-controller-manager.yaml
defaultconf: /etc/kubernetes/manifests/kube-controller-manager.yaml
kubeconfig:
- /etc/kubernetes/controller-manager.conf
- /var/lib/kube-controller-manager/kubeconfig
- /system/secrets/kubernetes/kube-controller-manager/kubeconfig
defaultkubeconfig: /etc/kubernetes/controller-manager.conf

etcd:
optional: true
bins:
- "etcd"
- "openshift start etcd"
confs:
- /etc/kubernetes/manifests/etcd.yaml
- /etc/kubernetes/manifests/etcd.yml
- /etc/kubernetes/manifests/etcd.manifest
- /etc/etcd/etcd.conf
- /var/snap/etcd/common/etcd.conf.yml
- /var/snap/etcd/common/etcd.conf.yaml
- /var/snap/microk8s/current/args/etcd
- /usr/lib/systemd/system/etcd.service
defaultconf: /etc/kubernetes/manifests/etcd.yaml

flanneld:
optional: true
bins:
- flanneld
defaultconf: /etc/sysconfig/flanneld

kubelet:
optional: true
bins:
- "hyperkube kubelet"
- "kubelet"

node:
components:
- kubelet
- proxy
# kubernetes is a component to cover the config file /etc/kubernetes/config that is referred to in the benchmark
- kubernetes

kubernetes:
defaultconf: "/etc/kubernetes/config"

kubelet:
cafile:
- "/etc/kubernetes/pki/ca.crt"
- "/etc/kubernetes/certs/ca.crt"
- "/etc/kubernetes/cert/ca.pem"
- "/var/snap/microk8s/current/certs/ca.crt"
svc:
# These paths must also be included
# in the 'confs' property below
- "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
- "/etc/systemd/system/kubelet.service"
- "/lib/systemd/system/kubelet.service"
- "/etc/systemd/system/snap.kubelet.daemon.service"
- "/etc/systemd/system/snap.microk8s.daemon-kubelet.service"
- "/etc/systemd/system/atomic-openshift-node.service"
- "/etc/systemd/system/origin-node.service"
bins:
- "hyperkube kubelet"
- "kubelet"
kubeconfig:
- "/etc/kubernetes/kubelet.conf"
- "/etc/kubernetes/kubelet-kubeconfig.conf"
- "/var/lib/kubelet/kubeconfig"
- "/etc/kubernetes/kubelet-kubeconfig"
- "/etc/kubernetes/kubelet/kubeconfig"
- "/var/snap/microk8s/current/credentials/kubelet.config"
- "/etc/kubernetes/kubeconfig-kubelet"
confs:
- "/etc/kubernetes-kubelet/kubelet_config.yaml"
- "/etc/kubernetes/kubelet-config.yaml"
- "/var/lib/kubelet/config.yaml"
- "/var/lib/kubelet/config.yml"
- "/etc/kubernetes/kubelet/kubelet-config.json"
- "/etc/kubernetes/kubelet/config"
- "/home/kubernetes/kubelet-config.yaml"
- "/home/kubernetes/kubelet-config.yml"
- "/etc/default/kubeletconfig.json"
- "/etc/default/kubelet"
- "/var/lib/kubelet/kubeconfig"
- "/var/snap/kubelet/current/args"
- "/var/snap/microk8s/current/args/kubelet"
## Due to the fact that the kubelet might be configured
## without a kubelet-config file, we use a work-around
## of pointing to the systemd service file (which can also
## hold kubelet configuration).
## Note: The following paths must match the one under 'svc'
- "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
- "/etc/systemd/system/kubelet.service"
- "/lib/systemd/system/kubelet.service"
- "/etc/systemd/system/snap.kubelet.daemon.service"
- "/etc/systemd/system/snap.microk8s.daemon-kubelet.service"
- "/etc/kubernetes/kubelet.yaml"
defaultconf: "/var/lib/kubelet/config.yaml"
defaultsvc: "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
defaultkubeconfig: "/etc/kubernetes/kubelet.conf"
defaultcafile: "/etc/kubernetes/pki/ca.crt"

proxy:
optional: true
bins:
- "kube-proxy"
- "hyperkube proxy"
- "hyperkube kube-proxy"
- "proxy"
- "openshift start network"
confs:
- /etc/kubernetes/proxy
- /etc/kubernetes/addons/kube-proxy-daemonset.yaml
- /etc/kubernetes/addons/kube-proxy-daemonset.yml
- /var/snap/kube-proxy/current/args
- /var/snap/microk8s/current/args/kube-proxy
kubeconfig:
- "/etc/kubernetes/kubelet-kubeconfig"
- "/etc/kubernetes/kubelet-kubeconfig.conf"
- "/etc/kubernetes/kubelet/config"
- "/var/lib/kubelet/kubeconfig"
- "/var/snap/microk8s/current/credentials/proxy.config"
svc:
- "/lib/systemd/system/kube-proxy.service"
- "/etc/systemd/system/snap.microk8s.daemon-proxy.service"
defaultconf: /etc/kubernetes/addons/kube-proxy-daemonset.yaml
defaultkubeconfig: "/etc/kubernetes/proxy.conf"

etcd:
components:
- etcd

etcd:
bins:
- "etcd"
confs:
- /etc/kubernetes/manifests/etcd.yaml
- /etc/kubernetes/manifests/etcd.yml
- /etc/kubernetes/manifests/etcd.manifest
- /etc/etcd/etcd.conf
- /var/snap/etcd/common/etcd.conf.yml
- /var/snap/etcd/common/etcd.conf.yaml
- /var/snap/microk8s/current/args/etcd
- /usr/lib/systemd/system/etcd.service
defaultconf: /etc/kubernetes/manifests/etcd.yaml

controlplane:
components:
- apiserver

apiserver:
bins:
- "kube-apiserver"
- "hyperkube apiserver"
- "hyperkube kube-apiserver"
- "apiserver"

policies:
components: []

managedservices:
components: []

version_mapping:
"1.15": "cis-1.5"
"1.16": "cis-1.6"
"1.17": "cis-1.6"
"1.18": "cis-1.6"
"1.19": "cis-1.20"
"1.20": "cis-1.20"
"1.21": "cis-1.20"
"1.22": "cis-1.23"
"1.23": "cis-1.23"
"eks-1.0.1": "eks-1.0.1"
"eks-1.1.0": "eks-1.1.0"
"gke-1.0": "gke-1.0"
"gke-1.2.0": "gke-1.2.0"
"ocp-3.10": "rh-0.7"
"ocp-3.11": "rh-0.7"
"ocp-4.0": "rh-1.0"
"aks-1.0": "aks-1.0"
"ack-1.0": "ack-1.0"
"cis-1.6-k3s": "cis-1.6-k3s"

target_mapping:
"cis-1.5":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
"cis-1.6":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
"cis-1.6-k3s":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
"cis-1.20":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
"cis-1.23":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
"gke-1.0":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
- "managedservices"
"gke-1.2.0":
- "master"
- "node"
- "controlplane"
- "policies"
- "managedservices"
"eks-1.0.1":
- "master"
- "node"
- "controlplane"
- "policies"
- "managedservices"
"eks-1.1.0":
- "master"
- "node"
- "controlplane"
- "policies"
- "managedservices"
"rh-0.7":
- "master"
- "node"
"aks-1.0":
- "master"
- "node"
- "controlplane"
- "policies"
- "managedservices"
"ack-1.0":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
- "managedservices"
"rh-1.0":
- "master"
- "node"
- "controlplane"
- "policies"
- "etcd"
"eks-stig-kubernetes-v1r6":
- "node"
- "controlplane"
- "policies"
- "managedservices"`

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]
Pod Specification
apiVersion: v1 kind: Pod metadata: name: kube-bench-pod namespace: kube-bench labels: name: kube-bench-pod spec: hostPID: true restartPolicy: Never containers: - name: kube-bench image: lnkdin.cr/temp/infosec/kube-bench:0.6.8 command: ["/bin/sh","-c"] args: ["kube-bench run -v 3 --targets node"] resources: requests: cpu: "2" memory: "2Gi" limits: cpu: "2" memory: "2Gi" volumeMounts: - name: var-lib-kubelet mountPath: /var/lib/kubelet readOnly: true - name: etc-systemd mountPath: /etc/systemd readOnly: true - name: lib-systemd mountPath: /lib/systemd readOnly: true - name: srv-kubernetes mountPath: /srv/kubernetes readOnly: true - name: etc-kubernetes mountPath: /etc/kubernetes readOnly: true # /usr/local/mount-from-host/bin is mounted to access kubectl / kubelet, for auto-detecting the Kubernetes version. # You can omit this mount if you specify --version as part of the command. - name: usr-bin mountPath: /usr/local/mount-from-host/bin readOnly: true - name: config-volume mountPath: /opt/kube-bench/cfg readOnly: true volumes: - name: var-lib-kubelet hostPath: path: "/var/lib/kubelet" - name: etc-systemd hostPath: path: "/etc/systemd" - name: lib-systemd hostPath: path: "/lib/systemd" - name: srv-kubernetes hostPath: path: "/srv/kubernetes" - name: etc-kubernetes hostPath: path: "/etc/kubernetes" - name: usr-bin hostPath: path: "/usr/bin" - name: config-volume configMap: name: my-config

[mozillazg]

@Algoss We can't use mount to overwrite config.yaml, because it will clean up all files and dirs under the /opt/kube-bench/cfg. The recommended way is to do it when building the docker image.

[Algoss]

@mozillazg I am not doing docker build. I am running this kube-bench pod in the k8s cluster.
I have followed #948 (comment) and tried the same solution.
But I am not sure about the mountPath where the configMap has to be mounted. I just followed this https://github.com/aquasecurity/kube-bench/blob/main/docs/running.md#running-inside-a-container and used /opt/kube-bench/cfg/ but I feel it is wrong for my use case as I am running in the k8s cluster and not in the container.

I have asked the doubt here #948 (comment)

Please provide your insights. Thanks