opnfv/functest-kubernetes-security:v1.23 run_tests -t kube_bench_node FAILED

[sultetveny]

Overview

I've faced with two issue running the kubernetes security test using kub_bench.

1. On cluster which has predefined strict PSP is used, it's simply unable to start the containers from the job. The question here shouldn't be noted somewhere (maybe in the failed output) in case PSP used in the cluster this test is not applicable? Maybe the test should check it first, then reject the test without even start to run it?! This is just a theoretical question, what is the expected behaviour in such case.
2. Just to make sure without the PSP the test is running fine, I disabled the PSP on the cluster. Removed the below plugin from kube-apiserver. However, without PSP it still failed and I can't figure out why. Please help!

# remove from /etc/kubernetes/kube-apiserver-config
--enable-admission-plugins=PodSecurityPolicy

How did you run kube-bench?

podman run -it --env-file ~/opnfv/env \
-v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z \
-v ~/opnfv/config:/root/.kube/config:Z \
-v ~/opnfv/results:/home/opnfv/functest/results:Z \
-v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z \
-v ~/opnfv/cluster-admin.pem:/home/opnfv/functest/cluster-admin.pem:Z \
-v ~/opnfv/cluster-admin-key.pem:/home/opnfv/functest/cluster-admin-key.pem:Z \
opnfv/functest-kubernetes-security:v1.23 /bin/bash

# then within the container
run_tests -t kube_bench_node

What happened?

Test case failed. For more information please check attached files.
functest-kubernetes.debug.log

cat opnfv/results/functest-kubernetes.debug.log 

...

2023-02-22 08:08:30,957 - kubernetes.client.rest - DEBUG - response body: 
failed to get a set of executables needed for tests: unable to detect running programs for component "kubelet"

2023-02-22 08:08:30,958 - functest_kubernetes.security.security - INFO - 


failed to get a set of executables needed for tests: unable to detect running programs for component "kubelet"

2023-02-22 08:08:30,959 - xtesting.ci.run_tests - ERROR - 

Please fix the testcase kube_bench_node.
All exceptions should be caught by the testcase instead!

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/xtesting/ci/run_tests.py", line 171, in run_test
    test_case.run(**kwargs)
  File "/usr/lib/python3.9/site-packages/functest_kubernetes/security/security.py", line 212, in run
    self.details["report"] = ast.literal_eval(self.pod_log)
  File "/usr/lib/python3.9/ast.py", line 62, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 2
    failed to get a set of executables needed for tests: unable to detect running programs for component "kubelet"
           ^
SyntaxError: invalid syntax
2023-02-22 08:08:30,962 - xtesting.ci.run_tests - ERROR - The test case 'kube_bench_node' failed.
2023-02-22 08:08:30,962 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_ERROR

kubectl logs -n kube-bench-phqn5 kube-bench-node-84zg7

failed to get a set of executables needed for tests: unable to detect running programs for component "kubelet"

kubectl get pod -n kube-bench-phqn5 -o wide

NAME                    READY   STATUS   RESTARTS   AGE   IP               NODE                  NOMINATED NODE   READINESS GATES
kube-bench-node-5k66d   0/1     Error    0          9h    192.168.72.66    cbis-sut1-worker-02   <none>           <none>
kube-bench-node-84zg7   0/1     Error    0          9h    192.168.72.104   cbis-sut1-worker-02   <none>           <none>
kube-bench-node-8t65k   0/1     Error    0          9h    192.168.72.123   cbis-sut1-worker-02   <none>           <none>
kube-bench-node-f2z9s   0/1     Error    0          9h    192.168.72.90    cbis-sut1-worker-02   <none>           <none>
kube-bench-node-mdhrw   0/1     Error    0          9h    192.168.72.92    cbis-sut1-worker-02   <none>           <none>
kube-bench-node-qgqnt   0/1     Error    0          9h    192.168.72.78    cbis-sut1-worker-02   <none>           <none>
kube-bench-node-wxl8h   0/1     Error    0          9h    192.168.72.89    cbis-sut1-worker-02   <none>           <none>

What did you expect to happen:

I expected the test case executed successfully.

Environment

[What is your version of kube-bench? (run kube-bench version)]

[What is your version of Kubernetes? (run kubectl version or oc version on OpenShift.)]

kubectl version

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:49:13Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:42:57Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

Running processes

[Please include the output from running ps -eaf | grep kube on the affected node. This will allow us to check what Kubernetes processes are running, and how this compares to what kube-bench detected.]

# worker-01
ps -eaf | grep kube

root       65980       1  1 Feb07 ?        05:45:56 /usr/local/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/kubelet-config.yml --register-node=true --hostname-override=cbis-sut1-worker-01 --node-labels=is_control=false,is_worker=true,is_edge=false,is_storage=false,bcmt_storage_node=false,rook_storage=false,rook_storage2=false,cpu_pooler_active=false,dynamic_local_storage_node=false,local_storage_node=true,ncs.nokia.com/group=group_02 --register-with-taints= --node-ip=172.20.110.3 --cloud-provider=external --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --v=1
root       66370   66265  0 Feb07 ?        00:05:33 /usr/local/bin/kube-proxy --config=/etc/kubernetes/kube-proxy-config.yml --oom-score-adj=-998
root       78307   78129  0 Feb07 ?        00:00:31 /csi-node-driver-registrar --csi-address=/csi/csi.sock --kubelet-registration-path=/var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock --v=1
99         82177   82051  0 Feb07 ?        01:01:20 /usr/bin/agent -metrics-addr 0 -server-addr 127.0.0.1:65432 -v 1 -log-level info -kubelet-endpoint-point 10248 -kubelet-endpoint-ip 127.0.0.1 -kubelet-endpoint-scheme http
root     2990842 2990798  0 17:02 pts/1    00:00:00 grep --color=auto kube

# worker-02
ps -eaf | grep kube

root       66019       1  1 Feb07 ?        06:07:14 /usr/local/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/kubelet-config.yml --register-node=true --hostname-override=cbis-sut1-worker-02 --node-labels=is_control=false,is_worker=true,is_edge=false,is_storage=false,bcmt_storage_node=false,rook_storage=false,rook_storage2=false,cpu_pooler_active=false,dynamic_local_storage_node=false,local_storage_node=true,ncs.nokia.com/group=group_02 --register-with-taints= --node-ip=172.20.110.23 --cloud-provider=external --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --v=1
root       66408   66304  0 Feb07 ?        00:06:12 /usr/local/bin/kube-proxy --config=/etc/kubernetes/kube-proxy-config.yml --oom-score-adj=-998
root       78171   78013  0 Feb07 ?        00:00:30 /csi-node-driver-registrar --csi-address=/csi/csi.sock --kubelet-registration-path=/var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock --v=1
99         81804   81680  0 Feb07 ?        01:02:26 /usr/bin/agent -metrics-addr 0 -server-addr 127.0.0.1:65432 -v 1 -log-level info -kubelet-endpoint-point 10248 -kubelet-endpoint-ip 127.0.0.1 -kubelet-endpoint-scheme http
root     2978125 2978090  0 17:02 pts/1    00:00:00 grep --color=auto kube

Configuration files

[If kube-bench is reporting an issue related to the settings defined in a config file, please attach the file, or include an extract showing the settings that are being detected incorrectly.]

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]