Trouble in debugging in k8s pod with limited privilege

[rickyzhang82]

I'm using PyStack to debug deadlock issue in Python code. But I found it hard to use PyStack in the following scenario:

The target Python process is inside a k8s pod (let's call it the target pod). The target pod didn't run as root user, removed apt-get, pip, made the root FS read-only. So I can't launch PyStack from the k8s pod at the runtime.

I used the tool kubectl-exec to attach a separate pod with the root privilege to the k8s node. But I can't remote attach PyStack from the separate pod to the target Python process because the Python binary in the target pod is not accessible to the separate pod.

I can use GDB to remote attach to the target Python process from the separate pod. But the stack frame from PVM is not quite helpful. Because PVM is compiled with the optimization flag enabled. Most the parameters are optimized out. There are two hundred levels deep stack frame. I'm not familiar with PVM source code. I found it very hard to use GDB to debug Python as well.

Any suggestion to use PyStack in this k8s env to make my debug life easier?

[sarahmonod]

Hi @rickyzhang82, did you try to use the core file feature instead of trying to work with a running process? gcore for example can get you a core of a running process, which you can then analyze in a different environment that has more debug symbols perhaps.

[rickyzhang82]

That's exactly what I did! I used gcore in GDB to create core dump files and then replicate the same binary environment as the target pod by extending the new Dockerfile from the base image of the target pod.

One thing I want to mention that there is python-gdb.py source file in the cpython source code. Adding it to ~/.gdbinit never work. I have to use source python-gdb.py inside the GDB. Now I can use py-bt to explore Python source code stack frame, instead of PVM stack frame.

I didn't know how pystack implements. I used dlv for Go. It is very handy Go debug tool that doesn't require root privilege. Perhaps, we should aim for non-privilege user for pystack as well.

Thank you!

[rickyzhang82]

I ran it inside a security hardening k8s pod. I'm not sure how I can bypass it.

WARNING(process_remote): Failed to attach to thread 141: Operation not permitted
💀 Engine error: Operation not permitted 💀

The specified process cannot be traced. This could be because the tracer
has insufficient privileges (the required capability is CAP_SYS_PTRACE).
Unprivileged processes cannot trace processes that they cannot send signals
to or those running set-user-ID/set-group-ID programs, for security reasons.
Alternatively, the process may already be being traced.

If your uid matches the uid of the target process you want to analyze, you
can do one of the following to get 'ptrace' scope permissions:

* If you are running inside a Docker container, you need to make sure you
  start the container using the '--cap-add=SYS_PTRACE' or '--privileged'
  command line arguments. Notice that this may not be enough if you are not
  running as 'root' inside the Docker container as you may need to disable
  hardening (see next points).

* Try running again with elevated permissions by running 'sudo -E !!'.

* You can disable kernel hardening for the current session temporarily (until
  a reboot happens) by running 'echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope'.

[pablogsal]

Unfortunately I don't particulary have a lot of experience managing kubernetes so I cannot point you to the right way to fix it but I would check if you at least have SYS_PTRACE permissions. You need to find how to add --cap-add=SYS_PTRACE in your k8s. if not. This is the same requirement as to attach with gdb and to use other profilers so maybe you can search that way.

[rickyzhang82]

I can't add any permission to the existing pod. It is security hardening by default.

But I can attach a new pod with no security limit to the k8s node that the target pod runs on. In this case, pystack can't attach to a remote process but GDB can.

[godlygeek]

Maybe the GDB process that you're running is root, and the pystack one isn't? Perhaps your GDB executable is setuid-root... Does running pystack with sudo make any difference?

[rickyzhang82]

When I launch a new pod to the k8s node, the pod runs without any security restraints. It runs as root.