"/dev/termination-log must have noexec so unpriviledged user cannot exec from it"

[pentago]

What happened?

An unprivileged process/user running inside of a pod is able to write to /dev/termination-log file.

I thought this was preventable with both pod/container securityContext but that didn't turn out to be the case.

I didn't test it till the end but I tried redirecting the output of /dev/urandom from within the container to /dev/termination-log and filled it with gibberish until it reached the size of 2GB.

I suspect this is a way to compromise/crash node.

Is this an exploitable scenario and what can be done to mount the termination log in a way that doesn't allow unprivileged user/process to write arbitrary stuff to it?

What did you expect to happen?

Permissions to be denied when attempting to write /dev/termination-log file.

How can we reproduce it (as minimally and precisely as possible)?

Exec pod container with maximally unprivileged securityContext and write /dev/termination-log file with:
cat /dev/urandom > /dev/termination-log

Anything else we need to know?

No response

Kubernetes version

Any

Cloud provider

N/A

OS version

Any

Install tools

N/A

Container runtime (CRI) and version (if applicable)

N/A

Related plugins (CNI, CSI, ...) and versions (if applicable)

N/A

[pentago]

/sig Security
/committee Security Response

[k8s-ci-robot]

@pentago: The label(s) committee/security, committee/response cannot be applied, because the repository doesn't have them.

In response to this:

/sig Security
/committee Security Response

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kannon92]

Maybe a duplicate of #81116

[kannon92]

/sig node

[hlx-a1]

To add some detail to this, I think /dev/termination-log/ has to be writable in order to capture container output, but what exacerbates the issue is that it's mounted without noexec like the rest of /dev which means it can be executed by an unprivileged user. The best case here, I think, is to mount it noexec and impose a user configurable limit on its size.

[SergeyKanzhelev]

also another dup: #108076

/retitle "/dev/termination-log must have noexec so unpriviledged user cannot exec from it"

[SergeyKanzhelev]

/triage accepted

with the scope of just noexec I think it is reasonable

[SergeyKanzhelev]

/priority backlog

[mochizuki875]

I have a question.
There are some file these mount conditions are similar to termination-log.
These are also bind-mounted to node files with rw and without noexec.
Is there a problem with this?

$ kubectl exec -it nginx -- mount
...
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/hosts type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /dev/termination-log type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/hostname type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/resolv.conf type ext4 (rw,relatime)
...

[hlx-a1]

Maybe /etc/hosts, /etc/hostname and /etc/resolv.conf could maybe mounted ro instead of rw? Is there a case for writing to these mounts at runtime? I don't think there's a problem per-se, but maybe they can be mounted with lower privileges.

[mochizuki875]

Maybe /etc/hosts, /etc/hostname and /etc/resolv.conf could maybe mounted ro instead of rw?

I thought that, but the result was that (these access mode were rw).
I used this manifest to deploy Pod so no securityContext is set.

nginx-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx

[anguslees]

I suspect this is a way to compromise/crash node.

Just to dial back the alarm - unless someone can propose a mechanism for this, I think this is untrue. /dev/termination-log is just a writeable file, and the kubelet just reads opaque content from it and makes that available through the pod status. The contents of this file are not parsed, nor executed outside the container's security boundary.

I can imagine a DoS attack scenario where we fill up the host's disk. This is similar to writing to other ephemeral-storage (eg emptyDir, or your container's writeable top layer), and the solution is the same: ephemeral-storage quota and kubelet's existing full rootfs/imagefs recovery mechanisms.

[identw]

the solution is the same: ephemeral-storage quota and kubelet's existing full rootfs/imagefs recovery mechanisms.

Hi @anguslees
I attempted to work with the ephemeral-storage quota. I'm afraid this isn't a solution.

For instance, I deployed this Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: debug
  labels:
    app: debug
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: debug
  template:
    metadata:
      labels:
        app: debug
    spec:
      terminationGracePeriodSeconds: 5
      enableServiceLinks: false
      imagePullSecrets:
        - name: registry
      containers:
        - name: debug
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
              ephemeral-storage: 1Mi
            limits:
              cpu: 100m
              memory: 128Mi
              ephemeral-storage: 1Mi
          image: docker.io/alpine:latest
          command:
            - /bin/sleep
            - "infinity"
          workingDir: /app

If I run dd if=/dev/urandom of=/app/violation_quota bs=1M count=1024, then kubelet will evict this pod (error: Pod ephemeral local storage usage exceeds the total limit of containers 1Mi). This is the correct behavior.

However, when I run dd if=/dev/urandom of=/dev/termination-log bs=1M count=1024 kubelet doesn't take any action. Instead, I observe filled space on the worker node:

root@kube-worker133-1 ~ $ du -sh /var/lib/kubelet/pods/b7611457-55f2-4efa-8c6f-35315364f28d/containers/debug/449783f9 
1.1G	/var/lib/kubelet/pods/b7611457-55f2-4efa-8c6f-35315364f28d/containers/debug/449783f9
root@kube-worker133-1 ~ $ ls -la /var/lib/kubelet/pods/b7611457-55f2-4efa-8c6f-35315364f28d/containers/debug/449783f9
-rw-rw-rw- 1 root root 1073741824 Apr 20 07:50 /var/lib/kubelet/pods/b7611457-55f2-4efa-8c6f-35315364f28d/containers/debug/449783f9
root@kube-worker133-1 ~ $ file /var/lib/kubelet/pods/b7611457-55f2-4efa-8c6f-35315364f28d/containers/debug/449783f9
/var/lib/kubelet/pods/b7611457-55f2-4efa-8c6f-35315364f28d/containers/debug/449783f9: data

If I run dd if=/dev/urandom of=/dev/termination-log bs=1M (without the count), the node's disk will fill up to approximately 90% until the kubelet sends an eviction signal because the node needs to resolve the DiskPressure status. However, kubelet's GC will not clean the node until the pod is deleted manually or another solution is applied to remove evicted pods.

[kannon92]

@identw thanks for the detailed investigation. Yes, if /dev/termination-log is not monitored by kubelet then GC won't remove it.

We have #122224 which if solved should solve the DDOS approach.

That issue is looking for an owner if you are interested.

[anguslees]

Hi @anguslees
I attempted to work with the ephemeral-storage quota. I'm afraid this isn't a solution.

Right, I agree you've found a bug - and we should include this file in the ephemeral quota calculation. I was trying to say we don't need a radically new mechanism here, and the implication is a DoS not compromise.