K8SSAND-1866 ⁃ Stargate Stuck in ContainerCreating Status Following "Single-cluster install with helm"

[compensator]

## Bug Report

Describe the bug
I followed the instructions at Install Prerequisites and Single-cluster install with helm and it repeatedly results in the Stargate deployment being stuck in the Status: ContainerCreating. I have destroyed and recreated the cluster and gone through the steps multiple times with the same result.

To Reproduce
Steps to reproduce the behavior:
Follow all instructions on Install Prerequisites and Single-cluster install with helm and after executing:
kubectl apply -n k8ssandra-operator -f k8c1.yml

I then get the following. As you can see, I let this sit for the past nearly 3 days.

$ kubectl get pods -n k8ssandra-operator | grep stargate
NAME                                                    READY   STATUS              RESTARTS   AGE
demo-dc1-default-stargate-deployment-7d999b7b9f-7znfd   0/1     ContainerCreating   0          2d20h

Expected behavior
After completing the step:

`kubectl apply -n k8ssandra-operator -f k8c1.yml`

I expect the following command to eventually show

$ kubectl get pods -n k8ssandra-operator | grep stargate
NAME                                                    READY   STATUS              RESTARTS   AGE
demo-dc1-default-stargate-deployment-7d999b7b9f-7znfd   1/1     Running   0          2d20h

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Helm charts version info

$ helm ls -A
NAME              	NAMESPACE         	REVISION	UPDATED                               	STATUS  	CHART                    	APP VERSION
cert-manager      	cert-manager      	1       	2022-10-30 20:32:30.65968193 -0400 EDT	deployed	cert-manager-v1.10.0     	v1.10.0    
k8ssandra-operator	k8ssandra-operator	1       	2022-10-30 20:33:10.48590982 -0400 EDT	deployed	k8ssandra-operator-0.38.2	1.2.1  

• Helm charts user-supplied values

helm install k8ssandra-operator k8ssandra/k8ssandra-operator -n k8ssandra-operator --create-namespace

• Kubernetes version information:

$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.3
Kustomize Version: v4.5.7
Server Version: v1.24.2

• Kubernetes cluster kind:

kind

Additional context

Operating System
$ lsb_release -d
Description: Ubuntu 20.04.5 LTS

kubectl get event --namespace k8ssandra-operator --field-selector involvedObject.name=demo-dc1-default-stargate-deployment-7d999b7b9f-7znfd
LAST SEEN   TYPE      REASON        OBJECT                                                      MESSAGE
9m6s        Warning   FailedMount   pod/demo-dc1-default-stargate-deployment-7d999b7b9f-7znfd   Unable to attach or mount volumes: unmounted volumes=[kube-api-access-jtlm8], unattached volumes=[cassandra-config kube-api-access-jtlm8]: timed out waiting for the condition
2m26s       Warning   FailedMount   pod/demo-dc1-default-stargate-deployment-7d999b7b9f-7znfd   (combined from similar events): MountVolume.SetUp failed for volume "kube-api-access-jtlm8" : write /var/lib/kubelet/pods/7e877716-8d85-451a-8b4b-7c1ebca14fe1/volumes/kubernetes.io~projected/kube-api-access-jtlm8/..2022_11_02_21_57_25.1550523734/namespace: no space left on device
47m         Warning   FailedMount   pod/demo-dc1-default-stargate-deployment-7d999b7b9f-7znfd   Unable to attach or mount volumes: unmounted volumes=[kube-api-access-jtlm8], unattached volumes=[kube-api-access-jtlm8 cassandra-config]: timed out waiting for the condition

There is no /var/lib/kubelet/pods directory and there's plenty of storage space available. It smells of an access issue.

$ df -h | grep sda5
/dev/sda5 457G 56G 379G 13% /

Much research on this since my first attempt at executing this install last week on Thursday. The local disk is a relatively new SSD.

┆Issue is synchronized with this Jira Task by Unito
┆Issue Number: K8SSAND-1866
┆Priority: Medium
┆Link To Issue: https://k8ssandra.atlassian.net/browse/K8SSAND-1866