cgroups v1 isn't supported

[zimnx]

On Bottlerocket OS (AWS), memory limit cannot be read by Seastart due to wrong implementation, resulting in no memory limit causing OOMs.

If there's a single line in /proc/<pid>/cgroup Seastart uses cgroupv2 logic, but on Bottlerocket there're more lines, so it falls back to cgroupsv1.

$ cat /proc/1/cgroup 
12:hugetlb:/
11:memory:/init.scope
10:pids:/init.scope
9:cpuset:/
8:freezer:/
7:devices:/init.scope
6:perf_event:/
5:blkio:/init.scope
4:net_cls,net_prio:/
3:misc:/
2:cpu,cpuacct:/init.scope
1:name=systemd:/init.scope
0::/init.scope

Seastar shouldn't assume this path contains single line, as manual page https://man7.org/linux/man-pages/man7/cgroups.7.html (search for /proc/pid/cgroup) suggests it may contain more.
Seastart should follow manual guide for this file.

[xemul]

I updated the issue title to better reflect what the problem is

[zimnx]

This is actually a hybrid setup, where both cgroups v1 and v2 are available. Seastar shouldn't fallback to using v1 just because this file has more than single line. It should use memory controller from version in which it's deployed, either v1 or v2.

[avikivity]

@zimnx can you add instructions on how to reproduce this? It's simple to fix but hard to get a reproducing configuration.

Maybe it's easiest to enable both v1 and v2 on some kernel and rebuild it, rather than attempt to teach someone how to login to bottlerocket.

[avikivity]

This is actually a hybrid setup, where both cgroups v1 and v2 are available. Seastar shouldn't fallback to using v1 just because this file has more than single line. It should use memory controller from version in which it's deployed, either v1 or v2.

So should the title be "cgroups isn't supported when both v1 and v2 are available"?

[zimnx]

Deploying a single node of bottlerocket on EKS is probably much simpler than preparing the right kernel. And working on same setup where issue was found, would give us more trust.
To set up a single instance of bottlerocket on EKS, install dependencies from: https://github.com/bottlerocket-os/bottlerocket/blob/develop/QUICKSTART-EKS.md#dependencies
Create new EC2 key pair in correct region (us-west-1 in this example) (or use existing one if you have private key).
Save following cluster spec in bottlerocket.yaml, replace EC2 key pair name with one you created:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: bottlerocket
  region: us-west-1
  version: '1.24'

nodeGroups:
  - name: ng-bottlerocket
    instanceType: m5.large
    desiredCapacity: 1
    amiFamily: Bottlerocket
    disableIMDSv1: true
    iam:
       attachPolicyARNs:
          - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
          - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
          - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
          - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    ssh:
        allow: true
        publicKeyName: YOUR_EC2_KEYPAIR_NAME
    tags:
        keep: alive

And create EKS cluster via

eksctl create cluster --config-file ./bottlerocket.yaml

Output should contain IP address of a node, if not, it can be extracted via kubectl describe nodes (ExternalIP).

[zimnx]

If needed I can set it up for you for the time of development.

[avikivity]

Those yamls don't help me, I don't know what to do with them.

Also, it's not enough to set up an instance, I want to know how to connect to it, and how to start a non-admin container that I can connect to.

[zimnx]

Those yamls don't help me, I don't know what to do with them.

It's three step guide, it's hard to make it simpler.
I can set up infrastructure for you, and give you access to it.

Also, it's not enough to set up an instance, I want to know how to connect to it, and how to start a non-admin container that I can connect to.

Running Pods on Kubernetes is very similar to how you do it using podman.
kubectl run and kubectl exec and --help parameter in these should be all you need.