Broker vm.swappiness configuration is lost after VM reboot.

[mcolomerc]

Describe the issue
sysctl configuration was lost after Broker VM reboot.

https://man7.org/linux/man-pages/man5/sysctl.d.5.html

Default value: kafka_broker_sysctl_file: /usr/lib/sysctl.d/sysctl.conf

https://github.com/confluentinc/cp-ansible/blob/7.1.6-post/roles/kafka_broker/defaults/main.yml#L59

To Reproduce
Steps to reproduce the behaviour:

Kafka Broker sysctl configuration: hosts.yaml

 kafka_broker_sysctl:
  vm.swappiness: 1
  vm.dirty_background_ratio: 5
  vm.dirty_ratio: 80
  vm.max_map_count: 262144

When cp-ansible ends, from the Broker shell:

$ sysctl vm.swappiness
    vm.swappiness = 1

Reboot the Broker VM:

$ sysctl vm.swappiness
    vm.swappiness = 30

Expected behaviour

 $ sysctl vm.swappiness
    vm.swappiness = 1

Inventory File

 kafka_broker_sysctl:
  vm.swappiness: 1
  vm.dirty_background_ratio: 5
  vm.dirty_ratio: 80
  vm.max_map_count: 262144

Logs

TASK [confluent.platform.kafka_broker : Tune virtual memory settings] *********************************************************
changed: [cp-node] => (item={'key': 'vm.swappiness', 'value': 1})
changed: [cp-node] => (item={'key': 'vm.dirty_background_ratio', 'value': 5})
changed: [cp-node] => (item={'key': 'vm.dirty_ratio', 'value': 80})
changed: [cp-node] => (item={'key': 'vm.max_map_count', 'value': 262144})

Environment (please complete the following information):

• OS: RHEL 8
• CP-Ansible Branch: 7.1.6
• Ansible Version: 2.14.3

Additional context
Adding the following configuration to the hosts.yaml:

kafka_broker_sysctl_file: /etc/sysctl.d/90-kafka-broker.conf

keeps the value vm.swappiness=1 after a VM reboot.

[sverrehu]

We see the same, and have found that tuned is the culprit. In our /usr/lib/tuned/virtual-guest/tuned.conf we find vm.swappiness = 30. Tuned's plugin_sysctl will check /run/sysctl.d/*.conf, /etc/sysctl.d/*.conf and /etc/sysctl.conf for values, and not override those found in these files. But it will not, however, check /usr/lib/sysctl.d/*.conf, where your file is.

This appears to be by design, and not really cp-ansible's problem. But a small note in the documentation of kafka_broker_sysctl_file would be nice.