Wait condition fails if more than one metallb controller replicaset exists

[TylerAngelier]

Expected Behavior

When running the playbook it completes successfully.

Current Behavior

An error occurs when running the playbook if you had more than one metallb controller replicaset exists. Below is the exception I was getting:

TASK [k3s/post : Wait for MetalLB resources] **************************************************************************************************
Wednesday 09 November 2022  21:24:30 -0500 (0:00:01.214)       0:03:15.201 **** 
ok: [192.168.1.51] => (item=controller)
ok: [192.168.1.51] => (item=webhook service)
ok: [192.168.1.51] => (item=pods in replica sets)
failed: [192.168.1.51] (item=ready replicas of controller) => changed=false 
  ansible_loop_var: item
  cmd:
  - k3s
  - kubectl
  - wait
  - replicaset
  - --namespace=metallb-system
  - --selector=component=controller,app=metallb
  - --for=jsonpath={.status.readyReplicas}=1
  - --timeout=120s
  delta: '0:00:00.211156'
  end: '2022-11-09 21:24:32.305627'
  item:
    condition: --for=jsonpath='{.status.readyReplicas}'=1
    description: ready replicas of controller
    resource: replicaset
    selector: component=controller,app=metallb
  msg: non-zero return code
  rc: 1
  start: '2022-11-09 21:24:32.094471'
  stderr: |-
    readyReplicas is not found
    readyReplicas is not found
  stderr_lines: <omitted>
  stdout: replicaset.apps/controller-6c58495cbb condition met
  stdout_lines: <omitted>
failed: [192.168.1.51] (item=fully labeled replicas of controller) => changed=false 
  ansible_loop_var: item
  cmd:
  - k3s
  - kubectl
  - wait
  - replicaset
  - --namespace=metallb-system
  - --selector=component=controller,app=metallb
  - --for=jsonpath={.status.fullyLabeledReplicas}=1
  - --timeout=120s
  delta: '0:00:00.174107'
  end: '2022-11-09 21:24:32.766300'
  item:
    condition: --for=jsonpath='{.status.fullyLabeledReplicas}'=1
    description: fully labeled replicas of controller
    resource: replicaset
    selector: component=controller,app=metallb
  msg: non-zero return code
  rc: 1
  start: '2022-11-09 21:24:32.592193'
  stderr: |-
    fullyLabeledReplicas is not found
    fullyLabeledReplicas is not found
  stderr_lines: <omitted>
  stdout: replicaset.apps/controller-6c58495cbb condition met
  stdout_lines: <omitted>

Output of replicaset with selectors used:

k get rs -n metallb-system --selector=component=controller,app=metallb
NAME                    DESIRED   CURRENT   READY   AGE
controller-5bd9496b89   0         0         0       100d
controller-6c58495cbb   1         1         1       66m
controller-6d5cb87f6    0         0         0       52d

Delete both the old replicasets:

k delete rs controller-5bd9496b89 -n metallb-system
k delete rs controller-6d5cb87f6 -n metallb-system

Run the wait command again:

k3s kubectl wait replicaset --namespace=metallb-system --selector=component=controller,app=metallb --for=jsonpath='{.status.readyReplicas}'=1 --timeout=10s
replicaset.apps/controller-6c58495cbb condition met

Steps to Reproduce

Have more than one replicaset of the metallb-system controller and run the playbook. Not sure why I had multiple ¯\_(ツ)_/¯

Context (variables)

Operating system: MacBook Pro

Hardware: M1 Pro

Variables Used

all.yml

---
k3s_version: v1.24.7+k3s1
# this is the user that has ssh access to these machines
ansible_user: serveradmin
ansible_ssh_private_key_file: <redacted>
systemd_dir: /etc/systemd/system

# Set your timezone
system_timezone: "America/New_York"

# interface which will be used for flannel
flannel_iface: "eth0"

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "192.168.1.222"

# k3s_token is required  masters can talk together securely
# this token should be alphanumeric only
k3s_token: "<redacted>"

# The IP on which the node is reachable in the cluster.
# Here, a sensible default is provided, you can still override
# it for each of your hosts, though.
k3s_node_ip: '{{ ansible_facts[flannel_iface]["ipv4"]["address"] }}'

# Disable the taint manually by setting: k3s_master_taint = false
k3s_master_taint: "{{ true if groups['node'] | default([]) | length >= 1 else false }}"

# these arguments are recommended for servers as well as agents:
extra_args: >-
  --flannel-iface={{ flannel_iface }}
  --node-ip={{ k3s_node_ip }}

# change these to your liking, the only required one is--no-deploy servicelb
extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik
  --write-kubeconfig-mode 644
  --kube-controller-manager-arg bind-address=0.0.0.0
  --kube-proxy-arg metrics-bind-address=0.0.0.0
  --kube-scheduler-arg bind-address=0.0.0.0
  --etcd-expose-metrics true
  --kubelet-arg containerd=/run/k3s/containerd/containerd.sock
  --kube-apiserver-arg default-not-ready-toleration-seconds=30
  --kube-apiserver-arg default-unreachable-toleration-seconds=30
  --kube-controller-arg node-monitor-period=20s
  --kube-controller-arg node-monitor-grace-period=20s
  --kubelet-arg node-status-update-frequency=5s

extra_agent_args: >-
  {{ extra_args }}
  --kubelet-arg node-status-update-frequency=10s

# image tag for kube-vip
kube_vip_tag_version: "v0.5.5"

# image tag for metal lb
metal_lb_speaker_tag_version: "v0.13.7"
metal_lb_controller_tag_version: "v0.13.7"

# metallb ip range for load balancer
metal_lb_ip_range: "192.168.1.40-192.168.1.50"

Hosts

host.ini

[master]
192.168.30.38
192.168.30.39
192.168.30.40

[node]
192.168.30.41
192.168.30.42

[k3s_cluster:children]
master
node

Possible Solution

• [ x] I've checked the General Troubleshooting Guide

Include more search criteria or cleanup previous metallb controllers that may exist.

[CameronMunroe]

Linking the discussion, so the answer for those seeing this article can find it.

#366 (comment)