I can create a PR but a sanity check on the above conclusions first would be appreciated

@nijave FTR, we haven't gotten complaints from other users about the guide. and yes, it has users.

@mbert do the above suggestions seem good to you?

/kind documentation
/area ha

Is this also relevant to SIG Docs? I can't tell.

If kubeadm is OK and a page inside https://k8s.io/docs/ is out of date: transfer this to k/website and label it for SIG Cluster Lifecycle.

If kubeadm is OK and a page inside https://k8s.io/docs/ is out of date: transfer this to k/website and label it for SIG Cluster Lifecycle.

This one which I think official docs link to (but not seeing the link at the moment) https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md

yes, https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md ended up in this repo.

the link to it is here:

Read the Options for Software Load Balancing guide for more details.

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/#first-steps-for-both-methods

If we can move that doc into the main website, I think that benefits our users. Not required though.

moving it might break existing URL references.

also frankly, i would prefer if kubeadm docs eventually start moving in the other direction - i.e. to this repo. if this repo starts hosting kubeadm source code and it has versioned branches.

the kubeadm docs can be hosted similarly to other projects with netlify / hugo:

• https://kind.sigs.k8s.io/
• https://cluster-api.sigs.k8s.io/
• https://minikube.sigs.k8s.io/docs/

but that's a discussion for another time.
and it's a lot of work.

@nijave FTR, we haven't gotten complaints from other users about the guide. and yes, it has users.

@mbert do the above suggestions seem good to you?

/kind documentation /area ha

From first glance (have been unable so far to thoroughly review things in my development environment): it is true that HAProxy has undergone some changes leading to different configuration files. If this turns out to be an issue here, then providing two sets of example configuration would make sense.

As the rest of the report is concerned I will need to take a closer look. Improvements are always welcome. I hope to be able to do this on the weekend.

thanks @mbert
if we can include only the latest config, that seems better to me, so that we can avoid being in the business of tracking N versions.

thanks @mbert if we can include only the latest config, that seems better to me, so that we can avoid being in the business of tracking N versions.

Let's see. For now still both versions are used in the field, because the older one is still present in EL distros.

thanks @mbert if we can include only the latest config, that seems better to me, so that we can avoid being in the business of tracking N versions.

Let's see. For now still both versions are used in the field, because the older one is still present in EL distros.

that's a good point. users sometimes just install from the distro packaging.

I have now had some time to read through everything. First of all: I totally agree with @nijave - the guide is outdated here, and I think a PR with the proposed changes would be very welcome.

Actually the examples in the guide were, IIRC, created using HAProxy 1.8 on an EL7 platform, and given the changes in HAProxy since then the configuration example should really get updated. Since what I still have (I am not actively using the setup in my environment, hence experimenting would require me setting things up again before) is all based on that "ancient" HAProxy I cannot quickly provide a configuration for version 2.1, because I have never had one in use. Long story short: I propose to provide the configuration example for 2.8 as seen above (assuming that it has been tested and works) along with mentioning the version and the fact that this may not work for other versions.

Regarding the health check: again, I totally agree that the HAProxy should be checked on all nodes, not only the one with the VIP. Thank you for noticing!

@nijave would you send a PR for this as you suggested earlier?

I'm running HA Proxy 2.4 (default in Ubuntu 22.04) without any custom healthcheck setups - and it works just fine:

frontend kube-cph
        bind :::6443 v4v6
        mode tcp
        timeout client 600s
        option tcplog
        default_backend kube-cph-api

backend kube-cph-api
        mode tcp
        timeout server 600s
        option log-health-checks
        server kube01 kube01:6443 check
        server kube02 kube02:6443 check
        server kube03 kube03:6443 check

I've been running an HA Proxy loadbalancer in front of my K8s clusters with similar configurations for at least four years, on several clusters, never had any problems.

@nijave would you send a PR for this as you suggested earlier?

Yeah, give me a few days

I'm running HA Proxy 2.4 (default in Ubuntu 22.04) without any custom healthcheck setups - and it works just fine:

frontend kube-cph
        bind :::6443 v4v6
        mode tcp
        timeout client 600s
        option tcplog
        default_backend kube-cph-api

backend kube-cph-api
        mode tcp
        timeout server 600s
        option log-health-checks
        server kube01 kube01:6443 check
        server kube02 kube02:6443 check
        server kube03 kube03:6443 check

I've been running an HA Proxy loadbalancer in front of my K8s clusters with similar configurations for at least four years, on several clusters, never had any problems.

I was able to get a working setup when using the current guide, however it didn't handle failures scenarios correctly (so it was load balanced, but not really highly available). With your setup, it doesn't look like you're checking api-server health, only if it accepts TCP connections. If api-server health check is failing, haproxy will still route traffic to those instances. An easy test is killing etcd on a node and observing api-server is still running but returning an error code for /healthz (in which case it should be removed from active backends in haproxy)

You're probably right. At home I have a bare metal cluster, and I routinely update the nodes, meaning that Kubernetes will be shutdown during reboots.

So far HA Proxy have handled this situation without any hickups - but let me have a look when I return home.

Did some further testing: This works for me.

backend kube-cph-api
        mode tcp
        timeout server 600s

        option log-health-checks
        option httpchk GET /healthz

        server kube01 kube01:6443 check check-ssl verify none
        server kube02 kube02:6443 check check-ssl verify none
        server kube03 kube03:6443 check check-ssl verify none

Poked around and it looks like Ubuntu and recent version of RHEL (clones) are on HAProxy v2.4 (LTS) which appears to also work with the config I mentioned above

OS - version (EOL date for community support)

RHEL 7 - v1.5.18 (30 Jun 2024)
RHEL 8 - v1.8.27 (31 May 2029)
RHEL 9 - v2.4.22 (31 May 2032)

Ubuntu 20.04 - v2.0.33 (02 Apr 2025)
Ubuntu 22.04 - v2.4.24 (01 Apr 2027)
Ubuntu 24.04 - v2.8.5 (25 Apr 2036)