Allow to disable public IP assignment on nodes

[beslovas]

Description

I believe it is not necessary for nodes to have public IP if they are placed under load-balancer, or communicating using private network. Terraform provider for hetzner-cloud supports this: https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/server#basic-server-creation

[mysticaltech]

@beslovas It is still necessary for changes and setup. But you can completely isolate the node via the firewall rules, see kube.tf.example, it's basically equivalent of having no public IPs.

However, in future versions of the project definitly ship this if we can.

[beslovas]

In my case(s), I first setup a VPN, and use private network to manage k8s. So public IPs for nodes are absolutely unnecessary.

[WebSpider]

@mysticaltech yes, at current it is. You could, however, solve this with either a (temporary?) box that serves as an SSH JumpHost, or use the control-plane loadbalancer to use a control-plane node as SSH JumpHost

[mysticaltech]

@WebSpider Makes sense indeed, if you want to add that feature via a flag, PR welcome 🙏

[WebSpider]

It's been a while since I did terraform development, but I welcome the challenge 👍

I've also found that we could pass a flag to cluster-autoscaler to not request a IPv4 address for an autoscaled worker. Would you like that as a seperate PR, or included?

[mysticaltech]

@WebSpider Sorry for the delay in repliying, separates PRs would be good unless they are related in which case you can definitely do both in one. Thanks!

[mkreis]

Hetzner has a limit on public IPs (by default 20) and each is billed with € 1.70/month. it seems quite wasteful to assign them and then hide them behind a firewall.

[gerwim]

Yes, I'd rather have a public IPv6 version only support. I don't need an IPv4.

[WebSpider]

I've been experimenting with this. There are some small practical issues that make this a bit more complex than you'd expect.
The main one, is that K3s resides on GitHub. Both GitHub and GitHub's Container Registry do not have an IPv6 address.

You could probably work around the limitation of GHCR by proxying the container images using K3's internal registry proxy, but as it is, the new nodes cannot download K3s without having IPv4 connectivity to GitHub.

[mkreis]

Wow, I had to double check that because I couldn't believe that Github isn't supporting IPv6 yet...

[mkreis]

https://isgithubipv6.live/

[pgruener]

Perhaps it could be solved with combination of egress node_pool as a requirement, that the cluster is able to communicate outside with a floating_ip but not all nodes need to have its own?

[WebSpider]

I have solved this for myself with a combination of 2 options:

1. k3s registry proxy option
2. a seperate vm that runs tinyproxy and has both ipv4 and ipv6 active added as http/https proxy in the hosts environment

This solves the case of getting a cluster up and running, but there still are some caveats getting terraform to do it, that mainly have to do with accessing the cluster while it is being built