Moving "Agentless Servers" out of Experimental Feature for K3S Project

[karthik1729]

As a user of K3S, I have been exploring the available features and have been particularly interested in the "Agentless Servers" functionality. I have found this feature to be quite useful and have been successfully utilizing it in my projects. However, I noticed that it is still classified as an experimental feature.

I would like to inquire about the current status of the "Agentless Servers" feature and the potential issues that may be preventing it from moving out of the experimental stage. I understand that experimental features may not be fully stable or production-ready, and I would like to know about any known issues or limitations associated with the "Agentless Servers" feature.

Additionally, I would like to understand the process of moving a feature out of the experimental stage in the K3S project. This information would be helpful for the community members who are interested in contributing to the project and ensuring the stability and reliability of the "Agentless Servers" feature.

To facilitate this discussion, I kindly request the following information:

Known Issues: Please provide a list of known issues or limitations associated with the "Agentless Servers" feature. This will help us understand the current state of the feature and the improvements needed before it can be considered for a stable release.

Process for Moving Out of Experimental Stage: Could you outline the process for moving a feature from experimental to stable in the K3S project? This will help the community understand the steps and requirements necessary for the "Agentless Servers" feature to graduate from experimental status.

How to Contribute: If there are any specific areas where the community can contribute to help expedite the process of moving "Agentless Servers" out of the experimental stage, please let us know. This information will enable us to actively participate in the project and work towards a stable release of the feature.

I believe that the "Agentless Servers" feature has the potential to greatly benefit the K3S project and its users, and I am eager to see it become a stable and reliable feature. I appreciate your time and consideration in addressing this request, and I look forward to your response.

[brandond]

You're asking for a fair bit of documentation and personal labor on our side to produce the detailed materials you're asking for. I'll do my best to answer, but I do want to mention that verbose requests for written materials, even when phrased politely, can still be irritating to open-source project development staff.

provide a list of known issues or limitations associated with the "Agentless Servers" feature.

This is covered in the docs: https://docs.k3s.io/advanced#running-agentless-servers-experimental

Because they do not host a kubelet, they cannot run pods or be managed by operators that rely on enumerating cluster nodes, including the embedded etcd controller and the system upgrade controller.

Consider also any other controller or operator that wants to manage hosts via their node object - Rancher for example.

There are other issues with apiserver access to webhooks and apiserver aggregation endpoints within the cluster (metrics-server for example), but the egress selector code that was added a while back (--egress-selector-mode=pod/cluster) handles most of that.

outline the process for moving a feature from experimental to stable in the K3S project?

In general, we need an identified use case that we can ask our QA team to test and support. K3s is a CNCF project, but the development and QA staff are currently employed by SUSE/Rancher, so we do prioritize use cases that we see our community and customers taking advantage of, or those prioritized by our product management team. That includes the ability to use a feature alongside other ecosystem projects such as Rancher, System Upgrade Controller, Longhorn, and so on.

I'm not aware of any customers that are using it, and I have only seen a few community members using it. The only one I can think of off the top of my head was the vcluster folks at #5118

specific areas where the community can contribute to help expedite the process of moving "Agentless Servers" out of the experimental stage

Community members can help by identifying use cases and speaking up if they are actively using this feature in their environment. The more we understand about how the community is using it, the better we can support it on our side.

I believe that the "Agentless Servers" feature has the potential to greatly benefit the K3S project and its users.

Can you expand upon this? What sort of potential do you see for this feature? How would you like to see it used and supported?

[mwthink]

@brandond I know its a bit late of a reply but I've been using this functionality and wanted to share some thoughts. I've been running an agentless server in my homelab, in a docker container on my NAS.

A bit of background: I run a small homelab that consists of mix-n-match consumer hardware, both x86-64 and ARM64 architectures. At the core I run a stateful, snowflake NAS which runs a few services consumes by other machines; notably a postgres server and some network-boot configs (DHCP/TFTP). Most of the other instances are k3s nodes and they all load their boot/installation media over the network which allows me to do some "cloud-like" functionality (i.e. Re-imaging machines without physically handling them).

Historically I would always have to have some special snowflake config for my server node as it would need to get the database credentials. Everything else would follow the same, 1-size-fits-all boot process to load the OS and join the cluster as a Node, but the server was always something special. I wanted to remove this additional snowflake config as I don't like having state in places other than the 1 special snowflake meant for storing state (the NAS).

I have tried to run traditional k3s (with an agent) on the NAS before but the OS (Synology DSM) never played nicely with it and running it in a VM on the NAS was undesired overhead. So I lived with this setup for awhile, and it was okay. Until I saw that the --disable-agent flag existed.

With this flag, my thinking was that I could get-by running the k3s-server in a docker container on the NAS. Since it wouldn't need to run into any of the agent components, I figured it would avoid the failures caused by the OS (read-only filesystem, non-standard dir structure). After some tinkering, I eventually landed on running my server with

DATA_ROOT="/volume1/k3s/homelab"

docker run -d --name k3s-server --restart=always \
  -net=host \
  -v ${DATA_ROOT}/home:/home/k3s \
  -v ${DATA_ROOT}/etc-rancher:/etc/rancher \
  -v ${DATA_ROOT}/var-lib-rancher:/var/lib/rancher \
  --user 1000 \
  -e K3S_TOKEN=mytoken \
  -e HOME=/home/k3s \
  docker.io/rancher/k3s:v1.27.4-k3s1 \
  server \
    --datastore-endpoint=postgres://username:password@dbhost:5432/dbname?sslmode=require \
    --disable=traefik --disable=servicelb \
    --flannel-backend=wireguard-native \
    --egress-selector-mode=pod \
    --disable-agent

This setup has worked great and I hope to continue using it going forward. My only problems encountered were solved by

• the addition of the --egress-selector-mode=pod option
• running the container as --net=host instead of just passing the port -p 6443:6443

I suspect both of these problems are related but I have not done a deep-dive to analyze why exactly this is. I've read the docs on --egress-selector-mode multiple times and I still can't wrap my head around what exactly is being determined there, I think the documentation could be massively improved in that regard.

[ysolis]

do you have one more thing ... you are using an external etcd datastore, in your case, a postgresql database. In my experience, multimaster k3s with the embedded etcd give problems for some network communication problem (master servers in different datacenters, networking issues...) and this is the normal way to deploy k3s for several users.

personally i prefer to include a taint for only run in the server cluster instances tasks related to cluster/orchestation/management.