This is home to my personal Kubernetes lab cluster. Flux watches this Git repository and makes the changes to my cluster based on the manifests in the kubernetes directory. Renovate also watches this Git repository and creates pull requests when it finds updates to Docker images, Helm charts, and other dependencies.
There is a template over at onedr0p/flux-cluster-template if you wanted to try and follow along with some of the practices I use here.
My cluster is talos running on proxmox VMs. This is a semi hyper-converged cluster, workloads are sharing the same available resources on my nodes while I have a separate server for data storage.
🔸 Click here to see my Ansible playbooks and roles.
- cilium: Internal Kubernetes networking plugin
- cert-manager: Creates SSL certificates for services in my Kubernetes cluster
- external-dns: Automatically manages DNS records from my cluster in a cloud DNS provider
- external-secrets: Managed Kubernetes secrets using 1Password Connect
- ingress-nginx: Ingress controller to expose HTTP traffic to pods over DNS
- sops: Managed secrets for Kubernetes, Ansible and Terraform which are commited to Git
- Democratic CSI: Provides block and NFS storage provisioning
- volsync: Backup and recovery of persistent volume claims
Flux watches my kubernetes folder (see Directories below) and makes the changes to my cluster based on the YAML manifests.
Renovate watches my entire repository looking for dependency updates, when they are found a PR is automatically created. When some PRs are merged Flux applies the changes to my cluster.
This Git repository contains the following directories under kubernetes.
📁 kubernetes # Kubernetes cluster defined as code
├── 📁 talos # main cluster
│ ├─📁 apps # applications
│ ├─📁 bootstrap # bootstrap procedures
│ └─📁 flux # core flux configuration
└── 📁 truenas # Truenas k3s
├─📁 apps # applications
├─📁 bootstrap # bootstrap procedures
└─📁 flux # core flux configuration| Name | CIDR |
|---|---|
| Network VLAN | 10.1.1.0/24 |
| Servers VLAN | 10.1.2.0/24 |
| TrueNAS external services (BGP) | 10.84.1.0/24 |
| Talos external services (BGP) | 10.84.2.0/24 |
| Kubernetes pods | 172.16.0.0/16 |
| Kubernetes services | 10.100.0.0/16 |
While most of my infrastructure and workloads are selfhosted I do rely upon the cloud for certain key parts of my setup. This saves me from having to worry about two things. (1) Dealing with chicken/egg scenarios and (2) services I critically need whether my cluster is online or not.
The alternative solution to these two problems would be to host a Kubernetes cluster in the cloud and deploy applications like HCVault, Vaultwarden, ntfy, and Gatus. However, maintaining another cluster and monitoring another group of workloads is a lot more time and effort than I am willing to put in and only saves me roughly $18/month.
| Service | Use | Cost |
|---|---|---|
| 1Password | Secrets with External Secrets | 73Eur/yr |
| Cloudflare | Domain, DNS and proxy management | Free |
| Fastmail | Email hosting | $75/yr |
| GitHub | Hosting this repository and continuous integration/deployments | Free |
| Terraform Cloud | Storing Terraform state | Free |
| Total: $12.75/mo |
On my Vyos router I have Bind9 and dnsdist deployed as containers. In my cluster external-dns is deployed with the RFC2136 provider which syncs DNS records to bind9.
Downstream DNS servers configured in dnsdist such as bind9 (above) and AdGuard DNS. All my clients use dnsdist as the upstream DNS server, this allows for more granularity with configuring DNS across my networks. These could be things like giving each of my VLANs a specific adguard profile, or having all requests for my domain forward to bind9 on certain networks, or only using 1.1.1.1 instead of adguard on certain networks where adblocking isn't required.
Outside the external-dns instance mentioned above another instance is deployed in my cluster and configured to sync DNS records to Cloudflare. The only ingress this external-dns instance looks at to gather DNS records to put in Cloudflare are ones that have an ingress class name of external and contain an ingress annotation external-dns.alpha.kubernetes.io/target.
| Device | Count | OS Disk Size | Data Disk Size | Ram | Operating System | Purpose |
|---|---|---|---|---|---|---|
| GOWIN GW-BS-1UR2 | 1 | 512GB NVMe | - | 32GB | VyOS | Router |
| Unifi USW-Enterprise-24-POE | 1 | - | - | - | - | Network Switch |
| Dell Optiplex 7040 | 4 | 256GB NVMe | - | 64GB | Debian 12 (PVE) | Virtualization Host |
| Cyberpower OR600ERM1U | 1 | - | - | - | - | UPS |
| QNAP TVS-682 | 1 | 2x256GB SATA | 2x512GB SSD + 4x4TB HDD | 32GB | TrueNAS Scale | NAS |
| ESP32+Ebyte 72 POE adapter | 1 | - | - | - | ESPHome | Zigbee adapter |
Thanks to all the people who donate their time to the Home Operations Discord community. A lot of inspiration for my cluster comes from the people that have shared their clusters using the k8s-at-home GitHub topic. Be sure to check out the kubesearch.dev for ideas on how to deploy applications or get ideas on what you can deploy.
See the awful commit history
See LICENSE