Release Proposal v2.16

[floryut]

Below is a proposal for release notes:

Announcements

We are looking for maintainers, reach out in #5432.

Deprecation / Removal

• Remove contrib/vault (Outdated since 2018) (Remove vault #7400)
• Drop support for calico version 3.15.x (Drop calico 3.15 #7545)

Major changes

• Replace inventory group kube-master with kube_control_plane (Replace the inventory group kube-master with kube_control_plane #7256) (see Notes 5)
• Move kubernetes/master to kubernetes/control-plane (Move kubernetes/master to kubernetes/control-plane #7218) (see Notes 1)
• Move recover_control_plane/master to control-plane (Move recover_control_plane/master to control-plane #7236) (see Notes 2)
• Replace KUBE_MASTERS with KUBE_CONTROL_HOSTS (Replace KUBE_MASTERS with KUBE_CONTROL_HOSTS #7257) (see Notes 3)
• Rename ansible groups to use _ instead of - (Rename ansible groups to use _ instead of - #7552) (see Notes 7)
• Add AlmaLinux support (Initial AlmaLinux support #7538)
• Add terraform support for Exoscale (Added terraform support for Exoscale #7141)
• Add terraform support for Vsphere (Vsphere #7306)
• Add terraform support for UpCloud (Terraform support for UpCloud #7360)
• Support for CentOS 8 and derivatives is considered stable (remove experimental note about CentOS 8 and derivatives #7615)
• Support dual stack IPv4 & IPv6 networking (Support dual stack IPv4 & IPv6 networking #6859)
• Auto renew control plane certificates (Auto renew control plane certificates #7358) (see Notes 4)
• Add auto_renew_certificates_systemd_calendar to configure when K8S certificates renewal runs (Add auto_renew_certificates_systemd_calendar #7490)
• Specify runAsGroup, allow safe sysctls by default (specify runAsGroup, allow safe sysctls by default #7399)
• Add KubeSchedulerConfiguration for k8s 1.19 and up (Add KubeSchedulerConfiguration for k8s 1.19 and up #7351) (see Notes 6)
• Add script for generate download files and images list (Add script for generate download files and images list #7561)
• Terraform 0.12+ is now required to run scripts under contrib/terraform/aws (Terraform no longer supports the map function #7576)
• Allow using ansible 2.10.x to deploy Kubespray (add support for using ansible 2.10.x for deploying kubespray #7600)
• Add a contrib playbook (os-manage) to disable service firewall for Kubespray development and test (Add contrib playbook to disable service firewall #7431)

Applications

• [Krew] Add krew support (Add krew support #7464)
• [Openstack] Make sure worker rules is applied on workers ([Openstack] Make sure worker nodes have the worker extra security group attached #7279)
• [Openstack] Write openstack controller manifests with correct perms (Write openstack cloud controller manifests with correct permission #7284)
• [Openstack] Allow users to set image_uuid instead of name, this allows the use of openstack community images ([Openstack] Make it possible to define image using uuid #7283)
• [Openstack] Use image id instad of name ([Openstack] Make python hosts output image id instead of name #7293)
• [Openstack] Update Cinder CSI driver to v1.20.0 ([Openstack] Update Cinder CSI driver to v1.20.0 #7280)
• [Openstack] Add most_recent = true while retrieving the latest image (Add most_recent = true while retrieving the latest image #7376)
• [Openstack] Add external_openstack_enable_ingress_hostname option for external-openstack-cloud-controller-manager (Add external_openstack_enable_ingress_hostname option for external-openstack-cloud-controller-manager #7572)
• [Metallb] Introduces optional tolerations and nodeSelector for metallb components (controller and speaker) (Add nodeselector and tolerations for metallb #7334)
• [CSI] Add suport of Vsphere CSI driver 2.X versions (Add suport of Vsphere CSI driver 2.X versions #7480)
• [External-Provisioner] Add new variable "local_volume_provisioner_use_node_name_only" to configure local volume provisioner "useNodeNameOnly" option (allow local provisioner 'useNodeNameOnly' option to be configured #7421)

Container managers

• [CRI-O] Add experimental cri-o support for Amazon Linux 2 (Added experimental cri-o support for Amazon Linux 2 #7353)
• [CRI-O] Add support for configuring cri-o pids_limit (add support for configuring cri-o pids_limit #7525)
• [CRI-O] Fix support for cri-o on OracleLinux and add support for AlmaLinux (Fix cri-o support for Oracle and AlmaLinux #7541)
• [Containerd] Fix reset.yml failing when using containerd (Fix reset when using containerd #7308)
• [Containerd] Add privileged_without_host_devices support (Add privileged_without_host_devices support #7343)
• [Containerd] Update config.toml to V2 and set default runtime to io.containerd.runc.v2 and cgroup to systemd (Set containerd version to 1.4.4 #7398)
• [Containerd] Add containerd_extra_args (Add containerd_extra_args #7461)
• [Containerd] Add nerdctl cli tool for containerd users (Add nerdctl cli tool for containerd user #7500)
• [Containerd] Add support for Amazon Linux 2(add containerd support for Amazon Linux 2 #7595)
• [Docker] docker_dns_servers_strict had different default values, the default is now the same everywhere: false (Update some docker defaults #7499)
• [Docker] Add enablerepo: amzn2extra-docker to allow docker installation on Amazon linux (add enablerepo: amzn2extra-docker for docker install on aws 2 #7507)
• [crun] Update and changed the default crun version to v0.19 (Crun v0.19 support #7433)
• [crictl] Change the owner of /etc/crictl.yaml to root (Change the owner of the /etc/crictl.yaml to root #7254)

Network

• [Calico] Fixup check when ipipMode / vxlanMode is not present (Calico: fixup check when ipipMode / vxlanMode is not present #7195)
• [Calico] Support for dual stack (IPv4 & IPv6) network deployment using Calico is introduced as an opt-in feature (Support dual stack IPv4 & IPv6 networking #6859)
• [Calico] Add option to use calico with azure when using calico in vxlan (add option to use calico with azure when using calico in vxlan #7300)
• [Calico] Download Calico KDD CRDs (Download Calico KDD CRDs #7372)
• [Calico] Add the ability to customize calico's bird port, via calico_bird_listen_port variable (add support for custom bird calico port #7419)
• [Calico] Add new variable calico_node_startup_loglevel to configure CALICO_STARTUP_LOGLEVEL (Default to error) (Fixes issue #7528 - allow configuring CALICO_STARTUP_LOGLEVEL #7530)
• [Calico] Allow specifying overriding BGP peer name (allow overriding calico peers names and avoid ipv6 naming issues #7591)
• [Calico] Enables Calico serviceAccount token monitoring and update of /etc/cni/net.d/calico-kubeconfig if need be (Enable calico CNI kubeconfig token monitor #7586)
• [Calico] Add support to advertise MetalLB allocated IPs through Calico when using Calico 3.18 and greater (Support Calico advertisement of MetalLB LoadBalancer IPs #7593)
• [Cilium] Allow cilium to be deployed with transparent encryption (Add support for cilium ipsec #7342)
• [Cilium] Add cilium_ipam_mode variable (Add cilium_ipam_mode variable #7418)
• [Cilium] Move cilium kvstore settings to configmap (Move cilium kvstore settings to configmap #7462)
• [Cilium] Update Cilium documentation and overall update of cilium role (Upgrade cilium role #7521)
• [Ambassador] Add ingress_ambassador_multi_namespace setting, allows Ambassador operator to watch all namespaces for AmbassadorInstallation CRD resources (Ambassador can watch multiple namespaces #7516)
• [Flannel] Add image_arch in image tag (Add image_arch in flannel image tag #7560)

Other note worthy changes

• Added the ping_access_ip variable to enable(default)/disable ping test during preinstall (Add ping_access_ip to control ping test #7020)
• Rework proxy support (Rework proxy support #7095)
• Remove ignore_errors from drain tasks and enable retires (Remove ignore_errors from drain tasks and enable retires #7151)
• Add other masters sequentially, not in parallel (Adding other masters sequentially, not in parallel #7166)
• Add 2 variables for upgrade, to prompt (upgrade_node_confirm, default false) and delay (upgrade_node_pause_seconds, default 0 seconds) (Add prompt to upgrade node or delay before upgrade #7168)
• Change node-role.kubernetes.io from master to control-plane (Change node-role.kubernetes.io from master to control-plane #7183)
• Add retries to drain during upgrade. Allow leaving nodes cordoned after drain failure. Allow continuing upgrade if drain fails (FIX: Bastion undefined variable #7227)
• Vagrantfile: always recreate inventory symlink (Vagrantfile: always recreate inventory symlink #7245)
• Updated etcd cert check tasks to detect when new cert gen is required (Updated etcd cert check tasks to detect when new cert gen is required #7219)
• Only use stat get_checksum: yes when needed (Only use stat get_checksum: yes when needed #7270)
• Match on os-release ID / VARIANT_ID (bootstrap-os: match on os-release ID / VARIANT_ID #7269)
• Fix issue with kubeadm when *_PROXY variables are present in the environment (Ensure kubeadm doesn't use proxy #7275)
• Kubespray now ignores *_PROXY vars found in your environment and only uses proxy configuration from the inventory (Fix proxy usage when *_PROXY are present in environment #7309)
• Facts.yaml: reduce the number of setup calls by ~7x (facts.yaml: reduce the number of setup calls by ~7x #7286)
• Fixup kubelet.conf to point to kubelet-client-current.pem (Fixup kubelet.conf to point to kubelet-client-current.pem #7347)
• Check for dummy kernel module (Check for dummy kernel module #7348)
• Disable gather_facts for correctly work via bastion (Disable gather_facts for correctly work via bastion #7265)
• Add etcd max snapshot and wals (Add etcd max snapshot and wals #7382)
• Add cryptography module installation (Add cryptography installation #7404)
• Allow connecting to bastion via non-standard SSH port (Allow connecting to bastion via non-standard SSH port #7396)
• Remove local lb privileged securityContext (remove local lb privileged #7437)
• Regenerate apiserver.crt on all controle-plane nodes when needed instead of just the first one (Regenerate apiserver.crt on all control-plane nodes #7463)
• Check if python netaddr is installed and if Jinja is recent enough (Check if python netaddr and recent enough jinja are installed #7486)
• Add ingress controller ingress-class var (Add ingress controller ingress-class var #7522)
• Update Dockerfile to reduce Kubespray image size (Update Dockerfile for reduce image size #7556)
• Change kubeadm coredns addon images name to coredns/coredns (change kubeadm coredns addon images name to coredns/coredns #7570)
• Allow usage of jinja2_native=True (Fix pull_by_digest variable type to boolean instead of str #7612 / Fix cloud_resolver type from str to list (issue #7605) #7606)

Component versions:

• Kubernetes v1.20.7
• Etcd 3.4.13
• Docker 19.03
• Containerd 1.4.4
• CRI-O 1.20
• CNI-plugins v0.9.1
• Calico v3.17.4
• Cilium 1.8.9
• Flannel 0.13.0
• Kube-Router 1.2.2
• Multus 3.7
• Kube-ovn 1.6.2
• Weave 2.8.1
• CoreDNS 1.7.0
• Nodelocaldns 1.17.1
• Helm 3.5.4
• Nginx-ingress 0.43.0
• Cert-manager 1.0.4
• Kubernetes Dashboard v2.2.0

Known issues

• Ansible 2.11 is not supported and using it will results in errors
• Using Docker container engine could prompt "PLEG IS NOT HEALTHY" error, due to a runc bug, please see this issue for more information.

Notes

1. The role kubernetes/master has been renamed to kubernetes/control-plane, if using the role kubernetes/master solely on previous Kubespray, it is necessary to update the specified role.
2. The role recover_control_plane/master has been renamed to recover_control_plane/control-plane. If using the role recover_control_plane/master solely on previous Kubespray, it is necessary to update the specified role.
3. inventory_builder starts referring the environment variable KUBE_CONTROL_HOSTS to get the number of control-plane nodes, it still refers KUBE_MASTERS but it will be not referred after some deprecation cycles. Please specify KUBE_CONTROL_HOSTS if now specifying KUBE_MASTERS
4. You can enable control plane certificates automatic renewal using auto_renew_certificates, or manually use k8s-certs-renew.sh force_certificate_regeneration is removed as it was only renewing the api server certs and not all the other ones
5. The inventory group kube-master has been renamed to kube_control_plane. Please update your inventory file by replacing kube-master if continuing to use the existing inventory file.
6. New vars for configuring kube-scheduler were introduced (including extenders and profiles). Default vaules can be found at roles/kubernetes/control-plane/defaults/main/kube-scheduler.yml
7. Ansible groups were updated to be more consistent with dynamic inventory plugins: k8s-cluster -> k8s_cluster / k8s-node -> k8s_node / calico-rr -> calico_rr / no-floating -> no_floating

[cristicalin]

Should we consider support for centos 8 / alma 8 stable in this release?

[oomichi]

I'd like to add #7431 also for this release if possible.
That is a contrib playbook and that doesn't change the existing features on the main playbook, but useful I think.

[oomichi]

/cc @oomichi

[floryut]

Should we consider support for centos 8 / alma 8 stable in this release?

Nothing against that on my end, not using centos 8 though but the experimental part has been there for like 6 months so I guess, not that much experimental now 😄

[champtar]

Most of my productions are CentOS 8 or RHEL 8 + containerd + Calico so it works for me ;)

[f9n]

I'd like to add #7622 also for this release if possible.

[bac-w]

And #7521 also please add

[floryut]

And #7521 also please add

Done

I'd like to add #7622 also for this release if possible.

Added to milestone

[fungusakafungus]

I believe on of the links to PRs is wrong, it probably should be #7398:

-[Containerd] Update config.toml to V2 and set default runtime to io.containerd.runc.v2 and cgroup to systemd (#7343)
+[Containerd] Update config.toml to V2 and set default runtime to io.containerd.runc.v2 and cgroup to systemd (#7398)

[floryut]

I believe on of the links to PRs is wrong, it probably should be #7398:

-[Containerd] Update config.toml to V2 and set default runtime to io.containerd.runc.v2 and cgroup to systemd (#7343)
+[Containerd] Update config.toml to V2 and set default runtime to io.containerd.runc.v2 and cgroup to systemd (#7398)

Thanks, some release note not up to date in PR leads to manual error from my end 😄

[jvleminc]

Do you have a (vague) release date for this version? We're impatiently waiting for the K8S 1.20.* integration in a formal release. :-)

[floryut]

Right now ;)

[jvleminc]

Thank you very much!

[oomichi]

@floryut Thank you!

[floryut]

@floryut Thank you!

Thanks to you too man 🙇