Upgrading kubeadm clusters

This blog will help you to upgrade the Kubernetes cluster from v1.24.3 to v1.25.0

Prerequisites

• Make sure the cluster health

• Make sure you read the release notes carefully. release notes

• Make sure to back up any important components. ETCD backup and restoration

Upgradation workflow

• Upgrade a primary control plane node.
• Upgrade an additional control plane node
• Upgrade worker nodes

Upgrade master node

1. Upgrade kubeadm on the Control Plane node
2. Drain the Control Plane node
3. Plan the upgrade (kubeadm upgrade plan)
4. Apply the upgrade (kubeadm upgrade apply)
5. Upgrade kubelet & kubectl on the control Plane node
6. Uncordon the Control Plane node

Update the package manager

apt update
apt-cache madison kubeadm

Find the Latest version

root@Kubernet-Master:~# apt-cache madison kubeadm
   kubeadm |  1.25.0-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.24.4-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.24.3-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.24.2-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.24.1-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.24.0-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm | 1.23.10-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages

Check the Kubeadm current version

root@Kubernet-Master:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:29:09Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes node status and version

root@Kubernet-Master:~# kubectl get node
NAME                STATUS   ROLES           AGE   VERSION
kubernet-master     Ready    control-plane   36d   v1.24.3
kubernet-worker     Ready    worker          36d   v1.24.3
kubernet-worker02   Ready    worker02        20d   v1.24.3

Upgrade the kubeadm

apt-mark unhold kubeadm &&  apt-get update && apt-get install -y kubeadm=1.25.0-00 &&  apt-mark hold kubeadm

Verify the latest version

root@Kubernet-Master:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:43:25Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}

Verify the upgrade plan

root@Kubernet-Master:~# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     3 x v1.24.3   v1.25.0

Upgrade to the latest stable version:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.24.3   v1.25.0
kube-controller-manager   v1.24.3   v1.25.0
kube-scheduler            v1.24.3   v1.25.0
kube-proxy                v1.24.3   v1.25.0
CoreDNS                   v1.8.6    v1.9.3
etcd                      3.5.3-0   3.5.4-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.25.0

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

Choose the Version and run the upgrade command

root@Kubernet-Master:~#  kubeadm upgrade apply v1.25.0
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.25.0"
[upgrade/versions] Cluster version: v1.24.3
[upgrade/versions] kubeadm version: v1.25.0
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.25.0". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

Drain the master node

root@Kubernet-Master:~# kubectl drain kubernet-master --ignore-daemonsets
node/kubernet-master cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-5cptw, kube-system/kube-proxy-gfhms
node/kubernet-master drained

Verifying the drain status

root@Kubernet-Master:~# kubectl get node
NAME                STATUS                     ROLES           AGE   VERSION
kubernet-master     Ready,SchedulingDisabled   control-plane   37d   v1.24.3
kubernet-worker     Ready                      worker          37d   v1.24.3
kubernet-worker02   Ready                      worker02        20d   v1.24.3

Upgrade kubelet and kubectl

root@Kubernet-Master:~# apt-mark unhold kubelet kubectl && \
> apt-get update && apt-get install -y kubelet=1.25.0-00 kubectl=1.25.0-00 && \
> apt-mark hold kubelet kubectl
kubelet was already not hold.
kubectl was already not hold.
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:3 http://ci.archive.ubuntu.com/ubuntu focal InRelease
Get:4 http://ci.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 DEP-11 Metadata [40.8 kB]
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.25.0-00 [9,500 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.25.0-00 [19.5 MB]
Fetched 29.0 MB in 9s (3,311 kB/s)
(Reading database ... 184662 files and directories currently installed.)
Preparing to unpack .../kubectl_1.25.0-00_amd64.deb ...
Unpacking kubectl (1.25.0-00) over (1.24.3-00) ...
Preparing to unpack .../kubelet_1.25.0-00_amd64.deb ...
Unpacking kubelet (1.25.0-00) over (1.24.3-00) ...
Setting up kubectl (1.25.0-00) ...
Setting up kubelet (1.25.0-00) ...
kubelet set on hold.
kubectl set on hold.

Restart the kubelet:

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Uncordon the node

root@Kubernet-Master:~# kubectl uncordon kubernet-master
node/kubernet-master uncordoned

Verifying the master node status and check the version

root@Kubernet-Master:~# kubectl get node
NAME                STATUS   ROLES           AGE   VERSION
kubernet-master     Ready    control-plane   37d   v1.25.0
kubernet-worker     Ready    worker          37d   v1.24.3
kubernet-worker02   Ready    worker02        20d   v1.24.3
root@Kubernet-Master:~#

Upgrading the worker nodes

The upgrade procedure on worker nodes should be run one node at a time or a few nodes at a time, without compromising the minimum required capacity for running your workloads.

Upgrading the worker nodes consist of the following steps:

1. Drain the node
2. Upgrade kubeadm on the node
3. Upgrade kubelet & kubectl
4. Uncordon the node

ssh into the worker node

Upgrade kubeadm

apt-mark unhold kubeadm && \
 apt-get update && apt-get install -y kubeadm=1.25.0-00 && \
 apt-mark hold kubeadm

Kubeadm upgrade

root@Kubernet-Worker:~# sudo kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

Drain the worker node

kubectl drain kuernet-worker --ignore-daemonsets

Upgrade kubelet and kubectl

apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.25.0-00 kubectl=1.25.0-00 && \
apt-mark hold kubelet kubectl

Restart the kubelet

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Uncordon the node

root@Kubernet-Master:~# kubectl uncordon kubernet-worker
node/kubernet-worker uncordoned

Verify the status of the cluster

root@Kubernet-Master:~# kubectl get node
NAME                STATUS   ROLES           AGE   VERSION
kubernet-master     Ready    control-plane   37d   v1.25.0
kubernet-worker     Ready    worker          37d   v1.25.0
kubernet-worker02   Ready    worker02        20d   v1.24.3

For the other worker node upgrade follow the same steps:

root@Kubernet-Master:~# kubectl get node
NAME                STATUS   ROLES           AGE   VERSION
kubernet-master     Ready    control-plane   37d   v1.25.0
kubernet-worker     Ready    worker          37d   v1.25.0
kubernet-worker02   Ready    worker02        20d   v1.25.0

Hope you got an idea of how to upgrade the kubernetes cluster

Thank you!

Related/References

blog.cloudnloud.com/series/kubernetes-deep-.. kubernetes.io/docs/tasks/administer-cluster..