• 一次升级 k8s 集群版本的过程

    • 2023-04-07 22:49
    • 字数 2,622
    • 阅读 588

    本文记录了一次升级 k8s 集群版本的过程,包括 k8s-master、k8s-node1 及 k8s-node2 一主两从架构。为了保证版本的兼容性,每次迭代一个版本,并且升级流程为从 k8s-master 到 k8s-node 节点。

    一、升级 k8s-master

    1.1 查看当前节点

    [root@k8s-master ~]# kubectl get nodes
    NAME         STATUS   ROLES           AGE    VERSION
    k8s-master   Ready    control-plane   128d   v1.24.8
    k8s-node1    Ready    <none>          128d   v1.24.8
    k8s-node2    Ready    <none>          128d   v1.24.8

    1.2 查找最新版本

    [root@k8s-master ~]# yum list --showduplicates kubeadm
    已加载插件:fastestmirror
    Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
    Determining fastest mirrors
     * base: mirrors.aliyun.com
     * extras: mirrors.aliyun.com
     * updates: mirrors.aliyun.com
    已安装的软件包
    kubeadm.x86_64                       1.24.8-0                        @kubernetes
    可安装的软件包
    kubeadm.x86_64                       1.25.0-0                        kubernetes
    kubeadm.x86_64                       1.25.1-0                        kubernetes
    kubeadm.x86_64                       1.25.2-0                        kubernetes
    kubeadm.x86_64                       1.25.3-0                        kubernetes
    kubeadm.x86_64                       1.25.4-0                        kubernetes
    
    1.3 升级 kubeadm
    上面查看的版本是 1.24.8,我们升级到 1.25.4
    [root@k8s-master ~]# yum install -y kubeadm-1.25.4-0 --disableexcludes=kubernetes
    已加载插件:fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirrors.aliyun.com
     * extras: mirrors.aliyun.com
     * updates: mirrors.aliyun.com
    base                                                     | 3.6 kB     00:00
    docker-ce-stable                                         | 3.5 kB     00:00
    extras                                                   | 2.9 kB     00:00
    kubernetes                                               | 1.4 kB     00:00
    updates                                                  | 2.9 kB     00:00
    (1/3): kubernetes/primary                                  | 126 kB   00:00
    (2/3): docker-ce-stable/7/x86_64/primary_db                | 102 kB   00:00
    (3/3): updates/7/x86_64/primary_db                         |  20 MB   00:02
    kubernetes                                                              941/941
    正在解决依赖关系
    --> 正在检查事务
    ---> 软件包 kubeadm.x86_64.0.1.24.8-0 将被 升级
    ---> 软件包 kubeadm.x86_64.0.1.25.4-0 将被 更新
    --> 解决依赖关系完成
    
    依赖关系解决
    
    ================================================================================
     Package          架构            版本                源                   大小
    ================================================================================
    正在更新:
     kubeadm          x86_64          1.25.4-0            kubernetes          9.8 M
    
    事务概要
    ================================================================================
    升级  1 软件包
    
    总下载量:9.8 M
    Downloading packages:
    Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
    40c1f30871f010cdc338ee1dfe78f25cb389e17cce6067fb2b9c3e0c55 | 9.8 MB   00:01
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      正在更新    : kubeadm-1.25.4-0.x86_64                                     1/2
      清理        : kubeadm-1.24.8-0.x86_64                                     2/2
      验证中      : kubeadm-1.25.4-0.x86_64                                     1/2
      验证中      : kubeadm-1.24.8-0.x86_64                                     2/2
    
    更新完毕:
      kubeadm.x86_64 0:1.25.4-0
    
    完毕!

    1.4 驱逐要升级节点

    [root@k8s-master ~]# kubectl drain k8s-master --ignore-daemonsets
    node/k8s-master cordoned
    WARNING: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-72vpn, kube-system/
    evicting pod kube-system/coredns-74586cf9b6-9wq4f
    evicting pod kube-system/coredns-74586cf9b6-qpn2s
    pod/coredns-74586cf9b6-qpn2s evicted
    pod/coredns-74586cf9b6-9wq4f evicted
    node/k8s-master drained

    1.5 检查当前节点,发现不可调度

    [root@k8s-master ~]# kubectl get nodes
    NAME         STATUS                     ROLES           AGE    VERSION
    k8s-master   Ready,SchedulingDisabled   control-plane   128d   v1.24.8
    k8s-node1    Ready                      <none>          128d   v1.24.8
    k8s-node2    Ready                      <none>          128d   v1.24.8

    1.6 检查集群是否可以升级,并获取升级的版本

    [root@k8s-master ~]# kubeadm upgrade plan
    [upgrade/config] Making sure the configuration is correct:
    [upgrade/config] Reading configuration from the cluster...
    [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
    [preflight] Running pre-flight checks.
    [upgrade] Running cluster health checks
    [upgrade] Fetching available versions to upgrade to
    [upgrade/versions] Cluster version: v1.24.8
    [upgrade/versions] kubeadm version: v1.25.4
    I0407 21:52:56.699876    9349 version.go:256] remote version is much newer: v1.26.3; falling back to: stable-1.25
    [upgrade/versions] Target version: v1.25.8
    [upgrade/versions] Latest version in the v1.24 series: v1.24.12
    
    Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
    COMPONENT   CURRENT       TARGET
    kubelet     3 x v1.24.8   v1.24.12
    
    Upgrade to the latest version in the v1.24 series:
    
    COMPONENT                 CURRENT   TARGET
    kube-apiserver            v1.24.8   v1.24.12
    kube-controller-manager   v1.24.8   v1.24.12
    kube-scheduler            v1.24.8   v1.24.12
    kube-proxy                v1.24.8   v1.24.12
    CoreDNS                   v1.8.6    v1.9.3
    etcd                      3.5.5-0   3.5.5-0
    
    You can now apply the upgrade by executing the following command:
    
            kubeadm upgrade apply v1.24.12
    
    _____________________________________________________________________
    
    Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
    COMPONENT   CURRENT       TARGET
    kubelet     3 x v1.24.8   v1.25.8
    
    Upgrade to the latest stable version:
    
    COMPONENT                 CURRENT   TARGET
    kube-apiserver            v1.24.8   v1.25.8
    kube-controller-manager   v1.24.8   v1.25.8
    kube-scheduler            v1.24.8   v1.25.8
    kube-proxy                v1.24.8   v1.25.8
    CoreDNS                   v1.8.6    v1.9.3
    etcd                      3.5.5-0   3.5.5-0
    
    You can now apply the upgrade by executing the following command:
    
            kubeadm upgrade apply v1.25.8
    
    Note: Before you can perform this upgrade, you have to update kubeadm to v1.25.8.
    
    _____________________________________________________________________
    
    
    The table below shows the current state of component configs as understood by this version of kubeadm.
    Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
    resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
    upgrade to is denoted in the "PREFERRED VERSION" column.
    
    API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
    kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
    kubelet.config.k8s.io     v1beta1           v1beta1             no
    _____________________________________________________________________

    1.7 根据提示,执行升级命令

    [root@k8s-master ~]# kubeadm upgrade apply v1.25.8
    [upgrade/config] Making sure the configuration is correct:
    [upgrade/config] Reading configuration from the cluster...
    [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
    [preflight] Running pre-flight checks.
    [upgrade] Running cluster health checks
    [upgrade/version] You have chosen to change the cluster version to "v1.25.8"
    [upgrade/versions] Cluster version: v1.24.8
    [upgrade/versions] kubeadm version: v1.25.4
    [upgrade/version] FATAL: the --version argument is invalid due to these errors:
    
            - Specified version to upgrade to "v1.25.8" is higher than the kubeadm version "v1.25.4". Upgrade kubeadm first using the tool you used to install kubeadm
    
    Can be bypassed if you pass the --force flag
    To see the stack trace of this error execute with --v=5 or higher

    提示升级的版本高于 kubeadm 的版本,需要使用安装 kubeadm 工具来升级 kubeadm

    [root@k8s-master ~]# yum install -y kubelet-1.25.8 kubeadm-1.25.8 kubectl-1.25.8
    已加载插件:fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirrors.aliyun.com
     * extras: mirrors.aliyun.com
     * updates: mirrors.aliyun.com
    正在解决依赖关系
    --> 正在检查事务
    ---> 软件包 kubeadm.x86_64.0.1.25.4-0 将被 升级
    ---> 软件包 kubeadm.x86_64.0.1.25.8-0 将被 更新
    ---> 软件包 kubectl.x86_64.0.1.24.8-0 将被 升级
    ---> 软件包 kubectl.x86_64.0.1.25.8-0 将被 更新
    ---> 软件包 kubelet.x86_64.0.1.24.8-0 将被 升级
    ---> 软件包 kubelet.x86_64.0.1.25.8-0 将被 更新
    --> 解决依赖关系完成
    
    依赖关系解决
    
    ==========================================================================================
     Package             架构               版本                 源                      大小
    ==========================================================================================
    正在更新:
     kubeadm             x86_64             1.25.8-0             kubernetes             9.8 M
     kubectl             x86_64             1.25.8-0             kubernetes              10 M
     kubelet             x86_64             1.25.8-0             kubernetes              21 M
    
    事务概要
    ==========================================================================================
    升级  3 软件包
    
    总下载量:41 M
    Downloading packages:
    Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
    (1/3): 6191f93924eb019f334fa96f179bc84cea9c21db1a74777f4fb8d43a8d1 | 9.8 MB  00:00:05
    (2/3): ec583fbf5524335d3f0083d82d5b924b93d72420c80d762547d273f179d |  10 MB  00:00:05
    (3/3): d9fa64f20de8fc2b81ae95713e40a06afba3af27ee5477a0a2682166b5b |  21 MB  00:00:09
    ------------------------------------------------------------------------------------------
    总计                                                      2.7 MB/s |  41 MB  00:00:15
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      正在更新    : kubectl-1.25.8-0.x86_64                                               1/6
      正在更新    : kubelet-1.25.8-0.x86_64                                               2/6
      正在更新    : kubeadm-1.25.8-0.x86_64                                               3/6
      清理        : kubeadm-1.25.4-0.x86_64                                               4/6
      清理        : kubectl-1.24.8-0.x86_64                                               5/6
      清理        : kubelet-1.24.8-0.x86_64                                               6/6
      验证中      : kubeadm-1.25.8-0.x86_64                                               1/6
      验证中      : kubelet-1.25.8-0.x86_64                                               2/6
      验证中      : kubectl-1.25.8-0.x86_64                                               3/6
      验证中      : kubectl-1.24.8-0.x86_64                                               4/6
      验证中      : kubeadm-1.25.4-0.x86_64                                               5/6
      验证中      : kubelet-1.24.8-0.x86_64                                               6/6
    
    更新完毕:
      kubeadm.x86_64 0:1.25.8-0    kubectl.x86_64 0:1.25.8-0    kubelet.x86_64 0:1.25.8-0
    
    完毕!

    然后再次执行升级命令: 

    [root@k8s-master ~]# kubeadm upgrade apply v1.25.8
    [upgrade/config] Making sure the configuration is correct:
    [upgrade/config] Reading configuration from the cluster...
    [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
    [preflight] Running pre-flight checks.
    [upgrade] Running cluster health checks
    [upgrade/version] You have chosen to change the cluster version to "v1.25.8"
    [upgrade/versions] Cluster version: v1.24.8
    [upgrade/versions] kubeadm version: v1.25.8
    [upgrade] Are you sure you want to proceed? [y/N]: y
    [upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
    [upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
    [upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
    [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.25.8" (timeout: 5m0s)...
    [upgrade/etcd] Upgrading to TLS for etcd
    [upgrade/staticpods] Preparing for "etcd" upgrade
    [upgrade/staticpods] Renewing etcd-server certificate
    [upgrade/staticpods] Renewing etcd-peer certificate
    [upgrade/staticpods] Renewing etcd-healthcheck-client certificate
    [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-04-07-22-04-09/etcd.yaml"
    [upgrade/staticpods] Waiting for the kubelet to restart the component
    [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
    [apiclient] Found 1 Pods for label selector component=etcd
    [upgrade/staticpods] Component "etcd" upgraded successfully!
    [upgrade/etcd] Waiting for etcd to become available
    [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests2071430521"
    [upgrade/staticpods] Preparing for "kube-apiserver" upgrade
    [upgrade/staticpods] Renewing apiserver certificate
    [upgrade/staticpods] Renewing apiserver-kubelet-client certificate
    [upgrade/staticpods] Renewing front-proxy-client certificate
    [upgrade/staticpods] Renewing apiserver-etcd-client certificate
    [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-04-07-22-04-09/kube-apiserver.yaml"
    [upgrade/staticpods] Waiting for the kubelet to restart the component
    [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
    [apiclient] Found 1 Pods for label selector component=kube-apiserver
    [upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
    [upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
    [upgrade/staticpods] Renewing controller-manager.conf certificate
    [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-04-07-22-04-09/kube-controller-manager.yaml"
    [upgrade/staticpods] Waiting for the kubelet to restart the component
    [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
    [apiclient] Found 1 Pods for label selector component=kube-controller-manager
    [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
    [upgrade/staticpods] Preparing for "kube-scheduler" upgrade
    [upgrade/staticpods] Renewing scheduler.conf certificate
    [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-04-07-22-04-09/kube-scheduler.yaml"
    [upgrade/staticpods] Waiting for the kubelet to restart the component
    [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
    [apiclient] Found 1 Pods for label selector component=kube-scheduler
    [upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
    [upgrade/postupgrade] Removing the old taint &Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,} from all control plane Nodes. After this step only the &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} taint will be present on control plane Nodes.
    [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
    [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
    [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
    [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
    [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
    [addons] Applied essential addon: CoreDNS
    [addons] Applied essential addon: kube-proxy
    
    [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.25.8". Enjoy!
    
    [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

    1.8 取消 k8s-master 节点不可调度

    [root@k8s-master ~]# kubectl uncordon k8s-master
    node/k8s-master uncordoned

    1.9 重启 kubelet

    [root@k8s-master ~]# systemctl daemon-reload
    [root@k8s-master ~]# systemctl restart kubelet
    [root@k8s-master ~]# kubectl get nodes
    NAME         STATUS   ROLES           AGE    VERSION
    k8s-master   Ready    control-plane   128d   v1.25.8
    k8s-node1    Ready    <none>          128d   v1.24.8
    k8s-node2    Ready    <none>          128d   v1.24.8

    二、升级 k8s-node

    2.1 驱逐 k8s-node1

    [root@k8s-master ~]# kubectl drain k8s-node1 --ignore-daemonsets
    node/k8s-node1 already cordoned

    2.2 查看节点状态

    [root@k8s-master ~]# kubectl get nodes
    NAME         STATUS                     ROLES           AGE    VERSION
    k8s-master   Ready                      control-plane   128d   v1.25.8
    k8s-node1    Ready,SchedulingDisabled   <none>          128d   v1.24.8
    k8s-node2    Ready                      <none>          128d   v1.24.8

    2.3 登录要升级的节点,执行升级

    [root@k8s-master ~]# ssh k8s-node1
    [root@k8s-node1 ~]# yum install -y kubelet-1.25.8 kubeadm-1.25.8 kubectl-1.25.8
    已加载插件:fastestmirror
    Determining fastest mirrors
     * base: mirrors.aliyun.com
     * extras: mirrors.aliyun.com
     * updates: mirrors.aliyun.com
    base                                                               | 3.6 kB  00:00:00
    docker-ce-stable                                                   | 3.5 kB  00:00:00
    extras                                                             | 2.9 kB  00:00:00
    kubernetes                                                         | 1.4 kB  00:00:00
    updates                                                            | 2.9 kB  00:00:00
    (1/3): docker-ce-stable/7/x86_64/primary_db                        | 102 kB  00:00:00
    (2/3): kubernetes/primary                                          | 126 kB  00:00:00
    (3/3): updates/7/x86_64/primary_db                                 |  20 MB  00:00:07
    kubernetes                                                                        941/941
    正在解决依赖关系
    --> 正在检查事务
    ---> 软件包 kubeadm.x86_64.0.1.24.8-0 将被 升级
    ---> 软件包 kubeadm.x86_64.0.1.25.8-0 将被 更新
    ---> 软件包 kubectl.x86_64.0.1.23.5-0 将被 升级
    ---> 软件包 kubectl.x86_64.0.1.25.8-0 将被 更新
    ---> 软件包 kubelet.x86_64.0.1.24.8-0 将被 升级
    ---> 软件包 kubelet.x86_64.0.1.25.8-0 将被 更新
    --> 解决依赖关系完成
    
    依赖关系解决
    
    ==========================================================================================
     Package             架构               版本                 源                      大小
    ==========================================================================================
    正在更新:
     kubeadm             x86_64             1.25.8-0             kubernetes             9.8 M
     kubectl             x86_64             1.25.8-0             kubernetes              10 M
     kubelet             x86_64             1.25.8-0             kubernetes              21 M
    
    事务概要
    ==========================================================================================
    升级  3 软件包
    
    总下载量:41 M
    Downloading packages:
    Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
    (1/3): 6191f93924eb019f334fa96f179bc84cea9c21db1a74777f4fb8d43a8d1 | 9.8 MB  00:00:03
    (2/3): ec583fbf5524335d3f0083d82d5b924b93d72420c80d762547d273f179d |  10 MB  00:00:04
    (3/3): d9fa64f20de8fc2b81ae95713e40a06afba3af27ee5477a0a2682166b5b |  21 MB  00:00:08
    ------------------------------------------------------------------------------------------
    总计                                                      3.6 MB/s |  41 MB  00:00:11
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      正在更新    : kubectl-1.25.8-0.x86_64                                               1/6
      正在更新    : kubelet-1.25.8-0.x86_64                                               2/6
      正在更新    : kubeadm-1.25.8-0.x86_64                                               3/6
      清理        : kubeadm-1.24.8-0.x86_64                                               4/6
      清理        : kubectl-1.23.5-0.x86_64                                               5/6
      清理        : kubelet-1.24.8-0.x86_64                                               6/6
      验证中      : kubeadm-1.25.8-0.x86_64                                               1/6
      验证中      : kubelet-1.25.8-0.x86_64                                               2/6
      验证中      : kubectl-1.25.8-0.x86_64                                               3/6
      验证中      : kubectl-1.23.5-0.x86_64                                               4/6
      验证中      : kubeadm-1.24.8-0.x86_64                                               5/6
      验证中      : kubelet-1.24.8-0.x86_64                                               6/6
    
    更新完毕:
      kubeadm.x86_64 0:1.25.8-0    kubectl.x86_64 0:1.25.8-0    kubelet.x86_64 0:1.25.8-0
    
    完毕!

    重启 kubelet

    [root@k8s-node1 ~]# systemctl daemon-reload
    [root@k8s-node1 ~]# systemctl restart kubelet

    2.4 取消节点 k8s-node1 不可调度

    [root@k8s-master ~]# kubectl uncordon k8s-node1
    node/k8s-node1 uncordoned
    [root@k8s-master ~]# kubectl get nodes
    NAME         STATUS   ROLES           AGE    VERSION
    k8s-master   Ready    control-plane   128d   v1.25.8
    k8s-node1    Ready    <none>          128d   v1.25.8
    k8s-node2    Ready    <none>          128d   v1.24.8

    2.5 其他节点同样处理

    [root@k8s-master ~]# kubectl uncordon k8s-node2
    node/k8s-node2 uncordoned
    [root@k8s-master ~]# kubectl get nodes
    NAME         STATUS   ROLES           AGE    VERSION
    k8s-master   Ready    control-plane   128d   v1.25.8
    k8s-node1    Ready    <none>          128d   v1.25.8
    k8s-node2    Ready    <none>          128d   v1.25.8
back_to_top