kubernetes学习笔记-离线升级kubernetes集群

一、kubernetes集群跨版本升级

kubernetes集群需要升级那些组件:

  • 升级管理节点,管理节点上的kube-apiserver,kuber-controller-manager,kube-scheduler,etcd等
  • worker工作节点,升级工作节点上的Container Runtime如docker,kubelet和kube-proxy
  • 其他管理节点,管理节点如果以高可用的方式部署,多个高可用节点需要一并升级

以离线的方式将1.14.1升级至1.1.5.1版本,升级前需要满足条件如下

  • 关闭swap空间
  • 备份数据,将etcd数据备份,以及一些重要目录如/etc/kubernetes,/var/lib/kubelet
  • 升级过程中pod需要重启,确保应用使用RollingUpdate滚动升级策略,避免业务有影响
  • 升级过程中pod需要重启,确保应用使用RollingUpdate滚动升级策略,避免业务有影响

注意,升级一般有两种情况,比如由v1.14.1升级到v1.15.1属于垮版本升级,而v1.15.1升级到v1.15.3属于小版本升级,这里采用离线升级大版本,在线升级小版本

1.1.升级前准备

关闭swap分区,其实阿里云ECS服务器默认关闭了swap,如果没有关闭请按照以下操作关闭swap分区

  • swapoff /mnt/swap关闭分区
  • 删除文件/etc/fstab中swap分区挂载的那一行配置
  • free-m确认swap关闭,均显示0表示关闭成功
  • 永久关闭echo "vm.swappiness=0" >> /etc/sysctl.conf && sysctl -p

查看当前集群版本
查看apiserver处于可用状态
kubectl version

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

查看节点版本
kubectl get nodes

NAME     STATUS   ROLES    AGE    VERSION
node-1   Ready    master   7d7h   v1.14.1
node-2   Ready    <none>   7d3h   v1.14.1
node-3   Ready    <none>   7d3h   v1.14.1

查看其他组件状态
kubectl get componentstatuses

NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   

kubectl get deployments --all-namespaces

NAMESPACE     NAME             READY   UP-TO-DATE   AVAILABLE   AGE
default       nginx-app-demo   4/4     4            4           36h
kube-system   coredns          2/2     2            2           7d7h

查看kubernetes最新可用版本
yum list --showduplicates kubeadm --disableexcludes=kubernetes

kubeadm.x86_64                                             1.13.11-0                                              kubernetes 
kubeadm.x86_64                                             1.13.12-0                                              kubernetes 
kubeadm.x86_64                                             1.14.0-0                                               kubernetes 
kubeadm.x86_64                                             1.14.1-0                                               kubernetes 
kubeadm.x86_64                                             1.14.2-0                                               kubernetes 
kubeadm.x86_64                                             1.14.3-0                                               kubernetes 
kubeadm.x86_64                                             1.14.4-0                                               kubernetes 
kubeadm.x86_64                                             1.14.5-0                                               kubernetes 
kubeadm.x86_64                                             1.14.6-0                                               kubernetes 
kubeadm.x86_64                                             1.14.7-0                                               kubernetes 
kubeadm.x86_64                                             1.14.8-0                                               kubernetes 
kubeadm.x86_64                                             1.14.9-0                                               kubernetes 
kubeadm.x86_64                                             1.14.10-0                                              kubernetes 
kubeadm.x86_64                                             1.15.0-0                                               kubernetes 
kubeadm.x86_64                                             1.15.1-0                                               kubernetes 
kubeadm.x86_64                                             1.15.2-0                                               kubernetes 
kubeadm.x86_64                                             1.15.3-0                                               kubernetes 
kubeadm.x86_64                                             1.15.4-0                                               kubernetes 
kubeadm.x86_64                                             1.15.5-0                                               kubernetes 
kubeadm.x86_64                                             1.15.6-0                                               kubernetes 
kubeadm.x86_64                                             1.15.7-0                                               kubernetes 
kubeadm.x86_64                                             1.15.8-0                                               kubernetes 
kubeadm.x86_64                                             1.15.9-0                                               kubernetes 
kubeadm.x86_64                                             1.15.10-0                                              kubernetes 
kubeadm.x86_64                                             1.16.0-0                                               kubernetes 
kubeadm.x86_64                                             1.16.1-0                                               kubernetes 
kubeadm.x86_64                                             1.16.2-0                                               kubernetes 
kubeadm.x86_64                                             1.16.3-0                                               kubernetes 
kubeadm.x86_64                                             1.16.4-0                                               kubernetes 
kubeadm.x86_64                                             1.16.5-0                                               kubernetes 
kubeadm.x86_64                                             1.16.6-0                                               kubernetes 
kubeadm.x86_64                                             1.16.7-0                                               kubernetes 
kubeadm.x86_64                                             1.17.0-0                                               kubernetes 
kubeadm.x86_64                                             1.17.1-0                                               kubernetes 
kubeadm.x86_64                                             1.17.2-0                                               kubernetes 
kubeadm.x86_64                                             1.17.3-0                                               kubernetes 

1.2.master节点升级

前面准备工作已经做完
获取kubernetes版本v1.15.1的镜像
v1.15.1下载地址
将获取的镜像在三台节点分别导入
先查看已有的镜像
docker images

k8s.gcr.io/kube-proxy                v1.14.1             20a2d7035165        11 months ago       82.1MB
k8s.gcr.io/kube-apiserver            v1.14.1             cfaa4ad74c37        11 months ago       210MB
k8s.gcr.io/kube-scheduler            v1.14.1             8931473d5bdb        11 months ago       81.6MB
k8s.gcr.io/kube-controller-manager   v1.14.1             efb3887b411d        11 months ago       158MB
quay.io/coreos/flannel               v0.11.0-amd64       ff281650a721        13 months ago       52.6MB
k8s.gcr.io/coredns                   1.3.1               eb516548c180        13 months ago       40.3MB
k8s.gcr.io/etcd                      3.3.10              2c4adeb21b4f        15 months ago       258MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        2 years ago         742kB

使用脚本导入新镜像
可以使用前面提供的地址下载,我这里是从个人阿里云镜像仓库里拉取的
cat image_pull.sh

#!/bin/bash

a=(kube-apiserver kube-scheduler kube-proxy kube-controller-manager)

for i in ${a[@]}
do
    docker pull registry.cn-shanghai.aliyuncs.com/linuxwt/$i:v1.15.1
    docker tag registry.cn-shanghai.aliyuncs.com/linuxwt/$i:v1.15.1 k8s.gcr.io/$i:v1.15.1
    docker rmi registry.cn-shanghai.aliyuncs.com/linuxwt/$i:v1.15.1
done
docker images

导入后

k8s.gcr.io/kube-proxy                v1.15.1             89a062da739d        7 months ago        82.4MB
k8s.gcr.io/kube-apiserver            v1.15.1             68c3eb07bfc3        7 months ago        207MB
k8s.gcr.io/kube-scheduler            v1.15.1             b0b3c4c404da        7 months ago        81.1MB
k8s.gcr.io/kube-controller-manager   v1.15.1             d75082f1d121        7 months ago        159MB
k8s.gcr.io/kube-proxy                v1.14.1             20a2d7035165        11 months ago       82.1MB
k8s.gcr.io/kube-apiserver            v1.14.1             cfaa4ad74c37        11 months ago       210MB
k8s.gcr.io/kube-scheduler            v1.14.1             8931473d5bdb        11 months ago       81.6MB
k8s.gcr.io/kube-controller-manager   v1.14.1             efb3887b411d        11 months ago       158MB
quay.io/coreos/flannel               v0.11.0-amd64       ff281650a721        13 months ago       52.6MB
k8s.gcr.io/coredns                   1.3.1               eb516548c180        13 months ago       40.3MB
k8s.gcr.io/etcd                      3.3.10              2c4adeb21b4f        15 months ago       258MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        2 years ago         742kB

更新kubeadm到1.15.1
yum -y install kubeadm-1.15.1-0 --disableexcludes=kuberne
kubeadm version查看kubeadm是否升级到1.15.1-0这个版本

通过kubeadm可以查看当前集群的升级计划,会显示当前小版本最新的版本以及社区最新的版
kubeadm upgrade plan

[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.14.1
[upgrade/versions] kubeadm version: v1.15.1
I0305 00:46:18.340774   32368 version.go:248] remote version is much newer: v1.17.3; falling back to: stable-1.15
[upgrade/versions] Latest stable version: v1.15.10
[upgrade/versions] Latest version in the v1.14 series: v1.14.10

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     3 x v1.14.1   v1.14.10

Upgrade to the latest version in the v1.14 series:

COMPONENT            CURRENT   AVAILABLE
API Server           v1.14.1   v1.14.10
Controller Manager   v1.14.1   v1.14.10
Scheduler            v1.14.1   v1.14.10
Kube Proxy           v1.14.1   v1.14.10
CoreDNS              1.3.1     1.3.1
Etcd                 3.3.10    3.3.10

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.14.10

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     3 x v1.14.1   v1.15.10

Upgrade to the latest stable version:

COMPONENT            CURRENT   AVAILABLE
API Server           v1.14.1   v1.15.10
Controller Manager   v1.14.1   v1.15.10
Scheduler            v1.14.1   v1.15.10
Kube Proxy           v1.14.1   v1.15.10
CoreDNS              1.3.1     1.3.1
Etcd                 3.3.10    3.3.10

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.15.10

Note: Before you can perform this upgrade, you have to update kubeadm to v1.15.10.

更新
kubeadm upgrade apply v1.15.1

[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/version] You have chosen to change the cluster version to "v1.15.1"
[upgrade/versions] Cluster version: v1.14.1
[upgrade/versions] kubeadm version: v1.15.1
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.15.1"...
Static pod: kube-apiserver-node-1 hash: 02c953be08e3d60c338766a3e9937cbd
Static pod: kube-controller-manager-node-1 hash: f4e6a574ceea76f0807a77e19a4d3b6c
Static pod: kube-scheduler-node-1 hash: f44110a0ca540009109bfc32a7eb0baa
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests453843779"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-03-05-00-48-18/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-node-1 hash: 02c953be08e3d60c338766a3e9937cbd
Static pod: kube-apiserver-node-1 hash: 9b4cf35107cc9054ee58a41394efe4ea
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-03-05-00-48-18/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-node-1 hash: f4e6a574ceea76f0807a77e19a4d3b6c
Static pod: kube-controller-manager-node-1 hash: 17b23c8c6fcf9b9f8a3061b3a2fbf633
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-03-05-00-48-18/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-node-1 hash: f44110a0ca540009109bfc32a7eb0baa
Static pod: kube-scheduler-node-1 hash: 18859150495c74ad1b9f283da804a3db
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.15.1". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

升级kubelet版本并重启kubelet服务,至此,master节点版本升级完毕
yum install -y kubelet-1.15.1-0 kubectl-1.15.1-0 --disableexcludes=kubernetes
systemctl daemon-reload
systemctl restart kubelet

1.3.升级worker

先升级node-2
升级kubeadm和kubelet软件包
yum -y install kubelet-1.15.1-0 --disableexcludes=kubernetes
yum install -y kubeadm-1.15.1-0 --disableexcludes=kubernetes
yum install -y kubectl-1.15.1-0 --disableexcludes=kubernetes

设置节点进入维护模式并驱逐worker节点上的应用,会将出了DaemonSets之外的其他应用迁移到其他节点上
kubectl drain node-2 --ignore-daemonsets

WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-amd64-sfglq, kube-system/kube-proxy-jhxm9
evicting pod "coredns-5c98db65d4-j5jh9"
evicting pod "nginx-app-demo-7bdfd97dcd-76l5r"
evicting pod "nginx-app-demo-7bdfd97dcd-qn9vx"
pod/nginx-app-demo-7bdfd97dcd-qn9vx evicted
pod/nginx-app-demo-7bdfd97dcd-76l5r evicted
pod/coredns-5c98db65d4-j5jh9 evicted
node/node-2 evicted

kubectl get nodes

NAME     STATUS                     ROLES    AGE    VERSION
node-1   Ready                      master   7d8h   v1.15.1
node-2   Ready,SchedulingDisabled   <none>   7d4h   v1.14.1
node-3   Ready                      <none>   7d4h   v1.14.1   

kubectl get pods --all-namespqces -o wide
可以看到应用pod均已迁移到node-3上

NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE     IP             NODE     NOMINATED NODE   READINESS GATES
default       nginx-app-demo-7bdfd97dcd-5swrn   1/1     Running   0          4m29s   10.244.2.42    node-3   <none>           <none>
default       nginx-app-demo-7bdfd97dcd-9dd2l   1/1     Running   0          4m29s   10.244.2.41    node-3   <none>           <none>
default       nginx-app-demo-7bdfd97dcd-mksq5   1/1     Running   1          37h     10.244.2.39    node-3   <none>           <none>
default       nginx-app-demo-7bdfd97dcd-trc48   1/1     Running   1          37h     10.244.2.38    node-3   <none>           <none>
kube-system   coredns-5c98db65d4-4gsxm          1/1     Running   0          4m29s   10.244.0.6     node-1   <none>           <none>
kube-system   coredns-5c98db65d4-5rcv2          1/1     Running   2          18m     10.244.2.40    node-3   <none>           <none>
kube-system   etcd-node-1                       1/1     Running   0          13m     172.19.159.7   node-1   <none>           <none>
kube-system   kube-apiserver-node-1             1/1     Running   0          13m     172.19.159.7   node-1   <none>           <none>
kube-system   kube-controller-manager-node-1    1/1     Running   0          13m     172.19.159.7   node-1   <none>           <none>
kube-system   kube-flannel-ds-amd64-5qxcf       1/1     Running   1          7d4h    172.19.159.9   node-3   <none>           <none>
kube-system   kube-flannel-ds-amd64-sfglq       1/1     Running   1          7d4h    172.19.159.8   node-2   <none>           <none>
kube-system   kube-flannel-ds-amd64-vjkx8       1/1     Running   2          7d4h    172.19.159.7   node-1   <none>           <none>
kube-system   kube-proxy-8chvb                  1/1     Running   0          18m     172.19.159.9   node-3   <none>           <none>
kube-system   kube-proxy-jhxm9                  1/1     Running   0          18m     172.19.159.8   node-2   <none>           <none>
kube-system   kube-proxy-lsw9f                  1/1     Running   1          18m     172.19.159.7   node-1   <none>           <none>
kube-system   kube-scheduler-node-1             1/1     Running   0          13m     172.19.159.7   node-1   <none>           <none>

升级node-2
[root@node-2 ~]# kubeadm upgrade node

[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Skipping phase. Not a control plane node[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

systemctl daemon-reload
systemctl restart kubelet

取消节点调度标志,确保worker节点可正常调度
kubectl uncordon node-2
kubectl get pods

NAME     STATUS   ROLES    AGE    VERSION
node-1   Ready    master   7d8h   v1.15.1
node-2   Ready    <none>   7d5h   v1.15.1
node-3   Ready    <none>   7d5h   v1.14.1

按照上面步骤升级node-3

最后升级成功需要达到以下两点
kubectl get nodes

NAME     STATUS   ROLES    AGE    VERSION
node-1   Ready    master   7d8h   v1.15.1
node-2   Ready    <none>   7d5h   v1.15.1
node-3   Ready    <none>   7d5h   v1.15.1

kubectl get daemonsets --all-spaces

NAMESPACE     NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE
kube-system   kube-flannel-ds-amd64   3         3         3       3            3           beta.kubernetes.io/arch=amd64   7d4h
kube-system   kube-proxy              3         3         3       3            3           beta.kubernetes.io/os=linux     7d8h

二、小版本升级

查看升级计划

kubeadm upgrade plan
可以看到小版本最多只能升级到v1.15.10进行小版本的升级
可以根据提示的命令进行升级,这里需要即时从网上拉取对应的kubernetes镜像,需要科学上网

最后的建议:还是以离线升级的方式升级比较稳妥