一、kubernetes集群跨版本升级
kubernetes集群需要升级那些组件：
- 升级管理节点，管理节点上的kube-apiserver，kuber-controller-manager，kube-scheduler，etcd等
- worker工作节点，升级工作节点上的Container Runtime如docker，kubelet和kube-proxy
- 其他管理节点，管理节点如果以高可用的方式部署，多个高可用节点需要一并升级
以离线的方式将1.14.1升级至1.1.5.1版本，升级前需要满足条件如下
- 关闭swap空间
- 备份数据，将etcd数据备份，以及一些重要目录如/etc/kubernetes,/var/lib/kubelet
- 升级过程中pod需要重启，确保应用使用RollingUpdate滚动升级策略，避免业务有影响
- 升级过程中pod需要重启，确保应用使用RollingUpdate滚动升级策略，避免业务有影响
注意，升级一般有两种情况，比如由v1.14.1升级到v1.15.1属于垮版本升级，而v1.15.1升级到v1.15.3属于小版本升级，这里采用离线升级大版本，在线升级小版本
1.1.升级前准备
关闭swap分区，其实阿里云ECS服务器默认关闭了swap，如果没有关闭请按照以下操作关闭swap分区
- swapoff /mnt/swap关闭分区
- 删除文件/etc/fstab中swap分区挂载的那一行配置
- free-m确认swap关闭，均显示0表示关闭成功
- 永久关闭echo "vm.swappiness=0" >> /etc/sysctl.conf && sysctl -p
查看当前集群版本
查看apiserver处于可用状态
kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
查看节点版本
kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-1 Ready master 7d7h v1.14.1
node-2 Ready <none> 7d3h v1.14.1
node-3 Ready <none> 7d3h v1.14.1
查看其他组件状态
kubectl get componentstatuses
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
kubectl get deployments --all-namespaces
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default nginx-app-demo 4/4 4 4 36h
kube-system coredns 2/2 2 2 7d7h
查看kubernetes最新可用版本
yum list --showduplicates kubeadm --disableexcludes=kubernetes
kubeadm.x86_64 1.13.11-0 kubernetes
kubeadm.x86_64 1.13.12-0 kubernetes
kubeadm.x86_64 1.14.0-0 kubernetes
kubeadm.x86_64 1.14.1-0 kubernetes
kubeadm.x86_64 1.14.2-0 kubernetes
kubeadm.x86_64 1.14.3-0 kubernetes
kubeadm.x86_64 1.14.4-0 kubernetes
kubeadm.x86_64 1.14.5-0 kubernetes
kubeadm.x86_64 1.14.6-0 kubernetes
kubeadm.x86_64 1.14.7-0 kubernetes
kubeadm.x86_64 1.14.8-0 kubernetes
kubeadm.x86_64 1.14.9-0 kubernetes
kubeadm.x86_64 1.14.10-0 kubernetes
kubeadm.x86_64 1.15.0-0 kubernetes
kubeadm.x86_64 1.15.1-0 kubernetes
kubeadm.x86_64 1.15.2-0 kubernetes
kubeadm.x86_64 1.15.3-0 kubernetes
kubeadm.x86_64 1.15.4-0 kubernetes
kubeadm.x86_64 1.15.5-0 kubernetes
kubeadm.x86_64 1.15.6-0 kubernetes
kubeadm.x86_64 1.15.7-0 kubernetes
kubeadm.x86_64 1.15.8-0 kubernetes
kubeadm.x86_64 1.15.9-0 kubernetes
kubeadm.x86_64 1.15.10-0 kubernetes
kubeadm.x86_64 1.16.0-0 kubernetes
kubeadm.x86_64 1.16.1-0 kubernetes
kubeadm.x86_64 1.16.2-0 kubernetes
kubeadm.x86_64 1.16.3-0 kubernetes
kubeadm.x86_64 1.16.4-0 kubernetes
kubeadm.x86_64 1.16.5-0 kubernetes
kubeadm.x86_64 1.16.6-0 kubernetes
kubeadm.x86_64 1.16.7-0 kubernetes
kubeadm.x86_64 1.17.0-0 kubernetes
kubeadm.x86_64 1.17.1-0 kubernetes
kubeadm.x86_64 1.17.2-0 kubernetes
kubeadm.x86_64 1.17.3-0 kubernetes
1.2.master节点升级
前面准备工作已经做完
获取kubernetes版本v1.15.1的镜像
v1.15.1下载地址
将获取的镜像在三台节点分别导入
先查看已有的镜像
docker images
k8s.gcr.io/kube-proxy v1.14.1 20a2d7035165 11 months ago 82.1MB
k8s.gcr.io/kube-apiserver v1.14.1 cfaa4ad74c37 11 months ago 210MB
k8s.gcr.io/kube-scheduler v1.14.1 8931473d5bdb 11 months ago 81.6MB
k8s.gcr.io/kube-controller-manager v1.14.1 efb3887b411d 11 months ago 158MB
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 13 months ago 52.6MB
k8s.gcr.io/coredns 1.3.1 eb516548c180 13 months ago 40.3MB
k8s.gcr.io/etcd 3.3.10 2c4adeb21b4f 15 months ago 258MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB
使用脚本导入新镜像
可以使用前面提供的地址下载，我这里是从个人阿里云镜像仓库里拉取的
cat image_pull.sh
#!/bin/bash
a=(kube-apiserver kube-scheduler kube-proxy kube-controller-manager)
for i in ${a[@]}
do
docker pull registry.cn-shanghai.aliyuncs.com/linuxwt/$i:v1.15.1
docker tag registry.cn-shanghai.aliyuncs.com/linuxwt/$i:v1.15.1 k8s.gcr.io/$i:v1.15.1
docker rmi registry.cn-shanghai.aliyuncs.com/linuxwt/$i:v1.15.1
done
docker images
导入后
k8s.gcr.io/kube-proxy v1.15.1 89a062da739d 7 months ago 82.4MB
k8s.gcr.io/kube-apiserver v1.15.1 68c3eb07bfc3 7 months ago 207MB
k8s.gcr.io/kube-scheduler v1.15.1 b0b3c4c404da 7 months ago 81.1MB
k8s.gcr.io/kube-controller-manager v1.15.1 d75082f1d121 7 months ago 159MB
k8s.gcr.io/kube-proxy v1.14.1 20a2d7035165 11 months ago 82.1MB
k8s.gcr.io/kube-apiserver v1.14.1 cfaa4ad74c37 11 months ago 210MB
k8s.gcr.io/kube-scheduler v1.14.1 8931473d5bdb 11 months ago 81.6MB
k8s.gcr.io/kube-controller-manager v1.14.1 efb3887b411d 11 months ago 158MB
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 13 months ago 52.6MB
k8s.gcr.io/coredns 1.3.1 eb516548c180 13 months ago 40.3MB
k8s.gcr.io/etcd 3.3.10 2c4adeb21b4f 15 months ago 258MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB
更新kubeadm到1.15.1
yum -y install kubeadm-1.15.1-0 --disableexcludes=kuberne
kubeadm version查看kubeadm是否升级到1.15.1-0这个版本
通过kubeadm可以查看当前集群的升级计划，会显示当前小版本最新的版本以及社区最新的版
kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.14.1
[upgrade/versions] kubeadm version: v1.15.1
I0305 00:46:18.340774 32368 version.go:248] remote version is much newer: v1.17.3; falling back to: stable-1.15
[upgrade/versions] Latest stable version: v1.15.10
[upgrade/versions] Latest version in the v1.14 series: v1.14.10
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 3 x v1.14.1 v1.14.10
Upgrade to the latest version in the v1.14 series:
COMPONENT CURRENT AVAILABLE
API Server v1.14.1 v1.14.10
Controller Manager v1.14.1 v1.14.10
Scheduler v1.14.1 v1.14.10
Kube Proxy v1.14.1 v1.14.10
CoreDNS 1.3.1 1.3.1
Etcd 3.3.10 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.14.10
_____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 3 x v1.14.1 v1.15.10
Upgrade to the latest stable version:
COMPONENT CURRENT AVAILABLE
API Server v1.14.1 v1.15.10
Controller Manager v1.14.1 v1.15.10
Scheduler v1.14.1 v1.15.10
Kube Proxy v1.14.1 v1.15.10
CoreDNS 1.3.1 1.3.1
Etcd 3.3.10 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.15.10
Note: Before you can perform this upgrade, you have to update kubeadm to v1.15.10.
更新
kubeadm upgrade apply v1.15.1
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/version] You have chosen to change the cluster version to "v1.15.1"
[upgrade/versions] Cluster version: v1.14.1
[upgrade/versions] kubeadm version: v1.15.1
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.15.1"...
Static pod: kube-apiserver-node-1 hash: 02c953be08e3d60c338766a3e9937cbd
Static pod: kube-controller-manager-node-1 hash: f4e6a574ceea76f0807a77e19a4d3b6c
Static pod: kube-scheduler-node-1 hash: f44110a0ca540009109bfc32a7eb0baa
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests453843779"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-03-05-00-48-18/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-node-1 hash: 02c953be08e3d60c338766a3e9937cbd
Static pod: kube-apiserver-node-1 hash: 9b4cf35107cc9054ee58a41394efe4ea
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-03-05-00-48-18/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-node-1 hash: f4e6a574ceea76f0807a77e19a4d3b6c
Static pod: kube-controller-manager-node-1 hash: 17b23c8c6fcf9b9f8a3061b3a2fbf633
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2020-03-05-00-48-18/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-node-1 hash: f44110a0ca540009109bfc32a7eb0baa
Static pod: kube-scheduler-node-1 hash: 18859150495c74ad1b9f283da804a3db
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.15.1". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
升级kubelet版本并重启kubelet服务，至此，master节点版本升级完毕
yum install -y kubelet-1.15.1-0 kubectl-1.15.1-0 --disableexcludes=kubernetes
systemctl daemon-reload
systemctl restart kubelet
1.3.升级worker
先升级node-2
升级kubeadm和kubelet软件包
yum -y install kubelet-1.15.1-0 --disableexcludes=kubernetes
yum install -y kubeadm-1.15.1-0 --disableexcludes=kubernetes
yum install -y kubectl-1.15.1-0 --disableexcludes=kubernetes
设置节点进入维护模式并驱逐worker节点上的应用，会将出了DaemonSets之外的其他应用迁移到其他节点上
kubectl drain node-2 --ignore-daemonsets
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-amd64-sfglq, kube-system/kube-proxy-jhxm9
evicting pod "coredns-5c98db65d4-j5jh9"
evicting pod "nginx-app-demo-7bdfd97dcd-76l5r"
evicting pod "nginx-app-demo-7bdfd97dcd-qn9vx"
pod/nginx-app-demo-7bdfd97dcd-qn9vx evicted
pod/nginx-app-demo-7bdfd97dcd-76l5r evicted
pod/coredns-5c98db65d4-j5jh9 evicted
node/node-2 evicted
kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-1 Ready master 7d8h v1.15.1
node-2 Ready,SchedulingDisabled <none> 7d4h v1.14.1
node-3 Ready <none> 7d4h v1.14.1
kubectl get pods --all-namespqces -o wide
可以看到应用pod均已迁移到node-3上
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default nginx-app-demo-7bdfd97dcd-5swrn 1/1 Running 0 4m29s 10.244.2.42 node-3 <none> <none>
default nginx-app-demo-7bdfd97dcd-9dd2l 1/1 Running 0 4m29s 10.244.2.41 node-3 <none> <none>
default nginx-app-demo-7bdfd97dcd-mksq5 1/1 Running 1 37h 10.244.2.39 node-3 <none> <none>
default nginx-app-demo-7bdfd97dcd-trc48 1/1 Running 1 37h 10.244.2.38 node-3 <none> <none>
kube-system coredns-5c98db65d4-4gsxm 1/1 Running 0 4m29s 10.244.0.6 node-1 <none> <none>
kube-system coredns-5c98db65d4-5rcv2 1/1 Running 2 18m 10.244.2.40 node-3 <none> <none>
kube-system etcd-node-1 1/1 Running 0 13m 172.19.159.7 node-1 <none> <none>
kube-system kube-apiserver-node-1 1/1 Running 0 13m 172.19.159.7 node-1 <none> <none>
kube-system kube-controller-manager-node-1 1/1 Running 0 13m 172.19.159.7 node-1 <none> <none>
kube-system kube-flannel-ds-amd64-5qxcf 1/1 Running 1 7d4h 172.19.159.9 node-3 <none> <none>
kube-system kube-flannel-ds-amd64-sfglq 1/1 Running 1 7d4h 172.19.159.8 node-2 <none> <none>
kube-system kube-flannel-ds-amd64-vjkx8 1/1 Running 2 7d4h 172.19.159.7 node-1 <none> <none>
kube-system kube-proxy-8chvb 1/1 Running 0 18m 172.19.159.9 node-3 <none> <none>
kube-system kube-proxy-jhxm9 1/1 Running 0 18m 172.19.159.8 node-2 <none> <none>
kube-system kube-proxy-lsw9f 1/1 Running 1 18m 172.19.159.7 node-1 <none> <none>
kube-system kube-scheduler-node-1 1/1 Running 0 13m 172.19.159.7 node-1 <none> <none>
升级node-2
[root@node-2 ~]# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Skipping phase. Not a control plane node[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
systemctl daemon-reload
systemctl restart kubelet
取消节点调度标志，确保worker节点可正常调度
kubectl uncordon node-2
kubectl get pods
NAME STATUS ROLES AGE VERSION
node-1 Ready master 7d8h v1.15.1
node-2 Ready <none> 7d5h v1.15.1
node-3 Ready <none> 7d5h v1.14.1
按照上面步骤升级node-3
最后升级成功需要达到以下两点
kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-1 Ready master 7d8h v1.15.1
node-2 Ready <none> 7d5h v1.15.1
node-3 Ready <none> 7d5h v1.15.1
kubectl get daemonsets --all-spaces
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system kube-flannel-ds-amd64 3 3 3 3 3 beta.kubernetes.io/arch=amd64 7d4h
kube-system kube-proxy 3 3 3 3 3 beta.kubernetes.io/os=linux 7d8h
二、小版本升级
查看升级计划
kubeadm upgrade plan
可以看到小版本最多只能升级到v1.15.10进行小版本的升级
可以根据提示的命令进行升级，这里需要即时从网上拉取对应的kubernetes镜像，需要科学上网
最后的建议：还是以离线升级的方式升级比较稳妥
