kubernetes故障汇总

故障一
kubectl get pods

NAME     STATUS     ROLES    AGE   VERSION
node-1   Ready      master   32d   v1.14.1
node-2   Ready      <none>   32d   v1.14.1
node-3   NotReady   <none>   32d   v1.14.1

故障节点docker ps发现所有容器无法随集群正常启动,kubelet也无法启动
查看日志
journactl -f -u kubelet

Mar 30 09:42:33 node-3 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Mar 30 09:42:33 node-3 systemd[1]: Unit kubelet.service entered failed state.
Mar 30 09:42:33 node-3 systemd[1]: kubelet.service failed.
Mar 30 09:42:44 node-3 systemd[1]: kubelet.service holdoff time over, scheduling restart.
Mar 30 09:42:44 node-3 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Mar 30 09:42:44 node-3 systemd[1]: Starting kubelet: The Kubernetes Node Agent...
Mar 30 09:42:44 node-3 kubelet[21157]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Mar 30 09:42:44 node-3 kubelet[21157]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Mar 30 09:42:44 node-3 kubelet[21157]: I0330 09:42:44.152946 21157 server.go:417] Version: v1.14.1
Mar 30 09:42:44 node-3 kubelet[21157]: I0330 09:42:44.153184 21157 plugins.go:103] No cloud provider specified.
Mar 30 09:42:44 node-3 kubelet[21157]: I0330 09:42:44.153207 21157 server.go:754] Client rotation is on, will bootstrap in background
Mar 30 09:42:44 node-3 kubelet[21157]: I0330 09:42:44.165326 21157 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Mar 30 09:42:44 node-3 kubelet[21157]: I0330 09:42:44.206290 21157 server.go:625] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
Mar 30 09:42:44 node-3 kubelet[21157]: F0330 09:42:44.206680 21157 server.go:265] failed to run Kubelet: Running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename Type Size Used Priority /data/swap file 8330984 14848 -1]
Mar 30 09:42:44 node-3 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Mar 30 09:42:44 node-3 systemd[1]: Unit kubelet.service entered failed state.
Mar 30 09:42:44 node-3 systemd[1]: kubelet.service failed.

可以看到这句错误

failed to run Kubelet: Running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps containe

突然记起前几天因为做测试内存不够开启了swap分区,而kubelet是不支持swap功能的,导致kubelet无法启动
解决方式:
swapoff掉swap分区
systemctl restart kubelet
systemctl restart docker