kubernetes

kubernetes学习笔记-离线部署1.14.1集群

linuxwt Estimate 10 min read

   

一、安装概述

安装主要有三种方式：

  • MiniKube工具安装，主要用于学习
  • 二进制安装，需要配置参数
  • Kubeadm安装，以镜像的方式部署

二、部署k8s集群

02k8s1

2.1.环境准备

【软件版本】

软件 版本
OS CentOS Linux release 7.5.1804 (Core)
Dokcer docker-ce-18.03.1.ce-1.el7.centos.x86_64
Kubernetes 1.14.1
etcd 3.3.10
fiannel v0.11.0
Kubeadm kubeadm-1.14.1-0.x86_64
【环境说明】 三台机器均为阿里云ECS服务器，配置为2cpu4g内存
主机名 角色 ip 软件
node-1 master 172.19.159.7 docker,kubelet,etcd,kube-apiserver,kube-controller-manager,kube-scheduler
node-2 worker 172.19.159.8 docker,kubelet,kube-proxy,flannel
node-3 worker 172.19.159.9 docker,kubelet,kube-proxy,flannel
【环境准备】

以下操作需要在三台服务器上进行
1.修改三台服务器hostname
hostnamectl set-hostname node-1
hostnamectl set-hostname node-2
hostnamectl set-hostname node-3
2.设置hosts文件
vi /etc/hosts

172.19.159.7    node-1
172.19.159.8    node-2
172.19.159.9    node-3

3.设置SSH无密码登录，并通过ssh-copy-id将公钥拷贝到对端
比如在node-1上进行如下操作
ssh-keygen
ssh-copy-id -i /root/.ssh/id_rsa.pub node-2
ssh-copy-id -i /root/.ssh/id_rsa.pub node-3
4.关闭SELINUX
sed -i 's/enforcing/disabled/g' /etc/selinux/config
systemctl stop firewalld
systemctl disable firewalld

2.2.安装docker环境

三台服务器均需要安装
安装脚本
cat docker_install.sh

#!/bin/bash

installdocker()
{

yum install -y yum-utils device-mapper-persistent-data lvm2

yum-config-manager  --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

yum makecache fast

yum -y install docker-ce

}

docker version

if [ $? -eq 127 ];then

        echo "we can install docker-ce"

        sleep 5

        installdocker

        docker version

        if [ $? -lt 127 ];then

                echo "the installation of docker-ce is ok."

		rpm -qa | grep docker | xargs rpm -e --nodeps 

                yum -y install docker-ce-18.03*

        else

                echo "the installation of docker-ce failed ,please reinstall"

                exit -1

        fi

else

        echo "docker have installed，pleae uninstall old version"

        sleep 5

        rpm -qa | grep docker | xargs rpm -e --nodeps

        docker version

        if [ $? -eq 127 ];then

                echo "old docker have been uninstalled and you can install docker-ce"

                sleep 5

		installdocker

		docker version

		if [ $? -lt 127 ];then

			echo "the installation of docker-ce is ok."

			rpm -qa | grep docker | xargs rpm -e --nodeps

		        yum -y install docker-ce-18.03*	

		else

			echo "the installation of docker-ce failed anad please reinstall."

			exit -1

		fi

	else

		echo "the old docker uninstalled conpletely and please uninstall again."

		exit -1

	fi

fi

systemctl start docker && systemctl enable docker && systemctl daemon-reload 

docker_version=$(docker version | grep "Version" | awk '{print $2}' | head -n 2 | sed -n '2p')

if [ $? -eq 0 ];then

	echo "docker start successfully and the version is ${docker_version}"

fi

# 配置docker加速拉取

echo {\"registry-mirrors\":[\"https://nr630v1c.mirror.aliyuncs.com\"]} > /etc/docker/daemon.json

chmod +x docker_install.sh
bash docker_install.sh

设置cggroup driver类型为systemd
cat /etc/docker/daemon.json

{
    "registry-mirrors":["https://nr630v1c.mirror.aliyuncs.com"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
    "max-size": "100m"
    },
    "storage-driver": "overlay2",
    "storage-opts": [
    "overlay2.override_kernel_check=true"
    ]
}

2.3.安装kubeadm组件

三台服务器均需要安装
1.设置kubenertes源
cat /etc/yum.repos.d/kubernetes.repo

name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

2.安装kubeadm，kubelet，kubectl
yum install kubeadm-1.14.1-0 kubectl-1.14.1-0 kubelet-1.14.1-0 --disableexcludes=kubernetes -y
3.配置网桥
cat /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

sysctl --system加载配置
4.将kubelet加入系统服务
systemctl enable kubelet
注意这里不要使用sysytemctl start kubelet，后面进行集群初始化的时候会自动启动该服务
一定要注意kubelet所使用的cgroup driver要与docker一致，否则kubelet无法启动，前面我们设置了docker的cgroup driver为systemd，下面设置kubelet
cat /etc/default/kubelet

KUBELET_KUBEADM_EXTRA_ARGS=--cgroup-driver=systemd

2.4.导入kubernetes镜像

首先下载离线镜像包
kubernetes-v1.14.1离线镜像包
导入镜像
为了方便，三台服务器都导入全部kubernetes镜像
for i in $(ls);do docker load -i $i;done

k8s.gcr.io/kube-proxy                                      v1.14.1             20a2d7035165        10 months ago       82.1MB
k8s.gcr.io/kube-apiserver                                  v1.14.1             cfaa4ad74c37        10 months ago       210MB
registry.cn-shanghai.aliyuncs.com/linuxwt/kube-apiserver   v1.14.1             cfaa4ad74c37        10 months ago       210MB
k8s.gcr.io/kube-controller-manager                         v1.14.1             efb3887b411d        10 months ago       158MB
k8s.gcr.io/kube-scheduler                                  v1.14.1             8931473d5bdb        10 months ago       81.6MB
quay.io/coreos/flannel                                     v0.11.0-amd64       ff281650a721        13 months ago       52.6MB
k8s.gcr.io/coredns                                         1.3.1               eb516548c180        13 months ago       40.3MB
k8s.gcr.io/etcd                                            3.3.10              2c4adeb21b4f        15 months ago       258MB
k8s.gcr.io/pause                                           3.1                 da86e6ba6ca1        2 years ago         742kB

2.5.集群初始化

初始化
node-1上
执行下面的命令

[root@node-1 ~]# kubeadm init --apiserver-advertise-address 172.19.159.7 --apiserver-bind-port 6443 --kubernetes-version 1.14.1 --pod-network-cidr 10.244.0.0/16
[init] Using Kubernetes version: v1.14.1
[preflight] Running pre-flight checks
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.03.1-ce. Latest validated version: 18.09
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [node-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.19.159.7]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [node-1 localhost] and IPs [172.19.159.7 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [node-1 localhost] and IPs [172.19.159.7 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 18.002772 seconds
[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --experimental-upload-certs
[mark-control-plane] Marking the node node-1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node node-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: fl5itt.m5aidmf451jbgeq6
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.19.159.7:6443 --token fl5itt.m5aidmf451jbgeq6 \
    --discovery-token-ca-cert-hash sha256:872d799f5a16950e9c9582aa4ed638537ae069a207563ff4d6e50af1c520ef26

上面的命令显示了kubeadm安装过程中的一些重要步骤：下载镜像，生成证书，生成配置文件，配置RBAC授权认证，配置环境变量，安装网络插件指引，添加node指引配置文件
生成kube环境配置文件
node-1上
mkdir /root/.kube
cp -i /etc/kubernetes/admin.conf /root/.kube/config
kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-1 NotReady master 159m v1.14.1
添加node节点
node-2上执行命令

kubeadm join 172.19.159.7:6443 --token fl5itt.m5aidmf451jbgeq6 \
    --discovery-token-ca-cert-hash sha256:872d799f5a16950e9c9582aa4ed638537ae069a207563ff4d6e50af1c520ef26

node-3上执行上面的命令
node-1上查看

kubectl get nodes   
NAME     STATUS     ROLES    AGE     VERSION
node-1   NotReady   master   3h33m   v1.14.1
node-2   NotReady   <none>   5m3s    v1.14.1
node-3   NotReady   <none>   3m18s   v1.14.1

安装网络plugin
kubernetes支持多种类型网络插件，要求网络支持CNI插件即可，CNI是Container Network Interface，要求kubernetes的中pod网络访问方式:

  • node与node之间互通
  • pod与pod之间互通
  • node与pod之间互通

kubernetes支持多种开源的网络CNI插件，常见的有flannel、calico、canal、weave等，flannel是一种overlay的网络模型，通过vxlan隧道方式构建tunnel网络，实现k8s中网络的互联

node-1上

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml   

podsecuritypolicy.extensions/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

通过上述输出可知道，部署flannel 需要RBAC授权，配置configmap和daemonset，其中Daemonset能够适配各种类型的CPU架构，默认安装了多个，一般是adm64即可，可以将上述的url下载编辑，保留kube-flannel-ds-amd64这个daemonset即可，或者将其删除

kubectl get daemonsets -n kube-system

删除不需要的damonsets

kubectl delete daemonsets kube-flannel-ds-arm kube-flannel-ds-arm64 kube-flannel-ds-ppc64le kube-flannel-ds-s390x -n kube-system   
    
kubectl get nodes   
 
NAME     STATUS   ROLES    AGE     VERSION
node-1   Ready    master   4h28m   v1.14.1
node-2   Ready    <none>   60m     v1.14.1
node-3   Ready    <none>   58m     v1.14.1

2.6.验证kubernetes组件

验证node状态
kubectl get nodes获取各个节点的状态、角色、运行时长、版本等信息
查看kubernetes服务组件状态
kubectl get componentstatuses

NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-0               Healthy   {"health":"true"}

查看pod情况
kubectl get pods -n kube-system

NAME                             READY   STATUS    RESTARTS   AGE
coredns-fb8b8dccf-mhkkk          1/1     Running   0          7h17m
coredns-fb8b8dccf-vz65l          1/1     Running   0          7h17m
etcd-node-1                      1/1     Running   0          7h16m
kube-apiserver-node-1            1/1     Running   0          7h16m
kube-controller-manager-node-1   1/1     Running   0          7h16m
kube-flannel-ds-amd64-5qxcf      1/1     Running   0          174m
kube-flannel-ds-amd64-sfglq      1/1     Running   0          174m
kube-flannel-ds-amd64-vjkx8      1/1     Running   0          174m
kube-proxy-8gjl7                 1/1     Running   0          7h17m
kube-proxy-pt922                 1/1     Running   0          3h49m
kube-proxy-zldlm                 1/1     Running   0          3h47m
kube-scheduler-node-1            1/1     Running   0          7h16m

2.7.配置kubectl补全命令

安装bash-completion
yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
配置补全
kubectl completion bash >/etc/kubernetes/kubectl.sh
echo "source /etc/kubernetes/kubectl.sh" >>/root/.bashrc
cat /root/.bashrc

# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
source /etc/kubernetes/kubectl.sh

source /etc/kubernetes/kubectl.sh
使用命令进行补全校验
kubectl get co TAB

[root@node-1 ~]# kubectl get co
componentstatuses         configmaps                controllerrevisions.apps

其实kubernetes除了补全方式还支持简写
比如
查看节点kubectl get no
查看组件状态kubectl get cs

2.8.补充

kubernetes集群搭建好后发现不知道该怎么去停止，只能用下面的本办法了
先停掉每一个节点的kubelet,否则停掉的容器会重启
systemctl stop kubelet
docker stop $(docker ps -a -q)

kubernetes
赏 支付宝扫码打赏 微信打赏

若你觉得我的文章对你有帮助,欢迎点击上方按钮对我打赏

扫描二维码,分享此文章

linuxwt's Picture
linuxwt

我叫王腾，来自武汉，2016年毕业后在上海做了一年helpdesk，自学了linux后回武汉从事系统运维的工作，从2017年开始写博客记录自己的学习工作，现在正在进行数据迁移到此博客，目前就职于中国移动设计院有限公司，个人的座右铭是：逃脱舒适区才能在闲暇的时候惬意的玩耍。

武汉光谷 https://linuxwt.com

Subscribe to 今晚打老虎

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!

Comments