一、概念
kubernetes启动的容器中的数据是临时的，重启容器后数据丢失，但是kubernetes提供了volume存储的抽象，volume后端能够支持多种不同的plugin驱动，通过spec.volumes定义一个存储，然后在容器中spec.containers.volumeMounts中调用，最终在容器中以目录的形式呈现，kubernetes内置支持多种驱动类型，主要分为四种：公有云和私有云驱动接口、开源存储驱动如ceph rbd、本地临时存储hostPath、kubernetes对象API驱动接口如configmap
这里主要介绍一下本地临时存储：
- hostPath 宿主机文件
- emptyDir 容器内临时文件
二、emptyDir临时存储
emptyDir是一种临时存储，pod在创建的时候在node上为容器申请一个临时的目录，跟随容器的生命周期，如果容器删除，emptyDir定义的临时存储空间也会删除，容器发生crash则不受影响，如果容器发生迁移，其上的数据也丢失，emptyDir一般用于测试或者缓存场景
定义一个emptyDir类型的临时存储，大小1G，将其挂在到容器内部redis的/data目录中
cat emptydir-redis.yaml
apiVersion: v1
kind: Pod
metadata:
name: emptydir-redis
labels:
volume: emptydir
annotations:
kubernetes.io/stroage: emptyDir
spec:
containers:
- name: emptydir-redis
image: redis:latest
imagePullPolicy: IfNotPresent
ports:
- name: redis-6379-port
protocol: TCP
containerPort: 6379
volumeMounts: # 定义的驱动emptydir-redis挂载到容器的/data目录
- name: emptydir-redis
mountPath: /data
volumes: # 定义一个存储
- name: emptydir-redis # 存储名字
emptyDir: # 存储驱动类型为emptyDir
sizeLimit: 1Gi
应用配置生成redis pod
kubectl apply -f emptydir-redis.yaml
查看pod详细信息
kubectl describe pods emptydir-redis
Name: emptydir-redis
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: node-3/172.19.159.9
Start Time: Wed, 11 Mar 2020 18:35:05 +0800
Labels: volume=emptydir
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"kubernetes.io/stroage":"emptyDir"},"labels":{"volume":"emptydir"},"name":"empt...
kubernetes.io/stroage: emptyDir
Status: Running
IP: 10.244.2.58
Containers:
emptydir-redis:
Container ID: docker://c8d559d2f6d10e7b6347b90d5e3db229327b1a8d16e393ae16574dde091a598c
Image: redis:latest
Image ID: docker-pullable://redis@sha256:6b9920bdc913ebeeed5cd5327decabe9fa829de425b52b3f28a7215ee7c7c457
Port: 6379/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 11 Mar 2020 23:28:08 +0800
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 11 Mar 2020 18:35:22 +0800
Finished: Wed, 11 Mar 2020 23:28:07 +0800
Ready: True
Restart Count: 1
Environment: <none>
Mounts: #挂载信息，将emptydir-redis挂载到/data目录，且是rw读写状态
/data from emptydir-redis (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-hg24n (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes: #定义了一个EmptyDir类型的存储，大小为1Gi
emptydir-redis:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: 1Gi
default-token-hg24n:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-hg24n
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
下面做一个测试，写入数据，然后把redis进程杀掉，pod异常默认会五分钟后重启，看是否数据还在
宿主机安装redis客户端
yum -y install redis
redis-cli -h 10.244.2.58
10.244.2.58:6379> set username wangteng
OK
10.244.2.58:6379> set volumes emptyDir
OK
10.244.2.58:6379>
先进到容器里面
kubectl exec -it emptydir-redis bash
杀掉redis server进程前需要知道它的pid号，需要临时在容器中安装procps工具
更换国内apt源
cp /etc/apt/sources.list /tmp/sources.list.bak
sed -i '1,$s/^/#/g' /etc/apt/sources.list
echo "
deb http://mirrors.163.com/debian/ buster main non-free contrib
deb http://mirrors.163.com/debian/ buster-updates main non-free contrib
deb http://mirrors.163.com/debian/ buster-backports main non-free contrib
deb-src http://mirrors.163.com/debian/ buster main non-free contrib
deb-src http://mirrors.163.com/debian/ buster-updates main non-free contrib
deb-src http://mirrors.163.com/debian/ buster-backports main non-free contrib
deb http://mirrors.163.com/debian-security/ buster/updates main non-free contrib
deb-src http://mirrors.163.com/debian-security/ buster/updates main non-free contrib
" >> /etc/apt/sources.list
apt-get update && apt-get install procps
top查看
1 redis 20 0 46328 10924 1688 S 0.0 0.3 0:01.04 redis-ser+
杀掉redis
kill 1 马上就会退出容器，接着会重启pod
redis-cli -h 10.244.2.58
0.244.2.59:6379> get username
"wangteng"
10.244.2.59:6379> get volume
(nil)
10.244.2.59:6379> get volumes
"emptyDir"
10.244.2.59:6379>
发现数据还在这是因为emptyDir实际是宿主机上创建的一个目录，将目录以bind mount的形势挂载到容器中，跟随容器的生命周期，到pod所在的node去查看容器详情
docker ps | grep redis
docker inspect containerID查看Mounts
{
"Type": "bind",
"Source": "/var/lib/kubelet/pods/d102e9b0-63fc-11ea-a525-00163e0855fc/volumes/kubernetes.io~empty-dir/emptydir-redis",
"Destination": "/data",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
ls -l /var/lib/kubelet/pods/d102e9b0-63fc-11ea-a525-00163e0855fc/volumes/kubernetes.io~empty-dir/emptydir-redis查看目录
total 4
-rw-r--r-- 1 polkitd input 134 Mar 12 09:12 dump.rdb
emptyDir是host上定义的一块临时存储，通过bind mount的形式挂载到容器中使用，容器重启数据会保留，容器删除则volume会随之删除
三、hostPath主机存储
与emptyDir存储驱动不同的是hostPath可以用指定的宿主机目录或文件来挂载到容器中指定的目录,用于单机测试场景，此外适用于一些容器业务需要访问宿主机目录，如监控系统访问/proc和/sys目录，日志系统访问/var/lib/docker目录的一些场景。支持设置不同的type类型
- Directory 本地存在的目录
- DirectoryOrCreate 不存在则创建，权限默认755，属主和组与kubelet一致
- File 本地文件存在
- FileOrCreate 不存在则创建，权限默认644，属主和组与kubelet一致
- Socket 已存在的Socket文件
- BlockDevice 本地已存在的Block块设备
- CharDevice 本地已存在的Char字符设备
cat hostpath-demo.yaml
apiVersion: v1
kind: Pod
metadata:
name: hostpath-demo # pod名称
labels:
stroage: hostpath
annotations:
kubernetes.io/stroage: "hostpath"
spec:
containers:
- name: nginx
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- name: nginx-http-port
protocol: TCP
containerPort: 80
volumeMounts:
- name: hostpath-demo # 通过该名称来联系宿主机目录挂在容器目录
mountPath: /usr/share/nginx/html # 容器目录
volumes: # 定义存储驱动类型
- name: hostpath-demo # 通过该名称来联系宿主机目录挂在容器目录
hostPath: # 存储驱动类型
type: DirectoryOrCreate # 表示如果宿主机不存在指定的目录则创建
path: /mnt/data # 宿主机目录
创建pod
kubectl apply -f hostpath-demo.yaml
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hostpath-demo 1/1 Running 0 20m 10.244.2.61 node-3 <none> <none>
在pod所在的node上作如下处理
echo "test page" >> /mnt/data/index.html
curl 10.244.2.61
输出test page
查看容器挂载存储的情况，以bind mount的形式挂载到容器中
在pod所在的节点上
docker ps | grep hostpath-demo
0c2a46a37ff8 6678c7c2e56c "nginx -g 'daemon of…" 24 minutes ago Up 24 minutes k8s_nginx_hostpath-demo_default_1adf9163-6424-11ea-a525-00163e0855fc_0
4a09fac2973d k8s.gcr.io/pause:3.1 "/pause" 24 minutes ago Up 24 minutes k8s_POD_hostpath-demo_default_1adf9163-6424-11ea-a525-00163e0855fc_0
docker inspect 0c2a46a37ff8查看Mounts信息
"Type": "bind",
"Source": "/mnt/data",
"Destination": "/usr/share/nginx/html",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
下面做个测试，我们在docker层面杀掉进程
docker kill 0c2a46a37ff8
再次查看pod
kubectl get pods hostpath-demo -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hostpath-demo 1/1 Running 1 29m 10.244.2.61 node-3 <none> <none>
获取pod的地址，通过RESTART可知，容器重启过一次，测试数据依旧保留
curl 10.244.2.61
输出test page
容器删除，volume还存在
四、NFS存储对接
三台节点上安装nfs
yum -y install nfs-utils
在node-1上配置nfs server
cat /etc/exports
/mnt/data 172.19.159.0/24(rw)
systemctl restart nfs
showmount -e node-1
Export list for node-1:
/mnt/data 172.19.159.0/24
cat nfs-demo.yaml
metadata:
name: nfs-demo
labels:
storage: nfs
annotations:
kubernetes.io/stroage: nfs
spec:
containers:
- name: nginx
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- name: nginx-http-port
protocol: TCP
containerPort: 80
volumeMounts:
- name: nfs-demo
mountPath: /usr/share/nginx/html
volumes:
- name: nfs-demo
nfs:
server: 172.19.159.7
path: /mnt/data # 该目录需在宿主机提前创建
kubectl apply -f nfs-demo.yaml
kubectl get pods nfs-demo -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nfs-demo 1/1 Running 0 26s 10.244.2.63 node-3 <none> <none>
echo "test page 2" > /mnt/data/index.html
curl 10.244.2.63
输出test page 2
kubectl delete pods nfs-demo
mount.nfs node-1:/mnt/data /media
ls -l /media
total 4
-rw-r--r-- 1 root root 12 Mar 12 21:36 index.html
五、总结
kubernetes的存储驱动类型emptyDir、hostPath、nfs都可以将宿主机文件目录以bind mount方式挂载到容器里面，但是emptyDir只是临时挂载，删掉pod也就丢失了数据，hostPath和nfs可以在删掉pod后数据还在
