kubernetes

kubernetes学习笔记-pod存储

一、概念

kubernetes启动的容器中的数据是临时的,重启容器后数据丢失,但是kubernetes提供了volume存储的抽象,volume后端能够支持多种不同的plugin驱动,通过spec.volumes定义一个存储,然后在容器中spec.containers.volumeMounts中调用,最终在容器中以目录的形式呈现,kubernetes内置支持多种驱动类型,主要分为四种:公有云和私有云驱动接口、开源存储驱动如ceph rbd、本地临时存储hostPath、kubernetes对象API驱动接口如configmap
这里主要介绍一下本地临时存储:

  • hostPath 宿主机文件
  • emptyDir 容器内临时文件

二、emptyDir临时存储

emptyDir是一种临时存储,pod在创建的时候在node上为容器申请一个临时的目录,跟随容器的生命周期,如果容器删除,emptyDir定义的临时存储空间也会删除,容器发生crash则不受影响,如果容器发生迁移,其上的数据也丢失,emptyDir一般用于测试或者缓存场景
定义一个emptyDir类型的临时存储,大小1G,将其挂在到容器内部redis的/data目录中
cat emptydir-redis.yaml

apiVersion: v1
kind: Pod
metadata:
  name: emptydir-redis
  labels:
    volume: emptydir
  annotations:
    kubernetes.io/stroage: emptyDir
spec:
  containers:
  - name: emptydir-redis
    image: redis:latest
    imagePullPolicy: IfNotPresent
    ports:
    - name: redis-6379-port
      protocol: TCP
      containerPort: 6379
    volumeMounts:         # 定义的驱动emptydir-redis挂载到容器的/data目录
    - name: emptydir-redis
      mountPath: /data
  volumes:               # 定义一个存储
  - name: emptydir-redis # 存储名字
    emptyDir:            # 存储驱动类型为emptyDir
      sizeLimit: 1Gi 

应用配置生成redis pod
kubectl apply -f emptydir-redis.yaml
查看pod详细信息
kubectl describe pods emptydir-redis

Name:               emptydir-redis
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               node-3/172.19.159.9
Start Time:         Wed, 11 Mar 2020 18:35:05 +0800
Labels:             volume=emptydir
Annotations:        kubectl.kubernetes.io/last-applied-configuration:
                      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"kubernetes.io/stroage":"emptyDir"},"labels":{"volume":"emptydir"},"name":"empt...
                    kubernetes.io/stroage: emptyDir
Status:             Running
IP:                 10.244.2.58
Containers:
  emptydir-redis:
    Container ID:   docker://c8d559d2f6d10e7b6347b90d5e3db229327b1a8d16e393ae16574dde091a598c
    Image:          redis:latest
    Image ID:       docker-pullable://redis@sha256:6b9920bdc913ebeeed5cd5327decabe9fa829de425b52b3f28a7215ee7c7c457
    Port:           6379/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 11 Mar 2020 23:28:08 +0800
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 11 Mar 2020 18:35:22 +0800
      Finished:     Wed, 11 Mar 2020 23:28:07 +0800
    Ready:          True
    Restart Count:  1
    Environment:    <none>
    Mounts:                #挂载信息,将emptydir-redis挂载到/data目录,且是rw读写状态
      /data from emptydir-redis (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hg24n (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:                #定义了一个EmptyDir类型的存储,大小为1Gi
  emptydir-redis:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  1Gi
  default-token-hg24n:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-hg24n
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

下面做一个测试,写入数据,然后把redis进程杀掉,pod异常默认会五分钟后重启,看是否数据还在
宿主机安装redis客户端
yum -y install redis
redis-cli -h 10.244.2.58

10.244.2.58:6379> set username wangteng
OK
10.244.2.58:6379> set volumes emptyDir
OK
10.244.2.58:6379>

先进到容器里面
kubectl exec -it emptydir-redis bash
杀掉redis server进程前需要知道它的pid号,需要临时在容器中安装procps工具
更换国内apt源
cp /etc/apt/sources.list /tmp/sources.list.bak
sed -i '1,$s/^/#/g' /etc/apt/sources.list
echo "

deb http://mirrors.163.com/debian/ buster main non-free contrib
deb http://mirrors.163.com/debian/ buster-updates main non-free contrib
deb http://mirrors.163.com/debian/ buster-backports main non-free contrib
deb-src http://mirrors.163.com/debian/ buster main non-free contrib
deb-src http://mirrors.163.com/debian/ buster-updates main non-free contrib
deb-src http://mirrors.163.com/debian/ buster-backports main non-free contrib
deb http://mirrors.163.com/debian-security/ buster/updates main non-free contrib
deb-src http://mirrors.163.com/debian-security/ buster/updates main non-free contrib
" >> /etc/apt/sources.list

apt-get update && apt-get install procps
top查看

   1 redis     20   0   46328  10924   1688 S   0.0   0.3   0:01.04 redis-ser+ 

杀掉redis
kill 1 马上就会退出容器,接着会重启pod
redis-cli -h 10.244.2.58

0.244.2.59:6379> get username
"wangteng"
10.244.2.59:6379> get volume
(nil)
10.244.2.59:6379> get volumes
"emptyDir"
10.244.2.59:6379>

发现数据还在这是因为emptyDir实际是宿主机上创建的一个目录,将目录以bind mount的形势挂载到容器中,跟随容器的生命周期,到pod所在的node去查看容器详情
docker ps | grep redis
docker inspect containerID查看Mounts

{
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/d102e9b0-63fc-11ea-a525-00163e0855fc/volumes/kubernetes.io~empty-dir/emptydir-redis",
                "Destination": "/data",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },

ls -l /var/lib/kubelet/pods/d102e9b0-63fc-11ea-a525-00163e0855fc/volumes/kubernetes.io~empty-dir/emptydir-redis查看目录

total 4
-rw-r--r-- 1 polkitd input 134 Mar 12 09:12 dump.rdb

emptyDir是host上定义的一块临时存储,通过bind mount的形式挂载到容器中使用,容器重启数据会保留,容器删除则volume会随之删除

三、hostPath主机存储

与emptyDir存储驱动不同的是hostPath可以用指定的宿主机目录或文件来挂载到容器中指定的目录,用于单机测试场景,此外适用于一些容器业务需要访问宿主机目录,如监控系统访问/proc和/sys目录,日志系统访问/var/lib/docker目录的一些场景。支持设置不同的type类型

  • Directory 本地存在的目录
  • DirectoryOrCreate 不存在则创建,权限默认755,属主和组与kubelet一致
  • File 本地文件存在
  • FileOrCreate 不存在则创建,权限默认644,属主和组与kubelet一致
  • Socket 已存在的Socket文件
  • BlockDevice 本地已存在的Block块设备
  • CharDevice 本地已存在的Char字符设备

cat hostpath-demo.yaml

apiVersion: v1
kind: Pod
metadata:
  name: hostpath-demo  # pod名称
  labels:
    stroage: hostpath
  annotations:
    kubernetes.io/stroage: "hostpath"
spec:
  containers:
  - name: nginx
    image: nginx:latest
    imagePullPolicy: IfNotPresent
    ports:
    - name: nginx-http-port
      protocol: TCP
      containerPort: 80
    volumeMounts:  
    - name: hostpath-demo  # 通过该名称来联系宿主机目录挂在容器目录
      mountPath: /usr/share/nginx/html # 容器目录
  volumes:    # 定义存储驱动类型
  - name: hostpath-demo   # 通过该名称来联系宿主机目录挂在容器目录
    hostPath:   # 存储驱动类型
      type: DirectoryOrCreate   # 表示如果宿主机不存在指定的目录则创建 
      path: /mnt/data  # 宿主机目录

创建pod
kubectl apply -f hostpath-demo.yaml
kubectl get pods -o wide

NAME            READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
hostpath-demo   1/1     Running   0          20m   10.244.2.61   node-3   <none>           <none>

在pod所在的node上作如下处理
echo "test page" >> /mnt/data/index.html
curl 10.244.2.61
输出test page
查看容器挂载存储的情况,以bind mount的形式挂载到容器中
在pod所在的节点上
docker ps | grep hostpath-demo

0c2a46a37ff8        6678c7c2e56c           "nginx -g 'daemon of…"   24 minutes ago      Up 24 minutes                           k8s_nginx_hostpath-demo_default_1adf9163-6424-11ea-a525-00163e0855fc_0
4a09fac2973d        k8s.gcr.io/pause:3.1   "/pause"                 24 minutes ago      Up 24 minutes                           k8s_POD_hostpath-demo_default_1adf9163-6424-11ea-a525-00163e0855fc_0

docker inspect 0c2a46a37ff8查看Mounts信息

                "Type": "bind",
                "Source": "/mnt/data",
                "Destination": "/usr/share/nginx/html",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"

下面做个测试,我们在docker层面杀掉进程
docker kill 0c2a46a37ff8
再次查看pod
kubectl get pods hostpath-demo -o wide

NAME            READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
hostpath-demo   1/1     Running   1          29m   10.244.2.61   node-3   <none>           <none>

获取pod的地址,通过RESTART可知,容器重启过一次,测试数据依旧保留
curl 10.244.2.61
输出test page
容器删除,volume还存在

四、NFS存储对接

三台节点上安装nfs
yum -y install nfs-utils
在node-1上配置nfs server
cat /etc/exports

/mnt/data 172.19.159.0/24(rw)

systemctl restart nfs
showmount -e node-1

Export list for node-1:
/mnt/data 172.19.159.0/24

cat nfs-demo.yaml

metadata:
  name: nfs-demo
  labels:
    storage: nfs
  annotations:
    kubernetes.io/stroage: nfs
spec:
  containers:
  - name: nginx
    image: nginx:latest
    imagePullPolicy: IfNotPresent
    ports:
    - name: nginx-http-port
      protocol: TCP
      containerPort: 80
    volumeMounts:
    - name: nfs-demo
      mountPath: /usr/share/nginx/html
  volumes:
  - name: nfs-demo
    nfs:
      server: 172.19.159.7
      path: /mnt/data   # 该目录需在宿主机提前创建

kubectl apply -f nfs-demo.yaml
kubectl get pods nfs-demo -o wide

NAME       READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
nfs-demo   1/1     Running   0          26s   10.244.2.63   node-3   <none>           <none>

echo "test page 2" > /mnt/data/index.html
curl 10.244.2.63
输出test page 2
kubectl delete pods nfs-demo
mount.nfs node-1:/mnt/data /media
ls -l /media

total 4
-rw-r--r-- 1 root root 12 Mar 12 21:36 index.html

五、总结
kubernetes的存储驱动类型emptyDir、hostPath、nfs都可以将宿主机文件目录以bind mount方式挂载到容器里面,但是emptyDir只是临时挂载,删掉pod也就丢失了数据,hostPath和nfs可以在删掉pod后数据还在

支付宝扫码打赏 微信打赏

若你觉得我的文章对你有帮助,欢迎点击上方按钮对我打赏

扫描二维码,分享此文章

linuxwt's Picture
linuxwt

我叫王腾,来自武汉,2016年毕业后在上海做了一年helpdesk,自学了linux后回武汉从事系统运维的工作,从2017年开始写博客记录自己的学习工作,现在正在进行数据迁移到此博客,目前就职于中国移动设计院有限公司,个人的座右铭是:逃脱舒适区才能在闲暇的时候惬意的玩耍。

武汉光谷 https://linuxwt.com

Subscribe to 今晚打老虎

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!

Comments