公网服务器搭建Docker私有镜像仓库Registry

一、Registry介绍

Registry是官方推荐的私有镜像服务,适用于在局域网内自定义装载镜像,以往都是简单部署一个registry通过http的方式来访问,事实上它是可以有用户认证、ssl认证、开源web管理方案的,这里结合工作以及网络中的一些资料做一个整理,由浅入深的来搭建一个功能齐全的私有仓库

二、搭建开始

2.1.准备工作

  • 阿里云ECS服务器一台,其地址为172.19.159.10
  • ssl免费证书

2.2.搭建基本registry

先简单实现一个registry服务
cat docker-compose.yml

registry_linuxwt:
    restart: always
    image: registry:latest
    container_name: registry_linuxwt
    volumes:
        - ./registry:/var/lib/registry
        - /etc/localtime:/etc/localtime
        - /etc/timezone:/etc/timezone
    privileged: true
    ports:
       - "5000:5000"

docker-compose up -d
浏览器访问http://172.19.159.10:5000/v2/_catalog 可以看到空仓库,这说明一个基本的私有镜像仓库搭建完成
但是由于registry默认是需要通过https来访问,如果想要进行镜像推送拉取,需要在目标服务器上的文件/etc/docker/daemon.json中设置,否则在其他机器上无法推送和拉取镜像,需要添加insecure-registries参数来忽略对registry域名证书的审核
cat /etc/docker/daemon.json

{
    "registry-mirrors":["https://nr630v1c.mirror.aliyuncs.com"],
    "insecure-registries": [ "172.19.159.10:5000"]
}

2.3.设置用户认证

一般来讲我们需要对用registry设置账号密码以此来限制使用者,这样更安全更好管理
在前面的基础环境上来设置账号密码
mkdir auth && cd auth
echo "user:wangteng passwd:123456" >htpasswd
cd ..
docker run --entrypoint htpasswd registry:latest -Bbn wangteng 123456 >auth/htpasswd
cat /root/Registry/docker-compose.yml

egistry_linuxwt:
    restart: always
    image: registry:latest
    container_name: registry_linuxwt
    volumes:       
        - ./auth:/auth       
        - ./registry:/var/lib/registry
        - /etc/localtime:/etc/localtime
        - /etc/timezone:/etc/timezone
    privileged: true
    environment:
       REGISTRY_AUTH: "htpasswd"
       REGISTRY_AUTH_HTPASSWD_REALM: "Registry Realm"
       REGISTRY_AUTH_HTPASSWD_PATH: "/auth/htpasswd"
    ports:
       - "5000:5000"

同理也要设置/etc/docker/daemon.json(同上)
在目标服务器不论是拉取还是推送镜像都需要先登录私有仓库
docker login --username="wangteng" --password="123456" 172.19.159.10:5000

2.4.SSL证书添加

前面已经基本实现了一个registry私有仓库服务,可以通过http的方式来访问,但是这还不够,而且在目标服务器(需要从registry拉取镜像的服务器)上去配置/etc/docker/daemon.json必须使用insecure-registries,为了能使https来访问,需要进行SSL证书添加

免费证书获取
这次在一台全新的阿里云服务器上申请时利用上面链接的文章来申请出现了下面的错误

The client lacks sufficient authorization :: Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See

这是因为从2019年11月份letsencrypt已经不支持ACMEv1了,需要升级到ACMEv2,但是我看了下服务器上是没有安装acme客户端的,最后在github上找到了acme项目
下载acme.sh
安装acme
./acme.sh --install
输出如下信息

[Sat Apr  4 12:20:32 CST 2020] Installing to /root/.acme.sh
[Sat Apr  4 12:20:32 CST 2020] Installed to /root/.acme.sh/acme.sh
[Sat Apr  4 12:20:32 CST 2020] Installing alias to '/root/.bashrc'
[Sat Apr  4 12:20:32 CST 2020] OK, Close and reopen your terminal to start using acme.sh
[Sat Apr  4 12:20:32 CST 2020] Installing alias to '/root/.cshrc'
[Sat Apr  4 12:20:32 CST 2020] Installing alias to '/root/.tcshrc'
[Sat Apr  4 12:20:33 CST 2020] Installing cron job
42 0 * * * "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null
[Sat Apr  4 12:20:33 CST 2020] Good, bash is found, so change the shebang to use bash as preferred.
[Sat Apr  4 12:20:33 CST 2020] OK

获取证书
这次直接利用acme.sh脚本来获取,不用前面那个链接里的文章来获取了,感觉这个更方便
./.acme.sh/acme.sh --issue -d registry.linuxwt.com --standalone
输出

[Sat Apr  4 12:40:07 CST 2020] Your cert is in  /root/.acme.sh/registry.linuxwt.com/registry.linuxwt.com.cer 
[Sat Apr  4 12:40:07 CST 2020] Your cert key is in  /root/.acme.sh/registry.linuxwt.com/registry.linuxwt.com.key 
[Sat Apr  4 12:40:07 CST 2020] The intermediate CA cert is in  /root/.acme.sh/registry.linuxwt.com/ca.cer 
[Sat Apr  4 12:40:07 CST 2020] And the full chain certs is there:  /root/.acme.sh/registry.linuxwt.com/fullchain.cer 

看一下获取的证书所在目录详情
ls -la /root/.acme.sh/registry.linuxwt.com/

total 40
drwxr-xr-x 3 root root 4096 Apr  4 13:26 .
drwx------ 4 root root 4096 Apr  4 13:19 ..
drwxr-xr-x 2 root root 4096 Apr  4 13:26 backup
-rw-r--r-- 1 root root 1648 Apr  4 12:40 ca.cer
-rw-r--r-- 1 root root 3571 Apr  4 12:40 fullchain.cer
-rw-r--r-- 1 root root 1923 Apr  4 12:40 registry.linuxwt.com.cer
-rw-r--r-- 1 root root  813 Apr  4 13:26 registry.linuxwt.com.conf
-rw-r--r-- 1 root root  993 Apr  4 12:39 registry.linuxwt.com.csr
-rw-r--r-- 1 root root  215 Apr  4 12:39 registry.linuxwt.com.csr.conf
-rw-r--r-- 1 root root 1675 Apr  4 12:39 registry.linuxwt.com.key

注意上面的文件是用于内部,不是直接使用这些文件,使用以下的命令在目标目录下生成证书
acme.sh --installcert -d registry.linuxwt.com --key-file /root/Registry/registry.linuxwt.com.key --fullchain-file /root/Registry/registry.linuxwt.com.cer

同时registry需要开启ssl,它需要的证书必须是以crt和key结尾
cat registry.linuxwt.com.cer > registry.linuxwt.com.crt
来看一下整个服务的目录
cd /root/Registry
ls -l

drwxr-xr-x 2 root root 4096 Mar 22 18:14 auth
-rw-r--r-- 1 root root 1371 Apr  4 15:59 docker-compose.yml
drwxr-xr-x 2 root root 4096 Apr  4 13:13 html
-rw-r--r-- 1 root root 1191 Apr  4 13:47 nginx.conf
-rw-r--r-- 1 root root  762 Apr  4 16:09 nginx_flask.conf
drwxr-xr-x 3 root root 4096 Apr  4 16:02 registry
-rw-r--r-- 1 root root 3571 Apr  4 13:26 registry.linuxwt.com.cer
-rw-r--r-- 1 root root 3571 Apr  4 14:50 registry.linuxwt.com.crt
-rw------- 1 root root 1675 Apr  4 13:26 registry.linuxwt.com.key

cat docker-compose.yml

registry_linuxwt:
    restart: always
    image: registry:latest
    container_name: registry_linuxwt
    volumes:       
        - ./auth:/auth       
        - ./registry:/var/lib/registry
        - ./registry.linuxwt.com.crt:/cert/registry.linuxwt.com.crt
        - ./registry.linuxwt.com.key:/cert/registry.linuxwt.com.key
        - /etc/localtime:/etc/localtime
        - /etc/timezone:/etc/timezone
    privileged: true
    environment:
       REGISTRY_AUTH: "htpasswd"
       REGISTRY_AUTH_HTPASSWD_REALM: "Registry Realm"
       REGISTRY_AUTH_HTPASSWD_PATH: "/auth/htpasswd"
       REGISTRY_HTTP_TLS_CERTIFICATE: "/cert/registry.linuxwt.com.crt"
       REGISTRY_HTTP_TLS_KEY: "/cert/registry.linuxwt.com.key"
    ports:
       - "5000:5000"
nginx_linuxwt:
     restart: always
     image: nginx:1.15
     container_name: nginx_linuxwt
     volumes:
        - ./registry.linuxwt.com.crt:/etc/registry.linuxwt.com.crt
        - ./registry.linuxwt.com.key:/etc/registry.linuxwt.com.key
        - ./nginx.conf:/etc/nginx/nginx.conf
        - ./nginx_flask.conf:/etc/nginx/conf.d/default.conf
        - /etc/localtime:/etc/localtime
        - /etc/timezone:/etc/timezone
     privileged: true
     links:
        - registry_linuxwt
     ports:
        - "80:80"
        - "443:443"

cat nginx.conf

user  root;  

worker_processes  1;
# 错误日志
error_log  /var/log/nginx/error.log warn;  
pid        /var/run/nginx.pid;

events {  
    worker_connections  1024;
}
http {  
    include       /etc/nginx/mime.types;
        server_tokens off;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
# 访问日志
    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    #tcp_nopush     on;
    keepalive_timeout  3000;

    autoindex off;

    gzip  on;

    gzip_min_length  1k;

    gzip_buffers     4 16k;

    gzip_http_version 1.1;

    gzip_comp_level 2;

    #  下面是一整段

    gzip_types text/plain image/png  application/javascript application/x-javascript  text/javascript text/css application/xml image/x-icon application/xml+rss 

    application/json; 

    gzip_vary on;

    gzip_proxied   expired no-cache no-store private auth;

    gzip_disable   "MSIE [1-6]\.";

    include /etc/nginx/conf.d/*.conf;

cat nginx_flask.conf

server {
    listen 443 ssl;
    server_name registry.linuxwt.com;
    ssl on;
    root html;
    index index.html index.htm;
    ssl_certificate   /etc/registry.linuxwt.com.crt;
    ssl_certificate_key  /etc/registry.linuxwt.com.key;
    ssl_session_timeout 5m;
    ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    location ~  {
        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_pass https://registry;    
    }
    client_max_body_size 0;
}
    upstream registry {
        server registry_linuxwt:5000;
    }

这里的proxy_pass必须使用https,因为registry启用了ssl,使用http是无法访问的

配置了ssl来访问私有镜像仓库就不用在目标服务器上设置/etc/docker/daemon.json里的insecure-registries了
到这里registry就部署完成了,访问https://registry.linuxwt.com/v2/_catalog 验证是否部署成功

2.5.registry实践管理

推送
docker login registry.linuxwt.com
docker tag imageID registry.linuxwt.com/imagename:version
拉取
docker pull registry.linuxwt.com/imagename:version
查看有哪些镜像
访问https://registry.linuxwt.com/v2/_catalog 或者
curl --user wangteng:123456 https://registry.linuxwt.com/v2/_catalog

这是当时测试推送上去的两个镜像
{"repositories":["loader","mysql"]}

查看镜像具体有哪些版本
访问https://registry.linuxwt.com/v2/loader/tags/list
{"name":"loader","tags":["1.3"]}
或者
curl https://registry.linuxwt.com/v2/loader/tags/list --user wangteng:123456

删除镜像
删除镜像有三层意思

  • 删除某个镜像
  • 删除某个镜像的特定版本
  • 删除所有镜像

以已上传的镜像loader:1.3为例
删除镜像的特定版本
获取该镜像的digest

curl -v --silent --user wangteng:123456 -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET  https://registry.linuxwt.com/v2/loader/manifests/1.3 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'

sha256:ff1520254de7cc4df8a1b47a56d489040e90cde95c0540c77b6085ec03fe4a69
根据上面的digest来删除loader:1.3

curl -v --silent --user wangteng:123456 -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X DELETE https://registry.linuxwt.com/v2/loader/manifests/sha256:ff1520254de7cc4df8a1b47a56d489040e90cde95c0540c77b6085ec03fe4a69   

报错

{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}

不支持删除目前,只能利用第三方工具来进行删除

删除镜像所有版本
docker exec registry_linuxwt rm -Rf /var/lib/registry/docker/registry/v2/repositories/loader
还没完,还需要进行垃圾回收
docker exec registry_linuxwt bin/registry garbage-collect /etc/docker/registry/config.yml

清空仓库
docker exec registry_linuxwt rm -Rf /var/lib/registry/docker/registry/v2/repositories/*
同理垃圾回收

2.6.docker registry webUI

docker-registry-web是一个开源的管理私有仓库的工具,前面无法删除固定版本的镜像可以通过这个工具来删除,需要添加一些配置,分别在config.yml和web-config.yml
cat docker-compose.yml

registry_linuxwt:
    restart: always
    image: registry:latest
    container_name: registry_linuxwt
    volumes:
        - ./auth:/auth       
        - ./registry:/var/lib/registry
        - ./registry.linuxwt.com.crt:/cert/registry.linuxwt.com.crt
        - ./registry.linuxwt.com.key:/cert/registry.linuxwt.com.key
        - /etc/localtime:/etc/localtime
        - /etc/timezone:/etc/timezone
        - ./config.yml:/etc/docker/registry/config.yml
    privileged: true
    environment:
         REGISTRY_AUTH: "htpasswd"
         REGISTRY_AUTH_HTPASSWD_REALM: "Registry Realm"
         REGISTRY_AUTH_HTPASSWD_PATH: "/auth/htpasswd
         REGISTRY_HTTP_TLS_CERTIFICATE: "/cert/registry.linuxwt.com.crt"
         REGISTRY_HTTP_TLS_KEY: "/cert/registry.linuxwt.com.key"
    ports:
       - "5000:5000"
nginx_linuxwt:
     restart: always
     image: nginx:1.15
     container_name: nginx_linuxwt
     volumes:
        - ./registry.linuxwt.com.crt:/etc/registry.linuxwt.com.crt
        - ./registry.linuxwt.com.key:/etc/registry.linuxwt.com.key
        - ./nginx.conf:/etc/nginx/nginx.conf
        - ./nginx_flask.conf:/etc/nginx/conf.d/default.conf
        - /etc/localtime:/etc/localtime
        - /etc/timezone:/etc/timezone
     privileged: true
     links:
        - registry_linuxwt
     ports:
        - "80:80"
        - "443:443"
webui:
    restart: always
    image: hyper/docker-registry-web
    container_name: registry_webui
    privileged: true
    volumes:
       - ./web-config.yml:/conf/config.yml
       - ./db:/data
    environment:
       REGISTRY_URL: "https://registry.linuxwt.com/v2"
       REGISTRY_TRUST_ANY_SSL: "true"
       REGISTRY_NAME: "registry.linuxwt.com"
       REGISTRY_BASIC_AUTH: "d2FuZ3Rlbmc6MTIzNDU2"
    ports:
      - "8080:8080"   

如果不加上config.yml与web-config.yml里的配置是无法删除镜像的,同时docker-compose.yml文件里的 REGISTRY_BASIC_AUTH变量可以通过/root/.docker/config.json查看,这个文件会在登录镜像仓库后自动生成

具体配置
cat config.yml

version: 0.1
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
http:
  addr: 0.0.0.0:5000

cat web-config.yml

registry:
  # Docker registry url
  url: https://registry.linuxwt.com/v2
  # Docker registry fqdn
  name: registry.linuxwt.com:5000
  # To allow image delete, should be false
  readonly: false
  auth:
    # Disable authentication
    enabled: false

访问http://registry.linuxwt.com:8080 可以看到
04reg1
04reg2

部署registry-web主要是为了方便删除镜像,但会有一个小的显示bug,以删除图中的这个镜像为例
04web1
先看一下registry里该镜像有哪些版本
curl https://registry.linuxwt.com/v2/registry-web/tags/list --user wangteng:123456
{"name":"registry-web","tags":["latest"]}

使用registry-web进行删除
04web2
垃圾清理

再次查看镜像版本
{"name":"registry-web","tags":null}

查看镜像仓库
{"repositories":["centos","goalng","golang","registry-web"]}
这说明使用registry-web删除了镜像,但是在registry里面还保留了镜像名的标签,虽然已经没有镜像了(无法进行镜像拉取)
如果要彻底删除标签,还是使用上面的仓库清空命令,清空registry-web镜像仓库

修改registry以及web-registry对registry的认证
修改registry账号密码参照前面设置其账号密码的过程,重点是修改web-registry对registry的认证
每一次对registry修改账号密码后下图的basic认证失效
04basic
docker logout https://registry.linuxwt.com 后重新登录即可重新生成basic认证,该认证可以在文件/root/.docker/config.json中查看

2.7.总结

1、本文同时开启了nginx和registry的ssl认证,要注意证书在nginx与registry中的证书名后缀问题
2、nginx反向代理的地址一定要使用https(proxy_pass https://xxx),因为registry开启了ssl
3、在推送镜像到registry的镜像最好使用无缓存模式来构建,这样每次构建的镜像的ID是唯一的,这对于在registry-web上删除镜像更为安全,比如源码级别来构建一个应用,假设你更改了代码并push到Git仓库。新代码不会check out下来,因为git clone命令没有更改。在Docker看来git clone的步骤一样,所以使用了缓存,在这种情况下,你可能不想开启docker的缓存了
docker build -t imagename:version --no-cache .
4、虽然本文中的服务器使用的阿里云ECS,域名解析的该服务器,但是如果和该registry服务器处于同一局域网的时候,添加insecure-registries参数后是可以采用内网地址来推送的
docker login https://registry.linuxwt.com
docker push 172.19.159.10:5000/imagename:version
这样比走公网地址docker push registry.linuxwt.com/imagename:version 更快
5、本文中registry账户认证使用的是htpasswd,其实还可以使用token以及nginx认证
6、本文中使用的域名进行证书申请并在公网服务器上部署,如果是在局域网内可否使用https来访问,是否需要在局域网内部署dns,还是通过host来进行简单映射
7、可以使用内网地址来登录私有仓库,但是无法通过curl命令来展示镜像细节,比如使用命令
curl 172.19.159.10:5000/v2/_catalog,这是因为registry是开启了ssl认证的,必须使用https协议来访问,但是可以推送镜像

2.8.参考

docker-registry-web
acme
本文github