容器云项目-CICD组件配置

一、cicd原理

持续集成测试前需要对各个组件进行配置,在测试前需要理解各个组件在持续集成中所起的作用,下面用一张简略图来说明,这有助于理解接下来的配置

07cicd

其过程描述如下
1、推送代码至gitlab
2、推送事件触发gitlab配中配置的关联jenkins的webhook
3、jenkins检出目的分支源码至项目工作空间
4、jenkins在项目工作空间中对源码进行编译、打包、发布,生成应用,在编译过程中会按照本地仓库、私有仓库、中央仓库的优先级进行依赖包拉取
5、jenkins在项目工作空间将应用封装到docker镜像中,推送镜像至harbor私有仓库
6、jenkins在项目工作空间执行部署脚本发布应用至kubernetes集群

ps:在整个持续集成测试过程当中步骤3~6都是通过pipeline工程里Jenkinsfile来实现

二、组件配置

在进行流水线持续集成测试的时候各个组件之间的通信是需要进行相关的配置,这些配置如下
1、jenkins访问gitlab的凭据
2、jenkins访问gitlab中项目的凭据
3、jenkins访问harbor私有仓库的凭据
4、jenkins全局工具配置,包括git maven docker docker registry
5、gitlab中触发jenkins拉取代码的webhook配置
6、nexus私有仓库配置
7、jenkins中maven工具配置私有仓库
8、jenkins中docker工具配置http方式访问harbor、访问jenkins所在宿主机的docker daemon
9、配置K8s集群拉取harbor镜像仓库中需要的secrets权限
10、jenkins系统配置

2.1、凭据配置

gitlab凭据配置
获取gitlab token
gitlab主页-右上角的administrator下拉的Preference-左边的Access Tokens
07pj

配置凭据
07pj3

gitlab项目凭据配置
07pj4
这里要注意和gitlab凭据区分

harbor凭据配置
类似于gitlab项目凭据,最好通过凭据描述与其他凭据区分开来

最终呈现的画面如下
07pj5

jenkins全局工具配置
maven配置文件、jdk配置,这里jdk是镜像里自带的
07h5
git
07h6
maven
07h7
docker
07h8

2.2、webhook配置

本身webhook的配置是需要在jenkins配置具体的项目时才能配置,这里暂时先拿过来作为示例展示

jenkins上的配置
07pj6

上面的URL地址是需要在jenkins系统设置界面进行设置
07pj8

07pj7

gitlab上的配置
07pj9

2.3、maven私有仓库配置

和linux系统类似,为了更快速的在编译java源码(这里以java项目为例)时能快速拉取依赖包,可以在本地局域网创建私有maven仓库

首次编译需要通过网络去下载包,那么最好将maven源替换为国内源,使用阿里的maven源,同时还需要在nexus作相应的配置

配置中央仓库
更改maven源
cat settings.xml

...
<mirror>
    <id>alimaven</id>
    <name>aliyun maven</name>
    <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    <mirrorOf>central</mirrorOf>        
</mirror>
...   

设置本地maven仓库路径
cat settings.xml

...
<localRepository>/var/jenkins_home/repository</localRepository>
...

首次编译完后下载的包都会存入该路径,后面再次编译会优先从本地仓库拉取包

同时配置源码
cat pom.xml

...
<repositories>
		<repository>
			<id>alimaven</id>
			<name>aliyun maven</name>
			<url>http://maven.aliyun.com/nexus/content/repositories/central/</url>
			<releases>
				<enabled>true</enabled>
			</releases>
			<snapshots>
				<enabled>false</enabled>
			</snapshots>
		</repository
</repositories>
...

nexus中创建私有库
创建maven私有仓库的存储
07pj54

创建maven私有仓库
07h1

将私有仓库加入公共仓库
07h3

上传依赖包到私有仓库
第一次编译完成后可以在本地maven仓库路径获取下载下来的jar包,上传这些包到私有仓库

cat import.sh

cd /var/jenkins_home/repository   
#!/bin/bash
# copy and run this script to the root of the repository directory containing files
# this script attempts to exclude uploading itself explicitly so the script name is important
# Get command line params
while getopts ":r:u:p:" opt; do
        case $opt in
                r) REPO_URL="$OPTARG"
                ;;
                u) USERNAME="$OPTARG"
                ;;
                p) PASSWORD="$OPTARG"
                ;;
        esac
done

find . -type f -not -path './mavenimport\.sh*' -not -path '*/\.*' -not -path '*/\^archetype\-catalog\.xml*' -not -path '*/\^maven\-metadata\-local*\.xml' -not -path '*/\^maven\-metadata\-deployment*\.xml' | sed "s|^\./||" | xargs -I '{}' curl -u "$USERNAME:$PASSWORD" -X PUT -v -T {} ${REPO_URL}/{} ;

bash import.sh -u admin -r http://nexus-svc:8081/repository/maven-local/ -p xxxxxxx导入包

ps:这里如果nexus密码含有特殊字符,比如@ !等请加上转移字符''

maven中配置私有库
cat settings.xml

...
 <mirror>
      <id>maven-public</id>
      <mirrorOf>*</mirrorOf>
      <name>maven-public</name>
      <url>http://nexus-svc:8081/repository/maven-public/</url>
 </mirror>
...  
<server>
        <id>maven-public</id>
        <username>admin</username>
        <password>123456</password>
</server>
<server>
        <id>maven-releases</id>
        <username>admin</username>
        <password>123456</password>
</server>
<server>
        <id>maven-snapshots</id>
        <username>admin</username>
        <password>123456</password>
</server>
...

同时配置源码
cat pom.xml

...
<!-- 设置deploy的地址 -->
    <distributionManagement>
        
        <repository>
            <id>maven-releases</id>                    <!--库的ID-->
            <name>maven-releases</name>   <!--库的名字-->
            <url>http://nexus-svc:8081/repository/maven-releases/</url>
        </repository>

        <snapshotRepository>
            <id>maven-snapshots</id>
            <name>maven-snapshot</name>
            <url>http://nexus-svc:8081/repository/maven-snapshots/</url>
        </snapshotRepository>

    </distributionManagement>
...   

2.4、jenkins应用中的docker配置

需要配置三个地方

  • 配置http方式访问harbor
  • 配置docker工具访问宿主机docker api
  • 配置secret允许jenkins拉取私有镜像

为什么要配置这三个地方,下面依次简单解释
1、harbor私有镜像仓库默认使用https方式访问,这里使用http访问需要配置

2、docker image需要访问docker api,本身jenkins镜像自带了一个docker客户端,但是没有服务端,所以可以通过挂载宿主机的docker daemon 文件/var/run/docker.sock至Pod内,这样可以使用构建命令来构建镜像,当然这个镜像是生成在宿主机上,同时为了与服务端保持一致,最好也将宿主机的docker客户端文件也挂载到jenkins的pod内,与服务端保持版本的一致

3、在kubernetes集群中如果要私有仓库拉取镜像需要创建secret来实现

以上三点都需要配置jenkins的部署文件实现,这里使用configMap来实现以上三点,不过在此之前还需要做两件事
实现http方式访问harbor
node19创建文件daemon.json
cat /etc/docker/daemon.json

{
    "registry-mirrors":["https://nr630v1c.mirror.aliyuncs.com"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "insecure-registries": [ "10.0.0.19:30002"],
    "log-driver": "json-file",
    "log-opts": {
    "max-size": "100m"
    },
    "storage-driver": "overlay2",
    "storage-opts": [
    "overlay2.override_kernel_check=true"
    ]
}

重启docker生效
docker login 10.0.0.19:30002首次登录私有仓库
scp /etc/docker/daemon.json node20:/etc/docker
scp /etc/docker/daemon.json node21:/etc/docker
scp .docker node20:/root
scp .docker node21:/root
重启node20 node21上docker

创建能实现kubernetes能拉取镜像的secret
/root/.docker/config.json | base64 -w 0生成config.json的编码字符串
cat harbor-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: harbor-secret
  namespace: devops
data:
  .dockerconfigjson: xxxxxxxxxxxxxxxxxxxxxxxx

准备工作就绪,下面在jenkins中添加相应配置
cat jenkins-configmap.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: jenkins-configmap
  namespace: devops
data:
  daemon.json: |-
    {
    "registry-mirrors":["https://nr630v1c.mirror.aliyuncs.com"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "insecure-registries": [ "10.0.0.19:30002"],
    "log-driver": "json-file",
    "log-opts": {
    "max-size": "100m"
    },
    "storage-driver": "overlay2",
    "storage-opts": [
    "overlay2.override_kernel_check=true"
    ]
    }
  settings.xml: |-
    <?xml version="1.0" encoding="UTF-8"?>
    
    <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
    
        http://www.apache.org/licenses/LICENSE-2.0
    
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
    
    <!--
     | This is the configuration file for Maven. It can be specified at two levels:
     |
     |  1. User Level. This settings.xml file provides configuration for a single user,
     |                 and is normally provided in ${user.home}/.m2/settings.xml.
     |
     |                 NOTE: This location can be overridden with the CLI option:
     |
     |                 -s /path/to/user/settings.xml
     |
     |  2. Global Level. This settings.xml file provides configuration for all Maven
     |                 users on a machine (assuming they're all using the same Maven
     |                 installation). It's normally provided in
     |                 ${maven.conf}/settings.xml.
     |
     |                 NOTE: This location can be overridden with the CLI option:
     |
     |                 -gs /path/to/global/settings.xml
     |
     | The sections in this sample file are intended to give you a running start at
     | getting the most out of your Maven installation. Where appropriate, the default
     | values (values used when the setting is not specified) are provided.
     |
     |-->
    <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
      <!-- localRepository
       | The path to the local repository maven will use to store artifacts.
       |
       | Default: ${user.home}/.m2/repository
      <localRepository>/var/jenkins_home/repository</localRepository>
      -->
      <localRepository>/var/jenkins_home/repository</localRepository>
    
      <!-- interactiveMode
       | This will determine whether maven prompts you when it needs input. If set to false,
       | maven will use a sensible default value, perhaps based on some other setting, for
       | the parameter in question.
       |
       | Default: true
      <interactiveMode>true</interactiveMode>
      -->
    
      <!-- offline
       | Determines whether maven should attempt to connect to the network when executing a build.
       | This will have an effect on artifact downloads, artifact deployment, and others.
       |
       | Default: false
      <offline>false</offline>
      -->
    
      <!-- pluginGroups
       | This is a list of additional group identifiers that will be searched when resolving plugins by their prefix, i.e.
       | when invoking a command line like "mvn prefix:goal". Maven will automatically add the group identifiers
       | "org.apache.maven.plugins" and "org.codehaus.mojo" if these are not already contained in the list.
       |-->
      <pluginGroups>
        <!-- pluginGroup
         | Specifies a further group identifier to use for plugin lookup.
        <pluginGroup>com.your.plugins</pluginGroup>
        -->
      </pluginGroups>
    
      <!-- proxies
       | This is a list of proxies which can be used on this machine to connect to the network.
       | Unless otherwise specified (by system property or command-line switch), the first proxy
       | specification in this list marked as active will be used.
       |-->
      <proxies>
        <!-- proxy
         | Specification for one proxy, to be used in connecting to the network.
         |
        <proxy>
          <id>optional</id>
          <active>true</active>
          <protocol>http</protocol>
          <username>proxyuser</username>
          <password>proxypass</password>
          <host>proxy.host.net</host>
          <port>80</port>
          <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
        </proxy>
        -->
      </proxies>
    
      <!-- servers
       | This is a list of authentication profiles, keyed by the server-id used within the system.
       | Authentication profiles can be used whenever maven must make a connection to a remote server.
       |-->
      <servers>
        <!-- server
         | Specifies the authentication information to use when connecting to a particular server, identified by
         | a unique name within the system (referred to by the 'id' attribute below).
         |
         | NOTE: You should either specify username/password OR privateKey/passphrase, since these pairings are
         |       used together.
         |
        <server>
          <id>deploymentRepo</id>
          <username>repouser</username>
          <password>repopwd</password>
        </server>
        -->
    
        <!-- Another sample, using keys to authenticate.
        <server>
          <id>siteServer</id>
          <privateKey>/path/to/private/key</privateKey>
          <passphrase>optional; leave empty if not used.</passphrase>
        </server>
        -->
        <server>
          <id>maven-public</id>
          <username>admin</username>
          <password>123456</password>
        </server>
 
        <server>
          <id>maven-releases</id>
          <username>admin</username>
          <password>123456</password>
        </server>

        <server>
          <id>maven-snapshots</id>
          <username>admin</username>
          <password>123456</password>
        </server>
      </servers>
    
      <!-- mirrors
       | This is a list of mirrors to be used in downloading artifacts from remote repositories.
       |
       | It works like this: a POM may declare a repository to use in resolving certain artifacts.
       | However, this repository may have problems with heavy traffic at times, so people have mirrored
       | it to several places.
       |
       | That repository definition will have a unique id, so we can create a mirror reference for that
       | repository, to be used as an alternate download site. The mirror site will be the preferred
       | server for that repository.
       |-->
      <mirrors>
        <!-- mirror
         | Specifies a repository mirror site to use instead of a given repository. The repository that
         | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
         | for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
         |
        <mirror>
          <id>mirrorId</id>
          <mirrorOf>repositoryId</mirrorOf>
          <name>Human Readable Name for this Mirror.</name>
          <url>http://my.repository.com/repo/path</url>
        </mirror>
         -->
      
        <!-- mirror
        <mirror>
          <id>alimaven</id>
          <name>aliyun maven</name>
          <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
          <mirrorOf>central</mirrorOf>        
        </mirror>
         -->
        
        <mirror>
          <id>maven-public</id>
          <mirrorOf>*</mirrorOf>
          <name>maven-public</name>
          <url>http://10.0.0.19:30100/nexus/repository/maven-public/</url>
        </mirror>
      </mirrors>
    
      <!-- profiles
       | This is a list of profiles which can be activated in a variety of ways, and which can modify
       | the build process. Profiles provided in the settings.xml are intended to provide local machine-
       | specific paths and repository locations which allow the build to work in the local environment.
       |
       | For example, if you have an integration testing plugin - like cactus - that needs to know where
       | your Tomcat instance is installed, you can provide a variable here such that the variable is
       | dereferenced during the build process to configure the cactus plugin.
       |
       | As noted above, profiles can be activated in a variety of ways. One way - the activeProfiles
       | section of this document (settings.xml) - will be discussed later. Another way essentially
       | relies on the detection of a system property, either matching a particular value for the property,
       | or merely testing its existence. Profiles can also be activated by JDK version prefix, where a
       | value of '1.4' might activate a profile when the build is executed on a JDK version of '1.4.2_07'.
       | Finally, the list of active profiles can be specified directly from the command line.
       |
       | NOTE: For profiles defined in the settings.xml, you are restricted to specifying only artifact
       |       repositories, plugin repositories, and free-form properties to be used as configuration
       |       variables for plugins in the POM.
       |
       |-->
      <profiles>
        <!-- profile
         | Specifies a set of introductions to the build process, to be activated using one or more of the
         | mechanisms described above. For inheritance purposes, and to activate profiles via <activatedProfiles/>
         | or the command line, profiles have to have an ID that is unique.
         |
         | An encouraged best practice for profile identification is to use a consistent naming convention
         | for profiles, such as 'env-dev', 'env-test', 'env-production', 'user-jdcasey', 'user-brett', etc.
         | This will make it more intuitive to understand what the set of introduced profiles is attempting
         | to accomplish, particularly when you only have a list of profile id's for debug.
         |
         | This profile example uses the JDK version to trigger activation, and provides a JDK-specific repo.
        <profile>
          <id>jdk-1.4</id>
    
          <activation>
            <jdk>1.4</jdk>
          </activation>
    
          <repositories>
            <repository>
              <id>jdk14</id>
              <name>Repository for JDK 1.4 builds</name>
              <url>http://www.myhost.com/maven/jdk14</url>
              <layout>default</layout>
              <snapshotPolicy>always</snapshotPolicy>
            </repository>
          </repositories>
        </profile>
        -->
    
        <!--
         | Here is another profile, activated by the system property 'target-env' with a value of 'dev',
         | which provides a specific path to the Tomcat instance. To use this, your plugin configuration
         | might hypothetically look like:
         |
         | ...
         | <plugin>
         |   <groupId>org.myco.myplugins</groupId>
         |   <artifactId>myplugin</artifactId>
         |
         |   <configuration>
         |     <tomcatLocation>${tomcatPath}</tomcatLocation>
         |   </configuration>
         | </plugin>
         | ...
         |
         | NOTE: If you just wanted to inject this configuration whenever someone set 'target-env' to
         |       anything, you could just leave off the <value/> inside the activation-property.
         |
        <profile>
          <id>env-dev</id>
    
          <activation>
            <property>
              <name>target-env</name>
              <value>dev</value>
            </property>
          </activation>
    
          <properties>
            <tomcatPath>/path/to/tomcat/instance</tomcatPath>
          </properties>
        </profile>
        -->
      </profiles>
    
      <!-- activeProfiles
       | List of profiles that are active for all builds.
       |
      <activeProfiles>
        <activeProfile>alwaysActiveProfile</activeProfile>
        <activeProfile>anotherAlwaysActiveProfile</activeProfile>
      </activeProfiles>
      -->
    </settings>

cat jenkins-deploy.yaml

apiVersion: apps/v1
kind: Deployment 
metadata: 
  name: jenkins
  namespace: devops
spec: 
  selector:
    matchLabels:
      name: jenkins
  template: 
    metadata: 
      labels: 
        name: jenkins
    spec: 
      serviceAccountName: jenkins
      containers: 
      - name: jenkins 
        image: jenkinsci/blueocean:master
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 1000m
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 1Gi
        env:
        - name: "JENKINS_OPTS"
          value: "--prefix=/jenkins"
        ports: 
        - containerPort: 8080
        volumeMounts:
        - name: jenkins
          mountPath: "/var/jenkins_home"
        - name: dockersock
          mountPath: "/var/run/docker.sock"
        - name: dockercli
          mountPath: "/usr/bin/docker"
        - name: dockerdaemon
          mountPath: /etc/docker/daemon.json
          subPath: daemon.json
        - name: mvn-config
          mountPath: /usr/local/maven/apache-maven-3.5.4/conf/settings.xml
          subPath: settings.xml
      volumes:
      - name: jenkins
        persistentVolumeClaim:
          claimName: jenkins-pvc
      - name: dockersock
        hostPath:
          path: /var/run/docker.sock
      - name: dockercli
        hostPath:
          path: /usr/bin/docker
      - name: mvn-config
        configMap:
          name: jenkins-configmap
          items:
          - key: settings.xml
            path: settings.xml
      - name: dockerdaemon
        configMap:
          name: jenkins-configmap
          items:
          - key: daemon.json
            path: daemon.json
      imagePullSecrets:
      - name: harbor-secret
      nodeSelector:
        app: devops

2.5、jenkins系统配置

配置jenkins url
07h9

配置gitlab
07h10

配置docker registry
07h11