容器云项目-日志采集实践(一)

一、实践说明

前面已经部署了elk日志工具链,现在需要采集日志进行存储查询,这里以nginx访问日志为例,通过filebeat来采集,再k8s中采集应用日志一般有两种做法,一种是应用与filebeat封装在同一pod,共享同一日志目录,另一种先将应用日志持久化到本地,然后使用filebeat去挂载应用日志,前者采集工具与应用紧耦合配置相对简单,后者松耦合配置相对麻烦,根据不同情况不同处理,本文采用前者

二、nginx访问日志采集测试

采集日志需要部署相应的工具,包括filebeat、nginx、cerbro

2.1、filebeat/nginx部署

cat filebeat-nginx.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: beat-config
data:
  filebeat.yml: |-
    #=========================== Filebeat inputs =============================
    filebeat.inputs:
    # 收集nginx日志
    - type: log
      enabled: true
      paths:
        - /usr/local/nginx/logs/access.log
    # 日志是json开启这个
      json.keys_under_root: true
      json.overwrite_keys: true
      json.add_error_key: true

    #-------------------------- Redis output ------------------------------
    output.redis:
      hosts: ["redis.monitor:6379"]   #输出到redis的机器
      password: "Qazwsx!23"
      key: "nginx_logs"   #redis中日志数据的key值? 
      db: 0
      timeout: 5

  nginx.conf: |-   
    user  nginx;
    worker_processes  auto;

    error_log  /var/log/nginx/error.log notice;
    pid        /var/run/nginx.pid;


    events {
        worker_connections  1024;
    }


    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;
        log_format aka_logs
            '{"@timestamp":"$time_iso8601",'
            '"host":"$hostname",'
            '"server_ip":"$server_addr",'
            '"client_ip":"$remote_addr",'
            '"xff":"$http_x_forwarded_for",'
            '"domain":"$host",'
            '"url":"$uri",'
            '"referer":"$http_referer",'
            '"args":"$args",'
            '"upstreamtime":"$upstream_response_time",'
            '"responsetime":"$request_time",'
            '"request_method":"$request_method",'
            '"status":"$status",'
            '"size":"$body_bytes_sent",'
            '"request_body":"$request_body",'
            '"request_length":"$request_length",'
            '"protocol":"$server_protocol",'
            '"upstreamhost":"$upstream_addr",'
            '"file_dir":"$request_filename",'
            '"http_user_agent":"$http_user_agent"'
            '}';
        access_log  /var/log/nginx/access.log  aka_logs;

        sendfile        on;
        #tcp_nopush     on;

        keepalive_timeout  65;

        #gzip  on;

        include /etc/nginx/conf.d/*.conf;
        }

cat filebeat-nginx-deploy.yaml

apiVersion: apps/v1
kind: Deployment 
metadata: 
  name: nginx-gfs
spec: 
  replicas: 1
  selector:
    matchLabels:
      name: nginx
  template: 
    metadata: 
      labels: 
        name: nginx 
    spec: 
      containers: 
        - name: nginx 
          image: nginx 
          imagePullPolicy: IfNotPresent
          ports: 
            - containerPort: 80
          volumeMounts:
            - name: nginx-logs
              mountPath: /var/log/nginx
            - name: nginx-conf
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
            - name: tz-config
              mountPath: /etc/localtime
        - name: filebeat
          image: elastic/filebeat:7.9.1
          imagePullPolicy: IfNotPresent
          args: [
            "-c","/etc/filebeat.yml",
            "-e",
          ]
          resources:
            limits:
              memory: 500Mi
            requests:
              cpu: 100m
              memory: 100Mi
          securityContext:
            runAsUser: 0
          volumeMounts:
            - name: filebeat-conf
              mountPath: /etc/filebeat.yml
              subPath: filebeat.yml
            - name: nginx-logs
              mountPath: /usr/local/nginx/logs
            - name: tz-config
              mountPath: /etc/localtime
      volumes:
        - name: nginx-logs
          emptyDir: {}
        - name: nginx-conf
          configMap:
            name: beat-config
            items:
            - key: nginx.conf
              path: nginx.conf
        - name: filebeat-conf
          configMap:
            name: beat-config
            items:
            - key: filebeat.yml
              path: filebeat.yml
        - name: tz-config
          hostPath:
            path: /usr/share/zoneinfo/Asia/Shanghai
---
kind: Service
apiVersion: v1
metadata:
  name: nginx-svc
  labels:
    glusterfs: nginx-service
    deploy-nginx: support
  annotations:
    description: Exposes Nginx Service
spec:
  type: NodePort
  selector:     
    name: nginx    
  ports:
    - name: nginx
      port: 80
      nodePort: 30006
      targetPort: 80

ps:上面采集nginx日志的时候对nginx日志格式处理成json格式,这也是为了方便后面logstash好处理以及kibana展示日志信息更为清晰

2.2、日志展示

如果上面部署成功,那么通过展示面板始可以查询到相应的日志信息的,下面通过三种方式来查看存入的日志数据
cerebro展示

node18上操作
cat cerebro.conf

server {
         listen       30004;
         server_name  xj.local;
         charset utf-8;
         location / {
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header HOST $http_host;
         proxy_set_header X-NginX-Proxy true;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Proto $scheme;
         proxy_pass http://10.0.0.19:9200;
         proxy_redirect off;
         client_max_body_size 35m;
      }
    }

node19上操作
docker run -id -p 9200:9000 --name cerebro-xj lmenezes/cerebro:0.8.4部署cerebro

本地访问
http://xj.local:30004访问
09pn1

上面的地址是elasticsearch的地址(这里的试通过nodeport方式访问)
09pn2

cerebro一般只用于简单验证日志是否进入elasticsearch,如果要具体查看日志还是需要通过kibana
kibana展示
http://xj.local:30008访问
09pn3

grafana展示
首先需要再grafana上配置elasticsearch
09pn4

09pn5

http://xj.local:30010访问
导入模板AKA ES Nginx Logs

打开对应的面板
09pn6

上面的模板还可以展示热点地图,但需要再grafana上安装对应的插件,这可以通过环境变量实现
GF_INSTALL_PLUGIN: grafana-piechart-panel,grafana-worldmap-panel

同时需要将地图默认的地址替换成国内的,否则地图无法展示

/var/lib/grafana/plugins/grafana-worldmap-panel   

sed -i 's/https:\/\/cartodb-basemaps{s}.global.ssl.fastly.net\/light_all\/{z}\/{x}\/{y}.png/http:\/\/{s}.basemaps.cartocdn.com\/light_all\/{z}\/{x}\/{y}.png/' module.js  \
module.js.map   

sed -i 's/https:\/\/cartodb-basemaps-{s}.global.ssl.fastly.net\/dark_all\/{z}\/{x}\/{y}.png/http:\/\/{s}.basemaps.cartocdn.com\/dark_all\/{z}\/{x}\/{y}.png/' module.js \
module.js.map