记忆角落

  • {{ item.name }}
  • ELK-k8s部署ELK集群
  • 系统架构
  • 创建StorageClass和Namespace
  • 创建StorageClass
  • sc.yaml
  • 创建Namespace
  • ns.yaml
  • 创建证书
  • 部署ES集群
  • elasticsearch.configmap.yaml
  • elasticsearch.pvc.yaml
  • elasticsearch.statefulset.yaml
  • elasticsearch.service.yaml
  • 设置ES集群密码
  • 部署 kibana
  • kibana.yaml
  • 部署 logstash
  • logstash.yaml
  • 部署 filebeat
  • 首页
  • 关于
  • 归档
  • 邻居
  • 捐赠
  • 速查表

ELK-k8s部署ELK集群

  • 郭良俊只狗
  • 2023-09-13
  • 0

ELK-k8s部署ELK集群

系统架构

image-20230913154432607

因测试or生产集群模式都是3master，2node，因此采用基本上2工作节点上部署。

创建StorageClass和Namespace

创建StorageClass

sc.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: logging-local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

因为使用是本地磁盘，需要在node节点创建目录：mkdir -p /data/elk/es

创建Namespace

ns.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: logging
  labels:
    app: elk

创建证书

docker run --name es-certutil -i -w /tmp elasticsearch:7.17.9 /bin/sh -c  \
    "elasticsearch-certutil ca --out /tmp/es-ca.p12 --pass '' && \
    elasticsearch-certutil cert --name security-master --dns \
    security-master --ca /tmp/es-ca.p12 --pass '' --ca-pass '' --out /tmp/elastic-certificates.p12"

docker cp es-certutil:/tmp/elastic-certificates.p12 ./

docker rm -f es-certutil

生成 ssl 认证要使用的 secret elastic-certificates：

# 生成 ssl 认证要使用的 secret elastic-certificates
sudo kubectl -n logging create secret generic elastic-certificates --from-file=./elastic-certificates.p12

部署ES集群

elasticsearch.configmap.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: logging
  name: elk-elasticsearch-config
  labels:
    app: elk-elasticsearch
data:
  elasticsearch.yml: |-
    cluster.name: elk-elasticsearch-cluster
    # 发现节点的地址，discovery.seed_hosts的值应包括所有master候选节点
    discovery.seed_hosts: elk-elasticsearch
    # 初始化集群时，ES从中选出master节点
    # 对应metadata.name名称加编号，编号从0开始
    # cluster.initial_master_nodes: elk-elasticsearch-0,elk-elasticsearch-1,elk-elasticsearch-2,elk-elasticsearch-3,elk-elasticsearch-4
    cluster.initial_master_nodes: elk-elasticsearch-0,elk-elasticsearch-1
    discovery.zen.minimum_master_nodes: 2

    http.cors.enabled: true
    http.cors.allow-origin: "*"
    network.host: 0.0.0.0

    node.master: true
    node.data: true
    node.ingest: true
    ingest.geoip.downloader.enabled: false

    xpack.security.enabled: true
    xpack.monitoring.collection.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/elastic-certificates.p12

elasticsearch.pvc.yaml

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: logging-local-storage-elk-es-pv-0
  namespace: logging
  labels:
    name: logging-local-storage-elk-es-pv-0
spec:
  capacity:
    storage: 250Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: logging-local-storage
  local:
    path: /data/elk/es
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - k8s-n01

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: logging-local-storage-elk-es-pv-1
  namespace: logging
  labels:
    name: logging-local-storage-elk-es-pv-1
spec:
  capacity:
    storage: 250Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: logging-local-storage
  local:
    path: /data/elk/es
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - k8s-n02

#---
#apiVersion: v1
#kind: PersistentVolume
#metadata:
#  name: logging-local-storage-elk-es-pv-2
#  namespace: logging
#  labels:
#    name: logging-local-storage-elk-es-pv-2
#spec:
#  capacity:
#    storage: 250Gi
#  accessModes:
#    - ReadWriteOnce
#  persistentVolumeReclaimPolicy: Retain
#  storageClassName: logging-local-storage
#  local:
#    path: /data/elk/es
#  nodeAffinity:
#    required:
#      nodeSelectorTerms:
#        - matchExpressions:
#            - key: kubernetes.io/hostname
#              operator: In
#              values:
#                - k8s-m01
#
#---
#apiVersion: v1
#kind: PersistentVolume
#metadata:
#  name: logging-local-storage-elk-es-pv-3
#  namespace: logging
#  labels:
#    name: logging-local-storage-elk-es-pv-3
#spec:
#  capacity:
#    storage: 250Gi
#  accessModes:
#    - ReadWriteOnce
#  persistentVolumeReclaimPolicy: Retain
#  storageClassName: logging-local-storage
#  local:
#    path: /data/elk/es
#  nodeAffinity:
#    required:
#      nodeSelectorTerms:
#        - matchExpressions:
#            - key: kubernetes.io/hostname
#              operator: In
#              values:
#                - k8s-m02
#
#---
#apiVersion: v1
#kind: PersistentVolume
#metadata:
#  name: logging-local-storage-elk-es-pv-4
#  namespace: logging
#  labels:
#    name: logging-local-storage-elk-es-pv-4
#spec:
#  capacity:
#    storage: 250Gi
#  accessModes:
#    - ReadWriteOnce
#  persistentVolumeReclaimPolicy: Retain
#  storageClassName: logging-local-storage
#  local:
#    path: /data/elk/es
#  nodeAffinity:
#    required:
#      nodeSelectorTerms:
#        - matchExpressions:
#            - key: kubernetes.io/hostname
#              operator: In
#              values:
#                - k8s-m03

elasticsearch.statefulset.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elk-elasticsearch
  namespace: logging
  labels:
    app: elk-elasticsearch
spec:
  serviceName: elk-elasticsearch
  replicas: 2
  selector:
    matchLabels:
      app: elk-elasticsearch
  template:
    metadata:
      labels:
        app: elk-elasticsearch
    spec:
      initContainers:
      - name: elasticsearch-logging-init
        image: 10.194.24.53/k8s-component/alpine:3.18
        command: [ "/sbin/sysctl", "-w", "vm.max_map_count=262144" ]
        securityContext:
          privileged: true

      - name: elasticsearch-volume-init
        image: 10.194.24.53/k8s-component/alpine:3.18
        command:
          - chmod
          - -R
          - "777"
          - /usr/share/elasticsearch/data/
        volumeMounts:
          - name: elasticsearch-storage-data
            mountPath: /usr/share/elasticsearch/data/

      containers:
        - name: elasticsearch
          image: 10.194.24.53/k8s-component/elk/elasticsearch:7.17.9
          command:
            - /bin/bash
            - '-c'
            - |-
              #!/bin/bash
              ulimit -l unlimited
              echo ulimit
              exec su elasticsearch /usr/local/bin/docker-entrypoint.sh

          resources:
            limits: #最大
              memory: 6144Mi
            requests:
              memory: 5120Mi
          ports:
            - containerPort: 9200
              name: http
              protocol: TCP

            - containerPort: 9300
              name: transport
              protocol: TCP
          env:
            - name: node.name
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

            - name: ES_JAVA_OPTS
              value: "-Xms5120m -Xmx5120m"

            - name: TZ
              value: Asia/Shanghai

            - name: TIME_ZONE
              value: Asia/Shanghai

          volumeMounts:
            - name: elastic-certificates
              readOnly: true
              mountPath: /usr/share/elasticsearch/config/elastic-certificates.p12
              subPath: elastic-certificates.p12

            - name: config
              mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
              readOnly: true
              subPath: elasticsearch.yml

            - name: elasticsearch-storage-data
              mountPath: /usr/share/elasticsearch/data

            - name: localtime
              mountPath: /etc/localtime


      volumes:
        - name: elastic-certificates
          secret:
            secretName: elastic-certificates

        - name: localtime
          hostPath:
            path: /etc/localtime

        - name: config
          configMap:
            name: elk-elasticsearch-config

      tolerations:
        - operator: Exists

  volumeClaimTemplates:
    - metadata:
        name: elasticsearch-storage-data
        namespace: logging
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "logging-local-storage"
        resources:
          requests:
            storage: 250Gi

elasticsearch.service.yaml

---
apiVersion: v1
kind: Service
metadata:
  namespace: logging
  name: elk-elasticsearch
  labels:
    app: elk-elasticsearch-svc
spec:
  type: NodePort
  ports:
    - port: 9200
      name: http
      nodePort: 39200
    - port: 9300
      name: transport
      nodePort: 39201
  selector:
    app: elk-elasticsearch

设置ES集群密码

#执行：自动生成随机密码并打印终端
# kubectl -n logging exec -it $(kubectl -n logging get pods | grep elk-elasticsearch | sed -n 1p | awk '{print $1}') -- bin/elasticsearch-setup-passwords auto -b
# 自行配置密码
kubectl -n logging exec -it $(kubectl -n logging get pods | grep elk-elasticsearch | sed -n 1p | awk '{print $1}') -- bin/elasticsearch-setup-passwords  interactive


Changed password for user apm_system
PASSWORD apm_system = xxx

Changed password for user kibana_system
PASSWORD kibana_system = xxx

Changed password for user kibana
PASSWORD kibana = xxx

Changed password for user logstash_system
PASSWORD logstash_system = xxx

Changed password for user beats_system
PASSWORD beats_system = xxx

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = xxx

Changed password for user elastic
PASSWORD elastic = xxx

部署 kibana

kibana.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: logging
  name: kibana-config
  labels:
    app: kibana
data:
  kibana.yml: |-
    server.host: 0.0.0.0
    i18n.locale: "zh-CN"
    monitoring.ui.container.elasticsearch.enabled: true
    elasticsearch:
      hosts: http://elk-elasticsearch:9200
      username: elastic
      password: xxx

    xpack.encryptedSavedObjects.encryptionKey: min-32-byte-long-strong-encryption-key
---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logging-kibana-data-pvc
  namespace: logging
  labels:
    app: logging-kibana-data-pvc
spec:
  storageClassName: k8s-data
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: logging
  labels:
    k8s-app: kibana
    srv: srv-kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana
  template:
    metadata:
      labels:
        k8s-app: kibana
    spec:
      containers:
      - name: kibana
        image: 10.194.24.53/k8s-component/elk/kibana:7.17.9
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: SERVER_PUBLICBASEURL
            value: "http://0.0.0.0:5601"
          - name: TZ
            value: Asia/Shanghai
          - name: TIME_ZONE
            value: Asia/Shanghai

        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
        volumeMounts:

          - name: kibana-config
            mountPath: /usr/share/kibana/config/kibana.yml
            readOnly: true
            subPath: kibana.yml

          - mountPath: /etc/localtime
            name: localtime

          - name: kibana-data
            mountPath: /usr/share/kibana/plugins
      volumes:
        - name: kibana-config
          configMap:
            name: kibana-config

        - hostPath:
            path: /etc/localtime
          name: localtime

        - name: kibana-data
          persistentVolumeClaim:
            claimName: logging-kibana-data-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: kibana-svc
  namespace: logging
  labels:
    k8s-app: kibana
    kubernetes.io/name: "Kibana"
    svc: kibana-svc
spec:
  type: ClusterIP
  ports:
    - port: 5601
      protocol: TCP
      targetPort: ui
  selector:
    k8s-app: kibana

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  namespace: logging
  name: kibana-ing
spec:
  rules:
    - host: kibana.xxx.com
      http:
        paths:
          - backend:
              service:
                name: kibana-svc
                port:
                  number: 5601
            pathType: Prefix
            path: /

部署 logstash

因后续会采集Nginx日志，也提前存放了GeoLite2-City.mmdbip数据库

logstash.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-configmap
  namespace: logging
  labels:
    app: logstash
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
    config.reload.automatic: true
    config.reload.interval: 300s
  logstash.conf: |-
    input {
      beats {
        type => "beat-nginx-logs"
        port => 5044
        codec => json
        client_inactivity_timeout => 3600
      }
    }

    filter {
      if [type] == "beat-nginx-logs" {
        # nginx 日志
        if [xff] != ""{
          geoip {
            target => "geoip"
            source => "xff"
            database => "/usr/share/logstash/GeoLite2-City.mmdb"
            add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
            add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
            # 去掉显示 geoip 显示的多余信息
            remove_field => ["[geoip][latitude]", "[geoip][longitude]", "[geoip][country_code]", "[geoip][country_code2]", "[geoip][country_code3]", "[geoip][timezone]", "[geoip][continent_code]", "[geoip][region_code]"]
          }
        }else{
          geoip {
            target => "geoip"
            source => "client_ip"
            database => "/usr/share/logstash/GeoLite2-City.mmdb"
            add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
            add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
            # 去掉显示 geoip 显示的多余信息
            remove_field => ["[geoip][latitude]", "[geoip][longitude]", "[geoip][country_code]", "[geoip][country_code2]", "[geoip][country_code3]", "[geoip][timezone]", "[geoip][continent_code]", "[geoip][region_code]"]
          }
        }

        mutate {
          convert => [ "size", "integer" ]
          convert => [ "status", "integer" ]
          convert => [ "responsetime", "float" ]
          convert => [ "upstreamtime", "float" ]
          convert => [ "[geoip][coordinates]", "float" ]
          # 过滤 filebeat 没用的字段,这里过滤的字段要考虑好输出到es的，否则过滤了就没法做判断
          # remove_field => [ "ecs","agent","host","cloud","@version","input","logs_type" ]
          remove_field => [ "ecs","agent","cloud","@version","input" ]
        }
        # 根据 http_user_agent来自动处理区分用户客户端系统与版本
        useragent {
          source => "http_user_agent"
          target => "ua"
          # 过滤useragent没用的字段
          remove_field => [ "[ua][minor]","[ua][major]","[ua][build]","[ua][patch]","[ua][os_minor]","[ua][os_major]" ]
        }
      }
    }

    output {
      if [type] == "beat-nginx-logs" {
        elasticsearch {
          hosts => "elk-elasticsearch:9200"
          user => "elastic"
          password => "xxx"
          index => "logstash-nginx-log-%{+yyyy.MM.dd}"
        }
      } else {
        stdout { }
      }
    }

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elk-logstash
  namespace: logging
spec:
  selector:
    matchLabels:
      app: logstash
  replicas: 2
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
        - name: logstash
          image: 10.194.24.53/k8s-component/elk/logstash:7.17.9
          ports:
            - containerPort: 5044
              name: nginx-port

            - containerPort: 5045
              name: tcp-test-port

          volumeMounts:
            - name: config-volume
              mountPath: /usr/share/logstash/config/logstash.yml
              readOnly: true
              subPath: logstash.yml

            - name: logstash-pipeline-volume
              mountPath: /usr/share/logstash/pipeline/logstash.conf
              readOnly: true
              subPath: logstash.conf

            - mountPath: /etc/localtime
              name: localtime

            - mountPath: /usr/share/logstash/GeoLite2-City.mmdb
              name: geolite2

          env:
            - name: TZ
              value: Asia/Shanghai

            - name: TIME_ZONE
              value: Asia/Shanghai

      volumes:
        - name: config-volume
          configMap:
            name: logstash-configmap
            items:
              - key: logstash.yml
                path: logstash.yml

        - name: logstash-pipeline-volume
          configMap:
            name: logstash-configmap
            items:
              - key: logstash.conf
                path: logstash.conf

        - hostPath:
            path: /etc/localtime
          name: localtime

        - hostPath:
            path: /glj_k8s/GeoLite2-City.mmdb
          name: geolite2

---
kind: Service
apiVersion: v1
metadata:
  name: logstash-svc
  namespace: logging
  labels:
    app: logstash
spec:
  selector:
    app: logstash
  type: NodePort
  ports:
    - name: nginx-port
      port: 5044
      nodePort: 35044
    - name: tcp-test-port
      port: 5045
      nodePort: 35045

部署 filebeat

后续加入采集容器日志，暂时只是部署Nginx服务器采集日志分析

参考：https://blog.csdn.net/weixin_43334786/article/details/122435701

© 2012 - 2024 记忆角落 网站统计    由又拍云CDN赞助 提供云储存服务
Theme by Wing 粤ICP备14056850号