kube-Prometheus监控zookeeper

因生产存在一套zookeeper需要进行监控，但因为版本只有3.4.11，秉承没挂我就躺平不升级原则就不能碰。

又因查看了 zookeeper 文档发送从3.6.0版本才开始有原生支持开放指标接口供Prometheus采集。地址：https://zookeeper.apache.org/doc/r3.6.0/zookeeperMonitor.html

那么我生产集群原则上是不支持上面的配置滴，那么就只能使用zookeeper-exporter进行采集咯

zookeeper-exporter部署监控

开源项目：https://github.com/dabealu/zookeeper-exporter

镜像地址：https://docker.aityp.com/image/docker.io/dabealu/zookeeper-exporter:v0.1.13

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-zookeeper-dyck-exporter
  namespace: monitoring
  labels:
    app: external-zookeeper-dyck-exporter
    type: zookeeper-exporter
    system: dyck
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: external-zookeeper-dyck-exporter
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: external-zookeeper-dyck-exporter
        type: zookeeper-exporter
        system: dyck
    spec:
      containers:
        - image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/dabealu/zookeeper-exporter:v0.1.13
          imagePullPolicy: IfNotPresent
          name: zookeeper-exporter
          args:
            - -zk-hosts
            - 10.196.68.43:2181,10.196.68.44:2181,10.196.68.45:2181
            - -location
            - /metrics
          env:
            - name: TZ
              value: Asia/Shanghai
          ports:
            - containerPort: 9141
              protocol: TCP
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 64Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File

Service

apiVersion: v1
kind: Service
metadata:
  name: external-zookeeper-dyck-exporter
  labels:
    app: zookeeper
    type: zookeeper-exporter
    system: dyck
  namespace: monitoring
spec:
  type: ClusterIP
  ports:
    - name: http-metrics
      port: 9141
      protocol: TCP
      targetPort: 9141
  selector:
    app: external-zookeeper-dyck-exporter
    type: zookeeper-exporter
    system: dyck

ServiceMonitor

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: external-zookeeper-exporter
  labels:
    app: zookeeper-exporter
    release: prometheus-community
  namespace: monitoring
spec:
  endpoints:
    - port: http-metrics
      interval: 30s
      path: /metrics
      relabelings:
        - sourceLabels: [ __meta_kubernetes_service_label_app, __meta_kubernetes_service_label_system ]
          targetLabel: job
          separator: "-"  # 用连字符连接两个标签值
          action: replace
      metricRelabelings:
        - sourceLabels: [ zk_host ]
          targetLabel: instance
          action: replace
          regex: "(.+):.+"
          replacement: "$1"
  selector:
    matchLabels:
      app: zookeeper
      type: zookeeper-exporter
  namespaceSelector:
    matchNames:
      - monitoring

注意了我这边设置metricRelabelings是因为一个Pod监控了多个zk_host服务，需要进行区分替换instance，不然grafana dashboard界面不好看了。

执行以上配置完成后，在Prometheus的target页面能看到指标成功采集

grafana dashboard配置

grafana id: 11442，地址：https://grafana.com/grafana/dashboards/11442-zookeeper-exporter-dabealu/