记忆角落

  • {{ item.name }}
  • 二进制或者kubeadm搭建的k8s部署kube-prometheus解决默认告警问题
  • Watchdog
  • KubeControllerManagerDown
  • KubeSchedulerDown
  • KubeProxyDown
  • 首页
  • 关于
  • 归档
  • 邻居
  • 捐赠
  • 速查表

二进制或者kubeadm搭建的k8s部署kube-prometheus解决默认告警问题

  • 郭良俊只狗
  • 2023-06-19
  • 0

二进制或者kubeadm搭建的k8s部署kube-prometheus解决默认告警问题

二进制或者kubeadm搭建的k8s在kube-promethesu部署完成之后，会出现KubeControllerManagerDown、KubeProxyDown、KubeSchedulerDown、Watchdog四个告警

img

Watchdog

查看Watchdog的告警描述：This is an alert meant to ensure that the entire alerting pipeline is functional. This alert is always firing, therefore it should always be firing in Alertmanager and always fire against a receiver. There are integrations with various notification mechanisms that send a notification when this alert is not firing. For example the "DeadMansSnitch" integration in PagerDuty.，简单来说，这个告警是为了保证告警功能可以正常使用，如果这个告警消失，那就表示告警系统出了问题。如果不需要这个功能，可以修改prometheus目录中的相关文件kubePrometheus-prometheusRule.yaml，注释或删除相关的内容。

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-prometheus
    app.kubernetes.io/part-of: kube-prometheus
    prometheus: k8s
    role: alert-rules
  name: kube-prometheus-rules
  namespace: monitoring
spec:
  groups:
  - name: general.rules
    ...
#    - alert: Watchdog
#      annotations:
#        description: |
#          This is an alert meant to ensure that the entire alerting pipeline is functional.
#          This alert is always firing, therefore it should always be firing in Alertmanager
#          and always fire against a receiver. There are integrations with various notification
#          mechanisms that send a notification when this alert is not firing. For example the
#          "DeadMansSnitch" integration in PagerDuty.
#        runbook_url: https://runbooks.prometheus-operator.dev/runbooks/general/watchdog
#        summary: An alert that should always be firing to certify that Alertmanager
#          is working properly.
#      expr: vector(1)
#      labels:
#        severity: none
  - name: node-network
    ...
  - name: kube-prometheus-node-recording.rules
    ...
  - name: kube-prometheus-general.rules
    ...

修改完成后，使用kubectl apply -f prometheus/kubePrometheus-prometheusRule.yaml更新配置，稍等片刻，就可以看到告警Watchdog已从告警列表消失。

KubeControllerManagerDown

KubeControllerManagerDown告警描述：KubeControllerManager has disappeared from Prometheus target discovery.，翻译：KubeControllerManager已经从Prometheus目标发现中消失。

原因是这个文件kubernetesControlPlane-serviceMonitorKubeControllerManager.yaml定义了一个ServiceMonitor资源对象，最后一段从名称空间kube-system中匹配包含labelapp.kubernetes.io/name: kube-controller-manager的svc

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: kube-controller-manager
    app.kubernetes.io/part-of: kube-prometheus
  name: kube-controller-manager
  namespace: monitoring
spec:
  ...
  jobLabel: app.kubernetes.io/name
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-controller-manager

创建一个新目录other，在这个目录中新建一个yaml文件kube-system-Service.yaml，文件内容如下：

apiVersion: v1
kind: Service
metadata:
  name: kube-controller-manager
  namespace: kube-system
  labels:
    # 这里定义的标签要保证跟ServiceMonitor资源对象中selector的标签一致
    app.kubernetes.io/name: kube-controller-manager
spec:
  selector:
    # 这里选择的标签要保证是kube-controller-manager Pod中定义的标签之一
    component: kube-controller-manager
  ports:
    - port: 10257
      targetPort: 10257
      name: https-metrics

修改完成后，使用kubectl apply -f other/创建kube-controller-manager Service资源，稍等片刻，就可以看到Service已经创建成功

 ➜  ~ kubectl get svc -n kube-system
NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                        AGE
kube-controller-manager   ClusterIP   10.98.64.108   <none>        10257/TCP                      110s
kube-dns                  ClusterIP   10.96.0.10     <none>        53/UDP,53/TCP,9153/TCP         11d
kubelet                   ClusterIP   None           <none>        10250/TCP,10255/TCP,4194/TCP   10d

然而，告警并没有消失，但是告警内容发生了改变：PrometheusTargetMissing，这时在prometheus Web查看targets，跟kube-controller-manager相关的三个全都不在线，

img

原因是kube-controller-manager默认监听地址是127.0.0.1，导致无法外部访问。在k8s集群每个master节点修改/etc/kubernetes/manifests/kube-controller-manager.yaml，将- --bind-address=127.0.0.1改为- --bind-address=0.0.0.0，保存退出，对应节点的pod会自动重启（如果没有pod没有自动重启，重启kubelet服务或手动重启）。等待pod重启完成，刷新Web，可以看到target都已在线，告警KubeControllerManagerDown已从告警列表消失。

img

KubeSchedulerDown

KubeSchedulerDown同理，在kube-system-Service.yaml后面追加kube-scheduler Service。

apiVersion: v1
kind: Service
metadata:
  name: kube-controller-manager
  namespace: kube-system
  labels:
    app.kubernetes.io/name: kube-controller-manager
spec:
  selector:
    component: kube-controller-manager
  ports:
    - port: 10257
      targetPort: 10257
      name: https-metrics
---
apiVersion: v1
kind: Service
metadata:
  name: kube-scheduler
  namespace: kube-system
  labels:
    app.kubernetes.io/name: kube-scheduler
spec:
  selector:
    component: kube-scheduler
  ports:
    - port: 10259
      targetPort: 10259
      name: https-metrics

使用kubectl apply -f other/创建kube-scheduler Service资源。

在k8s集群每个master节点修改/etc/kubernetes/manifests/kube-scheduler.yaml，将- --bind-address=127.0.0.1改为- --bind-address=0.0.0.0，保存退出，对应节点的pod会自动重启（如果没有pod没有自动重启，重启kubelet服务或手动重启）。重启完成，刷新Web，可以看到target已在线，告警KubeSchedulerDown已从告警列表消失。

img

KubeProxyDown

KubeProxyDown跟KubeControllerManagerDown和KubeSchedulerDown类似，不同的是kube-prometheus并没有为kube-proxy创建一个serviceMonitor对象。

首先参考其它serviceMonitor资源对象的yaml文件在kubernetesControlPlane目录下新建一个yaml文件kubernetesControlPlane-serviceMonitorKubeProxy.yaml，定义一个名为kube-proxy的ServiceMonitor对象，文件内容如下：

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: kube-proxy
    app.kubernetes.io/part-of: kube-prometheus
  name: kube-proxy
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: http-metrics
    scheme: http
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: app.kubernetes.io/name
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-proxy

使用kubectl apply -f kubernetesControlPlane/kubernetesControlPlane-serviceMonitorKubeProxy.yaml创建资源。

继续在kube-system-Service.yaml后面追加kube-proxy Service。

apiVersion: v1
kind: Service
metadata:
  name: kube-controller-manager
  namespace: kube-system
  labels:
    app.kubernetes.io/name: kube-controller-manager
spec:
  selector:
    component: kube-controller-manager
  ports:
    - port: 10257
      targetPort: 10257
      name: https-metrics
---
apiVersion: v1
kind: Service
metadata:
  name: kube-scheduler
  namespace: kube-system
  labels:
    app.kubernetes.io/name: kube-scheduler
spec:
  selector:
    component: kube-scheduler
  ports:
    - port: 10259
      targetPort: 10259
      name: https-metrics
---
apiVersion: v1
kind: Service
metadata:
  name: kube-proxy
  namespace: kube-system
  labels:
    app.kubernetes.io/name: kube-proxy
spec:
  selector:
    k8s-app: kube-proxy
  ports:
    - port: 10249
      targetPort: 10249
      name: http-metrics

使用kubectl apply -f other/创建kube-proxy Service资源。

kube-proxy也需要修改监听地址，在任意一台master节点修改kube-proxy ConfigMapkubectl edit cm/kube-proxy -n kube-system将metricsBindAddress: ""改为metricsBindAddress: "0.0.0.0:10249"，重启Pod。重启完成，稍等片刻，就可以在Web看到kube-proxy的target都已经在线，告警KubeProxyDown已从告警列表消失。

img

参考：https://blog.tearth.cn/index.php/archives/37/

© 2012 - 2024 记忆角落 网站统计    由又拍云CDN赞助 提供云储存服务
Theme by Wing 粤ICP备14056850号