kube-prometheus-stack 中grafana-sc-datasources 错误记录

前言

最近arm架构部署的二进制k8s，使用kube-prometheus-stack部署全家桶，其他业务正常情况下，就grafana下 grafana-sc-dashboard和grafana-sc-datasources出现异常，因为是二进制部署CA证书都说自签名的，导致pod（k8s-sidecar）出现CERTIFICATE_VERIFY_FAILED错误，数据源与大盘图都无法进行正常加载。搞了一下午也没修复，后续也是看出helm的对应chart 源码才解决。

{"time": "2025-08-26T07:11:48.733288+00:00", "level": "ERROR", "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_dashboard%3D1&timeoutSeconds=60&watch=True (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:992)')))\n"} {"time": "2025-08-26T07:11:48.733467+00:00", "level": "ERROR", "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_dashboard%3D1&timeoutSeconds=60&watch=True (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:992)')))\n"}

解决

通过查看helm chart源码，发现直接添加此参数就行。

{{- if .Values.sidecar.skipTlsVerify }}
- name: SKIP_TLS_VERIFY
value: "{{ .Values.sidecar.skipTlsVerify }}"
{{- end }}

在value.yaml添加再出现跑即可解决：

grafana:
  sidecar:
   skipTlsVerify: true

错误，不生效记录

下午通过各种验证方法都不行其中包括设置env也不生效，升级版本也不行

无效1：

sidecar:
  args:
    - --insecure-skip-tls-verify=true

无效2：

sidecar:
  args:
    - --kube-ca-file=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    - --kube-token-file=/var/run/secrets/kubernetes.io/serviceaccount/token

无效3：

grafana:
  sidecar:
    dashboards:
      enabled: true
      env:
        - name: SKIP_TLS_VERIFY
          value: "true"
    extraArgs:
      - --insecure-skip-tls-verify

无效4：

sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard
      extraEnv:
        - name: KUBERNETES_SKIP_TLS_VERIFY
          value: "true"
        - name: KUBERNETES_SERVICE_HOST
          value: "10.96.0.1"
        - name: KUBERNETES_SERVICE_PORT
          value: "443"
    datasources:
      enabled: true
      label: grafana_datasource
      extraEnv:
        - name: KUBERNETES_SKIP_TLS_VERIFY
          value: "true"
        - name: KUBERNETES_SERVICE_HOST
          value: "10.96.0.1"
        - name: KUBERNETES_SERVICE_PORT
          value: "443"