记忆角落

  • {{ item.name }}
  • Jenkins集成Kubernetes集群(Master-Slave分布式构建方案)
  • 1.Kubernetes 环境安装 Jenkins
  • 2.Jenkins 安装插件
  • 插件加速
  • 插件下载
  • 3.实现Master-Slave分布式构建方案(云配置)
  • 4.Template 模板配置
  • 构建Jenkins-Slave自定义镜像
  • 模板配置
  • maven-pv持久化yaml
  • 5.Jenkins测试
  • 首页
  • 关于
  • 归档
  • 邻居
  • 捐赠
  • 速查表

Jenkins集成Kubernetes集群(Master-Slave分布式构建方案)

  • 郭良俊只狗
  • 2023-02-28
  • 0

Jenkins集成Kubernetes集群(Master-Slave分布式构建方案)

image-20230228223656590

1.Kubernetes 环境安装 Jenkins

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins-admin
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["*"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: devops

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins-admin
subjects:
  - kind: ServiceAccount
    name: jenkins-admin
    namespace: devops
---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv-data
  namespace: devops
  labels:
    release: stable
spec:
  capacity:
    storage: 20Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: jenkins-pv-data #自定义定义存储的类名,特定类的PV只能绑定到请求该类的PVC。没有storageClassName的PV没有类，只能绑定到不请求特定类的PVC
  nfs:
    path: /data/nfs_data/jenkins_data #绑定主机的的路径
    server: 192.168.111.154 #指定nfs主机的ip地址
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc-data
  namespace: devops
  labels:
    release: stable
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 20Gi
  storageClassName: jenkins-pv-data
  selector:
    matchLabels:
      release: stable

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: devops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-server
  template:
    metadata:
      labels:
        app: jenkins-server
    spec:
      securityContext:
        fsGroup: 1000
        runAsUser: 1000
      serviceAccountName: jenkins-admin
      containers:
        - name: jenkins
          image: reg-hub.gzeport.com/gzeport/jenkins/jenkins:lts-centos7-jdk8
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              memory: "2Gi"
              cpu: "1000m"
            requests:
              memory: "500Mi"
              cpu: "500m"
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlpport
              containerPort: 50000
          env:
            - name: LIMITS_MEMORY
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
                  divisor: 1Mi
            - name: JAVA_OPTS
              value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
          livenessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home

      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
              claimName: jenkins-pvc-data
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: devops
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/path:   /
    prometheus.io/port:   '8080'
spec:
  selector:
    app: jenkins-server
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
#      nodePort: 32000
      name: httpport
    - port: 50000
      targetPort: 50000
      protocol: TCP
      name: jnlpport
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins-gzeport-com
  namespace: devops
  labels:
    app: jenkins-gzeport-com
  annotations:
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
spec:
  rules:
    - host: jenkins.gzeport.com
      http:
        paths:
          - backend:
              service:
                name: jenkins-service
                port:
                  number: 8080
            path: /
            pathType: Prefix

2.Jenkins 安装插件

插件加速

cd /var/lib/jenkins/updates
cp default.json default.json.bak
sed -i 's/http:\/\/updates.jenkinsci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json && sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json
sed -i 's/http:\/\/updates.jenkins-ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json && sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json

插件下载

暂时使用到的插件

Chinese
Role-based Authorization Strategy
Credentials Binding
kubenetes
Config File Provider
pipeline
Git
Git Parameter
Extended Choice Parameter

3.实现Master-Slave分布式构建方案(云配置)

image-20230228225518357

Dashboard > 系统管理 > 节点管理 > configureClouds

image-20230228224446877

image-20230228225839458

image-20230228230051538

这里是配置连接Kubernetes集群，启动 Jenkins Slave 代理的相关配置。

  • 名称： kubernetes
  • Kubernetes 地址： https://kubernetes.default.svc.cluster.local(默认集群内调用 k8s api 地址)
  • 禁用 HTTPS 证书检查： 勾选 (不验证https)
  • 凭据： 新增凭据—>Secret text—>Secret 设置 kubernetes 的 Token (进入 k8s dashboard 的 token 等都行)
  • Jenkins地址： http://jenkins-service.devops.svc.cluster.local:8080 (用于代理与 Jenkins 连接的地址，用的是 k8s 集群中 jenkins 服务)，我这边就是用无头服务地址
  • Jenkins 通道: jenkins-service.devops.svc.cluster.local:50000slave连接专用
  • 其他： 默认即可

4.Template 模板配置

这里配置 Jenkins Slave 在 kubernetes 集群中启动的 Pod 的配置

构建Jenkins-Slave自定义镜像

Jenkins-Master在构建Job的时候，Kubernetes会创建Jenkins-Slave的Pod来完成Job的构建

我们选择运行Jenkins-Slave的镜像为官方推荐镜像：官方镜像是 jenkins/inbound-agent（以前是 jenkinsci/jnlp-slave and jenkins/jnlp-slave），但是这个镜像里面并没有Maven环境，为了方便使用，我自定义一个新的镜像，Dockerfile文件内容：

image-20230228230619787

FROM centos:7.9.2009
MAINTAINER gzeport.com glj

USER root

# 安装 maven
ADD apache-maven-3.9.0-bin.tar.gz /usr/local/
ADD jdk-8u341-linux-x64.tar.gz /usr/local/

RUN mkdir -p /usr/share/jenkins && \
    ln -s /usr/local/apache-maven-3.9.0/bin/mvn /usr/bin/mvn && \
    ln -s /usr/local/apache-maven-3.9.0 /usr/local/maven && \
    ln -s /usr/local/jdk1.8.0_341/bin/java /usr/bin/java && \
    ln -s /usr/local/jdk1.8.0_341 /usr/local/jdk && \
    ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    yum install -y git curl && \
    yum clean all && rm -rf /var/cache/yum/* 

COPY settings.xml /usr/local/maven/conf/settings.xml
COPY kubectl /usr/bin/
COPY remoting-4.13.3.jar /usr/share/jenkins/agent.jar 
COPY jenkins-agent /usr/bin/jenkins-slave
RUN chmod +x /usr/bin/jenkins-slave && \
    chmod +x  /usr/bin/kubectl

# 设置环境变量
ENV JAVA_HOME /usr/local/jdk
ENV MAVEN_HOME /usr/local/maven
ENV PATH ${JAVA_HOME}/bin:${MAVEN_HOME}/bin:$PATH

ENTRYPOINT ["jenkins-slave"]

# docker build -t reg-hub.gzeport.com/gzeport/jenkins-slave-jdk8:v1 .
# docker push reg-hub.gzeport.com/gzeport/jenkins-slave-jdk8:v1

remoting.jar具体重命名和所放目录,要参照jenkins-slave脚本的执行命令参数来修改

remoting.jar下载地址：https://github.com/jenkinsci/remoting/tags?after=remoting-3.31

jenkins-agent 下载地址：https://github.com/jenkinsci/docker-inbound-agent

kubectl参考集群版本

settings.xml文件是maven配置，可根据公司自行替换

模板配置

推送到私有参考后，可进行模板配置

image-20230228231646716

image-20230228231840883

maven-pv持久化yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: maven-pv-data
  namespace: devops
  labels:
    release: stable
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: maven-pv-data #自定义定义存储的类名,特定类的PV只能绑定到请求该类的PVC。没有storageClassName的PV没有类，只能绑定到不请求特定类的PVC
  nfs:
    path: /data/nfs_data/maven #绑定主机的的路径
    server: 192.168.111.154 #指定nfs主机的ip地址

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: maven-pvc-data
  namespace: devops
  labels:
    release: stable
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi
  storageClassName: maven-pv-data
  selector:
    matchLabels:
      release: stable

5.Jenkins测试

一个demo测试

#!/usr/bin/env groovy
//公共
// 项目
def GIT_URL = "http://gitlab-svc.devops.svc.cluster.local/guoliangjun/gzeport-gitlab-demo.git"
// 认证
def Git_AUTH = "fdbad6cf-2ea2-4e82-82b2-55921dcd9c73"

//创建一个Pod的模板，label为jenkins-slave
podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
        containerTemplate(
                name: 'jnlp',
                image: "reg-hub.gzeport.com/gzeport/jenkins-slave:jdk8"
        )
])
{

    //参数列表
    parameters{
        gitParameter branch: '', branchFilter: '.*', defaultValue: 'origin/master', description: '选择发布的分支', listSize: '10', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'ASCENDING', tagFilter: '*', type: 'PT_BRANCH'
    }

    //引用jenkins-slave的pod模块来构建Jenkins-Slave的pod
    node("jenkins-slave"){
        stage('拉取代码'){
            checkout([$class: 'GitSCM',
                      branches: [[name: "${params.Branch}"]],
                      extensions: [], userRemoteConfigs: [[credentialsId: "${Git_AUTH}", url: "${GIT_URL}"]]])
        }
    }
}

image-20230228232549341

image-20230228232742277

参考：

https://blog.csdn.net/wangshui898/article/details/112306310

https://blog.csdn.net/heian_99/article/details/124986269

https://cloud.tencent.com/developer/article/2138129

© 2012 - 2024 记忆角落 网站统计    由又拍云CDN赞助 提供云储存服务
Theme by Wing 粤ICP备14056850号