Jenkins构建多架构镜像手札

前言

很久没用jenkins 流水线构建部署了，最近公司也出现amd与arm的双架构服务器（信创），为了后续便于适配就有了过程吧。

部署过程

1.解决buildkit拉取和上传镜像对证书的不信任

将harbor 的ca文件添加到buildkit的镜像中，为了解决证书的不信任.

Dockerfile 文件:

FROM alpine:3.20 AS prepare
RUN apk add --no-cache ca-certificates git
COPY ca.crt /usr/local/share/ca-certificates/harbor.crt
RUN update-ca-certificates && mkdir -p /root/.docker
COPY config.json /root/.docker/config.json

FROM moby/buildkit:v0.24.0
COPY --from=prepare /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=prepare /root/.docker /root/.docker

构建多架构镜像,运行命令：

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t reg-hub.gzeport.com/cicd/moby/buildkit:v0.24.0 \
  --push .

2.初始化K8S worker节点支持多架构

binfmt-daemonset.yaml文件

# binfmt-daemonset.yaml
# ls -l /proc/sys/fs/binfmt_misc
# 执行一次即可
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: binfmt-setup
  namespace: devops
spec:
  selector:
    matchLabels:
      name: binfmt-setup
  template:
    metadata:
      labels:
        name: binfmt-setup
    spec:
      tolerations:
        - operator: Exists
      hostPID: true
      containers:
        - name: binfmt-installer
          image: tonistiigi/binfmt:latest
          command: ["binfmt", "--install", "all"]
          securityContext:
            privileged: true

kubectl apply -f binfmt-daemonset.yaml 所有的pod都Completed以后可以把这个daemonset删除

3.创建buildkit的服务端

buildkit-deployment.yaml文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: buildkitd
  namespace: devops
  labels:
    app: buildkitd
spec:
  replicas: 1
  selector:
    matchLabels:
      app: buildkitd
  template:
    metadata:
      labels:
        app: buildkitd
    spec:
      hostAliases:
        - ip: "192.168.111.156"
          hostnames:
            - "reg-hub.gzeport.com"
      containers:
        - name: buildkitd
          image: reg-hub.gzeport.com/cicd/moby/buildkit:v0.24.0
          imagePullPolicy: Always
          args:
            - --addr
            - unix:///run/buildkit/buildkitd.sock
            - --addr
            - tcp://0.0.0.0:1234
            - --oci-worker-gc
            - --oci-worker-gc-keepstorage=53687091200
          env:
            - name: BUILDKIT_LOG_FORMAT
              value: json
          ports:
            - containerPort: 1234
          readinessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          livenessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2"
              memory: "4Gi"
---

apiVersion: v1
kind: Service
metadata:
  name: buildkitd
  namespace: devops
  labels:
    app: buildkitd
spec:
  selector:
    app: buildkitd
  ports:
    - name: tcp
      port: 1234
      targetPort: 1234
      protocol: TCP