鲲鹏ARM64服务器适配k8s组件, 宿主机中间件

鲲鹏ARM64服务器适配k8s组件, 宿主机中间件

以下记录在鲲鹏920服务器， openEuler 24.03 (LTS) 操作系统适配imom依赖的k8s组件, 宿主机中间件过程。

1. k8s适配

1.1 创建k8s离线制品包

创建 01-manifest-kubernetes-kubesphere.yaml 文件:

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
  name: sample
spec:
  arches:
  - arm64
  operatingSystems: []
  kubernetesDistributions:
  - type: kubernetes
    version: v1.22.12
  components:
    helm:
      version: v3.14.3
    cni:
      version: v1.2.0
    etcd:
      version: v3.5.13
    containerRuntimes:
    - type: docker
      version: 24.0.9
    calicoctl:
      version: v3.27.4
    crictl:
      version: v1.29.0
  images:
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.5
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.22.12
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.22.12
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.22.12
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.22.12
  - registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.8.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.22.20
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.27.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.27.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/typha:v3.27.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/snapshot-controller:v4.0.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-installer:v3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-apiserver:v3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-console:v3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.22.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node-exporter:v1.3.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager:v2.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-tenant-sidecar:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager-operator:v2.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.11.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-state-metrics:v2.6.0
  - swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/mirrorgooglecontainers/defaultbackend-arm64:1.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/alertmanager:v0.23.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/openldap:1.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus:v2.39.1
#  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-config-reloader:v0.55.1
  - swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/prometheus-operator/prometheus-config-reloader:v0.60.1-linuxarm64
#  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-operator:v0.55.1
  - swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/prometheus-operator/prometheus-operator:v0.60.1-linuxarm64

使用以下的命令创建出arm64的离线包01-manifest-kubernetes-kubesphere.tar.gz :

./kk artifact export -m 01-manifest-kubernetes-kubesphere.yaml -o 01-manifest-kubernetes-kubesphere.tar.gz

1.2 初始化k8s宿主机依赖环境

1.2.1 在线安装基础软件

dnf install -y   gpm-libs   gssproxy   keyutils   libbasicobjects   libcollection   libevent   libini_config   libnfsidmap   libpath_utils   libref_array   libtirpc   libverto-libevent   net-tools   nfs-utils   perl   perl-Carp   perl-Encode   perl-Exporter   perl-File-Path   perl-File-Temp   perl-Filter   perl-Getopt-Long   perl-HTTP-Tiny   perl-PathTools   perl-Pod-Escapes   perl-Pod-Perldoc   perl-Pod-Simple   perl-Pod-Usage   perl-Scalar-List-Utils   perl-Socket   perl-Storable   perl-Text-ParseWords   perl-Time-HiRes   perl-Time-Local   perl-constant   perl-libs   perl-macros   perl-parent   perl-podlators   perl-threads   perl-threads-shared   quota   quota-nls   rpcbind   tcp_wrappers   tree   unzip

1.2.2 在线安装k8s依赖软件

dnf install -y   conntrack-tools   ipvsadm   libnetfilter_conntrack   libnetfilter_cthelper   libnetfilter_cttimeout   libnetfilter_queue   openssl   openssl-libs   readline   socat

1.2.3 安装kk

tar -xvf kubekey-v3.1.8-linux-arm64.tar.gz
./kk

1.2.4 安装docker

cat install_docker.sh

#!/bin/sh
set -e

DOCKER_TAR_PATH=./docker-24.0.9.tgz
DOCKER_SERVICE_PATH=./docker.service
DOCKER_COMPOSE_PATH=./docker-compose
# 环境变量传入
DOCKER_REGISTRY_ADDRESS="${ANSIBLE_DEPLOY_HOST_IP:-192.168.1.104}:5000"

if which docker > /dev/null 2>&1; then
    echo "Docker 已经安装并可以在系统路径中找到。"
    exit 0
fi

echo " ==> DOCKER_REGISTRY_ADDRESS: ${DOCKER_REGISTRY_ADDRESS}"

echo '解压tar包...'
tar -xvf ${DOCKER_TAR_PATH}
echo '将docker目录移到/usr/bin目录下...'
cp docker/* /usr/bin/

echo '添加http白名单'
mkdir -p /etc/docker
if [[ ! -f /etc/docker/daemon.json ]]; then
echo "当前部署机器ip 加入到insecure-registries"
cat > /etc/docker/daemon.json << EOF
{
  "insecure-registries": ["${DOCKER_REGISTRY_ADDRESS}"]
}
EOF
fi


echo '将docker.service 移到/etc/systemd/system/ 目录...'
cp ${DOCKER_SERVICE_PATH} /etc/systemd/system/
echo '添加文件权限...'
chmod +x /etc/systemd/system/docker.service
echo '重新加载配置文件...'
systemctl daemon-reload
echo '启动docker...'
systemctl start docker
echo '设置开机自启...'
systemctl enable docker.service
echo 'docker安装成功...'
docker -v

echo '将docker-compose移到/usr/bin目录下...'
cp ${DOCKER_COMPOSE_PATH} /usr/bin/
chmod 755 /usr/bin/docker-compose
echo 'docker-compose安装成功...'
docker-compose  -v

执行安装:

./install_docker.sh

1.2.5 安装docker-registry

cat docker-compose.yml, 其中172.38.3.12 为当前宿主机的IP:

version: '3.8'

services:
  registry-ui:
    image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/joxit/docker-registry-ui:main-linuxarm64
    restart: always
    ports:
      - 5001:80
    environment:
      - SINGLE_REGISTRY=true
      - REGISTRY_TITLE=Docker Registry UI
      - DELETE_IMAGES=true
      - SHOW_CONTENT_DIGEST=true
      - NGINX_PROXY_PASS_URL=http://172.38.3.12:5000
      - SHOW_CATALOG_NB_TAGS=true
      - CATALOG_MIN_BRANCHES=1
      - CATALOG_MAX_BRANCHES=1
      - TAGLIST_PAGE_SIZE=100
      - REGISTRY_SECURED=false
      - CATALOG_ELEMENTS_LIMIT=1000
    container_name: registry-ui

  registry-server:
    image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/registry:2-linuxarm64
    restart: always
    ports:
      - 5000:5000
    environment:
#      REGISTRY_HTTP_HEADERS_Access-Control-Allow-Origin: '*'
#      REGISTRY_HTTP_HEADERS_Access-Control-Allow-Methods: '[HEAD,GET,OPTIONS,DELETE]'
#      REGISTRY_HTTP_HEADERS_Access-Control-Allow-Credentials: '[true]'
#      REGISTRY_HTTP_HEADERS_Access-Control-Allow-Headers: '[Authorization,Accept,Cache-Control]'
#      REGISTRY_HTTP_HEADERS_Access-Control-Expose-Headers: '[Docker-Content-Digest]'
#      REGISTRY_STORAGE_DELETE_ENABLED: 'true'
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry

    volumes:
      - /home/docker_registry/image_registry:/var/lib/registry
    container_name: registry-server

启动docker-registry:

docker-compose  up -d
# 查看服务是否正常: docker ps

1.2.6 安装nfs服务

cat /etc/exports

/data/nfs/ *(insecure,rw,sync,no_root_squash)

启动nfs-server, 放行防火墙

mkdir -p /data/nfs
service nfs-server restart
systemctl enable nfs-server
firewall-cmd --add-service=nfs --zone=public --permanent
firewall-cmd --add-service=rpc-bind --zone=public --permanent

1.2.7 在线安装mysql数据库服务

卸载原有的数据库, 安装mysql8.0 yum源:

rpm -qa | grep -E 'mysql|mariadb'
rpm -e --nodeps  mariadb-connector-c-3.3.8-1.oe2403.aarch64
cd /etc/yum.repos.d/
wget https://dev.mysql.com/get/mysql80-community-release-el8-3.noarch.rpm
rpm -ivh mysql80-community-release-el8-3.noarch.rpm

安装mysql 8.0 数据库:

yum install -y mysql-community-server --nogpgcheck

配置数据库:

cat /etc/my.cnf

[mysqld]
#lower_case_table_names=1
# 虚拟机搭建一定要挂载单独数据盘，不要用系统盘，要检查数据盘是否有初始化和挂载，可以扩容
skip-name-resolve
explicit_defaults_for_timestamp
max_allowed_packet=16M
character-set-server=UTF8
slow_query_log=1
long_query_time=3.0
# 最大连接数，默认151，最大设置3000
max_connections=1000
interactive_timeout=60
# lower_case_table_names 参数详解：
# 0: 区分大小写
# 1: 不区分大小写
# MySQL在Linux下数据库名、表名、列名、别名大小写规则是这样的：
#   1) 数据库名与表名是严格区分大小写的；
#   2) 表的别名是严格区分大小写的；
#   3) 列名与列的别名在所有情况下均忽略大小写；
#   4) 变量名是严格区分大小写的；
lower_case_table_names=1
# sql_mode: 删除 ONLY_FULL_GROUP_BY
sql_mode=IGNORE_SPACE,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION


datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

启动数据库:

systemctl start mysqld
systemctl enable mysqld
systemctl status mysqld
firewall-cmd --zone=public --add-port=3306/tcp --permanent
firewall-cmd --reload
firewall-cmd --list-all

配置数据库用户:

grep 'temporary password' /var/log/mysqld.log
# 进入mysql
mysql -u root -p
ALTER USER 'root'@'localhost' IDENTIFIED BY '123.nhdTaVMSAC';
set global validate_password.policy=0;
ALTER USER 'root'@'localhost' IDENTIFIED BY 'nhdTaVMSAC';
create user 'root'@'%' IDENTIFIED WITH mysql_native_password BY 'nhdTaVMSAC';


# 配置远程连接

# ALTER USER 'root'@'%' IDENTIFIED BY 'abcabC!9090';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%';

FLUSH PRIVILEGES;

1.3 安装k8s与kubesphere

创建k8s安装配置
cat kk-config.yaml

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sie-kubesphere
spec:
  arches:
  - arm64
  hosts:
    - name: node1
      address: "172.38.3.12"
      internalAddress: "172.38.3.12"
      user: "root"
      password: "your_password"
      arch: arm64
    - name: node2
      address: "172.38.3.27"
      internalAddress: "172.38.3.27"
      user: "root"
      password: "your_password"
      arch: arm64
    - name: node3
      address: "172.38.3.28"
      internalAddress: "172.38.3.28"
      user: "root"
      password: "your_password"
      arch: arm64
  roleGroups:
    etcd:
      - node1
    master:
      - node1
    control-plane:
      - node1
    worker:
      - node2
      - node3
    registry:
      - node1

  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers
    # internalLoadbalancer: haproxy

    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.22.12
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: docker
  etcd:
    type: kubekey
  network:
    plugin: calico
    calico:
      version: v3.23.2
  kubePodsCIDR: 10.233.64.0/18
  kubeServiceCIDR: 10.233.0.0/18
  ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
  multusCNI:
    enabled: false
  registry:
    # 如需使用 kk 部署 harbor, 可将该参数设置为 harbor，不设置该参数且需使用 kk 创建容器镜像仓库，将默认使用 docker registry。
    # type: harbor
    # 如使用 kk 部署的 harbor 或其他需要登录的仓库，需设置对应仓库的 auths，如使用 kk 创建默认的 docker registry 仓库，则无需配置 auths 参数。
    auths:
      "172.38.3.12:5000":
        # 部署 harbor 时需指定 harbor 帐号密码
        # username: admin
        # password: Harbor12345
        skipTLSVerify: true
        plainHTTP: true
    # 设置集群部署时使用的私有仓库地址。如果您已有可用的镜像仓库，请替换为您的实际镜像仓库地址。
    # 如果离线包中为原始 dockerhub 镜像（即 manifest 文件中的镜像地址为 docker.io/***），可以将该参数设置为 dockerhub.kubekey.local/ks, 表示将镜像全部推送至名为 ks 的 harbor 项目中。
    privateRegistry: "172.38.3.12:5000"
    # 如果构建离线包时 Kubernetes 镜像使用的是阿里云仓库镜像，需配置该参数。如果使用 dockerhub 镜像，则无需配置此参数。
    namespaceOverride: "kubesphereio"
    registryMirrors: []
    insecureRegistries: ["192.168.0.0/16","10.0.0.0/8","172.16.0.0/12","172.38.3.12:5000"]
    insecure: true
  addons:
    - name: nfs-client
      namespace: kube-system
      sources:
        chart:
          local:
            path: /root/ansible_install/packages/k8s/assets/addons/nfs-client-provisioner-1.2.8.tgz
        valuesFile: /root/ansible_install/packages/k8s/assets/addons/sie-nfs-client-values.yaml

---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    version: v3.4.1
spec:
  persistence:
    storageClass: ""
  authentication:
    jwtSecret: ""
  zone: ""
  local_registry: ""
  namespace_override: ""
  # dev_tag: ""
  etcd:
    monitoring: false
    endpointIps: localhost
    port: 2379
    tlsEnable: true
  common:
    core:
      console:
        enableMultiLogin: true
        port: 30880
        type: NodePort
    # apiserver:
    #  resources: {}
    # controllerManager:
    #  resources: {}
    redis:
      enabled: false
      volumeSize: 2Gi
    openldap:
      enabled: false
      volumeSize: 2Gi
    minio:
      volumeSize: 20Gi
    monitoring:
      # type: external
      endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
      GPUMonitoring:
        enabled: false
    gpu:
      kinds:
        - resourceName: "nvidia.com/gpu"
          resourceType: "GPU"
          default: true
    es:
      # master:
      #   volumeSize: 4Gi
      #   replicas: 1
      #   resources: {}
      # data:
      #   volumeSize: 20Gi
      #   replicas: 1
      #   resources: {}
      logMaxAge: 7
      elkPrefix: logstash
      basicAuth:
        enabled: false
        username: ""
        password: ""
      externalElasticsearchHost: ""
      externalElasticsearchPort: ""
  alerting:
    enabled: false
    # thanosruler:
    #   replicas: 1
    #   resources: {}
  auditing:
    enabled: false
    # operator:
    #   resources: {}
    # webhook:
    #   resources: {}
  devops:
    enabled: false
    # resources: {}
    jenkinsMemoryLim: 8Gi
    jenkinsMemoryReq: 4Gi
    jenkinsVolumeSize: 8Gi
  events:
    enabled: false
    # operator:
    #   resources: {}
    # exporter:
    #   resources: {}
    # ruler:
    #   enabled: true
    #   replicas: 2
    #   resources: {}
  logging:
    enabled: false
    logsidecar:
      enabled: true
      replicas: 2
      # resources: {}
  metrics_server:
    enabled: false
  monitoring:
    storageClass: ""
    node_exporter:
      port: 9100
      # resources: {}
    # kube_rbac_proxy:
    #   resources: {}
    # kube_state_metrics:
    #   resources: {}
    # prometheus:
    #   replicas: 1
    #   volumeSize: 20Gi
    #   resources: {}
    #   operator:
    #     resources: {}
    # alertmanager:
    #   replicas: 1
    #   resources: {}
    # notification_manager:
    #   resources: {}
    #   operator:
    #     resources: {}
    #   proxy:
    #     resources: {}
    gpu:
      nvidia_dcgm_exporter:
        enabled: false
        # resources: {}
  multicluster:
    clusterRole: none
  network:
    networkpolicy:
      enabled: false
    ippool:
      type: none
    topology:
      type: none
  openpitrix:
    store:
      enabled: false
  servicemesh:
    enabled: false
    istio:
      components:
        ingressGateways:
          - name: istio-ingressgateway
            enabled: false
        cni:
          enabled: false
  edgeruntime:
    enabled: false
    kubeedge:
      enabled: false
      cloudCore:
        cloudHub:
          advertiseAddress:
            - ""
        service:
          cloudhubNodePort: "30000"
          cloudhubQuicNodePort: "30001"
          cloudhubHttpsNodePort: "30002"
          cloudstreamNodePort: "30003"
          tunnelNodePort: "30004"
        # resources: {}
        # hostNetWork: false
      iptables-manager:
        enabled: true
        mode: "external"
        # resources: {}
      # edgeService:
      #   resources: {}
  terminal:
    timeout: 600

使用以下的命令解压离线包，安装:

# 解压离线包, 推送镜像
./kk artifact image push -f kk-config.yaml -a 01-manifest-kubernetes-kubesphere.tar.gz
# 安装k8s集群
./kk create cluster -f kk-config.yaml -a 01-manifest-kubernetes-kubesphere.tar.gz --with-local-storage

修复镜像无法拉取的问题:

# 修复 defaultbackend 镜像
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/mirrorgooglecontainers/defaultbackend-arm64:1.4  172.38.3.12:5000/ddn-k8s/docker.io/mirrorgooglecontainers/defaultbackend-arm64:1.4
docker push 172.38.3.12:5000/ddn-k8s/docker.io/mirrorgooglecontainers/defaultbackend-arm64:1.4
# 修复 prometheus-config-reloader 镜像
docker tag  172.38.3.12:5000/ddn-k8s/quay.io/prometheus-operator/prometheus-config-reloader:v0.60.1-linuxarm64  172.38.3.12:5000/kubesphereio/prometheus-config-reloader:v0.55.1
docker push 172.38.3.12:5000/kubesphereio/prometheus-config-reloader:v0.55.1
# 修复 prometheus-operator 镜像
docker tag  172.38.3.12:5000/ddn-k8s/quay.io/prometheus-operator/prometheus-operator:v0.60.1-linuxarm64  172.38.3.12:5000/kubesphereio/prometheus-operator:v0.55.1
docker push 172.38.3.12:5000/kubesphereio/prometheus-operator:v0.55.1

至此, k8s集群已安装完毕。

2. k8s 业务中间件适配

2.1 适配imom中间件

拉取arm64版本的中间件镜像到本地仓库上:

   # 1. 处理 NFS Provisioner 镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.0-linuxarm64
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.0-linuxarm64 172.38.3.12:5000/ddn-k8s/registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.0-linuxarm64
docker push 172.38.3.12:5000/ddn-k8s/registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.0-linuxarm64
   # 2. 处理 Redis 7.0.12 镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/bitnami/redis:7.0.12-debian-11-r0-linuxarm64
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/bitnami/redis:7.0.12-debian-11-r0-linuxarm64 172.38.3.12:5000/ddn-k8s/docker.io/bitnami/redis:7.0.12-debian-11-r0-linuxarm64
docker push 172.38.3.12:5000/ddn-k8s/docker.io/bitnami/redis:7.0.12-debian-11-r0-linuxarm64
   # 3. 处理 Redis 8.0.2 镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/bitnami/redis:8.0.2-linuxarm64
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/bitnami/redis:8.0.2-linuxarm64 172.38.3.12:5000/ddn-k8s/docker.io/bitnami/redis:8.0.2-linuxarm64
docker push 172.38.3.12:5000/ddn-k8s/docker.io/bitnami/redis:8.0.2-linuxarm64
   # 4. 处理 Redis Sentinel 镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/bitnamisecure/redis-sentinel:latest-linuxarm64
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/bitnamisecure/redis-sentinel:latest-linuxarm64 172.38.3.12:5000/ddn-k8s/docker.io/bitnamisecure/redis-sentinel:latest-linuxarm64
docker push 172.38.3.12:5000/ddn-k8s/docker.io/bitnamisecure/redis-sentinel:latest-linuxarm64
   # 5. 处理 PowerJob Server 镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/powerjob/powerjob-server:4.3.9-linuxarm64
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/powerjob/powerjob-server:4.3.9-linuxarm64 172.38.3.12:5000/ddn-k8s/docker.io/powerjob/powerjob-server:4.3.9-linuxarm64
docker push 172.38.3.12:5000/ddn-k8s/docker.io/powerjob/powerjob-server:4.3.9-linuxarm64
   # 6. 处理 MinIO 镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/bitnami/minio:2025.3.12-debian-12-r0-linuxarm64
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/bitnami/minio:2025.3.12-debian-12-r0-linuxarm64 172.38.3.12:5000/ddn-k8s/docker.io/bitnami/minio:2025.3.12-debian-12-r0-linuxarm64
docker push 172.38.3.12:5000/ddn-k8s/docker.io/bitnami/minio:2025.3.12-debian-12-r0-linuxarm64
   # 7. 处理 KKFileView 镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/pi4k8s/kkfileview:4.4.0-arm64-linuxarm64
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/pi4k8s/kkfileview:4.4.0-arm64-linuxarm64 172.38.3.12:5000/ddn-k8s/docker.io/pi4k8s/kkfileview:4.4.0-arm64-linuxarm64
docker push 172.38.3.12:5000/ddn-k8s/docker.io/pi4k8s/kkfileview:4.4.0-arm64-linuxarm64
   # 8. 处理 Nexus3 镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/sonatypecommunity/nexus3:3.73.0-linuxarm64
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/sonatypecommunity/nexus3:3.73.0-linuxarm64 172.38.3.12:5000/ddn-k8s/docker.io/sonatypecommunity/nexus3:3.73.0-linuxarm64
docker push 172.38.3.12:5000/ddn-k8s/docker.io/sonatypecommunity/nexus3:3.73.0-linuxarm64
   # 9. 处理 HAProxy 镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/haproxy:3.0.12-alpine-linuxarm64
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/haproxy:3.0.12-alpine-linuxarm64 172.38.3.12:5000/ddn-k8s/docker.io/haproxy:3.0.12-alpine-linuxarm64
docker push 172.38.3.12:5000/ddn-k8s/docker.io/haproxy:3.0.12-alpine-linuxarm64

编辑 ~/ansible_install/vars.env 文件内容:

export ANSIBLE_OS='openEuler2403'
export ANSIBLE_IMOM_NAMESPACE='imom'
export ANSIBLE_IMOM_COMMON_NAMESPACE='common'
export ANSIBLE_IMOM_ELK_NAMESPACE='elk'
export ANSIBLE_IMOM_DISABLE_ELK='1'
export ANSIBLE_HW_NAMESPACE='mbm'
export ANSIBLE_IMAGE_PULL_SECRETS='default-secret'
export ANSIBLE_DEFAULT_SECRETS_CONTENT='eyJhdXRocyI6eyJodHRwczovL3N3ci5jbi1zb3V0aC0xLm15aHVhd2VpY2xvdWQuY29tIjp7InVzZXJuYW1lIjoiY24tc291dGgtMUBDNERKUlA0M1haQkJESFMwMUhGTyIsInBhc3N3b3JkIjoiOGYzMGUwNjk2NmI5OWQ4ZWNlN2JhODZjOTczNjE3NGYxNjliZTk4ZDliYzEyY2RjOTgzNTU3NzZjNzAyYmQzOSIsImVtYWlsIjoiIiwiYXV0aCI6IlkyNHRjMjkxZEdndE1VQkRORVJLVWxBME0xaGFRa0pFU0ZNd01VaEdUem80WmpNd1pUQTJPVFkyWWprNVpEaGxZMlUzWW1FNE5tTTVOek0yTVRjMFpqRTJPV0psT1Roa09XSmpNVEpqWkdNNU9ETTFOVGMzTm1NM01ESmlaRE01In19fQ=='
export ANSIBLE_DEFAULT_STORAGE_CLASS='nfs-client'
export ANSIBLE_EXTERNAL_BASE_URL='http://172.38.3.12:32555'
export ANSIBLE_IMOM_GATEWAY_HOST='172.38.3.12:32556'
export ANSIBLE_ENABLE_IMOM_APPS='imom-gateway,imom-kernel'
export ANSIBLE_DOCKER_REGISTRY_ADDRESS='172.38.3.12:5000'
export ANSIBLE_NFS_ADDRESS='172.38.3.12'
export ANSIBLE_NEXUS3_ADDRESS='192.168.1.105:8081'
export ANSIBLE_MYSQL_COMMON_PORT='3306'
export ANSIBLE_MYSQL_ADDRESS='172.38.3.12'
export ANSIBLE_MYSQL_USER='root'
export ANSIBLE_MYSQL_PASSWORD='your_password'
export ANSIBLE_REDIS_ADDRESS='haproxy-redis-sv.common'
export ANSIBLE_REDIS_MASTER_ADDRESS='haproxy-redis-svc.common'
export ANSIBLE_REDIS_PASSWORD='Dme_your_password'
export ANSIBLE_REDIS_MODE='SENTINEL'
export ANSIBLE_MINIO_ADDRESS='sie-minio-broker.common'
export ANSIBLE_MINIO_USER='admin'
export ANSIBLE_MINIO_PASSWORD='Minio_your_password'
export ANSIBLE_POWERJOB_ADDRESS='powerjob-svc.common'
export ANSIBLE_IDME_ADDRESS='http://192.168.1.99:30083'
export ANSIBLE_IDME_SUB_APP_ID='rdm_e3231ab62e9d4771bd0df793f62d2282_app'
export ANSIBLE_IDME_APP_ID='e3231ab62e9d4771bd0df793f62d2282'
export ANSIBLE_IDME_MYSQL_ADDRESS='172.38.3.12'
export ANSIBLE_IDME_MYSQL_USER='root'
export ANSIBLE_IDME_MYSQL_PASSWORD='your_password'
export ANSIBLE_IDME_DEPLOY_VERSION='2.25.060.8.20250626.6'

export ANSIBLE_HW_MBM_ADDRESS='http://mbm-api-gateway.mbm:9090'
export ANSIBLE_ENABLE_MBM_APPS='mbm-api-gateway-service,mbm-mdm-service,mbm-msm-service,mbm-orche-web,mbm-mom-web'
export ANSIBLE_HW_TENANT_ID='067cfa58c8404a908780bda7e934e1b1'
export ANSIBLE_HW_ORG_NAME='赛意ARM64测试'
export ANSIBLE_HW_REDIS_ADDRESS='redis-mbm-master.mbm'
export ANSIBLE_HW_REDIS_PASSWORD='Rdis_your_password'
export ANSIBLE_HW_REDIS_MODE='CLUSTER'
export ANSIBLE_HW_MYSQL_COMMON_PORT='3306'
export ANSIBLE_HW_MYSQL_ADDRESS='172.38.3.12'
export ANSIBLE_HW_MYSQL_USER='root'
export ANSIBLE_HW_MYSQL_PASSWORD='your_password'
export ANSIBLE_HW_MINIO_ADDRESS='sie-minio-broker.common'
export ANSIBLE_HW_MINIO_USER='admin'
export ANSIBLE_HW_MINIO_PASSWORD='Minio_your_password'
export ANSIBLE_HW_GEN_QM_XDM_DB='0'
export ANSIBLE_AFFINITY_ENABLE='0'
export ANSIBLE_AFFINITY_COMMON_NODE='node2'
export ANSIBLE_AFFINITY_MBM_NODE='node2'
export ANSIBLE_AFFINITY_IMOM_NODE='node13'
export ANSIBLE_AFFINITY_TAINT_POLICY='NoSchedule'
# 添加目标机器 IP
export ANSIBLE_DEPLOY_HOST_IP='172.38.3.12'

执行以下的命令安装imom中间件:

cd /root/ansible_install/packages/k8s_app_broker

bash -x install.sh  nfs
bash -x install.sh  redis
# 先修复MYSQL_CLIENT的路径, 之后执行
bash -x install.sh  powerjob
bash -x install.sh  kkfileview
bash -x install.sh  minio

执行以下的命令安装华为mbm中间件:

cd k8s_hw_app_broker/
# 先修复镜像地址, 之后执行
bash -x install.sh  redis

若中间件无法启动，在kubesphere 界面修复镜像地址问题。