企业级别部署 Harbor 私有镜像仓库

Posted 2025-03-13 Updated 13 days ago

By Administrator

110~141 min read

一、Harbor 简介

1. 什么是 harbor

Harbor是一个开源的企业级Docker Registry管理项目，由VMware公司开源。

Harbor提供了比Docker官方公共镜像仓库更为丰富和安全的功能，尤其适合企业环境使用。以下是Harbor的一些关键特性：

权限管理（RBAC）：通过基于角色的访问控制，确保只有授权用户才能访问特定的镜像仓库。

LDAP集成：支持与LDAP服务集成，便于在企业环境中管理和认证用户身份。

日志审计：提供详细的日志记录功能，帮助企业进行操作跟踪和安全审计。

管理界面：提供直观的Web管理界面，方便用户进行镜像仓库的管理和维护。

自我注册：允许用户自行注册到Harbor，以便更好地管理自己的镜像和配置信息。

镜像复制：支持跨存储库的镜像复制，便于在不同位置部署和同步镜像。

漏洞扫描：新版本的Harbor增加了扫描镜像中漏洞的功能，并将镜像标记为受信任，增强了安全性。

HTTPS支持：支持通过HTTPS协议来保护数据传输的安全性。

2. 组件的功能和作用

Harbor是一个复杂的系统，它由多个组件构成，每个组件都有其特定的功能和作用。以下是Harbor主要组件的详细描述：

Docker Registry:

功能：存储和分发Docker镜像。

作用：Registry是Harbor的核心组件，它提供了镜像的存储、检索、分发等功能。在Harbor中，Registry负责处理镜像的上传、下载和管理请求。

Database:

功能：存储Harbor系统中的数据，包括用户信息、镜像元数据、权限配置等。

作用：数据库是Harbor的信息存储中心，确保了数据的持久化和快速查询。

Web UI (User Interface):

功能：提供图形化的界面，方便用户进行操作和管理。

作用：通过Web UI，用户可以浏览镜像仓库、管理项目、配置权限和使用Harbor的其它功能。

API Server:

功能：提供RESTful API接口，允许程序化访问和控制Harbor。

作用：API Server使得其他软件和工具可以与Harbor集成，并对其进行编程控制。

Advanced Logging & Auditing:

功能：记录操作日志，提供审计跟踪功能。

作用：这个组件帮助企业监控Harbor的使用情况，并满足合规性要求。

LDAP Service:

功能：提供用户身份验证和授权服务。

作用：通过与现有的LDAP服务集成，Harbor可以方便地管理用户账户和权限，无需维护独立的用户目录。

Job Service:

功能：执行后台任务，如镜像扫描和垃圾回收。

作用：Job Service确保Harbor能够高效地处理耗时的任务，而不会干扰到前台的用户操作。

Clair Security Scanner (在较新的Harbor版本中提供):

功能：对Docker镜像进行漏洞扫描。

作用：Clair安全扫描器增强了Harbor的安全性，可以在镜像被推送到仓库之前检测潜在的安全漏洞。

Notary Service:

功能：为Docker镜像提供签名和验真服务。

作用：确保镜像的来源和完整性，提高镜像分发的安全性。

Token Service:

功能：管理访问令牌，提供用户认证。

作用：Token Service用于处理用户的登录和认证过程，生成和管理用于访问Harbor的令牌。

二、Docker-compose 方式部署

1. 准备目录

[root@k8s-master1 ~]# mkdir /usr/local/etc/harbor-tls && cd /usr/local/etc/harbor-tls

2. 签证书

[root@k8s-node01 harbor-tls]# cat script.sh 
#!/bin/bash

# 生成 CA 证书
echo "正在生成 CA 证书..."
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=CN/ST=Beijing/L=Beijing/O=Example CA/OU=CA/CN=My Root CA" -key ca.key -out ca.crt
echo "✅ CA 证书已生成: ca.crt"

# 获取域名
while true; do
    read -e -p "请输入要签发的域名 (支持泛域名，如 *.example.com): " DOMAIN
    if [[ -n "$DOMAIN" ]]; then
        break
    else
        echo "❌ 域名不能为空，请重新输入！"
    fi
done

# 生成服务器私钥
openssl genrsa -out "${DOMAIN}.key" 4096

# 生成 CSR
openssl req -sha512 -new -subj "/C=CN/ST=Beijing/L=Beijing/O=Example/OU=Server/CN=${DOMAIN}" -key "${DOMAIN}.key" -out "${DOMAIN}.csr"

# 让用户输入多个 IP
read -e -p "请输入服务器的 IP 地址（多个 IP 用空格分隔，留空则不添加 IP）: " -a IPS

# 生成 SAN 配置文件
cat > v3.ext <<EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=127.0.0.1
DNS.2=${DOMAIN}
DNS.3=localhost
EOF

# 添加 IP 地址，每个 IP 单独占一行
ip_index=1
for ip in "${IPS[@]}"; do
    echo "IP.${ip_index}=${ip}" >> v3.ext
    ((ip_index++))
done

# 输出 v3.ext 确保格式正确
echo "✅ 生成的 v3.ext 配置文件如下:"
cat v3.ext

# 使用 CA 签发服务器证书
openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in "${DOMAIN}.csr" -out "${DOMAIN}.crt"

# 生成 PEM 格式证书
openssl x509 -inform PEM -in "${DOMAIN}.crt" -out "${DOMAIN}.cert"

# 验证证书
openssl verify -CAfile ca.crt "${DOMAIN}.crt"

# 显示证书信息
openssl x509 -in "${DOMAIN}.crt" -noout -text

[root@k8s-master1 harbor-tls]# bash script.sh

3. 安装 harbor

[root@k8s-master1 harbor]# wget https://ghfast.top/https://github.com/goharbor/harbor/releases/download/v2.5.5/harbor-offline-installer-v2.5.5.tgz

[root@k8s-master1 harbor]# tar xvf harbor-offline-installer-v2.5.5.tgz

[root@k8s-master1 harbor]# vim harbor.yml

# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: harbor.linux.cn

# http related config
#http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
#  port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /usr/local/etc/harbor-tls/harbor.linux.cn.crt
  private_key: /usr/local/etc/harbor-tls/harbor.linux.cn.key


# 安装 harbor
[root@k8s-master1 harbor]# ./prepare --with-trivy
[root@k8s-master1 harbor]# curl -SL https://ghfast.top/https://github.com/docker/compose/releases/download/v2.29.1/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
[root@k8s-master1 harbor]# chmod +x /usr/local/bin/docker-compose
[root@k8s-master1 harbor]# ./install.sh --with-trivy
[root@k8s-master1 harbor]# docker-compose ps
[root@mirros harbor]# docker-compose ps
NAME                IMAGE                                  COMMAND                  SERVICE         CREATED          STATUS                             PORTS
harbor-core         goharbor/harbor-core:v2.5.5            "/harbor/entrypoint.…"   core            18 seconds ago   Up 16 seconds (health: starting)   
harbor-db           goharbor/harbor-db:v2.5.5              "/docker-entrypoint.…"   postgresql      18 seconds ago   Up 17 seconds (health: starting)   
harbor-jobservice   goharbor/harbor-jobservice:v2.5.5      "/harbor/entrypoint.…"   jobservice      18 seconds ago   Up 16 seconds (health: starting)   
harbor-log          goharbor/harbor-log:v2.5.5             "/bin/sh -c /usr/loc…"   log             18 seconds ago   Up 17 seconds (health: starting)   127.0.0.1:1514->10514/tcp
harbor-portal       goharbor/harbor-portal:v2.5.5          "nginx -g 'daemon of…"   portal          18 seconds ago   Up 17 seconds (health: starting)   
nginx               goharbor/nginx-photon:v2.5.5           "nginx -g 'daemon of…"   proxy           18 seconds ago   Up 16 seconds (health: starting)   0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp
redis               goharbor/redis-photon:v2.5.5           "redis-server /etc/r…"   redis           18 seconds ago   Up 17 seconds (health: starting)   
registry            goharbor/registry-photon:v2.5.5        "/home/harbor/entryp…"   registry        18 seconds ago   Up 17 seconds (health: starting)   
registryctl         goharbor/harbor-registryctl:v2.5.5     "/home/harbor/start.…"   registryctl     18 seconds ago   Up 17 seconds (health: starting)   
trivy-adapter       goharbor/trivy-adapter-photon:v2.5.5   "/home/scanner/entry…"   trivy-adapter   18 seconds ago   Up 16 seconds (health: starting)

4. 拷贝证书到 docker 信任目录

[root@k8s-master1 harbor-tls]# cp ca.crt /etc/docker/certs.d/harbor.linux.cn/
[root@k8s-master1 harbor-tls]# cp harbor.linux.cn.cert /etc/docker/certs.d/harbor.linux.cn/
[root@k8s-master1 harbor-tls]# cp harbor.linux.cn.key /etc/docker/certs.d/harbor.linux.cn/

5. 登录验证

[root@k8s-master1 harbor-tls]# docker login https://harbor.linux.cn -u admin -p Harbor12345
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

其余机器也没问题

[root@k8s-master2 ~]# vim /etc/hosts

10.0.0.11 harbor.linux.cn

[root@k8s-master2 ~]# docker login https://harbor.linux.cn -u admin -p Harbor12345
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

6. 验证功能

[root@k8s-master1 harbor-tls]# docker pull nginx:latest
latest: Pulling from library/nginx
7cf63256a31a: Pull complete 
bf9acace214a: Pull complete 
513c3649bb14: Pull complete 
d014f92d532d: Pull complete 
9dd21ad5a4a6: Pull complete 
943ea0f0c2e4: Pull complete 
103f50cb3e9f: Pull complete 
Digest: sha256:9d6b58feebd2dbd3c56ab5853333d627cc6e281011cfd6050fa4bcf2072c9496
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest
[root@k8s-master1 harbor-tls]# docker tag nginx:latest harbor.linux.cn/library/nginx:latest
[root@k8s-master1 harbor-tls]# docker push harbor.linux.cn/library/nginx:latest
The push refers to repository [harbor.linux.cn/library/nginx]
55e9644f21c3: Pushed 
7d22e2347c12: Pushed 
f6d5815f290e: Pushed 
791f0a07985c: Pushed 
cabea05c000e: Pushed 
c68632c455ae: Pushed 
5f1ee22ffb5e: Pushed 
latest: digest: sha256:b7f8d15543a82805cfa01884032133e78bd839f5bb21ad41e235552ed04a5657 size: 1778

7. containerd 配置使用

找到 [plugins."io.containerd.grpc.v1.cri".registry]下的config_path，然后指定证书存储目录

[root@k8s-node01 containerd]# vim /etc/containerd/config.toml
146     [plugins."io.containerd.grpc.v1.cri".registry]
147       config_path = "/etc/containerd/certs.d"

[root@k8s-master1 harbor.linux.cn]# mkdir -pv /etc/containerd/certs.d/harbor.linux.cn && cd /etc/containerd/certs.d/harbor.linux.cn
[root@k8s-master1 harbor.linux.cn]# cp /usr/local/etc/harbor-tls/ca.crt .
[root@k8s-master1 harbor.linux.cn]# cp /usr/local/etc/harbor-tls/harbor.linux.cn.cert .
[root@k8s-master1 harbor.linux.cn]# cp /usr/local/etc/harbor-tls/harbor.linux.cn.key .
[root@k8s-master1 harbor.linux.cn]# vim hosts.toml 
[host."https://harbor.linux.cn"]
  capabilities = ["pull", "resolve","push"]
  skip_verify = false
  cert = "/etc/containerd/certs.d/harbor.linux.cn/harbor.linux.cn.cert"
  key = "/etc/containerd/certs.d/harbor.linux.cn/harbor.linux.cn.key"
  ca = "/etc/containerd/certs.d/harbor.linux.cn/ca.crt"

验证，如果 crictl 能正常拉取，k8s 也就能正常拉取

[root@k8s-master1 harbor.linux.cn]# crictl pull harbor.linux.cn/library/busybox:1.35.0

但是直接使用 ctr 拉取是会报错的：tls: failed to verify certificate: x509: certificate signed by unknown authority

或者使用 nerdctl 命令也能正常拉取

[root@k8s-master1 harbor.linux.cn]# wget https://github.com/containerd/nerdctl/releases/download/v1.7.6/nerdctl-1.7.6-linux-amd64.tar.gz
[root@k8s-master1 harbor.linux.cn]# mkdir nerdctl 
[root@k8s-master1 harbor.linux.cn]# tar xvf nerdctl-1.7.6-linux-amd64.tar.gz -C nerdctl
[root@k8s-master1 harbor.linux.cn]# mv nerdctl/nerdctl /usr/bin/
[root@k8s-master1 harbor.linux.cn]# nerdctl pull harbor.linux.cn/library/busybox:1.35.0

三、helm 方式部署

1. 前期准备

官方提供四种方式暴露 Harbor service:

Ingress: 借助Ingress暴露服务，K8S集群中已经部署ingress nginx controller。
ClusterIP: 使用ClusterIP暴露服务，只能在集群内部进行访问。
NodePort: 使用NodePort暴露服务，通过NodeIP:NodePort进行访问。
LoadBalancer: 使用云供应商提供的LB进行访问。

部署harbor仓库，使用ingress暴露服务。ingress-nginx 使用的是 NodePort 方式暴露自身，需要在 externalURL 中配置其 NodePort 端口号: 30443

[root@k8s-master1 harbor-2.5.5]# kubectl get svc -n ingress-nginx 
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
ingress-nginx-controller             NodePort    10.96.5.64      <none>        8080:30080/TCP,8043:30443/TCP   19h
ingress-nginx-controller-admission   ClusterIP   10.96.124.243   <none>        443/TCP                         19h

关于存储

呜 harbor 中的组建，无状态服务可以多副本启动，所以要看自己的存储是否支持多个 pod 同时挂载，

如果支持那么 accessMode: ReadWriteMany 是必须的，否则多副本 Pod 无法同时挂载存储卷。

我的存储如下，使用的 local 本地类型的存储，所以是无法支持多副本的

[root@k8s-master1 harbor-2.5.5]# kubectl get sc
NAME               PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
openebs-hostpath   openebs.io/local   Delete          WaitForFirstConsumer   false                  29d

2. 配置 harbor 的 helm 仓库

[root@k8s-master1 harbor-2.5.5]# helm repo add harbor https://helm.goharbor.io

3. 准备 https 证书

[root@k8s-master1 harbor-2.5.5]# kubectl -n harbor create secret tls harbor-tls-secret --key=\*.tianxiang.love.key --cert=\*.tianxiang.love.crt

4. 启动服务

我这边没有修改 values 文件，而是直接启动的，如有需要可以先拉取到本地然后修改完再启动

[root@k8s-master1 harbor-2.5.5]# helm pull harbor/harbor --version 1.9.5

命令详情

helm upgrade --install harbor harbor/harbor --namespace harbor --create-namespace \
  --version 1.9.5 \  # 显式指定 Chart 版本
  --set expose.type=ingress \
  --set expose.ingress.className=nginx \
  --set expose.ingress.hosts.core=harbor.tianxiang.love \
  --set expose.ingress.hosts.notary=notary.tianxiang.love \
  --set expose.ingress.annotations."nginx\.ingress\.kubernetes\.io/proxy-body-size"='"4096"' \  # 允许最大 4G 文件上传
  --set expose.ingress.annotations."nginx\.ingress\.kubernetes\.io/ssl-redirect"='"true"' \  # 强制 HTTPS
  --set expose.ingress.annotations."nginx\.ingress\.kubernetes\.io/force-ssl-redirect"='"true"' \  # 强制跳转 HTTPS
  --set expose.tls.enabled=true \  # 开启 https
  --set expose.tls.certSource=secret \  # 使用 secret 证书
  --set expose.tls.secret.secretName="harbor-tls-secret" \  # 指定已有 TLS Secret
  --set expose.tls.secret.notarySecretName="harbor-tls-secret" \  # 指定已有 TLS Secret
  --set externalURL=https://harbor.tianxiang.love:30443 \  # harbor 访问地址, 由于我使用的 nginx 的入口端口为 30443 
  --set harborAdminPassword="TianHarbor12345" \  # harbor 登录密码
  --set trivy.enabled=true \  # 开启镜像漏洞扫描
  --set trivy.ignoreUnfixed=false \  # 扫描包含未修复漏洞
  --set trivy.skipUpdate=false \  # 定期更新漏洞数据库
  # --- 资源限制 ---
  --set chartmuseum.resources.limits.cpu="500m" \
  --set chartmuseum.resources.limits.memory="2Gi" \
  --set trivy.resources.limits.cpu="500m" \
  --set trivy.resources.limits.memory="2Gi" \
  --set portal.resources.limits.cpu="1000m" \
  --set portal.resources.limits.memory="2Gi" \
  --set core.resources.limits.cpu="1000m" \
  --set core.resources.limits.memory="2Gi" \
  --set jobservice.resources.limits.cpu="2000m" \
  --set jobservice.resources.limits.memory="4Gi" \
  --set registry.resources.limits.cpu="1000m" \
  --set registry.resources.limits.memory="4Gi" \
  --set database.resources.limits.cpu="1000m" \
  --set database.resources.limits.memory="2Gi" \
  --set redis.resources.limits.cpu="500m" \
  --set redis.resources.limits.memory="1Gi" \
  # --- 存储配置 ---
  --set persistence.persistentVolumeClaim.registry.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.registry.size=500Gi \
  --set persistence.persistentVolumeClaim.chartmuseum.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.chartmuseum.size=50Gi \
  --set persistence.persistentVolumeClaim.jobservice.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.jobservice.size=50Gi \
  --set persistence.persistentVolumeClaim.database.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.database.size=50Gi \
  --set persistence.persistentVolumeClaim.redis.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.redis.size=50Gi \
  --set persistence.persistentVolumeClaim.trivy.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.trivy.size=50Gi

直接启动如下

[root@k8s-master1 harbor-2.5.5]# helm upgrade --install harbor harbor/harbor --namespace harbor --create-namespace \
  --version 1.9.5 \
  --set expose.type=ingress \
  --set expose.ingress.className=nginx \
  --set expose.ingress.hosts.core=harbor.tianxiang.love \
  --set expose.ingress.hosts.notary=notary.tianxiang.love \
  --set expose.ingress.annotations."nginx\.ingress\.kubernetes\.io/proxy-body-size"='"4096"' \
  --set expose.ingress.annotations."nginx\.ingress\.kubernetes\.io/ssl-redirect"='"true"' \
  --set expose.ingress.annotations."nginx\.ingress\.kubernetes\.io/force-ssl-redirect"='"true"' \
  --set expose.tls.enabled=true \
  --set expose.tls.certSource=secret \
  --set expose.tls.secret.secretName="harbor-tls-secret" \
  --set expose.tls.secret.notarySecretName="harbor-tls-secret" \
  --set externalURL=https://harbor.tianxiang.love:30443 \
  --set harborAdminPassword="TianHarbor12345" \
  --set trivy.enabled=true \
  --set trivy.ignoreUnfixed=false \
  --set trivy.skipUpdate=false \
  --set chartmuseum.resources.limits.cpu="500m" \
  --set chartmuseum.resources.limits.memory="2Gi" \
  --set trivy.resources.limits.cpu="500m" \
  --set trivy.resources.limits.memory="2Gi" \
  --set portal.resources.limits.cpu="1000m" \
  --set portal.resources.limits.memory="2Gi" \
  --set core.resources.limits.cpu="1000m" \
  --set core.resources.limits.memory="2Gi" \
  --set jobservice.resources.limits.cpu="2000m" \
  --set jobservice.resources.limits.memory="4Gi" \
  --set registry.resources.limits.cpu="1000m" \
  --set registry.resources.limits.memory="4Gi" \
  --set database.resources.limits.cpu="1000m" \
  --set database.resources.limits.memory="2Gi" \
  --set redis.resources.limits.cpu="500m" \
  --set redis.resources.limits.memory="1Gi" \
  --set persistence.persistentVolumeClaim.registry.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.registry.size=500Gi \
  --set persistence.persistentVolumeClaim.chartmuseum.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.chartmuseum.size=50Gi \
  --set persistence.persistentVolumeClaim.jobservice.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.jobservice.size=50Gi \
  --set persistence.persistentVolumeClaim.database.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.database.size=50Gi \
  --set persistence.persistentVolumeClaim.redis.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.redis.size=50Gi \
  --set persistence.persistentVolumeClaim.trivy.storageClass="openebs-hostpath" \
  --set persistence.persistentVolumeClaim.trivy.size=50Gi

5. 检查启动情况

检查 pod 启动情况

[root@k8s-master1 harbor-2.5.5]# kubectl -n harbor get pod
NAME                                    READY   STATUS    RESTARTS   AGE
harbor-chartmuseum-754fbc9457-m2xp9     1/1     Running   0          3h46m
harbor-core-65bf6d9ccf-4n4jr            1/1     Running   0          3h46m
harbor-database-0                       1/1     Running   0          3h46m
harbor-jobservice-5dbb5d5fbb-nzjq8      1/1     Running   0          3h46m
harbor-notary-server-7d98cfdbf8-m4gzb   1/1     Running   0          3h46m
harbor-notary-signer-5ffc867985-rn5rj   1/1     Running   0          3h46m
harbor-portal-78899b57fd-kf4kn          1/1     Running   0          3h46m
harbor-redis-0                          1/1     Running   0          3h46m
harbor-registry-7bb77fb895-m4p56        2/2     Running   0          3h46m
harbor-trivy-0                          1/1     Running   0          3h46m

检查 svc 和 ingress

[root@k8s-master1 harbor-2.5.5]# kubectl -n harbor get svc
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
harbor-chartmuseum     ClusterIP   10.96.140.37    <none>        80/TCP              3h47m
harbor-core            ClusterIP   10.96.16.101    <none>        80/TCP              3h47m
harbor-database        ClusterIP   10.96.142.173   <none>        5432/TCP            3h47m
harbor-jobservice      ClusterIP   10.96.86.125    <none>        80/TCP              3h47m
harbor-notary-server   ClusterIP   10.96.222.249   <none>        4443/TCP            3h47m
harbor-notary-signer   ClusterIP   10.96.90.234    <none>        7899/TCP            3h47m
harbor-portal          ClusterIP   10.96.131.214   <none>        80/TCP              3h47m
harbor-redis           ClusterIP   10.96.55.5      <none>        6379/TCP            3h47m
harbor-registry        ClusterIP   10.96.129.165   <none>        5000/TCP,8080/TCP   3h47m
harbor-trivy           ClusterIP   10.96.100.64    <none>        8080/TCP            3h47m
[root@k8s-master1 harbor-2.5.5]# kubectl -n harbor get ingress
NAME                    CLASS   HOSTS                   ADDRESS      PORTS     AGE
harbor-ingress          nginx   harbor.tianxiang.love   10.96.5.64   80, 443   3h47m
harbor-ingress-notary   nginx   notary.tianxiang.love   10.96.5.64   80, 443   3h47m

[root@k8s-master1 harbor-2.5.5]# cat /etc/hosts
192.168.233.246 harbor.tianxiang.love

配置可信任证书

[root@k8s-master1 harbor-2.5.5]# mkdir -pv /etc/docker/certs.d/harbor.tianxiang.love:30443

[root@k8s-master1 harbor-2.5.5]# kubectl -n harbor get secrets harbor-tls-secret -o jsonpath="{.data.tls\.crt}" | base64 -d > /etc/docker/certs.d/harbor.tianxiang.love:30443/ca.crt

[root@k8s-master1 harbor-2.5.5]# curl -k https://harbor.tianxiang.love:30443
<!doctype html>
<html>

<head>
    <meta charset="utf-8">
    <title>Harbor</title>
    <base href="/">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="icon" type="image/x-icon" href="favicon.ico?v=2">
    <link rel="preload" as="style" href="./light-theme.css?buildTimestamp=1673512572393">
    <link rel="preload" as="style" href="./dark-theme.css?buildTimestamp=1673512572393">
<link rel="stylesheet" href="styles.f21a150d0d914470.css"></head>
<body>
    <harbor-app>
        <div class="spinner spinner-lg app-loading app-loading-fixed">
            Loading...
        </div>
    </harbor-app>
<script src="runtime.c3926a0e4b77e829.js" type="module"></script><script src="polyfills.40b0dd9c16ed9dc8.js" type="module"></script><script src="scripts.a459d5a2820e9a99.js" defer></script><script src="main.abc826d0e4e6964c.js" type="module"></script></body>

</html>

[root@k8s-master1 harbor-2.5.5]# docker login harbor.tianxiang.love:30443 -u admin -p TianHarbor12345
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

7. 镜像异地同步脚本工具

[root@k8s-master1 replication-images]# cat replication-images.sh 
#!/usr/bin/env bash
#
#****************************************************************************
# 作者:               甄天祥
# QQ:                 2099637909
# 日期:               2022-09-21
# 网址:               http://blog.tianxiang.love
# 描述:               同步本地Harbor仓库中没有的远端Harbor仓库中的镜像
#*****************************************************************************

set -eo pipefail

# === 默认配置 ===
DEFAULT_SOURCE_USER="admin"
DEFAULT_SOURCE_PASS="TianHarbor12345"
DEFAULT_SOURCE_ADDR="harbor.tianxiang.love:30443"
DEFAULT_LOCAL_USER="admin"
DEFAULT_LOCAL_PASS="TianHarbor12345"
DEFAULT_LOCAL_ADDR="harbor-m6.tianxiang.love"
DEFAULT_THREADS=5
DEFAULT_WATCH_INTERVAL=300  # 默认每 5 分钟轮询

# === 初始化变量 ===
SOURCE_HARBOR_USER="$DEFAULT_SOURCE_USER"
SOURCE_HARBOR_PASSWD="$DEFAULT_SOURCE_PASS"
SOURCE_HARBOR_ADDRESS="$DEFAULT_SOURCE_ADDR"
LOCAL_HARBOR_USER="$DEFAULT_LOCAL_USER"
LOCAL_HARBOR_PASSWD="$DEFAULT_LOCAL_PASS"
LOCAL_HARBOR_ADDRESS="$DEFAULT_LOCAL_ADDR"
THREADS=$DEFAULT_THREADS
WATCH_MODE=false
WATCH_INTERVAL=$DEFAULT_WATCH_INTERVAL
SYNC_ALL=true
SPECIFIC_PROJECTS=()
EXCLUDE_PROJECTS=()
DRY_RUN=false
LOG_LEVEL="info"
TIMESTAMP=$(date '+%Y-%m-%d-%H-%M')
LOG_FILE="harbor-sync-${TIMESTAMP}.log"
FAILED_FILE="failed-images.txt"

# === 颜色定义 ===
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
NC='\033[0m'

# === 日志函数 ===
log() {
    local level=$1
    local message=$2
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')

    case $level in
        "debug")
            [[ "$LOG_LEVEL" == "debug" ]] && echo -e "${BLUE}[DEBUG]${NC} ${timestamp} - ${message}" | tee -a "$LOG_FILE"
            ;;
        "info")
            echo -e "${GREEN}[INFO]${NC} ${timestamp} - ${message}" | tee -a "$LOG_FILE"
            ;;
        "warn")
            echo -e "${YELLOW}[WARN]${NC} ${timestamp} - ${message}" | tee -a "$LOG_FILE"
            ;;
        "error")
            echo -e "${RED}[ERROR]${NC} ${timestamp} - ${message}" | tee -a "$LOG_FILE"
            ;;
        *)
            echo -e "[${level}] ${timestamp} - ${message}" | tee -a "$LOG_FILE"
            ;;
    esac
}

# === 帮助信息 ===
usage() {
    echo "使用方法: $0 [选项]"
    echo
    echo "选项:"
    echo "  --source-user 用户       源Harbor用户名"
    echo "  --source-pass 密码       源Harbor密码"
    echo "  --source-addr 地址       源Harbor地址"
    echo "  --local-user 用户        本地Harbor用户名"
    echo "  --local-pass 密码        本地Harbor密码"
    echo "  --local-addr 地址        本地Harbor地址"
    echo "  --project 项目名         同步指定项目 (可多次使用)"
    echo "  --exclude 项目名         排除指定项目 (可多次使用)"
    echo "  --threads 数量           并发线程数 (默认: $DEFAULT_THREADS)"
    echo "  --dry-run                试运行，不实际同步镜像"
    echo "  --debug                  启用DEBUG日志"
    echo "  --watch                  启用守护模式"
    echo "  --watch-interval 秒数     守护模式下轮询间隔 (默认: $DEFAULT_WATCH_INTERVAL)"
    echo "  -h, --help               显示帮助信息"
    exit 0
}

# === 参数解析 ===
parse_args() {
    while [[ $# -gt 0 ]]; do
        case "$1" in
            --source-user) SOURCE_HARBOR_USER="$2"; shift 2;;
            --source-pass) SOURCE_HARBOR_PASSWD="$2"; shift 2;;
            --source-addr) SOURCE_HARBOR_ADDRESS="$2"; shift 2;;
            --local-user) LOCAL_HARBOR_USER="$2"; shift 2;;
            --local-pass) LOCAL_HARBOR_PASSWD="$2"; shift 2;;
            --local-addr) LOCAL_HARBOR_ADDRESS="$2"; shift 2;;
            --project) SYNC_ALL=false; SPECIFIC_PROJECTS+=("$2"); shift 2;;
            --exclude) EXCLUDE_PROJECTS+=("$2"); shift 2;;
            --threads) THREADS="$2"; shift 2;;
            --dry-run) DRY_RUN=true; shift;;
            --debug) LOG_LEVEL="debug"; shift;;
            --watch) WATCH_MODE=true; shift;;
            --watch-interval) WATCH_INTERVAL="$2"; shift 2;;
            -h|--help) usage;;
            *) log error "未知选项: $1"; usage; exit 1;;
        esac
    done
}

check_dependencies() {
    for cmd in curl docker jq; do
        command -v "$cmd" >/dev/null || { log error "缺少依赖: $cmd"; exit 1; }
    done
}

harbor_login() {
    docker login "$1" -u "$2" -p "$3" >/dev/null || { log error "登录 $1 失败"; exit 1; }
}

fetch_projects() {
    curl -s -k -u "$1:$2" -X GET "https://$3/api/v2.0/projects?page_size=100" | jq -r '.[].name'
}

fetch_image_tags() {
    curl -s -k -u "$1:$2" -X GET "https://$3/api/v2.0/projects/$4/repositories?page_size=100" |
        jq -r '.[].name' | while read -r repo; do
        curl -s -k -u "$1:$2" -X GET "https://$3/v2/$repo/tags/list" |
            jq -r '.tags[]?' | while read -r tag; do
                echo "$repo:$tag"
            done
    done
}

create_project() {
    curl -s -k -u "$LOCAL_HARBOR_USER:$LOCAL_HARBOR_PASSWD" -X POST "https://$LOCAL_HARBOR_ADDRESS/api/v2.0/projects" \
        -H "Content-Type: application/json" \
        -d '{"project_name": "'$1'", "metadata": {"public": "false"}}' >/dev/null 2>&1 || true
}

sync_image() {
    local image=$1
    for attempt in {1..3}; do
        if docker pull "$SOURCE_HARBOR_ADDRESS/$image" && \
           docker tag "$SOURCE_HARBOR_ADDRESS/$image" "$LOCAL_HARBOR_ADDRESS/$image" && \
           docker push "$LOCAL_HARBOR_ADDRESS/$image"; then
            docker rmi "$SOURCE_HARBOR_ADDRESS/$image" "$LOCAL_HARBOR_ADDRESS/$image" >/dev/null 2>&1
            log info "同步成功: $image"
            echo "$image" >> synced-images.txt
            return 0
        else
            log warn "第 $attempt 次尝试失败: $image"
            sleep 2
        fi
    done
    log error "同步失败: $image"
    echo "$image" >> "$FAILED_FILE"
}

sync_images_concurrent() {
    local image_list=("$@")
    local -i index=0 total=${#image_list[@]}

    while [[ $index -lt $total ]]; do
        for ((i=0; i<THREADS && index<total; i++, index++)); do
            sync_image "${image_list[$index]}" &
        done
        wait
    done
}

main() {
    start_time=$(date +%s)
    check_dependencies

    log info "登录 Harbor 仓库..."
    harbor_login "$SOURCE_HARBOR_ADDRESS" "$SOURCE_HARBOR_USER" "$SOURCE_HARBOR_PASSWD"
    harbor_login "$LOCAL_HARBOR_ADDRESS" "$LOCAL_HARBOR_USER" "$LOCAL_HARBOR_PASSWD"

    log info "获取源项目列表..."
    readarray -t ALL_PROJECTS < <(fetch_projects "$SOURCE_HARBOR_USER" "$SOURCE_HARBOR_PASSWD" "$SOURCE_HARBOR_ADDRESS")

    FILTERED_PROJECTS=()
    for proj in "${ALL_PROJECTS[@]}"; do
        [[ "$SYNC_ALL" == false && ! " ${SPECIFIC_PROJECTS[*]} " =~ " $proj " ]] && continue
        [[ " ${EXCLUDE_PROJECTS[*]} " =~ " $proj " ]] && continue
        FILTERED_PROJECTS+=("$proj")
    done

    > synced-images.txt
    > "$FAILED_FILE"

    TO_SYNC_IMAGES=()
    for proj in "${FILTERED_PROJECTS[@]}"; do
        log info "处理项目: $proj"
        create_project "$proj"

        readarray -t SRC_IMAGES < <(fetch_image_tags "$SOURCE_HARBOR_USER" "$SOURCE_HARBOR_PASSWD" "$SOURCE_HARBOR_ADDRESS" "$proj")
        readarray -t DST_IMAGES < <(fetch_image_tags "$LOCAL_HARBOR_USER" "$LOCAL_HARBOR_PASSWD" "$LOCAL_HARBOR_ADDRESS" "$proj")

        for image in "${SRC_IMAGES[@]}"; do
            if ! printf '%s\n' "${DST_IMAGES[@]}" | grep -qx "$image"; then
                TO_SYNC_IMAGES+=("$image")
            fi
        done
    done

    if [[ "$DRY_RUN" == true ]]; then
        log info "[Dry Run] 需要同步的镜像:"
        printf '%s\n' "${TO_SYNC_IMAGES[@]}"
        exit 0
    fi

    log info "共需同步 ${#TO_SYNC_IMAGES[@]} 个镜像，开始并发执行..."
    sync_images_concurrent "${TO_SYNC_IMAGES[@]}"

    end_time=$(date +%s)
    duration=$((end_time - start_time))

    log info "✅ 同步完成: 成功 ${#TO_SYNC_IMAGES[@]} 个镜像, 耗时 ${duration} 秒"

    if [[ -s "$FAILED_FILE" ]]; then
        log warn "❌ 失败镜像列表如下 (保存在 $FAILED_FILE):"
        cat "$FAILED_FILE"
    fi

    log info "📦 同步成功的镜像列表："
    sort synced-images.txt || true
}

watch_mode() {
    log info "进入守护模式，每 ${WATCH_INTERVAL} 秒轮询同步..."
    while true; do
        main "${ORIGINAL_ARGS[@]}"
        log info "等待 ${WATCH_INTERVAL} 秒后继续同步..."
        sleep "$WATCH_INTERVAL"
    done
}

# 入口
ORIGINAL_ARGS=("$@")
parse_args "$@"

if [[ "$WATCH_MODE" == true ]]; then
    watch_mode
else
    main "$@"
fi

1. 脚本帮助信息

root@k8s-master1:/data/script/harbor# ./replication-images.sh -h
使用方法: ./replication-images.sh [选项]

选项:
  --source-user 用户       源Harbor用户名
  --source-pass 密码       源Harbor密码
  --source-addr 地址       源Harbor地址
  --local-user 用户        本地Harbor用户名
  --local-pass 密码        本地Harbor密码
  --local-addr 地址        本地Harbor地址
  --project 项目名         同步指定项目 (可多次使用)
  --exclude 项目名         排除指定项目 (可多次使用)
  --threads 数量           并发线程数 (默认: 5)
  --dry-run                试运行，不实际同步镜像
  --debug                  启用DEBUG日志
  --watch                  启用守护模式
  --watch-interval 秒数     守护模式下轮询间隔 (默认: 300)
  -h, --help               显示帮助信息

2. systemd 管理

root@k8s-master1:/data/script/harbor# cat /etc/systemd/system/harbor-sync.service
[Unit]
Description=Harbor 镜像同步守护进程
After=network.target docker.service
Requires=docker.service

[Service]
Type=simple
ExecStart=/data/script/harbor/replication-images.sh --watch --watch-interval 300
WorkingDirectory=/data/script/harbor
StandardOutput=append:/var/log/harbor-sync.log
StandardError=append:/var/log/harbor-sync.err
Restart=always
RestartSec=10
User=root

[Install]
WantedBy=multi-user.target

启动服务

root@k8s-master1:/data/script/harbor# sudo systemctl daemon-reexec
root@k8s-master1:/data/script/harbor# sudo systemctl daemon-reload
root@k8s-master1:/data/script/harbor# sudo systemctl enable harbor-sync.service --now

3. 日志信息

root@k8s-master1:/data/script/harbor# systemctl status harbor-sync.service 
● harbor-sync.service - Harbor 镜像同步守护进程
     Loaded: loaded (/etc/systemd/system/harbor-sync.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2025-08-06 15:10:37 CST; 48min ago
   Main PID: 189835 (bash)
      Tasks: 2 (limit: 4557)
     Memory: 19.3M
        CPU: 45.917s
     CGroup: /system.slice/harbor-sync.service
             ├─189835 bash /data/script/harbor/replication-images.sh --watch --watch-interval 300
             └─239536 sleep 300

Aug 06 15:10:37 k8s-master1 systemd[1]: Started Harbor 镜像同步守护进程.

root@k8s-master1:/data/script/harbor# tail -f harbor-sync-2025-08-06-15-10.log 
[INFO] 2025-08-06 15:54:49 - 登录 Harbor 仓库...
[INFO] 2025-08-06 15:54:49 - 获取源项目列表...
[INFO] 2025-08-06 15:54:50 - 处理项目: csiplugin
[INFO] 2025-08-06 15:54:50 - 处理项目: kubesphere
[INFO] 2025-08-06 15:54:57 - 处理项目: library
[INFO] 2025-08-06 15:54:58 - 处理项目: mysql-innodb-cluster
[INFO] 2025-08-06 15:54:58 - 共需同步 0 个镜像，开始并发执行...
[INFO] 2025-08-06 15:54:58 - ✅ 同步完成: 成功 0 个镜像, 耗时 9 秒
[INFO] 2025-08-06 15:54:58 - 📦 同步成功的镜像列表：
[INFO] 2025-08-06 15:54:58 - 等待 300 秒后继续同步...

root@k8s-master1:/data/script/harbor# tail -f /var/log/harbor-sync.log 
[INFO] 2025-08-06 15:54:49 - 获取源项目列表...
[INFO] 2025-08-06 15:54:50 - 处理项目: csiplugin
[INFO] 2025-08-06 15:54:50 - 处理项目: kubesphere
[INFO] 2025-08-06 15:54:57 - 处理项目: library
[INFO] 2025-08-06 15:54:58 - 处理项目: mysql-innodb-cluster
[INFO] 2025-08-06 15:54:58 - 共需同步 0 个镜像，开始并发执行...
[INFO] 2025-08-06 15:54:58 - ✅ 同步完成: 成功 0 个镜像, 耗时 9 秒
[INFO] 2025-08-06 15:54:58 - 📦 同步成功的镜像列表：
[INFO] 2025-08-06 15:54:58 - 等待 300 秒后继续同步...
[INFO] 2025-08-06 15:59:58 - 登录 Harbor 仓库...
[INFO] 2025-08-06 15:59:59 - 获取源项目列表...
[INFO] 2025-08-06 15:59:59 - 处理项目: csiplugin
[INFO] 2025-08-06 15:59:59 - 处理项目: kubesphere

四、本地信息证书

1. windows 信任证书

本地配置信任自签证书时期浏览器显示安全小绿🔒

1.1 脚本导入证书

保存命名为 install-ca.ps1 注意文件后缀为 ps1 格式

# install-ca.ps1
# 将自签名 CA 根证书导入 Windows 受信任的根证书颁发机构（带检测，支持 PEM 格式）

param(
    [string]$CertPath = ".\ca.crt"
)

if (-Not (Test-Path $CertPath)) {
    Write-Error "Certificate file not found: $CertPath"
    exit 1
}

try {
    # 读取 PEM 格式证书
    $certContent = Get-Content -Path $CertPath -Raw
    $bytes = [System.Text.Encoding]::ASCII.GetBytes($certContent)
    $cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
    $cert.Import($bytes)

    $thumbprint = $cert.Thumbprint.ToUpper()
    Write-Output "Detected certificate: $CertPath"
    Write-Output "Thumbprint: $thumbprint"
    Write-Output "Subject: $($cert.Subject)"
    Write-Output "Issuer: $($cert.Issuer)"
}
catch {
    Write-Error "Failed to load certificate: $_"
    exit 1
}

# 检查系统是否已有相同证书
$store = New-Object System.Security.Cryptography.X509Certificates.X509Store("Root","LocalMachine")
$store.Open("ReadOnly")
$existing = $store.Certificates | Where-Object { $_.Thumbprint -eq $thumbprint }
$store.Close()

if ($existing) {
    Write-Output "Certificate already exists in 'Trusted Root Certification Authorities'. No action taken."
} else {
    Write-Output "Importing certificate..."
    Import-Certificate -FilePath $CertPath -CertStoreLocation Cert:\LocalMachine\Root | Out-Null
    if ($?) {
        Write-Output "Certificate successfully imported into 'Trusted Root Certification Authorities'."
    } else {
        Write-Error "Certificate import failed. Try running PowerShell as Administrator."
    }
}

1.2 以管理员身份打开 PowerShell

执行如下命令：

PS E:\> Set-ExecutionPolicy Bypass -Scope Process -Force
PS E:\> .\install-ca.ps1 -CertPath .\ca.crt
Detected certificate: .\ca.crt
Thumbprint: 239E417C126F59C04A2945ACC14A46EFB706C6A7
Subject: CN=My Root CA, OU=CA, O=Example CA, L=Beijing, S=Beijing, C=CN
Issuer: CN=My Root CA, OU=CA, O=Example CA, L=Beijing, S=Beijing, C=CN
Importing certificate...
Certificate successfully imported into 'Trusted Root Certification Authorities'.
PS E:\>