创建账号

进入ceph-tools pod

kubectl -n rook-ceph exec -it rook-ceph-tools-6ccb958485-j7pvb bash

查看可用的对象存储

[root@rook-ceph-tools-6ccb958485-j7pvb /]# radosgw-admin realm list  
{
    "default_info": "11f77019-6723-4932-9bd4-d253077d8bca",
    "realms": [
        "my-store"
    ]
}

查看可用的对象存储组

[root@rook-ceph-tools-6ccb958485-j7pvb /]# radosgw-admin zonegroup list
{
    "default_info": "fcdf8b04-bfba-492b-b1db-9f89771c85cc",
    "zonegroups": [
        "my-store"
    ]
}

创建账号

radosgw-admin user create --uid rgw-gitlab --display-name "user for gitlab runner cache" --rgw-realm=my-store --rgw-zonegroup=my-store

得到AK/SK

{
            "user": "rgw-gitlab",
            "access_key": "06SHK9ONNRLXB7KSQ4P3",
            "secret_key": "AudsraBSwjVm5hjLZ7XoBmA6Pe5Cj7JBYnT4GyTC"
}

查看账号

[root@rook-ceph-tools-6ccb958485-j7pvb /]# radosgw-admin user list
[
    "rgw-gitlab"
]

创建bucket

注：以下操作都是在k8s集群内的node上操作的

安装s5cmd

在集群内安装s5cmd，用它来和ceph交互

curl https://ghproxy.com/https://github.com/peak/s5cmd/releases/download/v2.0.0-beta.2/s5cmd_2.0.0-beta.2_Linux-64bit.tar.gz -O
tar -zxvf s5cmd_2.0.0-beta.2_Linux-64bit.tar.gz
mv s5cmd /usr/local/bin/

找到ceph对象存储的service

root@lvs-172-30-3-192:~# kubectl -n rook-ceph get svc|grep rgw
rook-ceph-rgw-my-store     ClusterIP   x.x.x.x    <none>        80/TCP              2d21h

配置s5cmd

export S3_ENDPOINT_URL="http://x.x.x.x:80"
export AWS_ACCESS_KEY_ID=06SHK9ONNRLXB7KSQ4P3
export AWS_SECRET_ACCESS_KEY=AudsraBSwjVm5hjLZ7XoBmA6Pe5Cj7JBYnT4GyTC

• url来自ceph对象存储的service
• AK/SK来自创建账号得到的AK/SK

创建bucket

root@node1:~# s5cmd mb s3://gitlab-runner-cache-bucket
mb s3://gitlab-runner-cache-bucket

查看bucket

root@node1:~# s5cmd ls
2022/05/13 07:54:24  s3://gitlab-runner-cache-bucket

20220924更新

Gitlab CI/CD 已在我司使用了 4 个月，期间不断迭代优化，现在基本稳定了。Gitlab CI/CD 系列 的文章是早期的版本，这里更新下最新的情况。

不用 Gitlab 流水线的缓存，性能很差，直接用挂载目录替代。参考Gitlab CI/CD 实践三：Docker 安装 Gitlab Runner