使用ceph rdb做k8s后端存储(ceph-csi storageclass

环境:
Os:Centos 7
k8s:v1.28

ceph:14.2.22

 

###############################################ceph端分配存储##############################
1.分配osd pool和创建用户
[root@master tmp]#ceph osd pool create dynamics-pool 8
[root@master tmp]#ceph auth get-or-create client.dynamics-user mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=dynamics-pool' -o ceph.client.dynamics-user.keyring

这里我们创建了用户 dynamics-user,后面的配置会使用到

2.查看 dynamics-user 的秘钥：
[root@master tmp]# ceph auth get-key client.dynamics-user
AQAmu+xo7MsnJhAAUzvdoEWnunVNGQhdSz1o6Q==

 

#######################################storageclass部署#########################
1.查看k8s版本
[root@master my_kubernetes]# kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.13

 

2.ceph版本
[root@master my_kubernetes]# ceph version
ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)

 

3.下载ceph-csi
下载地址:
https://github.com/ceph/ceph-csi/releases/tag/v3.11.0
我这里下载的是:v3.11.0

4.下载后进行解压
解压后找到如下目录:
/root/ceph-csi/ceph-csi-3.11.0/deploy/rbd/kubernetes
我们会使用该目录下的部署yaml文件

[root@master rbd]#tar -xvf ceph-csi-3.11.0.tar.gz
[root@master rbd]# pwd
/root/ceph-csi/ceph-csi-3.11.0/deploy/rbd
[root@master rbd]# ls -al
total 4
drwxrwxr-x 4 root root 45 Oct 17 09:41 .
drwxrwxr-x 6 root root 173 Apr 2 2024 ..
drwxrwxr-x 2 root root 184 Apr 2 2024 kubernetes

 

我们拷贝一份出来,根据自己的情况进行修改
[root@master rbd]#cp -r -p kubernetes my_kubernetes

 

5.编辑csi-config-map.yaml

[root@master rbd]# more /root/ceph-csi/ceph-csi-3.11.0/deploy/rbd/my_kubernetes/csi-config-map.yaml
#
# /!\ DO NOT MODIFY THIS FILE
#
# This file has been automatically generated by Ceph-CSI yamlgen.
# The source for the contents can be found in the api/deploy directory, make
# your modifications there.
#
---
apiVersion: v1
kind: ConfigMap
metadata:name: "ceph-csi-config"
data:config.json: |-[{"clusterID": "1508a2da-5991-487a-836c-d6e6527b1dc7","monitors": ["192.168.1.108:6789","192.168.1.105:6789","192.168.1.106:6789"]}]

 

 

6.创建名称空间并部署csi-config-map
kubectl create ns ceph-csi ##创建namespace --done
kubectl -n ceph-csi create -f csi-config-map.yaml

 

7.新建kms-config文件

内容保留空白

[root@master my_kubernetes]# more hxl_csi-kms-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:config.json: |-{}
metadata:name: ceph-csi-encryption-kms-config[root@master my_kubernetes]# kubectl create -n ceph-csi -f hxl_csi-kms-config-map.yaml

 

8.新建一个ceph-config文件，ceph.conf中的内容与ceph集群中 /etc/ceph/ceph.conf 的内容保持一致

[root@master my_kubernetes]# more hxl_ceph-config-map.yaml 
---
apiVersion: v1
kind: ConfigMap
data:ceph.conf: |[global]fsid = 1508a2da-5991-487a-836c-d6e6527b1dc7public_network = 192.168.1.0/24mon_initial_members = master, node1, node2mon_host = 192.168.1.108,192.168.1.105,192.168.1.106auth_cluster_required = cephxauth_service_required = cephxauth_client_required = cephxmon_allow_pool_delete = trueosd pool default size = 2 mon clock drift allowed = 2 mon clock drift warn backoff = 30 # keyring is a required key and its value should be emptykeyring: |
metadata:name: ceph-config[root@master my_kubernetes]#kubectl -n ceph-csi create -f hxl_ceph-config-map.yaml

 

9.新建一个secret文件

[root@master my_kubernetes]# more hxl_csi-rbd-secret.yaml
apiVersion: v1
kind: Secret
metadata:name: csi-rbd-secretnamespace: ceph-csi
stringData:userID: dynamics-useruserKey: AQAmu+xo7MsnJhAAUzvdoEWnunVNGQhdSz1o6Q==
[root@k8s-master02 kubernetes]# kubectl create -f hxl_csi-rbd-secret.yaml

这里的userID和userKey填写上面步骤生成的

 

10.将所有yaml文件的namespace从default改成ceph-csi

[root@master my_kubernetes]# sed -i "s/namespace: default/namespace: ceph-csi/g" $(grep -rl "namespace: default" ./)

##检查配置文件中namespace是否更改成功
[root@master my_kubernetes]# cat csi-provisioner-rbac.yaml

 

11.创建rbac权限

[root@master my_kubernetes]#kubectl create -f csi-provisioner-rbac.yaml

 

12.部署ceph-csi相关容器

替换yaml文件中的镜像,原有镜像可能下载不了
[root@k8s-master02 kubernetes]# sed -i 's#quay.io/cephcsi/#registry.cn-shenzhen.aliyuncs.com/hxlk8s/#' csi-rbdplugin*
[root@k8s-master02 kubernetes]# sed -i 's#registry.k8s.io/sig-storage/#registry.cn-shenzhen.aliyuncs.com/hxlk8s/#' csi-rbdplugin*

[root@master my_kubernetes]# kubectl -n ceph-csi create -f csi-rbdplugin-provisioner.yaml ##这个部署默认是3个副本，我这里的环境修改成2个副本
[root@master my_kubernetes]# kubectl -n ceph-csi create -f csi-rbdplugin.yaml
[root@master my_kubernetes]# kubectl -n ceph-csi create -f csidriver.yaml
[root@master my_kubernetes]# kubectl -n ceph-csi create -f csi-nodeplugin-rbac.yaml

 

##检查pod是否都已启动

[root@master my_kubernetes]# kubectl get pod -n ceph-csi
NAME                                         READY   STATUS    RESTARTS   AGE
csi-rbdplugin-6xczb                          3/3     Running   0          142m
csi-rbdplugin-f7smn                          3/3     Running   0          142m
csi-rbdplugin-provisioner-7dbb74fb75-6kv56   7/7     Running   0          142m
csi-rbdplugin-provisioner-7dbb74fb75-rtlnt   7/7     Running   0          142m

 

13.创建StorageClass

[root@master my_kubernetes]# more hxl_storageclass.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:clusterID: 1508a2da-5991-487a-836c-d6e6527b1dc7pool: dynamics-poolimageFeatures: layeringcsi.storage.k8s.io/provisioner-secret-name: csi-rbd-secretcsi.storage.k8s.io/provisioner-secret-namespace: ceph-csicsi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secretcsi.storage.k8s.io/controller-expand-secret-namespace: ceph-csicsi.storage.k8s.io/node-stage-secret-name: csi-rbd-secretcsi.storage.k8s.io/node-stage-secret-namespace: ceph-csicsi.storage.k8s/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:- discard[root@k8s-master02 kubernetes]# kubectl create -f hxl_storageclass.yaml
storageclass.storage.k8s.io/csi-rbd-sc created[root@master my_kubernetes]# kubectl get sc
NAME         PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-rbd-sc   rbd.csi.ceph.com   Delete          Immediate           true                   5s

 

14.创建PV

可以直接使用安装包自带的pvc.yaml文件,目录如下:

cd /root/ceph-csi/ceph-csi-3.11.0/examples/rbd
[root@master rbd]# more pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: rbd-pvc
spec:accessModes:- ReadWriteOnceresources:requests:storage: 1GistorageClassName: csi-rbd-sc[root@master rbd]#kubectl create -f pvc.yaml[root@master rbd]# kubectl get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rbd-pvc   Bound    pvc-aa4c6cc7-6a3c-42d9-8a7c-53b162162073   1Gi        RWO            csi-rbd-sc     2s

 

15.测试 pod 挂载 pvc

15.1 准备yaml文件

[root@master ceph_test]# more /root/ceph-csi/ceph-csi-3.11.0/deploy/rbd/my_kubernetes/rbd-test-pod1.yml
apiVersion: v1
kind: Pod
metadata:name: rbd-test-pod1
spec:containers:- name: nginximage: registry.cn-shenzhen.aliyuncs.com/hxlk8s/nginx:latestimagePullPolicy: IfNotPresentvolumeMounts:- name: data-volumemountPath: /usr/share/nginx/html/volumes:- name: data-volumepersistentVolumeClaim:claimName: rbd-pvcreadOnly: false

 

15.2 应用
[root@master my_kubernetes]# kubectl apply -f rbd_nginx_deployment.yml

 

15.3 查看运行情况

[root@master my_kubernetes]# kubectl get pods
NAME                  READY   STATUS    RESTARTS   AGE
rbd-d6799c7fc-44cp2   1/1     Running   0          30s

 

15.4 进入容器查看

[root@master my_kubernetes]#kubectl exec -it rbd-d6799c7fc-44cp2 -- bash
root@rbd-d6799c7fc-44cp2:/# df -hT |grep /dev/rbd0
/dev/rbd0               ext4     976M  2.6M  958M   1% /usr/share/nginx/html

可以看到/dev/rbd0已经被格式化成ext4并且挂载到了/usr/share/nginx/html

 

15.5 进入容器修改内容

[root@master my_kubernetes]#kubectl exec -it rbd-d6799c7fc-44cp2 -- bash
root@rbd-d6799c7fc-44cp2:/# cd /usr/share/nginx/html
root@rbd-d6799c7fc-44cp2:/usr/share/nginx/html# ls -al
total 20
drwxr-xr-x 3 root root  4096 Oct 17 07:51 .
drwxr-xr-x 3 root root    18 Aug 13 18:52 ..
drwx------ 2 root root 16384 Oct 17 07:51 lost+found
root@rbd-d6799c7fc-44cp2:/usr/share/nginx/html# echo 123 > index.html
root@rbd-d6799c7fc-44cp2:/usr/share/nginx/html# chmod 644 index.html

 

15.6 访问容器查看内容

[root@master my_kubernetes]# kubectl get pods -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP              NODE    NOMINATED NODE   READINESS GATES
rbd-d6799c7fc-44cp2   1/1     Running   0          3m    10.244.104.37   node2   <none>           <none>[root@master my_kubernetes]# curl 10.244.104.37
123

内容可以正常被访问到,我们将容器删除,然后让他自己重新启动一个来看看文件是否还存在

 

[root@master my_kubernetes]# kubectl delete pods rbd-d6799c7fc-44cp2
pod "rbd-d6799c7fc-44cp2" deleted

一会后自动创建一个新的pod

[root@master my_kubernetes]# kubectl get pods -owide
NAME                  READY   STATUS    RESTARTS   AGE   IP              NODE    NOMINATED NODE   READINESS GATES
rbd-d6799c7fc-5bnbl   1/1     Running   0          84s   10.244.104.38   node2   <none>           <none>

 

再次访问
[root@master my_kubernetes]# curl 10.244.104.38
123
可以看到，也是没有问题的，这样k8s就正常的使用了rbd存储