ceph的块存储太简单了。所以不做演示

查看集群

创建一个 CephFS 文件系统
# ceph fs volume create cephfs01
需要创建一个子卷# ceph fs subvolume create cephfs01 my-subvol
-----------------#以下全部自动创建好
# ceph fs ls       
name: cephfs01, metadata pool: cephfs.cephfs01.meta, data pools: [cephfs.cephfs01.data ]# ceph osd pool ls
.mgr
cephfs.cephfs01.meta
cephfs.cephfs01.data

最后配置K8S yaml即可

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm install -n ceph-csi ceph-csi-cephfs ceph-csi-cephfs-3.13.1.tgz -f values.yaml

以下我的yaml参考

# egrep -v "^[[:space:]]*#|^$" values.yaml
---
rbac:create: trueleastPrivileges: true
serviceAccounts:nodeplugin:create: truename:provisioner:create: truename:
csiConfig:- clusterID: "51edffbe-03fd-11f0-8abe-000c29f6d983"monitors:- "172.16.8.100:6789"- "172.16.8.102:6789"- "172.16.8.103:6789"cephFS:subvolumeGroup: "myfsg"  ----你的子卷
encryptionKMSConfig: {}
commonLabels: {}
logLevel: 5
sidecarLogLevel: 1
logSlowOperationInterval: 30s
CSIDriver:fsGroupPolicy: "File"seLinuxMount: false
nodeplugin:name: nodepluginupdateStrategy: RollingUpdatepriorityClassName: system-node-criticalhttpMetrics:enabled: truecontainerPort: 8081service:enabled: trueservicePort: 8080type: ClusterIPannotations: {}clusterIP: ""externalIPs: []loadBalancerIP: ""loadBalancerSourceRanges: []imagePullSecrets: []profiling:enabled: falseregistrar:image:repository: registry.aliyuncs.com/google_containers/csi-node-driver-registrartag: v2.13.0pullPolicy: IfNotPresentresources: {}plugin:image:repository: quay.io/cephcsi/cephcsitag: canarypullPolicy: IfNotPresentresources: {}nodeSelector: {}tolerations: []affinity: {}annotations: {}podSecurityContext: {}podAnnotations: {}kernelmountoptions: ""fusemountoptions: ""
provisioner:name: provisionerreplicaCount: 1strategy:type: RollingUpdaterollingUpdate:maxUnavailable: 50%timeout: 60spriorityClassName: system-cluster-criticalenableHostNetwork: falsehttpMetrics:enabled: truecontainerPort: 8081service:enabled: trueservicePort: 8080type: ClusterIPannotations: {}clusterIP: ""externalIPs: []loadBalancerIP: ""loadBalancerSourceRanges: []imagePullSecrets: []profiling:enabled: falseprovisioner:image:repository: registry.aliyuncs.com/google_containers/csi-provisionertag: v5.1.0pullPolicy: IfNotPresentresources: {}args:extraArgs: []setmetadata: trueresizer:name: resizerenabled: trueimage:repository: registry.aliyuncs.com/google_containers/csi-resizertag: v1.13.1pullPolicy: IfNotPresentresources: {}args:extraArgs: []snapshotter:image:repository: registry.aliyuncs.com/google_containers/csi-snapshottertag: v8.2.0pullPolicy: IfNotPresentresources: {}args:enableVolumeGroupSnapshots: falseextraArgs: []nodeSelector: {}tolerations: []affinity: {}annotations: {}podSecurityContext: {}podAnnotations: {}
selinuxMount: false
storageClass:create: truename: csi-cephfs-scannotations: {}clusterID: 51edffbe-03fd-11f0-8abe-000c29f6d983fsName: cephfs01   ----------------------你的名字pool: "cephfs.cephfs01.data"  fuseMountOptions: ""kernelMountOptions: ""mounter: ""volumeNamePrefix: ""encrypted: ""encryptionKMSID: ""provisionerSecret: csi-cephfs-secretprovisionerSecretNamespace: ""controllerExpandSecret: csi-cephfs-secretcontrollerExpandSecretNamespace: ""nodeStageSecret: csi-cephfs-secretnodeStageSecretNamespace: ""reclaimPolicy: DeleteallowVolumeExpansion: truemountOptions:- _netdev  ------------------重点在这里cephfs必须使用这个参数 块存储 设置为discard 启用 TRIM 功能 文件存储中无效或冗余，可能导致挂载参数冲突（如 CephFS 报错 Invalid argument）
secret:create: truename: csi-cephfs-secretannotations: {}adminID: admin  ###----------这里建议全部使用adminadminKey: AQDDddln7+UvHRAAwZVyl1UEJOpY3fWG2Oev+A==userID: "admin"userKey: "AQDDddln7+UvHRAAwZVyl1UEJOpY3fWG2Oev+A=="
cephconf: |[global]auth_cluster_required = cephxauth_service_required = cephxauth_client_required = cephx
extraDeploy: []
provisionerSocketFile: csi-provisioner.sock
pluginSocketFile: csi.sock
kubeletDir: /var/lib/kubelet
driverName: cephfs.csi.ceph.com
configMapName: ceph-csi-config
externallyManagedConfigmap: false
cephConfConfigMapName: ceph-config
kmsConfigMapName: ceph-csi-encryption-kms-config

下面测试

apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: csi-cephfs-pvc
spec:accessModes:- ReadWriteManyresources:requests:storage: 1GistorageClassName: csi-cephfs-sc
---
apiVersion: v1
kind: Pod
metadata:name: fs-pod
spec:containers:- name: web-serverimage: docker.1ms.run/nginx:alpinevolumeMounts:- name: mypvcmountPath: /usr/share/nginx/htmlvolumes:- name: mypvcpersistentVolumeClaim:claimName: csi-cephfs-pvcreadOnly: false

默认是开启内核挂载模式的