K8s Pod 创建埋点处理(Mutating Admission Webhook)

写在前面

工作中涉及到相关的知识
在实际的生产中，我们可以有需求对创建的资源做一些类似 埋点 相关的操作，比如添加一些 Pod 创建时的自定义验证逻辑，类似表单提交验证那样，或者希望对创建的资源对象进行加工，在比如给资源对象添加对应的 zone 标签，涉及到 SC 相关的标签，或者根据命名空间动态织入亲和性和拓扑相关约束,添加一些卷
上面的这些需求我们可以通过 k8s Admission Webhook来完成，博文为查阅资料整理笔记，内容为涉及
一个大佬写好的自定义准入控制器 Demo 学习
理解不足小伙伴帮忙指正

对每个人而言，真正的职责只有一个：找到自我。然后在心中坚守其一生，全心全意，永不停息。所有其它的路都是不完整的，是人的逃避方式，是对大众理想的懦弱回归，是随波逐流，是对内心的恐惧 ——赫尔曼·黑塞《德米安》

学习之前，建议看看这两篇官网的文章：

准入控制器参考:

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

Kubernetes 准入控制器指南:

https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

学习的项目文章：

https://didil.medium.com/building-a-kubernetes-mutating-admission-webhook-7e48729523ed

翻译版本：

https://cloudnative.to/blog/mutating-admission-webhook/

通过 Admission Webhook 为每个创建的 Pod(打了指定标签) 添加一个 CM ，以卷的方式挂载，类似为每个命名空间中的 Pod 自动挂载当前命名默认生成 SA 的 token 一样。

克隆一下作者的项目：

https://github.com/didil/k8s-hello-mutating-webhook

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook]
└─$git clone  https://github.com/didil/k8s-hello-mutating-webhook.git

确认一下对应的 Webhook 钩子对应的镜像能不能使用

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$docker pull quay.io/didil/hello-webhook:0.1.9
0.1.9: Pulling from didil/hello-webhook
df20fa9351a1: Pull complete
c5ef763ec908: Pull complete
242c22b4c8b8: Pull complete
f201599daacc: Pull complete
Digest: sha256:8c6ef414e4df46f15b9b63f83bd24f7c6a2cc5b5b937c53f0811fd509969ad05
Status: Downloaded newer image for quay.io/didil/hello-webhook:0.1.9
quay.io/didil/hello-webhook:0.1.9

学习的环境

┌──[root@vms100.liruilongs.github.io]-[~]
└─$kubectl get node
NAME                          STATUS   ROLES           AGE    VERSION
vms100.liruilongs.github.io   Ready    control-plane   292d   v1.25.1
vms101.liruilongs.github.io   Ready    control-plane   292d   v1.25.1
vms102.liruilongs.github.io   Ready    control-plane   292d   v1.25.1
vms103.liruilongs.github.io   Ready    <none>          292d   v1.25.1
vms105.liruilongs.github.io   Ready    <none>          292d   v1.25.1
vms106.liruilongs.github.io   Ready    <none>          292d   v1.25.1
┌──[root@vms100.liruilongs.github.io]-[~]
└─$

看下作者的 yaml 文件，可以看到，当前作者使用 kustomization 来管理 k8s yaml 资源文件, kustomize 是一个生成 k8s 资源 yaml 文件的插件，支持继承组合等一些面向对象思维方面的 yaml 文件生成

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$ls
csr  deployment  other
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$tree .
.
├── csr
│   ├── csr-job.yaml
│   ├── csr-rolebinding.yaml
│   ├── csr-role.yaml
│   ├── csr-sa.yaml
│   └── kustomization.yaml
├── deployment
│   ├── deployment.yaml
│   └── kustomization.yaml
└── other├── configmap.yaml├── kustomization.yaml├── service.yaml└── webhookconf.yaml3 directories, 11 files
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$

需要安装 kustomize ,嫌麻烦当然也可以直接看

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$kustomize version
{Version:kustomize/v4.5.7 GitCommit:56d82a8378dfc8dc3b3b1085e5a6e67b82966bd7 BuildDate:2022-08-02T16:35:54Z GoOs:linux GoArch:amd64}

这里分别看下作者的三个目录里面放了什么

deployment 目录主要为注入逻辑的工作负载hello-webhook，这里需要注意的是挂载了一个证书文件，由于 Webhook 必须通过 HTTPS 提供，因此我们需要为服务器提供适当的自签名 CA 签名。证书的公用名（CN）必须与 Kubernetes API 服务器使用的服务器名称匹配。

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$ls
deployment.yaml  kustomization.yaml
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$kubectl kustomize ./
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: hello-webhookname: hello-webhook-deployment
spec:replicas: 1selector:matchLabels:app: hello-webhooktemplate:metadata:labels:app: hello-webhookspec:containers:- image: quay.io/didil/hello-webhook:0.1.9name: hello-webhookports:- containerPort: 8000resources:limits:cpu: 500mmemory: 128MivolumeMounts:- mountPath: /tlsname: hello-tls-secretreadOnly: truevolumes:- name: hello-tls-secretsecret:secretName: hello-tls-secret
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$

下面为 webhook 的核心逻辑,用于对 Pod 添加对应的 CM 卷 ,添加对应的操作记录，返回修改后的对象

func (app *App) HandleMutate(w http.ResponseWriter, r *http.Request) {admissionReview := &admissionv1.AdmissionReview{}// read the AdmissionReview from the request json bodyerr := readJSON(r, admissionReview)if err != nil {app.HandleError(w, r, err)return}// unmarshal the pod from the AdmissionRequestpod := &corev1.Pod{}if err := json.Unmarshal(admissionReview.Request.Object.Raw, pod); err != nil {app.HandleError(w, r, fmt.Errorf("unmarshal to pod: %v", err))return}// add the volume to the podpod.Spec.Volumes = append(pod.Spec.Volumes, corev1.Volume{Name: "hello-volume",VolumeSource: corev1.VolumeSource{ConfigMap: &corev1.ConfigMapVolumeSource{LocalObjectReference: corev1.LocalObjectReference{Name: "hello-configmap",},},},})// add volume mount to all containers in the podfor i := 0; i < len(pod.Spec.Containers); i++ {pod.Spec.Containers[i].VolumeMounts = append(pod.Spec.Containers[i].VolumeMounts, corev1.VolumeMount{Name:      "hello-volume",MountPath: "/etc/config",})}containersBytes, err := json.Marshal(&pod.Spec.Containers)if err != nil {app.HandleError(w, r, fmt.Errorf("marshall containers: %v", err))return}volumesBytes, err := json.Marshal(&pod.Spec.Volumes)if err != nil {app.HandleError(w, r, fmt.Errorf("marshall volumes: %v", err))return}// build json patchpatch := []JSONPatchEntry{JSONPatchEntry{OP:    "add",Path:  "/metadata/labels/hello-added",Value: []byte(`"OK"`),},JSONPatchEntry{OP:    "replace",Path:  "/spec/containers",Value: containersBytes,},JSONPatchEntry{OP:    "replace",Path:  "/spec/volumes",Value: volumesBytes,},}patchBytes, err := json.Marshal(&patch)if err != nil {app.HandleError(w, r, fmt.Errorf("marshall jsonpatch: %v", err))return}patchType := admissionv1.PatchTypeJSONPatch// build admission responseadmissionResponse := &admissionv1.AdmissionResponse{UID:       admissionReview.Request.UID,Allowed:   true,Patch:     patchBytes,PatchType: &patchType,}respAdmissionReview := &admissionv1.AdmissionReview{TypeMeta: metav1.TypeMeta{Kind:       "AdmissionReview",APIVersion: "admission.k8s.io/v1",},Response: admissionResponse,}jsonOk(w, &respAdmissionReview)
}

上述代码主要做了如下事情：

将来自 Http 请求中的 AdmissionReview json 输入反序列化。
读取 Pod 的 spec 信息。
将 hello-configmap 作为数据源，添加 hello-volume 卷到 Pod。
挂载卷至 Pod 容器中。
以 JSON PATCH 的形式记录变更信息，包括卷的变更，卷挂载信息的变更。顺道为容器添加一个“hello-added=true”的标签。
构建 json 格式的响应结果，结果中包含了这次请求中的被修改的部分。

other 目录主要放了需要动态织入的 CM，以卷的方式使用，以及 MutatingWebhookConfiguration 准入控制器的定义，以及 webhook 的 SVC(SVC 名字太长，没办法生成证书，需要修改一下)

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/other]
└─$kubectl kustomize .
apiVersion: v1
data:hello.txt: "\n /$$$$$$$$ /$$   /$$ /$$$$$$  /$$$$$$        /$$$$$$  /$$$$$$        /$$\  /$$  /$$$$$$   /$$$$$$ \n|__  $$__/| $$  | $$|_  $$_/ /$$__  $$      |_  $$_//$$__  $$      | $$  /$$/ /$$__  $$ /$$__  $$\n   | $$   | $$  | $$  | $$  | $$\ \\__/        | $$  | $$  \\__/      | $$ /$$/ | $$  \\ $$| $$  \\__/\n   | $$\  | $$$$$$$$  | $$  |  $$$$$$         | $$  |  $$$$$$       | $$$$$/  |  $$$$$$/|\ $$$$$$ \n   | $$   | $$__  $$  | $$   \\____  $$        | $$   \\____  $$      |$$  $$   >$$__  $$ \\____  $$\n   | $$   | $$  | $$  | $$   /$$  \\ $$        |$$   /$$  \\ $$      | $$\\  $$ | $$  \\ $$ /$$  \\ $$\n   | $$   | $$  | $$ /$$$$$$|\ $$$$$$/       /$$$$$$|  $$$$$$/      | $$ \\  $$|  $$$$$$/|  $$$$$$/\n   |__/\  |__/  |__/|______/ \\______/       |______/ \\______/       |__/  \\__/ \\______/\ \\______/ \n                                                                                                  \n\                                                                                                 \n\                                                                                                 \n"
kind: ConfigMap
metadata:name: hello-configmap
---
apiVersion: v1
kind: Service
metadata:name: hello-webhook-service
spec:ports:- port: 443protocol: TCPtargetPort: 8000selector:app: hello-webhooktype: ClusterIP
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:name: hello-webhook.leclouddev.com
webhooks:
- admissionReviewVersions:- v1- v1beta1clientConfig:service:name: hello-webhook-servicenamespace: defaultpath: /mutatename: hello-webhook.leclouddev.comobjectSelector:matchLabels:hello: "true"rules:- apiGroups:- ""apiVersions:- v1operations:- CREATEresources:- podsscope: NamespacedsideEffects: NonetimeoutSeconds: 10
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/other]
└─$

简单分析一下 MutatingWebhookConfiguration ,主要的配置 webhooks 配置

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:name: hello-webhook.leclouddev.com
webhooks:
- admissionReviewVersions:- v1- v1beta1clientConfig: #指定了客户端配置，用于指定 Webhook 的服务信息。service:name: hello-webhook-servicenamespace: defaultpath: /mutatename: hello-webhook.leclouddev.com #指定了 Webhook 的名称objectSelector: # 指定了对象选择器，用于选择要应用 Webhook 的对象。matchLabels:hello: "true"rules: #指定了 Webhook 的规则- apiGroups:- ""apiVersions:- v1operations:- CREATEresources:- podsscope: NamespacedsideEffects: NonetimeoutSeconds: 10 #指定了 Webhook 的超时时间，此处为 10 秒。

这里可以看到当前准入控制器 webhook 只处理打了标签 hello=true 的 pod

csr 目录为权限，生成证书的 Job，SA 以及通过 SA 添加对应的集群权限

生成的相关证书,用于 K8s Webhook 通信，相关项目地址：

https://github.com/didil/k8s-webhook-cert-manager

当前作者的 csr-job.yaml 封装的 pod 比较旧，shell 脚本需要调整，所有我们不需要，直接本地生成证书和对应的秘密

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$ls
csr-job.yaml  csr-rolebinding.yaml  csr-role.yaml  csr-sa.yaml  kustomization.yaml
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl kustomize  ./
apiVersion: v1
kind: ServiceAccount
metadata:name: webhook-cert-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: webhook-cert-cluster-role
rules:
- apiGroups:- admissionregistration.k8s.ioresources:- mutatingwebhookconfigurationsverbs:- get- create- patch
- apiGroups:- certificates.k8s.ioresources:- certificatesigningrequestsverbs:- create- get- delete
- apiGroups:- certificates.k8s.ioresources:- certificatesigningrequests/approvalverbs:- update
- apiGroups:- ""resources:- secretsverbs:- create- get- patch
- apiGroups:- ""resources:- configmapsverbs:- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: webhook-cert-cluster-role-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: webhook-cert-cluster-role
subjects:
- kind: ServiceAccountname: webhook-cert-sanamespace: default
---
apiVersion: batch/v1
kind: Job
metadata:name: webhook-cert-setup
spec:backoffLimit: 3template:spec:containers:- args:- --service- hello-webhook-service- --webhook- hello-webhook.leclouddev.com- --secret- hello-tls-secret- --namespace- defaultcommand:- ./generate_certificate.shimage: quay.io/didil/k8s-webhook-cert-manager:0.13.19-1-aname: webhook-cert-setuprestartPolicy: OnFailureserviceAccountName: webhook-cert-sa
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$

部署项目

在部署之前，我们需要改一下命名空间,创建一个新的命名空间，mutating-webhook

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl create  ns mutating-webhook
namespace/mutating-webhook created

然后修改一下每个目录的 kustomization.yaml 文件,类似下面这样，添加 namespace: mutating-webhook

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/other]
└─$cat kustomization.yaml
namespace: mutating-webhook
resources:
- configmap.yaml
- service.yaml
- webhookconf.yaml

然后还需要修改一下证书生成的的命名空间

./generate_certificate.sh  --service hello-webhook-service --webhook hello-webhook.leclouddev.com --secret hello-tls-secret --namespace mutating-webhook

之后就可以部署了

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl apply  -k ./
serviceaccount/webhook-cert-sa created
clusterrole.rbac.authorization.k8s.io/webhook-cert-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/webhook-cert-cluster-role-binding created
job.batch/webhook-cert-setup created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl apply  -k ../other/
configmap/hello-configmap created
service/hello-webhook-service created
mutatingwebhookconfiguration.admissionregistration.k8s.io/hello-webhook.leclouddev.com created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl apply  -k ../deployment/
deployment.apps/hello-webhook-deployment created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl config  set-context --current --namespace mutating-webhook
Context "kubernetes-admin@kubernetes" modified.

webhook 一直没办法正常运行

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl get all
NAME                                            READY   STATUS              RESTARTS      AGE
pod/hello-webhook-deployment-7f599b95c4-pg9w5   0/1     ContainerCreating   0             55s
pod/webhook-cert-setup-v62rt                    0/1     CrashLoopBackOff    2 (15s ago)   79sNAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/hello-webhook-service   ClusterIP   10.103.24.30   <none>        443/TCP   59sNAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-webhook-deployment   0/1     1            0           55sNAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-webhook-deployment-7f599b95c4   1         1         0       55sNAME                           COMPLETIONS   DURATION   AGE
job.batch/webhook-cert-setup   0/1           79s        79s
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$

这里发现作者生成证书的 job 中脚本使用的镜像中的 kubectl 版本太低了，没办法正常执行，所以我这么直接在的本地生成证书，创建对应的 secret ,任然使用作者的脚本generate_certificate.sh

作者原来的证书签名请求对应的 API 资源对象使用的是bata 版本，现在已经更新为正式版本，需要替换一下,其他部分逻辑也需要调整，下面为调整后的

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$cat generate_certificate.sh
#!/usr/bin/env shset -eusage() {cat <<EOF
Generate certificate suitable for use with any Kubernetes Mutating Webhook.
This script uses k8s' CertificateSigningRequest API to a generate a
certificate signed by k8s CA suitable for use with any Kubernetes Mutating Webhook service pod.
This requires permissions to create and approve CSR. See
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
detailed explantion and additional instructions.
The server key/cert k8s CA cert are stored in a k8s secret.
usage: ${0} [OPTIONS]
The following flags are required.--service          Service name of webhook.--webhook          Webhook config name.--namespace        Namespace where webhook service and secret reside.--secret           Secret name for CA certificate and server certificate/key pair.
The following flags are optional.--webhook-kind     Webhook kind, either MutatingWebhookConfiguration orValidatingWebhookConfiguration (defaults to MutatingWebhookConfiguration)
EOFexit 1
}while [ $# -gt 0 ]; docase ${1} in--service)service="$2"shift;;--webhook)webhook="$2"shift;;--secret)secret="$2"shift;;--namespace)namespace="$2"shift;;--webhook-kind)kind="$2"shift;;*)usage;;esacshift
done[ -z "${service}" ] && echo "ERROR: --service flag is required" && exit 1
[ -z "${webhook}" ] && echo "ERROR: --webhook flag is required" && exit 1
[ -z "${secret}" ] && echo "ERROR: --secret flag is required" && exit 1
[ -z "${namespace}" ] && echo "ERROR: --namespace flag is required" && exit 1fullServiceDomain="${service}.${namespace}.svc"# THE CN has a limit of 64 characters. We could remove the namespace and svc
# and rely on the Subject Alternative Name (SAN), but there is a bug in EKS
# that discards the SAN when signing the certificates.
#
# https://github.com/awslabs/amazon-eks-ami/issues/341
if [ ${#fullServiceDomain} -gt 64 ] ; thenecho "ERROR: common name exceeds the 64 character limit: ${fullServiceDomain}"exit 1
fiif [ ! -x "$(command -v openssl)" ]; thenecho "ERROR: openssl not found"exit 1
ficsrName=${service}.${namespace}
tmpdir=$(mktemp -d)
echo "creating certs in tmpdir ${tmpdir} "cat <<EOF >> "${tmpdir}/csr.conf"
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${fullServiceDomain}
DNS.4 = ${fullServiceDomain}.cluster.local
EOF
echo "/CN=${fullServiceDomain}"
openssl genrsa -out "${tmpdir}/server-key.pem" 2048
#openssl req -new -key "${tmpdir}/server-key.pem" -subj "/CN=${fullServiceDomain}" -out "${tmpdir}/server.csr" -config "${tmpdir}/csr.conf"
openssl req -new -key "${tmpdir}/server-key.pem" -subj "/CN=system:node:${fullServiceDomain};/O=system:nodes" -out "${tmpdir}/server.csr" -config "${tmpdir}/csr.conf"
set +e
# clean-up any previously created CSR for our service. Ignore errors if not present.
if kubectl delete csr "${csrName}"; thenecho "WARN: Previous CSR was found and removed."
fi
set -e# create server cert/key CSR and send it to k8s api
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:name: ${csrName}
spec:#signerName: kubernetes.io/kube-apiserver-clientsignerName: kubernetes.io/kubelet-servinggroups:- system:authenticatedrequest: $(base64 < "${tmpdir}/server.csr" | tr -d '\n')usages:- server auth- digital signature- key encipherment
EOFset +e
# verify CSR has been created
while true; doif kubectl get csr "${csrName}"; thenecho "CertificateSigningRequest create succsee"breakfi
done
set -e# approve and fetch the signed certificate . !! not working with k8s 1.19.1, running the command separately outside of the container / node
set +e
while true; doif kubectl certificate approve "${csrName}"; thenecho "${csrName} certificate approve"breakfi
doneset -eset +e
# verify certificate has been signed
i=1
while [ "$i" -ne 10 ]
doserverCert=$(kubectl get csr "${csrName}" -o jsonpath='{.status.certificate}')if [ "${serverCert}" != '' ]; thenbreakfisleep 5i=$((i + 1))
doneset -e
if [ "${serverCert}" = '' ]; thenecho "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2exit 1
fiecho "${serverCert}" | openssl base64 -d -A -out "${tmpdir}/server-cert.pem"# create the secret with CA cert and server cert/key
kubectl create secret tls "${secret}" \--key="${tmpdir}/server-key.pem" \--cert="${tmpdir}/server-cert.pem" \--dry-run -o yaml |kubectl -n "${namespace}" apply -f -#caBundle=$(base64 < /run/secrets/kubernetes.io/serviceaccount/ca.crt  | tr -d '\n')
caBundle=$(cat ${tmpdir}/server-cert.pem)
set +e
# Patch the webhook adding the caBundle. It uses an `add` operation to avoid errors in OpenShift because it doesn't set
# a default value of empty string like Kubernetes. Instead, it doesn't create the caBundle key.
# As the webhook is not created yet (the process should be done manually right after this job is created),
# the job will not end until the webhook is patched.
while true; doecho "INFO: Trying to patch webhook adding the caBundle."if kubectl patch "${kind:-mutatingwebhookconfiguration}" "${webhook}" --type='json' -p "[{'op': 'add', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${serverCert}'}]"; thenbreakfiecho "INFO: webhook not patched. Retrying in 5s..."sleep 5
done
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$

生成证书，并且创建 csr secret ，同时更新 mutatingwebhookconfiguration 的 caBundle 字段

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$./generate_certificate.sh  --service webhook-svc --webhook hello-webhook.leclouddev.com --secret hello-tls-secret --namespace mutating-webhook
creating certs in tmpdir /tmp/tmp.Di367dgyMz
/CN=webhook-svc.mutating-webhook.svc
Generating RSA private key, 2048 bit long modulus
........................................+++
.........+++
e is 65537 (0x10001)
Error from server (NotFound): certificatesigningrequests.certificates.k8s.io "webhook-svc.mutating-webhook" not found
certificatesigningrequest.certificates.k8s.io/webhook-svc.mutating-webhook created
NAME                           AGE   SIGNERNAME                      REQUESTOR          REQUESTEDDURATION   CONDITION
webhook-svc.mutating-webhook   0s    kubernetes.io/kubelet-serving   kubernetes-admin   <none>              Pending
CertificateSigningRequest create succsee
certificatesigningrequest.certificates.k8s.io/webhook-svc.mutating-webhook approved
webhook-svc.mutating-webhook certificate approve
W1115 17:25:40.403890   66211 helpers.go:663] --dry-run is deprecated and can be replaced with --dry-run=client.
secret/hello-tls-secret created
INFO: Trying to patch webhook adding the caBundle.
mutatingwebhookconfiguration.admissionregistration.k8s.io/hello-webhook.leclouddev.com patched

如果命名空间或者 svc 名字太长的话，会报下面的错，需要调整短一点

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$./generate_certificate.sh  --service hello-webhook-service --webhook hello-webhook.leclouddev.com --secret hello-tls-secret --namespace k8s-hello-mutating-webhook
creating certs in tmpdir /tmp/tmp.GcNh5TwKXP
/CN=hello-webhook-service.k8s-hello-mutating-webhook.svc
Generating RSA private key, 2048 bit long modulus
....+++
...................................................+++
e is 65537 (0x10001)
problems making Certificate Request
140682165290896:error:0D07A097:asn1 encoding routines:ASN1_mbstring_ncopy:string too long:a_mbstr.c:158:maxsize=64

测试结果

创建指定标签的 Pod ,自动挂载 CM

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl run busybox-1 --image=busybox  --restart=Never -l=app=busybox,hello=true -- sleep 3600
pod/busybox-1 created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl exec busybox-1 -it -- sh -c "ls /etc/config/hello.txt"
/etc/config/hello.txt
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl exec busybox-1 -it -- sh -c "cat /etc/config/hello.txt"/$$$$$$$$ /$$   /$$ /$$$$$$  /$$$$$$        /$$$$$$  /$$$$$$        /$$   /$$  /$$$$$$   /$$$$$$
|__  $$__/| $$  | $$|_  $$_/ /$$__  $$      |_  $$_/ /$$__  $$      | $$  /$$/ /$$__  $$ /$$__  $$| $$   | $$  | $$  | $$  | $$  \__/        | $$  | $$  \__/      | $$ /$$/ | $$  \ $$| $$  \__/| $$   | $$$$$$$$  | $$  |  $$$$$$         | $$  |  $$$$$$       | $$$$$/  |  $$$$$$/|  $$$$$$| $$   | $$__  $$  | $$   \____  $$        | $$   \____  $$      | $$  $$   >$$__  $$ \____  $$| $$   | $$  | $$  | $$   /$$  \ $$        | $$   /$$  \ $$      | $$\  $$ | $$  \ $$ /$$  \ $$| $$   | $$  | $$ /$$$$$$|  $$$$$$/       /$$$$$$|  $$$$$$/      | $$ \  $$|  $$$$$$/|  $$$$$$/|__/   |__/  |__/|______/ \______/       |______/ \______/       |__/  \__/ \______/  \______/

没有标签的没有自动挂载

┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl run busybox-2 --image=busybox --restart=Never -l=app=busybox -- sleep 3600
pod/busybox-2 created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl get pod -l=app=busybox -L=hello-added
NAME        READY   STATUS    RESTARTS   AGE   HELLO-ADDED
busybox-1   1/1     Running   0          30m   OK
busybox-2   1/1     Running   0          28s
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl exec busybox-2 -it -- sh -c "ls /etc/config/hello.txt"
ls: /etc/config/hello.txt: No such file or directory
command terminated with exit code 1
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$