为Kubernetes集群添加NFS provisioner-GX博客-分享个人 Full-Stack JavaScript 项目开发经验、前端在线工具、Sass、Bootstrap、Grunt、Gulp、webpack、React、Node.js、Git、Koa、MySQL、Nginx、Docker、Kubernetes、Jenkins、SEO、网络安全、CentOS 7

为Kubernetes集群添加NFS provisioner

创建 2020-07-09 01:18，修改 2020-08-29 11:45，作者 leeguangxing.cn，阅读数：1075

Kubernetes 提供了动态卷配置 StorageClass 和卷声明 PersistentVolumeClaim 来将 Pod 与挂载卷解耦，将挂载卷与集群解耦。

一般地，云上 Kubernetes 方案都会配备各自的持久磁盘置备程序 provisioner。不过本文要介绍的是在小型集群中手动搭建 NFS provisioner，以供各节点上的应用程序持久化存储需求。

网上有一些解决方案，但不完备，有些甚至有些过时，所以在这里作一个补充。

首先要注意，本文的实现环境如下：

CentOS 7
kubernetes 集群 v1.18.4（由 kubeadm 搭建）
kubernetes_incubator/nfs-provisioner 镜像 v2.3.0
CentOS 7 默认使用 NFS V4 协议，NFS 客户端由 nfs-utils、rpcbind 工具包提供

下面开始介绍部署步骤。

添加 ServiceAccount 并绑定权限角色

后面 provisioner 将会部署在默认的 default 命名空间下：

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-provisioner
  namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: nfs-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "endpoints"]
    verbs: ["get"]
  - apiGroups: ["extensions"]
    resources: ["podsecuritypolicies"]
    resourceNames: ["nfs-provisioner"]
    verbs: ["use"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: run-nfs-provisioner
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: leader-locking-nfs-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: leader-locking-nfs-provisioner
roleRef:
  kind: Role
  name: leader-locking-nfs-provisioner
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    namespace: default

---

部署 provisioner Deployment

以 Pod 形式部署 provisioner 到指定节点上：

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-provisioner
  template:
    metadata:
      labels:
        app: nfs-provisioner
    spec:
      serviceAccountName: nfs-provisioner
      # 通过节点选择器将 provisioner 的 Pod 部署到指定节点
      nodeSelector:
        role: nfs-provisioner
      containers:
        - name: nfs-provisioner
          # 写这文章时，镜像最新版本为 v2.3.0
          image: quay.io/kubernetes_incubator/nfs-provisioner:v2.3.0
          ports:
            - name: nfs
              containerPort: 2049
            - name: nfs-udp
              containerPort: 2049
              protocol: UDP
            - name: mountd
              containerPort: 20048
            - name: mountd-udp
              containerPort: 20048
              protocol: UDP
            - name: rpcbind
              containerPort: 111
            - name: rpcbind-udp
              containerPort: 111
              protocol: UDP
            - name: port-a
              containerPort: 662
            - name: port-a-udp
              containerPort: 662
              protocol: UDP
            - name: port-b
              containerPort: 875
            - name: port-b-udp
              containerPort: 875
              protocol: UDP
            - name: port-c
              containerPort: 32803
            - name: port-c-udp
              containerPort: 32803
              protocol: UDP
          # 在容器安全上下文中添加相应内核功能
          securityContext:
            capabilities:
              add:
                - DAC_READ_SEARCH
                - SYS_RESOURCE
          args:
            # 该参数与 StorageClass 的 provisioner 字段相对应
            - "-provisioner=leeguangxing.cn/nfs"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: SERVICE_NAME
              value: nfs-provisioner
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: export-volume
              mountPath: /export
      volumes:
        - name: export-volume
          hostPath:
            # 节点主机的 NFS 挂载目录
            path: /var/storage/dynamic

---

要查看最新 nfs-provisioner 镜像版本，请点击这里。

公开 provisioner 服务

将 provisioner 以集群服务形式公开：

---

apiVersion: v1
kind: Service
metadata:
  name: nfs-provisioner
  labels:
    app: nfs-provisioner
spec:
  # 端口问题
  # https://github.com/kubernetes-incubator/external-storage/issues/1262
  ports:
    - name: nfs
      port: 2049
    - name: nfs-udp
      port: 2049
      protocol: UDP
    - name: mountd
      port: 20048
    - name: mountd-udp
      port: 20048
      protocol: UDP
    - name: rpcbind
      port: 111
    - name: rpcbind-udp
      port: 111
      protocol: UDP
    - name: port-a
      port: 662
    - name: port-a-udp
      port: 662
      protocol: UDP
    - name: port-b
      port: 875
    - name: port-b-udp
      port: 875
      protocol: UDP
    - name: port-c
      port: 32803
    - name: port-c-udp
      port: 32803
      protocol: UDP
  selector:
    app: nfs-provisioner

---

github 上的 deployment.yaml 已经过时，除 NFS 相关端口外，还遗漏了部分端口未公开，导致 Pod 运行日志上的错误，要查看具体 issue，请点击这里。

添加动态卷配置和卷声明

StorageClass：

---

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: nfs-dynamic
provisioner: leeguangxing.cn/nfs

---

测试用的 PersistentVolumeClaim（大小为 100M）：

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-pvc
spec:
  resources:
    requests:
      storage: 100Mi
  accessModes:
    - ReadWriteMany
  storageClassName: nfs-dynamic

---

挂载卷的 yaml 片段：

spec:
  volumes:
    - name: view
      persistentVolumeClaim:
        claimName: nfs-pvc
  containers:
    - volumeMounts:
        - name: view
          mountPath: /var/web/views
          subPath: ""

到这里为止，我们就完成了 NFS provisioner 的部署和具体挂载。

一篇关于 IBM 的 NFS provisioner 部署的介绍文章，可点击这里查看。
文中使用的 nfs-provisioner 镜像的 github 地址，可点击这里查看。

GX博客