K8s高可用部署至少需要2台Master主机,为此我们复用上次做etcd集群的4台机器,2台做Master,2台做Worker。也就是说要完成k8s集群的部署,先完成etcd集群的独立部署工作。

参考以前的文章:

/2020/12/09121959-k8s-config.html

/2021/04/23183930-k8s-etcd.html

环境准备

我们用4台centos8.3虚拟机来做配置试验,每台服务都做下面的配置:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
10.10.200.146 DXHJZW146 (master)
10.10.200.147 DXHJZW147 (master)
10.10.200.148 DXHJZW148 (worker)
10.10.200.149 DXHJZW149 (worker)

# 1. 两台master上安装下面软件
yum install -y keepalived
yum install -y haproxy
# 添加ipvs支持
yum install -y nfs-utils ipset ipvsadm

# 2. 下载证书生成工具
https://github.com/cloudflare/cfssl/releases
mv cfssl_1.5.0_linux_amd64 cfssl
mv cfssljson_1.5.0_linux_amd64 cfssljson
mv cfssl-certinfo_1.5.0_linux_amd64 cfssl-certinfo
# 下载相应平台最新版解压之后,将 cfssl、cfssljson、cfssl-certinfo 放入 /usr/bin
# 可执行权限 chmod +x /usr/bin/cfssl*

# 3. 新建立目录
mkdir /opt/etcd/ssl
mkdir /opt/etcd/cfg
mkdir /opt/etcd/data
mkdir /opt/etcd/check

配置外部ETCD集群的方法参考以前的文章。

注意事项

要想做K8s HA部署,一般启用 ipvs 做路由处理,两台K8s Master服务器一定要启用相关模块:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# 安装
yum install -y nfs-utils ipset ipvsadm # 安装ipvs支持

# 配置
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
modprobe -- nf_conntrack
EOF

# 执行下面三条,2条配置1条检查
chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules 
lsmod | grep -e ip_vs -e nf_conntrack_ipv4
# 或者 centos 8.3 的系统,用下面检测
lsmod | grep -e ip_vs -e nf_conntrack

因为内核版本高(centos 8.3以上)的系统不会再有nf_conntrack_ipv4模块, 它和 nf_conntrack_ipv6合并了,统一到nf_conntrack中了。

初始化第一个Master

先构建下面的初始化配置文件:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
# vi k8s-init.yml

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
  - groups:
      - system:bootstrappers:kubeadm:default-node-token
    token: abcdef.0123456789abcdef
    ttl: 24h0m0s
    usages:
      - signing
      - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.10.200.146
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: DXHJZW146
  taints: null
---
apiServer:
  certSANs:
    - DXHJZW145
    - DXHJZW146
    - DXHJZW147
    - DXHJZW148
    - DXHJZW149
    - DXHJZW157
    - 127.0.0.1
    - 10.10.200.145
    - 10.10.200.146
    - 10.10.200.147
    - 10.10.200.148
    - 10.10.200.149
    - cluster.local
    - kubernetes
    - kubernetes.default
    - kubernetes.default.svc
    - kubernetes.default.svc.cluster.local
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: { }
dns:
  type: CoreDNS
etcd:
  external:
    endpoints:
      - https://10.10.200.146:2379
      - https://10.10.200.147:2379
      - https://10.10.200.148:2379
      - https://10.10.200.149:2379
    caFile: /opt/etcd/ssl/ca.pem
    certFile: /opt/etcd/ssl/server.pem
    keyFile: /opt/etcd/ssl/server-key.pem
imageRepository: 10.10.200.11:5000
kind: ClusterConfiguration
kubernetesVersion: 1.21.0
controlPlaneEndpoint: 10.10.200.145:6444
networking:
  dnsDomain: cluster.local
  podSubnet: 10.243.0.0/16
  serviceSubnet: 10.19.0.0/16
scheduler: { }
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

## 1. 开启 IPVS 模式
##  (以下6行适合 k8s 1.19前版本)
#---
#apiVersion: kubeproxy.config.k8s.io/v1alpha1
#kind: KubeProxyConfiguration
#featureGates:
#  SupportIPVSProxyMode: true
#mode: ipvs

## 2. (以下5行适合 k8s 1.20之后版本)
#...
#kubeProxy:
#  config:
#    mode: ipvs
#...

执行K8s初始化:

1
kubeadm init --config=k8s-init.yml

成功之后按提示做好设置即可。

特别注意

上面配置文件末位给出的两种配置都不正确,只能用下面列出的4行才可以正确启动 K8s IPVS 模式。

1
2
3
4
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

安装网络插件

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
# vi kube-flannel.yml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }    
  net-conf.json: |
    {
      "Network": "10.243.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }    
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: 10.10.200.11:5000/coreos/flannel:v0.14.0-rc1
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: 10.10.200.11:5000/coreos/flannel:v0.14.0-rc1
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface-regex="eth*|ens*|eno*"
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

添加网络插件:

1
kubectl apply -f kube-flannel.yml

添加Master节点

先将第一个Master中 /etc/kubernetes/pki/*所有文件copy到新Master节点对应目录中 /etc/kubernetes/pki/。之后再执行下面的命令:

kubeadm join 10.10.200.145:6444 --token abcdef.0123456789abcdef \
--apiserver-advertise-address 10.10.200.147 \
--discovery-token-ca-cert-hash sha256:xxxxxx \
--control-plane

添加Worker节点

添加Worker节点非常简单,只需要节点装好了docker和kubeadm相关程序,用下面的命令执行即可。

kubeadm join 10.10.200.145:6444 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:xxxxxx

之后需要将master节点中的 /etc/kubernetes/admin.conf 加入worker节点中对应目录,即可实现worker节点查询集群的状态。

常见问题

上面的方法都没有解决问题,看下面的根本解法:

https://blog.csdn.net/qingyafan/article/details/93519196

https://blog.csdn.net/kenkao/article/details/83960951

https://blog.csdn.net/lswzw/article/details/102543498

其它参考:

https://blog.csdn.net/lswzw/article/details/109027255

https://blog.51cto.com/u_3241766/2467865

https://blog.51cto.com/u_3241766/2094750

https://www.cnblogs.com/ants/p/11489598.html

(完)