K8s高可用部署至少需要2台Master主机，为此我们复用上次做etcd集群的4台机器，2台做Master，2台做Worker。也就是说要完成k8s集群的部署，先完成etcd集群的独立部署工作。
参考以前的文章：
/2020/12/09121959-k8s-config.html
/2021/04/23183930-k8s-etcd.html
环境准备
我们用4台centos8.3虚拟机来做配置试验，每台服务都做下面的配置：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
10.10.200.146 DXHJZW146 （master）
10.10.200.147 DXHJZW147 （master）
10.10.200.148 DXHJZW148 （worker）
10.10.200.149 DXHJZW149 （worker）
# 1. 两台master上安装下面软件
yum install -y keepalived
yum install -y haproxy
# 添加ipvs支持
yum install -y nfs-utils ipset ipvsadm
# 2. 下载证书生成工具
https://github.com/cloudflare/cfssl/releases
mv cfssl_1.5.0_linux_amd64 cfssl
mv cfssljson_1.5.0_linux_amd64 cfssljson
mv cfssl-certinfo_1.5.0_linux_amd64 cfssl-certinfo
# 下载相应平台最新版解压之后，将 cfssl、cfssljson、cfssl-certinfo 放入 /usr/bin
# 可执行权限 chmod +x /usr/bin/cfssl*
# 3. 新建立目录
mkdir /opt/etcd/ssl
mkdir /opt/etcd/cfg
mkdir /opt/etcd/data
mkdir /opt/etcd/check
|
配置外部ETCD集群的方法参考以前的文章。
注意事项
要想做K8s HA部署，一般启用 ipvs 做路由处理，两台K8s Master服务器一定要启用相关模块：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
# 安装
yum install -y nfs-utils ipset ipvsadm # 安装ipvs支持
# 配置
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
modprobe -- nf_conntrack
EOF
# 执行下面三条，2条配置1条检查
chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules
lsmod | grep -e ip_vs -e nf_conntrack_ipv4
# 或者 centos 8.3 的系统，用下面检测
lsmod | grep -e ip_vs -e nf_conntrack
|
因为内核版本高（centos 8.3以上）的系统不会再有nf_conntrack_ipv4模块， 它和 nf_conntrack_ipv6合并了，统一到nf_conntrack中了。
初始化第一个Master
先构建下面的初始化配置文件：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
|
# vi k8s-init.yml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 10.10.200.146
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: DXHJZW146
taints: null
---
apiServer:
certSANs:
- DXHJZW145
- DXHJZW146
- DXHJZW147
- DXHJZW148
- DXHJZW149
- DXHJZW157
- 127.0.0.1
- 10.10.200.145
- 10.10.200.146
- 10.10.200.147
- 10.10.200.148
- 10.10.200.149
- cluster.local
- kubernetes
- kubernetes.default
- kubernetes.default.svc
- kubernetes.default.svc.cluster.local
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: { }
dns:
type: CoreDNS
etcd:
external:
endpoints:
- https://10.10.200.146:2379
- https://10.10.200.147:2379
- https://10.10.200.148:2379
- https://10.10.200.149:2379
caFile: /opt/etcd/ssl/ca.pem
certFile: /opt/etcd/ssl/server.pem
keyFile: /opt/etcd/ssl/server-key.pem
imageRepository: 10.10.200.11:5000
kind: ClusterConfiguration
kubernetesVersion: 1.21.0
controlPlaneEndpoint: 10.10.200.145:6444
networking:
dnsDomain: cluster.local
podSubnet: 10.243.0.0/16
serviceSubnet: 10.19.0.0/16
scheduler: { }
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
## 1. 开启 IPVS 模式
## （以下6行适合 k8s 1.19前版本）
#---
#apiVersion: kubeproxy.config.k8s.io/v1alpha1
#kind: KubeProxyConfiguration
#featureGates:
# SupportIPVSProxyMode: true
#mode: ipvs
## 2. （以下5行适合 k8s 1.20之后版本）
#...
#kubeProxy:
# config:
# mode: ipvs
#...
|
执行K8s初始化：
1
|
kubeadm init --config=k8s-init.yml
|
成功之后按提示做好设置即可。
特别注意
上面配置文件末位给出的两种配置都不正确，只能用下面列出的4行才可以正确启动 K8s IPVS 模式。
1
2
3
4
|
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
|
安装网络插件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
|
# vi kube-flannel.yml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.243.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: 10.10.200.11:5000/coreos/flannel:v0.14.0-rc1
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: 10.10.200.11:5000/coreos/flannel:v0.14.0-rc1
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface-regex="eth*|ens*|eno*"
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
|
添加网络插件：
1
|
kubectl apply -f kube-flannel.yml
|
添加Master节点
先将第一个Master中 /etc/kubernetes/pki/*所有文件copy到新Master节点对应目录中 /etc/kubernetes/pki/。之后再执行下面的命令：
kubeadm join 10.10.200.145:6444 --token abcdef.0123456789abcdef \
--apiserver-advertise-address 10.10.200.147 \
--discovery-token-ca-cert-hash sha256:xxxxxx \
--control-plane
添加Worker节点
添加Worker节点非常简单，只需要节点装好了docker和kubeadm相关程序，用下面的命令执行即可。
kubeadm join 10.10.200.145:6444 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:xxxxxx
之后需要将master节点中的 /etc/kubernetes/admin.conf 加入worker节点中对应目录，即可实现worker节点查询集群的状态。
常见问题
上面的方法都没有解决问题，看下面的根本解法：
https://blog.csdn.net/qingyafan/article/details/93519196
https://blog.csdn.net/kenkao/article/details/83960951
https://blog.csdn.net/lswzw/article/details/102543498
其它参考：
https://blog.csdn.net/lswzw/article/details/109027255
https://blog.51cto.com/u_3241766/2467865
https://blog.51cto.com/u_3241766/2094750
https://www.cnblogs.com/ants/p/11489598.html
（完）