K8s脱坑前的挣扎——k8s群集部署--多节点及高可用（nginx负载均衡+keepalived双机热备）
·
前言：本次部署将在单节点之上增加一个master以及配置负载均衡和双机热备实现高可用，部署细节可参考前面三张单节点部署系列单节点部署1-Etcd、单节点部署2-Flannel、单节点部署3-Master及Node。
文章目录
一、环境部署
1.节点分配
| 服务器名称 | IP地址 | 分配 |
|---|---|---|
| master01 | 192.168.170.128/24 | etcd，apiserver，controller-manager，scheduler |
| master02 | 192.168.170.129/24 | etcd证书，apiserver，controller-manager，scheduler |
| node1 | 192.168.170.145/24 | etcd+flannel |
| node2 | 192.168.170.136/24 | etcd+flannel |
| lb1 | 192.168.170.134/24 | nginx+keepalived |
| lb2 | 192.168.170.131/24 | nginx+keepalived |
2.拓扑图
- 多节点拓扑图
- 在生产环境当中，大多是采用的多master的群集部署，所以需要建立vip，再加入node节点后只要找vip就行，它会自动会请求某一台固定的master，master里的apiserver会给新添加的节点颁发证书，让节点加入到群集中
- 所有node地址全部指向vip，由vip地址作为调度器去调配给下面的某一台master去管理、调度节点
二、部署
1.master02部署
- 在单节点部署完成后，添加一台master服务器就简单了
1）环境优化
- 关闭防火墙，增强型安全功能，清空防火墙
[root@promote ~]# hostnamectl set-hostname master02
[root@promote ~]# su
[root@master02 ~]# systemctl stop NetworkManager
[root@master02 ~]# systemctl disable NetworkManager
[root@master02 ~]# iptables -F
[root@master02 ~]# systemctl stop firewalld.service
[root@master02 ~]# systemctl disable firewalld.service
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@master02 ~]# setenforce 0
[root@master02 ~]# vim /etc/selinux/config
SELINUX=disabled
2）部署
- 直接在master01上将k8s工作目录以及启动脚本直接复制过来
[root@master01 ~]# scp -r /opt/kubernetes/ root@192.168.170.129:/opt
root@192.168.170.129's password:
token.csv 100% 84 16.6KB/s 00:00
kube-apiserver 100% 939 703.7KB/s 00:00
kube-scheduler 100% 94 106.8KB/s 00:00
kube-controller-manager 100% 483 579.3KB/s 00:00
kubectl 100% 55MB 94.6MB/s 00:00
kube-controller-manager 100% 155MB 88.3MB/s 00:01
kube-scheduler 100% 55MB 89.5MB/s 00:00
kube-apiserver 100% 184MB 94.8MB/s 00:01
ca-key.pem 100% 1675 1.7MB/s 00:00
ca.pem 100% 1359 2.0MB/s 00:00
server-key.pem 100% 1679 1.4MB/s 00:00
server.pem 100% 1643 1.2MB/s 00:00
[root@master02 ~]# cd /opt/kubernetes/
[root@master02 kubernetes]# ls
bin cfg ssl
[root@master02 kubernetes]# ls bin/
kube-apiserver kube-controller-manager kubectl kube-scheduler
[root@master02 kubernetes]# ls cfg/
kube-apiserver kube-controller-manager kube-scheduler token.csv
[root@master02 kubernetes]# ls ssl/
ca-key.pem ca.pem server-key.pem server.pem
//复制master中的三个组件启动脚本
[root@master01 ~]# scp -r /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.170.129:/usr/lib/systemd/system
root@192.168.170.129's password:
kube-apiserver.service 100% 282 374.8KB/s 00:00
kube-controller-manager.service 100% 317 378.9KB/s 00:00
kube-scheduler.service 100% 281 327.8KB/s 00:00
[root@master02 opt]# ls /usr/lib/systemd/system | grep kube
kube-apiserver.service
kube-controller-manager.service
kube-scheduler.service
- 修改地址
//修改配置文件kube-apiserver中的ip
[root@master02 opt]# cd /opt/kubernetes/cfg/
[root@master02 cfg]# vim kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.170.128:2379,https://192.168.170.145:2379,https://192.168.170.136:2379 \
--bind-address=192.168.170.129 \ //指向自己
--secure-port=6443 \
--advertise-address=192.168.170.129 \ //指向自己
- 启动apiserver服务需要etcd证书，所以将master01上的证书复制过来
//拷贝master01上已有的etcd证书给master02使用
[root@master01 ~]# scp -r /opt/etcd/ root@192.168.170.129:/opt/
root@192.168.170.129's password:
etcd 100% 523 158.9KB/s 00:00
etcd 100% 18MB 70.6MB/s 00:00
etcdctl 100% 15MB 84.6MB/s 00:00
ca-key.pem 100% 1675 591.2KB/s 00:00
ca.pem 100% 1265 414.4KB/s 00:00
server-key.pem 100% 1679 1.5MB/s 00:00
server.pem 100% 1338 1.9MB/s 00:00
[root@master02 cfg]# ls /opt/etcd/
bin cfg ssl
3）启动
- 启动服务并增加环境变量使系统能够识别
//启动master02中的三个组件服务
[root@master02 cfg]# systemctl start kube-apiserver.service
[root@master02 cfg]# systemctl start kube-controller-manager.service
[root@master02 cfg]# systemctl start kube-scheduler.service
[root@master02 cfg]# systemctl enable kube-apiserver.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
[root@master02 cfg]# systemctl enable kube-controller-manager.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
[root@master02 cfg]# systemctl enable kube-scheduler.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
//增加环境变量
[root@master02 cfg]# vim /etc/profile
export PATH=$PATH:/opt/kubernetes/bin/
[root@master02 cfg]# source /etc/profile
- 查看群集
[root@master02 cfg]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.170.136 Ready <none> 1h v1.12.3
192.168.170.145 Ready <none> 2h v1.12.3
- master02部署成功
2.部署负载均衡及双机热备
1）负载均衡
- Kubernetes在设计之初就充分考虑了针对容器的服务发现与负载均衡机制，提供了Service资源，并通过kube-proxy配合cloud provider来适应不同的应用场景。随着kubernetes用户的激增，用户场景的不断丰富，产生了一些新的负载均衡机制
1.1 环境优化
- 优化两台服务器lb1和lb2，操作相同，这里只展示lb1配置
[root@promote ~]# hostnamectl set-hostname lb1
[root@promote ~]# su
[root@lb1 ~]# systemctl stop firewalld.service
[root@lb1 ~]# systemctl disable firewalld.service
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@lb1 ~]# setenforce 0
[root@lb1 ~]# vim /etc/selinux/config
SELINUX=disabled
[root@lb1 ~]# iptables -F
- 构建yum源环境
[root@lb1 ~]# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
[root@lb1 ~]# yum list //重新加载
[root@lb1 ~]# yum install nginx -y
- 部署模块，实现负载均衡
//添加四层转发
[root@lb1 ~]# vim /etc/nginx/nginx.conf //添加stream模块
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent'; //加载日志文件的格式
access_log /var/log/nginx/k8s-access.log main; //日志文件路径
upstream k8s-apiserver { //连接的master节点
server 192.168.170.128:6443;
server 192.168.170.129:6443;
}
server {
listen 6443;
proxy_pass k8s-apiserver;
}
}
- 启动并检查
[root@lb1 ~]# nginx -t //验证语法格式是否正确
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
//更改访问内容
//lb1
[root@lb1 ~]# cd /usr/share/nginx/html/
[root@lb1 html]# ls
50x.html index.html
[root@lb1 html]# vim index.html
<h1>Welcome to master nginx!</h1>
[root@lb1 html]# systemctl start nginx
//lb2
[root@lb1 ~]# cd /usr/share/nginx/html/
[root@lb1 html]# ls
50x.html index.html
[root@lb1 html]# vim index.html
<h1>Welcome to backup nginx!</h1>
[root@lb1 html]# systemctl start nginx
- 访问lb1
- 访问lb2
2）双机热备
2.1 部署
- 部署keepalived服务，操作大致相同，这里仅展示lb1操作
[root@lb1 html]# yum install keepalived -y
[root@lb1 ~]# cp keepalived.conf /etc/keepalived/keepalived.conf
cp：是否覆盖"/etc/keepalived/keepalived.conf"？ yes
//修改配置文件
[root@lb1 ~]# vim /etc/keepalived/keepalived.conf
vrrp_script check_nginx {
script "/etc/nginx/check_nginx.sh"
}
vrrp_instance VI_1 {
state MASTER #状态为master，lb2设置为BACKUP
interface ens33 ##接口eth0改为ens33
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 100 # 优先级，备服务器设置 90 lb2设置为90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.170.100/24 //VIP地址
}
track_script {
check_nginx
}
}
//删除下面所有内容
mkdir /usr/local/nginx/sbin/ -p
vim /usr/local/nginx/sbin/check_nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
/etc/init.d/keepalived stop
fi
chmod +x /usr/local/nginx/sbin/check_nginx.sh
- 编写统计nginx进程脚本
//监控信息，统计nginx进程，如果为0，则关闭keeplived服务，实现地址漂移
[root@lb1 ~]# vim /etc/nginx/check_nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi
[root@lb1 ~]# chmod +x /etc/nginx/check_nginx.sh
[root@lb1 ~]# systemctl start keepalived.service
[root@lb1 ~]# ip a
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:20:3c:4a brd ff:ff:ff:ff:ff:ff
inet 192.168.170.134/24 brd 192.168.170.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.170.100/24 scope global secondary ens33 //可以看到VIP地址
。。。省略部分内容
- 可以访问VIP地址查看是否在lb1节点上
2.2 模拟故障
- kill掉nginx服务
[root@lb1 ~]# pkill nginx
[root@lb1 ~]# systemctl status nginx //查看服务状态为关闭
● nginx.service - nginx - high performance web server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since 二 2020-05-05 19:05:58 CST; 7s ago
[root@lb1 ~]# ip a //可以看到VIP地址没了
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:20:3c:4a brd ff:ff:ff:ff:ff:ff
inet 192.168.170.134/24 brd 192.168.170.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe20:3c4a/64 scope link
valid_lft forever preferred_lft forever
[root@lb1 ~]# systemctl status keepalived.service //keepalived服务也挂了，说明之前的脚本奏效了
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: inactive (dead)
- 进入lb2服务器查看地址
[root@lb2 ~]# ip a //可以看到VIP地址转移过来了
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:83:95:21 brd ff:ff:ff:ff:ff:ff
inet 192.168.170.131/24 brd 192.168.170.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.170.100/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::a3a8:f538:1fef:ae39/64 scope link
valid_lft forever preferred_lft forever
2.3 恢复
- 恢复步骤：先恢复nginx，再恢复keepalived
[root@lb1 ~]# systemctl start nginx
[root@lb1 ~]# systemctl start keepalived.service
[root@lb1 ~]# ip a
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:20:3c:4a brd ff:ff:ff:ff:ff:ff
inet 192.168.170.134/24 brd 192.168.170.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.170.100/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe20:3c4a/64 scope link
valid_lft forever preferred_lft forever
//优先级高，所以又回来了
2.4 node节点绑定漂移地址
- node1和node2节点配置相同，这里仅展示node1操作
//开始修改node节点配置文件统一VIP
[root@node1 ~]# vim /opt/kubernetes/cfg/bootstrap.kubeconfig
[root@node1 ~]# vim /opt/kubernetes/cfg/kubelet.kubeconfig
[root@node1 ~]# vim /opt/kubernetes/cfg/kube-proxy.kubeconfig
server: https://192.168.170.100:6443 //地址都修改为VIP地址
[root@node1 ~]# systemctl restart kubelet.service
[root@node1 ~]# systemctl restart kube-proxy.service
[root@node1 ~]# cat /opt/kubernetes/cfg/bootstrap.kubeconfig | grep 100
server: https://192.168.170.100:6443
[root@node1 ~]# cat /opt/kubernetes/cfg/kubelet.kubeconfig | grep 100
server: https://192.168.170.100:6443
[root@node1 ~]# cat /opt/kubernetes/cfg/kube-proxy.kubeconfig | grep 100
server: https://192.168.170.100:6443
- 在lb1上查看ningx的k8s日志
[root@lb1 ~]# tail /var/log/nginx/k8s-access.log //采用轮询调度算法
192.168.170.145 192.168.170.128:6443 - [05/May/2020:19:21:28 +0800] 200 1121
192.168.170.145 192.168.170.129:6443 - [05/May/2020:19:21:28 +0800] 200 1122
192.168.170.136 192.168.170.128:6443 - [05/May/2020:19:28:51 +0800] 200 1120
192.168.170.136 192.168.170.129:6443 - [05/May/2020:19:28:51 +0800] 200 1121
3.测试
- 在master1上创建一个pod
[root@master01 ~]# kubectl run nginx1 --image=nginx //在线下载
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx created
[root@master01 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-dbddb74b8-qj8c8 0/1 ContainerCreating 0 10s //正在创建中
[root@master01 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE //创建完成
nginx-dbddb74b8-qj8c8 1/1 Running 0 20s
- 查看日志
[root@master01 ~]# kubectl logs nginx-dbddb74b8-qj8c8
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx1-84ccd956fb-xl7ws)
//这里可以看到，报了error
//需要开启权限，绑定，绑定的是cluster-system系统群集中的一个匿名用户，赋予管理员权限，指令的用户为匿名
[root@master01 ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created
//查看pod网络
[root@master01 ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
nginx-dbddb74b8-qj8c8 1/1 Running 1 4h36m 172.17.7.2 192.168.170.145 <none>
[root@master01 ~]# kubectl logs nginx-dbddb74b8-qj8c8 //日志文件是空的，但是能正常访问了
- 在对应网段的node节点上操作可以直接访问，这里是node1节点
//在对应网段的node节点上操作可以直接访问
[root@node1 ~]# curl 172.17.7.2 //可以看到nginx的信息
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
//访问就会产生日志
[root@master01 ~]# kubectl logs nginx-dbddb74b8-qj8c8
172.17.7.1 - - [05/May/2020:16:15:15 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
2020/05/05 16:15:15 [error] 6#6: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.17.7.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "172.17.7.2"
172.17.7.1 - - [05/May/2020:16:15:15 +0000] "GET /favicon.ico HTTP/1.1" 404 154 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
172.17.12.0 - - [05/May/2020:16:16:19 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
2020/05/05 16:16:19 [error] 6#6: *2 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.17.12.0, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "172.17.7.2"
172.17.12.0 - - [05/May/2020:16:16:19 +0000] "GET /favicon.ico HTTP/1.1" 404 154 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
172.17.7.1 - - [05/May/2020:16:20:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" "-"
- 也可以在节点服务器上访问，因为在两个node节点上都做过flannel网络，所以都可以访问
- 至此，k8s的多节点部署以及负载均衡和双机热备实现高可用结束
北京人形旗下天工造物具身智能开源社区，聚焦具身天工与慧思开物两大平台
更多推荐
0
0- 0
已为社区贡献1条内容
所有评论(0)