【go-zero】api与rpc使用k8s服务发现和部署

k8s安装

参考
debian12极简快速安装k8s
【Kubernetes】linux centos安装部署Kubernetes集群

1、代码准备

参考上一篇【go-zero】api与rpc使用etcd服务发现 代码搬过来，因为要用到k8s环境换到linux了 ，做一下修改

rpc

修改一下rpc的logic文件，增加返回主机名，用于后面查看负载均衡效果

func (l *GetUserInfoLogic) GetUserInfo(in *pb.GetUserInfoReq) (*pb.GetUserInfoResp, error) {
	hostname, _ := os.Hostname()
	return &pb.GetUserInfoResp{
		Id:       in.Id,
		Nickname: "zero-rpc----->" + hostname,
	}, nil
}

user.yaml 文件

Name: user.rpc
ListenOn: 0.0.0.0:8980
Mode: dev

api 配置rpc的发现

user-api.api 文件
user-rpc-svc 是rpc的k8s yaml部署文件的serviceName

Name: user-api
Host: 0.0.0.0
Port: 8988
UserRpcConf:
  Target: k8s://go-zero-demo/user-rpc-svc:8980

2、使用goctl 生成rpc、api 的 Dockerfile，通过Dockerfile构建镜像

user-rpc目录下

goctl docker -go user.go

根据提示，执行 docker build 要在/home/goproject/domo目录下 我们把 Dockerfile 复制到/home/goproject/domo

cp Dockerfile ../Dockerfile

/home/goproject/domo目录下 执行 docker构建镜像

docker build -t gozerok8s-user-rpc:v1 .

执行完查看 docker images

user-api目录下

goctl docker -go user.go

生成成功 把Dockerfile挪到/home/goproject/domo目录下

构建镜像

docker build -t gozerok8s-user-api:v1 .

镜像构建成功

3、创建serviceAccount账号，要具有k8s内部的endpoints资源的list、watch、get权限

k8s创建命名空间

[root@k8smaster k8s]# kubectl create namespace go-zero-demo
namespace/go-zero-demo created
[root@k8smaster k8s]# 
[root@k8smaster k8s]# kubectl apply -f auth.yaml 
serviceaccount/find-endpoints created
clusterrole.rbac.authorization.k8s.io/discov-endpoints unchanged
clusterrolebinding.rbac.authorization.k8s.io/find-endpoints-discov-endpoints unchanged
[root@k8smaster k8s]# 

auth.yaml文件

#创建账号
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: go-zero-demo
  name: find-endpoints

#创建角色对应操作
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: discov-endpoints
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get","list","watch"]

#给账号绑定角色
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: find-endpoints-discov-endpoints
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: discov-endpoints
subjects:
- kind: ServiceAccount
  name: find-endpoints
  namespace: go-zero-demo

查看命名空间

[root@k8smaster k8s]# kubectl get sa -n go-zero-demo
NAME             SECRETS   AGE
default          0         3m26s
find-endpoints   0         3m24s

4、使用goctl生成k8s的 rpc、api yaml部署文件，制定创建好的serviceAccount

1、user-rpc 目录下执行

goctl kube deploy -replicas 3 -requestCpu 200 -requestMem 50 -limitCpu 300 -limitMem 100 -name user-rpc -namespace go-zero-demo -image gozerok8s-user-rpc:v1 -o user-rpc.yaml -port 8980 --serviceAccount find-endpoints

[root@k8smaster user-rpc]# goctl kube deploy -replicas 3 -requestCpu 200 -requestMem 50 -limitCpu 300 -limitMem 100 -name user-rpc -namespace go-zero-demo -image gozerok8s-user-rpc:v1 -o user-rpc.yaml -port 8980 --serviceAccount find-endpoints
Done.

user-rpc.yaml 文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-rpc
  namespace: go-zero-demo
  labels:
    app: user-rpc
spec:
  replicas: 3
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: user-rpc
  template:
    metadata:
      labels:
        app: user-rpc
    spec:
      serviceAccountName: find-endpoints
      containers:
      - name: user-rpc
        image: gozerok8s-user-rpc:v1
        ports:
        - containerPort: 8980
        readinessProbe:
          tcpSocket:
            port: 8980
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8980
          initialDelaySeconds: 15
          periodSeconds: 20
        resources:
          requests:
            cpu: 200m
            memory: 50Mi
          limits:
            cpu: 300m
            memory: 100Mi
        volumeMounts:
        - name: timezone
          mountPath: /etc/localtime
      volumes:
        - name: timezone
          hostPath:
            path: /usr/share/zoneinfo/Asia/Shanghai

---

apiVersion: v1
kind: Service
metadata:
  name: user-rpc-svc
  namespace: go-zero-demo
spec:
  ports:
  - port: 8980
    targetPort: 8980
  selector:
    app: user-rpc

---

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: user-rpc-hpa-c
  namespace: go-zero-demo
  labels:
    app: user-rpc-hpa-c
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: user-rpc
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

---

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: user-rpc-hpa-m
  namespace: go-zero-demo
  labels:
    app: user-rpc-hpa-m
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: user-rpc
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

遇到问题，HorizontalPodAutoscaler ，autoscaling/v2beta2 这个版本不存在

[root@k8smaster user-rpc]# kubectl apply -f user-rpc.yaml 
deployment.apps/user-rpc created
service/user-rpc-svc created
resource mapping not found for name: "user-rpc-hpa-c" namespace: "go-zero-demo" from "user-rpc.yaml": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2"
ensure CRDs are installed first
resource mapping not found for name: "user-rpc-hpa-m" namespace: "go-zero-demo" from "user-rpc.yaml": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2"
ensure CRDs are installed first

查询 HorizontalPodAutoscaler 的版本，是v2，修改下user-rpc.yaml文件，把 v2beta2 改成v2

kubectl explain HorizontalPodAutoscaler

[root@k8smaster user-rpc]# kubectl explain HorizontalPodAutoscaler
GROUP:      autoscaling
KIND:       HorizontalPodAutoscaler
VERSION:    v2

DESCRIPTION:
    HorizontalPodAutoscaler is the configuration for a horizontal pod
    autoscaler, which automatically manages the replica count of any resource
    implementing the scale subresource based on the metrics specified.
    
FIELDS:
  apiVersion	<string>
    APIVersion defines the versioned schema of this representation of an object.
    Servers should convert recognized schemas to the latest internal value, and
    may reject unrecognized values. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

  kind	<string>
    Kind is a string value representing the REST resource this object
    represents. Servers may infer this from the endpoint the client submits
    requests to. Cannot be updated. In CamelCase. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

  metadata	<ObjectMeta>
    metadata is the standard object metadata. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

  spec	<HorizontalPodAutoscalerSpec>
    spec is the specification for the behaviour of the autoscaler. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status.

  status	<HorizontalPodAutoscalerStatus>
    status is the current information about the autoscaler.

执行成功

[root@k8smaster user-rpc]# kubectl apply -f user-rpc.yaml 
deployment.apps/user-rpc unchanged
service/user-rpc-svc unchanged
horizontalpodautoscaler.autoscaling/user-rpc-hpa-c created
horizontalpodautoscaler.autoscaling/user-rpc-hpa-m created

2、user-api 目录下执行

nodePort 暴露到外面的端口
-port 8988 api服务的端口

goctl kube deploy -nodePort 32000 -replicas 3 -requestCpu 200 -requestMem 50 -limitCpu 300 -limitMem 100 -name user-api -namespace go-zero-demo -image gozerok8s-user-api:v1 -o user-api.yaml -port 8988 --serviceAccount find-endpoints

编辑user-api.yaml 文件，改成v2

部署

[root@k8smaster user-api]# kubectl apply -f user-api.yaml 
deployment.apps/user-api created
service/user-api-svc created
horizontalpodautoscaler.autoscaling/user-api-hpa-c created
horizontalpodautoscaler.autoscaling/user-api-hpa-m created

查看服务

[root@k8smaster user-api]# kubectl get svc -n go-zero-demo
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
user-api-svc   NodePort    10.140.120.5     <none>        8988:32000/TCP   89s
user-rpc-svc   ClusterIP   10.140.199.229   <none>        8980/TCP         31m

查看pod，发现状态没起来，镜像问题

kubectl get pod -n go-zero-demo

[root@k8smaster user-api]# kubectl get pod -n go-zero-demo
NAME                        READY   STATUS             RESTARTS   AGE
user-api-5c4f9b8554-g4fzh   0/1     ImagePullBackOff   0          18h
user-api-5c4f9b8554-nmgm9   0/1     ImagePullBackOff   0          18h
user-api-5c4f9b8554-rgwmw   0/1     ImagePullBackOff   0          18h
user-rpc-6dc5f57977-djsz5   0/1     ImagePullBackOff   0          18h
user-rpc-6dc5f57977-gm8q6   0/1     ImagePullBackOff   0          18h
user-rpc-6dc5f57977-qqw4k   0/1     ImagePullBackOff   0          18h

查看信息，确认镜像问题

kubectl describe pod -n go-zero-demo

Events:
Type Reason Age From Message

---

Warning Failed 38m (x220 over 15h) kubelet (combined from similar events): Failed to pull image “gozerok8s-user-rpc:v1”: failed to pull and unpack image “docker.io/library/gozerok8s-user-rpc:v1”: failed to resolve reference “docker.io/library/gozerok8s-user-rpc:v1”: failed to do request: Head “https://registry-1.docker.io/v2/library/gozerok8s-user-rpc/manifests/v1”: dial tcp 173.244.209.150:443: connect: connection refused
Normal BackOff 3m17s (x4484 over 18h) kubelet Back-off pulling image “gozerok8s-user-rpc:v1”

从Kuberetes 1.20版本开始，Kuberetes官方已经将默认的容器运行时从Docker改为Containerd。

3、镜像问题

1、把docker镜像打包，然后构建为Containerd镜像

[root@k8smaster user-api]# docker save -o gozerok8s-user-rpc.tar gozerok8s-user-rpc
[root@k8smaster user-api]# ll
总用量 49168
-rw-r--r-- 1 root root      756 9月  19 14:35 Dockerfile
drwxr-xr-x 2 root root       27 9月  19 14:13 etc
-rw------- 1 root root 50330112 9月  20 10:31 gozerok8s-user-rpc.tar
drwxr-xr-x 7 root root       72 9月  19 14:13 internal
-rw-r--r-- 1 root root      482 9月   4 14:27 user-api.api
-rw-r--r-- 1 root root     2105 9月  19 16:39 user-api.yaml
-rw-r--r-- 1 root root      614 9月   4 14:03 user.go
[root@k8smaster user-api]# ctr -n k8s.io image import gozerok8s-user-rpc.tar 
unpacking docker.io/library/gozerok8s-user-rpc:v1 (sha256:77769e8edc98f5847ad37959282290b9ea4735a71fb17be40dcba9ed8f7b5dc6)...done

2、查看Containerd镜像

[root@k8smaster user-api]# ctr -n k8s.io image list |grep gozerok8s-user-rpc
docker.io/library/gozerok8s-user-rpc:v1                                                                                                 application/vnd.oci.image.manifest.v1+json                sha256:77769e8edc98f5847ad37959282290b9ea4735a71fb17be40dcba9ed8f7b5dc6 48.0 MiB  linux/amd64                                                                   io.cri-containerd.image=managed                                 

3、把tar镜像包复制到其他子节点上

[root@k8smaster user-api]# scp gozerok8s-user-rpc.tar root@192.168.59.149:/data/
root@192.168.59.149's password: 
gozerok8s-user-rpc.tar                                                   100%   48MB  12.9MB/s   00:03    
[root@k8smaster user-api]# scp gozerok8s-user-rpc.tar root@192.168.59.150:/data/
root@192.168.59.150's password: 
gozerok8s-user-rpc.tar                                                   100%   48MB  12.8MB/s   00:03    

4、子节点也构建

[root@k8snode1 data]# ctr -n k8s.io image import gozerok8s-user-rpc.tar
unpacking docker.io/library/gozerok8s-user-rpc:v1 (sha256:77769e8edc98f5847ad37959282290b9ea4735a71fb17be40dcba9ed8f7b5dc6)...done
[root@k8snode1 data]# ctr -n k8s.io image import gozerok8s-user-api.tar
unpacking docker.io/library/gozerok8s-user-api:v1 (sha256:51fb675792fb6f1cc587dbb29184193a2aeaeb9b58874d37fbc18f0d0c7aba0a)...done

5、查看状态

api和rpc镜像都这样操作完后，查看状态就都启动了

[root@k8smaster user-rpc]# kubectl get pod -n go-zero-demo
NAME                        READY   STATUS    RESTARTS   AGE
user-api-5c4f9b8554-g4fzh   1/1     Running   0          18h
user-api-5c4f9b8554-nmgm9   1/1     Running   0          18h
user-api-5c4f9b8554-rgwmw   1/1     Running   0          18h
user-rpc-6dc5f57977-djsz5   1/1     Running   0          18h
user-rpc-6dc5f57977-gm8q6   1/1     Running   0          18h
user-rpc-6dc5f57977-qqw4k   1/1     Running   0          18h
[root@k8smaster user-rpc]# kubectl get svc -n go-zero-demo
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
user-api-svc   NodePort    10.140.120.5     <none>        8988:32000/TCP   19h
user-rpc-svc   ClusterIP   10.140.199.229   <none>        8980/TCP         19h

5、访问测试

多请求几次可以看到服务名会变，负载均衡有效

至此api和rpc使用k8s服务发现和部署就算ok了