coredns CrashLoopBackOff 报错
·
# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-apiserver calico-apiserver-58b7cdd685-mnnqk 1/1 Running 38 (32m ago) 7h43m
calico-apiserver calico-apiserver-58b7cdd685-mw626 1/1 Running 37 (31m ago) 7h43m
calico-system calico-kube-controllers-58d9bdcc64-5brpp 1/1 Running 6 (7h42m ago) 2d
calico-system calico-node-29wrk 1/1 Running 10 (7h42m ago) 2d
calico-system calico-node-bv2mv 1/1 Running 0 2d
calico-system calico-node-sjbms 1/1 Running 0 2d
calico-system calico-typha-57bfdd9d4b-d688b 1/1 Running 0 2d
calico-system calico-typha-57bfdd9d4b-pfv9b 1/1 Running 11 (7h42m ago) 2d
calico-system csi-node-driver-5j5cc 2/2 Running 8 (6h45m ago) 2d
calico-system csi-node-driver-l69hz 2/2 Running 6 (47h ago) 2d
calico-system csi-node-driver-ttk2f 2/2 Running 10 (6h45m ago) 2d
kube-system coredns-666fb66584-48ccf 0/1 CrashLoopBackOff 4 (86s ago) 3m38s
kube-system coredns-666fb66584-5sgz7 0/1 Error 5 (97s ago) 3m38s
kube-system etcd-k8s-master 1/1 Running 4 (7h42m ago) 2d
kube-system kube-apiserver-k8s-master 1/1 Running 5 (7h42m ago) 2d
kube-system kube-controller-manager-k8s-master 1/1 Running 19 (7h42m ago) 2d
kube-system kube-proxy-jncnm 1/1 Running 4 (7h42m ago) 2d
kube-system kube-proxy-tlktl 1/1 Running 4 (6h45m ago) 2d
kube-system kube-proxy-wbmrv 1/1 Running 5 (6h45m ago) 2d
kube-system kube-scheduler-k8s-master 1/1 Running 20 (7h42m ago) 2d
kubernetes-dashboard dashboard-metrics-scraper-795895d745-r2frw 1/1 Running 5 (31m ago) 7h43m
kubernetes-dashboard kubernetes-dashboard-78f95ff46f-fkbjl 1/1 Running 18 (31m ago) 7h43m
tigera-operator tigera-operator-67d96f44b5-pnw2l 1/1 Running 19 (7h39m ago) 2d
看日志
kubectl logs -n kube-system coredns-666fb66584-48ccf
看 Pod 事件
kubectl describe pod -n kube-system coredns-666fb66584-48ccf | tail -n 20
[root@k8s-master ~]# kubectl logs -n kube-system coredns-666fb66584-48ccf
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:kube-system:coredns" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:kube-system:coredns" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:kube-system:coredns" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:kube-system:coredns" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:kube-system:coredns" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:kube-system:coredns" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
.:53
[INFO] plugin/reload: Running configuration SHA512 = ec91819f27159ab6f9b5a4d2641b5a2249737263164e5ddd77080beb1f629377519c744940a0115fb4b36ce24c272598b5c883a48bf84bdf64993e0caf026b1c
CoreDNS-1.10.1
linux/amd64, go1.20, 055b2c3
[FATAL] plugin/loop: Loop (192.168.174.130:16813 -> :53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 2374223895899837454.5830436550668310787."
[root@k8s-master ~]# kubectl describe pod -n kube-system coredns-666fb66584-48ccf | tail -n 20
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: CriticalAddonsOnly op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m10s default-scheduler Successfully assigned kube-system/coredns-666fb66584-48ccf to k8s-node2
Warning FailedCreatePodSandBox 5m7s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "662b145239cfcf026c241c86e066642c9adb44b1c60c80bb7e4c8dc335172343" network for pod "coredns-666fb66584-48ccf": networkPlugin cni failed to set up pod "coredns-666fb66584-48ccf_kube-system" network: plugin type="calico" failed (add): error adding host side routes for interface: calibc5b56b0f80, error: route (Ifindex: 32, Dst: 10.244.169.140/32, Scope: link) already exists for an interface other than 'calibc5b56b0f80': route (Ifindex: 6, Dst: 10.244.169.140/32, Scope: link, Iface: cali40c0748fc8e)
Warning FailedCreatePodSandBox 5m4s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c99bc0bf41699076de587ef97d7d80ff5df5a05561e3e310859d8fcb0df28eb7" network for pod "coredns-666fb66584-48ccf": networkPlugin cni failed to set up pod "coredns-666fb66584-48ccf_kube-system" network: plugin type="calico" failed (add): error adding host side routes for interface: calibc5b56b0f80, error: route (Ifindex: 33, Dst: 10.244.169.141/32, Scope: link) already exists for an interface other than 'calibc5b56b0f80': route (Ifindex: 7, Dst: 10.244.169.141/32, Scope: link, Iface: cali5989ef1d8fa)
Normal SandboxChanged 5m2s (x3 over 5m6s) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 5m2s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "543adc97b3e32707478898e0b139e1b50ca32417859e404aa03894fbc4b58320" network for pod "coredns-666fb66584-48ccf": networkPlugin cni failed to set up pod "coredns-666fb66584-48ccf_kube-system" network: plugin type="calico" failed (add): error adding host side routes for interface: calibc5b56b0f80, error: route (Ifindex: 34, Dst: 10.244.169.142/32, Scope: link) already exists for an interface other than 'calibc5b56b0f80': route (Ifindex: 8, Dst: 10.244.169.142/32, Scope: link, Iface: cali8b137f2363b)
Normal Pulled 3m50s (x4 over 5m) kubelet Container image "coredns/coredns:1.10.1" already present on machine
Normal Created 3m50s (x4 over 5m) kubelet Created container coredns
Normal Started 3m50s (x4 over 5m) kubelet Started container coredns
Warning BackOff 2s (x24 over 4m48s) kubelet Back-off restarting failed container coredns in pod coredns-666fb66584-48ccf_kube-system(4beb07eb-d22b-400c-abee-c42d762e2387)
问题 现象 根因 解决
RBAC 缺失 Pod 日志里大量 endpointslices is forbidden CoreDNS 1.10 需要额外权限，但你的 ClusterRole 没有给 给 system:coredns 加一条 endpointslices 权限
DNS 转发死循环 plugin/loop: Loop ... detected CoreDNS 把请求又转给自己（/etc/resolv.conf 里出现本机地址） 把 forward . /etc/resolv.conf 改成真实上游 DNS
1. 修复 RBAC（一次性命令）
bash
复制
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:coredns
rules:
- apiGroups: [""]
resources: ["endpoints","services","pods","namespaces"]
verbs: ["list","watch"]
- apiGroups: ["discovery.k8s.io"]
resources: ["endpointslices"]
verbs: ["list","watch"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
EOF
上面只是 增量打补丁，不会破坏现有权限。
🔧 2. 修复 DNS 转发死循环
1.查看你节点上 真正可用的上游 DNS。
一般节点 /etc/resolv.conf 里除了 127.0.0.53 还会有真正 DNS，比如：
复制
nameserver 192.168.174.2
nameserver 8.8.8.8
选一个可用的，例如 192.168.174.2（你的网关/公司 DNS）。
2.修改 CoreDNS ConfigMap，把 forward 语句改掉：
kubectl edit cm coredns -n kube-system
把 forward . /etc/resolv.conf 改成
forward . 192.168.174.2 # 或 8.8.8.8、114.114.114.114 等
3.重启 CoreDNS Pod：
kubectl rollout restart deployment coredns -n kube-system
✅ 验证
等 20 秒后检查：
kubectl get pod -n kube-system -l k8s-app=kube-dns
应该看到：
NAME READY STATUS RESTARTS AGE
coredns-xxxxxxxx-xxxxx 1/1 Running 0 40s
coredns-xxxxxxxx-yyyyy 1/1 Running 0 40s
能解析出 kubernetes.default 的 IP 就彻底 OK 了。
北京人形旗下天工造物具身智能开源社区，聚焦具身天工与慧思开物两大平台
更多推荐
3
0- 0
已为社区贡献8条内容
所有评论(0)