kubelet报错如下：
kubelet[3807]: E1215 18:01:25.516905 3807 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
12月 15 18:01:25 localhost.localdomain kubelet[3807]: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
12月 15 18:01:25 localhost.localdomain kubelet[3807]: E1215 18:01:25.543568 3807 kubelet.go:1306] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
12月 15 18:01:25 localhost.localdomain kubelet[3807]: E1215 18:01:25.671787 3807 kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
12月 15 18:01:25 localhost.localdomain kubelet[3807]: E1215 18:01:25.772456 3807 kubelet.go:1846] skipping pod synchronization - container runtime status check may not have completed yet
12月 15 18:02:00 localhost.localdomain kubelet[3807]: E1215 18:02:00.080546 3807 pod_workers.go:191] Error syncing pod 47533b1b-9f9b-4f80-a706-13830cf1952c ("kube-flannel-ds-amd64-qqpxj_kube-system(47533b1b-9f9b-4f80-a706-13830cf1952c)"), skipping: failed to "StartContainer" for "kube-flannel" with CrashLoopBackOff: "back-off 10s restarting failed container=kube-flannel pod=kube-flannel-ds-amd64-qqpxj_kube-system(47533b1b-9f9b-4f80-a706-13830cf1952c)"
简单描述：
创建flannel网络时，有两个节点创建flannel.1网络失败，报错信息如上：
具体操作：
1.把之前三个节点无关的网卡全部下线，删除
ifconfig cni0 down
ip link delete cni0
ip a
ifconfig nodelocaldns down
ip link delete nodelocaldns
只保留最简单的几个
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a1:72:53 brd ff:ff:ff:ff:ff:ff
inet 10.182.0.103/24 brd 10.182.0.255 scope global noprefixroute ens192
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:9c:f2:30:33 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
2.删除三个节点之前的网络数据缓存
rm -rf /var/lib/cni/*
rm -rf /run/flannel/*
rm -rf /var/run/containerd/
3.重启docker kubelet kube-proxy
systemctl restart docker kubelet kube-proxy
4.重新部署flannel网络；cat kube-flannel.yaml
---
kind: Namespace
apiVersion: v1
metadata:
name: kube-system
labels:
pod-security.kubernetes.io/enforce: privileged
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
annotations:
deprecated.daemonset.template.generation: '1'
labels:
app: flannel
tier: node
name: kube-flannel-ds-amd64
namespace: kube-system
resourceVersion: '112554485'
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
app: flannel
template:
metadata:
creationTimestamp: null
labels:
app: flannel
tier: node
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
containers:
- args:
- '--ip-masq'
- '--kube-subnet-mgr'
# - '--kubeconfig-file=/etc/kubernetes/cfg/flannel.conf'
command:
- /opt/bin/flanneld
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: 'easzlab/flannel:v0.13.0-amd64'
imagePullPolicy: IfNotPresent
name: kube-flannel
resources:
limits:
cpu: 100m
memory: 50Mi
requests:
cpu: 100m
memory: 50Mi
securityContext:
capabilities:
add:
- NET_ADMIN
privileged: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /run/flannel
name: run
- mountPath: /etc/kube-flannel/
name: flannel-cfg
dnsPolicy: ClusterFirst
hostNetwork: true
initContainers:
- args:
- '-f'
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
command:
- cp
image: 'easzlab/flannel:v0.13.0-amd64'
imagePullPolicy: IfNotPresent
name: install-cni
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/cni/net.d
name: cni
- mountPath: /etc/kube-flannel/
name: flannel-cfg
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: flannel
serviceAccountName: flannel
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
operator: Exists
volumes:
- hostPath:
path: /run/flannel
type: ''
name: run
- hostPath:
path: /etc/cni/net.d
type: ''
name: cni
- configMap:
defaultMode: 420
name: kube-flannel-cfg
name: flannel-cfg
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
# 部署命令
kubectl apply -f kube-flannel.yaml
[root@localhost lkpconf]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-flannel-ds-amd64-6bhjl 1/1 Running 0 33m
kube-system kube-flannel-ds-amd64-h6t4v 1/1 Running 0 33m
kube-system kube-flannel-ds-amd64-rfgqc 1/1 Running 0 33m
kuboard kuboard-agent-3ebiqh-2-55748b6d98-4k7n7 1/1 Running 1 4h40m
kuboard kuboard-agent-3ebiqh-8d699dfcb-8bfcf 1/1 Running 1 4h42m
flannel日志
I1216 06:20:52.477397 1 main.go:518] Determining IP address of default interface
I1216 06:20:52.479067 1 main.go:531] Using interface with name ens192 and address 10.182.0.102
I1216 06:20:52.479135 1 main.go:548] Defaulting external address to interface address (10.182.0.102)
W1216 06:20:52.479200 1 client_config.go:517] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1216 06:20:52.692185 1 kube.go:119] Waiting 10m0s for node controller to sync
I1216 06:20:52.692254 1 kube.go:306] Starting kube subnet manager
I1216 06:20:53.692573 1 kube.go:126] Node controller sync successful
I1216 06:20:53.692655 1 main.go:246] Created subnet manager: Kubernetes Subnet Manager - 10.182.0.102
I1216 06:20:53.692681 1 main.go:249] Installing signal handlers
I1216 06:20:53.692918 1 main.go:390] Found network config - Backend type: vxlan
I1216 06:20:53.693136 1 vxlan.go:121] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
I1216 06:20:53.877516 1 main.go:355] Current network or subnet (10.244.0.0/16, 10.244.1.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
I1216 06:20:54.683635 1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
I1216 06:20:54.783668 1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I1216 06:20:54.786251 1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
I1216 06:20:54.788710 1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j MASQUERADE --random-fully
I1216 06:20:54.877429 1 main.go:305] Setting up masking rules
I1216 06:20:54.879632 1 main.go:313] Changing default FORWARD chain policy to ACCEPT
I1216 06:20:54.879882 1 main.go:321] Wrote subnet file to /run/flannel/subnet.env
I1216 06:20:54.879925 1 main.go:325] Running backend.
I1216 06:20:54.879961 1 main.go:343] Waiting for all goroutines to exit
I1216 06:20:54.880071 1 vxlan_network.go:60] watching for new subnet leases
I1216 06:20:55.076905 1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I1216 06:20:55.076982 1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I1216 06:20:55.077481 1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I1216 06:20:55.077525 1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 -j ACCEPT
I1216 06:20:55.176643 1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I1216 06:20:55.177007 1 iptables.go:167] Deleting iptables rule: -d 10.244.0.0/16 -j ACCEPT
I1216 06:20:55.179838 1 iptables.go:167] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.1.0/24 -j RETURN
I1216 06:20:55.277677 1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 -j ACCEPT
I1216 06:20:55.279603 1 iptables.go:167] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
I1216 06:20:55.282139 1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I1216 06:20:55.476441 1 iptables.go:155] Adding iptables rule: -d 10.244.0.0/16 -j ACCEPT
I1216 06:20:55.478564 1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I1216 06:20:55.579364 1 iptables.go:155] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.1.0/24 -j RETURN
I1216 06:20:55.677691 1 iptables.go:155] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
以上操作亲测有效。
