[root@server2 ~]# kubectl get pod -o wide 查看pod 的ip
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deployment-2-77cd76f9c5-2zrkd 1/1 Running 1 (7h40m ago) 25h 10.244.2.68 server4 <none> <none>
deployment-2-77cd76f9c5-96cwp 1/1 Running 1 (7h40m ago) 25h 10.244.2.69 server4 <none> <none>
deployment-2-77cd76f9c5-stppk 1/1 Running 1 (7h40m ago) 25h 10.244.1.77 server3 <none> <none>
deployment-678fcbc488-2lf22 1/1 Running 1 (7h40m ago) 30h 10.244.1.80 server3 <none> <none>
deployment-678fcbc488-5ms5s 1/1 Running 1 (7h40m ago) 30h 10.244.2.70 server4 <none> <none>
deployment-678fcbc488-sgcd6 1/1 Running 1 30h 10.244.1.79 server3 <none> <none>
[root@server2 ~]# kubectl exec -it deployment-2-77cd76f9c5-stppk -- sh 进入其中一个pod
/ # route -n 查看容器的网关
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.244.1.1 0.0.0.0 UG 0 0 0 eth0
10.244.0.0 10.244.1.1 255.255.0.0 UG 0 0 0 eth0
10.244.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
/ # ping 10.244.1.80 可以看出同一节点server3上不同pod可以通信，原理是数据包通过默认网关连接网桥cni0通信，然后网桥连接另一个pod
PING 10.244.1.80 (10.244.1.80): 56 data bytes
64 bytes from 10.244.1.80: seq=0 ttl=64 time=0.212 ms
64 bytes from 10.244.1.80: seq=1 ttl=64 time=0.071 ms
flannel网络原理图：
[root@server2 ~]# kubectl exec -it deployment-2-77cd76f9c5-stppk -- sh
/ # ping 10.244.2.70 此节点地址在server4上，
PING 10.244.2.70 (10.244.2.70): 56 data bytes
64 bytes from 10.244.2.70: seq=0 ttl=62 time=1.059 ms
64 bytes from 10.244.2.70: seq=1 ttl=62 time=0.445 ms
节点上的pod的ip地址是由是由subnet划分的,如下
[root@server3 ~]# cd /var/run/flannel
[root@server3 flannel]# cat subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.1.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
封装信息如何得知?
知道原地址10.244.1.77和目标地址10.244.2.70,现在需要知道原mac地址和目标mac地址,原mac地址好找
[root@server3 flannel]# ip addr show flannel.1 可以查看原mac地址
但是server3是如何知道server4上的mac地址
[root@server3 flannel]# route -n 可以看出0网段和2网段的都要走隧道模式到flannel.1
[root@server3 flannel]# arp -an
后端mac地址的获取是由后台程序 flanneld运行在后端，他在分配子网段后，他还会获取整个集群节点的内部信息，他还会把每个设备的mac地址缓存到本地
[root@server3 flannel]# ps ax | grep flanneld
6163 ? Ssl 0:38 /opt/bin/flanneld --ip-masq --kube-subnet-mgr
17409 pts/0 S+ 0:00 grep --color=auto flannel
知道原地址和mac地址，目标地址和mac地址，然后通过vxlan进行1此封装，现在还需要知道对端flannel.1连接的eth0 ip地址和mac地址，需要查询转发数据库
[root@server3 flannel]# bridge fdb 查询，可以查出目标eth0的设备地址为172.25.50.4
然后再通过[root@server3 flannel]# arp -an 查寻172.25.50.4的mac地址
知道eth0这些地址后进行一个正常的udp封装，内部数据帧承载着正常的udp数据包传到对端eth0上，然后对端开始解包，匹配到内部数据帧是flannel.1d的然后由对端flannel.1 vtep进行解包，拿到原地址和目标地址
[root@server4 ~]# route -n 可以看到目标地址走的是2网段所以开始走cni0，这样就可以通信了
[root@server2 ~]# kubectl -n kube-system get cm 查看配置文件
[root@server2 ~]# kubectl -n kube-system edit cm kube-flannel-cfg 编辑flannel配置文件
[root@server2 ~]# kubectl -n kube-system get pod | grep flannel | awk '{system("kubectl -n kube-system delete pod "$1"")}' 删除之前pod
pod "kube-flannel-ds-gss9t" deleted
pod "kube-flannel-ds-ltggs" deleted
pod "kube-flannel-ds-z4sbd" deleted
[root@server2 ~]# kubectl exec -it deployment-2-77cd76f9c5-stppk -- sh
/ # ping 10.244.2.72 访问2网段也可以ping通
PING 10.244.2.72 (10.244.2.72): 56 data bytes
64 bytes from 10.244.2.72: seq=0 ttl=62 time=0.708 ms
64 bytes from 10.244.2.72: seq=1 ttl=62 time=0.450 ms
[root@server3 ~]# route -n 可以发现访问10.224.2.0网段时直接访问172.25.50.4
[root@server4 ~]# route -n 可以发现访问10.224.0.0网段时直接访问172.25.50.2;可以发现访问10.224.1.0网段时直接访问172.25.50.3,直接跨过flannel.1
注意：在一个网段可以用host-gw,不在同一个网段用vxlan模式，如果集群里即有同一网段，也有不同网段则将vxlan和直接路由同时打开，让集群自主选者，如下设置：
flannel只负责通信但是没有网络策略，比如不可以定义那些网段的pod可以访问，那些网段的pod不可以访问，做策勒比较好的插件calico
calico官网
calico安装
[root@server2 ~]# mkdir calico
[root@server2 ~]# cd calico/
[root@server2 calico]# wget https://docs.projectcalico.org/manifests/calico.yaml
[root@server2 calico]# ls
calico.yaml
单纯使用calico不使用flannel：
[root@server2 ~]# kubectl delete -f kube-flannel.yml 删除flannel
[root@server2 cni]# cd /etc/cni/net.d/
[root@server2 net.d]# ls
10-flannel.conflist
[root@server2 net.d]# mv 10-flannel.conflist /mnt/ 为避免和calico冲突，将各个节点flannel网络插件配置文件移走
[root@server3 ~]# cd /etc/cni/net.d/
[root@server3 net.d]# ls
10-flannel.conflist
[root@server3 net.d]# mv 10-flannel.conflist /mnt/ 为避免和calico冲突，将各个节点flannel网络插件配置文件移走
[root@server4 net.d]# mv 10-flannel.conflist /mnt/ 为避免和calico冲突，将各个节点flannel网络插件配置文件移走
calico部署文档
当前部署50个节点以下的
[root@server2 calico]# vim calico.yaml 修改yaml文件
拉取所需镜像:
[root@server1 harbor]# docker pull calico/kube-controllers:v3.22.1 拉取所需镜像
[root@server1 harbor]# docker pull calico/typha:v3.22.1 拉取所需镜像
[root@server1 harbor]# docker pull calico/cni:v3.22.1 拉取所需镜像
[root@server1 harbor]# docker pull calico/pod2daemon-flexvol:v3.22.1 拉取所需镜像
[root@server1 harbor]# docker pull calico/node:v3.22.1 拉取所需镜像
在habor仓库新建项目
改标签：
[root@server1 ~]# docker tag calico/kube-controllers:v3.22.1 reg.westos.org/calico/kube-controllers:v3.22.1
[root@server1 ~]# docker tag calico/typha:v3.22.1 reg.westos.org/calico/typha:v3.22.1
[root@server1 ~]# docker tag calico/cni:v3.22.1 reg.westos.org/calico/cni:v3.22.1
[root@server1 ~]# docker tag calico/pod2daemon-flexvol:v3.22.1 reg.westos.org/calico/pod2daemon-flexvol:v3.22.1
[root@server1 ~]# docker tag calico/node:v3.22.1 reg.westos.org/calico/node:v3.22.1
上传镜像到仓库：
[root@server1 ~]# docker push reg.westos.org/calico/kube-controllers:v3.22.1
[root@server1 ~]# docker push reg.westos.org/calico/typha:v3.22.1
[root@server1 ~]# docker push reg.westos.org/calico/cni:v3.22.1
[root@server1 ~]# docker push reg.westos.org/calico/pod2daemon-flexvol:v3.22.1
[root@server1 ~]# docker push reg.westos.org/calico/node:v3.22.1
将镜像地址改为仓库地址
[root@server2 calico]# kubectl apply -f calico.yaml 运行
[root@server2 calico]# kubectl -n kube-system get pod 查看pod节点
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-56b559bc95-chqb4 1/1 Running 0 5m57s 运行成功
calico-node-9l6bn 1/1 Running 0 5m58s 运行成功
calico-node-jmvsx 1/1 Running 0 5m58s 运行成功
calico-node-z2gxs 1/1 Running 0 5m58s 运行成功
calico-typha-84bffd4667-vkr2s 1/1 Running 0 4m17s 运行成功
coredns-7b56f6bc55-hlts6 1/1 Running 21 (61m ago) 12d
coredns-7b56f6bc55-k6dzz 1/1 Running 21 12d
etcd-server2 1/1 Running 25 (61m ago) 12d
kube-apiserver-server2 1/1 Running 25 (61m ago) 12d
kube-controller-manager-server2 1/1 Running 46 (61m ago) 12d
kube-proxy-2vsgk 1/1 Running 3 (61m ago) 3d12h
kube-proxy-cjppk 1/1 Running 3 (61m ago) 3d12h
kube-proxy-khgfn 1/1 Running 4 (61m ago) 3d12h
kube-scheduler-server2 1/1 Running 32 (61m ago) 12d
[root@server2 ~]# kubectl delete pod --all 删除之前pod，更新一下pod
pod "deployment-2-77cd76f9c5-2zrkd" deleted
pod "deployment-2-77cd76f9c5-96cwp" deleted
pod "deployment-2-77cd76f9c5-stppk" deleted
pod "deployment-678fcbc488-2lf22" deleted
pod "deployment-678fcbc488-5ms5s" deleted
pod "deployment-678fcbc488-sgcd6" deleted
[root@server2 ~]# kubectl get pod -o wide 查看pod的ip，各节点pod已经分到ip
[root@server3 ~]# route -n 各个节点直接添加网关
[root@foundation50 network-scripts]# curl www2.westos.org 访问，可以访问访问成功
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>
[root@foundation50 network-scripts]# curl www2.westos.org/hostname.html
deployment-2-77cd76f9c5-clp52
[root@foundation50 network-scripts]# curl www2.westos.org/hostname.html
deployment-2-77cd76f9c5-6hlkk
一般情况我们是将flannel和calico结合使用，flannel负责网络通信，calico负责网络策略
网络策略文档
例1：限制访问指定服务
[root@server2 ~]# cd calico/
[root@server2 calico]# vim deny-nginx.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-nginx
spec:
podSelector:
matchLabels:
app: nginx
[root@server2 ~]# kubectl apply -f deny-nginx.yaml 运行
networkpolicy.networking.k8s.io/deny-nginx created
[root@server2 ~]# kubectl get networkpolicies 查看策略，策略已经添加
NAME POD-SELECTOR AGE
deny-nginx app=nginx 83s
测试：
[root@foundation50 ~]# curl www2.westos.org 此时外部访问www2就已经无法访问了
[root@server2 ~]# kubectl describe svc nginx-svc 详细查看nginx-svc服务
Name: nginx-svc
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=nginx
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.110.63.4
IPs: 10.110.63.4
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.141.202:80,10.244.141.203:80,10.244.22.8:80 后端pod ip
[root@server2 ~]# curl 10.244.141.202 访问集群内部后端pod ip也不能访问
[root@server2 ~]# kubectl describe svc myservice 查看myservice后端服务
Name: myservice
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=myapp
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.100.178.184
IPs: 10.100.178.184
Port: <unset> 80/TCP
TargetPort: 80/TCP
Endpoints: 10.244.141.204:80,10.244.22.10:80,10.244.22.9:80 后端pod ip
[root@server2 ~]# curl 10.244.141.204 可以访问
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
例2：允许指定访问的pod服务
[root@server2 ~]# kubectl run demo -it --image=busyboxplus -l app=demo 现运行带有app=demo标签的pod
/ #curl 10.244.141.202 访问带有nginx标签的pod，访问不了
[root@server2 calico]# vim deny-nginx.yaml
--- 添加
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: acces-nginx
spec:
podSelector:
matchLabels:
app: nginx
ingress:
- from:
- podSelector: 表示如果数据包从 app: demo 这样的pod出来，可以访问
matchLabels:
app: demo
[root@server2 calico]# kubectl apply -f deny-nginx.yaml 运行
networkpolicy.networking.k8s.io/deny-nginx unchanged
networkpolicy.networking.k8s.io/acces-nginx created
[root@server2 calico]# kubectl attach demo -it 进入demo
If you don't see a command prompt, try pressing enter.
/ # curl 10.244.141.202 数据包从带有demo标签的pod里，访问带有nginx标签的pod
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a> 可以访问
[root@server2 calico]# curl 10.244.141.202 直接访问，访问不了
例3：禁止同一namespace中所有pod之间的pod相互访问
[root@server2 calico]# kubectl create namespace test 创建一个test空间
namespace/test created
[root@server2 calico]# kubectl run test1 -it --image=busyboxplus -n test 运行pod
[root@server2 calico]# kubectl run test2 -it --image=busyboxplus -n test 再运行一个pod
If you don't see a command prompt, try pressing enter.
/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
link/ether 32:b9:86:f4:26:b4 brd ff:ff:ff:ff:ff:ff
inet 10.244.141.206/32 scope global eth0 分到的ip为10.244.141.206
valid_lft forever preferred_lft forever
[root@server2 calico]# kubectl -n test get pod --show-labels 创建了两个pod
NAME READY STATUS RESTARTS AGE LABELS
test1 1/1 Running 1 4m40s run=test1
test2 1/1 Running 1 (112s ago) 2m55s run=test2
[root@server2 calico]# kubectl -n test get pod --show-labels -o wide 查看ip，不在一个结点上，同在一个namespace空间
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS
test1 1/1 Running 1 5m40s 10.244.22.11 server4 <none> <none> run=test1
test2 1/1 Running 1 (2m52s ago) 3m55s 10.244.141.206 server3 <none> <none> run=test2
[root@server2 calico]# kubectl attach test2 -it -n test 进入test2的pod里
If you don't see a command prompt, try pressing enter.
/ # ping 10.244.22.11 ping test1的pod，可以ping通
PING 10.244.22.11 (10.244.22.11): 56 data bytes
64 bytes from 10.244.22.11: seq=0 ttl=62 time=0.784 ms
64 bytes from 10.244.22.11: seq=1 ttl=62 time=0.364 ms
[root@server2 calico]# vim deny-nginx.yaml 编辑
--- 添加 ,禁止namespace中所有pod之间的pod相互访问
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: default-deny
namespace: test
spec:
podSelector: {} 不指定标签就是匹配所有pod
[root@server2 calico]# kubectl apply -f deny-nginx.yaml 运行
networkpolicy.networking.k8s.io/deny-nginx unchanged
networkpolicy.networking.k8s.io/acces-nginx unchanged
networkpolicy.networking.k8s.io/default-deny created
[root@server2 calico]# kubectl attach test2 -it -n test
If you don't see a command prompt, try pressing enter.
/ # ping 10.244.22.18 ping不通
例4：禁止其他namespace访问服务
[root@server2 calico]# kubectl attach test2 -it -n test
If you don't see a command prompt, try pressing enter.
/ # ping 10.244.141.213 同一namespace不能访问，但是不通namespace可以访问
PING 10.244.141.213 (10.244.141.213): 56 data bytes
64 bytes from 10.244.141.213: seq=0 ttl=63 time=0.193 ms
64 bytes from 10.244.141.213: seq=1 ttl=63 time=0.087 ms
[root@server2 calico]# vim deny-nginx.yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: deny-namespace
spec:
podSelector:
matchLabels:
ingress:
- from:
- podSelector: {}
[root@server2 calico]# kubectl apply -f deny-nginx.yaml 运行
networkpolicy.networking.k8s.io/deny-nginx unchanged
networkpolicy.networking.k8s.io/acces-nginx unchanged
networkpolicy.networking.k8s.io/default-deny unchanged
networkpolicy.networking.k8s.io/deny-namespace created
[root@server2 calico]# kubectl attach test2 -it -n test 进入
If you don't see a command prompt, try pressing enter.
/ # ping 10.244.141.213 彼此namespace被隔离不能访问
PING 10.244.141.213 (10.244.141.213): 56 data bytes
例5：只允许指定namespace访问服务
[root@server2 calico]# kubectl label namespaces test role=prod 修改namespaces标签为role=prod
namespace/test labeled
[root@server2 calico]# kubectl get ns --show-labels 查看ns标签，标签已经添加成功
[root@server2 calico]# vim deny-nginx.yaml
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: access-namespace
spec:
podSelector:
matchLabels:
app: myapp
ingress:
- from:
- namespaceSelector: 表示数据包从带有 role: prod 标签的namespace，访问带有app：myapp的pod，可以访问
matchLabels:
role: prod
[root@server2 calico]# kubectl apply -f deny-nginx.yaml 运行
networkpolicy.networking.k8s.io/deny-nginx unchanged
networkpolicy.networking.k8s.io/acces-nginx unchanged
networkpolicy.networking.k8s.io/default-deny unchanged
networkpolicy.networking.k8s.io/deny-namespace configured
networkpolicy.networking.k8s.io/access-namespace created
[root@server2 calico]# kubectl get pod -o wide --show-labels 查看标签为myapp的ip
[root@server2 calico]# kubectl attach test2 -it -n test 进入pod
If you don't see a command prompt, try pressing enter.
/ # ping 10.244.22.16
PING 10.244.22.16 (10.244.22.16): 56 data bytes
64 bytes from 10.244.22.16: seq=0 ttl=62 time=0.610 ms
64 bytes from 10.244.22.16: seq=1 ttl=62 time=0.361 ms
64 bytes from 10.244.22.16: seq=2 ttl=62 time=0.431 ms
例6：允许外网访问服务
--- 添加
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: web-allow-external
spec:
podSelector:
matchLabels:
app: myapp
ingress:
- ports:
- port: 80 暴露内部端口为80
from: []
[root@server2 calico]# kubectl apply -f deny-nginx.yaml 运行
networkpolicy.networking.k8s.io/deny-nginx unchanged
networkpolicy.networking.k8s.io/acces-nginx unchanged
networkpolicy.networking.k8s.io/default-deny unchanged
networkpolicy.networking.k8s.io/deny-namespace configured
networkpolicy.networking.k8s.io/access-namespace unchanged
networkpolicy.networking.k8s.io/web-allow-external created
注意：写多个策略时，加不加“-”有很大区别，加上“-”表示或者，满足一个条件即可；不加“-”表示并且，所有条件必须同时满足。
策略文档