calico分配网络使k8s节点指定固定网段
需求：测试环境的k8s的pod网络需要和阿里云zk通信（10.188.0.0）（pod的ip注册到zk），这里要固定node节点docker网段，避免当node节点重启后docker的网段会改变，避免对应的路由也需要重新指定网络。
| ip地址 |
指定网段 |
备注 |
| 10.188.28.7 |
10.188.248.0/22 |
固定地址段 |
| 10.188.28.8 |
10.188.252.0/22 |
固定地址段 |
1.配置calicoctl
1.1 下载calicoctl
# wget https://github.com/projectcalico/calicoctl/releases/download/v3.5.4/calicoctl -O /usr/bin/calicoctl
# chmod +x /usr/local/bin/calicoctl
1.2 配置calicoctl
# cat /etc/calico/calicoctl.cfg
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
datastoreType: "kubernetes"
kubeconfig: "/root/.kube/config"
1.3 测试calicoctl
# calicoctl get ippool
NAME CIDR SELECTOR
default-ipv4-ippool 192.168.0.0/16 all()
# calicoctl get ippool -o json default-ipv4-ippool
{
"kind": "IPPool",
"apiVersion": "projectcalico.org/v3",
"metadata": {
"name": "default-ipv4-ippool",
"uid": "e0b3c2f2-6d2c-4552-95df-cb75506b4e65",
"resourceVersion": "1613",
"creationTimestamp": "2021-10-26T11:11:00Z"
},
"spec": {
"cidr": "192.168.0.0/16",
"ipipMode": "Always",
"natOutgoing": true,
"blockSize": 26,
"nodeSelector": "all()"
}
}
2.配置ippool
使分配到k8s02上的pod，分配的ip池是10.188.252.0/22，分配到k8s01上的pod，分配的ip池是10.188.248.0/22
# cat 248-ippool.yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: 248-ipv4pool
spec:
blockSize: 26
cidr: 10.188.248.0/22
ipipMode: Always
nodeSelector: calico-network == 'k8s01'
natOutgoing: true
# cat 252-ippool.yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: 252-ipv4pool
spec:
blockSize: 26
cidr: 10.188.252.0/22
ipipMode: Always
nodeSelector: calico-network == 'k8s02'
natOutgoing: true
下一步给node节点打上标签
# kubectl label nodes bj-smzy-ce-k8s01-28-7 calico-network=k8s01
# kubectl label nodes bj-smzy-ce-k8s02-28-8 calico-network=k8s02
# kubectl get nodes --show-labels #查看label是否添加成功
3.添加ippool
# calicoctl create -f 248-ippool.yaml
# calicoctl create -f 252-ippool.yaml
# calicoctl get ippool
NAME CIDR SELECTOR
248-ipv4pool 10.188.248.0/22 calico-network == 'k8s01'
252-ipv4pool 10.188.252.0/22 calico-network == 'k8s02'
default-ipv4-ippool 10.5.0.0/16 all()
删除默认的ippool
# kubectl delete ippool default-ipv4-ippool
4.创建pod测试ippool网段划分
# cat nginx-deplay.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 10
template:
metadata:
labels:
app: nginx
#annotations: #可以不添加注释（已经把默认ipool删除）
# "cni.projectcalico.org/ipv4pools": "[\"248-ipv4pool\",\"252-ipv4pool\"]"
spec:
containers:
- name: nginx
image: nginx:alpine
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
# kubectl create -f nginx-deplay
# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-7fb7fd49b4-5w982 1/1 Running 0 117m 10.188.252.10 bj-smzy-ce-k8s02-28-8 <none> <none>
nginx-deployment-7fb7fd49b4-8qqrx 1/1 Running 0 117m 10.188.248.205 bj-smzy-ce-k8s01-28-7 <none> <none>
nginx-deployment-7fb7fd49b4-hnw76 1/1 Running 0 117m 10.188.248.208 bj-smzy-ce-k8s01-28-7 <none> <none>
nginx-deployment-7fb7fd49b4-jsq2r 1/1 Running 0 117m 10.188.252.15 bj-smzy-ce-k8s02-28-8 <none> <none>
nginx-deployment-7fb7fd49b4-kbwqz 1/1 Running 0 117m 10.188.248.206 bj-smzy-ce-k8s01-28-7 <none> <none>
nginx-deployment-7fb7fd49b4-klbhj 1/1 Running 0 117m 10.188.252.14 bj-smzy-ce-k8s02-28-8 <none> <none>
nginx-deployment-7fb7fd49b4-mwd7z 1/1 Running 0 117m 10.188.252.12 bj-smzy-ce-k8s02-28-8 <none> <none>
nginx-deployment-7fb7fd49b4-s7rwx 1/1 Running 0 117m 10.188.252.11 bj-smzy-ce-k8s02-28-8 <none> <none>
nginx-deployment-7fb7fd49b4-sgr6l 1/1 Running 0 117m 10.188.248.207 bj-smzy-ce-k8s01-28-7 <none> <none>
nginx-deployment-7fb7fd49b4-zhrfz 1/1 Running 0 117m 10.188.252.13 bj-smzy-ce-k8s02-28-8 <none> <none>
可以看到创建的pod的ip地址
分配到bj-smzy-ce-k8s01-28-7节点的pod是10.188.248.0/22段
分配到bj-smzy-ce-k8s02-28-8节点的pod是10.188.252.0/22段
注意：当新节点接入集群时，需要再次添加新节点的网段的ippool，不然分配的pod落在新节点上是启动不成功的（无法分配pod的ip）
5.核心交换机上添加静态路由
# ip route 10.188.248.0/22（node的docker网段） 10.188.28.7（node节点ip）
# ip route 10.188.252.0/22（node的docker网段） 10.188.28.8（node节点ip）
查看node节点的路由，如若不通信，还需要手动指定路由
# route add -net 10.180.0.0/16 gw 10.188.28.1
6.测试
在10.180.0.0网段的机器上ping podIP
在pod里ping 10.180.0.0网段的ip
