1、总结Underlay和Overlay网络的的区别及优缺点
Overlay网络：
Overlay 叫叠加网络也叫覆盖网络，指的是在物理网络的
基础之上叠加实现新的虚拟网络，即可使网络的中的容器可
以相互通信。
优点是对物理网络的兼容性比较好，可以实现pod的夸宿
主机子网通信。
calico与flannel等网络插件都支持overlay网络
缺点是有额外的封装与解封性能开销
目前私有云使用比较多。
VTEP(VXLAN Tunnel Endpoint vxlan隧道端点),VTEP是VXLAN网络的边缘设备，是VXLAN隧道的起点和终点，VXLAN对用户原始数据帧的封装和解封装均在VTEP上进行,用于VXLAN报文的封装和解封装,VTEP与物理网络相连,VXLAN报文中源IP地址为本节点的VTEP地址，VXLAN报文中目的IP地址为对端节点的VTEP地址，一对VTEP地址就对应着一个VXLAN隧道，服务器上的虚拟交换机(隧道flannel.1 就是VTEP)，比如一个虚拟机网络中的多个vxlan就需要多个VTEP对不同网络的报文进行封装与解封装
VNI（VXLAN Network Identifier）：VXLAN网络标识VNI类似VLAN ID,用于区分VXLAN段,不同VXLAN段的虚拟机不能直接二层相互通信,一个VNI表示一个租户,即使多个终端用户属于同一个VNI,也表示一个租户。
NVGRE：Network Virtualization using Generic Routing Encapsulation，主要支持者是Microsoft，与VXLAN不同的是，NVGRE没有采用标准传输协议（TCP/UDP），而是借助通用路由封装协议（GRE），NVGRE使用GRE头部的第24位作为租户网络标识符（TNI），与VXLAN一样可以支持1777216个vlan。
源宿主机VTEP添加或者封装VXLAN、UDP及IP头部报文。
网络层设备将封装后的报文通过标准的报文在三层网络进行转发到目标主机
目标宿主机VTEP删除或者解封装VXLAN、UDP及IP头部。
Underlay网络：
Underlay网络就是传统IT基础设施网络，由交换机和路由器等设备组成，借助以太网协议、路由协议和VLAN协议等驱动，它还是Overlay网络的底层网络，为Overlay网络提供数据通信服务。容器网络中的Underlay网络是指借助驱动程序将宿主机的底层网络接口直接暴露给容器使用的一种网络构建技术，较为常见的解决方案有MAC VLAN、IP VLAN和直接路由等。
Mac Vlan模式：
MAC VLAN：支持在同一个以太网接口上虚拟出多个网络接口(子接口)，每个虚拟接口都拥有唯一的MAC地址并可配置网卡子接口IP。
IP VLAN模式：
IP VLAN类似于MAC VLAN，它同样创建新的虚拟网络接口并为每个接口分配唯一的IP地址，不同之处在于，每个虚拟接口将共享使用物理接口的MAC地址。
网络通信-MAC Vlan工作模式：
bridge模式：
在bridge这种模式下，使用同一个宿主机网络的macvlan容器可以直接实现通信，推荐使用此模式。
网络通信-总结：
Overlay：基于VXLAN、NVGRE等封装技术实现overlay叠加网络。
Underlay(Macvlan)：基于宿主机物理网卡虚拟出多个网络接口(子接口)，每个虚拟接口都拥有唯一的MAC地址并可配置网卡子接口IP。
2、在kubernetes集群实现underlay网络
##在包括master及node节点在内的每一个节点安装docker或containerd运行时，先安装必要的一些系统工具，只展示master节点
root@k8s-master1:~# apt-get update
Get:1 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease [8,993 B]
Hit:2 http://cn.archive.ubuntu.com/ubuntu jammy InRelease
Get:3 http://cn.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 http://cn.archive.ubuntu.com/ubuntu jammy-backports InRelease [107 kB]
Get:5 http://cn.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:6 http://cn.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [948 kB]
Get:7 http://cn.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [890 kB]
Fetched 2,183 kB in 9s (248 kB/s)
Reading package lists... Done
W: https://mirrors.aliyun.com/kubernetes/apt/dists/kubernetes-xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
root@k8s-master1:~# apt -y install apt-transport-https ca-certificates curl software-properties-common
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
ca-certificates is already the newest version (20211016ubuntu0.22.04.1).
ca-certificates set to manually installed.
curl is already the newest version (7.81.0-1ubuntu1.8).
curl set to manually installed.
apt-transport-https is already the newest version (2.4.8).
The following additional packages will be installed:
python3-software-properties
The following packages will be upgraded:
python3-software-properties software-properties-common
2 upgraded, 0 newly installed, 0 to remove and 70 not upgraded.
Need to get 42.9 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://cn.archive.ubuntu.com/ubuntu jammy-updates/main amd64 software-properties-common all 0.99.22.6 [14.1 kB]
Get:2 http://cn.archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-software-properties all 0.99.22.6 [28.8 kB]
Fetched 42.9 kB in 1s (39.6 kB/s)
(Reading database ... 73785 files and directories currently installed.)
Preparing to unpack .../software-properties-common_0.99.22.6_all.deb ...
Unpacking software-properties-common (0.99.22.6) over (0.99.22.2) ...
Preparing to unpack .../python3-software-properties_0.99.22.6_all.deb ...
Unpacking python3-software-properties (0.99.22.6) over (0.99.22.2) ...
Setting up python3-software-properties (0.99.22.6) ...
Setting up software-properties-common (0.99.22.6) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for dbus (1.12.20-2ubuntu4.1) ...
Scanning processes...
Scanning linux images...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
##安装GPG证书
root@k8s-master1:~# curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
##写入软件源信息
root@k8s-master1:~# add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
Repository: 'deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy stable'
Description:
Archive for codename: jammy components: stable
More info: http://mirrors.aliyun.com/docker-ce/linux/ubuntu
Adding repository.
Press [ENTER] to continue or Ctrl-c to cancel.
Adding deb entry to /etc/apt/sources.list.d/archive_uri-http_mirrors_aliyun_com_docker-ce_linux_ubuntu-jammy.list
Adding disabled deb-src entry to /etc/apt/sources.list.d/archive_uri-http_mirrors_aliyun_com_docker-ce_linux_ubuntu-jammy.list
Get:1 http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy InRelease [48.9 kB]
Hit:2 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease
Hit:3 http://cn.archive.ubuntu.com/ubuntu jammy InRelease
Get:4 http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages [13.6 kB]
Hit:5 http://cn.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:6 http://cn.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:7 http://cn.archive.ubuntu.com/ubuntu jammy-security InRelease
Fetched 62.5 kB in 1s (53.7 kB/s)
Reading package lists... Done
W: https://mirrors.aliyun.com/kubernetes/apt/dists/kubernetes-xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: http://mirrors.aliyun.com/docker-ce/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
##更新软件
root@k8s-master1:~# apt-get -y update
Hit:1 http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy InRelease
Hit:2 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease
Hit:3 http://cn.archive.ubuntu.com/ubuntu jammy InRelease
Hit:4 http://cn.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:5 http://cn.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:6 http://cn.archive.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
W: http://mirrors.aliyun.com/docker-ce/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://mirrors.aliyun.com/kubernetes/apt/dists/kubernetes-xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
##查看docker可安装的版本
root@k8s-master1:~# apt-cache madison docker-ce docker-ce-cli
docker-ce | 5:23.0.1-1~ubuntu.22.04~jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:23.0.0-1~ubuntu.22.04~jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.23~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.22~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.21~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.20~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.19~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.18~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.17~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.16~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.15~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.14~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce | 5:20.10.13~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:23.0.1-1~ubuntu.22.04~jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:23.0.0-1~ubuntu.22.04~jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:20.10.23~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:20.10.22~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:20.10.21~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:20.10.20~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:20.10.19~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:20.10.18~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:20.10.17~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:20.10.16~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:20.10.15~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:20.10.14~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
docker-ce-cli | 5:20.10.13~3-0~ubuntu-jammy | http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 Packages
##安装docker
root@k8s-master1:~# apt install -y docker-ce=5:20.10.23~3-0~ubuntu-jammy docker-ce-cli=5:20.10.23~3-0~ubuntu-jammy
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
containerd.io docker-ce-rootless-extras docker-scan-plugin libltdl7 libslirp0 pigz slirp4netns
Suggested packages:
aufs-tools cgroupfs-mount | cgroup-lite
The following NEW packages will be installed:
containerd.io docker-ce docker-ce-cli docker-ce-rootless-extras docker-scan-plugin libltdl7 libslirp0 pigz slirp4netns
0 upgraded, 9 newly installed, 0 to remove and 70 not upgraded.
Need to get 104 MB of archives.
After this operation, 389 MB of additional disk space will be used.
Get:1 http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 containerd.io amd64 1.6.18-1 [28.2 MB]
Get:2 http://cn.archive.ubuntu.com/ubuntu jammy/universe amd64 pigz amd64 2.6-1 [63.6 kB]
Get:3 http://cn.archive.ubuntu.com/ubuntu jammy/main amd64 libltdl7 amd64 2.4.6-15build2 [39.6 kB]
Get:4 http://cn.archive.ubuntu.com/ubuntu jammy/main amd64 libslirp0 amd64 4.6.1-1build1 [61.5 kB]
Get:5 http://cn.archive.ubuntu.com/ubuntu jammy/universe amd64 slirp4netns amd64 1.0.1-2 [28.2 kB]
15% [1 containerd.io 7,889 kB/28.2 MB 28%] 106 kB/s 15min 0s^Get:6 http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 docker-ce-cli amd64 5:20.10.23~3-0~ubuntu-jammy [42.6 MB]
Get:7 http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 docker-ce amd64 5:20.10.23~3-0~ubuntu-jammy [20.5 MB]
Get:8 http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 docker-ce-rootless-extras amd64 5:23.0.1-1~ubuntu.22.04~jammy [8,760 kB]
Get:9 http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy/stable amd64 docker-scan-plugin amd64 0.23.0~ubuntu-jammy [3,623 kB]
Fetched 104 MB in 16min 33s (105 kB/s)
Selecting previously unselected package pigz.
(Reading database ... 73785 files and directories currently installed.)
Preparing to unpack .../0-pigz_2.6-1_amd64.deb ...
Unpacking pigz (2.6-1) ...
Selecting previously unselected package containerd.io.
Preparing to unpack .../1-containerd.io_1.6.18-1_amd64.deb ...
Unpacking containerd.io (1.6.18-1) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../2-docker-ce-cli_5%3a20.10.23~3-0~ubuntu-jammy_amd64.deb ...
Unpacking docker-ce-cli (5:20.10.23~3-0~ubuntu-jammy) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../3-docker-ce_5%3a20.10.23~3-0~ubuntu-jammy_amd64.deb ...
Unpacking docker-ce (5:20.10.23~3-0~ubuntu-jammy) ...
Selecting previously unselected package docker-ce-rootless-extras.
Preparing to unpack .../4-docker-ce-rootless-extras_5%3a23.0.1-1~ubuntu.22.04~jammy_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:23.0.1-1~ubuntu.22.04~jammy) ...
Selecting previously unselected package docker-scan-plugin.
Preparing to unpack .../5-docker-scan-plugin_0.23.0~ubuntu-jammy_amd64.deb ...
Unpacking docker-scan-plugin (0.23.0~ubuntu-jammy) ...
Selecting previously unselected package libltdl7:amd64.
Preparing to unpack .../6-libltdl7_2.4.6-15build2_amd64.deb ...
Unpacking libltdl7:amd64 (2.4.6-15build2) ...
Selecting previously unselected package libslirp0:amd64.
Preparing to unpack .../7-libslirp0_4.6.1-1build1_amd64.deb ...
Unpacking libslirp0:amd64 (4.6.1-1build1) ...
Selecting previously unselected package slirp4netns.
Preparing to unpack .../8-slirp4netns_1.0.1-2_amd64.deb ...
Unpacking slirp4netns (1.0.1-2) ...
Setting up docker-scan-plugin (0.23.0~ubuntu-jammy) ...
Setting up containerd.io (1.6.18-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up libltdl7:amd64 (2.4.6-15build2) ...
Setting up docker-ce-cli (5:20.10.23~3-0~ubuntu-jammy) ...
Setting up libslirp0:amd64 (4.6.1-1build1) ...
Setting up pigz (2.6-1) ...
Setting up docker-ce-rootless-extras (5:23.0.1-1~ubuntu.22.04~jammy) ...
Setting up slirp4netns (1.0.1-2) ...
Setting up docker-ce (5:20.10.23~3-0~ubuntu-jammy) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
Scanning processes...
Scanning linux images...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
##启动docker并加入开机自启动
root@k8s-node1:~# systemctl start docker && systemctl enable docker
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
##参数优化，配置镜像加速并使用systemd
root@k8s-node1:~# mkdir -p /etc/docker
root@k8s-node1:~# tee /etc/docker/daemon.json <<-'EOF'
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": ["https://9916w1ow.mirror.aliyuncs.com"]
}
EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": ["https://9916w1ow.mirror.aliyuncs.com"]
}
##重启docker使配置生效
root@k8s-node1:~# systemctl daemon-reload && sudo systemctl restart docker
root@k8s-node1:~# cat /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": ["https://9916w1ow.mirror.aliyuncs.com"]
}
##下载二进制安装包
root@k8s-master1:~# wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.1/cri-dockerd-0.3.1.amd64.tgz
##解压安装包
root@k8s-node1:~# tar xvf cri-dockerd-0.3.1.amd64.tgz
cri-dockerd/
cri-dockerd/cri-dockerd
##拷贝二进制文件到可执行目录
root@k8s-node1:~# cp cri-dockerd/cri-dockerd /usr/local/bin/
##编写cri-docker.service
root@k8s-node1:~# vim /lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket
[Service]
Type=notify
ExecStart=/usr/local/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
##配置cri-docker.socket文件
root@k8s-node1:~# vim /etc/systemd/system/cri-docker.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service
[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
##启动服务
root@k8s-node1:~# systemctl daemon-reload && systemctl restart cri-docker && systemctl enable cri-docker
root@k8s-node1:~# systemctl enable --now cri-docker.socket
Created symlink /etc/systemd/system/sockets.target.wants/cri-docker.socket → /etc/systemd/system/cri-docker.socket.
##查看cri-docker启动状态
root@k8s-node1:~# systemctl status cri-docker.service
● cri-docker.service - CRI Interface for Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/cri-docker.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2023-03-19 05:09:25 UTC; 57s ago
TriggeredBy: ● cri-docker.socket
Docs: https://docs.mirantis.com
Main PID: 6443 (cri-dockerd)
Tasks: 7
Memory: 8.8M
CPU: 63ms
CGroup: /system.slice/cri-docker.service
└─6443 /usr/local/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9
Mar 19 05:09:25 k8s-node1 cri-dockerd[6443]: time="2023-03-19T05:09:25Z" level=info msg="The binary conntrack is not installed, this can cause fail>
Mar 19 05:09:25 k8s-node1 cri-dockerd[6443]: time="2023-03-19T05:09:25Z" level=info msg="The binary conntrack is not installed, this can cause fail>
Mar 19 05:09:25 k8s-node1 cri-dockerd[6443]: time="2023-03-19T05:09:25Z" level=info msg="Loaded network plugin cni"
Mar 19 05:09:25 k8s-node1 cri-dockerd[6443]: time="2023-03-19T05:09:25Z" level=info msg="Docker cri networking managed by network plugin cni"
Mar 19 05:09:25 k8s-node1 cri-dockerd[6443]: time="2023-03-19T05:09:25Z" level=info msg="Docker Info: &{ID:JKKP:U24V:NLTT:V37V:Y3BB:7L5O:4VTR:DMVC:>
Mar 19 05:09:25 k8s-node1 cri-dockerd[6443]: time="2023-03-19T05:09:25Z" level=info msg="Setting cgroupDriver systemd"
Mar 19 05:09:25 k8s-node1 cri-dockerd[6443]: time="2023-03-19T05:09:25Z" level=info msg="Docker cri received runtime config &RuntimeConfig{NetworkC>
Mar 19 05:09:25 k8s-node1 cri-dockerd[6443]: time="2023-03-19T05:09:25Z" level=info msg="Starting the GRPC backend for the Docker CRI interface."
Mar 19 05:09:25 k8s-node1 cri-dockerd[6443]: time="2023-03-19T05:09:25Z" level=info msg="Start cri-dockerd grpc backend"
Mar 19 05:09:25 k8s-node1 systemd[1]: Started CRI Interface for Docker Application Container Engine.
##使用阿里云源
root@k8s-node1:~# apt-get update && apt-get install -y apt-transport-https
Hit:1 http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy InRelease
Hit:2 http://cn.archive.ubuntu.com/ubuntu jammy InRelease
Get:3 http://cn.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 http://cn.archive.ubuntu.com/ubuntu jammy-backports InRelease [107 kB]
Get:5 http://cn.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Fetched 336 kB in 2s (144 kB/s)
Reading package lists... Done
W: http://mirrors.aliyun.com/docker-ce/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
apt-transport-https is already the newest version (2.4.8).
0 upgraded, 0 newly installed, 0 to remove and 69 not upgraded.
root@k8s-node1:~# curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
100 1210 100 1210 0 0 3710 0 --:--:-- --:--:-- --:--:-- 3711
OK
root@k8s-node1:~# cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
##检查更新
root@k8s-node1:~# apt-get update
Hit:1 http://mirrors.aliyun.com/docker-ce/linux/ubuntu jammy InRelease
Get:2 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease [8,993 B]
Hit:3 http://cn.archive.ubuntu.com/ubuntu jammy InRelease
Ign:4 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages
Get:4 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages [64.5 kB]
Get:5 http://cn.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:6 http://cn.archive.ubuntu.com/ubuntu jammy-backports InRelease [107 kB]
Get:7 http://cn.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Fetched 409 kB in 3s (143 kB/s)
Reading package lists... Done
W: http://mirrors.aliyun.com/docker-ce/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://mirrors.aliyun.com/kubernetes/apt/dists/kubernetes-xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
##安装kubeadm
root@k8s-node1:~# apt-get install kubelet=1.24.10-00 kubeadm=1.24.10-00 kubectl=1.24.10-00
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
conntrack cri-tools ebtables kubernetes-cni socat
The following NEW packages will be installed:
conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 8 newly installed, 0 to remove and 69 not upgraded.
Need to get 85.0 MB of archives.
After this operation, 332 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 cri-tools amd64 1.26.0-00 [18.9 MB]
Get:2 http://cn.archive.ubuntu.com/ubuntu jammy/main amd64 conntrack amd64 1:1.4.6-2build2 [33.5 kB]
Get:3 http://cn.archive.ubuntu.com/ubuntu jammy/main amd64 ebtables amd64 2.0.11-4build2 [84.9 kB]
Get:4 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 1.2.0-00 [27.6 MB]
Get:5 http://cn.archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB]
Get:6 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubelet amd64 1.24.10-00 [19.4 MB]
Get:7 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubectl amd64 1.24.10-00 [9,432 kB]
Get:8 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubeadm amd64 1.24.10-00 [9,114 kB]
Ign:5 http://cn.archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4
Get:5 http://cn.archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB]
Fetched 84.9 MB in 1min 46s (803 kB/s)
Selecting previously unselected package conntrack.
(Reading database ... 73975 files and directories currently installed.)
Preparing to unpack .../0-conntrack_1%3a1.4.6-2build2_amd64.deb ...
Unpacking conntrack (1:1.4.6-2build2) ...
Selecting previously unselected package cri-tools.
Preparing to unpack .../1-cri-tools_1.26.0-00_amd64.deb ...
Unpacking cri-tools (1.26.0-00) ...
Selecting previously unselected package ebtables.
Preparing to unpack .../2-ebtables_2.0.11-4build2_amd64.deb ...
Unpacking ebtables (2.0.11-4build2) ...
Selecting previously unselected package kubernetes-cni.
Preparing to unpack .../3-kubernetes-cni_1.2.0-00_amd64.deb ...
Unpacking kubernetes-cni (1.2.0-00) ...
Selecting previously unselected package socat.
Preparing to unpack .../4-socat_1.7.4.1-3ubuntu4_amd64.deb ...
Unpacking socat (1.7.4.1-3ubuntu4) ...
Selecting previously unselected package kubelet.
Preparing to unpack .../5-kubelet_1.24.10-00_amd64.deb ...
Unpacking kubelet (1.24.10-00) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../6-kubectl_1.24.10-00_amd64.deb ...
Unpacking kubectl (1.24.10-00) ...
Selecting previously unselected package kubeadm.
Preparing to unpack .../7-kubeadm_1.24.10-00_amd64.deb ...
Unpacking kubeadm (1.24.10-00) ...
Setting up conntrack (1:1.4.6-2build2) ...
Setting up kubectl (1.24.10-00) ...
Setting up ebtables (2.0.11-4build2) ...
Setting up socat (1.7.4.1-3ubuntu4) ...
Setting up cri-tools (1.26.0-00) ...
Setting up kubernetes-cni (1.2.0-00) ...
Setting up kubelet (1.24.10-00) ...
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /lib/systemd/system/kubelet.service.
Setting up kubeadm (1.24.10-00) ...
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...
Scanning linux images...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
##用脚本提前下载好镜像
root@k8s-node1:~# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.24.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.24.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.24.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.24.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.7
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.6-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.8.6
v1.24.10: Pulling from google_containers/kube-apiserver
fc251a6e7981: Pull complete
bffbec874b09: Pull complete
6374258c2a53: Pull complete
Digest: sha256:425a9728de960c4fe2d162669fb91e2b28061026d77bb48c8730c0ad9ee15149
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.24.10
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.24.10
v1.24.10: Pulling from google_containers/kube-controller-manager
fc251a6e7981: Already exists
bffbec874b09: Already exists
63cf3ce8a444: Pull complete
Digest: sha256:9d78dd19663d9eb3b9f660d3bae836c4cda685d0f1debc69f8a476cb9d796b66
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.24.10
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.24.10
v1.24.10: Pulling from google_containers/kube-scheduler
fc251a6e7981: Already exists
bffbec874b09: Already exists
a83371edc6c5: Pull complete
Digest: sha256:77f9414f4003e3c471a51dce48a6b2ef5c6753fd41ac336080d1ff562a3862a3
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.24.10
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.24.10
v1.24.10: Pulling from google_containers/kube-proxy
318b659927b4: Pull complete
dd22051c691b: Pull complete
294b864c5e01: Pull complete
Digest: sha256:5d5748d409be932fba3db18b86673e9b3542dff373d9c26ed65e4c89add6102b
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.24.10
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.24.10
3.7: Pulling from google_containers/pause
7582c2cc65ef: Pull complete
Digest: sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.7
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.7
3.5.6-0: Pulling from google_containers/etcd
8fdb1fc20e24: Pull complete
436b7dc2bc75: Pull complete
05135444fe12: Pull complete
e462d783cd7f: Pull complete
ecff8ef6851d: Pull complete
Digest: sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.6-0
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.6-0
1.8.6: Pulling from google_containers/coredns
d92bdee79785: Pull complete
6e1b7c06e42d: Pull complete
Digest: sha256:5b6ec0d6de9baaf3e92d0f66cd96a25b9edbce8716f5f15dcd1a616b3abd590e
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.8.6
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.8.6
##初始化kubernetes
root@k8s-master1:~# kubeadm init --apiserver-advertise-address=172.31.6.201 --apiserver-bind-port=6443 --kubernetes-version=v1.24.10 --pod-network-cidr=10.200.0.0/16 --service-cidr=172.31.5.0/24 --service-dns-domain=cluster.local --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --ignore-preflight-errors=swap --cri-socket unix:///var/run/cri-dockerd.sock
[init] Using Kubernetes version: v1.24.10
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [172.31.0.1 172.31.6.201]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master1 localhost] and IPs [172.31.6.201 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master1 localhost] and IPs [172.31.6.201 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 33.025391 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: zcpj0c.ynp9jbztcml2v55c
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.31.6.201:6443 --token zcpj0c.ynp9jbztcml2v55c \
--discovery-token-ca-cert-hash sha256:d311556d7062595e4b34cf19adbb39d2f193e4d2b7cb87e49f3c6bc1fbed25d9
##node 节点执行kubectl命令验证
root@k8s-node1:~# kubeadm join 172.31.6.201:6443 --token zcpj0c.ynp9jbztcml2v55c \
--discovery-token-ca-cert-hash sha256:d311556d7062595e4b34cf19adbb39d2f193e4d2b7cb87e49f3c6bc1fbed25d9 --cri-socket unix:///var/run/cri-dockerd.sock
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
root@k8s-node2:~# kubeadm join 172.31.6.201:6443 --token zcpj0c.ynp9jbztcml2v55c \
--discovery-token-ca-cert-hash sha256:d311556d7062595e4b34cf19adbb39d2f193e4d2b7cb87e49f3c6bc1fbed25d9 --cri-socket unix:///var/run/cri-dockerd.sock
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
root@k8s-node3:~# kubeadm join 172.31.6.201:6443 --token zcpj0c.ynp9jbztcml2v55c \
--discovery-token-ca-cert-hash sha256:d311556d7062595e4b34cf19adbb39d2f193e4d2b7cb87e49f3c6bc1fbed25d9 --cri-socket unix:///var/run/cri-dockerd.sock
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
##分发kubeconfig认证文件
root@k8s-node1:~# mkdir /root/.kube -p
root@k8s-node2:~# mkdir /root/.kube -p
root@k8s-node3:~# mkdir /root/.kube -p
##配置master kubectl配置
root@k8s-master1:~# mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
root@k8s-master1:~# export KUBECONFIG=/etc/kubernetes/admin.conf
root@k8s-master1:~# scp /root/.kube/config 172.31.6.202:/root/.kube/
root@k8s-master1:~# scp /root/.kube/config 172.31.6.203:/root/.kube/
root@k8s-master1:~# scp /root/.kube/config 172.31.6.204:/root/.kube/
##下载二进制安装包
root@k8s-master1:/usr/local/src# wget https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz
--2023-03-19 06:28:09-- https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz
##解压
root@k8s-master1:/usr/local/src# tar xvf helm-v3.11.1-linux-amd64.tar.gz
linux-amd64/
linux-amd64/helm
linux-amd64/LICENSE
linux-amd64/README.md
root@k8s-master1:/usr/local/src# mv linux-amd64/helm /usr/local/bin/
##添加helm源
root@k8s-master1:/usr/local/bin# helm repo add hybridnet https://alibaba.github.io/hybridnet/
"hybridnet" has been added to your repositories
root@k8s-master1:/usr/local/bin# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hybridnet" chart repository
Update Complete. ⎈Happy Helming!⎈
##配置overlay pod网络(使用kubeadm初始化时指定的pod网络)，如果不指定--set init.cidr=10.200.0.0/16默认会使用100.64.0.0/16
root@k8s-master1:/usr/local/bin# helm install hybridnet hybridnet/hybridnet -n kube-system --set init.cidr=10.200.0.0/16
W0319 06:36:24.544931 34262 warnings.go:70] spec.template.spec.nodeSelector[beta.kubernetes.io/os]: deprecated since v1.14; use "kubernetes.io/os" instead
W0319 06:36:24.545018 34262 warnings.go:70] spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead
NAME: hybridnet
LAST DEPLOYED: Sun Mar 19 06:36:21 2023
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
##查看pod状态
root@k8s-master1:/usr/local/bin# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-typha-59bf6c985c-c7tmk 0/1 ContainerCreating 0 33s
kube-system calico-typha-59bf6c985c-w7qsc 0/1 ContainerCreating 0 33s
kube-system calico-typha-59bf6c985c-xnh77 0/1 ContainerCreating 0 33s
kube-system coredns-7f74c56694-76kpr 0/1 Pending 0 21m
kube-system coredns-7f74c56694-gklzd 0/1 Pending 0 21m
kube-system etcd-k8s-master1 1/1 Running 0 21m
kube-system hybridnet-daemon-5tnd7 0/2 Init:0/1 0 33s
kube-system hybridnet-daemon-mkkx8 0/2 Init:0/1 0 33s
kube-system hybridnet-daemon-qgvzw 0/2 Init:0/1 0 33s
kube-system hybridnet-daemon-v98vj 0/2 Init:0/1 0 33s
kube-system hybridnet-manager-57f55874cf-94grf 0/1 Pending 0 33s
kube-system hybridnet-manager-57f55874cf-llzrc 0/1 Pending 0 33s
kube-system hybridnet-manager-57f55874cf-ng6h7 0/1 Pending 0 33s
kube-system hybridnet-webhook-7f4fcb5646-dj964 0/1 Pending 0 36s
kube-system hybridnet-webhook-7f4fcb5646-p5rrj 0/1 Pending 0 36s
kube-system hybridnet-webhook-7f4fcb5646-pmwvn 0/1 Pending 0 36s
kube-system kube-apiserver-k8s-master1 1/1 Running 0 21m
kube-system kube-controller-manager-k8s-master1 1/1 Running 0 21m
kube-system kube-proxy-49kkc 1/1 Running 0 21m
kube-system kube-proxy-q6m5f 1/1 Running 0 15m
kube-system kube-proxy-w8pl6 1/1 Running 0 15m
kube-system kube-proxy-xjfdv 1/1 Running 0 15m
kube-system kube-scheduler-k8s-master1 1/1 Running 0 21m
root@k8s-master1:/usr/local/bin# kubectl describe pod hybridnet-manager-57f55874cf-94grf -n kube-system
Name: hybridnet-manager-57f55874cf-94grf
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: <none>
Labels: app=hybridnet
component=manager
pod-template-hash=57f55874cf
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/hybridnet-manager-57f55874cf
Containers:
hybridnet-manager:
Image: docker.io/hybridnetdev/hybridnet:v0.8.0
Port: 9899/TCP
Host Port: 9899/TCP
Command:
/hybridnet/hybridnet-manager
--default-ip-retain=true
--feature-gates=MultiCluster=false,VMIPRetain=false
--controller-concurrency=Pod=1,IPAM=1,IPInstance=1
--kube-client-qps=300
--kube-client-burst=600
--metrics-port=9899
Environment:
DEFAULT_NETWORK_TYPE: Overlay
DEFAULT_IP_FAMILY: IPv4
NAMESPACE: kube-system (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mt2z4 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-mt2z4:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: node-role.kubernetes.io/master=
Tolerations: :NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m6s default-scheduler 0/4 nodes are available: 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
##给节点打标签
root@k8s-master1:/usr/local/bin# kubectl label node k8s-node1 node-role.kubernetes.io/master=
node/k8s-node1 labeled
root@k8s-master1:/usr/local/bin# kubectl label node k8s-node2 node-role.kubernetes.io/master=
node/k8s-node2 labeled
root@k8s-master1:/usr/local/bin# kubectl label node k8s-node3 node-role.kubernetes.io/master=
node/k8s-node3 labeled
##为node主机添加underlay network标签
root@k8s-master1:/usr/local/bin# kubectl label node k8s-node1 network=underlay-nethost
node/k8s-node1 labeled
root@k8s-master1:/usr/local/bin# kubectl label node k8s-node2 network=underlay-nethost
node/k8s-node2 labeled
root@k8s-master1:/usr/local/bin# kubectl label node k8s-node3 network=underlay-nethost
node/k8s-node3 labeled
##创建underlay网络
root@k8s-master1:~/underlay-cases-files# ls
1.create-underlay-network.yaml 2.tomcat-app1-overlay.yaml 3.tomcat-app1-underlay.yaml 4-pod-underlay.yaml
root@k8s-master1:~/underlay-cases-files# ll
total 24
drwxr-xr-x 2 root root 4096 Mar 19 06:56 ./
drwx------ 9 root root 4096 Mar 19 06:56 ../
-rw-r--r-- 1 root root 464 Mar 5 12:12 1.create-underlay-network.yaml
-rw-r--r-- 1 root root 1383 Mar 5 12:12 2.tomcat-app1-overlay.yaml
-rw-r--r-- 1 root root 1532 Mar 5 12:12 3.tomcat-app1-underlay.yaml
-rw-r--r-- 1 root root 219 Mar 5 12:12 4-pod-underlay.yaml
root@k8s-master1:~/underlay-cases-files# vim 1.create-underlay-network.yaml
root@k8s-master1:~/underlay-cases-files# kubectl apply -f 1.create-underlay-network.yaml
network.networking.alibaba.com/underlay-network1 created
subnet.networking.alibaba.com/underlay-network1 created
root@k8s-master1:~/underlay-cases-files# kubectl get network
NAME NETID TYPE MODE V4TOTAL V4USED V4AVAILABLE LASTALLOCATEDV4SUBNET V6TOTAL V6USED V6AVAILABLE LASTALLOCATEDV6SUBNET
init 4 Overlay 65534 2 65532 init 0 0 0
underlay-network1 0 Underlay 254 0 254 underlay-network1 0 0 0
##创建namaspace
root@k8s-master1:~/underlay-cases-files# kubectl create ns myserver
namespace/myserver created
root@k8s-master1:~/underlay-cases-files# ls
1.create-underlay-network.yaml 2.tomcat-app1-overlay.yaml 3.tomcat-app1-underlay.yaml 4-pod-underlay.yaml
root@k8s-master1:~/underlay-cases-files# vim 2.tomcat-app1-overlay.yaml
##创建测试pod和service
root@k8s-master1:~/underlay-cases-files# kubectl apply -f 2.tomcat-app1-overlay.yaml
deployment.apps/myserver-tomcat-app1-deployment-overlay created
service/myserver-tomcat-app1-service-overlay created
##查看pod和service
root@k8s-master1:~/underlay-cases-files# kubectl get pod,svc -o wide -n myserver
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/myserver-tomcat-app1-deployment-overlay-f8dbf4964-hrhmg 0/1 ContainerCreating 0 28s <none> k8s-node2 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/myserver-tomcat-app1-service-overlay NodePort 172.31.5.64 <none> 80:30003/TCP 28s app=myserver-tomcat-app1-overlay-selector
##验证overlay pod通信
root@k8s-master1:~/underlay-cases-files# kubectl exec -it myserver-tomcat-app1-service-overlay bash -n myserver
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Error from server (NotFound): pods "myserver-tomcat-app1-service-overlay" not found
root@k8s-master1:~/underlay-cases-files# kubectl exec -it myserver-tomcat-app1-deployment-overlay-f8dbf4964-hrhmg bash -n myserver
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@myserver-tomcat-app1-deployment-overlay-f8dbf4964-hrhmg /]# ping www.baidu.com
PING www.a.shifen.com (14.119.104.189) 56(84) bytes of data.
64 bytes from 14.119.104.189 (14.119.104.189): icmp_seq=1 ttl=127 time=35.7 ms
64 bytes from 14.119.104.189 (14.119.104.189): icmp_seq=2 ttl=127 time=33.3 ms
64 bytes from 14.119.104.189 (14.119.104.189): icmp_seq=3 ttl=127 time=40.1 ms
^C
--- www.a.shifen.com ping statistics ---
4 packets transmitted, 3 received, 25% packet loss, time 3004ms
rtt min/avg/max/mdev = 33.373/36.433/40.181/2.825 ms
##underlay pod可以与overlaypod共存(混合使用)
root@k8s-master1:~/underlay-cases-files# kubectl apply -f 3.tomcat-app1-underlay.yaml
deployment.apps/myserver-tomcat-app1-deployment-underlay created
service/myserver-tomcat-app1-service-underlay created
##查看创建的pod和service
root@k8s-master1:~/underlay-cases-files# kubectl get pod,svc -o wide -n myserver
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/myserver-tomcat-app1-deployment-overlay-f8dbf4964-hrhmg 1/1 Running 0 14m 10.200.0.3 k8s-node2 <none> <none>
pod/myserver-tomcat-app1-deployment-underlay-5f7dd46d56-n7lcc 1/1 Running 0 86s 172.31.6.1 k8s-node3 <none> <none>
##访问pod IP
#查看service信息
root@k8s-master1:~/underlay-cases-files# kubectl get svc -o wide -n myserver
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
myserver-tomcat-app1-service-overlay NodePort 172.31.5.109 <none> 80:30003/TCP 20m app=myserver-tomcat-app1-overlay-selector
myserver-tomcat-app1-service-underlay ClusterIP 172.31.5.208 <none> 80/TCP 6m31s app=myserver-tomcat-app1-underlay-selector
注意：后期如果要访问SVC则需要在网络设备配置静态路由，打通从客户端到SVC的通信，把请求转发到具体的kubernetes node节点响应，如：
#windows10 以管理员身份运行cmd添加路由
route add 172.31.5.0 MASK 255.255.255.0 -p 172.31.7.102
~# route add -net 172.31.5.0 netmask 255.255.255.0 gateway 172.31.6.204
3、总结网络组件flannel vxlan模式的网络通信流程
flannel通信流程-1.pod->2.cni->3.eth0：
1：源pod发起请求，此时报文中源IP为pod的eth0 的ip,源mac 为pod的eth0的mac，目的Pod为目的Pod的IP，目的mac为网关(cni0)的MAC
2：数据报文通过veth peer发送给网关cni0,检查目的mac就是发给自己的，cni0进行目标IP检查,如果是同一个网桥的报文就直接转发，不是的话就发送给flannel.1, 此时报文被修改如下：
源IP: Pod IP，10.100.2.2
目的IP: Pod IP，10.100.1.6
源MAC： 源POD MAC
目的MAC：cni mAC
3：到达flannel.1,检查目的mac就是发给自己的，开始匹配路由表,先实现overlay报文的内层封装(主要是修改目的Pod的对端flannel.1的 MAC、源MAC为当前宿主机flannel.1的MAC)：
源IP: Pod IP，10.100.2.2
目的IP: Pod IP，10.100.1.6
源MAC: 9e:68:76:c4:64:0f，源Pod所在宿主机flannel.1的MAC
目的MAC：de:34:a2:06:d3:56，目的Pod所在主机flannel.1的MAC
4.源宿主机基于UDP封装Vxlan报文：
VXLAN ID：1
UDP 源端口： 随机
UDP 目的端口： 8472
源IP #源Pod所在宿主机的物理网卡IP#
目的IP #目的Pod所在宿主机的物理网卡IP
源MAC: 00:0c:29:bc:fe:d8 #源Pod所在宿主机的物理网卡
目的MAC: 00:0c:29:56:e7:1b #目的Pod所在宿主机的物理网卡
5.报文到达目的宿主机物理网卡，接开始解封装报文：
目的主机解封装报文：
外层目的IP为本机物理网卡，解开后发现里面还有一层目的IP和目的MAC，发现目的IP为10.100.1.6，目的MAC为de:34:a2:06:d3:56(目的
6.报文到达目的宿主机flannel.1：
flannel.1检查报文的目的IP，发现是去往本机cni0的子网，将请求报文转发至cni0
目的IP：10.100.1.6 #目的Pod
源IP： 10.100.2.2 #源Pod
目的 MAC：de:34:a2:06:d3:56 #目的pod所在宿主机的flannel.1
源MAC：9e:68:76:c4:64:0f #源pod所在宿主机flannel.1的MAC
flannel.1的MAC)，然后将报文发送给flannel.1：
7.报文到达目的宿主机cni0:
cni0基于目的IP检查mac地址表，修改目的MAC为目的MAC后将来请求转给pod：
源IP： 10.100.2.2 #源Pod
目的IP： 10.100.1.6 #目的Pod
源MAC：cni0的MAC, b2:12:0d:e4:eb:46
目的MAC: 目的Pod的MAC f2:50:98:b4:ea:01
8.报文到达目的宿主机pod:
cni0收到报文返现去往10.100.1.6,检查MAC地址表发现是本地接口，然后通过网桥接口发给pod：
目的IP： 目的pod IP 10.100.1.6
源IP：源Pod IP 10.200.2.2
目的MAC： 目的pod MAC,f2:50:98:b4:ea:01
源MAC: cni0的MAC,b2:12:0d:e4:eb:46
4、总结网络组件calico IPIP模式的网络通信流程
calico通信流程- - 确 认pod虚拟网卡名
确认源pod与目的pod的宿主机网卡名称，用于下一步抓包
源pod：
# kubectl exec -it net-test2 bash #进入pod
[root@net-test2 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.200.218.62 netmask 255.255.255.255 broadcast 10.200.218.62
#ethtool -S eth0 #源pod确认网卡对儿
NIC statistics:
peer_ifindex: 8
rx_queue_0_xdp_packets: 0
8: cali2b2e7c9e43e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 3
root@k8s-node1:~# tcpdump -nn -vvv -i cali2b2e7c9e43e -vvv -nn ! port 22 and ! port 2379 and ! port 6443 and ! port 10250 and ! arp and ! port 53
目的pod：
#ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.200.151.205 netmask 255.255.255.255 broadcast 10.200.151.205
/# ethtool -S eth0 #目的pod网卡对儿
NIC statistics:
peer_ifindex: 16
rx_queue_0_xdp_packets: 0
16: cali32ecf57bfbe@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 7
root@k8s-node2:~# tcpdump -nn -vvv -i cali32ecf57bfbe -vvv -nn ! port 22 and ! port 2379 and ! port 6443 and ! port 10250 and ! arp and ! port 53
calico通信流程-pod产生请求报文到达宿主机虚拟网卡：
1.源pod发起请求, 报文到达宿主机与pod对应的网卡：
root@net-test2 /]# curl 10.200.151.205
2.报文到达在宿主机与pod对应的网卡：cali2b2e7c9e43e
root@k8s-node1:~# tcpdump -nn -vvv -i cali2b2e7c9e43e -vvv -nn ! port 22 and ! port 2379 and ! port 6443 and ! port 10250 and ! arp and ! port 53
calico通信流程- -报文到达源宿主机tun0 ：
3. 报文到达源宿主机tun0
root@k8s-node1:~# tcpdump -nn -vvv -i tunl0 -vvv -nn ! port 22 and ! port 2379 and ! port 6443 and ! port 10250 and ! arp and ! port 53 -w 2-tunl0.pcap
此时报文的源IP为源pod IP，目的IP为目的pod IP。没有MAC 地址。
calico通信流程- - 报文到达源宿主机eth0 ：
4.报文到达源宿主机eth0：
root@k8s-node1:~# tcpdump -nn -vvv -i eth0 -vvv -nn ! port 22 and ! port 2379 and ! port 6443 and ! port 10250 and ! arp and ! port 53 and ! port 2380 and ! host 172.31.7.101 -w 3.eth0.pcap
此时报文为IPinIP格式，外层为源宿主机和目的宿主机的源MAC目的MAC、源IP及目的IP，内部为源pod IP及目的pod的IP，没有使用pod的MAC地址。
calico 通 信 流 程- - 报 文 到 达 目 的 宿 主 机eth0 ：
5.报文到达目的宿主机eth0
此时收到的是源宿主机的IPinIP报文，外层为源宿主机和目的宿主机的源MAC目的MAC、源IP及目的IP，内部为源pod IP及目的pod的IP，没有使用MAC地址，
解封装后发现是去往10.200.151.205
calico 通 信 流 程- - 报 文 到 达 目 的 宿 主 机tunl0 ：
6.报文到达目的宿主机tunl0:
root@k8s-node2:~# tcpdump -nn -vvv -i tunl0 -vvv -nn ! port 22 and ! port 2379 and ! port 6443 and ! port 10250 and ! arp and !
port 53 and ! port 2380 and ! host 172.31.7.101 -w 5-tunl0.pcap
报文到达目的宿主机tunl0，此时源IP为源pod IP，目的IP为目的pod的IP，没有MAC地址。
calico 通 信 流 程- - 报 文 到 达 源 宿 主 机 虚 拟 网 卡
7 报 文 到 达 目 的P Po od d 与 目 的 宿 主 机 对 应 的 网 卡cali32ecf57bfbe
root@k8s-node2:~# tcpdump -nn -vvv -i cali32ecf57bfbe -vvv -nn ! port 22 and ! port 2379 and ! port 6443 and ! port 10250 and ! arp and ! port 53 and ! port 2380 and ! host 172.31.7.101 -w 6-cali32ecf57bfbe.pcap
报文到达目的宿主机cali32ecf57bfbe 网卡。此时源IP为源Pod IP，源MCA为tunl0 MAC，目的IP为目的Pod IP，目的MAC为目的Pod MAC，随后报文被转发被目的MAC(目的Pod) 。
calico 通 信 流 程- - 报 文 到 达 目 的Pod：
8.报文到达目的Pod：
nginx-deployment-58f6c4967b-x959s:/# tcpdump - i eth0 -vvv -nn - 7dst-pod.p cap
nginx-deployment-58f6c4967b-x959s:/# apt-get install openssh-server
ginx-deployment-58f6c4967b-x959s:/# scp 7-dst-pod.acp 172.31.7.112:/root/
报文到达目的pod，目的pod接受请求并构建响应报文并原路返回给源pod。
5、基于二进制实现高可用的K8S集群环境
#二进制安装docker
root@k8s-harbor1:/usr/local/src# tar xvf docker-20.10.19-binary-install.tar.gz
./
./docker.socket
./containerd.service
./docker.service
./sysctl.conf
./limits.conf
./docker-compose-Linux-x86_64_1.28.6
./daemon.json
./docker-20.10.19.tgz
./docker-install.sh
root@k8s-harbor1:/usr/local/src# ls
containerd.service docker-20.10.19-binary-install.tar.gz docker-compose-Linux-x86_64_1.28.6 docker.service limits.conf
daemon.json docker-20.10.19.tgz docker-install.sh docker.socket sysctl.conf
root@k8s-harbor1:/usr/local/src# bash docker-install.sh
当前系统是Ubuntu 20.04.3 LTS \n \l,即将开始系统初始化、配置docker-compose与安装docker
docker/
docker/docker-proxy
docker/docker-init
docker/containerd
docker/containerd-shim
docker/dockerd
docker/runc
docker/ctr
docker/docker
docker/containerd-shim-runc-v2
正在启动docker server并设置为开机自启动!
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
docker server安装完成,欢迎进入docker世界!
#docker info信息
root@k8s-harbor1:/usr/local/src# docker info
Client:
Context: default
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 20.10.19
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc version: v1.1.4-0-g5fd4c4d1
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-81-generic
Operating System: Ubuntu 20.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.913GiB
Name: k8s-harbor1.example.com
ID: XY4C:NFC5:ZLZM:GKFH:QW5O:HYAK:QYF4:XGMQ:DP22:QAGV:VUEM:O4P6
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
harbor.magedu.com
harbor.myserver.com
172.31.7.105
127.0.0.0/8
Registry Mirrors:
https://9916w1ow.mirror.aliyuncs.com/
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
#⾃⾏使⽤openssl签发、⽣成私有key
root@k8s-harbor1:/apps# tar xvf harbor-offline-installer-v2.6.3.tgz
harbor/harbor.v2.6.3.tar.gz
harbor/prepare
harbor/LICENSE
harbor/install.sh
harbor/common.sh
harbor/harbor.yml.tmpl
root@k8s-harbor1:/apps# cd harbor/
root@k8s-harbor1:/apps/harbor# mkdir certs
root@k8s-harbor1:/apps/harbor# cd certs/
root@k8s-harbor1:/apps/harbor/certs# openssl genrsa -out harbor-ca.key
Generating RSA private key, 2048 bit long modulus (2 primes)
..............+++++
.................................................................................................................................................................................................+++++
e is 65537 (0x010001)
root@k8s-harbor1:/apps/harbor/certs# ls
harbor-ca.key
root@k8s-harbor1:/apps/harbor/certs# openssl req -x509 -new -nodes -key harbor-ca.key -subj "/CN=harbor.linuxarchitect.io" -days 7120 -out harbor-ca.crt
root@k8s-harbor1:/apps/harbor/certs# ls
harbor-ca.crt harbor-ca.key
#创建/data/harbor文件系统，后面使用它作harbor数据目录
root@k8s-harbor1:/apps/harbor/certs# mkfs.xfs /dev/sdb
meta-data=/dev/sdb isize=512 agcount=4, agsize=3276800 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=13107200, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=6400, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@k8s-harbor1:/apps/harbor/certs# mkdir /data/harbor -p
root@k8s-harbor1:/apps/harbor/certs# vi /etc/fstab
root@k8s-harbor1:/apps/harbor/certs# mount -a
root@k8s-harbor1:/apps/harbor/certs# df -h
Filesystem Size Used Avail Use% Mounted on
udev 936M 0 936M 0% /dev
tmpfs 196M 1.3M 195M 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 119G 7.8G 112G 7% /
tmpfs 980M 0 980M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 980M 0 980M 0% /sys/fs/cgroup
/dev/sda2 976M 107M 803M 12% /boot
/dev/loop1 68M 68M 0 100% /snap/lxd/22753
/dev/loop2 62M 62M 0 100% /snap/core20/1581
/dev/loop0 56M 56M 0 100% /snap/core18/2714
/dev/loop3 56M 56M 0 100% /snap/core18/2538
/dev/loop4 92M 92M 0 100% /snap/lxd/24061
/dev/loop5 50M 50M 0 100% /snap/snapd/18596
tmpfs 196M 0 196M 0% /run/user/0
/dev/sdb 50G 390M 50G 1% /data/harbor
#修改harbor配置⽂件
root@k8s-harbor1:/apps/harbor# grep -v "#" harbor.yml | grep -v "^$"
hostname: harbor.linuxarchitect.io
http:
port: 80
https:
port: 443
certificate: /apps/harbor/certs/harbor-ca.crt
private_key: /apps/harbor/certs/harbor-ca.key
harbor_admin_password: 123456
database:
password: root123
max_idle_conns: 100
max_open_conns: 900
data_volume: /data/harbor
#执⾏安装
root@k8s-harbor1:/apps/harbor# ./install.sh --with-trivy --with-chartmuseum
[Step 0]: checking if docker is installed ...
Note: docker version: 20.10.19
[Step 1]: checking docker-compose is installed ...
Note: docker-compose version: 1.28.6
[Step 2]: loading Harbor images ...
ed6c4a2423e8: Loading layer [==================================================>] 37.72MB/37.72MB
c8776c17f955: Loading layer [==================================================>] 119.1MB/119.1MB
5f120e4803a2: Loading layer [==================================================>] 7.538MB/7.538MB
58dbf2e2daf0: Loading layer [==================================================>] 1.185MB/1.185MB
Loaded image: goharbor/harbor-portal:v2.6.3
1c09f6f53666: Loading layer [==================================================>] 127MB/127MB
6f4e5282749c: Loading layer [==================================================>] 3.584kB/3.584kB
aeaec4bb0bba: Loading layer [==================================================>] 3.072kB/3.072kB
2429e67351ee: Loading layer [==================================================>] 2.56kB/2.56kB
b410f00133a5: Loading layer [==================================================>] 3.072kB/3.072kB
e8f4e26fb91d: Loading layer [==================================================>] 3.584kB/3.584kB
6223393fe2f5: Loading layer [==================================================>] 20.99kB/20.99kB
Loaded image: goharbor/harbor-log:v2.6.3
026a81c8f4b3: Loading layer [==================================================>] 5.759MB/5.759MB
f6b33ea5f36a: Loading layer [==================================================>] 4.096kB/4.096kB
2153df6352bf: Loading layer [==================================================>] 17.11MB/17.11MB
875409599e37: Loading layer [==================================================>] 3.072kB/3.072kB
81d6dfdd2e4a: Loading layer [==================================================>] 29.71MB/29.71MB
f2da68271a27: Loading layer [==================================================>] 47.61MB/47.61MB
Loaded image: goharbor/harbor-registryctl:v2.6.3
884a1f5c9843: Loading layer [==================================================>] 8.902MB/8.902MB
2ddf1b4d43bf: Loading layer [==================================================>] 25.08MB/25.08MB
81eb1b23bbb5: Loading layer [==================================================>] 4.608kB/4.608kB
9e58c43954b6: Loading layer [==================================================>] 25.88MB/25.88MB
Loaded image: goharbor/harbor-exporter:v2.6.3
bc0c522810c4: Loading layer [==================================================>] 8.902MB/8.902MB
9fbf2bb4023c: Loading layer [==================================================>] 3.584kB/3.584kB
98b4939c60c8: Loading layer [==================================================>] 2.56kB/2.56kB
045d575b759b: Loading layer [==================================================>] 102.3MB/102.3MB
1f466184225a: Loading layer [==================================================>] 103.1MB/103.1MB
Loaded image: goharbor/harbor-jobservice:v2.6.3
566336e05ba3: Loading layer [==================================================>] 119.9MB/119.9MB
44facf87cb8c: Loading layer [==================================================>] 3.072kB/3.072kB
d9d4a35201e9: Loading layer [==================================================>] 59.9kB/59.9kB
94110a561bc1: Loading layer [==================================================>] 61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v2.6.3
cacdd58f45fd: Loading layer [==================================================>] 5.754MB/5.754MB
3bb8655c543f: Loading layer [==================================================>] 8.735MB/8.735MB
6a7367cd9912: Loading layer [==================================================>] 15.88MB/15.88MB
296e3d2fd99f: Loading layer [==================================================>] 29.29MB/29.29MB
ca60e55cc5ea: Loading layer [==================================================>] 22.02kB/22.02kB
dfcecee54d65: Loading layer [==================================================>] 15.88MB/15.88MB
Loaded image: goharbor/notary-server-photon:v2.6.3
89043341a274: Loading layer [==================================================>] 43.84MB/43.84MB
9d023d2a97b6: Loading layer [==================================================>] 66.03MB/66.03MB
61c36c0b0ac2: Loading layer [==================================================>] 18.21MB/18.21MB
91f5f58d0482: Loading layer [==================================================>] 65.54kB/65.54kB
4f6269635897: Loading layer [==================================================>] 2.56kB/2.56kB
a98564e3a1f8: Loading layer [==================================================>] 1.536kB/1.536kB
173bcb5d11b3: Loading layer [==================================================>] 12.29kB/12.29kB
bcf887d06d88: Loading layer [==================================================>] 2.613MB/2.613MB
f20d7a395ffa: Loading layer [==================================================>] 379.9kB/379.9kB
Loaded image: goharbor/prepare:v2.6.3
c08fb233db28: Loading layer [==================================================>] 8.902MB/8.902MB
7da1ae52a1a6: Loading layer [==================================================>] 3.584kB/3.584kB
83cf11f9e71b: Loading layer [==================================================>] 2.56kB/2.56kB
edda2f6a1e4c: Loading layer [==================================================>] 83.92MB/83.92MB
8742d75dc503: Loading layer [==================================================>] 5.632kB/5.632kB
c9b2016053d9: Loading layer [==================================================>] 106.5kB/106.5kB
b1803d4adaaa: Loading layer [==================================================>] 44.03kB/44.03kB
2f1a1ce6928d: Loading layer [==================================================>] 84.87MB/84.87MB
04316e994e93: Loading layer [==================================================>] 2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v2.6.3
dab3848549c0: Loading layer [==================================================>] 119.1MB/119.1MB
Loaded image: goharbor/nginx-photon:v2.6.3
036fd2d1c3c4: Loading layer [==================================================>] 5.759MB/5.759MB
01b862c716d4: Loading layer [==================================================>] 4.096kB/4.096kB
2cee2bc0cc47: Loading layer [==================================================>] 3.072kB/3.072kB
998cc489ba67: Loading layer [==================================================>] 17.11MB/17.11MB
9ac4078c0b12: Loading layer [==================================================>] 17.9MB/17.9MB
Loaded image: goharbor/registry-photon:v2.6.3
659fadc582e7: Loading layer [==================================================>] 1.097MB/1.097MB
9f40d7d801f0: Loading layer [==================================================>] 5.889MB/5.889MB
2c2d126c3429: Loading layer [==================================================>] 169.1MB/169.1MB
79926d613579: Loading layer [==================================================>] 16.91MB/16.91MB
ddefe0eb652b: Loading layer [==================================================>] 4.096kB/4.096kB
6fc9e4dd18a4: Loading layer [==================================================>] 6.144kB/6.144kB
8ed4b870f096: Loading layer [==================================================>] 3.072kB/3.072kB
081c5440b79c: Loading layer [==================================================>] 2.048kB/2.048kB
0eff6e82f1ce: Loading layer [==================================================>] 2.56kB/2.56kB
8b5978d0cdd0: Loading layer [==================================================>] 2.56kB/2.56kB
3319997d6569: Loading layer [==================================================>] 2.56kB/2.56kB
1bf8f6ab8e25: Loading layer [==================================================>] 8.704kB/8.704kB
Loaded image: goharbor/harbor-db:v2.6.3
1db48763bb6c: Loading layer [==================================================>] 5.754MB/5.754MB
d051f6488df9: Loading layer [==================================================>] 8.735MB/8.735MB
7ea2b3f7e6f6: Loading layer [==================================================>] 14.47MB/14.47MB
1cb265dac032: Loading layer [==================================================>] 29.29MB/29.29MB
144284088cbf: Loading layer [==================================================>] 22.02kB/22.02kB
5ac59fccf6fb: Loading layer [==================================================>] 14.47MB/14.47MB
Loaded image: goharbor/notary-signer-photon:v2.6.3
2a92cdbfe831: Loading layer [==================================================>] 6.287MB/6.287MB
02ccf0420a48: Loading layer [==================================================>] 4.096kB/4.096kB
ade777038c1d: Loading layer [==================================================>] 3.072kB/3.072kB
46b542154bbd: Loading layer [==================================================>] 180.6MB/180.6MB
0f964f621fcb: Loading layer [==================================================>] 13.38MB/13.38MB
b884febe7cf8: Loading layer [==================================================>] 194.7MB/194.7MB
Loaded image: goharbor/trivy-adapter-photon:v2.6.3
1b69ea382f2d: Loading layer [==================================================>] 5.759MB/5.759MB
cd7f5f488cf0: Loading layer [==================================================>] 90.88MB/90.88MB
393f6387bb7a: Loading layer [==================================================>] 3.072kB/3.072kB
53997c89d1fd: Loading layer [==================================================>] 4.096kB/4.096kB
73b80a5cb9ac: Loading layer [==================================================>] 91.67MB/91.67MB
Loaded image: goharbor/chartmuseum-photon:v2.6.3
[Step 3]: preparing environment ...
[Step 4]: preparing harbor configs ...
prepare base dir is set to /apps/harbor
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /config/trivy-adapter/env
Generated configuration file: /config/chartserver/env
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
Note: stopping existing Harbor instance ...
Removing network harbor_harbor
WARNING: Network harbor_harbor not found.
Removing network harbor_harbor-chartmuseum
WARNING: Network harbor_harbor-chartmuseum not found.
[Step 5]: starting Harbor ...
➜
Chartmusuem will be deprecated as of Harbor v2.6.0 and start to be removed in v2.8.0 or later.
Please see discussion here for more details. https://github.com/goharbor/harbor/discussions/15057
Creating network "harbor_harbor" with the default driver
Creating network "harbor_harbor-chartmuseum" with the default driver
Creating harbor-log ... done
Creating redis ... done
Creating chartmuseum ... done
Creating registryctl ... done
Creating harbor-portal ... done
Creating harbor-db ... done
Creating registry ... done
Creating trivy-adapter ... done
Creating harbor-core ... done
Creating harbor-jobservice ... done
Creating nginx ... done
✔ ----Harbor has been installed and started successfully.----
#访问harbor, 新建baseimages项目
#配置高可用负载均衡
##安装keepalived和haproxy
root@k8s-ha1:~# apt update
root@k8s-ha1:~# apt install keepalived haproxy -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
##修改配置
root@k8s-ha1:~# cp /usr/share/doc/keepalived/samples/keepalived.conf.vrrp /etc/keepalived/keepalived.conf
root@k8s-ha1:~# vim /etc/keepalived/keepalived.conf
virtual_ipaddress {
192.168.7.188 dev eth1 label eth1:0
192.168.7.189 dev eth1 label eth1:1
192.168.7.190 dev eth1 label eth1:2
192.168.7.191 dev eth1 label eth1:3
192.168.7.192 dev eth1 label eth1:4
}
root@k8s-ha1:~# systemctl restart keepalived
root@k8s-ha1:~# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.31.7.109 netmask 255.255.248.0 broadcast 172.31.7.255
inet6 fe80::20c:29ff:fe9f:ffb9 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:9f:ff:b9 txqueuelen 1000 (Ethernet)
RX packets 25920 bytes 38338087 (38.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3323 bytes 253165 (253.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.7.109 netmask 255.255.248.0 broadcast 192.168.7.255
inet6 fe80::20c:29ff:fe9f:ffc3 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:9f:ff:c3 txqueuelen 1000 (Ethernet)
RX packets 2247 bytes 172776 (172.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2420 bytes 358069 (358.0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.7.188 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:9f:ff:c3 txqueuelen 1000 (Ethernet)
eth1:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.7.189 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:9f:ff:c3 txqueuelen 1000 (Ethernet)
eth1:2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.7.190 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:9f:ff:c3 txqueuelen 1000 (Ethernet)
eth1:3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.7.191 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:9f:ff:c3 txqueuelen 1000 (Ethernet)
eth1:4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.7.192 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:9f:ff:c3 txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 240 bytes 20752 (20.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 240 bytes 20752 (20.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root@k8s-ha1:~# ping 192.168.7.188
PING 192.168.7.188 (192.168.7.188) 56(84) bytes of data.
64 bytes from 192.168.7.188: icmp_seq=1 ttl=64 time=0.030 ms
64 bytes from 192.168.7.188: icmp_seq=2 ttl=64 time=0.039 ms
root@k8s-ha1:~# systemctl enable keepalived
Synchronizing state of keepalived.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable keepalived
##修改haproxy配置
root@k8s-ha1:~# vim /etc/haproxy/haproxy.cfg
listen k8s_api_nodes_6443
bind 192.168.7.188:6443
mode tcp
#balance leastconn
server 192.168.7.101 192.168.7.101:6443 check inter 2000 fall 3 rise 5
server 192.168.7.102 192.168.7.102:6443 check inter 2000 fall 3 rise 5
server 192.168.7.103 192.168.7.103:6443 check inter 2000 fall 3 rise 5
root@k8s-ha1:~# systemctl restart haproxy.service
root@k8s-ha1:~# systemctl enable haproxy.service
Synchronizing state of haproxy.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable haproxy
#检查haproxy监听端口
root@k8s-ha1:~# ss -tnlp
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 64 0.0.0.0:2049 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:37029 0.0.0.0:* users:(("rpc.mountd",pid=842,fd=9))
LISTEN 0 4096 0.0.0.0:46471 0.0.0.0:* users:(("rpc.mountd",pid=842,fd=17))
LISTEN 0 64 0.0.0.0:41033 0.0.0.0:*
LISTEN 0 4096 192.168.7.188:6443 0.0.0.0:* users:(("haproxy",pid=4598,fd=7))
LISTEN 0 4096 0.0.0.0:51563 0.0.0.0:* users:(("rpc.mountd",pid=842,fd=13))
LISTEN 0 4096 0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=749,fd=4),("systemd",pid=1,fd=79))
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=840,fd=13))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=916,fd=3))
LISTEN 0 128 127.0.0.1:6010 0.0.0.0:* users:(("sshd",pid=1939,fd=10))
LISTEN 0 64 [::]:2049 [::]:*
LISTEN 0 64 [::]:42985 [::]:*
LISTEN 0 4096 [::]:52457 [::]:* users:(("rpc.mountd",pid=842,fd=11))
LISTEN 0 4096 [::]:111 [::]:* users:(("rpcbind",pid=749,fd=6),("systemd",pid=1,fd=81))
LISTEN 0 4096 [::]:58933 [::]:* users:(("rpc.mountd",pid=842,fd=19))
LISTEN 0 4096 [::]:60821 [::]:* users:(("rpc.mountd",pid=842,fd=15))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=916,fd=4))
LISTEN 0 128 [::1]:6010 [::]:* users:(("sshd",pid=1939,fd=9))
#使用haproxy1节点做部署节点
#安装pip和git
root@haproxy1:~# apt install python3-pip git -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
build-essential cpp-9 dpkg-dev fakeroot g++ g++-9 gcc-9 gcc-9-base libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl
libasan5 libdpkg-perl libexpat1 libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-9-dev libpython3-dev libpython3.8 libpython3.8-dev
libpython3.8-minimal libpython3.8-stdlib libstdc++-9-dev python-pip-whl python3-dev python3-wheel python3.8 python3.8-dev python3.8-minimal
Suggested packages:
gcc-9-locales debian-keyring g++-multilib g++-9-multilib gcc-9-doc gcc-9-multilib git-daemon-run | git-daemon-sysvinit git-doc git-el git-email
git-gui gitk gitweb git-cvs git-mediawiki git-svn bzr libstdc++-9-doc python3.8-venv python3.8-doc binfmt-support
The following NEW packages will be installed:
build-essential dpkg-dev fakeroot g++ g++-9 libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libdpkg-perl libexpat1-dev
libfakeroot libfile-fcntllock-perl libpython3-dev libpython3.8-dev libstdc++-9-dev python-pip-whl python3-dev python3-pip python3-wheel
python3.8-dev
The following packages will be upgraded:
cpp-9 gcc-9 gcc-9-base git libasan5 libexpat1 libgcc-9-dev libpython3.8 libpython3.8-minimal libpython3.8-stdlib python3.8 python3.8-minimal
12 upgraded, 20 newly installed, 0 to remove and 238 not upgraded.
Need to get 28.8 MB/49.7 MB of archives.
After this operation, 90.6 MB of additional disk space will be used.
Get:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 python3.8 amd64 3.8.10-0ubuntu1~20.04.7 [387 kB]
Get:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 libpython3.8 amd64 3.8.10-0ubuntu1~20.04.7 [1626 kB]
Get:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 libpython3.8-stdlib amd64 3.8.10-0ubuntu1~20.04.7 [1675 kB]
Get:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 python3.8-minimal amd64 3.8.10-0ubuntu1~20.04.7 [1903 kB]
Get:5 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 libpython3.8-minimal amd64 3.8.10-0ubuntu1~20.04.7 [717 kB]
Get:6 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 libexpat1 amd64 2.2.9-1ubuntu0.6 [74.6 kB]
Get:7 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 libstdc++-9-dev amd64 9.4.0-1ubuntu1~20.04.1 [1722 kB]
Get:8 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 g++-9 amd64 9.4.0-1ubuntu1~20.04.1 [8420 kB]
Get:9 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 g++ amd64 4:9.3.0-1ubuntu2 [1604 B]
Get:10 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 libdpkg-perl all 1.19.7ubuntu3.2 [231 kB]
Get:11 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 dpkg-dev all 1.19.7ubuntu3.2 [679 kB]
Get:12 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 build-essential amd64 12.8ubuntu1.1 [4664 B]
Get:13 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 libfakeroot amd64 1.24-1 [25.7 kB]
Get:14 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 fakeroot amd64 1.24-1 [62.6 kB]
Get:15 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.10 [4534 kB]
Get:16 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 libalgorithm-diff-perl all 1.19.03-2 [46.6 kB]
Get:17 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 libalgorithm-diff-xs-perl amd64 0.04-6 [11.3 kB]
Get:18 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 libalgorithm-merge-perl all 0.08-3 [12.0 kB]
Get:19 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 libexpat1-dev amd64 2.2.9-1ubuntu0.6 [116 kB]
Get:20 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 libfile-fcntllock-perl amd64 0.22-3build4 [33.1 kB]
Get:21 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 libpython3.8-dev amd64 3.8.10-0ubuntu1~20.04.7 [3953 kB]
Get:22 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 libpython3-dev amd64 3.8.2-0ubuntu2 [7236 B]
Get:23 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.8 [1805 kB]
Get:24 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 python3.8-dev amd64 3.8.10-0ubuntu1~20.04.7 [514 kB]
Get:25 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 python3-dev amd64 3.8.2-0ubuntu2 [1212 B]
Get:26 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/universe amd64 python3-wheel all 0.34.2-1ubuntu0.1 [23.9 kB]
Get:27 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.8 [231 kB]
Fetched 28.8 MB in 1min 41s (286 kB/s)
Extracting templates from packages: 100%
(Reading database ... 76746 files and directories currently installed.)
Preparing to unpack .../00-python3.8_3.8.10-0ubuntu1~20.04.7_amd64.deb ...
Unpacking python3.8 (3.8.10-0ubuntu1~20.04.7) over (3.8.10-0ubuntu1~20.04) ...
Preparing to unpack .../01-libpython3.8_3.8.10-0ubuntu1~20.04.7_amd64.deb ...
Unpacking libpython3.8:amd64 (3.8.10-0ubuntu1~20.04.7) over (3.8.10-0ubuntu1~20.04) ...
Preparing to unpack .../02-libpython3.8-stdlib_3.8.10-0ubuntu1~20.04.7_amd64.deb ...
Unpacking libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04.7) over (3.8.10-0ubuntu1~20.04) ...
Preparing to unpack .../03-python3.8-minimal_3.8.10-0ubuntu1~20.04.7_amd64.deb ...
Unpacking python3.8-minimal (3.8.10-0ubuntu1~20.04.7) over (3.8.10-0ubuntu1~20.04) ...
Preparing to unpack .../04-libpython3.8-minimal_3.8.10-0ubuntu1~20.04.7_amd64.deb ...
Unpacking libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.7) over (3.8.10-0ubuntu1~20.04) ...
Preparing to unpack .../05-libexpat1_2.2.9-1ubuntu0.6_amd64.deb ...
Unpacking libexpat1:amd64 (2.2.9-1ubuntu0.6) over (2.2.9-1build1) ...
Preparing to unpack .../06-libasan5_9.4.0-1ubuntu1~20.04.1_amd64.deb ...
Unpacking libasan5:amd64 (9.4.0-1ubuntu1~20.04.1) over (9.3.0-17ubuntu1~20.04) ...
Preparing to unpack .../07-gcc-9_9.4.0-1ubuntu1~20.04.1_amd64.deb ...
Unpacking gcc-9 (9.4.0-1ubuntu1~20.04.1) over (9.3.0-17ubuntu1~20.04) ...
Preparing to unpack .../08-libgcc-9-dev_9.4.0-1ubuntu1~20.04.1_amd64.deb ...
Unpacking libgcc-9-dev:amd64 (9.4.0-1ubuntu1~20.04.1) over (9.3.0-17ubuntu1~20.04) ...
Preparing to unpack .../09-cpp-9_9.4.0-1ubuntu1~20.04.1_amd64.deb ...
Unpacking cpp-9 (9.4.0-1ubuntu1~20.04.1) over (9.3.0-17ubuntu1~20.04) ...
Preparing to unpack .../10-gcc-9-base_9.4.0-1ubuntu1~20.04.1_amd64.deb ...
Unpacking gcc-9-base:amd64 (9.4.0-1ubuntu1~20.04.1) over (9.3.0-17ubuntu1~20.04) ...
Selecting previously unselected package libstdc++-9-dev:amd64.
Preparing to unpack .../11-libstdc++-9-dev_9.4.0-1ubuntu1~20.04.1_amd64.deb ...
Unpacking libstdc++-9-dev:amd64 (9.4.0-1ubuntu1~20.04.1) ...
Selecting previously unselected package g++-9.
Preparing to unpack .../12-g++-9_9.4.0-1ubuntu1~20.04.1_amd64.deb ...
Unpacking g++-9 (9.4.0-1ubuntu1~20.04.1) ...
Selecting previously unselected package g++.
Preparing to unpack .../13-g++_4%3a9.3.0-1ubuntu2_amd64.deb ...
Unpacking g++ (4:9.3.0-1ubuntu2) ...
Selecting previously unselected package libdpkg-perl.
Preparing to unpack .../14-libdpkg-perl_1.19.7ubuntu3.2_all.deb ...
Unpacking libdpkg-perl (1.19.7ubuntu3.2) ...
Selecting previously unselected package dpkg-dev.
Preparing to unpack .../15-dpkg-dev_1.19.7ubuntu3.2_all.deb ...
Unpacking dpkg-dev (1.19.7ubuntu3.2) ...
Selecting previously unselected package build-essential.
Preparing to unpack .../16-build-essential_12.8ubuntu1.1_amd64.deb ...
Unpacking build-essential (12.8ubuntu1.1) ...
Selecting previously unselected package libfakeroot:amd64.
Preparing to unpack .../17-libfakeroot_1.24-1_amd64.deb ...
Unpacking libfakeroot:amd64 (1.24-1) ...
Selecting previously unselected package fakeroot.
Preparing to unpack .../18-fakeroot_1.24-1_amd64.deb ...
Unpacking fakeroot (1.24-1) ...
Preparing to unpack .../19-git_1%3a2.25.1-1ubuntu3.10_amd64.deb ...
Unpacking git (1:2.25.1-1ubuntu3.10) over (1:2.25.1-1ubuntu3.1) ...
Selecting previously unselected package libalgorithm-diff-perl.
Preparing to unpack .../20-libalgorithm-diff-perl_1.19.03-2_all.deb ...
Unpacking libalgorithm-diff-perl (1.19.03-2) ...
Selecting previously unselected package libalgorithm-diff-xs-perl.
Preparing to unpack .../21-libalgorithm-diff-xs-perl_0.04-6_amd64.deb ...
Unpacking libalgorithm-diff-xs-perl (0.04-6) ...
Selecting previously unselected package libalgorithm-merge-perl.
Preparing to unpack .../22-libalgorithm-merge-perl_0.08-3_all.deb ...
Unpacking libalgorithm-merge-perl (0.08-3) ...
Selecting previously unselected package libexpat1-dev:amd64.
Preparing to unpack .../23-libexpat1-dev_2.2.9-1ubuntu0.6_amd64.deb ...
Unpacking libexpat1-dev:amd64 (2.2.9-1ubuntu0.6) ...
Selecting previously unselected package libfile-fcntllock-perl.
Preparing to unpack .../24-libfile-fcntllock-perl_0.22-3build4_amd64.deb ...
Unpacking libfile-fcntllock-perl (0.22-3build4) ...
Selecting previously unselected package libpython3.8-dev:amd64.
Preparing to unpack .../25-libpython3.8-dev_3.8.10-0ubuntu1~20.04.7_amd64.deb ...
Unpacking libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04.7) ...
Selecting previously unselected package libpython3-dev:amd64.
Preparing to unpack .../26-libpython3-dev_3.8.2-0ubuntu2_amd64.deb ...
Unpacking libpython3-dev:amd64 (3.8.2-0ubuntu2) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../27-python-pip-whl_20.0.2-5ubuntu1.8_all.deb ...
Unpacking python-pip-whl (20.0.2-5ubuntu1.8) ...
Selecting previously unselected package python3.8-dev.
Preparing to unpack .../28-python3.8-dev_3.8.10-0ubuntu1~20.04.7_amd64.deb ...
Unpacking python3.8-dev (3.8.10-0ubuntu1~20.04.7) ...
Selecting previously unselected package python3-dev.
Preparing to unpack .../29-python3-dev_3.8.2-0ubuntu2_amd64.deb ...
Unpacking python3-dev (3.8.2-0ubuntu2) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../30-python3-wheel_0.34.2-1ubuntu0.1_all.deb ...
Unpacking python3-wheel (0.34.2-1ubuntu0.1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../31-python3-pip_20.0.2-5ubuntu1.8_all.deb ...
Unpacking python3-pip (20.0.2-5ubuntu1.8) ...
Setting up libexpat1:amd64 (2.2.9-1ubuntu0.6) ...
Setting up libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.7) ...
Setting up libfile-fcntllock-perl (0.22-3build4) ...
Setting up libalgorithm-diff-perl (1.19.03-2) ...
Setting up python3-wheel (0.34.2-1ubuntu0.1) ...
Setting up libfakeroot:amd64 (1.24-1) ...
Setting up fakeroot (1.24-1) ...
update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode
Setting up libexpat1-dev:amd64 (2.2.9-1ubuntu0.6) ...
Setting up git (1:2.25.1-1ubuntu3.10) ...
Setting up libdpkg-perl (1.19.7ubuntu3.2) ...
Setting up python3.8-minimal (3.8.10-0ubuntu1~20.04.7) ...
Setting up python-pip-whl (20.0.2-5ubuntu1.8) ...
Setting up libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04.7) ...
Setting up python3.8 (3.8.10-0ubuntu1~20.04.7) ...
Setting up libalgorithm-diff-xs-perl (0.04-6) ...
Setting up gcc-9-base:amd64 (9.4.0-1ubuntu1~20.04.1) ...
Setting up libalgorithm-merge-perl (0.08-3) ...
Setting up dpkg-dev (1.19.7ubuntu3.2) ...
Setting up libasan5:amd64 (9.4.0-1ubuntu1~20.04.1) ...
Setting up libpython3.8:amd64 (3.8.10-0ubuntu1~20.04.7) ...
Setting up python3-pip (20.0.2-5ubuntu1.8) ...
Setting up cpp-9 (9.4.0-1ubuntu1~20.04.1) ...
Setting up libgcc-9-dev:amd64 (9.4.0-1ubuntu1~20.04.1) ...
Setting up libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04.7) ...
Setting up python3.8-dev (3.8.10-0ubuntu1~20.04.7) ...
Setting up gcc-9 (9.4.0-1ubuntu1~20.04.1) ...
Setting up libpython3-dev:amd64 (3.8.2-0ubuntu2) ...
Setting up libstdc++-9-dev:amd64 (9.4.0-1ubuntu1~20.04.1) ...
Setting up python3-dev (3.8.2-0ubuntu2) ...
Setting up g++-9 (9.4.0-1ubuntu1~20.04.1) ...
Setting up g++ (4:9.3.0-1ubuntu2) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up build-essential (12.8ubuntu1.1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for mime-support (3.64ubuntu1) ...
#安装ansible
root@haproxy1:~# pip3 install ansible -i https://mirrors.ustc.edu.cn/pypi/web/simple
Looking in indexes: https://mirrors.ustc.edu.cn/pypi/web/simple
Collecting ansible
Downloading https://mirrors.bfsu.edu.cn/pypi/web/packages/d2/ba/99b47f2c618eef5044e2d8cbb0c12d80ba9586492f3e85fe1367fc38523b/ansible-6.7.0-py3-none-any.whl (42.8 MB)
|████████████████████████████████| 42.8 MB 56 kB/s
Collecting ansible-core~=2.13.7
Downloading https://mirrors.bfsu.edu.cn/pypi/web/packages/48/ce/88f77cf767290a6b5f9e500a3f82dbea95e95cc5a20f8b930592815e5f48/ansible_core-2.13.8-py3-none-any.whl (2.1 MB)
|████████████████████████████████| 2.1 MB 63.2 MB/s
Requirement already satisfied: cryptography in /usr/lib/python3/dist-packages (from ansible-core~=2.13.7->ansible) (2.8)
Collecting packaging
Downloading https://mirrors.bfsu.edu.cn/pypi/web/packages/ed/35/a31aed2993e398f6b09a790a181a7927eb14610ee8bbf02dc14d31677f1c/packaging-23.0-py3-none-any.whl (42 kB)
|████████████████████████████████| 42 kB 1.6 MB/s
Requirement already satisfied: PyYAML>=5.1 in /usr/lib/python3/dist-packages (from ansible-core~=2.13.7->ansible) (5.3.1)
Collecting resolvelib<0.9.0,>=0.5.3
Downloading https://mirrors.bfsu.edu.cn/pypi/web/packages/98/c0/46cfa3f56e43033b705965120058c018375600fa8fdb44c4e53d75820673/resolvelib-0.8.1-py2.py3-none-any.whl (16 kB)
Collecting jinja2>=3.0.0
Downloading https://mirrors.bfsu.edu.cn/pypi/web/packages/bc/c3/f068337a370801f372f2f8f6bad74a5c140f6fda3d9de154052708dd3c65/Jinja2-3.1.2-py3-none-any.whl (133 kB)
|████████████████████████████████| 133 kB 33.7 MB/s
Collecting MarkupSafe>=2.0
Downloading https://mirrors.bfsu.edu.cn/pypi/web/packages/9d/80/8320f182d06a9b289b1a9f266f593feb91d3781c7e104bbe09e0c4c11439/MarkupSafe-2.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Installing collected packages: packaging, resolvelib, MarkupSafe, jinja2, ansible-core, ansible
Attempting uninstall: MarkupSafe
Found existing installation: MarkupSafe 1.1.0
Not uninstalling markupsafe at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'MarkupSafe'. No files were found to uninstall.
Attempting uninstall: jinja2
Found existing installation: Jinja2 2.10.1
Not uninstalling jinja2 at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'Jinja2'. No files were found to uninstall.
Successfully installed MarkupSafe-2.1.2 ansible-6.7.0 ansible-core-2.13.8 jinja2-3.1.2 packaging-23.0 resolvelib-0.8.1
#查看ansible 版本
root@haproxy1:~# ansible --version
ansible [core 2.13.8]
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.8.10 (default, Mar 13 2023, 10:26:41) [GCC 9.4.0]
jinja version = 3.1.2
libyaml = True
#生成密钥对
root@haproxy1:~# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:QPSbEXStXf0pduD6RV4t+xoeNI41l/64+G27XV5o9Dw root@haproxy1
The key's randomart image is:
+---[RSA 3072]----+
| .o.o .. . |
| . . o ... .|
| . o o... +|
| . +. .+.+=|
| S o @+o|
| . *oX |
| o BE=|
| = *O|
| ..**B|
+----[SHA256]-----+
#安装sshpass命令用于同步公钥到各k8s服务器
root@haproxy1:~# apt install sshpass -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
sshpass
0 upgraded, 1 newly installed, 0 to remove and 238 not upgraded.
Need to get 10.5 kB of archives.
After this operation, 30.7 kB of additional disk space will be used.
Get:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/universe amd64 sshpass amd64 1.06-1 [10.5 kB]
Fetched 10.5 kB in 0s (66.5 kB/s)
Selecting previously unselected package sshpass.
(Reading database ... 78554 files and directories currently installed.)
Preparing to unpack .../sshpass_1.06-1_amd64.deb ...
Unpacking sshpass (1.06-1) ...
Setting up sshpass (1.06-1) ...
Processing triggers for man-db (2.9.1-1) ...
#创建分发公钥脚本
root@haproxy1:~# cat scp-key.sh
#!/bin/bash
#目标主机地址
IP="
172.31.7.101
172.31.7.102
172.31.7.103
172.31.7.111
172.31.7.112
172.31.7.113
"
for node in ${IP};do
sshpass -p 123456 ssh-copy-id ${node} -o StrictHostKeyChecking=no
if [ $? -eq 0 ];then
echo "${node} 秘钥copy完成"
else
echo "${node} 秘钥copy失败"
fi
done
root@haproxy1:~# sed -i "s/172.31/192.168/g" scp-key.sh
#执行脚本
root@haproxy1:~# bash scp-key.sh
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'StrictHostKeyChecking=no' '192.168.7.101'"
and check to make sure that only the key(s) you wanted were added.
192.168.7.101 秘钥copy完成
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'StrictHostKeyChecking=no' '192.168.7.102'"
and check to make sure that only the key(s) you wanted were added.
192.168.7.102 秘钥copy完成
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'StrictHostKeyChecking=no' '192.168.7.103'"
and check to make sure that only the key(s) you wanted were added.
192.168.7.103 秘钥copy完成
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'StrictHostKeyChecking=no' '192.168.7.111'"
and check to make sure that only the key(s) you wanted were added.
192.168.7.111 秘钥copy完成
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'StrictHostKeyChecking=no' '192.168.7.112'"
and check to make sure that only the key(s) you wanted were added.
192.168.7.112 秘钥copy完成
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'StrictHostKeyChecking=no' '192.168.7.113'"
and check to make sure that only the key(s) you wanted were added.
192.168.7.113 秘钥copy完成
##指定kubeasz版本
root@haproxy1:~# export release=3.2.0
##下载安装包
root@haproxy1:~# wget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown
##添加可执行权限
root@haproxy1:~# chmod a+x ezdown
root@haproxy1:~# vi ezdown
#default settings, can be overridden by cmd line options, see usage
DOCKER_VER=20.10.9
KUBEASZ_VER=3.2.0
K8S_BIN_VER=v1.23.1
EXT_BIN_VER=1.0.0
SYS_PKG_VER=0.4.2
HARBOR_VER=v2.1.3
REGISTRY_MIRROR=CN
##下载kubeasz代码、二进制、默认容器镜像
root@haproxy1:~# ./ezdown -D
2023-03-04 03:55:00 INFO Action successed: download_all
##查看下载的文件
root@haproxy1:/etc/kubeasz# ll /etc/kubeasz/down/
total 1192960
drwxr-xr-x 3 root root 4096 Mar 4 03:55 ./
drwxrwxr-x 12 root root 4096 Mar 4 03:57 ../
-rw------- 1 root root 384354816 Mar 4 03:50 calico_v3.19.3.tar
-rw------- 1 root root 46967296 Mar 4 03:51 coredns_1.8.6.tar
-rw------- 1 root root 224458240 Mar 4 03:51 dashboard_v2.4.0.tar
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 4 03:52 docker/
-rw-r--r-- 1 root root 63350495 Oct 4 2021 docker-20.10.9.tgz
-rw------- 1 root root 70554112 Mar 4 03:53 flannel_v0.15.1.tar
-rw------- 1 root root 106171392 Mar 4 03:51 k8s-dns-node-cache_1.21.1.tar
-rw------- 1 root root 179055104 Mar 4 03:55 kubeasz_3.2.0.tar
-rw------- 1 root root 34463744 Mar 4 03:53 metrics-scraper_v1.0.7.tar
-rw------- 1 root root 65683968 Mar 4 03:53 metrics-server_v0.5.2.tar
-rw------- 1 root root 45085184 Mar 4 03:54 nfs-provisioner_v4.0.2.tar
-rw------- 1 root root 692736 Mar 4 03:54 pause_3.6.tar
-rw------- 1 root root 692736 Mar 4 03:54 pause.tar
root@haproxy1:~# cd /etc/kubeasz/
root@haproxy1:/etc/kubeasz# ./ezctl new k8s-01
vim clusters/k8s-01/hosts
root@haproxy1:/etc/kubeasz# cat clusters/k8s-01/
config.yml hosts
root@haproxy1:/etc/kubeasz# cat clusters/k8s-01/hosts
#'etcd' cluster should have odd member(s) (1,3,5,...)
[etcd]
192.168.7.101
192.168.7.102
192.168.7.103
#master node(s)
[kube_master]
192.168.7.101
192.168.7.102
#work node(s)
[kube_node]
192.168.7.111
192.168.7.112
#[optional] harbor server, a private docker registry
#'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one
[harbor]
#192.168.1.8 NEW_INSTALL=false
#[optional] loadbalance for accessing k8s from outside
[ex_lb]
#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#[optional] ntp server for the cluster
[chrony]
#192.168.1.1
[all:vars]
#--------- Main Variables ---------------
#Secure port for apiservers
SECURE_PORT="6443"
#Cluster container-runtime supported: docker, containerd
CONTAINER_RUNTIME="docker"
#Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="calico"
#Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"
#K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.100.0.0/16"
#Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="10.200.0.0/16"
#NodePort Range
NODE_PORT_RANGE="30000-60000"
#Cluster DNS Domain
CLUSTER_DNS_DOMAIN="magedu.local"
#-------- Additional Variables (don't change the default value right now) ---
#Binaries Directory
bin_dir="/usr/local/bin"
#Deploy Directory (kubeasz workspace)
base_dir="/etc/kubeasz"
#Directory for a specific cluster
cluster_dir="{{ base_dir }}/clusters/k8s-01"
#CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"
#编辑config.yml 文件
cat clusters/k8s-01/config.yml
root@haproxy1:/etc/kubeasz# cat clusters/k8s-01/hosts
#'etcd' cluster should have odd member(s) (1,3,5,...)
[etcd]
192.168.7.101
192.168.7.102
192.168.7.103
#master node(s)
[kube_master]
192.168.7.101
192.168.7.102
#work node(s)
[kube_node]
192.168.7.111
192.168.7.112
#[optional] harbor server, a private docker registry
#'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one
[harbor]
#192.168.1.8 NEW_INSTALL=false
#[optional] loadbalance for accessing k8s from outside
[ex_lb]
#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#[optional] ntp server for the cluster
[chrony]
#192.168.1.1
[all:vars]
#--------- Main Variables ---------------
#Secure port for apiservers
SECURE_PORT="6443"
#Cluster container-runtime supported: docker, containerd
CONTAINER_RUNTIME="docker"
#Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="calico"
#Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"
#K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.100.0.0/16"
#Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="10.200.0.0/16"
#NodePort Range
NODE_PORT_RANGE="30000-60000"
#Cluster DNS Domain
CLUSTER_DNS_DOMAIN="magedu.local"
#-------- Additional Variables (don't change the default value right now) ---
#Binaries Directory
bin_dir="/usr/local/bin"
#Deploy Directory (kubeasz workspace)
base_dir="/etc/kubeasz"
#Directory for a specific cluster
cluster_dir="{{ base_dir }}/clusters/k8s-01"
#CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"
root@haproxy1:/etc/kubeasz# cat clusters/k8s-01/
config.yml hosts
root@haproxy1:/etc/kubeasz# cat clusters/k8s-01/config.yml
############################
#prepare
############################
#可选离线安装系统软件包 (offline|online)
INSTALL_SOURCE: "online"
#可选进行系统安全加固 github.com/dev-sec/ansible-collection-hardening
OS_HARDEN: false
#设置时间源服务器【重要：集群内机器时间必须同步】
ntp_servers:
- "ntp1.aliyun.com"
- "time1.cloud.tencent.com"
- "0.cn.pool.ntp.org"
#设置允许内部时间同步的网络段，比如"10.0.0.0/8"，默认全部允许
local_network: "0.0.0.0/0"
############################
#role:deploy
############################
#default: ca will expire in 100 years
#default: certs issued by the ca will expire in 50 years
CA_EXPIRY: "876000h"
CERT_EXPIRY: "438000h"
#kubeconfig 配置参数
CLUSTER_NAME: "cluster1"
CONTEXT_NAME: "context-{{ CLUSTER_NAME }}"
#k8s version
K8S_VER: "1.23.1"
############################
#role:etcd
############################
#设置不同的wal目录，可以避免磁盘io竞争，提高性能
ETCD_DATA_DIR: "/var/lib/etcd"
ETCD_WAL_DIR: ""
############################
#role:runtime [containerd,docker]
############################
#------------------------------------------- containerd
#[.]启用容器仓库镜像
ENABLE_MIRROR_REGISTRY: true
#[containerd]基础容器镜像
SANDBOX_IMAGE: "easzlab/pause:3.6"
#[containerd]容器持久化存储目录
CONTAINERD_STORAGE_DIR: "/var/lib/containerd"
#------------------------------------------- docker
#[docker]容器存储目录
DOCKER_STORAGE_DIR: "/var/lib/docker"
#[docker]开启Restful API
ENABLE_REMOTE_API: false
#[docker]信任的HTTP仓库
INSECURE_REG: '["127.0.0.1/8"]'
############################
#role:kube-master
############################
#k8s 集群 master 节点证书配置，可以添加多个ip和域名（比如增加公网ip和域名）
MASTER_CERT_HOSTS:
- "10.1.1.1"
- "k8s.test.io"
#- "www.test.com"
#node 节点上 pod 网段掩码长度（决定每个节点最多能分配的pod ip地址）
#如果flannel 使用 --kube-subnet-mgr 参数，那么它将读取该设置为每个节点分配pod网段
#https://github.com/coreos/flannel/issues/847
NODE_CIDR_LEN: 24
############################
#role:kube-node
############################
#Kubelet 根目录
KUBELET_ROOT_DIR: "/var/lib/kubelet"
#node节点最大pod 数
MAX_PODS: 110
#配置为kube组件（kubelet,kube-proxy,dockerd等）预留的资源量
#数值设置详见templates/kubelet-config.yaml.j2
KUBE_RESERVED_ENABLED: "no"
#k8s 官方不建议草率开启 system-reserved, 除非你基于长期监控，了解系统的资源占用状况；
#并且随着系统运行时间，需要适当增加资源预留，数值设置详见templates/kubelet-config.yaml.j2
#系统预留设置基于 4c/8g 虚机，最小化安装系统服务，如果使用高性能物理机可以适当增加预留
#另外，集群安装时候apiserver等资源占用会短时较大，建议至少预留1g内存
SYS_RESERVED_ENABLED: "no"
#haproxy balance mode
BALANCE_ALG: "roundrobin"
############################
#role:network [flannel,calico,cilium,kube-ovn,kube-router]
############################
#------------------------------------------- flannel
#[flannel]设置flannel 后端"host-gw","vxlan"等
FLANNEL_BACKEND: "vxlan"
DIRECT_ROUTING: false
#[flannel] flanneld_image: "quay.io/coreos/flannel:v0.10.0-amd64"
flannelVer: "v0.15.1"
flanneld_image: "easzlab/flannel:{{ flannelVer }}"
#[flannel]离线镜像tar包
flannel_offline: "flannel_{{ flannelVer }}.tar"
#------------------------------------------- calico
#[calico]设置 CALICO_IPV4POOL_IPIP=“off”,可以提高网络性能，条件限制详见 docs/setup/calico.md
CALICO_IPV4POOL_IPIP: "Always"
#[calico]设置 calico-node使用的host IP，bgp邻居通过该地址建立，可手工指定也可以自动发现
IP_AUTODETECTION_METHOD: "can-reach={{ groups['kube_master'][0] }}"
#[calico]设置calico 网络 backend: brid, vxlan, none
CALICO_NETWORKING_BACKEND: "brid"
#[calico]更新支持calico 版本: [v3.3.x] [v3.4.x] [v3.8.x] [v3.15.x]
calico_ver: "v3.19.3"
#[calico]calico 主版本
calico_ver_main: "{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}"
#[calico]离线镜像tar包
calico_offline: "calico_{{ calico_ver }}.tar"
#------------------------------------------- cilium
#[cilium]CILIUM_ETCD_OPERATOR 创建的 etcd 集群节点数 1,3,5,7...
ETCD_CLUSTER_SIZE: 1
#[cilium]镜像版本
cilium_ver: "v1.4.1"
#[cilium]离线镜像tar包
cilium_offline: "cilium_{{ cilium_ver }}.tar"
#------------------------------------------- kube-ovn
#[kube-ovn]选择 OVN DB and OVN Control Plane 节点，默认为第一个master节点
OVN_DB_NODE: "{{ groups['kube_master'][0] }}"
#[kube-ovn]离线镜像tar包
kube_ovn_ver: "v1.5.3"
kube_ovn_offline: "kube_ovn_{{ kube_ovn_ver }}.tar"
#------------------------------------------- kube-router
#[kube-router]公有云上存在限制，一般需要始终开启 ipinip；自有环境可以设置为 "subnet"
OVERLAY_TYPE: "full"
#[kube-router]NetworkPolicy 支持开关
FIREWALL_ENABLE: "true"
#[kube-router]kube-router 镜像版本
kube_router_ver: "v0.3.1"
busybox_ver: "1.28.4"
#[kube-router]kube-router 离线镜像tar包
kuberouter_offline: "kube-router_{{ kube_router_ver }}.tar"
busybox_offline: "busybox_{{ busybox_ver }}.tar"
############################
#role:cluster-addon
############################
#coredns 自动安装
dns_install: "yes"
corednsVer: "1.8.6"
ENABLE_LOCAL_DNS_CACHE: true
dnsNodeCacheVer: "1.21.1"
#设置 local dns cache 地址
LOCAL_DNS_CACHE: "169.254.20.10"
#metric server 自动安装
metricsserver_install: "yes"
metricsVer: "v0.5.2"
#dashboard 自动安装
dashboard_install: "yes"
dashboardVer: "v2.4.0"
dashboardMetricsScraperVer: "v1.0.7"
#ingress 自动安装
ingress_install: "no"
ingress_backend: "traefik"
traefik_chart_ver: "10.3.0"
#prometheus 自动安装
prom_install: "no"
prom_namespace: "monitor"
prom_chart_ver: "12.10.6"
#nfs-provisioner 自动安装
nfs_provisioner_install: "no"
nfs_provisioner_namespace: "kube-system"
nfs_provisioner_ver: "v4.0.2"
nfs_storage_class: "managed-nfs-storage"
nfs_server: "192.168.1.10"
nfs_path: "/data/nfs"
############################
#role:harbor
############################
#harbor version，完整版本号
HARBOR_VER: "v2.1.3"
HARBOR_DOMAIN: "harbor.yourdomain.com"
HARBOR_TLS_PORT: 8443
#if set 'false', you need to put certs named harbor.pem and harbor-key.pem in directory 'down'
HARBOR_SELF_SIGNED_CERT: true
#install extra component
HARBOR_WITH_NOTARY: false
HARBOR_WITH_TRIVY: false
HARBOR_WITH_CLAIR: false
HARBOR_WITH_CHARTMUSEUM: true
root@haproxy1:/etc/kubeasz# ./ezctl help setup
Usage: ezctl setup <cluster> <step>
available steps:
01 prepare to prepare CA/certs & kubeconfig & other system settings
02 etcd to setup the etcd cluster
03 container-runtime to setup the container runtime(docker or containerd)
04 kube-master to setup the master nodes
05 kube-node to setup the worker nodes
06 network to setup the network plugin
07 cluster-addon to setup other useful plugins
90 all to run 01~07 all at once
10 ex-lb to install external loadbalance for accessing k8s from outside
11 harbor to install a new harbor server or to integrate with an existed one
examples: ./ezctl setup test-k8s 01 (or ./ezctl setup test-k8s prepare)
./ezctl setup test-k8s 02 (or ./ezctl setup test-k8s etcd)
./ezctl setup test-k8s all
./ezctl setup test-k8s 04 -t restart_master
##系统设置初始化
root@haproxy1:/etc/kubeasz# ./ezctl setup k8s-01 01
root@k8s-master1:~# export NODE_IPS="192.168.7.101 192.168.7.102 192.168.7.103"
root@k8s-master1:~# for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem endpoint health; done
https://192.168.7.101:2379 is healthy: successfully committed proposal: took = 17.322006ms
https://192.168.7.102:2379 is healthy: successfully committed proposal: took = 21.598359ms
https://192.168.7.103:2379 is healthy: successfully committed proposal: took = 20.850246ms
##将证书从harbor节点scp到部署节点，稍后从部署节点分发证书
root@haproxy1:/etc/kubeasz# mkdir -pv /etc/docker/certs.d/harbor.magedu.net/
mkdir: created directory '/etc/docker/certs.d'
mkdir: created directory '/etc/docker/certs.d/harbor.magedu.net/'
root@harbor1:~#cd /apps/harbor/certs
root@k8s-harbor1:/apps/harbor# scp /apps/harbor/certs/magedu.net.crt 192.168.7.109:/etc/docker/certs.d/harbor.magedu.net
root@192.168.7.109's password:
magedu.net.crt
#配置harbor客户端证书同步
root@haproxy1:/etc/kubeasz# vim roles/docker/tasks/main.yml
30 - block:
31 - name: 准备docker相关目录
32 file: name={{ item }} state=directory
33 with_items:
34 - "{{ bin_dir }}"
35 - "/etc/docker"
36 - "/etc/bash_completion.d"
37 - "/etc/docker/certs.d/harbor.linuxarchitect.io"
38 - name: docker证书分发
39 copy: src=magedu.net.crt dest=/etc/docker/certs.d/harbor.magedu.net/magedu.net.crt mode=0644
40
41 - name: 添加域名解析
42 shell: "echo '192.168.7.104 harbor.magedu.net' >>/etc/hosts"
root@haproxy1:/etc/kubeasz# ./ezctl setup k8s-01 03
##登录docker harbor
root@haproxy1:/etc/kubeasz# docker login harbor.magedu.net
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
##下载pause镜像，并传到本地搭建的harbor
root@haproxy1:/etc/kubeasz# docker tag registry.cn-hangzhou.aliyuncs.com/zhangshijie/pause:3.9 harbor.magedu.net/baseimages/pause:3.9
root@haproxy1:/etc/kubeasz#
root@haproxy1:/etc/kubeasz# docker push harbor.magedu.net/baseimages/pause:3.9
The push refers to repository [harbor.magedu.net/baseimages/pause]
e3e5579ddd43: Pushed
3.9: digest: sha256:3ec9d4ec5512356b5e77b13fddac2e9016e7aba17dd295ae23c94b2b901813de size: 527
##修改containerd pause镜像地址
root@haproxy1:/etc/kubeasz# vim clusters/k8s-01/config.yml
#[containerd]基础容器镜像
SANDBOX_IMAGE: "harbor.magedu.net/baseimages/pause:3.9"
##部署master节点
root@haproxy1:/etc/kubeasz# ./ezctl setup k8s-01 04
##验证master节点安装
root@haproxy1:/etc/kubeasz# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.7.101 Ready,SchedulingDisabled master 4m42s v1.23.1
192.168.7.102 Ready,SchedulingDisabled master 4m42s v1.23.1
##部署node节点
root@haproxy1:/etc/kubeasz# ./ezctl setup k8s-01 05
##验证node节点安装
root@haproxy1:/etc/kubeasz# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.7.101 Ready,SchedulingDisabled master 10m v1.23.1
192.168.7.102 Ready,SchedulingDisabled master 10m v1.23.1
192.168.7.111 Ready node 65s v1.23.1
192.168.7.112 Ready node 66s v1.23.1
##部署网络服务
root@haproxy1:/etc/kubeasz# ./ezctl setup k8s-01 06
##验证calico
root@k8s-master1:/etc# calicoctl node status
Calico process is running.
IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+---------------+-------------------+-------+----------+-------------+
| 192.168.7.102 | node-to-node mesh | up | 09:04:17 | Established |
| 192.168.7.111 | node-to-node mesh | up | 09:04:36 | Established |
| 192.168.7.112 | node-to-node mesh | up | 09:04:54 | Established |
+---------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
##添加一个node节点
root@haproxy1:/etc/kubeasz# ./ezctl add-node k8s-01 192.168.7.113
##查看节点
root@k8s-master1:/etc# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.7.101 Ready,SchedulingDisabled master 6h31m v1.23.1
192.168.7.102 Ready,SchedulingDisabled master 6h31m v1.23.1
192.168.7.111 Ready node 6h22m v1.23.1
192.168.7.112 Ready node 6h22m v1.23.1
192.168.7.113 Ready node 5m2s v1.23.1
##部署coredns
root@k8s-master1:~/coredns-v1.8.6# ls
coredns-1.8.6.yaml coredns-v1.8.6.yaml
root@k8s-master1:~/coredns-v1.8.6# kubectl apply -f coredns-1.8.6.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
##测试域名解析
root@k8s-master1:~/coredns-v1.8.6# kubectl run --rm -it busybox --image=busybox sh
/ # nslookup kubernetes.default.svc.yanggc.local
Server: 10.100.0.2
Address: 10.100.0.2:53
Name: kubernetes.default.svc.yanggc.local
Address: 10.100.0.1
##上传镜像到本地harbor
root@haproxy1:~# docker pull kubernetesui/dashboard:v2.5.1
v2.5.1: Pulling from kubernetesui/dashboard
d1d01ae59b08: Pull complete
a25bff2a339f: Pull complete
Digest: sha256:cc746e7a0b1eec0db01cbabbb6386b23d7af97e79fa9e36bb883a95b7eb96fe2
Status: Downloaded newer image for kubernetesui/dashboard:v2.5.1
docker.io/kubernetesui/dashboard:v2.5.1
root@haproxy1:~# docker tag kubernetesui/dashboard:v2.5.1 harbor.magedu.net/baseimages/dashboard:v2.5.1
root@haproxy1:~# docker push harbor.magedu.net/baseimages/dashboard:v2.5.1
The push refers to repository [harbor.magedu.net/baseimages/dashboard]
e98b3744f758: Pushed
dab46c9f5775: Pushed
v2.5.1: digest: sha256:0c82e96241aa683fe2f8fbdf43530e22863ac8bfaddb0d7d30b4e3a639d4e8c5 size: 736
root@haproxy1:~# docker pull kubernetesui/metrics-scraper:v1.0.7
v1.0.7: Pulling from kubernetesui/metrics-scraper
Digest: sha256:36d5b3f60e1a144cc5ada820910535074bdf5cf73fb70d1ff1681537eef4e172
Status: Image is up to date for kubernetesui/metrics-scraper:v1.0.7
docker.io/kubernetesui/metrics-scraper:v1.0.7
root@haproxy1:~# docker tag kubernetesui/metrics-scraper:v1.0.7 harbor.magedu.net/baseimages/metrics-scraper:v1.0.7
root@haproxy1:~# docker push harbor.magedu.net/baseimages/metrics-scraper:v1.0.7
The push refers to repository [harbor.magedu.net/baseimages/metrics-scraper]
7813555162f0: Pushed
10d0ebb29e3a: Pushed
v1.0.7: digest: sha256:76eb73afa0198ac457c760887ed7ebfa2f58adc09205bd9667b3f76652077a71 size: 736
##部署dashboard
root@k8s-master1:~/dashboard-v2.5.1# ls
admin-user.yaml dashboard-v2.5.1.yaml
root@k8s-master1:~/dashboard-v2.5.1# sed -e 's/harbor.linuxarchitect.io/harbor.magedu.net/g' -i dashboard-v2.5.1.yaml
root@k8s-master1:~/dashboard-v2.5.1# kubectl apply -f dashboard-v2.5.1.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
root@k8s-master1:~/dashboard-v2.5.1# kubectl get pod -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
dashboard-metrics-scraper-6cd948965c-8j5qw 1/1 Running 0 14s
kubernetes-dashboard-6bbb9fb9dc-c4pqp 1/1 Running 0 14s
root@k8s-master1:~/dashboard-v2.5.1# kubectl apply -f admin-user.yaml
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
root@k8s-master1:~/dashboard-v2.5.1# kubectl get secret -A | grep admin
kubernetes-dashboard admin-user-token-rwldb kubernetes.io/service-account-token 3 12s
root@k8s-master1:~/dashboard-v2.5.1# kubectl -n kubernetes-dashboard describe secret admin-user-token-rwldb
Name: admin-user-token-rwldb
Namespace: kubernetes-dashboard
Labels: <none>
Annotations: kubernetes.io/service-account.name: admin-user
kubernetes.io/service-account.uid: 97cc63db-6b38-458e-a542-14fe5c122b89
Type: kubernetes.io/service-account-token
Data
====
token: eyJhbGciOiJSUzI1NiIsImtpZCI6Il85emVudF9KTGRQTmVSeFFYMEZ2blNkNlMwXzAwOElpWDZFa3FTX3Z0RDAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXJ3bGRiIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI5N2NjNjNkYi02YjM4LTQ1OGUtYTU0Mi0xNGZlNWMxMjJiODkiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.EY4d7rf5gM9i1C-wh8N0RcOl8gGugu_xXczJK_m2ucs034gC1Auzc3RvMHVgfSb4ck0Af5-bfOI19UJr1nvcKcFsJ5ByQZ4RGoddfgxh2S61Hq7VwnXQZycx8bv6hlnv8yWHUwd11NgV0EuvBv5qVGmaUQdt_9cRsNtX51klMhaRMTl2aOkuNgZLhgskT9Tyq5DQh3Mg45aPdrE2GlK7LtYtpJVIkbyymKCXiARkxFXHfCtIHWHdmjvl2Va0MHxz-4z4j8YRlw5F93ryquiUlRDU_cGDSOnRtaO7WDYUBi7SsFSuESNLhK5RnAiezd-yJdsC5EcZ8A1tVG_mEY5c8Q
ca.crt: 1302 bytes
namespace: 20 bytes
root@k8s-master1:~/dashboard-v2.5.1# kubectl get svc -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.100.65.22 <none> 8000/TCP 2m42s
kubernetes-dashboard NodePort 10.100.93.115 <none> 443:30002/TCP 2m42s
root@k8s-master1:~/dashboard-v2.5.1# kubectl get pod -n kubernetes-dashboard -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dashboard-metrics-scraper-6cd948965c-8j5qw 1/1 Running 0 5m12s 10.200.218.66 192.168.7.113 <none> <none>
kubernetes-dashboard-6bbb9fb9dc-c4pqp 1/1 Running 0 5m12s 10.200.151.194 192.168.7.112 <none> <none>
root@k8s-master1:~/dashboard-v2.5.1# kubectl get svc -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.100.65.22 <none> 8000/TCP 6m18s
kubernetes-dashboard NodePort 10.100.93.115 <none> 443:30002/TCP 6m18s
#测试访问dashboard
