程序员经验分享-hwhale

k8s集群部署

2023-11-12

文章目录

1. 二进制部署三节点（复用）高可用 k8s 集群

1.1 环境规划阶段

1.1.1 实验架构图

image-20230506143854348

1.1.2 系统版本说明

OS 版本：CentOS Linux release 7.9.2009 (Core)

初始内核版本：3.10.0-1160.71.1.el7.x86_64

配置信息：2C2G 150G硬盘

文件系统：xfs

网络：外网权限

k8s 版本：1.25.9

1.1.3 环境基本信息

K8s集群角色 IP地址 主机名 组件信息
控制节点1（工作节点1） 192.168.204.10 k8s-001 apiserver、controller-manager、schedule、etcd、kube-proxy、容器运行时、keepalived、nginx、calico、coredns、kubelet
控制节点2（工作节点2） 192.168.204.11 k8s-002 apiserver、controller-manager、schedule、etcd、kube-proxy、容器运行时、keepalived、nginx、calico、coredns、kubelet
控制节点3（工作节点3） 192.168.204.12 k8s-003 apiserver、controller-manager、schedule、etcd、kube-proxy、容器运行时、calico、coredns、kubelet
VIP地址 192.168.204.13（k8s-vip）

1.1.4 k8s 网段划分

  • service 网段：10.165.0.0/16

1.2 基础安装及优化阶段

无特别说明，三台都要执行

1.2.1 系统信息检查

检查系统版本以及内核

cat /etc/redhat-release ; uname -r

image-20230505191050142

image-20230505191100158

image-20230505191106987

1.2.2 静态 IP 地址配置

服务器必须配置静态IP地址，不可变动

grep -E 'BOOTPROTO|IPADDR' /etc/sysconfig/network-scripts/ifcfg-ens32

image-20230505190923436

image-20230505190930684

image-20230505190937200

1.2.3 配置主机名

按照规划配置对应主机的主机名即可

hostnamectl set-hostname k8s-001
hostnamectl set-hostname k8s-002
hostnamectl set-hostname k8s-003

image-20230505191242757

image-20230505191249488

image-20230505191509440

1.2.4 配置/etc/hosts文件

cat >> /etc/hosts <<EOF
192.168.204.10 k8s-001
192.168.204.11 k8s-002
192.168.204.12 k8s-003
192.168.204.13 k8s-vip
EOF

image-20230505191456163

image-20230505191503030

image-20230505191509440

1.2.5 关闭 selinux

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

image-20230505191639625

image-20230505191646133

image-20230505191652640

1.2.6 配置主机互信

### 只需要配置k8s-001到三节点互信即可
ssh-keygen -t rsa -f ~/.ssh/id_rsa -N ''
ssh-copy-id k8s-001
ssh-copy-id k8s-002
ssh-copy-id k8s-003

image-20230505191831571

1.2.7 关闭交换分区

必须关闭

swapoff -a
sed --in-place=.bak 's/.*swap.*/#&/g' /etc/fstab

在这里插入图片描述

1.2.8 关闭 firewalld

systemctl stop firewalld ; systemctl disable firewalld

image-20230505192026594

1.2.9 关闭 NetworkManager

systemctl stop NetworkManager; systemctl disable NetworkManager

image-20230505192142782

1.2.10 设置资源限制

/etc/security/limits.conf 初始文件没有任何有效的参数内容

cat >> /etc/security/limits.conf << EOF
* soft nofile 65536
* hard nofile 131072
* soft nproc 65535
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

image-20230505192312175

1.2.11 配置时间同步

### 配置chrony.conf
cat > /etc/chrony.conf << EOF
server ntp.aliyun.com iburst
stratumweight 0
driftfile /var/lib/chrony/drift
rtcsync
makestep 10 3
bindcmdaddress 127.0.0.1
bindcmdaddress ::1
keyfile /etc/chrony.keys
commandkey 1
generatecommandkey
logchange 0.5
logdir /var/log/chrony
EOF

### 重启服务

image-20230505192652090

1.2.12 配置国内源

## centos 7 的yum和epel源
mkdir /etc/yum.repos.d.bak && mv /etc/yum.repos.d/*.repo /etc/yum.repos.d.bak

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo

sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

curl -o /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo

yum clean all && yum makecache

## 配置docker源
yum install -y yum-utils device-mapper-persistent-data lvm2

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo

yum makecache fast

## 配置k8s源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

1.2.13 升级内核

### 先升级一下软件包
yum update --exclude=kernel* -y

### 下载内核（4.19以上推荐，默认其实也可以）
curl -o kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm

curl -o kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm

### 安装内核（当前目录只有这2个rpm包）
yum localinstall -y *.rpm

### 更改内核启动顺序
grub2-set-default 0
grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

### 检查是否加载最新
grubby --default-kernel

### 重启服务器
reboot

image-20230505194246261

1.2.14 安装基础工具

就安装一些用得到的一些工具

yum install -y device-mapper-persistent-data net-tools nfs-utils jq psmisc git lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack telnet ipset sysstat libseccomp

image-20230505194721685

1.2.15 配置内核模块和参数

### 配置需要加载模块
cat > /etc/modules-load.d/k8s.conf << EOF
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
overlay
br_netfilter
EOF

## 开机自动加载
systemctl enable systemd-modules-load.service --now

## 配置内核参数优化
cat > /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
net.ipv4.conf.all.route_localnet = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF

## 生效加载
sysctl --system

### 重启服务，检查模块加载是否正常
reboot
lsmod | grep --color=auto -e ip_vs -e nf_conntrack

1.2.16 安装容器运行时

docker 引擎和 containerd 都安装上

安装 containerd

### 安装
yum install -y containerd.io-1.6.6

### 生成配置文件
containerd config default > /etc/containerd/config.toml

### 修改配置文件
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml

sed -i 's#sandbox_image = "k8s.gcr.io/pause:3.6"#sandbox_image="registry.aliyuncs.com/google_containers/pause:3.7"#g' /etc/containerd/config.toml

### 配置镜像加速
sed -i 's#config_path = ""#config_path = "/etc/containerd/certs.d"#g' /etc/containerd/config.toml

mkdir -p /etc/containerd/certs.d/docker.io

cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://registry-1.docker.io"
[host."https://xpd691zc.mirror.aliyuncs.com"]
  capabilities = ["pull", "resolve", "push"]
EOF

## 启动生效
systemctl daemon-reload ; systemctl enable containerd --now

**安装 crictl **

### 下载二进制包
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.25.0/crictl-v1.25.0-linux-amd64.tar.gz

### 解压
tar -xf crictl-v1.25.0-linux-amd64.tar.gz

### 移动位置
mv crictl /usr/local/bin/

### 配置
cat > /etc/crictl.yaml << EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

### 重启生效
systemctl restart containerd

安装 docker

yum install docker-ce -y
# 配置镜像加速
mkdir -p /etc/docker

cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://xpd691zc.mirror.aliyuncs.com"]
}
EOF

# 启动服务
systemctl enable docker --now

1.3 集群创建优化阶段

1.3.1 高可用组件安装 keepalived、nginx

安装组件

yum install nginx keepalived nginx-mod-stream -y

cat /dev/null > /etc/nginx/nginx.conf
cat /dev/null > /etc/keepalived/keepalived.conf

配置相关文件

### 三个节点都一样的 /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream kube-apiserver {
            server 192.168.204.10:6443 weight=5 max_fails=3 fail_timeout=30s;  
            server 192.168.204.11:6443 weight=5 max_fails=3 fail_timeout=30s;
            server 192.168.204.12:6443 weight=5 max_fails=3 fail_timeout=30s;  

    }
    server {
       listen 16443;
       proxy_pass kube-apiserver;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    server {
        listen       80 default_server;
        server_name  _;

        location / {
        }
    }
}


### /etc/keepalived/keepalived.conf 文件
global_defs {  
   notification_email {  
     acassen@firewall.loc  
     failover@firewall.loc  
     sysadmin@firewall.loc  
   }  
   notification_email_from Alexandre.Cassen@firewall.loc   
   smtp_server 127.0.0.1  
   smtp_connect_timeout 30  
   router_id NGINX_MASTER 
}  
 

vrrp_script check_nginx { 
    script "/etc/keepalived/check_nginx.sh" 
} 
vrrp_instance VI_1 {  
    state MASTER  
    interface ens32 ##注意网卡
    virtual_router_id 51  
    priority 100 # 主是100，其他2个节点是90、80
    advert_int 1
    authentication {  
        auth_type PASS       
        auth_pass 1111  
    }   
    virtual_ipaddress {  
        192.168.204.13/24 
    }  
    track_script { 
        check_nginx 
    }  
}

### /etc/keepalived/check_nginx.sh 文件内容

#!/bin/bash
#
counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )
if [ $counter -eq 0 ]; then
    service nginx start
    sleep 2
    counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )
    if [ $counter -eq 0 ]; then
        service  keepalived stop
    fi
fi
## 给执行权限
chmod +x /etc/keepalived/check_nginx.sh

启动验证

nginx -t
systemctl daemon-reload
systemctl enable nginx keepalived --now
ip -4 a

## 尝试停一下主节点的nginx，看是否漂移恢复
systemctl stop nginx

## 验证结果：vip 正常飘逸，恢复后会回到第一台

image-20230505203411277

1.3.2 cfssl 工具安装

## k8s-001 执行安装即可
# 下载地址：https://github.com/cloudflare/cfssl/releases
# 下载三个软件：cfssl-certinfo_linux-amd64 、cfssljson_linux-amd64 、cfssl_linux-amd64
# 下载最新版本1.6.4后操作
cd /root/cfssl
chmod +x cfssl*
mv cfssl_1.6.4_linux_amd64 /usr/local/bin/cfssl
mv cfssljson_1.6.4_linux_amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_1.6.4_linux_amd64 /usr/local/bin/cfssl-certinfo

image-20230506162452387

1.3.3 配置 CA 证书中心

我就在 k8s-001 上操作生成证书文件

先创建一个存放各个证书的文件夹

mkdir -p /root/cfssl/pki
cd /root/cfssl/pki/

配置 CA 证书

  1. 生成 CA 证书请求文件

vim /root/cfssl/pki/ca-csr.json 

{
  "CN": "kubernetes",
  "key": {
      "algo": "rsa",
      "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Sichuan",
      "L": "Chengdu",
      "O": "k8s",
      "OU": "system"
    }
  ],
  "ca": {
          "expiry": "87600h"
  }
}
  1. 生成证书

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

image-20230507100935968

  1. 生成 CA 证书的配置文件

vim /root/cfssl/pki/ca-config.json

{
  "signing": {
      "default": {
          "expiry": "87600h"
        },
      "profiles": {
          "kubernetes": {
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          }
      }
  }
}

1.3.4 生成 etcd 证书

  1. 配置证书请求文件

vim /root/cfssl/pki/etcd-csr.json

{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.204.10",
    "192.168.204.11",
    "192.168.204.12",
    "192.168.204.13"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "Sichuan",
    "L": "Chengdu",
    "O": "k8s",
    "OU": "system"
  }]
}
  1. 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

image-20230507101109274

1.3.5 生成 apiserver 证书

  1. 创建 token.csv 文件
cd /root/cfssl/pki/

cat > token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
  1. 创建证书请求文件

vim /root/cfssl/pki/kube-apiserver-csr.json

{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.204.10",
    "192.168.204.11",
    "192.168.204.12",
    "192.168.204.13",
    "10.165.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Sichuan",
      "L": "Chengdu",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
  1. 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver

image-20230507101215055

1.3.6 生成 kubectl 证书

  1. 创建证书请求文件

vim /root/cfssl/pki/admin-csr.json

{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Sichuan",
      "L": "Chengdu",
      "O": "system:masters",             
      "OU": "system"
    }
  ]
}
  1. 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

image-20230507101314421

1.3.7 生成 controller-manager 证书

  1. 创建证书请求文件

vim /root/cfssl/pki/kube-controller-manager-csr.json

{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.204.10",
      "192.168.204.11",
      "192.168.204.12",
      "192.168.204.13"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "Sichuan",
        "L": "Chengdu",
        "O": "system:kube-controller-manager",
        "OU": "system"
      }
    ]
}
  1. 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

image-20230507102240310

1.3.8 生成 scheduler 证书

  1. 创建证书请求文件

vim /root/cfssl/pki/kube-scheduler-csr.json

{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "192.168.204.10",
      "192.168.204.11",
      "192.168.204.12",
      "192.168.204.13"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "Sichuan",
        "L": "Chengdu",
        "O": "system:kube-scheduler",
        "OU": "system"
      }
    ]
}
  1. 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

image-20230507102324474

1.3.9 生成 kube-proxy 证书

  1. 创建证书请求文件

vim /root/cfssl/pki/kube-proxy-csr.json

{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Sichuan",
      "L": "Chengdu",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
  1. 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

image-20230507102418857

1.3.10 安装 etcd 高可用集群

k8s-001操作即可

准备 etcd 二进制安装包

# etcd 二进制下载地址：1.25.9版本对应的etcd版本：v3.5.5
https://github.com/etcd-io/etcd/releases
#得到的包：etcd-v3.5.5-linux-amd64.tar.gz

tar -xf etcd-v3.5.5-linux-amd64.tar.gz
cp -ar etcd-v3.5.5-linux-amd64/etcd* /usr/local/bin/
chmod +x /usr/local/bin/etcd*

## 将二进制包发送给其他节点
chmod +x etcd-v3.5.5-linux-amd64/etcd*
scp -r etcd-v3.5.5-linux-amd64/etcd* k8s-002:/usr/local/bin/
scp -r etcd-v3.5.5-linux-amd64/etcd* k8s-003:/usr/local/bin/

image-20230506164923009

创建 etcd 配置文件

## k8s-001节点执行，生成配置文件
mkdir -p /etc/etcd/

## /etc/etcd/etcd.conf文件内容如下
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.204.10:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.204.10:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.204.10:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.204.10:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.204.10:2380,etcd2=https://192.168.204.11:2380,etcd3=https://192.168.204.12:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"



## k8s-002节点执行，生成配置文件
mkdir -p /etc/etcd/

## /etc/etcd/etcd.conf文件内容如下
#[Member]
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.204.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.204.11:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.204.11:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.204.11:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.204.10:2380,etcd2=https://192.168.204.11:2380,etcd3=https://192.168.204.12:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"


## k8s-003节点执行，生成配置文件
mkdir -p /etc/etcd/

## /etc/etcd/etcd.conf文件内容如下
#[Member]
ETCD_NAME="etcd3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.204.12:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.204.12:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.204.12:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.204.12:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.204.10:2380,etcd2=https://192.168.204.11:2380,etcd3=https://192.168.204.12:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_NAME：节点名称，集群中唯一

ETCD_DATA_DIR：数据目录

ETCD_LISTEN_PEER_URLS：集群通信监听地址

ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址

ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址

ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址

ETCD_INITIAL_CLUSTER：集群节点地址

ETCD_INITIAL_CLUSTER_TOKEN：集群 Token

ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new 是新集群，existing 表示加入已有集群

创建 etcd 启动服务文件

###三台都执行

## vim /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

拷贝证书文件

## 我的证书放在k8s-001的/root/cfssl/pki下的
## 每台执行
mkdir -p /etc/etcd/ssl/
## k8s-001执行
cd /root/cfssl/pki
cp ca*.pem /etc/etcd/ssl/
cp etcd*.pem /etc/etcd/ssl/
scp -r ca*.pem etcd*.pem k8s-002:/etc/etcd/ssl/
scp -r ca*.pem etcd*.pem k8s-003:/etc/etcd/ssl/

image-20230506170243141

启动 etcd 集群

### 每台执行
mkdir -p /var/lib/etcd/default.etcd

### 每台执行
systemctl daemon-reload
systemctl enable etcd --now

如果先启动第一台的etcd服务，会发现启动会命令行会卡住，这是正常的，只需要再启动第二台的，就可以了

查看集群状态

### k8s-001执行
/usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.204.10:2379,https://192.168.204.11:2379,https://192.168.204.12:2379 endpoint status --cluster

image-20230506170814870

我试了下关闭第一台虚拟机后的状态

image-20230506171012199

重启会恢复，但是不会恢复到之前的主节点

image-20230506171111385

1.3.11 k8s 二进制包安装

# 第一台下载即可：https://www.downloadkubernetes.com/
# 下载的二进制包有：kube-apiserver、kube-controller-manager、kube-proxy、kube-scheduler、kubectl、kubelet
# 我放在k8s-001的/opt下面
ll

image-20230506172017244

### k8s-001拷贝二进制包到其他节点
chmod +x kube*

cp kube-apiserver kube-controller-manager kube-scheduler kubectl kubelet kube-proxy /usr/local/bin/

scp -r kube-apiserver kube-controller-manager kube-scheduler kubectl kubelet kube-proxy k8s-002:/usr/local/bin/

scp -r kube-apiserver kube-controller-manager kube-scheduler kubectl kubelet kube-proxy k8s-003:/usr/local/bin/

image-20230506172347135

1.3.12 安装 kube-apiserver

创建数据目录

### 每台执行
mkdir -p /etc/kubernetes/ssl
mkdir /var/log/kubernetes

创建服务配置文件

### k8s-001上的/etc/kubernetes/kube-apiserver.conf

KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.204.10 \
  --secure-port=6443 \
  --advertise-address=192.168.204.10 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.165.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-50000 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.204.10:2379,https://192.168.204.11:2379,https://192.168.204.12:2379 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"


### k8s-002上的/etc/kubernetes/kube-apiserver.conf

KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.204.11 \
  --secure-port=6443 \
  --advertise-address=192.168.204.11 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.165.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-50000 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.204.10:2379,https://192.168.204.11:2379,https://192.168.204.12:2379 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"


### k8s-003上的/etc/kubernetes/kube-apiserver.conf

KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.204.12 \
  --secure-port=6443 \
  --advertise-address=192.168.204.12 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.165.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-50000 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.204.10:2379,https://192.168.204.11:2379,https://192.168.204.12:2379 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"

创建服务启动文件

### 每台执行创建文件：/usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service

[Service]
EnvironmentFile=-/etc/kubernetes/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

拷贝证书

### k8s-001执行
cd /root/cfssl/pki
cp ca*.pem /etc/kubernetes/ssl
cp kube-apiserver*.pem /etc/kubernetes/ssl/
cp token.csv /etc/kubernetes/

scp -r ca*.pem kube-apiserver*.pem k8s-002:/etc/kubernetes/ssl/
scp -r ca*.pem kube-apiserver*.pem k8s-003:/etc/kubernetes/ssl/

scp token.csv k8s-002:/etc/kubernetes/
scp token.csv k8s-003:/etc/kubernetes/

image-20230506175352260

启动服务

### 每台执行
systemctl daemon-reload
systemctl enable kube-apiserver --now

测试验证

### 每台执行
systemctl status kube-apiserver
### k8s-001执行
curl --insecure https://192.168.204.10:6443/
curl --insecure https://192.168.204.11:6443/
curl --insecure https://192.168.204.12:6443/

image-20230506185114753

以下都是正常的，因为还没授权

image-20230506185216252

1.3.13 安装 kubectl

我只需要在一台主机上操作 k8s 集群资源，所以我就在k8s-001 安装这个即可

  1. 拷贝证书
cd /root/cfssl/pki
cp admin*.pem /etc/kubernetes/ssl/
  1. 配置安全上下文
### 设置集群参数
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.204.13:16443 --kubeconfig=kube.config

### 设置客户端认证参数
kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=kube.config

### 设置上下文参数
kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kube.config

### 设置当前上下文
kubectl config use-context kubernetes --kubeconfig=kube.config

### 设置
mkdir /root/.kube
cp kube.config /root/.kube/config
cp kube.config /etc/kubernetes/admin.conf

image-20230507131411160

  1. 授权 kubernetes 证书访问权限
kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

验证一下

  1. 查看集群状态

kubectl cluster-info

image-20230507131708054

  1. 查看组件状态

kubectl get componentstatuses

image-20230507131831398

1.3.14 安装 kube-controller-manager

  1. 配置 kubeconfig
### 设置集群参数
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.204.13:16443 --kubeconfig=kube-controller-manager.kubeconfig

### 设置客户端参数
kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig

### 设置上下文参数
kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

### 设置当前上下文
kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

image-20230507132811962

  1. 创建配置文件
### 三台都一样的/etc/kubernetes/kube-controller-manager.conf文件

KUBE_CONTROLLER_MANAGER_OPTS=" \
  --secure-port=10257 \
  --bind-address=127.0.0.1 \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --service-cluster-ip-range=10.165.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --allocate-node-cidrs=true \
  --cluster-cidr=10.166.0.0/16 \
  --root-ca-file=/etc/kubernetes/ssl/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --horizontal-pod-autoscaler-sync-period=10s \
  --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
  --use-service-account-credentials=true \
  --alsologtostderr=true \
  --logtostderr=false \
  --v=2"
  1. 创建服务启动文件
### 三台都一样的/usr/lib/systemd/system/kube-controller-manager.service文件

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
  1. 拷贝证书
### k8s-001执行
cd /root/cfssl/pki
scp kube-controller-manager*.pem k8s-001:/etc/kubernetes/ssl/
scp kube-controller-manager*.pem k8s-002:/etc/kubernetes/ssl/
scp kube-controller-manager*.pem k8s-003:/etc/kubernetes/ssl/

scp kube-controller-manager.kubeconfig k8s-001:/etc/kubernetes/
scp kube-controller-manager.kubeconfig k8s-002:/etc/kubernetes/
scp kube-controller-manager.kubeconfig k8s-003:/etc/kubernetes/

在这里插入图片描述

  1. 启动服务
### 3台都执行
systemctl daemon-reload
systemctl enable kube-controller-manager --now

image-20230507134732078

  1. 检查一下

kubectl get componentstatuses

image-20230507134848031

1.3.15 安装 kube-scheduler

  1. 配置 kubeconfig
### k8s-001执行
cd /root/cfssl/pki

### 设置集群参数
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.204.13:16443 --kubeconfig=kube-scheduler.kubeconfig

### 设置客户端认证参数
kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig

### 设置上下文参数
kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

### 设置当前上下文
kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

image-20230507135352823

  1. 创建配置文件
###每台都要/etc/kubernetes/kube-scheduler.conf，都是一样的文件

KUBE_SCHEDULER_OPTS="--bind-address=127.0.0.1 \
--kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
--leader-elect=true \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2"
  1. 创建服务启动文件
###每台都要/usr/lib/systemd/system/kube-scheduler.service，都是一样的文件

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
  1. 拷贝文件
### k8s-001执行
cd /root/cfssl/pki
scp kube-scheduler*.pem k8s-001:/etc/kubernetes/ssl/
scp kube-scheduler*.pem k8s-002:/etc/kubernetes/ssl/
scp kube-scheduler*.pem k8s-003:/etc/kubernetes/ssl/

scp kube-scheduler.kubeconfig k8s-001:/etc/kubernetes/
scp kube-scheduler.kubeconfig k8s-002:/etc/kubernetes/
scp kube-scheduler.kubeconfig k8s-003:/etc/kubernetes/

image-20230507140041688

  1. 启动服务
### 每台执行
systemctl daemon-reload
systemctl enable kube-scheduler --now

image-20230507140139170

  1. 验证一下

image-20230507140338488

1.3.16 安装 kubelet

  1. 创建 kubelet-bootstrap.kubeconfig
cd /root/cfssl/pki
BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv)

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.204.13:16443 --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

image-20230507141038272

  1. 创建配置文件
### k8s-001上的/etc/kubernetes/kubelet.json

{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "192.168.204.10",
  "port": 10250,
  "readOnlyPort": 10255,
  "cgroupDriver": "systemd",
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "featureGates": {
    "RotateKubeletServerCertificate": true
  },
  "clusterDomain": "cluster.local.",
  "clusterDNS": ["10.165.0.2"]
}

### k8s-002上的/etc/kubernetes/kubelet.json
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "192.168.204.11",
  "port": 10250,
  "readOnlyPort": 10255,
  "cgroupDriver": "systemd",
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "featureGates": {
    "RotateKubeletServerCertificate": true
  },
  "clusterDomain": "cluster.local.",
  "clusterDNS": ["10.165.0.2"]
}
### k8s-003上的/etc/kubernetes/kubelet.json
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "192.168.204.12",
  "port": 10250,
  "readOnlyPort": 10255,
  "cgroupDriver": "systemd",
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "featureGates": {
    "RotateKubeletServerCertificate": true
  },
  "clusterDomain": "cluster.local.",
  "clusterDNS": ["10.165.0.2"]
}
  1. 创建服务启动文件
### 每台执行这个文件：/usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
  --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
  --cert-dir=/etc/kubernetes/ssl \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --config=/etc/kubernetes/kubelet.json \
  --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
  --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7 \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
  1. 拷贝文件
### k8s-001执行
cd /root/cfssl/pki
scp kubelet-bootstrap.kubeconfig k8s-001:/etc/kubernetes/
scp kubelet-bootstrap.kubeconfig k8s-002:/etc/kubernetes/
scp kubelet-bootstrap.kubeconfig k8s-003:/etc/kubernetes/

scp ca.pem k8s-001:/etc/kubernetes/ssl
scp ca.pem k8s-002:/etc/kubernetes/ssl
scp ca.pem k8s-003:/etc/kubernetes/ssl
  1. 启动服务
### 每台执行
mkdir /var/lib/kubelet
systemctl daemon-reload
systemctl enable kubelet --now
  1. 授权
### k8s-001执行

### CSR请求
kubectl get csr

### 同意
kubectl certificate approve xxx

image-20230507144001943

image-20230507144756042

1.3.17 安装 kube-proxy

  1. 配置 kubeconfig
cd /root/cfssl/pki

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.204.13:16443 --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

image-20230507145136239

  1. 创建配置文件
### k8s-001的/etc/kubernetes/kube-proxy.yaml文件
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.204.10
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 192.168.204.0/24
healthzBindAddress: 192.168.204.10:10256
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.204.10:10249
mode: "ipvs"

### k8s-002的/etc/kubernetes/kube-proxy.yaml文件
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.204.11
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 192.168.204.0/24
healthzBindAddress: 192.168.204.11:10256
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.204.11:10249
mode: "ipvs"

### k8s-003的/etc/kubernetes/kube-proxy.yaml文件
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.204.12
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 192.168.204.0/24
healthzBindAddress: 192.168.204.12:10256
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.204.12:10249
mode: "ipvs"
  1. 创建服务启动文件
### 三个节点都是一样的：/usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy.yaml \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
  1. 拷贝文件
### k8s-001执行
cd /root/cfssl/pki
scp kube-proxy.kubeconfig k8s-001:/etc/kubernetes/
scp kube-proxy.kubeconfig k8s-002:/etc/kubernetes/
scp kube-proxy.kubeconfig k8s-003:/etc/kubernetes/

image-20230507150217306

  1. 启动服务
### 每台执行
mkdir -p /var/lib/kube-proxy
systemctl daemon-reload
systemctl enable kube-proxy --now

1.3.18 安装网络插件 calico

### k8s-001执行即可
mkdir /opt/k8s-yaml 
cd /opt/k8s-yaml
### 提前准备好了相关文件和镜像包：calico.tar.gz和calico.yaml
### 将镜像拷贝到其他节点
scp calico.tar.gz k8s-002:/root
scp calico.tar.gz k8s-003:/root
### 在每个节点对应位置导入镜像
ctr -n=k8s.io images import calico.tar.gz
### 在k8s-001节点启动
kubectl apply -f calico.yaml

image-20230507150852082

再看下集群状态

kubectl get nodes

image-20230505211948132

1.3.19 安装 coredns

### k8s-001执行即可
cd /opt/k8s-yaml
### 提前准备好了coredns.yaml
### 注意修改下 clusterIP地址：clusterIP: 10.165.0.2
kubectl apply -f coredns.yaml

image-20230507151640156

1.4 插件安装及优化阶段

1.4.1 kubectl 命令补全功能

### 任意一台执行
yum install bash-completion
source /usr/share/bash-completion/bash_completion
bash
type _init_completion
kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null
source ~/.bashrc
bash

image-20230505222649739

1.4.2 Dashboard UI 安装

准备的 yaml 文件

直接下载也是可以的：https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

  • 这个服务的镜像不好拉取，我自己做成了阿里云的仓库进行拉取了（这是一共公开的仓库）

  • 需要的是这2个镜像

    • kubernetesui/metrics-scraper:v1.0.8
    • kubernetesui/dashboard:v2.7.0

  • 我换成了阿里云的

    • registry.cn-hangzhou.aliyuncs.com/k8s_whale_images/dashboard:v2.7.0
    • registry.cn-hangzhou.aliyuncs.com/k8s_whale_images/metrics-scraper:v1.0.8
# 我们还得修改一下这个服务的暴露方式，这样我们就可以外部访问了
# 要将配置文件里面的 Service 暴露方式设置成 NodePort 方式，这样我们外部就可以直接访问了
# 在342行位置
type: NodePort

# 再把镜像改一下
sed -i 's#kubernetesui/metrics-scraper:v1.0.8#registry.cn-hangzhou.aliyuncs.com/k8s_whale_images/metrics-scraper:v1.0.8#g' recommended.yaml

sed -i 's#kubernetesui/dashboard:v2.7.0#registry.cn-hangzhou.aliyuncs.com/k8s_whale_images/dashboard:v2.7.0#g' recommended.yaml

image-20230506122156948

运行一下

kubectl apply -f recommended.yaml

image-20230507155855846

看下状态

kubectl get pod -A

image-20230507155935074

配置访问

### dashboard-admin.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

生效一下

kubectl delete -f dashboard-admin.yaml
kubectl create -f dashboard-admin.yaml

image-20230506122913833

生成登陆 token

这个版本的好像和之前的有所改变，需要这样操作才能获取到登陆 token

  • 创建服务账号
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
  • 集群角色绑定
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
  • 生成 token
kubectl -n kubernetes-dashboard create token admin-user

image-20230506123258352

登陆验证

##看一下访问端口
kubectl get svc -n kubernetes-dashboard

## 44255

image-20230507160258982

试试浏览器访问：https://192.168.204.13:44255/ （vip 地址也可）

输入上面生成的 token 即可访问

image-20230507160405831

1.4.3 安装 metrics

有关 metrics-server 地址：https://github.com/kubernetes-sigs/metrics-server#readme

Metrics Server是 k8s 内置自动缩放管道的可扩展、高效的容器资源度量源。

Metrics Server 从 Kubelets 收集资源度量，并通过Metrics API在Kubernetes apiserver中公开这些度量，供Horizontal Pod Autoscaler和Vertical Pod Autocaler使用。kubectl top还可以访问Metrics API，从而更容易调试自动缩放管道。

准备 yaml 文件

##### metrics-server.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --kubelet-insecure-tls
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/metrics-server:v0.4.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          periodSeconds: 10
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

部署一下

### k8s-001执行即可
kubectl apply -f metrics-server.yaml

image-20230506114600384

查看一下状态

kubectl get pod -n kube-system

image-20230506114646238

还需要额外配置才能生效

不然服务一直输出以下错误：

image-20230507172220925

### k8s-001执行
cd /root/cfssl/pki
### 生成proxy-client-csr.json内容

{
  "CN": "aggregator",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Sichuan",
      "L": "Chengdu",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

### 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client

### 复制证书
scp proxy-client*.pem k8s-001:/etc/kubernetes/ssl/
scp proxy-client*.pem k8s-002:/etc/kubernetes/ssl/
scp proxy-client*.pem k8s-003:/etc/kubernetes/ssl/

image-20230507172822304

修改kube-apiserver配置文件

### 增加以下内容
--runtime-config=api/all=true \
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem \
--proxy-client-cert-file=/etc/kubernetes/ssl/proxy-client.pem \
--proxy-client-key-file=/etc/kubernetes/ssl/proxy-client-key.pem \

重启生效

### 每个节点依次执行
systemctl daemon-reload && systemctl restart kube-apiserver

重新部署一下看是否正常

image-20230507173919614

image-20230507173951272

image-20230507174431068

1.5 集群验证阶段

1.5.1 节点验证

kubectl get nodes

image-20230507160434753

  • 节点正常处于 Ready 状态
  • 版本和安装无出入

1.5.2 Pod 验证

kubectl get pods -A

image-20230507160509961

  • 状态肯定是需要是 Running 状态
  • Pod 准备是否就绪，处于 1/1（前面数字和后面数字保持一致）
  • 对于重启次数，不用太纠结这个，虚拟机重启都会导致这个重启次数

1.5.3 k8s 网段验证

kubectl get svc
kubectl get pod -A -owide

image-20230507160556104

  • 核实一下对于的网段是否和我们规划的一致，是否存在冲突

  • 对于网络用的宿主机 IP 地址，是因为 pod 网络模式用的主机模式

1.5.4 创建资源验证

### 我们使用国内的一个debug工具镜像进行验证即可，包括后续的验证，准备这么一个yaml文件
apiVersion: apps/v1
kind: Deployment
metadata:
  name: debug-tools
  labels:
    app: debug-tools
spec:
  replicas: 3
  selector:
    matchLabels:
      app: debug-tools
  template:
    metadata:
      labels:
        app: debug-tools
    spec:
      containers:
      - name: debug-tools
        image: registry.cn-hangzhou.aliyuncs.com/k8s_whale_images/debug-tools:latest
        command: ["/bin/sh","-c","sleep 3600"]

运行一下

kubectl apply -f demo.yaml

查看一下是否正常部署

  • 副本数
  • pod 状态
  • 容器内部

image-20230507160959723

image-20230507161253664

删除资源是否正常

kubectl delete -f demo.yaml
  • 资源配置有限，稍微比较慢，只要资源能够正常创建删除即可

image-20230507161410779

1.5.5 pod 解析验证

验证是否正确解析 service

# 从上面的pod进行验证

# 同空间（容器内部验证）
nslookup kubernetes
# 跨空间
nslookup kube-dns.kube-system

image-20230507161719463

  • 有 IP 返回即可

1.5.6 节点访问验证

# 宿主机必须是能够访问k8s svc 443 和 kube-dns的53端口
# 每个节点
curl -k https://10.165.0.1:443
curl 10.165.0.10:53

image-20230507161902292

  • 有这些说明事正常通信的

1.5.7 Pod 之间的通信

同机器上的pod和不同机器上的都要去验证一下

准备2个ymal

第一个

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

第二个

apiVersion: apps/v1
kind: Deployment
metadata:
  name: debug-tools
  namespace: kube-system
  labels:
    app: debug-tools
spec:
  replicas: 1
  selector:
    matchLabels:
      app: debug-tools
  template:
    metadata:
      labels:
        app: debug-tools
    spec:
      containers:
      - name: debug-tools
        image: registry.cn-hangzhou.aliyuncs.com/k8s_whale_images/debug-tools:latest
        command: ["/bin/sh","-c","sleep 3600"]

跑起来

  • 我们跑了2个容器，在不同的空间下

image-20230507162047258

验证 Pod 之间通信

查看一下各 pod 地址

image-20230507162156041

实际去 ping 验证一下：

# 不同空间
kubectl exec -it debug-tools-7b466cf8dc-xmhl7 -n kube-system ping 10.166.6.10

在这里插入图片描述

相同的空间

image-20230507162411519

到现在，应该算是一个比较完美的过程了

本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系:hwhale#tublm.com(使用前将#替换为@)

Kubernetes运维

Kubernetes

运维

学习

云原生

k8s集群部署 的相关文章

随机推荐

  • 2023国赛数学建模B题思路代码 - 多波束测线问题

    1 赛题 B 题 多波束测线问题 单波束测深是利用声波在水中的传播特性来测量水体深度的技术 声波在均匀介质中作匀 速直线传播 在不同界面上产生反射 利用这一原理 从测量船换能器垂直向海底发射声波信 号 并记录从声波发射到信号接收的传播时间

  • C++ 中栈对象的使用总结、RAII

    背景 栈区用于存放函数的参数 局部变量 返回值等 栈区的数据由编译器自动进行分配 在作用域内有效 在超出变量作用域后 栈中数据由编译器自动释放 栈内存分配运算内置于处理器的指令集 效率高 但是分配的内存容量有限 栈对象 栈对象在创建时会自动

  • yagmail发送附件

    效果图 经测试代码 导入yagmail第三方库 import yagmail yagmail SMTP user 邮箱名 host SMTP服务器域名 yag yagmail SMTP user 284036658 qq com host

  • 程序员的自我修养——链接，装载与库（一）

    程序员的自我修养 链接 装载与库这本书看了差不多有一个多月了 这本书讲了很多计算机底层的知识 也补充了我的知识盲区 但是感觉看完以后前面有的知识有遗忘 因此就想好好的总结一下 也可以更好的理解这本书 计算机三个最重要的硬件是 中央处理器CP

  • Linux用户与用户组

    Linux目录详解 目录名 说明 bin 重要的二进制 binary 应用程序 包含二进制文件 系统的所有用户使用的命令都在这个目录下 boot 启动 boot 配置文件 包含引导加载程序相关的文件 开机时用到的引导文件 data 数据存储

  • 事件驱动框架（五）——框架的实现

    事件驱动框架 五 框架的实现 说明 这里先描述一下QP的一些策略和源码 因为某原因这个系列先停更 后面主要是内核介绍 实现 1 临界区 临界区内每次只准许一个线程 进程 进入 进入后不允许其他线程 进程 进入 因此临界区的代码不可分割 在嵌

  • 【axios】get/post请求params/data传参总结

    axios中get post请求方式 1 前言 最近突然发现post请求可以使用params方式传值 然后想总结一下其中的用法 2 1 分类 get请求中没有data传值方式 经过查阅资料 get请求是可以通过body传输数据的 但是许多工

  • jeecgboot接口限制每ip每分钟访问次数——限制ip请求频率【伸手党福利】

    代码借鉴的别人的 自己做过部分修改 1 新建文件夹并新建文件 jeecg boot base jeecg boot base core src main java org jeecg common accesslimit RequestLi

  • Spire.XLS 图表系列教程：C# 填充 Excel 图表中的图例背景色

    默认情况下 创建图表时 Excel会自动设置其坐标轴属性 这些属性包括坐标轴选项 例如边界最大值 边界最小值 主要单位 次要单位 刻度线标记的主要类型 次要类型 标签位置 横坐标轴交叉设置 是否逆序刻度值等 但是有时我们需要手动设置这些属性

  • nginx之配置文件niginx.conf（全网看这一篇就行）

    指定nginx的工作进程的用户及用户组 默认是nobody用户 user nobody 指定工作进程的个数 默认是1个 具体可以根据服务器cpu数量进行设置 比如cpu有4个 可以设置为4 如果不知道cpu的数量 可以设置为auto ngi

  • 【lwIP（第十三章）】WebServer协议

    目录 一 WebServer简介 二 Web服务器工作原理 三 CGI技术简介 四 CGI工作原理 五 SSI技术简介 六 SSI工作原理 一 WebServer简介 Web Server 就是提供 Web 服务的 Server 主要功能是

  • 【GRE】GRE普通考试改革前后区别

    参考张禄老师的视频 首先是官网的总结 从图中看出的几个点 写作 由 argument issue 改为 issue 删除 argument 这意味着写作想获得高分将更加困难 因为大陆考生基本都是靠 argument 拉分的 数学和语文 题量

  • 谈谈我的个人追求

    说到个人追求 我发现我自己都无法说出来 是我没有答案 还是不敢去追求呢 是我的心太浮 要的太多吧 反正这个问题也不是几分钟的思考就可以得到的答案 这个问题 或许需要我一辈子的努力去寻找 见识的东西太少了 视野过于狭隘 愿努力之 看到更大的世

  • 三、MySql 数据类型

    文章标题 Mysql数据类型 Int 类型 INT N 是什么 字符类型 排序规则 时间类型 前文 mysql权限 Mysql数据类型 Int 类型 有无符号 在项目中使用 BIGINT 而且是有符号的 演示 create table te

  • 网络编程——IO模型

    搭建select的TCP客户端 include

  • 100流明相当于多少w_20年前的100元，相当于现在的多少钱？说出来你可能不信

    最近 有网友提出一个问题 他说20年前的100元 相当于现在的多少钱 对此 专家们回答是 2000年我国GDP总量为1 2万亿美元 位居全球第六 到2019年 我国GDP已经高达14 3万亿美元 位居全球第二位 20年时间GDP增长了11

  • 汇编符号语言

    CSDN话题挑战赛第1期 活动详情地址 第1期话题PK赛 参赛话题 汇编知识分享 话题描述 我们的计算机知识就像一座金字塔 底层是数学 上面是数字电路 然后是汇编 再往上是操作系统 网络 数据库 高级编程语言 框架等等 我们不可能精通这个金

  • 前端项目uniapp小兔鲜儿微信小程序项目

    小兔鲜儿 项目起步 项目架构 项目架构图 拉取项目模板代码 项目模板包含 目录结构 项目素材 代码风格 模板地址 git clone http git itcast cn heimaqianduan erabbit uni app vue3

  • 由于找不到msvcp100.dll无法继续执行代码怎么解决

    当遇到程序无法正常运行 提示缺少msvcp100 dll文件时 最初的反应可能是困惑和不知所措 然而 通过修复msvcp100 dll文件 我发现这个问题实际上并不复杂 并且可以通过一些简单的步骤解决 在修复msvcp100 dll文件的时

  • k8s集群部署

    文章目录 1 二进制部署三节点 复用 高可用 k8s 集群 1 1 环境规划阶段 1 1 1 实验架构图 1 1 2 系统版本说明 1 1 3 环境基本信息 1 1 4 k8s 网段划分 1 2 基础安装及优化阶段 1 2 1 系统信息检查

热门标签