集群搭建

参考： Kubernetes 文档 / 入门 / 生产环境 / 使用部署工具安装 Kubernetes / 使用 kubeadm 引导集群 / 安装 kubeadm

流程图

准备开始

  • 一台兼容的 Linux 主机。Kubernetes 项目为基于 Debian 和 Red Hat 的 Linux 发行版以及一些不提供包管理器的发行版提供通用的指令
  • 每台机器 2 GB 或更多的 RAM （如果少于这个数字将会影响你应用的运行内存)
  • 2 CPU 核或更多
  • 集群中的所有机器的网络彼此均能相互连接(公网和内网都可以)
  • 节点之中不可以有重复的主机名、MAC 地址或 product_uuid。请参见这里了解更多详细信息。
  • 开启机器上的某些端口。请参见这里 了解更多详细信息。
  • 禁用交换分区。为了保证 kubelet 正常工作，你 必须 禁用交换分区。

U. 确保每个节点上 MAC 地址和 product_uuid 的唯一性

  • 你可以使用命令 ip link 或 ifconfig -a 来获取网络接口的 MAC 地址
  • 可以使用 sudo cat /sys/class/dmi/id/product_uuid 命令对 product_uuid 校验

准备工作

[k8s-master|k8s-worker1|k8s-worker2]$

  1. 设置当前用户sudo免密[选做]

    不想每次都输入密码 - 加速

    1
    2
    3
    4
    5
    6
    7
    # 缓存 sudo 密码
    echo ubuntu | sudo -v -S 

    # 
    sudo tee /etc/sudoers.d/$USER >/dev/null <<EOF
    $USER ALL=(ALL) NOPASSWD: ALL
    EOF
  2. 使用国内镜像仓库[选做]

    软件安装 - 加速

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    # 更换阿里云加速

    cp /etc/apt/sources.list /etc/apt/sources.list_bak
    vim /etc/apt/sources.list
    # 查看/etc/apt/sources.list中的URL是archive.ubuntu还是cn.archive.ubuntu 
    # 然后再执行：
    sudo sed -i 's/cn.archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list
    # 或 
    sudo sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list


  3. 编辑 hosts<必做> 
    1
    2
    3
    4
    5
    sudo tee -a /etc/hosts >/dev/null <<EOF
    192.168.147.131 k8s-master
    192.168.147.132 k8s-worker1
    192.168.147.133 k8s-worker2
    EOF

[k8s-master|k8s-worker1|k8s-worker2]$

  1. 禁用 swap<必做>

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    # 交换文件
    SWAPF=$(awk '/swap/ {print $1}' /etc/fstab)

    # 立即禁用
    sudo swapoff $SWAPF

    # 永久禁用
    sudo sed -i '/swap/d' /etc/fstab

    # 删除交换文件
    sudo rm $SWAPF

  2. 模块支持<必做>

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    # 安装
    sudo apt -y install bridge-utils

    # 立即生效
    sudo modprobe br_netfilter

    # 内核支持
    sudo tee /etc/sysctl.d/k8s.conf >/dev/null <<EOF
    net.ipv4.ip_forward=1
    vm.swappiness=0
    vm.overcommit_memory=1
    vm.panic_on_oom=0
    EOF

    # 立即生效
    sudo sysctl -p /etc/sysctl.d/k8s.conf

  3. 安装运行时<必做>

    这里不再采用docker作为k8s的运行时. 因为K8s自1.24 对docker支持改为 安装指定CRI才能访问. 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    # 安装 containerd https://download.docker.com/linux/ubuntu/dists/jammy/pool/stable/amd64/
    # 下载最新版本 containerd 因为在k8s后续版本废弃1.5.x 安装对应的deb版本
    sudo apt install -y containerd

    # 锁定版本
    sudo apt-mark hold containerd

    # 创建目录
    sudo mkdir /etc/containerd

    # 生成默认配置文件
    containerd config default | \
    sudo tee /etc/containerd/config.toml >/dev/null

    # 修改配置文件 加速
    sudo sed -i \
    -e '/sandbox_image/s?k8s.gcr.io?registry.aliyuncs.com/google_containers?' \
    -e '/SystemdCgroup/s?false?true?' \
    -e '/registry.mirrors/a\        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]' \
    -e '/registry.mirrors/a\          endpoint = ["https://docker.nju.edu.cn/"]' /etc/containerd/config.toml

    # 服务重启
    sudo systemctl restart containerd
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    # 运行时配置私服
    [plugins."io.containerd.grpc.v1.cri".registry]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
            [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
              endpoint = ["https://------.mirror.aliyuncs.com", "https://registry-1.docker.io"]
    # 运行时配置登录
    [plugins."io.containerd.grpc.v1.cri".registry]
       [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
           [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
              endpoint = ["https://registry-1.docker.io"]
           [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.cn-hangzhou.aliyuncs.com"]
              endpoint = ["https://registry.cn-hangzhou.aliyuncs.com"]
          [plugins."io.containerd.grpc.v1.cri".registry.configs]
            [plugins."io.containerd.grpc.v1.cri".registry.configs."registry.cn-hangzhou.aliyuncs.com".tls]
              insecure_skip_verify = true
            [plugins."io.containerd.grpc.v1.cri".registry.configs."registry.cn-hangzhou.aliyuncs.com".auth]
              username = "阿里云账户，类似xxx@aliyun.com"
              password = "上一步设置的固定密码"

安装 K8s

[kiosk@k8s-master|k8s-worker1|k8s-worker2]$

  1. 安装 kubeadm、kubelet 和 kubectl
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    # 更新 apt 包索引并安装使用 Kubernetes apt 仓库所需要的包
    sudo apt -y install apt-transport-https ca-certificates curl

    # 下载 Google Cloud 公开签名秘钥
    curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -

    # 添加 Kubernetes apt 仓库
    MIRROR_URL=https://mirrors.aliyun.com/kubernetes/apt/
    sudo tee /etc/apt/sources.list.d/kubernetes.list >/dev/null <<EOF
    deb $MIRROR_URL kubernetes-xenial main
    EOF

    # 更新 apt 包索引
    sudo cp /etc/apt/trusted.gpg /etc/apt/trusted.gpg.d
    sudo apt update -y

    # 查询指定版本
    sudo apt-cache madison kubelet | grep 1.25

    # 安装 kubelet、kubeadm 和 kubectl 指定版本
    sudo apt install -y kubelet=1.25.1-00 kubeadm=1.25.1-00 kubectl=1.25.1-00

    # 锁定版本
    sudo apt-mark hold kubelet kubeadm kubectl

    1
    2
    3
    4
    5
    6
    7
    8
    9
    # 安装完K8s以后会自带crictl
    # crictl 配置文件
    sudo tee /etc/crictl.yaml >/dev/null <<EOF
    runtime-endpoint: unix:///run/containerd/containerd.sock
    image-endpoint: unix:///run/containerd/containerd.sock
    timeout: 10
    debug: false
    pull-image-on-create: true
    EOF
  2. k8s 支持
    1
    2
    3
    4
    5
    6
    7
    # 增加 k8s 支持
    sudo sed -i '/ExecStart=\//s|$| --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --cgroup-driver=systemd|' \
      /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

    # 重启 kubelet 服务
    sudo systemctl daemon-reload
    sudo systemctl restart kubelet

[k8s-master]$

  1. 初始化
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    # 生成初始文件
    sudo kubeadm config print init-defaults > kubeadm-config.yaml

    # 修改文件
    NICP=$(ip a | awk '/inet / {print $2}' | grep -v ^127 | sed 's+/24++')
    sudo sed -i \
      -e "/advertiseAddress/s?:.*?: $NICP?" \
      -e "/name/s?:.*?: $(hostname -s)?" \
      -e "/clusterName/s?:.*?: k8s?" \
      -e "/imageRepository/s?:.*?: registry.aliyuncs.com/google_containers?" kubeadm-config.yaml

    # 使用初始文件，初始化集群
    sudo kubeadm init --config kubeadm-config.yaml

    Your Kubernetes control-plane has initialized successfully!
    PS: 普通用户管理集群

    To start using your cluster, you need to run the following as a regular user:
    bash
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

    PS：root 用户管理集群

    Alternatively, if you are the root user, you can run:

    bash
    export KUBECONFIG=/etc/kubernetes/admin.conf

    You should now deploy a pod network to the cluster.Run “kubectl apply -f [podnetwork].yaml“ with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/
    Then you can join any number of worker nodes by running the following on each as root:
    bash
    kubeadm join 192.168.147.128:6443 –token abcdef.0123456789abcdef –discovery-token-ca-cert-hash sha256:c4781194de65ebb47984fc5e7e64d4897875410825ce4d18df81da1a298afa1f

  2. 配置文件 - Client
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    # 创建目录
    mkdir -p $HOME/.kube

    # user 复制配置文件
    sudo \cp /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

    # root 变量
    sudo tee -a ~root/.bashrc >/dev/null <<EOF
    export KUBECONFIG=/etc/kubernetes/admin.conf
    EOF
  3. 创建网络
    1
    kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
  4. 命令补全 - Client[建议]
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    $ kubectl completion --help

    # 立即生效
    source <(kubectl completion bash)
        
    # 永久生效
    mkdir ~/.kube 2>/dev/null
    kubectl completion bash > ~/.kube/completion.bash.inc
    printf "
    # Kubectl shell completion
    source '$HOME/.kube/completion.bash.inc'
    " >> $HOME/.bashrc
    source $HOME/.bashrc
  5. 命令别名 - Client[建议]
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    # 网址 https://kubernetes.io/zh-cn/docs/reference/kubectl/cheatsheet/

    # 永久生效
    tee -a $HOME/.bashrc >/dev/null <<EOF
    alias k='kubectl'
    complete -F __start_kubectl k
    EOF

    # 立即生效
    source $HOME/.bashrc

[k8s-worker1|k8s-worker2]$

  1. 加入集群
    1
    2
    3
    4
    sudo \
      kubeadm join 192.168.147.128:6443 \
      --token abcdef.0123456789abcdef \
      --discovery-token-ca-cert-hash sha256:c4781194de65ebb47984fc5e7e64d4897875410825ce4d18df81da1a298afa1f

C. 确认环境正常

[k8s-master]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
$ kubectl get nodes
NAME          STATUS     ROLES           AGE     VERSION
k8s-master   `Ready`     control-plane   9m17s   `v1.25.1`
k8s-worker1  `Ready`     <none>          90s     `v1.25.1`
k8s-worker2  `Ready`     <none>          51s     `v1.25.1`

$ kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
scheduler           `Healthy`  ok
controller-manager  `Healthy`  ok
etcd-0              `Healthy`  {"health":"true","reason":""}

$ kubectl -n kube-system get pod -w
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-798cc86c47-dkdjl   1/1     Running   0          4m5s
calico-node-ftwk8                          1/1     Running   0          4m5s
calico-node-hstcg                          1/1     Running   0          109s
calico-node-lcnw6                          1/1     Running   0          2m28s
coredns-c676cc86f-mxpb8                    1/1     Running   0          10m
coredns-c676cc86f-vhzzh                    1/1     Running   0          10m
etcd-k8s-master                            1/1     Running   0          10m
kube-apiserver-k8s-master                  1/1     Running   0          10m
kube-controller-manager-k8s-master         1/1     Running   0          10m
kube-proxy-g2tz9                           1/1     Running   0          109s
kube-proxy-j4fgc                           1/1     Running   0          10m
kube-proxy-nz8vj                           1/1     Running   0          2m28s
kube-scheduler-k8s-master                  1/1     Running   0          10m
<Ctrl-C>