Gauss Boot

Appearance

Rancher 安装和配置(2.6.x) ​

此文档用于安装k3sv1.24.x和rancher2.6.x集群

公共配置 ​

请参照老版本的公共配置

K3S 安装 ​

安装k3s需要有互联网环境

安装master节点 ​

bash
curl -sfL https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_VERSION=v1.24.14+k3s1 sh -
curl  https://mirrors.aliyun.com/repo/epel-7.repo -o /etc/yum.repos.d/epel.repo
echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> ~/.bash_profile

注意, 如果需要使用docker作为后端引擎，需要修改参数为sh -s - --docker

高可用集群安装 ​

高可用集群可以使用mysql做的集群存储

bash
curl -sfL https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_VERSION=v1.24.14+k3s1 sh -s - server --datastore-endpoint="mysql://rancher:password@tcp(localhost:3306)/rancher"

安装agent节点 ​

首先获取master节点的配置信息, K3S_TOKEN在/var/lib/rancher/k3s/server/node-token下 通过以下命令安装节点，需求修改K3S_URL和K3S_TOKEN

curl -sfL https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_VERSION=v1.24.14+k3s1 K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken sh -

kubectl label node bigdata-02 node-role.kubernetes.io/worker=worker

k3s卸载 ​

bash
cd /usr/local/bin/
./k3s-uninstall.sh
./k3s-agent-uninstall.sh

升级k3s ​

注意需要使用相同的初始化参数初始化k3s

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_VERSION=v1.24.14+k3s1 sh -

升级高可用集群 ​

需要先升级k3s到1.24.x版本后，使用--cluster-init参数初始化master节点，升级前请先备份master节点

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_VERSION=v1.24.14+k3s1 sh -s - server --cluster-init

安装Rancher ​

安装Helm ​

source ~/.bash_profile
#
yum install -y wget
#yum update -y
#yum install -y snapd
#systemctl start snapd
#ln -s /var/lib/snapd/snap /snap
#snap install helm --classic
#export PATH=$PATH:/var/lib/snapd/snap/bin/
wget https://mirrors.huaweicloud.com/helm/v3.9.4/helm-v3.9.4-linux-amd64.tar.gz
tar -zxvf helm-v3.7.2-linux-amd64.tar.gz
cp linux-amd64/helm /usr/bin

安装Cert Manager ​

安装以下命令安装cert manager

bash
helm repo add rancher-stable http://rancher-mirror.oss-cn-beijing.aliyuncs.com/server-charts/stable
wget https://github.com/jetstack/cert-manager/releases/download/v1.7.3/cert-manager.crds.yaml
kubectl apply -f cert-manager.crds.yaml
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager   --namespace cert-manager   --create-namespace   --version v1.7.3

安装Rancher ​

运行以下命令安装rancher，注意需要修改hostname参数，该参数是一个可以指向rancher的域名

kubectl create namespace cattle-system
helm install rancher rancher-stable/rancher  --namespace cattle-system  --set hostname=rancher.10.108.6.28.nip.io  --set replicas=1 --version v2.6.10

升级rancher ​

helm repo update
helm fetch rancher-stable/rancher --version=v2.6.10
helm get values rancher -n cattle-system -o yaml > values.yaml
helm upgrade rancher rancher-stable/rancher --namespace cattle-system -f values.yaml --version v2.6.10

#helm upgrade rancher rancher-stable/rancher --namespace cattle-system --set hostname=rancher.10.108.6.28.nip.io --version=2.6.10

启用gpu ​

注意此处是用ubuntu系统作为示例

安装依赖 ​

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update

sudo apt-get install -y nvidia-container-toolkit

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml


apt install -y nvidia-container-runtime cuda-drivers-fabricmanager-515 nvidia-headless-515-server

编辑容器配置文件 ​

/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl

version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true
  default_runtime_name = "nvidia"


[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
  runtime_type = "io.containerd.runc.v2"
  privileged_without_host_devices = false
  runtime_engine = ""
  runtime_root = ""
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/bin/nvidia-container-runtime"