클러스터 자체 업그레이드 절차 정리.
v1.29.15에서 v1.35.1까지 순차 업그레이드.
업그레이드 경로.
v1.29.15
v1.30.14
v1.31.14
v1.32.12
v1.33.8
v1.34.4
v1.35.1
순서 중요.
master 먼저.
worker 나중.
worker가 master보다 높아지면 kubelet이 API 서버와 통신 안 될 수 있음.
현재 상태 확인.
kubectl get nodes
v1.30 master 업그레이드.
sudo apt-mark unhold kubelet kubeadm kubectl
sudo sed -i "s|v1\.[0-9]*|v1.30|g" /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y --allow-change-held-packages kubeadm
sudo kubeadm upgrade apply v1.30.14 -y
sudo apt-get install -y --allow-change-held-packages kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
worker1 drain.
kubectl drain k8s-worker1 --ignore-daemonsets --delete-emptydir-data
PDB가 있으면 이런 식으로 막아줌.
Cannot evict pod as it would violate the pod's disruption budget.
worker1 업그레이드.
ssh [email protected]
sudo apt-mark unhold kubelet kubeadm kubectl
sudo sed -i "s|v1\.[0-9]*|v1.30|g" /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y --allow-change-held-packages kubelet kubeadm kubectl
sudo systemctl restart kubelet
exit
worker1 uncordon.
kubectl uncordon k8s-worker1
worker2도 동일.
kubectl drain k8s-worker2 --ignore-daemonsets --delete-emptydir-data
ssh [email protected]
sudo apt-mark unhold kubelet kubeadm kubectl
sudo sed -i "s|v1\.[0-9]*|v1.30|g" /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y --allow-change-held-packages kubelet kubeadm kubectl
sudo systemctl restart kubelet
exit
kubectl uncordon k8s-worker2
v1.31 업그레이드.
sudo apt-mark unhold kubeadm
sudo sed -i "s|v1\.[0-9]*|v1.31|g" /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y --allow-change-held-packages kubeadm
sudo kubeadm upgrade apply v1.31.14 -y
sudo apt-get install -y --allow-change-held-packages kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
ConfigMap 버전이 남아있으면 수정.
kubectl -n kube-system get cm kubeadm-config -o yaml | \
sed "s/kubernetesVersion: v1.30.14/kubernetesVersion: v1.31.14/" | \
kubectl apply -f -
v1.32 예시.
sudo sed -i "s|v1\.[0-9]*|v1.32|g" /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y --allow-change-held-packages kubeadm=1.32.12-1.1
sudo kubeadm upgrade apply v1.32.12 -y
sudo apt-get install -y --allow-change-held-packages kubelet=1.32.12-1.1 kubectl=1.32.12-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet
worker 노드.
ssh [email protected]
sudo apt-mark unhold kubelet kubeadm kubectl
sudo sed -i "s|v1\.[0-9]*|v1.32|g" /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y --allow-change-held-packages kubelet=1.32.12-1.1 kubeadm=1.32.12-1.1 kubectl=1.32.12-1.1
sudo systemctl restart kubelet
exit
v1.33, v1.34도 같은 방식.
v1.35 master 업그레이드.
sudo sed -i "s|v1\.[0-9]*|v1.35|g" /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y --allow-change-held-packages kubeadm=1.35.1-1.1
sudo kubeadm upgrade apply v1.35.1 -y
sudo apt-get install -y --allow-change-held-packages kubelet=1.35.1-1.1 kubectl=1.35.1-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet
v1.35에서 kubelet 플래그 문제 발생.
에러.
failed to parse kubelet flag: unknown flag: --pod-infra-container-image
제거.
sudo sed -i "s/ --pod-infra-container-image=[^ ]*//g" /var/lib/kubelet/kubeadm-flags.env
sudo systemctl restart kubelet
systemctl status kubelet
worker1 v1.35.
kubectl drain k8s-worker1 --ignore-daemonsets --delete-emptydir-data
ssh [email protected]
sudo apt-mark unhold kubelet kubeadm kubectl
sudo sed -i "s|v1\.[0-9]*|v1.35|g" /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y --allow-change-held-packages kubelet=1.35.1-1.1 kubeadm=1.35.1-1.1 kubectl=1.35.1-1.1
sudo systemctl restart kubelet
sudo sed -i "s/ --pod-infra-container-image=[^ ]*//g" /var/lib/kubelet/kubeadm-flags.env
sudo systemctl restart kubelet
systemctl status kubelet
exit
kubectl uncordon k8s-worker1
worker2도 동일.
최종 확인.
kubectl get nodes
결과.
k8s-master
Ready
v1.35.1
k8s-worker1
Ready
v1.35.1
k8s-worker2
Ready
v1.35.1
Pod 확인.
kubectl get pods -l "app in (nginx-demo,apache-demo)"
서비스 확인.
for i in {1..20}; do
curl -s -o /dev/null -w "%{http_code} " http://192.168.255.100/
done
결과.
200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200
체크.
kubectl 명령은 master에서.
apt/systemctl은 각 노드에서.
버전은 한 단계씩.
순서는 master -> worker1 -> worker2.
v1.35는 pod-infra-container-image 플래그 제거.