Cluster update known issues¶
This section lists the cluster update known issues with workarounds for the Mirantis OpenStack for Kubernetes release 21.6.
 Cluster update failure with kubelet being stuck¶
Moving forward, the workaround for this issue will be moved from Release Notes to Mirantis Container Cloud documentation: MOS managed clusters update fails with stuck kubelet.
A MOS managed cluster may fail to update to the latest Cluster release with kubelet being stuck and reporting authorization errors.
The cluster is affected by the issue if you see the Failed to make webhook authorizer request: context canceled error in the kubelet logs:
docker logs ucp-kubelet --since 5m 2>&1 | grep 'Failed to make webhook authorizer request: context canceled'
As a workaround, restart the
ucp-kubelet container on the affected
ctr -n com.docker.ucp snapshot rm ucp-kubelet docker rm -f ucp-kubelet
Ignore failures in the output of the first command, if any.
 Cluster update fails at Ceph CSI pod eviction¶
An update of a MOS managed cluster may fail with the ceph csi-driver is not evacuated yet, waiting… error during the Ceph CSI pod eviction.
Scale the affected
StatefulSetof the pod that fails to init down to
0replicas. If it is the
nova-compute, it must not be scheduled on the affected node.
csi-rbdpluginpod, search for stuck
rbd device list | grep <csi-vol-uuid>
Unmap the affected
rbd unmap -o force /dev/rbd<i>
volumeattachmentof the affected pod:
kubectl get volumeattachments | grep <csi-vol-uuid> kubectl delete volumeattacmhent <id>
Scale the affected
StatefulSetback to the original number of replicas or until its state is
Running. If it is a
DaemonSet, run the pod on the affected node again.
 MySQL crashes during managed cluster update or instances live migration¶
MySQL may crash when performing instances live migration or during a managed
cluster update from version 6.19.0 to 6.20.0. After the crash, MariaDB cannot
connect to the cluster and gets stuck in the
Verify that other MariaDB replicas are up and running and have joined the cluster:
Verify that at least 2 pods are running and operational (
kubectl -n openstack get pods |grep maria
Example of system response where the pods
mariadb-controller-77b5ff47d5-ndj68 1/1 Running 0 39m mariadb-server-0 2/2 Running 0 39m mariadb-server-1 0/2 Running 0 39m mariadb-server-2 2/2 Running 0 39m
Log in to each operational pod and verify that the node is
Primaryand the cluster size is at least
2. For example:
mysql -u root -p$MYSQL_DBADMIN_PASSWORD -e "show status;" |grep -e \ wsrep_cluster_size -e "wsrep_cluster_status" -e "wsrep_local_state_comment"
Example of system response:
wsrep_cluster_size 2 wsrep_cluster_status Primary wsrep_local_state_comment Synced
Remove the content of the
kubectl -n openstack exec -it mariadb-server-1 – rm -rf /var/lib/mysql/*
Restart the MariaDB container:
kubectl -n openstack delete pod mariadb-server-1