Update a managed cluster

After you verify that the Mirantis Container Cloud management cluster is upgraded successfully as described in Verify the Container Cloud status before managed cluster update, proceed to update your managed clusters using the Container Cloud web UI.

Caution

During a baremetal-based cluster update, hosts can be restarted to apply the latest supported Ubuntu 18.04 or 20.04 packages. In this case:

  • Depending on the cluster configuration, applying security updates and host restart can increase the update time for each node to up to 1 hour.

  • Cluster nodes are updated one by one. Therefore, for large clusters, the update may take several days to complete.

Note

For a MOSK-based cluster update procedure, refer to MOSK documentation: Update a MOSK cluster.

Caution

During a baremetal-based cluster update, the false positive CalicoDataplaneFailuresHigh alert may be firing. Disregard this alert, which will disappear once cluster update succeeds.

The observed behavior is typical for calico-node during upgrades, as workload changes occur frequently. Consequently, there is a possibility of temporary desynchronization in the Calico dataplane. This can occasionally result in throttling when applying workload changes to the Calico dataplane.

To update a managed cluster:

  1. Log in to the Container Cloud web UI with the m:kaas:namespace@operator or m:kaas:namespace@writer permissions.

  2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

  3. Optional. Configure the update sequence of cluster machines:

  4. In the Clusters tab, select from the following options:

    • Available since Container Cloud 2.23.0. Click Upgrade next to the More action icon located in the last column for each cluster where available.

      Note

      If Upgrade is greyed out, the cluster is in maintenance mode that must be disabled before you can proceed with cluster update. For details, see Disable maintenance mode on a cluster and machine.

      If Upgrade does not display, your cluster is up-to-date.

    • Click the More action icon in the last column for each cluster and select Upgrade cluster where available.

  5. In the Release update window, select the required Cluster release to update your managed cluster to.

    The Description section contains the list of components versions to be installed with a new Cluster release. The release notes for each Container Cloud and Cluster release are available at Container Cloud releases and Cluster releases (managed).

  6. Click Update.

    Before the cluster update starts, Container Cloud performs a backup of MKE and Docker Swarm. The backup directory is located under:

    • /srv/backup/swarm on every Container Cloud node for Docker Swarm

    • /srv/backup/ucp on one of the controller nodes for MKE

    To monitor the cluster readiness, hover over the status icon of a specific cluster in the Status column of the Clusters page.

    Once the orange blinking status icon becomes green and Ready, the cluster deployment or update is complete.

    You can monitor live deployment status of the following cluster components:

    Component

    Description

    Bastion

    For the OpenStack-based management clusters, the Bastion node IP address status that confirms the Bastion node creation

    Helm

    Installation or upgrade status of all Helm releases

    Kubelet

    Readiness of the node in a Kubernetes cluster, as reported by kubelet

    Kubernetes

    Readiness of all requested Kubernetes objects

    Nodes

    Equality of the requested nodes number in the cluster to the number of nodes having the Ready LCM status

    OIDC

    Readiness of the cluster OIDC configuration

    StackLight

    Health of all StackLight-related objects in a Kubernetes cluster

    Swarm

    Readiness of all nodes in a Docker Swarm cluster

    LoadBalancer

    Readiness of the Kubernetes API load balancer

    ProviderInstance

    Readiness of all machines in the underlying infrastructure (virtual or bare metal, depending on the provider type)

    Graceful Reboot

    Readiness of a cluster during a scheduled graceful reboot, available since Cluster releases 15.0.1 and 14.0.0.

    Infrastructure Status

    Available since Container Cloud 2.25.0 for bare metal and OpenStack providers. Readiness of the following cluster components:

    • Bare metal: the MetalLBConfig object along with MetalLB and DHCP subnets.

    • OpenStack: cluster network, routers, load balancers, and Bastion along with their ports and floating IPs.

    LCM Operation

    Available since Container Cloud 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Health of all LCM operations on the cluster and its machines.

    For the history of a cluster deployment or update, refer to Inspect the history of a cluster and machine deployment or update.

  7. Available since Container Cloud 2.22.0 for baremetal-based clusters and since 2.24.2 for MOSK 23.2. In the Clusters tab, verify whether the required cluster has the One or more machines require a reboot warning icon. If so, reboot the corresponding hosts manually to apply the Ubuntu operating system updates.

    To identify the hosts to reboot:

    1. In the Clusters tab, click the required cluster name. The page with Machines opens.

    2. Hover over the status of every machine. A machine to reboot contains the Reboot > The machine requires a reboot notification in the Status tooltip.

Caution

During cluster update to the Cluster release 11.6.0 or 12.7.0 with StackLight logging enabled, a short outage of OpenSearch and its dependent components may occur with the following alerts firing on the cluster. This behavior is expected. Therefore, disregard these alerts.

StackLight alerts list firing during cluster update

Cluster size and outage probability level

Alert name

Label name and component

Any cluster with high probability

KubeStatefulSetOutage

statefulset=opensearch-master

KubeDeploymentOutage

  • deployment=opensearch-dashboards

  • deployment=metricbeat

Large cluster with average probability

KubePodsNotReady Removed in 17.0.0, 16.0.0, and 14.1.0

  • created_by_name="opensearch-master*"

  • created_by_name="opensearch-dashboards*"

  • created_by_name="metricbeat-*"

OpenSearchClusterStatusWarning

n/a

OpenSearchNumberOfPendingTasks

n/a

OpenSearchNumberOfInitializingShards

n/a

OpenSearchNumberOfUnassignedShards

n/a

Any cluster with low probability

KubeStatefulSetReplicasMismatch

statefulset=opensearch-master

KubeDeploymentReplicasMismatch

  • deployment=opensearch-dashboards

  • deployment=metricbeat

Note

MKE and Kubernetes API may return short-term 50x errors during the upgrade process. Ignore these errors.

Caution

If the cluster update includes MKE upgrade from 3.4 to 3.5 and you need to access the cluster while the update is in progress, use the admin kubeconfig instead of the existing one while OIDC settings are being reconfigured.

To obtain the admin kubeconfig:

kubectl --kubeconfig <pathToMgmtKubeconfig> get secret -n <affectedClusterNamespace> \
-o yaml <affectedClusterName>-kubeconfig | awk '/admin.conf/ {print $2}' | \
head -1 | base64 -d > clusterKubeconfig.yaml

See also

Reference Architecture: Upgrade workflow of a bare metal host operating system