Automatic upgrade workflow¶

A management cluster upgrade to a newer version is performed automatically once a new Container Cloud version is released. For more details about the Container Cloud release upgrade mechanism, see: Release Controller.

The Operator can delay the Container Cloud automatic upgrade procedure for a limited amount of time or schedule upgrade to run at desired hours or weekdays. For details, see Schedule Mirantis Container Cloud upgrades.

Container Cloud remains operational during the management cluster upgrade. Managed clusters are not affected during this upgrade. For the list of components that are updated during the Container Cloud upgrade, see the Components versions section of the corresponding Container Cloud release in Release Notes.

When Mirantis announces support of the newest versions of Mirantis Container Runtime (MCR) and Mirantis Kubernetes Engine (MKE), Container Cloud automatically upgrades these components as well. For the maintenance window best practices before upgrade of these components, see MKE Documentation.

Since Container Cloud 2.23.2, the release update train includes patch release updates being delivered between major releases. For details on the currently available patch releases, see Patch releases.

Note

MKE and Kubernetes API may return short-term 50x errors during the upgrade process. Ignore these errors.

Caution

If the cluster update includes MKE upgrade from 3.4 to 3.5 and you need to access the cluster while the update is in progress, use the admin kubeconfig instead of the existing one while OIDC settings are being reconfigured.

To obtain the admin kubeconfig:

kubectl --kubeconfig <pathToMgmtKubeconfig> get secret -n <affectedClusterNamespace> \
-o yaml <affectedClusterName>-kubeconfig | awk '/admin.conf/ {print $2}' | \
head -1 | base64 -d > clusterKubeconfig.yaml

Caution

During cluster upgrade from the release 2.21.1 to 2.22.0 with StackLight logging enabled, a short outage of OpenSearch and its dependent components may occur with the following alerts firing on the cluster. This behavior is expected. Therefore, disregard these alerts.

StackLight alerts list firing during cluster update

Cluster size and outage probability level	Alert name	Label name and component
Any cluster with high probability	`KubeStatefulSetOutage`	`statefulset=opensearch-master`
	`KubeDeploymentOutage`	`deployment=opensearch-dashboards` `deployment=metricbeat`
Large cluster with average probability	`KubePodsNotReady` ^{Removed in 17.0.0, 16.0.0, and 14.1.0}	`created_by_name="opensearch-master"` `created_by_name="opensearch-dashboards"` `created_by_name="metricbeat-*"`
	`OpenSearchClusterStatusWarning`	n/a
	`OpenSearchNumberOfPendingTasks`	n/a
	`OpenSearchNumberOfInitializingShards`	n/a
	`OpenSearchNumberOfUnassignedShards` ^{Removed in 2.27.0 (17.2.0 and 16.2.0)}	n/a
Any cluster with low probability	`KubeStatefulSetReplicasMismatch`	`statefulset=opensearch-master`
	`KubeDeploymentReplicasMismatch`	`deployment=opensearch-dashboards` `deployment=metricbeat`

To inspect the cluster upgrade progress or history, refer to Inspect the history of a cluster and machine deployment or update.

Once the management cluster is upgraded to the latest version, update the original bootstrap tarball for successful cluster management, such as collecting logs and so on.

To update the bootstrap tarball after an automatic cluster upgrade:

Select from the following options:

For clusters deployed using Container Cloud 2.11.0 or later:

./container-cloud bootstrap download --management-kubeconfig <pathToMgmtKubeconfig> \
--target-dir <pathToBootstrapDirectory>

For clusters deployed using the Container Cloud release earlier than 2.11.0 or if you deleted the kaas-bootstrap folder, download and run the Container Cloud bootstrap script:
```
wget https://binary.mirantis.com/releases/get_container_cloud.sh

chmod 0755 get_container_cloud.sh

./get_container_cloud.sh
```