Automatic upgrade workflow

A management cluster upgrade to a newer version is performed automatically once a new Container Cloud version is released. Regional clusters also upgrade automatically along with the management cluster. For more details about the Container Cloud release upgrade mechanism, see: Release Controller.

The Operator can delay the Container Cloud automatic upgrade procedure for a limited amount of time or schedule upgrade to run at desired hours or weekdays. For details, see Schedule Mirantis Container Cloud upgrades.

Container Cloud remains operational during the management and regional clusters upgrade. Managed clusters are not affected during this upgrade. For the list of components that are updated during the Container Cloud upgrade, see the Components versions section of the corresponding Container Cloud release in Release Notes.

When Mirantis announces support of the newest versions of Mirantis Container Runtime (MCR) and Mirantis Kubernetes Engine (MKE), Container Cloud automatically upgrades these components as well. For the maintenance window best practices before upgrade of these components, see MKE Documentation.

Since Container Cloud 2.23.2, the release update train includes patch release updates being delivered between major releases. For details on the currently available patch releases, see Patch releases.

Note

MKE and Kubernetes API may return short-term 50x errors during the upgrade process. Ignore these errors.

Caution

If the cluster update includes MKE upgrade from 3.4 to 3.5 and you need to access the cluster while the update is in progress, use the admin kubeconfig instead of the existing one while OIDC settings are being reconfigured.

To obtain the admin kubeconfig:

kubectl --kubeconfig <pathToMgmtKubeconfig> get secret -n <affectedClusterNamespace> \
-o yaml <affectedClusterName>-kubeconfig | awk '/admin.conf/ {print $2}' | \
head -1 | base64 -d > clusterKubeconfig.yaml

If the related cluster is regional, replace <pathToMgmtKubeconfig> with <pathToRegionalKubeconfig>.

Caution

During cluster upgrade from the release 2.21.1 to 2.22.0 with StackLight logging enabled, a short outage of OpenSearch and its dependent components may occur with the following alerts firing on the cluster. This behavior is expected. Therefore, disregard these alerts.

StackLight alerts list firing during cluster update

Cluster size and outage probability level

Alert name

Label name and component

Any cluster with high probability

KubeStatefulSetOutage

statefulset=opensearch-master

KubeDeploymentOutage

  • deployment=opensearch-dashboards

  • deployment=metricbeat

Large cluster with average probability

KubePodsNotReady Removed in 17.0.0, 16.0.0, and 14.1.0

  • created_by_name="opensearch-master*"

  • created_by_name="opensearch-dashboards*"

  • created_by_name="metricbeat-*"

OpenSearchClusterStatusWarning

n/a

OpenSearchNumberOfPendingTasks

n/a

OpenSearchNumberOfInitializingShards

n/a

OpenSearchNumberOfUnassignedShards

n/a

Any cluster with low probability

KubeStatefulSetReplicasMismatch

statefulset=opensearch-master

KubeDeploymentReplicasMismatch

  • deployment=opensearch-dashboards

  • deployment=metricbeat

To inspect the cluster upgrade progress or history, refer to Inspect the history of a cluster and machine deployment or update.

Once management and regional clusters are upgraded to the latest version, update the original bootstrap tarball for successful clusters management, such as collecting logs, deploying new regional clusters, and so on.

To update the bootstrap tarball after an automatic cluster upgrade:

Select from the following options:

  • For clusters deployed using Container Cloud 2.11.0 or later:

    ./container-cloud bootstrap download --management-kubeconfig <pathToMgmtKubeconfig> \
    --target-dir <pathToBootstrapDirectory>
    
  • For clusters deployed using the Container Cloud release earlier than 2.11.0 or if you deleted the kaas-bootstrap folder, download and run the Container Cloud bootstrap script:

    wget https://binary.mirantis.com/releases/get_container_cloud.sh
    
    chmod 0755 get_container_cloud.sh
    
    ./get_container_cloud.sh