Update notes

This section describes the specific actions you as a Cloud Operator need to complete to accurately plan and successfully perform your MOSK cluster update to version 26.2. This information supplements the generic update procedure published in Cluster update.

Cluster update schema

You can update to the 26.2 version from the following cluster versions (major update):

26.1 (2026-03-10)
26.1.2 (2026-08-03)

For a visual representation of supported update paths, refer to MOSK cluster update schema.

Update impact and maintenance windows planning

The following table provides details on the update impact on a MOSK cluster.

Impact during update to MOSK 26.2
Updated component	Impact on cloud users	Impact on cloud workloads
OpenStack and OpenSDN	~1% of read operations on cloud API resources may fail ~8% of create and update operations on cloud API resources may fail	Open vSwitch networking - interruption of the North-South connectivity, depending on the type of virtual routers used by a workload: Distributed (DVR) routers - no interruption Non-distributed routers, High Availability (HA) mode - interruption up to 1 minute, usually less than 5 seconds Non-distributed routers, non-HA mode - interruption up to 10 minutes OpenSDN networking - no impact
Ceph	~1% of read operations on object storage API may fail	IO performance degradation for Ceph-backed virtual storage devices. During the Ceph chart update that occurs during the Cluster release update, do not create or update Ceph VMs and PVCs. Otherwise, you may have issues during the update from Squid to Tentacle, which includes extraction of the Ceph CSI Operator from the Rook controller into a separate component that manages Ceph CSI driver resources. Therefore, the Ceph cluster must remain in read-only mode during the Cluster release update.
Migration to k0s (management cluster)	See MKE to k0s migration impact for management clusters below for details.	MOSK clusters are not affected and keep running MKE. Therefore, this migration has no impact on the cloud users and workloads of your MOSK clusters.
Host OS components	No expected performance impact	Instance network connectivity interruption up to 5 minutes
Host OS kernel	No expected performance impact	Restart of instances due to the hypervisor reboot [0]

MKE to k0s migration impact for management clusters

During the update to MOSK management 2.32.0, the Kubernetes underlay of the management cluster is migrated from Mirantis Kubernetes Engine (MKE) to k0s, and the Kubernetes version changes from 1.30 to 1.33. The migration is performed automatically as part of the regular management cluster update. No manual steps are required to start or drive it.

Important

The migration is a one-way operation. Once it is initiated, the management cluster cannot be rolled back to MKE.

The following table outlines the impact on the management cluster itself.

Impact during migration of the management cluster to k0s
Aspect	Impact
Kubernetes controller manager downtime	For 30-50 minutes, starting from the drain of the first control plane node and until k0s starts on that node, the Kubernetes controller manager of the management cluster is not running. During this window, the pods of the management cluster are not rescheduled, deployments do not maintain the replica count or perform rolling updates, new PersistentVolumeClaims do not bind, and CronJobs do not trigger. The MOSK management console, Keycloak, and StackLight remain available as long as their pods keep running.
Cluster networking	The conversion of the Calico datastore from etcd to KDD freezes cluster networking for about 20 seconds per node. During this window, new or replacement pods cannot start, while existing pods retain full network connectivity.
MKE-specific interfaces	The MKE web UI, the MKE API, and the MKE client bundle of the management cluster become permanently unavailable.

For the detailed migration flow, see Migration from MKE to k0s.

Known issues during the update

Before updating the cluster, review the potential issues that may arise during the process and the recommended solutions to address them, as outlined in Known issues: Cluster update.

StackLight alerts firing during the management cluster update

During the update of a management cluster from 2.31.6 to 2.32.0, the management cluster control plane is migrated from MKE to k0s, which involves a temporary Kubernetes API and control plane node disruption. As a result, several StackLight alerts may be triggered. This behavior is expected and does not indicate an issue with the update.

KubeAPI* alerts

During migration of the Kubernetes API from MKE to k0s, the Kubernetes API may be temporarily unavailable, which can trigger the following alerts:

KubeAPIOutage
KubeAPIDown
KubeAPIErrorsHighMajor
KubeAPIErrorsHighWarning

This behavior is expected. If they persist after update has completed, investigate the underlying cause.

NodeDown alert

During migration from MKE to k0s, control plane nodes may become temporarily unavailable, so the NodeDown alert may fire for those nodes.

This is expected during update and clears once the nodes are back. If it persists after the update has completed, investigate the underlying cause.

Uninhibited node target-down alerts on the first control plane node

During migration from MKE to k0s, the first control plane node, for example, master-0, may fail to fire NodeDown in time. As a result, the per-node (DaemonSet-based) target-down alerts for that node are not inhibited and may fire, for example:

cAdvisorTargetDown
CalicoTargetDown
CnncAgentDown
FluentdTargetDown
KubeletTargetDown
NodeExporterTargetDown
TelegrafSMARTTargetDown

These alerts are expected to fire for the duration of that node disruption and clear once the node is back and its targets are scraped again. If they persist after the node has recovered, investigate the underlying cause.

KubePodsCrashLooping for the metallb-speaker DaemonSet

During the control plane node disruption as part of the update, the KubePodsCrashLooping alert may fire for the mcc-metallb-speaker DaemonSet in the metallb-system namespace.

The speaker pods restart while the nodes are disrupted. This is expected during update and clears once the nodes are back. If it persists after the update has completed, investigate the underlying cause.

SystemTxPacketsDroppedTooHigh on the Calico VXLAN interface

During node disruption as part of the update, the SystemTxPacketsDroppedTooHigh alert may fire for the vxlan.calico interface.

Transient transmit packet drops occur on the Calico VXLAN overlay interface while the nodes are disrupted. This is expected during update and clears once the nodes are back. If it persists after the update has completed, investigate the underlying cause.

Pre-update actions

Audit the MKE configuration of the management cluster

The MKE configuration in the TOML format does not map one-to-one to the k0s configuration. The options that MOSK manages by default are reapplied automatically during the migration. However, the MKE settings that were applied directly through the MKE API are not carried over, and some of them have no k0s counterpart at all.

Before the update, collect the current MKE configuration of the management cluster and identify the settings that are not part of MOSK defaults:

Obtain the MKE configuration as described in MKE configuration managed directly by the MKE API.
Compare it with the list of MOSK-managed parameters described in MKE configuration managed by MOSK.
For every remaining custom setting, identify the k0s counterpart using k0s configuration management. Most Kubernetes component flags can be reapplied through the api, controllerManager, scheduler, etcd, and workerProfile parameters of the Cluster object.

Keep the collected configuration for the post-update actions.

Caution

The MKE API is not available after the migration, so this audit must be performed before the update.

Correct IAM configuration of baremetal-provider

Applies only to clusters deployed using 2.13.x or earlier

Management clusters deployed using an old Container Cloud release, earlier than 2.14.0, may contain an override of the IAM configuration of the baremetal-provider Helm release in the Cluster object with iam.enabled: false. This override is invalid and incompatible with the update to MOSK management 2.32.0. If the cluster is affected, correct the configuration and wait until the cluster becomes ready before starting the update. For the procedure, see Known issues: Management cluster update gets stuck with k0s failing to start.

Remove legacy-versioned OpenSearch indices

As part of this update, StackLight upgrades OpenSearch from major version 2.x to 3.x. The upgrade may get stuck if the cluster contains indices with a pre-2.0 index.version.created value, typically old system or history indices that have not been touched in four or more years.

Before starting the update, verify whether the cluster is affected. If so, remove the affected indices. Perform the following steps separately for every management and MOSK cluster before starting its update:

Exec into the opensearch-master-0 pod:

kubectl exec -it opensearch-master-0 -n stacklight -- bash

List the indices that will affect the OpenSearch 3.x startup:

curl -s "http://localhost:9200/_all/_settings?expand_wildcards=all&filter_path=*.settings.index.version.created&pretty" | jq -r 'to_entries[] | select(.value.settings.index.version.created | tonumber < 100000000) | "\(.key): \(.value.settings.index.version.created)"'

If the output is empty, the cluster is not affected and no further action is required.

Before removing any listed index, verify whether it is safe to delete. The following internal and system indices are always safe to remove:
- .async-search
- ilm-history-3-*
- .reporting-*
- .opendistro-reports-instances
- .opensearch-observability
All other indices must be assessed on a case-by-case basis, as they may hold data created by external integrations rather than by StackLight.

Every .kibana_* index is safe to remove except the one that the .kibana alias currently points to. Verify the alias target before removing any .kibana_* index:
```
curl -s "http://localhost:9200/_alias/.kibana"
```

Remove every index confirmed as safe to delete:

curl -X DELETE "http://localhost:9200/<indexName>"

Repeat step 2 to verify that no affected indices remain.

If the update has already stalled on the described error, recreate the stuck replica as described in Known issues: [65911] OpenSearch 3.x rolling upgrade fails on legacy-versioned indices.

Upgrade Ubuntu to 24.04

Ensure that all your MOSK clusters are running Ubuntu 24.04 to unblock the update of management clusters to 2.32.0.

MOSK 26.1 release series was the last one to support Ubuntu 22.04 as the host operating system. The Cluster release update of the Ubuntu 22.04-based MOSK clusters is no longer possible starting with MOSK management 2.32.0, where Ubuntu 24.04 is the only supported version.

A host operating system upgrade requires reboot of the servers and can be performed in small batches. For the detailed Ubuntu upgrade procedure, refer to Upgrade an operating system distribution.

Caution

Usage of third-party software, which is not part of Mirantis-supported configurations, for example, the use of custom DPDK modules, may block upgrade of an operating system distribution. Users are fully responsible for ensuring the compatibility of such custom components with the latest supported Ubuntu version.

Verify MetalLB address pool capacity

MOSK 26.2 introduces Application Gateway as a replacement for the NGINX Ingress service, and Application Gateway requires an additional LoadBalancer IP address in your cluster.

Verify that at least one IP address is available in the MetalLB address pool. To identify the available IP addresses in the MetalLB address pool for a MOSK cluster:

On the management cluster, obtain the IP address pool from the MetalLBConfig object in your MOSK cluster namespace:

kubectl -n <MOSK_CLUSTER_NAMESPACE> get metallbconfig <CONFIG_NAME> -o jsonpath='{.spec.ipAddressPools}'

Example of system response:

[{"name":"default","spec":{"addresses":["172.16.56.151-172.16.56.250"],"autoAssign":true,"avoidBuggyIPs":false}}]

On the MOSK cluster, obtain the number of IP addresses used by LoadBalancer services:
```
kubectl get services -A --output=jsonpath='{.items[?(@.spec.type=="LoadBalancer")].status.loadBalancer.ingress[*].ip}' | tr ' \t' '\n' | uniq | wc -l
```
Example of system response:
```
7
```
In this example, the IP address pool contains 99 addresses and 7 are in use, leaving 92 available.

Post-update actions

Reapply the custom Kubernetes underlay configuration

Using the MKE configuration collected during the pre-update audit, reapply the required custom settings to the Cluster object of the management cluster. The Kubernetes component flags are set through the api, controllerManager, scheduler, and etcd parameters, the kubelet settings through workerProfile, and the Kubernetes feature gates through featureGates. For example:

spec:
  ...
  providerSpec:
    value:
      ...
      api:
        extraArgs:
          event-ttl: 1440h
      workerProfile:
        config:
          imageGCHighThresholdPercent: 80

Warning

Any change to these parameters restarts the Kubernetes control plane on the control plane nodes sequentially, with a cordon and drain of one node at a time. Therefore, apply all the required settings in a single change.

For reference, see Reference Architecture: k0s configuration management and API Reference: Cluster resource specification.

Close the MKE-specific firewall ports on the management cluster nodes

After the migration of the management cluster to k0s, the MKE-specific ports are no longer used by any cluster component. For this reason, you can optionally close TCP 2376, 2377, 7946, 12376, 12379-12387, and UDP 7946 on the management cluster nodes. All other ports of the management cluster remain required, including TCP 179, 10250, 12388 and UDP 4792. For details, see MOSK management.

Switch to setting Ceph RGW DNS names using the ObjectStore spec definition

Note

Skip this action if your MOSK cluster has no customizations for TLS and DNS settings.

If your MOSK cluster has a non-default ingressConfig.tlsConfig.publicDomain configuration or the ingressConfig.tlsConfig.hostname parameter specified, switch to setting RGW DNS names using the ObjectStore spec definition. This requirement is part of the NGINX Ingress deprecation and migration to Application Gateway.

Configure the spec.objectStorage section of CephDeployment as follows:

spec:
  objectStorage:
    gatewayHTTPRoutes:
    - name: <name-of-the-gateway-http-route> # name of the Gateway HTTP route
      spec:
        hostnames:
        - <ingressConfig.tlsConfig.hostname>.<ingressConfig.tlsConfig.publicDomain> # custom TLS hostname and public domain values
    objectStores:
    - name: <object-store-name> # name of the ObjectStore (Ceph RGW) that migrated from the deprecated objectStorage.rgw
      spec:
        hosting:
          dnsNames:
          - <ingressConfig.tlsConfig.hostname>.<ingressConfig.tlsConfig.publicDomain> # custom TLS hostname and public domain values

Remove the existing StorageClass for RGW instances named rgw-storage-class

Optional

With the introduction of support for multiple ObjectStore (Ceph RGW) instances, the default StorageClass name for newly created buckets has changed to <object-store-name>-bucket. The previously created StorageClass for a single ObjectStore instance named rgw-storage-class remains for backward compatibility. You can optionally remove the existing StorageClass for RGW instances named rgw-storage-class if no ObjectBucketClaims exist on your cluster with a reference to the old storage class:

kubectl delete storageclass rgw-storage-class

All existing buckets are not affected, but for newly created ones, use StorageClass with the new name.

Migrate NGINX Ingress to Application Gateway

Migrate your OpenStack, Ceph, and OpenSDN public endpoints from NGINX Ingress to Application Gateway. For detailed instructions, refer to Migrate public endpoints to Envoy Proxy.