Known issues

This section describes the MOSK known issues with available workarounds in this release. For guidance on troubleshooting common issues with deploying and operating MOSK, refer to Troubleshooting Guide.

This section may also include known issues carried over from previous releases and still applicable to this release.

Bare metal

[62999] Master node replacement fails on MOSK clusters with BGP announcement enabled

On MOSK clusters with BGP announcement enabled (useBGPAnnouncement: true in the Cluster object), master nodes are deployed one per rack, and the Kubernetes API Virtual IP (VIP), for example, 10.0.30.100/32 is assigned to the loopback (lo) interface of each master node. For details on BGP announcement, see Configure BGP announcement for cluster API LB address.

During master node replacement, lcm-agent on the new node tries to reach the API VIP on the lo interface for the first time but cannot connect to it. This happens because cloud-init binds the API VIP to the lo interface before the local containerized API proxy starts listening on it, so the connection is refused. The lcm-agent cannot report back to the management cluster. As a result, the Machine object remains stuck in the PendingLCMAgent state.

As a workaround, temporarily remove the API VIP from the lo interface to force API traffic through the physical network to a healthy master node. Once the node finishes provisioning, the local VIP handling takes over automatically.

Workaround:

From the management cluster, obtain the IP address of the new node from its IpamHost object:

kubectl get ipamhost <NODE_NAME> -o jsonpath="{.status.serviceMap['ipam/SVC-k8s-lcm'][0].ipAddress}"

SSH to the new node:

ssh -i <path-to-ssh-key> mcc-user@<NEW_NODE_IP>

Wait for cloud-init to complete on the new node:
```
sudo cloud-init status --wait
```
Before proceeding, ensure that the output reports status: done.

Warning

Do not proceed with networking modifications until cloud-init has finished executing, so it does not overwrite your changes.

Verify that the API VIP is bound to the lo interface:

ip addr show dev lo

Example of system response on the affected node:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
    inet 10.0.30.100/32 scope global lo

Confirm that lcm-agent is failing with a connection refused error:
```
sudo journalctl -u lcm-agent* -n 20 --no-pager
```
Remove the /32 API VIP from the lo interface:
```
sudo ip addr del 10.0.30.100/32 dev lo
```
Substitute 10.0.30.100 with the actual API VIP of your cluster.
Verify outbound network connectivity to the API. It should now route across the network to one of the healthy master nodes:
```
curl -k -v https://10.0.30.100:443/api
```
An HTTP response such as HTTP/2 401 or 403 confirms that the traffic is reaching an active Kubernetes API server. For example:
```
* Connected to 10.0.30.100 (10.0.30.100) port 443
< HTTP/2 401
{
  "kind": "Status",
  "message": "Unauthorized",
  "code": 401
}
```
Reload systemd to clear the warnings about unit files that have changed on disk:
```
sudo systemctl daemon-reload
```
Restart the lcm-agent daemon to trigger an immediate check-in:
```
sudo systemctl restart lcm-agent*.service
```
Verify that the node recovers:
1. On the new node, inspect the agent logs to confirm a successful tick execution:
```
sudo journalctl -u lcm-agent* -f
```
  In the system response, find the tick finished messages without connection errors.
2. From the management or seed node, verify the machine phase transition:
```
kubectl get machine,lcmmachine -o wide
```
  The LCMPHASE for the new node should transition from PendingLCMAgent to Prepare, Deploy, and then to Ready.

[42889] Graceful reboot gets stuck when Kubernetes and OpenStack control planes are drained simultaneously

When a GracefulRebootRequest targets both the Kubernetes and OpenStack control plane machines, either by listing machines of both types in spec.machines or by leaving the list empty to reboot all cluster nodes, the rolling reboot may get stuck. This happens because both node groups are drained in parallel, and the OpenStack workload manager running on the Kubernetes control plane becomes unavailable while the OpenStack control plane nodes are simultaneously being drained.

Workaround:

Identify the machines that have not yet been rebooted.

Delete the stuck GracefulRebootRequest:

kubectl -n <projectName> delete gracefulrebootrequest <gracefulRebootRequestName>

Recreate the reboot requests in two sequential steps as described in Perform a rolling reboot of a cluster using CLI: first for the Kubernetes control plane machines, then, once that request completes and is deleted, for the remaining machines that still require a reboot.

Ceph

[62392] False negatives in the ceph_daemon_socket_up metric

When a pod with the ceph-rgw daemon restarts, the stale rgw socket may persist due to the upstream Ceph issue 76939, which results in a false-positive failed ceph_daemon_socket_up check in the ceph-exporter metrics. For example:

ceph_daemon_socket_up{ceph_daemon="ceph-osd.1", hostname="child-worker-1-storage-worker", instance="192.168.9.124:9926", job="prometheus-ceph-exporter"}     1
ceph_daemon_socket_up{ceph_daemon="ceph-client.rgw.openstack.store.a.1.107989349947776", hostname="managed-master-2-control-storage", instance="192.168.9.122:9926", job="prometheus-ceph-exporter"} 0
ceph_daemon_socket_up{ceph_daemon="ceph-osd.0", hostname="child-worker-2-storage-worker", instance="192.168.9.125:9926", job="prometheus-ceph-exporter"}     1
ceph_daemon_socket_up{ceph_daemon="ceph-client.rgw.openstack.store.a.1.95065910507904", hostname="managed-master-1-control-storage", instance="192.168.9.121:9926", job="prometheus-ceph-exporter"}  1
ceph_daemon_socket_up{ceph_daemon="ceph-client.rgw.openstack.store.a.1.104243065772416", hostname="managed-master-2-control-storage", instance="192.168.9.122:9926", job="prometheus-ceph-exporter"} 1

In the example above, ceph-exporter detects a failed socket with the 0 value for the ceph-client.rgw.openstack.store.a.1.107989349947776 daemon, which has already been replaced by the ceph-client.rgw.openstack.store.a.1.104243065772416 daemon. To mitigate the issue, the operator must manually remove the stale socket from the affected node.

Workaround:

SSH into a node where the affected daemon is placed. In the example above, it is managed-master-2-control-storage.

Locate the stale socket with the 0 value under the /var/lib/rook/exporter/ directory:

ls -l /var/lib/rook/exporter/

Example of system response:

total 0
srwxr-xr-x 1 root root 0 Jun 16 19:32 ceph-client.ceph-exporter.asok
srwxr-xr-x 1  167  167 0 Jun 16 19:22 ceph-client.rgw.openstack.store.a.1.104243065772416.asok
srwxr-xr-x 1  167  167 0 Jun 16 20:52 ceph-client.rgw.openstack.store.a.1.107989349947776.asok
srwxr-xr-x 1  167  167 0 Jun 16 19:29 ceph-mon.a.asok

Remove the stale socket. For example:

rm /var/lib/rook/exporter/ceph-client.rgw.openstack.store.a.1.107989349947776.asok

Cluster update

[65903] Management cluster update gets stuck with k0s failing to start

Applies only to clusters deployed using 2.13.x or earlier

Management clusters deployed using an old Container Cloud release, earlier than 2.14.0, may contain an override of the IAM configuration of the baremetal-provider Helm release in the Cluster object with iam.enabled: false. This override is invalid and incompatible with the update to MOSK management 2.32.0. It may cause the update to get stuck with the k0s control plane nodes failing to start with the following example errors:

err="open /etc/kubernetes/oidc_ca.crt: no such file or directory\"" component=kube-apiserver

msg="Failed to wait for process" component=kube-apiserver error="exit status 1"

To verify whether the cluster is affected:

Log in to a node that contains kubeconfig of the required management cluster. For example:
```
export KUBECONFIG=/path/to/mgmt-kubeconfig
```
Verify whether the baremetal-provider Helm release contains iam.enabled: false:
```
kubectl get cluster <mgmtClusterName> -o jsonpath='{.spec.providerSpec.value.kaas.regional[?(@.provider=="baremetal")].helmReleases[?(@.name=="baremetal-provider")].values.config.iam}'
```
Example output of the affected cluster:
```
{"enabled":false,"registrationUrl":"<ip>:8082"}
```
If the output is empty or does not contain "enabled":false, the cluster is not affected and you can skip this procedure. Otherwise, proceed to the next step.

Verify that HelmBundle of the management cluster has IAM enabled for the baremetal-provider release:

kubectl -n default get helmbundle <mgmtClusterName> -o jsonpath='{.spec.releases[?(@.name=="baremetal-provider")].values.config.iam}'

Example output of the affected cluster:

{"enabled":true,"registrationUrl":"<ip>:8082"}

Example of the affected HelmBundle snippet:

spec:
  releases:
  ...
  - name: baremetal-provider
    namespace: kaas
    values:
      ...
      config:
        ...
        iam:
          enabled: true
          registrationUrl: <ip>:8082

If the output is empty, does not contain iam.enabled, or contains "enabled":false, contact Mirantis support. Do not proceed to the next step.

Verify whether the Cluster object contains the Keycloak TLS configuration:

kubectl get cluster <mgmtClusterName> -o jsonpath='{.spec.providerSpec.value.tls.keycloak.tlsConfigRef}'

Example of the affected Cluster object:

spec:
  ...
  providerSpec:
    value:
      ...
      tls:
        ...
        keycloak:
          tlsConfigRef: keycloak

If the output is empty, contact Mirantis support. Do not proceed with the workaround below.

Workaround:

If the Keycloak TLS configuration is present and the HelmBundle has iam.enabled: true, remove enabled: false from the Cluster object:

Open the Cluster object of the affected management cluster for editing:
```
kubectl edit cluster <mgmtClusterName>
```

In the spec.providerSpec.value.kaas.regional section, find the baremetal-provider Helm release and delete only the enabled: false line under values.config.iam.

Caution

Do not remove registrationUrl or other IAM parameters.

spec:
  ...
  providerSpec:
    value:
      ...
      kaas:
        regional:
        - provider: baremetal
          helmReleases:
          - name: baremetal-provider
            values:
              ...
              config:
                iam:
                  enabled: false # delete this line
                  registrationUrl: <ip>:8082

Save and exit the object to apply changes.

Warning

After you remove the override, the management cluster nodes may be sequentially cordoned, drained, and updated with the new configuration. Do not start the update to MOSK management 2.32.0 until the cluster becomes ready.
Wait until every management cluster node completes the configuration update and the cluster becomes ready:
```
kubectl get cluster <mgmtClusterName> -o jsonpath='{.status.providerStatus.ready}{"\n"}'
```
The command must return true, which means the cluster is ready for the update to MOSK management 2.32.0.

[65106] Management cluster update hangs due to Keycloak database schema issue

A management cluster update may get stuck with the following error:

keycloak encountered an exception.: liquibase.exception.DatabaseException: (conn=538) \
Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. \
This includes storage overhead, check the manual. \
You have to change some columns to TEXT or BLOBs \
[Failed SQL: (1118) ALTER TABLE keycloak.CREDENTIAL ADD VERSION INT DEFAULT 0 NULL]

Workaround:

kubectl -n kaas exec -it mariadb-server-0 -- bash -c 'mysql -uroot -p$MYSQL_DBADMIN_PASSWORD'

Optimize the affected table:
```
OPTIMIZE TABLE keycloak.CREDENTIAL;
```

[64721] MOSK cluster upgrade gets stuck due to runc-ee package downgrade

Clusters deployed with MOSK 26.1 (Cluster release 21.1.0) starting from 2026-08-04 may have issues upgrading to more recent versions, such as MOSK 26.1.x (Cluster releases 21.1.x) patch releases or MOSK 26.2 (Cluster release 21.2.0). The issue occurs during lcm-ansible execution at the step that installs the runc-ee package.

Symptom: the first control plane node gets stuck in the deploy phase with the following error in the lcm-ansible logs:

TASK [containerd : Install containerd.io packages]
task path: .../lcm-ansible-<version>/roles/containerd/tasks/Debian.yml:62
The following packages will be upgraded:
  containerd.io* → 1.7.31m1+fips-0ubuntu0.24.04.1
The following packages will be DOWNGRADED:
  runc-ee → 1.4.2m1-0ubuntu0.24.04.1
E: Packages were downgraded and -y was used without --allow-downgrades.

The issue occurs because the runc-ee package version was not pinned and was installed from a constantly updated repository in MOSK 26.1 (Cluster release 21.1.0). Starting from MOSK 26.1.x (Cluster releases 21.1.x) patch releases and MOSK 26.2 (Cluster release 21.2.0), the package version is pinned.

Workaround:

SSH to any node of the affected cluster.

Install the pinned version of the runc-ee package:

apt-get --allow-downgrades install runc-ee=1.4.2m1-0ubuntu0.24.04.1 -y --allow-change-held-packages

Hold the package to prevent future automatic updates:
```
apt-mark hold runc-ee
```
Repeat steps 1-3 for the remaining nodes of the affected cluster.

[65911] OpenSearch 3.x rolling upgrade fails on legacy-versioned indices

During the update of a management cluster to 2.32.0 or a MOSK cluster to 26.2 (Cluster release 21.2.0), StackLight upgrades OpenSearch from major version 2.x to 3.x as part of a rolling restart of the opensearch-master StatefulSet. The rolling restart may get stuck if the cluster contains at least one index with a pre-2.0 index.version.created value, for example, an index that was originally created under Elasticsearch 7.10.2 and has not been reindexed since. Such indices are typically old system or history indices, for example, .async-search or ilm-history-3-*, that were created four or more years ago and have not been touched since.

Symptoms:

The first replica to receive the OpenSearch 3.x image, usually opensearch-master-2, fails to start and enters the CrashLoopBackOff state:

kubectl get pod -n stacklight -l app=opensearch-master
NAME                  READY   STATUS             RESTARTS      AGE
opensearch-master-0   1/1     Running            0             161m
opensearch-master-1   1/1     Running            0             157m
opensearch-master-2   0/1     CrashLoopBackOff   21 (13m ago)  99m

The opensearch-master-2 container logs contain the following error:

[opensearch-master-2] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Failed to parse value [7100299] for setting [index.version.created]

Before starting the update:

To avoid the issue, remove the affected indices before starting the update. Perform the following steps separately for every management and MOSK cluster before starting its update.

Exec into the opensearch-master-0 pod:

kubectl exec -it opensearch-master-0 -n stacklight -- bash

List the indices that will affect the OpenSearch 3.x startup:

curl -s "http://localhost:9200/_all/_settings?expand_wildcards=all&filter_path=*.settings.index.version.created&pretty" | jq -r 'to_entries[] | select(.value.settings.index.version.created | tonumber < 136217728) | "\(.key): \(.value.settings.index.version.created)"'

If the output is empty, the cluster is not affected and no further action is required.

Before removing any listed index, verify whether it is safe to delete. The following internal and system indices are always safe to remove:
- .async-search
- ilm-history-3-*
- .reporting-*
- .opendistro-reports-instances
- .opensearch-observability
All other indices must be assessed on a case-by-case basis, as they may hold data created by external integrations rather than by StackLight.

Every .kibana_* index is safe to remove except the one that the .kibana alias currently points to. Verify the alias target before removing any .kibana_* index:
```
curl -s "http://localhost:9200/_alias/.kibana"
```

Remove every index confirmed as safe to delete:

curl -X DELETE "http://localhost:9200/<indexName>"

Repeat step 2 to verify that no affected indices remain.

Workaround (if the update has already failed):

If the update has already stalled on the described error, recreate the stuck replica. Apply the following steps only if the OpenSearch cluster health is yellow. If the cluster health is red, even though 2 of 3 replicas are Running, do not apply this workaround and contact Mirantis support instead.

Exec into the opensearch-master-0 pod:

kubectl exec -it opensearch-master-0 -n stacklight -- bash

Verify the OpenSearch cluster health:
```
curl -s http://localhost:9200/_cluster/health | jq -r '.status'
```
Proceed only if the output is yellow.
Complete the steps of the Before starting the update procedure above for the affected cluster. This removes the affected indices from the local storage of the opensearch-master-0 and opensearch-master-1 replicas.
Delete the persistent volume claim and the pod of the stuck opensearch-master-2 replica:
```
kubectl -n stacklight delete persistentvolumeclaim/opensearch-master-opensearch-master-2 pod/opensearch-master-2
```
A new opensearch-master-2 pod starts with an empty volume, joins the cluster, and syncs the data from the two other replicas. Depending on the database size, the sync can take from several minutes to a few hours. While it is in progress, the pod remains Running but not Ready, which is expected. Once the sync completes, opensearch-master-2 becomes Ready and the remaining replicas are recreated one by one with the OpenSearch 3.x image.

MOSK management cluster

[65700] Management cluster gets stuck after a custom k0s certificate CA change

The issue applies only to management clusters that have custom k0s certificates configured as described in Workflow of custom k0s certificates configuration.

After you replace the CA certificate or change the host name of such a certificate, the cluster stops reconciling. The management cluster kubeconfig secret keeps the previous host name and CA certificate, and MOSK does not refresh it, so internal clients cannot verify the newly issued certificate.

Renewal of the certificate and private key alone, without changing the CA certificate or the host name, is not affected.

Symptoms:

The affected node stays cordoned and draining indefinitely, the LCM status of the corresponding machine remains in the Prepare phase with Machine evacuation pending: upgrade (current: pending-upgrade), while the NodeReady condition reported by kubelet stays True. The LCM agent logs contain an error similar to the following one:

Error: failed to check rook-ceph/csi-rbdplugin daemonset existence: \
Get "https://<previousHostName>/apis/apps/v1/namespaces/rook-ceph/daemonsets/csi-rbdplugin": \
tls: failed to verify certificate: x509: certificate is valid for \
<newHostName>, not <previousHostName>

The host name in the request is the one configured before the change, while the newly issued certificate is valid for the new host name only.

Workaround:

Once the admin.conf key is removed, the LCM agent recreates it with the current host name and CA certificate within a minute.

Obtain the name of the management cluster kubeconfig secret:

CLUSTER=<managementClusterName>
SECRET=$(kubectl -n default get lcmcluster "$CLUSTER" -o jsonpath='{.spec.kubeconfigSecretName}')

Back up the secret:

Warning

The backup contains a full administrative kubeconfig in clear text. Save it to a secure location and remove it once the cluster recovers.
```
kubectl -n default get secret "$SECRET" -o yaml > "${SECRET}.bak.yaml"
```

Remove the admin.conf key from the secret:

kubectl -n default patch secret "$SECRET" --type=json -p '[{"op":"remove","path":"/data/admin.conf"}]'

Verify that the cluster recovers:
1. Verify that the LCM agent has recreated the admin.conf key:
```
kubectl -n default get secret "$SECRET" -o jsonpath='{.data.admin\.conf}'
```
  The command must return a non-empty value within a minute.
2. Verify the machine phase transition:
```
kubectl -n default get machine,lcmmachine -o wide
```
  The LCMPHASE of the affected machine must leave Prepare and proceed to Ready, and the node must become uncordoned.
If the admin.conf key does not reappear or the machine phase does not change, contact Mirantis support.

[65108] Rotated Keycloak CA certificate is not applied by kube-apiserver

After rotation of a custom Keycloak certificate with a new CA, MOSK delivers the new CA certificate to every k0s control plane node of the management cluster, but the running kube-apiserver does not apply it. The Kubernetes API server reads its OIDC CA certificate only at startup, and MOSK does not restart k0s for this change.

The issue remains latent and then becomes permanent. The control plane keeps working on cached data until kube-apiserver restarts for any reason. From that moment, kube-apiserver cannot verify Keycloak and rejects every OIDC token with the HTTP 401 Unauthorized error, while the correct CA certificate remains unused on disk.

Workaround:

Note

For enabling and disabling maintenance mode on a cluster and machine, see Enable cluster and machine maintenance mode.

Obtain the required OIDC CA certificate that MOSK attempts to apply:

CLUSTER=<managementClusterName>
REF=$(kubectl get cluster "$CLUSTER" -o jsonpath='{.spec.providerSpec.value.tls.keycloak.tlsConfigRef}')
kubectl get tlsconfig "$REF" -o jsonpath='{.spec.caCertificate}' | base64 -d

Enable maintenance mode on the management cluster.
For each control plane node of the management cluster, one node at a time:
1. Verify that the on-disk CA certificate located at /etc/kubernetes/oidc_ca.crt matches the required OIDC CA certificate.
2. Enable maintenance mode on the corresponding machine.
3. Restart the k0s controller:
```
systemctl restart k0scontroller
```
4. Disable maintenance mode on the machine and wait until the node becomes Ready before proceeding to the next one.
Disable maintenance mode on the management cluster.

[64703] Missing WireGuard interface address on management cluster nodes

On new management clusters based on Cluster release 22.0.0 (MOSK 26.2) that have secureOverlay set to true, some management cluster nodes may be missing the WireGuard interface address. As a result, kube-apiserver or any other process running in the host network namespace on an affected node cannot reach pods running on healthy nodes.

Symptom: webhook calls handled by kube-apiserver on the affected node fail with a timeout if the pod serving the webhook runs on a healthy node. The error propagates back to the client that issued the original API request.

The following example illustrates a cert-manager log entry produced when the cert-manager pod sends a request to kube-apiserver on the affected node, while cert-manager-webhook runs on a healthy node:

E0817 23:38:37.501136       1 controller.go:167] cert-manager/issuers "msg"="re-queuing item due to error processing" "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.kaas.svc:443/mutate?timeout=10s\": context deadline exceeded" "key"="kaas/mcc-ca"

The following example illustrates the corresponding k0scontroller log entry on the affected node:

Aug 17 23:38:37 master-2 k0s[45673]: time="2026-08-17 23:38:37" level=info msg="W0817 23:38:37.496457   45763 dispatcher.go:225] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.kaas.svc:443/mutate?timeout=10s\": context deadline exceeded" component=kube-apiserver stream=stderr

Workaround:

Identify the affected nodes that have no IP address in the output of the following command:

kubectl get node -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.projectcalico\.org/IPv4WireguardInterfaceAddr}{"\n"}{end}'

Identify the names of the calico-node pods running on the affected nodes:
```
kubectl get pod -n kube-system -o wide | grep calico-node
```

Delete the affected pods to trigger their restart:

kubectl delete pod -n kube-system <calico-node-pod-name>

Repeat steps 1-3 until all nodes have the WireGuard interface address assigned.

[64454] Node gets stuck in the deploy phase

During bootstrap of a MOSK management cluster, a master node may get stuck in the deploy phase and never become Ready, even though the LCM deploy playbook has completed successfully.

Symptoms:

The lcm-agent logs on the affected nodes, including nodes that are not stuck, contain the following error:

{"level":"error","msg":"failed to check k0s manager health","error":"local etcd member health check failed: failed to get etcd member master-1: etcdmembers.etcd.k0sproject.io \"master-1\" not found"}

No etcdmember objects exist for any node, even though etcd itself is healthy:
```
kubectl get etcdmembers.etcd.k0sproject.io -A
```
Example of system response:
```
No resources found
```

Workaround:

Restart the k0scontroller service on one of the affected nodes:

systemctl restart k0scontroller

Wait for the node to become Ready and for its deployment to finish before proceeding to the next one. The remaining affected nodes will likely finish deployment successfully without requiring the same restart. Otherwise, restart the service on each remaining affected node as well.

[64407] MariaDB restore from backup causes Keycloak failure

During a full MariaDB unavailability caused by the restore procedure, Keycloak’s internal clustering mechanism cannot re-establish the cluster state, leaving the affected pods in the NOT READY condition even after MariaDB becomes available again.

Symptoms: the Keycloak pod logs contain the following errors:

ERROR [org.keycloak.jgroups.protocol.KEYCLOAK_JDBC_PING2] (Timer runner-2,iam-keycloak-0-4892(v=16.0.8)) iam-keycloak-0-4892(v=16.0.8): failed reading from the DB: java.sql.SQLNonTransientConnectionException: Socket fail to connect to mariadb. Connection refused
Caused by: java.net.ConnectException: Connection refused
    at org.mariadb.jdbc.client.impl.ConnectionHelper.connectSocket(ConnectionHelper.java:118)
    ... 12 more

INFO  [io.smallrye.health] (vert.x-eventloop-thread-0) SRHCK01001: Reporting health down status: {"status":"DOWN","checks":[{"name":"Graceful Shutdown","status":"UP"},{"name":"Keycloak database connections async health check","status":"UP"},{"name":"Keycloak Initialized","status":"UP"},{"name":"Keycloak cluster health check","status":"DOWN","data":{"Failing since":"2026-08-17 17:38:26,222"}}]}

Workaround:

Verify that the restore procedure completed successfully and that all MariaDB replicas are up and ready.

Identify the stuck Keycloak pods:

kubectl get pods -n kaas | grep iam-keycloak

Delete each stuck pod:

kubectl delete pod -n kaas <IAM_KEYCLOAK_POD_NAME>

Verify that the newly created Keycloak pods are up and in the READY state:
```
kubectl get pods -n kaas | grep iam-keycloak
```

MOSK management console

[50168] Inability to use a new project right after creation

A newly created project does not display all available tabs in the MOSK management console and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Networking

[64752] LoadBalancer IPs flap after a MetalLB speaker node shutdown

On clusters using MetalLB in L2 mode, an unexpected shutdown of a node running a MetalLB speaker can cause LoadBalancer service IP addresses to flap instead of failing over to a live node. The issue can occur on a management cluster or a MOSK cluster, on any node hosting a speaker, whether a control plane node or a worker node. The condition does not self-recover while the node remains down.

Affected services are those of type LoadBalancer that MetalLB advertises in L2 mode. For example, on the management cluster these include iam-keycloak-http, kaas-kaas-ui, and iam-proxy-*. As a result, access to Keycloak, the MOSK management console, and StackLight IAM proxies (Grafana, Prometheus, Alerta, Alertmanager, Kibana) becomes intermittently unavailable. On a MOSK cluster, the same failure mode affects the LoadBalancer services announced on the powered-off speaker node.

If the affected cluster is the management cluster, support dump collection may also fail because the systemLogs collector cannot complete:

failed to collect system logs: failed waiting for calls to execute

Symptoms:

After a hard power-off of a node running a MetalLB speaker, the external IP addresses of LoadBalancer services become intermittently reachable. Reachability can change every few seconds. For example, TCP probes to <ip>:443 from a live node, taken a few seconds apart, may look as follows:

Service              IP             Round 0   Round 1   Round 2
iam-keycloak-http    172.16.61.24   OPEN      CLOSED    OPEN
kaas-kaas-ui         172.16.61.11   CLOSED    CLOSED    OPEN
iam-proxy-alerta     172.16.61.13   CLOSED    OPEN      OPEN
iam-proxy-grafana    172.16.61.15   OPEN      CLOSED    OPEN
iam-proxy-prometheus 172.16.61.17   CLOSED    CLOSED    CLOSED

The MetalLB speaker pods on the surviving nodes log a memberlist join failure approximately every 60 seconds:

{"caller":"speakerlist.go:269","op":"memberDiscovery","level":"error",
 "msg":"partial join","expected":1,"joined":0,
 "error":"Failed to join 172.16.61.26:7947: dial tcp 172.16.61.26:7947: connect: no route to host"}

The ServiceL2Status objects still allocate LoadBalancer IP addresses to the powered-off node (kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587 in the example below):

kubectl -n metallb-system get servicel2status

Example of system response:

NAME       ALLOCATED NODE                                   SERVICE NAME                SERVICE NAMESPACE
l2-fwbbn   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   ironic-kaas-bm              kaas
l2-kx245   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   ironic-syslog               kaas
l2-m74nf   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   iam-proxy-prometheus        stacklight
l2-ndsz9   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   httpd-http                  kaas
l2-x6tw2   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   dhcp-lb                     kaas
l2-ztpjw   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   iam-proxy-grafana           stacklight

Service backends remain healthy: externalTrafficPolicy is Cluster, endpoints contain ready pods on the live nodes, and direct connections to backend pod IPs succeed. This rules out a backend failure.

The MetalLB speaker pod on the powered-off node remains Running (1/1 Ready). Pod status is push-based: when the node is powered off, kubelet dies and stops reporting container state, so the API server keeps the last known status. The node is tainted with node.kubernetes.io/unreachable:NoExecute, but the speaker DaemonSet tolerates that taint indefinitely and Kubernetes never evicts the pod.

MetalLB determines L2 IP ownership from speaker pods that Kubernetes still reports as Ready. Because the frozen pod still appears healthy, MetalLB treats the dead node as an eligible L2 speaker. Speaker membership never converges, L2 IP ownership keeps being re-elected, and gratuitous ARP announcements flap.

Workaround:

Force-delete the frozen MetalLB speaker pod on the powered-off node:

Using the kubeconfig of the affected cluster, identify the speaker pod on the powered-off node:
```
kubectl -n metallb-system get pod -o wide | grep speaker
```
The pod on the unreachable node remains Running while the node is NotReady.
Force-delete that speaker pod:

Note

A graceful deletion cannot complete because kubelet on the powered-off node is not running.
```
kubectl -n metallb-system delete pod <speakerPodName> --force --grace-period=0
```
Kubernetes immediately removes the frozen pod object. The DaemonSet creates a replacement pod that cannot become Ready on the dead node, so MetalLB stops considering that node for L2 IP ownership.
Verify that ServiceL2Status objects no longer list the powered-off node as the allocated node and that the LoadBalancer IP addresses stop flapping:
```
kubectl -n metallb-system get servicel2status
```

OpenSDN

[65652] DNS records are removed after instance migration

DNS records of an instance may be removed after the instance is migrated to another compute node. Both live migration and cold migration followed by openstack server migrate confirm are affected.

During migration, the vRouter agent on the destination compute node publishes the internal and external A records of the instance. About 50 seconds later, the cleanup from the source compute node deletes them again: the ownership check in the OpenSDN DNS service still identifies the source node as the owner of the records because the instance keeps its virtual machine interface across the migration.

As a result, the instance remains ACTIVE and reachable by its fixed and floating IP addresses, while DNS queries for its FQDN return NXDOMAIN. All OpenSDN control nodes perform the same add-then-delete sequence, for example:

13:17:18 ADD agent1.example.com A 10.31.32.14
13:18:09 DELETE agent1.example.com A

Workaround:

Republish the DNS records of the affected instance by triggering a new DNS update for it.

Verify that the FQDN of the instance does not resolve:
```
dig +short <instanceFQDN>
```
Republish the records using one of the following options:
- If the instance has a floating IP address assigned, detach and re-attach it:
```
openstack server remove floating ip <instanceName> <floatingIP>

openstack server add floating ip <instanceName> <floatingIP>
```
- If the instance does not have a floating IP address assigned, stop and start the instance:
  
  Warning
  
  This option implies downtime of the workload running on the instance.
```
openstack server stop <instanceName>

openstack server start <instanceName>
```
Verify that the FQDN of the instance resolves again:
```
dig +short <instanceFQDN>
```

[53818] Octavia load balancer is not functional after update

After a cluster update, an Octavia load balancer that uses the OpenSDN (tungstenfabric) provider may stop passing traffic, while both Octavia and OpenSDN keep reporting it as healthy. The load balancer members remain reachable by their own addresses, but the virtual IP address (VIP) of the load balancer does not respond and does not answer ARP requests.

The issue occurs because the load balancer network namespace fails to be recreated after the restart of the tf-vrouter-agent pod during the update: the name of the veth interface is still occupied by a stale interface left from the previous run. As a result, no HAProxy process is started for the affected load balancer. The failure is not logged, and the agent does not retry it.

Workaround:

Verify that the VIP of the load balancer does not respond while its members do, for example, from the keystone-client pod or from an instance in the same tenant network:
```
curl <loadBalancerVIP>:<port>

curl <memberIP>:<port>
```
Identify the compute nodes that host the service instances of the affected load balancer:
```
kubectl -n openstack exec -it <keystoneClientPodName> -- \
  openstack loadbalancer show <loadBalancerID>
```
On each OpenSDN compute node, the presence of the HAProxy process for the load balancer indicates whether the node is affected:
```
ps -ef | grep "haproxy -f /var/lib/contrail/loadbalancer/haproxy"
```
The node is affected if the output contains no entry with the ID of the load balancer, while the /var/lib/contrail/loadbalancer/<loadBalancerID>.conf configuration file exists on the node.

Restart the tf-vrouter-agent pod on each affected node:

kubectl -n tf delete pod <tfVrouterAgentPodName>

Wait until the pod becomes ready and verify that the VIP of the load balancer responds again:
```
curl <loadBalancerVIP>:<port>
```

[42896] Cassandra cluster contains extra node with outdated IP after replacement of TF control node

After replacing a failed Tungsten Fabric controller node as described in Replace a failed OpenSDN controller node, the first restart of the Cassandra pod on this node may cause an issue if the Cassandra node with the outdated IP address has not been removed from the cluster. Subsequent Cassandra pod restarts should not trigger this problem.

To verify if your Cassandra cluster is affected, run the nodetool status command specifying the config or analytics cluster and the replica number:

kubectl -n tf exec -it tf-cassandra-<CONFIG-OR-ANALYTICS>-dc1-rack1-<REPLICA-NUM> -c cassandra -- nodetool status

Example of the system response with outdated IP addresses:

Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load       Tokens       Owns    Host ID                               Rack
UN  192.168.201.144  509.43 KiB  256          ?       7e760a99-fae5-4921-b0c5-d9e6e1eca1c5  rack1
UN  192.168.50.146   534.18 KiB  256          ?       2248ea35-85d4-4887-820b-1fac4733021f  rack1
UN  192.168.145.147  484.19 KiB  256          ?       d988aaaa-44ae-4fec-a617-0b0a253e736d  rack1
DN  192.168.145.144  481.53 KiB  256          ?       c23703a1-6854-47a7-a4a2-af649d63af0c  rack1

An extra node will appear in the cluster with an outdated IP address (the IP of the terminated Cassandra pod) in the Down state.

To work around the issue, after replacing the Tungsten Fabric controller node, delete the Cassandra pod on the replaced node and remove the outdated node from the Cassandra cluster using nodetool:

kubectl -n tf exec -it tf-cassandra-<CONFIG-OR-ANALYTICS>-dc1-rack1-<REPLICA-NUM> -c cassandra -- nodetool removenode <HOST-ID>

[40032] tf-rabbitmq fails to start after rolling reboot

Occasionally, RabbitMQ instances in tf-rabbitmq pods fail to enable the tracking_records_in_ets during the initialization process.

To work around the issue, restart the affected pods manually.

OpenStack

[57473] OpenStack update fails due to neutron-ovs-agent-default start failure

During OpenStack update from Caracal to Epoxy, neutron-ovs-agent-default may fail to start with the The DaemonSet neutron-ovs-agent-default is not ready error in the Rockoon logs due to Kopf missing the OsDpl update events.

As a workaround, recreate the rockoon pod of the affected MOSK cluster:

kubectl -n osh-system rollout restart deployment rockoon

[53401] Credential rotation reports success without performing action

Occasionally, the password rotation procedure for admin or service credentials may incorrectly report success without actually initiating the rotation process. This can result in unchanged credentials despite the procedure indicating completion.

To work around the issue, restart the rotation procedure and verify that the credentials have been successfully updated.

[63801] Encrypted ephemeral VM disk becomes undecryptable

Rebooting a virtual machine (VM) with encrypted ephemeral storage after upgrading from OpenStack Caracal to Epoxy makes its data undecryptable.

Symptoms:

After upgrade from OpenStack Caracal to Epoxy, a VM can no longer access existing data on its encrypted ephemeral disk after a hard reboot or a stop/start. This issue affects VMs that meet both of the following conditions:

A VM uses an encrypted ephemeral storage that was created on OpenStack Caracal or earlier
A VM was created on OpenStack Caracal, and OpenStack was later upgraded to Epoxy

Warning

The data is not lost and remains recoverable as long as no application inside the affected VM writes to the ephemeral disk, for example, by reformatting it or re-creating partitions.

Cause:

OpenStack Caracal container images in MOSK are based on Ubuntu 22.04. OpenStack Epoxy images are based on Ubuntu 24.04, which includes cryptsetup 2.7. This cryptsetup version changes the default hashing algorithm for disk encryption from ripemd160 to sha256. For details, see Cryptsetup 2.7.0 Release Notes.

OpenStack Nova does not explicitly set the hashing algorithm, as per upstream known issue #1639221, when it configures disk encryption. As a result, after the upgrade to Epoxy, Nova sets up encryption with different parameters than those used when the disk was originally created, which breaks decryption. MOSK does not automatically re-encrypt affected disks with the original hashing algorithm.

Workaround:

MOSK 26.2 includes fixes that let you control the hashing algorithm that Nova uses for disk encryption:

Nova now explicitly specifies the hashing algorithm when it creates disk encryption. On Epoxy, the default is sha256, matching the cryptsetup default in Epoxy-based MOSK images. This default applies to all VMs on a given compute node. You can override it in the configuration of any nova-compute service.
You can also override the hashing algorithm for an individual VM using the dmcrypt_hash instance metadata key.

To fix VMs that are already broken or at risk of breaking after the upgrade to Epoxy:

Add the following metadata to each affected instance:

openstack server set --property dmcrypt_hash=ripemd160 <INSTANCE-ID>

If the instance is already broken (can not read its ephemeral disk), power-cycle the instance.

The disk becomes decryptable again.

To prevent existing VMs from breaking before you upgrade from Caracal to Epoxy:

While still on Caracal, set the following configuration option to ripemd160. The setting has no effect until you upgrade to Epoxy.

kind: OpenStackDeployment
spec:
  services:
    compute:
      nova:
        values:
          conf:
            nova:
              ephemeral_storage_encryption:
                hash: ripemd160

Upgrade to Epoxy.

Existing VMs continue to work without requiring per-instance metadata. New VMs also default to ripemd160, which is a less secure hashing algorithm than the sha256 default introduced in Epoxy.

[52320] Heat operations fail with Timed out waiting for a reply to message ID

Epoxy only

Under sustained or heavy load, Heat stack operations may fail with the following error:

CREATE_FAILED  Resource CREATE failed: MessagingTimeout: resources.<RESOURCE>:
Timed out waiting for a reply to message ID <ID>

The heat-engine logs also contain the following error:

AssertionError: do not call blocking functions from the mainloop

The issue is caused by an incompatibility between eventlet and the SQLAlchemy 2.0-style connection handling used by oslo.db in the OpenStack Epoxy releases of Heat, which can exhaust the database connection pool and cause RPC calls to time out. The issue is most likely to occur when a large number of stacks, or deeply nested stacks, are created, updated, or deleted concurrently.

As a workaround, periodically restart the heat-engine pods to release exhausted database connections and clear the timed-out RPC calls:

kubectl -n openstack rollout restart deployment heat-engine

[65683] Stale OVN routers block credential rotation after OVS to OVN migration

Gazpacho only

After migration from Neutron OVS to OVN, a Logical_Router row can be left behind in the OVN Northbound database for a router that Neutron has already deleted along with its associated external network. The Neutron OVN maintenance worker detects the stale row and tries to remove it, but the removal code unconditionally attempts to resync the HA chassis group for the router’s external network. Because that network no longer exists, the operation fails with NetworkNotFound and the stale router is never cleaned up.

As a result, the maintenance task never completes, and the neutron-periodic-workers pod, which waits for the maintenance task to finish before its startup probe can pass, crash-loops indefinitely. The deployment stays at 0/1 available, and OpenStackDeploymentStatus reports the neutron.periodic-workers service as unhealthy. Any operation that waits for full OpenStackDeployment health, including OpenStack credential rotation (osctl credentials rotate --wait), hangs and eventually times out.

Symptoms:

The ovn-maintenance container of the neutron-periodic-workers pod is in CrashLoopBackOff, and the neutron-periodic-workers deployment is 0/1 available.
OpenStackDeploymentStatus never reaches full health, with neutron.periodic-workers reported as the only unhealthy service.

The container startup probe log repeatedly includes the following error:

Waiting for maintenance jobs ['update_router_static_routes'] to be completed.
ERROR: Neutron server is not synced yet.

The ovn-maintenance container logs a traceback ending in:

neutron_lib.exceptions.NetworkNotFound: Network <network-id> could not be found.
...
File ".../ovn_client.py", line 1706, in delete_router
  self._sync_external_ports_hacg(context, network_id)

osctl credentials rotate --wait (or any operation that waits for full OpenStackDeployment health) times out.

Workaround:

Manually remove the stale OVN logical routers and restart neutron-periodic-workers:

Identify the OVN Northbound database pod:

kubectl -n openstack get pod | grep ovn-db

Compare the routers known to Neutron with the logical routers present in the OVN Northbound database to find the stale entries:
```
openstack router list -f value -c ID
kubectl -n openstack exec <OVN-DB-POD> -- ovn-nbctl lr-list
```
A logical router listed by ovn-nbctl but absent from the openstack router list output is stale.

Delete each stale logical router:

kubectl -n openstack exec <OVN-DB-POD> -- ovn-nbctl lr-del <STALE-ROUTER-UUID>

Restart the neutron-periodic-workers deployment:

kubectl rollout restart deploy/neutron-periodic-workers -n openstack

Alternatively, you can resynchronize Neutron with OVN, which also clears the stale entries:

neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf \
  --config-file /etc/neutron/plugins/ml2/ml2_conf.ini \
  --ovn-neutron_sync_mode repair

StackLight

[55317] The Dropped sample for series errors in the Prometheus logs

When the experimental feature memory-snapshot-on-shutdown, which is enabled by default, is used together with remote_write, Prometheus may emit multiple log messages, such as Dropped sample for series that was not explicitly dropped via relabelling. For more details, see the upstream issue description in the Prometheus GitHub project.

Workaround:

On the related management cluster, open the affected MOSK Cluster object for editing:

kubectl edit cluster <affectedMOSKClusterName> -n <affectedMOSKClusterProjectName>

Remove the memory-snapshot-on-shutdown feature from the prometheusServer:enabledFeatures list:

spec:
  ...
  providerSpec:
    ...
    value:
      ...
      helmReleases:
        ...
        - name: stacklight
          values:
            ...
            prometheusServer:
              enabledFeatures: []

Others

[65611] Support dump fails to collect MKE logs

During support dump collection using supportdump-controller, the underlayKubernetesLogs collector fails to gather per-node logs for MKE clusters through the MKE API. The issue affects MOSK 26.2.x clusters, which run Mirantis Container Runtime (MCR) 25.0.17 or later.

Symptoms:

The mke directory of the resulting support dump contains <nodeName>.error files with the following content:

Error copying support log dir from '<nodeName>': Error response from daemon: openat run/docker.sock: no such device or address

Workaround:

Request the systemLogs collector along with underlayKubernetesLogs, so that the SSH key for the cluster nodes becomes available to the collector. If system logs are not required, set a node label filter that no machine satisfies:

metadata:
  name: test
  namespace: mosk
spec:
  collectors:
  - kubernetesObjects: {}
  - underlayKubernetesLogs: {}
  - systemLogs:
      nodeLabels:
        fakelabel: fakevalue
  overallTimeout: 45m0s
  targetCluster: mosk

If, after requesting both systemLogs and underlayKubernetesLogs collectors, the SupportDump object obtains the Failed status with the following message, the per-node collector in MKE has set wrong permissions on the files in the cni folder:

multiple errors while collecting logs: Failed to create tarball: failed to create tarball for support dump <namespace>/<name>: open /dumps/<namespace>-<cluster>-<timestamp>/<cluster>/system/<nodeName>/mke/dsinfo/cni/calico/bgp_peers.txt: permission denied

In this case, perform the following steps:

Change the permissions for the files in the cni folder:

kubectl exec -n kaas <podName> -- /bin/sh -c \
'chmod -R 755 /dumps/<namespace>-<cluster>-<timestamp>/<cluster>/system/*/mke/dsinfo/cni/calico'

Create the archive:

kubectl exec -n kaas <podName> -- \
tar -zcf /dumps/<namespace>-<cluster>-<timestamp>.tar.gz \
/dumps/<namespace>-<cluster>-<timestamp>

Remove the directory with the raw collected data:

kubectl exec -n kaas <podName> -- \
rm -rf /dumps/<namespace>-<cluster>-<timestamp>

Copy the archive as usual:

kubectl cp \
kaas/<podName>:/dumps/<namespace>-<cluster>-<timestamp>.tar.gz ./logs.tar.gz