Known issues

This section lists MOSK known issues with workarounds for the MOSK release 26.2.1.

Bare metal

[62999] Master node replacement fails on MOSK clusters with BGP announcement enabled

On MOSK clusters with BGP announcement enabled (useBGPAnnouncement: true in the Cluster object), master nodes are deployed one per rack, and the Kubernetes API Virtual IP (VIP), for example, 10.0.30.100/32 is assigned to the loopback (lo) interface of each master node. For details on BGP announcement, see Configure BGP announcement for cluster API LB address.

During master node replacement, lcm-agent on the new node tries to reach the API VIP on the lo interface for the first time but cannot connect to it. This happens because cloud-init binds the API VIP to the lo interface before the local containerized API proxy starts listening on it, so the connection is refused. The lcm-agent cannot report back to the management cluster. As a result, the Machine object remains stuck in the PendingLCMAgent state.

As a workaround, temporarily remove the API VIP from the lo interface to force API traffic through the physical network to a healthy master node. Once the node finishes provisioning, the local VIP handling takes over automatically.

Workaround:

  1. From the management cluster, obtain the IP address of the new node from its IpamHost object:

    kubectl get ipamhost <NODE_NAME> -o jsonpath="{.status.serviceMap['ipam/SVC-k8s-lcm'][0].ipAddress}"
    
  2. SSH to the new node:

    ssh -i <path-to-ssh-key> mcc-user@<NEW_NODE_IP>
    
  3. Wait for cloud-init to complete on the new node:

    sudo cloud-init status --wait
    

    Before proceeding, ensure that the output reports status: done.

    Warning

    Do not proceed with networking modifications until cloud-init has finished executing, so it does not overwrite your changes.

  4. Verify that the API VIP is bound to the lo interface:

    ip addr show dev lo
    

    Example of system response on the affected node:

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        inet 127.0.0.1/8 scope host lo
        inet 10.0.30.100/32 scope global lo
    
  5. Confirm that lcm-agent is failing with a connection refused error:

    sudo journalctl -u lcm-agent* -n 20 --no-pager
    
  6. Remove the /32 API VIP from the lo interface:

    sudo ip addr del 10.0.30.100/32 dev lo
    

    Substitute 10.0.30.100 with the actual API VIP of your cluster.

  7. Verify outbound network connectivity to the API. It should now route across the network to one of the healthy master nodes:

    curl -k -v https://10.0.30.100:443/api
    

    An HTTP response such as HTTP/2 401 or 403 confirms that the traffic is reaching an active Kubernetes API server. For example:

    * Connected to 10.0.30.100 (10.0.30.100) port 443
    < HTTP/2 401
    {
      "kind": "Status",
      "message": "Unauthorized",
      "code": 401
    }
    
  8. Reload systemd to clear the warnings about unit files that have changed on disk:

    sudo systemctl daemon-reload
    
  9. Restart the lcm-agent daemon to trigger an immediate check-in:

    sudo systemctl restart lcm-agent*.service
    
  10. Verify that the node recovers:

    1. On the new node, inspect the agent logs to confirm a successful tick execution:

      sudo journalctl -u lcm-agent* -f
      

      In the system response, find the tick finished messages without connection errors.

    2. From the management or seed node, verify the machine phase transition:

      kubectl get machine,lcmmachine -o wide
      

      The LCMPHASE for the new node should transition from PendingLCMAgent to Prepare, Deploy, and then to Ready.

[42889] Graceful reboot gets stuck when Kubernetes and OpenStack control planes are drained simultaneously

When a GracefulRebootRequest targets both the Kubernetes and OpenStack control plane machines, either by listing machines of both types in spec.machines or by leaving the list empty to reboot all cluster nodes, the rolling reboot may get stuck. This happens because both node groups are drained in parallel, and the OpenStack workload manager running on the Kubernetes control plane becomes unavailable while the OpenStack control plane nodes are simultaneously being drained.

Workaround:

  1. Identify the machines that have not yet been rebooted.

  2. Delete the stuck GracefulRebootRequest:

    kubectl -n <projectName> delete gracefulrebootrequest <gracefulRebootRequestName>
    
  3. Recreate the reboot requests in two sequential steps as described in Perform a rolling reboot of a cluster using CLI: first for the Kubernetes control plane machines, then, once that request completes and is deleted, for the remaining machines that still require a reboot.

Ceph

[67812] Ceph OSD fails to start after a restart

After a restart of a Ceph OSD, for example, during a rolling restart of all Ceph OSDs, the Ceph OSD may fail to start. A race condition in Ceph corrupts a small allocation file that the Ceph OSD writes during shutdown. On the next start, the expand-bluefs init container of the Ceph OSD pod reads that file, detects one 64 KiB block in an unexpected location, and aborts. The data stored on the Ceph OSD is not damaged.

A similar behavior is described in the upstream Ceph issue 81213.

The Ceph cluster continues to operate with the affected Ceph OSD in the down and out state. Other Ceph OSDs, including those located on the same device, are not affected.

Symptoms:

  • The rook-ceph-osd-<ID> pod is stuck in the Init:Error or Init:CrashLoopBackOff state with a growing number of restarts.

  • The Rook Ceph Operator logs contain the following error:

    failed to update OSD deployment "rook-ceph-osd-<ID>": progress deadline exceeded
    
  • The logs of the expand-bluefs init container of the affected pod contain an assertion failure. To view the logs:

    kubectl -n rook-ceph logs <affectedOsdPodName> -c expand-bluefs
    

    Example of system response:

    inferring bluefs devices from bluestore path
    -1 hybrid::operator() lambda Uexpected extent:  0xd705000~10000
    HybridAllocator_impl.h: In function 'HybridAllocatorBase<AvlAllocator>::init_rm_free(uint64_t, uint64_t)::<lambda(uint64_t, uint64_t, bool)>'
    HybridAllocator_impl.h: 103: FAILED ceph_assert(false)
    

Workaround:

  1. Open the deployment of the affected Ceph OSD for editing:

    kubectl -n rook-ceph edit deployment rook-ceph-osd-<ID>
    
  2. In the spec.template.spec.initContainers section, add the following arguments of ceph-bluestore-tool to the expand-bluefs init container:

    initContainers:
    - name: expand-bluefs
      args:
      - bluefs-bdev-expand
      - --path
      - /var/lib/ceph/osd/ceph-<ID>
      - --bluestore-allocator=bitmap
      - --bluefs-allocator=bitmap
    
  3. Verify that the affected Ceph OSD pod is Running and the Ceph OSD is up and in:

    kubectl -n rook-ceph get pod -l ceph-osd-id=<ID>
    
    kubectl -n rook-ceph exec -it deploy/pelagia-ceph-toolbox -- ceph osd tree
    

If the issue persistently reproduces on multiple Ceph OSDs of a newly deployed cluster, configure Ceph OSDs to use the bitmap allocator and to store the allocation data in RocksDB instead of the allocation file, which eliminates the race condition:

  1. Open the CephDeployment custom resource for editing:

    kubectl -n ceph-lcm-mirantis edit cephdeployment
    
  2. In the spec.rookConfig section, add the following parameters:

    spec:
      rookConfig:
        osd|bluestore_allocator: "bitmap"
        osd|bluefs_allocator: "bitmap"
        osd|bluestore_allocation_from_file: "false"
    

    For details about rookConfig, see RookConfig parameters.

  3. Redeploy every affected Ceph OSD by removing it and adding it back as described in Add, remove, or reconfigure Ceph OSDs. The bluestore_allocation_from_file parameter applies only to redeployed Ceph OSDs.

[62392] False negatives in the ceph_daemon_socket_up metric

When a pod with the ceph-rgw daemon restarts, the stale rgw socket may persist due to the upstream Ceph issue 76939, which results in a false-positive failed ceph_daemon_socket_up check in the ceph-exporter metrics. For example:

ceph_daemon_socket_up{ceph_daemon="ceph-osd.1", hostname="child-worker-1-storage-worker", instance="192.168.9.124:9926", job="prometheus-ceph-exporter"}     1
ceph_daemon_socket_up{ceph_daemon="ceph-client.rgw.openstack.store.a.1.107989349947776", hostname="managed-master-2-control-storage", instance="192.168.9.122:9926", job="prometheus-ceph-exporter"} 0
ceph_daemon_socket_up{ceph_daemon="ceph-osd.0", hostname="child-worker-2-storage-worker", instance="192.168.9.125:9926", job="prometheus-ceph-exporter"}     1
ceph_daemon_socket_up{ceph_daemon="ceph-client.rgw.openstack.store.a.1.95065910507904", hostname="managed-master-1-control-storage", instance="192.168.9.121:9926", job="prometheus-ceph-exporter"}  1
ceph_daemon_socket_up{ceph_daemon="ceph-client.rgw.openstack.store.a.1.104243065772416", hostname="managed-master-2-control-storage", instance="192.168.9.122:9926", job="prometheus-ceph-exporter"} 1

In the example above, ceph-exporter detects a failed socket with the 0 value for the ceph-client.rgw.openstack.store.a.1.107989349947776 daemon, which has already been replaced by the ceph-client.rgw.openstack.store.a.1.104243065772416 daemon. To mitigate the issue, the operator must manually remove the stale socket from the affected node.

Workaround:

  1. SSH into a node where the affected daemon is placed. In the example above, it is managed-master-2-control-storage.

  2. Locate the stale socket with the 0 value under the /var/lib/rook/exporter/ directory:

    ls -l /var/lib/rook/exporter/
    

    Example of system response:

    total 0
    srwxr-xr-x 1 root root 0 Jun 16 19:32 ceph-client.ceph-exporter.asok
    srwxr-xr-x 1  167  167 0 Jun 16 19:22 ceph-client.rgw.openstack.store.a.1.104243065772416.asok
    srwxr-xr-x 1  167  167 0 Jun 16 20:52 ceph-client.rgw.openstack.store.a.1.107989349947776.asok
    srwxr-xr-x 1  167  167 0 Jun 16 19:29 ceph-mon.a.asok
    
  3. Remove the stale socket. For example:

    rm /var/lib/rook/exporter/ceph-client.rgw.openstack.store.a.1.107989349947776.asok
    

Cluster update

[68577] MOSK cluster update gets stuck during the MKE upgrade

During the update of a MOSK cluster to 26.2.1 (Cluster release 21.2.1), the k8s-controllers step of the ClusterUpdatePlan object may get stuck with the 0 out of 3 machines upgraded message because the MKE upgrade fails to start.

The issue occurs because the previous cluster update leaves the ucp-3-7-32-mosk upgrade service in Docker Swarm. During the update to MOSK 26.2.1, the MCR update restarts dockerd on the control plane node that this service is pinned to, and Docker Swarm reruns the previous MKE upgrade. As a result, an extra worker agent service is created, which blocks the upgrade to MKE 3.7.33.

Symptoms:

  • The ClusterUpdatePlan object has the Stuck status on the k8s-controllers step.

  • All attempts of the ucp-3-7-33-mosk MKE upgrade service fail with the following error:

    unable to initialize upgrader: unable to determine blue/green worker agent services: \
    both -x and -y worker node agent services exist. Is an upgrade already in progress?
    

Workaround:

  1. On an MKE manager node of the affected cluster, verify that the MKE upgrade fails with the error described above:

    sudo docker service logs --raw ucp-3-7-33-mosk 2>&1 | grep 'both -x and -y worker node agent services exist'
    

    If the output is empty, the update is stuck for a different reason. In this case, do not apply the workaround and contact Mirantis support.

  2. List the worker agent services:

    sudo docker service ls | grep ucp-worker-agent
    

    Example of system response:

    xg5qz6u9omt6   ucp-worker-agent-x   global   0/0   mirantis/ucp-agent:3.7.32-mosk
    hfaox4q10bb6   ucp-worker-agent-y   global   9/9   mirantis/ucp-agent:3.7.32-mosk
    

    On an affected cluster, both services exist and use the image of the MKE version that is not upgraded yet, ucp-agent:3.7.32-mosk. One service has all replicas running, and the other one has 0/0 replicas.

  3. Remove the service with 0/0 replicas, which can be either ucp-worker-agent-x or ucp-worker-agent-y. For example:

    sudo docker service rm ucp-worker-agent-x
    

    Warning

    Do not remove the service with 0/0 replicas if it uses the image of the new MKE version, ucp-agent:3.7.33-mosk.

  4. Remove the services left from the previous MKE upgrade. Complete this step before the MKE upgrade is retried.

    sudo docker service rm ucp-3-7-32-mosk ucp-images-check-3-7-32-mosk
    
  5. On the MOSK management cluster, increase the number of the MKE upgrade attempts for the affected cluster to retry the upgrade. Set mkeUpgradeAttempts to 7 or, if the parameter is already set, increase its value by at least 1:

    kubectl -n <projectName> patch lcmcluster <clusterName> --type merge -p '{"spec":{"mkeUpgradeAttempts":7}}'
    

    The MOSK cluster update proceeds automatically.

[65903] Management cluster update gets stuck with k0s failing to start

Applies only to clusters deployed using 2.13.x or earlier

Management clusters deployed using an old Container Cloud release, earlier than 2.14.0, may contain an override of the IAM configuration of the baremetal-provider Helm release in the Cluster object with iam.enabled: false. This override is invalid and incompatible with the update to MOSK management 2.32.0. It may cause the update to get stuck with the k0s control plane nodes failing to start with the following example errors:

err="open /etc/kubernetes/oidc_ca.crt: no such file or directory\"" component=kube-apiserver

msg="Failed to wait for process" component=kube-apiserver error="exit status 1"

To verify whether the cluster is affected:

  1. Log in to a node that contains kubeconfig of the required management cluster. For example:

    export KUBECONFIG=/path/to/mgmt-kubeconfig
    
  2. Verify whether the baremetal-provider Helm release contains iam.enabled: false:

    kubectl get cluster <mgmtClusterName> -o jsonpath='{.spec.providerSpec.value.kaas.regional[?(@.provider=="baremetal")].helmReleases[?(@.name=="baremetal-provider")].values.config.iam}'
    

    Example output of the affected cluster:

    {"enabled":false,"registrationUrl":"<ip>:8082"}
    

    If the output is empty or does not contain "enabled":false, the cluster is not affected and you can skip this procedure. Otherwise, proceed to the next step.

  3. Verify that HelmBundle of the management cluster has IAM enabled for the baremetal-provider release:

    kubectl -n default get helmbundle <mgmtClusterName> -o jsonpath='{.spec.releases[?(@.name=="baremetal-provider")].values.config.iam}'
    

    Example output of the affected cluster:

    {"enabled":true,"registrationUrl":"<ip>:8082"}
    

    Example of the affected HelmBundle snippet:

    spec:
      releases:
      ...
      - name: baremetal-provider
        namespace: kaas
        values:
          ...
          config:
            ...
            iam:
              enabled: true
              registrationUrl: <ip>:8082
    

    If the output is empty, does not contain iam.enabled, or contains "enabled":false, contact Mirantis support. Do not proceed to the next step.

  4. Verify whether the Cluster object contains the Keycloak TLS configuration:

    kubectl get cluster <mgmtClusterName> -o jsonpath='{.spec.providerSpec.value.tls.keycloak.tlsConfigRef}'
    

    Example of the affected Cluster object:

    spec:
      ...
      providerSpec:
        value:
          ...
          tls:
            ...
            keycloak:
              tlsConfigRef: keycloak
    

    If the output is empty, contact Mirantis support. Do not proceed with the workaround below.

Workaround:

If the Keycloak TLS configuration is present and the HelmBundle has iam.enabled: true, remove enabled: false from the Cluster object:

  1. Open the Cluster object of the affected management cluster for editing:

    kubectl edit cluster <mgmtClusterName>
    
  2. In the spec.providerSpec.value.kaas.regional section, find the baremetal-provider Helm release and delete only the enabled: false line under values.config.iam.

    Caution

    Do not remove registrationUrl or other IAM parameters.

    spec:
      ...
      providerSpec:
        value:
          ...
          kaas:
            regional:
            - provider: baremetal
              helmReleases:
              - name: baremetal-provider
                values:
                  ...
                  config:
                    iam:
                      enabled: false # delete this line
                      registrationUrl: <ip>:8082
    
  3. Save and exit the object to apply changes.

    Warning

    After you remove the override, the management cluster nodes may be sequentially cordoned, drained, and updated with the new configuration. Do not start the update to MOSK management 2.32.0 until the cluster becomes ready.

  4. Wait until every management cluster node completes the configuration update and the cluster becomes ready:

    kubectl get cluster <mgmtClusterName> -o jsonpath='{.status.providerStatus.ready}{"\n"}'
    

    The command must return true, which means the cluster is ready for the update to MOSK management 2.32.0.

[65106] Management cluster update hangs due to Keycloak database schema issue

A management cluster update may get stuck with the following error:

keycloak encountered an exception.: liquibase.exception.DatabaseException: (conn=538) \
Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. \
This includes storage overhead, check the manual. \
You have to change some columns to TEXT or BLOBs \
[Failed SQL: (1118) ALTER TABLE keycloak.CREDENTIAL ADD VERSION INT DEFAULT 0 NULL]

Workaround:

  1. Log in to the MariaDB console as admin:

    kubectl -n kaas exec -it mariadb-server-0 -- bash -c 'mysql -uroot -p$MYSQL_DBADMIN_PASSWORD'
    
  2. Optimize the affected table:

    OPTIMIZE TABLE keycloak.CREDENTIAL;
    

[64721] MOSK cluster upgrade gets stuck due to runc-ee package downgrade

Clusters deployed with MOSK 26.1 (Cluster release 21.1.0) starting from 2026-08-04 may have issues upgrading to more recent versions, such as MOSK 26.1.x (Cluster releases 21.1.x) patch releases or MOSK 26.2 (Cluster release 21.2.0). The issue occurs during lcm-ansible execution at the step that installs the runc-ee package.

Symptom: the first control plane node gets stuck in the deploy phase with the following error in the lcm-ansible logs:

TASK [containerd : Install containerd.io packages]
task path: .../lcm-ansible-<version>/roles/containerd/tasks/Debian.yml:62
The following packages will be upgraded:
  containerd.io* → 1.7.31m1+fips-0ubuntu0.24.04.1
The following packages will be DOWNGRADED:
  runc-ee → 1.4.2m1-0ubuntu0.24.04.1
E: Packages were downgraded and -y was used without --allow-downgrades.

The issue occurs because the runc-ee package version was not pinned and was installed from a constantly updated repository in MOSK 26.1 (Cluster release 21.1.0). Starting from MOSK 26.1.x (Cluster releases 21.1.x) patch releases and MOSK 26.2 (Cluster release 21.2.0), the package version is pinned.

Workaround:

  1. SSH to any node of the affected cluster.

  2. Install the pinned version of the runc-ee package:

    apt-get --allow-downgrades install runc-ee=1.4.2m1-0ubuntu0.24.04.1 -y --allow-change-held-packages
    
  3. Hold the package to prevent future automatic updates:

    apt-mark hold runc-ee
    
  4. Repeat steps 1-3 for the remaining nodes of the affected cluster.

MOSK management cluster

[64703] Missing WireGuard interface address on management cluster nodes

On new management clusters based on Cluster release 22.0.0 (MOSK 26.2) that have secureOverlay set to true, some management cluster nodes may be missing the WireGuard interface address. As a result, kube-apiserver or any other process running in the host network namespace on an affected node cannot reach pods running on healthy nodes.

Symptom: webhook calls handled by kube-apiserver on the affected node fail with a timeout if the pod serving the webhook runs on a healthy node. The error propagates back to the client that issued the original API request.

The following example illustrates a cert-manager log entry produced when the cert-manager pod sends a request to kube-apiserver on the affected node, while cert-manager-webhook runs on a healthy node:

E0817 23:38:37.501136       1 controller.go:167] cert-manager/issuers "msg"="re-queuing item due to error processing" "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.kaas.svc:443/mutate?timeout=10s\": context deadline exceeded" "key"="kaas/mcc-ca"

The following example illustrates the corresponding k0scontroller log entry on the affected node:

Aug 17 23:38:37 master-2 k0s[45673]: time="2026-08-17 23:38:37" level=info msg="W0817 23:38:37.496457   45763 dispatcher.go:225] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.kaas.svc:443/mutate?timeout=10s\": context deadline exceeded" component=kube-apiserver stream=stderr

Workaround:

  1. Identify the affected nodes that have no IP address in the output of the following command:

    kubectl get node -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.projectcalico\.org/IPv4WireguardInterfaceAddr}{"\n"}{end}'
    
  2. Identify the names of the calico-node pods running on the affected nodes:

    kubectl get pod -n kube-system -o wide | grep calico-node
    
  3. Delete the affected pods to trigger their restart:

    kubectl delete pod -n kube-system <calico-node-pod-name>
    
  4. Repeat steps 1-3 until all nodes have the WireGuard interface address assigned.

[64454] Node gets stuck in the deploy phase

During bootstrap of a MOSK management cluster, a master node may get stuck in the deploy phase and never become Ready, even though the LCM deploy playbook has completed successfully.

Symptoms:

  • The lcm-agent logs on the affected nodes, including nodes that are not stuck, contain the following error:

    {"level":"error","msg":"failed to check k0s manager health","error":"local etcd member health check failed: failed to get etcd member master-1: etcdmembers.etcd.k0sproject.io \"master-1\" not found"}
    
  • No etcdmember objects exist for any node, even though etcd itself is healthy:

    kubectl get etcdmembers.etcd.k0sproject.io -A
    

    Example of system response:

    No resources found
    

Workaround:

Restart the k0scontroller service on one of the affected nodes:

systemctl restart k0scontroller

Wait for the node to become Ready and for its deployment to finish before proceeding to the next one. The remaining affected nodes will likely finish deployment successfully without requiring the same restart. Otherwise, restart the service on each remaining affected node as well.

[64407] MariaDB restore from backup causes Keycloak failure

During a full MariaDB unavailability caused by the restore procedure, Keycloak’s internal clustering mechanism cannot re-establish the cluster state, leaving the affected pods in the NOT READY condition even after MariaDB becomes available again.

Symptoms: the Keycloak pod logs contain the following errors:

ERROR [org.keycloak.jgroups.protocol.KEYCLOAK_JDBC_PING2] (Timer runner-2,iam-keycloak-0-4892(v=16.0.8)) iam-keycloak-0-4892(v=16.0.8): failed reading from the DB: java.sql.SQLNonTransientConnectionException: Socket fail to connect to mariadb. Connection refused
Caused by: java.net.ConnectException: Connection refused
    at org.mariadb.jdbc.client.impl.ConnectionHelper.connectSocket(ConnectionHelper.java:118)
    ... 12 more

INFO  [io.smallrye.health] (vert.x-eventloop-thread-0) SRHCK01001: Reporting health down status: {"status":"DOWN","checks":[{"name":"Graceful Shutdown","status":"UP"},{"name":"Keycloak database connections async health check","status":"UP"},{"name":"Keycloak Initialized","status":"UP"},{"name":"Keycloak cluster health check","status":"DOWN","data":{"Failing since":"2026-08-17 17:38:26,222"}}]}

Workaround:

  1. Verify that the restore procedure completed successfully and that all MariaDB replicas are up and ready.

  2. Identify the stuck Keycloak pods:

    kubectl get pods -n kaas | grep iam-keycloak
    
  3. Delete each stuck pod:

    kubectl delete pod -n kaas <IAM_KEYCLOAK_POD_NAME>
    
  4. Verify that the newly created Keycloak pods are up and in the READY state:

    kubectl get pods -n kaas | grep iam-keycloak
    

MOSK management console

[50168] Inability to use a new project right after creation

A newly created project does not display all available tabs in the MOSK management console and contains different access denied errors during first five minutes after creation.

To work around the issue, refresh the browser in five minutes after the project creation.

Networking

[64752] LoadBalancer IPs flap after a MetalLB speaker node shutdown

On clusters using MetalLB in L2 mode, an unexpected shutdown of a node running a MetalLB speaker can cause LoadBalancer service IP addresses to flap instead of failing over to a live node. The issue can occur on a management cluster or a MOSK cluster, on any node hosting a speaker, whether a control plane node or a worker node. The condition does not self-recover while the node remains down.

Affected services are those of type LoadBalancer that MetalLB advertises in L2 mode. For example, on the management cluster these include iam-keycloak-http, kaas-kaas-ui, and iam-proxy-*. As a result, access to Keycloak, the MOSK management console, and StackLight IAM proxies (Grafana, Prometheus, Alerta, Alertmanager, Kibana) becomes intermittently unavailable. On a MOSK cluster, the same failure mode affects the LoadBalancer services announced on the powered-off speaker node.

If the affected cluster is the management cluster, support dump collection may also fail because the systemLogs collector cannot complete:

failed to collect system logs: failed waiting for calls to execute

Symptoms:

After a hard power-off of a node running a MetalLB speaker, the external IP addresses of LoadBalancer services become intermittently reachable. Reachability can change every few seconds. For example, TCP probes to <ip>:443 from a live node, taken a few seconds apart, may look as follows:

Service              IP             Round 0   Round 1   Round 2
iam-keycloak-http    172.16.61.24   OPEN      CLOSED    OPEN
kaas-kaas-ui         172.16.61.11   CLOSED    CLOSED    OPEN
iam-proxy-alerta     172.16.61.13   CLOSED    OPEN      OPEN
iam-proxy-grafana    172.16.61.15   OPEN      CLOSED    OPEN
iam-proxy-prometheus 172.16.61.17   CLOSED    CLOSED    CLOSED

The MetalLB speaker pods on the surviving nodes log a memberlist join failure approximately every 60 seconds:

{"caller":"speakerlist.go:269","op":"memberDiscovery","level":"error",
 "msg":"partial join","expected":1,"joined":0,
 "error":"Failed to join 172.16.61.26:7947: dial tcp 172.16.61.26:7947: connect: no route to host"}

The ServiceL2Status objects still allocate LoadBalancer IP addresses to the powered-off node (kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587 in the example below):

kubectl -n metallb-system get servicel2status

Example of system response:

NAME       ALLOCATED NODE                                   SERVICE NAME                SERVICE NAMESPACE
l2-fwbbn   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   ironic-kaas-bm              kaas
l2-kx245   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   ironic-syslog               kaas
l2-m74nf   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   iam-proxy-prometheus        stacklight
l2-ndsz9   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   httpd-http                  kaas
l2-x6tw2   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   dhcp-lb                     kaas
l2-ztpjw   kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587   iam-proxy-grafana           stacklight

Service backends remain healthy: externalTrafficPolicy is Cluster, endpoints contain ready pods on the live nodes, and direct connections to backend pod IPs succeed. This rules out a backend failure.

The MetalLB speaker pod on the powered-off node remains Running (1/1 Ready). Pod status is push-based: when the node is powered off, kubelet dies and stops reporting container state, so the API server keeps the last known status. The node is tainted with node.kubernetes.io/unreachable:NoExecute, but the speaker DaemonSet tolerates that taint indefinitely and Kubernetes never evicts the pod.

MetalLB determines L2 IP ownership from speaker pods that Kubernetes still reports as Ready. Because the frozen pod still appears healthy, MetalLB treats the dead node as an eligible L2 speaker. Speaker membership never converges, L2 IP ownership keeps being re-elected, and gratuitous ARP announcements flap.

Workaround:

Force-delete the frozen MetalLB speaker pod on the powered-off node:

  1. Using the kubeconfig of the affected cluster, identify the speaker pod on the powered-off node:

    kubectl -n metallb-system get pod -o wide | grep speaker
    

    The pod on the unreachable node remains Running while the node is NotReady.

  2. Force-delete that speaker pod:

    Note

    A graceful deletion cannot complete because kubelet on the powered-off node is not running.

    kubectl -n metallb-system delete pod <speakerPodName> --force --grace-period=0
    

    Kubernetes immediately removes the frozen pod object. The DaemonSet creates a replacement pod that cannot become Ready on the dead node, so MetalLB stops considering that node for L2 IP ownership.

  3. Verify that ServiceL2Status objects no longer list the powered-off node as the allocated node and that the LoadBalancer IP addresses stop flapping:

    kubectl -n metallb-system get servicel2status
    

OpenSDN

[68383] vRouter kernel module is not updated after OpenSDN upgrade to 25.1

Applies only to the upgrade of OpenSDN from 24.1 to 25.1

After the upgrade of OpenSDN from 24.1 to 25.1, the OpenSDN 24.1 vrouter kernel module remains loaded and installed on compute nodes, although all OpenSDN pods run 25.1 images and the TFOperator custom resource reports the upgrade as completed. As a result, the vRouter Agent of OpenSDN 25.1 runs on top of the 24.1 datapath. The issue persists after node reboot.

The issue occurs because Dynamic Kernel Module Support (DKMS) treats the version of the 24.1 module as newer than the 25.1 one and does not install the new module, while the kernel-build-init container of the tf-vrouter-agent pod completes successfully despite the failure.

Symptoms:

The logs of the kernel-build-init container contain the following error:

Error! Module version 26.2.r25.1.<buildID> for vrouter.ko.zst
is not newer than what is already found in kernel <kernelVersion> (v0+26.2.r24.1.<buildID>).

To verify whether a compute node is affected:

  1. Obtain the name of the tf-vrouter-agent pod running on the node:

    kubectl -n tf get pods -l app=tf-vrouter-agent -o wide
    
  2. Verify the version of the vrouter kernel module:

    POD=<tfVrouterAgentPodName>
    kubectl -n tf exec $POD -c agent -- sh -c \
      'cat /sys/module/vrouter/version; for f in /lib/modules/*/updates/dkms/vrouter.ko*; do echo "$f $(modinfo -F version $f)"; done'
    

    Example of system response on an affected node:

    v0+26.2.r24.1.20261103122747
    /lib/modules/6.8.0-134-generic/updates/dkms/vrouter.ko v0+26.2.r24.1.20261103122747
    /lib/modules/6.8.0-147-generic/updates/dkms/vrouter.ko v0+26.2.r24.1.20261103122747
    

    The node is affected if the version of the loaded or installed module starts with v0+ (OpenSDN 24.1), while the agent image has the 26.2-r25.1.<buildID> tag.

Workaround:

Perform the following procedure on one compute node at a time. Restarting the vRouter Agent briefly interrupts the workload traffic on the node, as during a regular vRouter update.

  1. Remove the stale 24.1 module files for all kernels. The loaded module is not affected.

    kubectl -n tf exec $POD -c agent -- sh -c 'rm -fv /lib/modules/*/updates/dkms/vrouter.ko*'
    
  2. Restart the vRouter Agent:

    kubectl -n tf delete pod $POD
    

    The kernel-build-init container unloads the 24.1 module, then builds and installs the 25.1 module for all installed kernels.

  3. Once the new pod is Ready, verify the module version:

    POD=<newTfVrouterAgentPodName>
    kubectl -n tf exec $POD -c agent -- sh -c \
      'cat /sys/module/vrouter/version; for f in /lib/modules/*/updates/dkms/vrouter.ko*; do echo "$f $(modinfo -F version $f)"; done'
    

    Both the loaded and installed versions must match the image version of the vRouter Agent. For example, 26.2.r25.1.20261103113743 for the 26.2-r25.1.20261103113743 image.

  4. If the module files on disk have the 25.1 version but the loaded module still has the v0+ version, the 24.1 module could not be unloaded. In this case, drain and reboot the compute node as described in Perform a graceful reboot of a cluster. The 25.1 module is loaded from disk at boot.

[53818] Octavia load balancer is not functional after update

After a cluster update, an Octavia load balancer that uses the OpenSDN (tungstenfabric) provider may stop passing traffic, while both Octavia and OpenSDN keep reporting it as healthy. The load balancer members remain reachable by their own addresses, but the virtual IP address (VIP) of the load balancer does not respond and does not answer ARP requests.

The issue occurs because the load balancer network namespace fails to be recreated after the restart of the tf-vrouter-agent pod during the update: the name of the veth interface is still occupied by a stale interface left from the previous run. As a result, no HAProxy process is started for the affected load balancer. The failure is not logged, and the agent does not retry it.

Workaround:

  1. Verify that the VIP of the load balancer does not respond while its members do, for example, from the keystone-client pod or from an instance in the same tenant network:

    curl <loadBalancerVIP>:<port>
    
    curl <memberIP>:<port>
    
  2. Identify the compute nodes that host the service instances of the affected load balancer:

    kubectl -n openstack exec -it <keystoneClientPodName> -- \
      openstack loadbalancer show <loadBalancerID>
    

    On each OpenSDN compute node, the presence of the HAProxy process for the load balancer indicates whether the node is affected:

    ps -ef | grep "haproxy -f /var/lib/contrail/loadbalancer/haproxy"
    

    The node is affected if the output contains no entry with the ID of the load balancer, while the /var/lib/contrail/loadbalancer/<loadBalancerID>.conf configuration file exists on the node.

  3. Restart the tf-vrouter-agent pod on each affected node:

    kubectl -n tf delete pod <tfVrouterAgentPodName>
    
  4. Wait until the pod becomes ready and verify that the VIP of the load balancer responds again:

    curl <loadBalancerVIP>:<port>
    

[42896] Cassandra cluster contains extra node with outdated IP after replacement of TF control node

After replacing a failed Tungsten Fabric controller node as described in Replace a failed OpenSDN controller node, the first restart of the Cassandra pod on this node may cause an issue if the Cassandra node with the outdated IP address has not been removed from the cluster. Subsequent Cassandra pod restarts should not trigger this problem.

To verify if your Cassandra cluster is affected, run the nodetool status command specifying the config or analytics cluster and the replica number:

kubectl -n tf exec -it tf-cassandra-<CONFIG-OR-ANALYTICS>-dc1-rack1-<REPLICA-NUM> -c cassandra -- nodetool status

Example of the system response with outdated IP addresses:

Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load       Tokens       Owns    Host ID                               Rack
UN  192.168.201.144  509.43 KiB  256          ?       7e760a99-fae5-4921-b0c5-d9e6e1eca1c5  rack1
UN  192.168.50.146   534.18 KiB  256          ?       2248ea35-85d4-4887-820b-1fac4733021f  rack1
UN  192.168.145.147  484.19 KiB  256          ?       d988aaaa-44ae-4fec-a617-0b0a253e736d  rack1
DN  192.168.145.144  481.53 KiB  256          ?       c23703a1-6854-47a7-a4a2-af649d63af0c  rack1

An extra node will appear in the cluster with an outdated IP address (the IP of the terminated Cassandra pod) in the Down state.

To work around the issue, after replacing the Tungsten Fabric controller node, delete the Cassandra pod on the replaced node and remove the outdated node from the Cassandra cluster using nodetool:

kubectl -n tf exec -it tf-cassandra-<CONFIG-OR-ANALYTICS>-dc1-rack1-<REPLICA-NUM> -c cassandra -- nodetool removenode <HOST-ID>

[40032] tf-rabbitmq fails to start after rolling reboot

Occasionally, RabbitMQ instances in tf-rabbitmq pods fail to enable the tracking_records_in_ets during the initialization process.

To work around the issue, restart the affected pods manually.

OpenStack

[57473] OpenStack update fails due to neutron-ovs-agent-default start failure

During OpenStack update from Caracal to Epoxy, neutron-ovs-agent-default may fail to start with the The DaemonSet neutron-ovs-agent-default is not ready error in the Rockoon logs due to Kopf missing the OsDpl update events.

As a workaround, recreate the rockoon pod of the affected MOSK cluster:

kubectl -n osh-system rollout restart deployment rockoon

[53401] Credential rotation reports success without performing action

Occasionally, the password rotation procedure for admin or service credentials may incorrectly report success without actually initiating the rotation process. This can result in unchanged credentials despite the procedure indicating completion.

To work around the issue, restart the rotation procedure and verify that the credentials have been successfully updated.

[63801] Encrypted ephemeral VM disk becomes undecryptable

Rebooting a virtual machine (VM) with encrypted ephemeral storage after upgrading from OpenStack Caracal to Epoxy makes its data undecryptable.

Symptoms:

After upgrade from OpenStack Caracal to Epoxy, a VM can no longer access existing data on its encrypted ephemeral disk after a hard reboot or a stop/start. This issue affects VMs that meet both of the following conditions:

  • A VM uses an encrypted ephemeral storage that was created on OpenStack Caracal or earlier

  • A VM was created on OpenStack Caracal, and OpenStack was later upgraded to Epoxy

Warning

The data is not lost and remains recoverable as long as no application inside the affected VM writes to the ephemeral disk, for example, by reformatting it or re-creating partitions.

Cause:

OpenStack Caracal container images in MOSK are based on Ubuntu 22.04. OpenStack Epoxy images are based on Ubuntu 24.04, which includes cryptsetup 2.7. This cryptsetup version changes the default hashing algorithm for disk encryption from ripemd160 to sha256. For details, see Cryptsetup 2.7.0 Release Notes.

OpenStack Nova does not explicitly set the hashing algorithm, as per upstream known issue #1639221, when it configures disk encryption. As a result, after the upgrade to Epoxy, Nova sets up encryption with different parameters than those used when the disk was originally created, which breaks decryption. MOSK does not automatically re-encrypt affected disks with the original hashing algorithm.

Workaround:

MOSK 26.2 includes fixes that let you control the hashing algorithm that Nova uses for disk encryption:

  • Nova now explicitly specifies the hashing algorithm when it creates disk encryption. On Epoxy, the default is sha256, matching the cryptsetup default in Epoxy-based MOSK images. This default applies to all VMs on a given compute node. You can override it in the configuration of any nova-compute service.

  • You can also override the hashing algorithm for an individual VM using the dmcrypt_hash instance metadata key.

To fix VMs that are already broken or at risk of breaking after the upgrade to Epoxy:

  1. Add the following metadata to each affected instance:

    openstack server set --property dmcrypt_hash=ripemd160 <INSTANCE-ID>
    
  2. If the instance is already broken (can not read its ephemeral disk), power-cycle the instance.

The disk becomes decryptable again.

To prevent existing VMs from breaking before you upgrade from Caracal to Epoxy:

  1. While still on Caracal, set the following configuration option to ripemd160. The setting has no effect until you upgrade to Epoxy.

    kind: OpenStackDeployment
    spec:
      services:
        compute:
          nova:
            values:
              conf:
                nova:
                  ephemeral_storage_encryption:
                    hash: ripemd160
    
  2. Upgrade to Epoxy.

Existing VMs continue to work without requiring per-instance metadata. New VMs also default to ripemd160, which is a less secure hashing algorithm than the sha256 default introduced in Epoxy.

[52320] Heat operations fail with Timed out waiting for a reply to message ID

Epoxy only

Under sustained or heavy load, Heat stack operations may fail with the following error:

CREATE_FAILED  Resource CREATE failed: MessagingTimeout: resources.<RESOURCE>:
Timed out waiting for a reply to message ID <ID>

The heat-engine logs also contain the following error:

AssertionError: do not call blocking functions from the mainloop

The issue is caused by an incompatibility between eventlet and the SQLAlchemy 2.0-style connection handling used by oslo.db in the OpenStack Epoxy releases of Heat, which can exhaust the database connection pool and cause RPC calls to time out. The issue is most likely to occur when a large number of stacks, or deeply nested stacks, are created, updated, or deleted concurrently.

As a workaround, periodically restart the heat-engine pods to release exhausted database connections and clear the timed-out RPC calls:

kubectl -n openstack rollout restart deployment heat-engine

StackLight

[55317] The Dropped sample for series errors in the Prometheus logs

When the experimental feature memory-snapshot-on-shutdown, which is enabled by default, is used together with remote_write, Prometheus may emit multiple log messages, such as Dropped sample for series that was not explicitly dropped via relabelling. For more details, see the upstream issue description in the Prometheus GitHub project.

Workaround:

  1. On the related management cluster, open the affected MOSK Cluster object for editing:

    kubectl edit cluster <affectedMOSKClusterName> -n <affectedMOSKClusterProjectName>
    
  2. Remove the memory-snapshot-on-shutdown feature from the prometheusServer:enabledFeatures list:

    spec:
      ...
      providerSpec:
        ...
        value:
          ...
          helmReleases:
            ...
            - name: stacklight
              values:
                ...
                prometheusServer:
                  enabledFeatures: []