Known issues
This section lists MOSK known issues with workarounds for the MOSK release 26.2.1.
Bare metal
[62999] Master node replacement fails on MOSK clusters with BGP announcement enabled
On MOSK clusters with BGP announcement enabled (useBGPAnnouncement: true in
the Cluster object), master nodes are deployed one per rack, and the
Kubernetes API Virtual IP (VIP), for example, 10.0.30.100/32 is assigned to
the loopback (lo) interface of each master node. For details on BGP
announcement, see Configure BGP announcement for cluster API LB address.
During master node replacement, lcm-agent on the new node tries to reach
the API VIP on the lo interface for the first time but cannot connect to
it. This happens because cloud-init binds the API VIP to the lo
interface before the local containerized API proxy starts listening on it, so
the connection is refused. The lcm-agent cannot report back to the
management cluster. As a result, the Machine object remains stuck in the
PendingLCMAgent state.
As a workaround, temporarily remove the API VIP from the lo interface to
force API traffic through the physical network to a healthy master node. Once
the node finishes provisioning, the local VIP handling takes over
automatically.
Workaround:
From the management cluster, obtain the IP address of the new node from its
IpamHostobject:kubectl get ipamhost <NODE_NAME> -o jsonpath="{.status.serviceMap['ipam/SVC-k8s-lcm'][0].ipAddress}"
SSH to the new node:
ssh -i <path-to-ssh-key> mcc-user@<NEW_NODE_IP>
Wait for
cloud-initto complete on the new node:sudo cloud-init status --wait
Before proceeding, ensure that the output reports
status: done.Warning
Do not proceed with networking modifications until
cloud-inithas finished executing, so it does not overwrite your changes.Verify that the API VIP is bound to the
lointerface:ip addr show dev lo
Example of system response on the affected node:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 inet 127.0.0.1/8 scope host lo inet 10.0.30.100/32 scope global loConfirm that
lcm-agentis failing with a connection refused error:sudo journalctl -u lcm-agent* -n 20 --no-pager
Remove the
/32API VIP from thelointerface:sudo ip addr del 10.0.30.100/32 dev lo
Substitute
10.0.30.100with the actual API VIP of your cluster.Verify outbound network connectivity to the API. It should now route across the network to one of the healthy master nodes:
curl -k -v https://10.0.30.100:443/api
An HTTP response such as
HTTP/2 401or403confirms that the traffic is reaching an active Kubernetes API server. For example:* Connected to 10.0.30.100 (10.0.30.100) port 443 < HTTP/2 401 { "kind": "Status", "message": "Unauthorized", "code": 401 }Reload
systemdto clear the warnings about unit files that have changed on disk:sudo systemctl daemon-reload
Restart the
lcm-agentdaemon to trigger an immediate check-in:sudo systemctl restart lcm-agent*.service
Verify that the node recovers:
On the new node, inspect the agent logs to confirm a successful tick execution:
sudo journalctl -u lcm-agent* -f
In the system response, find the
tick finishedmessages without connection errors.From the management or seed node, verify the machine phase transition:
kubectl get machine,lcmmachine -o wide
The
LCMPHASEfor the new node should transition fromPendingLCMAgenttoPrepare,Deploy, and then toReady.
[42889] Graceful reboot gets stuck when Kubernetes and OpenStack control planes are drained simultaneously
When a GracefulRebootRequest targets both the Kubernetes and OpenStack
control plane machines, either by listing machines of both types in
spec.machines or by leaving the list empty to reboot all cluster nodes,
the rolling reboot may get stuck. This happens because both node groups are
drained in parallel, and the OpenStack workload manager running on the
Kubernetes control plane becomes unavailable while the OpenStack control plane
nodes are simultaneously being drained.
Workaround:
Identify the machines that have not yet been rebooted.
Delete the stuck
GracefulRebootRequest:kubectl -n <projectName> delete gracefulrebootrequest <gracefulRebootRequestName>
Recreate the reboot requests in two sequential steps as described in Perform a rolling reboot of a cluster using CLI: first for the Kubernetes control plane machines, then, once that request completes and is deleted, for the remaining machines that still require a reboot.
Ceph
[67812] Ceph OSD fails to start after a restart
After a restart of a Ceph OSD, for example, during a rolling restart of all
Ceph OSDs, the Ceph OSD may fail to start. A race condition in Ceph corrupts
a small allocation file that the Ceph OSD writes during shutdown. On the next
start, the expand-bluefs init container of the Ceph OSD pod reads that
file, detects one 64 KiB block in an unexpected location, and aborts. The data
stored on the Ceph OSD is not damaged.
A similar behavior is described in the upstream Ceph issue 81213.
The Ceph cluster continues to operate with the affected Ceph OSD in the
down and out state. Other Ceph OSDs, including those located on the
same device, are not affected.
Symptoms:
The
rook-ceph-osd-<ID>pod is stuck in theInit:ErrororInit:CrashLoopBackOffstate with a growing number of restarts.The Rook Ceph Operator logs contain the following error:
failed to update OSD deployment "rook-ceph-osd-<ID>": progress deadline exceeded
The logs of the
expand-bluefsinit container of the affected pod contain an assertion failure. To view the logs:kubectl -n rook-ceph logs <affectedOsdPodName> -c expand-bluefs
Example of system response:
inferring bluefs devices from bluestore path -1 hybrid::operator() lambda Uexpected extent: 0xd705000~10000 HybridAllocator_impl.h: In function 'HybridAllocatorBase<AvlAllocator>::init_rm_free(uint64_t, uint64_t)::<lambda(uint64_t, uint64_t, bool)>' HybridAllocator_impl.h: 103: FAILED ceph_assert(false)
Workaround:
Open the deployment of the affected Ceph OSD for editing:
kubectl -n rook-ceph edit deployment rook-ceph-osd-<ID>
In the
spec.template.spec.initContainerssection, add the following arguments ofceph-bluestore-toolto theexpand-bluefsinit container:initContainers: - name: expand-bluefs args: - bluefs-bdev-expand - --path - /var/lib/ceph/osd/ceph-<ID> - --bluestore-allocator=bitmap - --bluefs-allocator=bitmap
Verify that the affected Ceph OSD pod is
Runningand the Ceph OSD isupandin:kubectl -n rook-ceph get pod -l ceph-osd-id=<ID> kubectl -n rook-ceph exec -it deploy/pelagia-ceph-toolbox -- ceph osd tree
If the issue persistently reproduces on multiple Ceph OSDs of a newly deployed
cluster, configure Ceph OSDs to use the bitmap allocator and to store the
allocation data in RocksDB instead of the allocation file, which eliminates
the race condition:
Open the
CephDeploymentcustom resource for editing:kubectl -n ceph-lcm-mirantis edit cephdeployment
In the
spec.rookConfigsection, add the following parameters:spec: rookConfig: osd|bluestore_allocator: "bitmap" osd|bluefs_allocator: "bitmap" osd|bluestore_allocation_from_file: "false"
For details about
rookConfig, see RookConfig parameters.Redeploy every affected Ceph OSD by removing it and adding it back as described in Add, remove, or reconfigure Ceph OSDs. The
bluestore_allocation_from_fileparameter applies only to redeployed Ceph OSDs.
[62392] False negatives in the ceph_daemon_socket_up metric
When a pod with the ceph-rgw daemon restarts, the stale rgw socket
may persist due to the upstream Ceph issue 76939,
which results in a false-positive failed ceph_daemon_socket_up check in
the ceph-exporter metrics. For example:
ceph_daemon_socket_up{ceph_daemon="ceph-osd.1", hostname="child-worker-1-storage-worker", instance="192.168.9.124:9926", job="prometheus-ceph-exporter"} 1
ceph_daemon_socket_up{ceph_daemon="ceph-client.rgw.openstack.store.a.1.107989349947776", hostname="managed-master-2-control-storage", instance="192.168.9.122:9926", job="prometheus-ceph-exporter"} 0
ceph_daemon_socket_up{ceph_daemon="ceph-osd.0", hostname="child-worker-2-storage-worker", instance="192.168.9.125:9926", job="prometheus-ceph-exporter"} 1
ceph_daemon_socket_up{ceph_daemon="ceph-client.rgw.openstack.store.a.1.95065910507904", hostname="managed-master-1-control-storage", instance="192.168.9.121:9926", job="prometheus-ceph-exporter"} 1
ceph_daemon_socket_up{ceph_daemon="ceph-client.rgw.openstack.store.a.1.104243065772416", hostname="managed-master-2-control-storage", instance="192.168.9.122:9926", job="prometheus-ceph-exporter"} 1
In the example above, ceph-exporter detects a failed socket with the 0
value for the ceph-client.rgw.openstack.store.a.1.107989349947776 daemon,
which has already been replaced by the
ceph-client.rgw.openstack.store.a.1.104243065772416 daemon. To mitigate
the issue, the operator must manually remove the stale socket from the
affected node.
Workaround:
SSH into a node where the affected daemon is placed. In the example above, it is
managed-master-2-control-storage.Locate the stale socket with the
0value under the/var/lib/rook/exporter/directory:ls -l /var/lib/rook/exporter/
Example of system response:
total 0 srwxr-xr-x 1 root root 0 Jun 16 19:32 ceph-client.ceph-exporter.asok srwxr-xr-x 1 167 167 0 Jun 16 19:22 ceph-client.rgw.openstack.store.a.1.104243065772416.asok srwxr-xr-x 1 167 167 0 Jun 16 20:52 ceph-client.rgw.openstack.store.a.1.107989349947776.asok srwxr-xr-x 1 167 167 0 Jun 16 19:29 ceph-mon.a.asok
Remove the stale socket. For example:
rm /var/lib/rook/exporter/ceph-client.rgw.openstack.store.a.1.107989349947776.asok
Cluster update
[68577] MOSK cluster update gets stuck during the MKE upgrade
During the update of a MOSK cluster to 26.2.1 (Cluster release 21.2.1), the
k8s-controllers step of the ClusterUpdatePlan object may get stuck with
the 0 out of 3 machines upgraded message because the MKE upgrade fails to
start.
The issue occurs because the previous cluster update leaves the
ucp-3-7-32-mosk upgrade service in Docker Swarm. During the update to MOSK
26.2.1, the MCR update restarts dockerd on the control plane node that this
service is pinned to, and Docker Swarm reruns the previous MKE upgrade. As a
result, an extra worker agent service is created, which blocks the upgrade to
MKE 3.7.33.
Symptoms:
The
ClusterUpdatePlanobject has theStuckstatus on thek8s-controllersstep.All attempts of the
ucp-3-7-33-moskMKE upgrade service fail with the following error:unable to initialize upgrader: unable to determine blue/green worker agent services: \ both -x and -y worker node agent services exist. Is an upgrade already in progress?
Workaround:
On an MKE manager node of the affected cluster, verify that the MKE upgrade fails with the error described above:
sudo docker service logs --raw ucp-3-7-33-mosk 2>&1 | grep 'both -x and -y worker node agent services exist'
If the output is empty, the update is stuck for a different reason. In this case, do not apply the workaround and contact Mirantis support.
List the worker agent services:
sudo docker service ls | grep ucp-worker-agent
Example of system response:
xg5qz6u9omt6 ucp-worker-agent-x global 0/0 mirantis/ucp-agent:3.7.32-mosk hfaox4q10bb6 ucp-worker-agent-y global 9/9 mirantis/ucp-agent:3.7.32-mosk
On an affected cluster, both services exist and use the image of the MKE version that is not upgraded yet,
ucp-agent:3.7.32-mosk. One service has all replicas running, and the other one has0/0replicas.Remove the service with
0/0replicas, which can be eitherucp-worker-agent-xorucp-worker-agent-y. For example:sudo docker service rm ucp-worker-agent-x
Warning
Do not remove the service with
0/0replicas if it uses the image of the new MKE version,ucp-agent:3.7.33-mosk.Remove the services left from the previous MKE upgrade. Complete this step before the MKE upgrade is retried.
sudo docker service rm ucp-3-7-32-mosk ucp-images-check-3-7-32-mosk
On the MOSK management cluster, increase the number of the MKE upgrade attempts for the affected cluster to retry the upgrade. Set
mkeUpgradeAttemptsto7or, if the parameter is already set, increase its value by at least1:kubectl -n <projectName> patch lcmcluster <clusterName> --type merge -p '{"spec":{"mkeUpgradeAttempts":7}}'
The MOSK cluster update proceeds automatically.
[65903] Management cluster update gets stuck with k0s failing to start
Applies only to clusters deployed using 2.13.x or earlier
Management clusters deployed using an old Container Cloud release, earlier than
2.14.0, may contain an override of the IAM configuration of the
baremetal-provider Helm release in the Cluster object with
iam.enabled: false. This override is invalid and incompatible with the
update to MOSK management 2.32.0. It may cause the update to get stuck with
the k0s control plane nodes failing to start with the following example errors:
err="open /etc/kubernetes/oidc_ca.crt: no such file or directory\"" component=kube-apiserver
msg="Failed to wait for process" component=kube-apiserver error="exit status 1"
To verify whether the cluster is affected:
Log in to a node that contains
kubeconfigof the required management cluster. For example:export KUBECONFIG=/path/to/mgmt-kubeconfig
Verify whether the
baremetal-providerHelm release containsiam.enabled: false:kubectl get cluster <mgmtClusterName> -o jsonpath='{.spec.providerSpec.value.kaas.regional[?(@.provider=="baremetal")].helmReleases[?(@.name=="baremetal-provider")].values.config.iam}'
Example output of the affected cluster:
{"enabled":false,"registrationUrl":"<ip>:8082"}
If the output is empty or does not contain
"enabled":false, the cluster is not affected and you can skip this procedure. Otherwise, proceed to the next step.Verify that
HelmBundleof the management cluster has IAM enabled for thebaremetal-providerrelease:kubectl -n default get helmbundle <mgmtClusterName> -o jsonpath='{.spec.releases[?(@.name=="baremetal-provider")].values.config.iam}'
Example output of the affected cluster:
{"enabled":true,"registrationUrl":"<ip>:8082"}
Example of the affected
HelmBundlesnippet:spec: releases: ... - name: baremetal-provider namespace: kaas values: ... config: ... iam: enabled: true registrationUrl: <ip>:8082
If the output is empty, does not contain
iam.enabled, or contains"enabled":false, contact Mirantis support. Do not proceed to the next step.Verify whether the
Clusterobject contains the Keycloak TLS configuration:kubectl get cluster <mgmtClusterName> -o jsonpath='{.spec.providerSpec.value.tls.keycloak.tlsConfigRef}'
Example of the affected
Clusterobject:spec: ... providerSpec: value: ... tls: ... keycloak: tlsConfigRef: keycloak
If the output is empty, contact Mirantis support. Do not proceed with the workaround below.
Workaround:
If the Keycloak TLS configuration is present and the HelmBundle has
iam.enabled: true, remove enabled: false from the Cluster object:
Open the
Clusterobject of the affected management cluster for editing:kubectl edit cluster <mgmtClusterName>
In the
spec.providerSpec.value.kaas.regionalsection, find thebaremetal-providerHelm release and delete only theenabled: falseline undervalues.config.iam.Caution
Do not remove
registrationUrlor other IAM parameters.spec: ... providerSpec: value: ... kaas: regional: - provider: baremetal helmReleases: - name: baremetal-provider values: ... config: iam: enabled: false # delete this line registrationUrl: <ip>:8082
Save and exit the object to apply changes.
Warning
After you remove the override, the management cluster nodes may be sequentially cordoned, drained, and updated with the new configuration. Do not start the update to MOSK management 2.32.0 until the cluster becomes ready.
Wait until every management cluster node completes the configuration update and the cluster becomes ready:
kubectl get cluster <mgmtClusterName> -o jsonpath='{.status.providerStatus.ready}{"\n"}'
The command must return
true, which means the cluster is ready for the update to MOSK management 2.32.0.
[65106] Management cluster update hangs due to Keycloak database schema issue
A management cluster update may get stuck with the following error:
keycloak encountered an exception.: liquibase.exception.DatabaseException: (conn=538) \
Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. \
This includes storage overhead, check the manual. \
You have to change some columns to TEXT or BLOBs \
[Failed SQL: (1118) ALTER TABLE keycloak.CREDENTIAL ADD VERSION INT DEFAULT 0 NULL]
Workaround:
Log in to the MariaDB console as admin:
kubectl -n kaas exec -it mariadb-server-0 -- bash -c 'mysql -uroot -p$MYSQL_DBADMIN_PASSWORD'
Optimize the affected table:
OPTIMIZE TABLE keycloak.CREDENTIAL;
[64721] MOSK cluster upgrade gets stuck due to runc-ee package downgrade
Clusters deployed with MOSK 26.1 (Cluster release 21.1.0) starting from
2026-08-04 may have issues upgrading to more recent versions, such as MOSK
26.1.x (Cluster releases 21.1.x) patch releases or MOSK 26.2 (Cluster release
21.2.0). The issue occurs during lcm-ansible execution at the step that
installs the runc-ee package.
Symptom: the first control plane node gets stuck in the deploy phase
with the following error in the lcm-ansible logs:
TASK [containerd : Install containerd.io packages]
task path: .../lcm-ansible-<version>/roles/containerd/tasks/Debian.yml:62
The following packages will be upgraded:
containerd.io* → 1.7.31m1+fips-0ubuntu0.24.04.1
The following packages will be DOWNGRADED:
runc-ee → 1.4.2m1-0ubuntu0.24.04.1
E: Packages were downgraded and -y was used without --allow-downgrades.
The issue occurs because the runc-ee package version was not pinned and was
installed from a constantly updated repository in MOSK 26.1 (Cluster release
21.1.0). Starting from MOSK 26.1.x (Cluster releases 21.1.x) patch releases and
MOSK 26.2 (Cluster release 21.2.0), the package version is pinned.
Workaround:
SSH to any node of the affected cluster.
Install the pinned version of the
runc-eepackage:apt-get --allow-downgrades install runc-ee=1.4.2m1-0ubuntu0.24.04.1 -y --allow-change-held-packages
Hold the package to prevent future automatic updates:
apt-mark hold runc-ee
Repeat steps 1-3 for the remaining nodes of the affected cluster.
MOSK management cluster
[64703] Missing WireGuard interface address on management cluster nodes
On new management clusters based on Cluster release 22.0.0 (MOSK 26.2) that
have secureOverlay set to true, some management cluster nodes may be
missing the WireGuard interface address. As a result, kube-apiserver or
any other process running in the host network namespace on an affected node
cannot reach pods running on healthy nodes.
Symptom: webhook calls handled by kube-apiserver on the affected node
fail with a timeout if the pod serving the webhook runs on a healthy node. The
error propagates back to the client that issued the original API request.
The following example illustrates a cert-manager log entry produced when
the cert-manager pod sends a request to kube-apiserver on the affected
node, while cert-manager-webhook runs on a healthy node:
E0817 23:38:37.501136 1 controller.go:167] cert-manager/issuers "msg"="re-queuing item due to error processing" "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.kaas.svc:443/mutate?timeout=10s\": context deadline exceeded" "key"="kaas/mcc-ca"
The following example illustrates the corresponding k0scontroller log
entry on the affected node:
Aug 17 23:38:37 master-2 k0s[45673]: time="2026-08-17 23:38:37" level=info msg="W0817 23:38:37.496457 45763 dispatcher.go:225] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.kaas.svc:443/mutate?timeout=10s\": context deadline exceeded" component=kube-apiserver stream=stderr
Workaround:
Identify the affected nodes that have no IP address in the output of the following command:
kubectl get node -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.projectcalico\.org/IPv4WireguardInterfaceAddr}{"\n"}{end}'
Identify the names of the
calico-nodepods running on the affected nodes:kubectl get pod -n kube-system -o wide | grep calico-node
Delete the affected pods to trigger their restart:
kubectl delete pod -n kube-system <calico-node-pod-name>
Repeat steps 1-3 until all nodes have the WireGuard interface address assigned.
[64454] Node gets stuck in the deploy phase
During bootstrap of a MOSK management cluster, a master node may get stuck in
the deploy phase and never become Ready, even though the LCM deploy
playbook has completed successfully.
Symptoms:
The
lcm-agentlogs on the affected nodes, including nodes that are not stuck, contain the following error:{"level":"error","msg":"failed to check k0s manager health","error":"local etcd member health check failed: failed to get etcd member master-1: etcdmembers.etcd.k0sproject.io \"master-1\" not found"}No
etcdmemberobjects exist for any node, even though etcd itself is healthy:kubectl get etcdmembers.etcd.k0sproject.io -A
Example of system response:
No resources found
Workaround:
Restart the k0scontroller service on one of the affected nodes:
systemctl restart k0scontroller
Wait for the node to become Ready and for its deployment to finish
before proceeding to the next one. The remaining affected nodes will likely
finish deployment successfully without requiring the same restart. Otherwise,
restart the service on each remaining affected node as well.
[64407] MariaDB restore from backup causes Keycloak failure
During a full MariaDB unavailability caused by the restore procedure,
Keycloak’s internal clustering mechanism cannot re-establish the cluster
state, leaving the affected pods in the NOT READY condition even
after MariaDB becomes available again.
Symptoms: the Keycloak pod logs contain the following errors:
ERROR [org.keycloak.jgroups.protocol.KEYCLOAK_JDBC_PING2] (Timer runner-2,iam-keycloak-0-4892(v=16.0.8)) iam-keycloak-0-4892(v=16.0.8): failed reading from the DB: java.sql.SQLNonTransientConnectionException: Socket fail to connect to mariadb. Connection refused
Caused by: java.net.ConnectException: Connection refused
at org.mariadb.jdbc.client.impl.ConnectionHelper.connectSocket(ConnectionHelper.java:118)
... 12 more
INFO [io.smallrye.health] (vert.x-eventloop-thread-0) SRHCK01001: Reporting health down status: {"status":"DOWN","checks":[{"name":"Graceful Shutdown","status":"UP"},{"name":"Keycloak database connections async health check","status":"UP"},{"name":"Keycloak Initialized","status":"UP"},{"name":"Keycloak cluster health check","status":"DOWN","data":{"Failing since":"2026-08-17 17:38:26,222"}}]}
Workaround:
Verify that the restore procedure completed successfully and that all MariaDB replicas are up and ready.
Identify the stuck Keycloak pods:
kubectl get pods -n kaas | grep iam-keycloak
Delete each stuck pod:
kubectl delete pod -n kaas <IAM_KEYCLOAK_POD_NAME>
Verify that the newly created Keycloak pods are up and in the
READYstate:kubectl get pods -n kaas | grep iam-keycloak
MOSK management console
[50168] Inability to use a new project right after creation
A newly created project does not display all available tabs in the MOSK
management console and contains different access denied errors during first
five minutes after creation.
To work around the issue, refresh the browser in five minutes after the project creation.
Networking
[64752] LoadBalancer IPs flap after a MetalLB speaker node shutdown
On clusters using MetalLB in L2 mode, an unexpected shutdown of a node running a MetalLB speaker can cause LoadBalancer service IP addresses to flap instead of failing over to a live node. The issue can occur on a management cluster or a MOSK cluster, on any node hosting a speaker, whether a control plane node or a worker node. The condition does not self-recover while the node remains down.
Affected services are those of type LoadBalancer that MetalLB
advertises in L2 mode. For example, on the management cluster these include
iam-keycloak-http, kaas-kaas-ui, and iam-proxy-*. As a result,
access to Keycloak, the MOSK management console, and StackLight IAM proxies
(Grafana, Prometheus, Alerta, Alertmanager, Kibana) becomes intermittently
unavailable. On a MOSK cluster, the same failure mode affects the
LoadBalancer services announced on the powered-off speaker node.
If the affected cluster is the management cluster, support dump collection
may also fail because the systemLogs collector cannot complete:
failed to collect system logs: failed waiting for calls to execute
Symptoms:
After a hard power-off of a node running a MetalLB speaker, the external
IP addresses of LoadBalancer services become intermittently reachable.
Reachability can change every few seconds. For example, TCP probes to
<ip>:443 from a live node, taken a few seconds apart, may look as
follows:
Service IP Round 0 Round 1 Round 2
iam-keycloak-http 172.16.61.24 OPEN CLOSED OPEN
kaas-kaas-ui 172.16.61.11 CLOSED CLOSED OPEN
iam-proxy-alerta 172.16.61.13 CLOSED OPEN OPEN
iam-proxy-grafana 172.16.61.15 OPEN CLOSED OPEN
iam-proxy-prometheus 172.16.61.17 CLOSED CLOSED CLOSED
The MetalLB speaker pods on the surviving nodes log a memberlist join failure approximately every 60 seconds:
{"caller":"speakerlist.go:269","op":"memberDiscovery","level":"error",
"msg":"partial join","expected":1,"joined":0,
"error":"Failed to join 172.16.61.26:7947: dial tcp 172.16.61.26:7947: connect: no route to host"}
The ServiceL2Status objects still allocate LoadBalancer IP addresses to
the powered-off node (kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587 in
the example below):
kubectl -n metallb-system get servicel2status
Example of system response:
NAME ALLOCATED NODE SERVICE NAME SERVICE NAMESPACE
l2-fwbbn kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587 ironic-kaas-bm kaas
l2-kx245 kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587 ironic-syslog kaas
l2-m74nf kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587 iam-proxy-prometheus stacklight
l2-ndsz9 kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587 httpd-http kaas
l2-x6tw2 kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587 dhcp-lb kaas
l2-ztpjw kaas-node-b889e3ae-9da5-4049-b33e-d219a7176587 iam-proxy-grafana stacklight
Service backends remain healthy: externalTrafficPolicy is Cluster,
endpoints contain ready pods on the live nodes, and direct connections to
backend pod IPs succeed. This rules out a backend failure.
The MetalLB speaker pod on the powered-off node remains Running
(1/1 Ready). Pod status is push-based: when the node is powered off,
kubelet dies and stops reporting container state, so the API server keeps
the last known status. The node is tainted with
node.kubernetes.io/unreachable:NoExecute, but the speaker DaemonSet
tolerates that taint indefinitely and Kubernetes never evicts the pod.
MetalLB determines L2 IP ownership from speaker pods that Kubernetes still
reports as Ready. Because the frozen pod still appears healthy, MetalLB
treats the dead node as an eligible L2 speaker. Speaker membership never
converges, L2 IP ownership keeps being re-elected, and gratuitous ARP
announcements flap.
Workaround:
Force-delete the frozen MetalLB speaker pod on the powered-off node:
Using the kubeconfig of the affected cluster, identify the speaker pod on the powered-off node:
kubectl -n metallb-system get pod -o wide | grep speaker
The pod on the unreachable node remains
Runningwhile the node isNotReady.Force-delete that speaker pod:
Note
A graceful deletion cannot complete because
kubeleton the powered-off node is not running.kubectl -n metallb-system delete pod <speakerPodName> --force --grace-period=0
Kubernetes immediately removes the frozen pod object. The DaemonSet creates a replacement pod that cannot become
Readyon the dead node, so MetalLB stops considering that node for L2 IP ownership.Verify that
ServiceL2Statusobjects no longer list the powered-off node as the allocated node and that the LoadBalancer IP addresses stop flapping:kubectl -n metallb-system get servicel2status
OpenSDN
[68383] vRouter kernel module is not updated after OpenSDN upgrade to 25.1
Applies only to the upgrade of OpenSDN from 24.1 to 25.1
After the upgrade of OpenSDN from 24.1 to 25.1, the OpenSDN 24.1 vrouter
kernel module remains loaded and installed on compute nodes, although all
OpenSDN pods run 25.1 images and the TFOperator custom resource reports the
upgrade as completed. As a result, the vRouter Agent of OpenSDN 25.1 runs on
top of the 24.1 datapath. The issue persists after node reboot.
The issue occurs because Dynamic Kernel Module Support (DKMS) treats the
version of the 24.1 module as newer than the 25.1 one and does not
install the new module, while the kernel-build-init container of the
tf-vrouter-agent pod completes successfully despite the failure.
Symptoms:
The logs of the kernel-build-init container contain the following error:
Error! Module version 26.2.r25.1.<buildID> for vrouter.ko.zst
is not newer than what is already found in kernel <kernelVersion> (v0+26.2.r24.1.<buildID>).
To verify whether a compute node is affected:
Obtain the name of the
tf-vrouter-agentpod running on the node:kubectl -n tf get pods -l app=tf-vrouter-agent -o wide
Verify the version of the
vrouterkernel module:POD=<tfVrouterAgentPodName> kubectl -n tf exec $POD -c agent -- sh -c \ 'cat /sys/module/vrouter/version; for f in /lib/modules/*/updates/dkms/vrouter.ko*; do echo "$f $(modinfo -F version $f)"; done'
Example of system response on an affected node:
v0+26.2.r24.1.20261103122747 /lib/modules/6.8.0-134-generic/updates/dkms/vrouter.ko v0+26.2.r24.1.20261103122747 /lib/modules/6.8.0-147-generic/updates/dkms/vrouter.ko v0+26.2.r24.1.20261103122747
The node is affected if the version of the loaded or installed module starts with
v0+(OpenSDN 24.1), while the agent image has the26.2-r25.1.<buildID>tag.
Workaround:
Perform the following procedure on one compute node at a time. Restarting the vRouter Agent briefly interrupts the workload traffic on the node, as during a regular vRouter update.
Remove the stale 24.1 module files for all kernels. The loaded module is not affected.
kubectl -n tf exec $POD -c agent -- sh -c 'rm -fv /lib/modules/*/updates/dkms/vrouter.ko*'
Restart the vRouter Agent:
kubectl -n tf delete pod $POD
The
kernel-build-initcontainer unloads the 24.1 module, then builds and installs the 25.1 module for all installed kernels.Once the new pod is
Ready, verify the module version:POD=<newTfVrouterAgentPodName> kubectl -n tf exec $POD -c agent -- sh -c \ 'cat /sys/module/vrouter/version; for f in /lib/modules/*/updates/dkms/vrouter.ko*; do echo "$f $(modinfo -F version $f)"; done'
Both the loaded and installed versions must match the image version of the vRouter Agent. For example,
26.2.r25.1.20261103113743for the26.2-r25.1.20261103113743image.If the module files on disk have the 25.1 version but the loaded module still has the
v0+version, the 24.1 module could not be unloaded. In this case, drain and reboot the compute node as described in Perform a graceful reboot of a cluster. The 25.1 module is loaded from disk at boot.
[53818] Octavia load balancer is not functional after update
After a cluster update, an Octavia load balancer that uses the OpenSDN
(tungstenfabric) provider may stop passing traffic, while both Octavia and
OpenSDN keep reporting it as healthy. The load balancer members remain
reachable by their own addresses, but the virtual IP address (VIP) of the load
balancer does not respond and does not answer ARP requests.
The issue occurs because the load balancer network namespace fails to be
recreated after the restart of the tf-vrouter-agent pod during the update:
the name of the veth interface is still occupied by a stale interface left
from the previous run. As a result, no HAProxy process is started for the
affected load balancer. The failure is not logged, and the agent does not
retry it.
Workaround:
Verify that the VIP of the load balancer does not respond while its members do, for example, from the
keystone-clientpod or from an instance in the same tenant network:curl <loadBalancerVIP>:<port> curl <memberIP>:<port>
Identify the compute nodes that host the service instances of the affected load balancer:
kubectl -n openstack exec -it <keystoneClientPodName> -- \ openstack loadbalancer show <loadBalancerID>
On each OpenSDN compute node, the presence of the HAProxy process for the load balancer indicates whether the node is affected:
ps -ef | grep "haproxy -f /var/lib/contrail/loadbalancer/haproxy"
The node is affected if the output contains no entry with the ID of the load balancer, while the
/var/lib/contrail/loadbalancer/<loadBalancerID>.confconfiguration file exists on the node.Restart the
tf-vrouter-agentpod on each affected node:kubectl -n tf delete pod <tfVrouterAgentPodName>
Wait until the pod becomes ready and verify that the VIP of the load balancer responds again:
curl <loadBalancerVIP>:<port>
[42896] Cassandra cluster contains extra node with outdated IP after replacement of TF control node
After replacing a failed Tungsten Fabric controller node as described in Replace a failed OpenSDN controller node, the first restart of the Cassandra pod on this node may cause an issue if the Cassandra node with the outdated IP address has not been removed from the cluster. Subsequent Cassandra pod restarts should not trigger this problem.
To verify if your Cassandra cluster is affected, run the
nodetool status command specifying the config or analytics cluster
and the replica number:
kubectl -n tf exec -it tf-cassandra-<CONFIG-OR-ANALYTICS>-dc1-rack1-<REPLICA-NUM> -c cassandra -- nodetool status
Example of the system response with outdated IP addresses:
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 192.168.201.144 509.43 KiB 256 ? 7e760a99-fae5-4921-b0c5-d9e6e1eca1c5 rack1
UN 192.168.50.146 534.18 KiB 256 ? 2248ea35-85d4-4887-820b-1fac4733021f rack1
UN 192.168.145.147 484.19 KiB 256 ? d988aaaa-44ae-4fec-a617-0b0a253e736d rack1
DN 192.168.145.144 481.53 KiB 256 ? c23703a1-6854-47a7-a4a2-af649d63af0c rack1
An extra node will appear in the cluster with an outdated IP address
(the IP of the terminated Cassandra pod) in the Down state.
To work around the issue, after replacing the Tungsten Fabric
controller node, delete the Cassandra pod on the replaced node and remove
the outdated node from the Cassandra cluster using nodetool:
kubectl -n tf exec -it tf-cassandra-<CONFIG-OR-ANALYTICS>-dc1-rack1-<REPLICA-NUM> -c cassandra -- nodetool removenode <HOST-ID>
[40032] tf-rabbitmq fails to start after rolling reboot
Occasionally, RabbitMQ instances in tf-rabbitmq pods fail to enable
the tracking_records_in_ets during the initialization process.
To work around the issue, restart the affected pods manually.
OpenStack
[57473] OpenStack update fails due to neutron-ovs-agent-default start failure
During OpenStack update from Caracal to Epoxy, neutron-ovs-agent-default
may fail to start with the The DaemonSet neutron-ovs-agent-default is not
ready error in the Rockoon logs due to Kopf missing the OsDpl update
events.
As a workaround, recreate the rockoon pod of the affected MOSK cluster:
kubectl -n osh-system rollout restart deployment rockoon
[53401] Credential rotation reports success without performing action
Occasionally, the password rotation procedure for admin or service
credentials may incorrectly report success without actually initiating
the rotation process. This can result in unchanged credentials despite
the procedure indicating completion.
To work around the issue, restart the rotation procedure and verify that the credentials have been successfully updated.
[63801] Encrypted ephemeral VM disk becomes undecryptable
Rebooting a virtual machine (VM) with encrypted ephemeral storage after upgrading from OpenStack Caracal to Epoxy makes its data undecryptable.
Symptoms:
After upgrade from OpenStack Caracal to Epoxy, a VM can no longer access existing data on its encrypted ephemeral disk after a hard reboot or a stop/start. This issue affects VMs that meet both of the following conditions:
A VM uses an encrypted ephemeral storage that was created on OpenStack Caracal or earlier
A VM was created on OpenStack Caracal, and OpenStack was later upgraded to Epoxy
Warning
The data is not lost and remains recoverable as long as no application inside the affected VM writes to the ephemeral disk, for example, by reformatting it or re-creating partitions.
Cause:
OpenStack Caracal container images in MOSK are based on Ubuntu 22.04.
OpenStack Epoxy images are based on Ubuntu 24.04, which includes
cryptsetup 2.7. This cryptsetup version changes the default
hashing algorithm for disk encryption from ripemd160 to sha256.
For details, see Cryptsetup 2.7.0 Release Notes.
OpenStack Nova does not explicitly set the hashing algorithm, as per upstream known issue #1639221, when it configures disk encryption. As a result, after the upgrade to Epoxy, Nova sets up encryption with different parameters than those used when the disk was originally created, which breaks decryption. MOSK does not automatically re-encrypt affected disks with the original hashing algorithm.
Workaround:
MOSK 26.2 includes fixes that let you control the hashing algorithm that Nova uses for disk encryption:
Nova now explicitly specifies the hashing algorithm when it creates disk encryption. On Epoxy, the default is
sha256, matching thecryptsetupdefault in Epoxy-based MOSK images. This default applies to all VMs on a given compute node. You can override it in the configuration of anynova-computeservice.You can also override the hashing algorithm for an individual VM using the
dmcrypt_hashinstance metadata key.
To fix VMs that are already broken or at risk of breaking after the upgrade to Epoxy:
Add the following metadata to each affected instance:
openstack server set --property dmcrypt_hash=ripemd160 <INSTANCE-ID>
If the instance is already broken (can not read its ephemeral disk), power-cycle the instance.
The disk becomes decryptable again.
To prevent existing VMs from breaking before you upgrade from Caracal to Epoxy:
While still on Caracal, set the following configuration option to
ripemd160. The setting has no effect until you upgrade to Epoxy.kind: OpenStackDeployment spec: services: compute: nova: values: conf: nova: ephemeral_storage_encryption: hash: ripemd160
Upgrade to Epoxy.
Existing VMs continue to work without requiring per-instance
metadata. New VMs also default to ripemd160, which is a less
secure hashing algorithm than the sha256 default introduced in
Epoxy.
[52320] Heat operations fail with Timed out waiting for a reply to message ID
Epoxy only
Under sustained or heavy load, Heat stack operations may fail with the following error:
CREATE_FAILED Resource CREATE failed: MessagingTimeout: resources.<RESOURCE>:
Timed out waiting for a reply to message ID <ID>
The heat-engine logs also contain the following error:
AssertionError: do not call blocking functions from the mainloop
The issue is caused by an incompatibility between eventlet and the
SQLAlchemy 2.0-style connection handling used by oslo.db in the
OpenStack Epoxy releases of Heat, which can exhaust the database
connection pool and cause RPC calls to time out. The issue is
most likely to occur when a large number of stacks, or deeply nested
stacks, are created, updated, or deleted concurrently.
As a workaround, periodically restart the heat-engine pods to
release exhausted database connections and clear the timed-out RPC calls:
kubectl -n openstack rollout restart deployment heat-engine
StackLight
[55317] The Dropped sample for series errors in the Prometheus logs
When the experimental feature memory-snapshot-on-shutdown, which is enabled
by default, is used together with remote_write, Prometheus may emit
multiple log messages, such as Dropped sample for series that was not
explicitly dropped via relabelling. For more details, see the upstream issue
description in the Prometheus GitHub project.
Workaround:
On the related management cluster, open the affected MOSK
Clusterobject for editing:kubectl edit cluster <affectedMOSKClusterName> -n <affectedMOSKClusterProjectName>
Remove the
memory-snapshot-on-shutdownfeature from theprometheusServer:enabledFeatureslist:spec: ... providerSpec: ... value: ... helmReleases: ... - name: stacklight values: ... prometheusServer: enabledFeatures: []