Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.3.0 including the Cluster release 5.10.0.


This section also outlines still valid known issues from previous Container Cloud releases.


[8013] Managed cluster deployment requiring PVs may fail

Fixed in the Cluster release 7.0.0


The issue below affects only the Kubernetes 1.18 deployments. Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshooting.

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.


The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.


  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'


    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...

Bare metal

[6988] LVM fails to deploy if the volume group name already exists

Fixed in Container Cloud 2.5.0

During a management or managed cluster deployment, LVM cannot be deployed on a new disk if an old volume group with the same name already exists on the target hardware node but on the different disk.


In the bare metal host profile specific to your hardware configuration, add the wipe: true parameter to the device that fails to be deployed. For the procedure details, see Operations Guide: Create a custom host profile.

[7655] Wrong status for an incorrectly configured L2 template

Fixed in 2.11.0

If an L2 template is configured incorrectly, a bare metal cluster is deployed successfully but with the runtime errors in the IpamHost object.


If you suspect that the machine is not working properly because of incorrect network configuration, verify the status of the corresponding IpamHost object. Inspect the l2RenderResult and ipAllocationResult object fields for error messages.

[8560] Manual deletion of BareMetalHost leads to its silent removal

Fixed in Container Cloud 2.5.0

If BareMetalHost is manually removed from a managed cluster, it is silently removed without a power-off and deprovision that leads to a managed cluster failures.


Do not manually delete a BareMetalHost that has the Provisioned status.

[9875] Full preflight fails with a timeout waiting for BareMetalHost

Fixed in Container Cloud 2.6.0

If you run preflight with KAAS_BM_FULL_PREFLIGHT=true, the script fails with the following message:

failed to create BareMetal objects: failed to wait for objects of kinds BareMetalHost
to become available: timed out waiting for the condition

As a workaround, unset full preflight using unset KAAS_BM_FULL_PREFLIGHT to run fast preflight instead.


[2757] IAM fails to start during management cluster deployment

Fixed in Container Cloud 2.4.0

During a management cluster deployment, IAM fails to start with the IAM pods being in the CrashLoopBackOff status.


  1. Log in to the bootstrap node.

  2. Remove the iam-mariadb-state configmap:

    kubectl delete cm -n kaas iam-mariadb-state
  3. Manually delete the mariadb pods:

    kubectl delete po -n kaas mariadb-server-{0,1,2}

    Wait for the pods to start. If the mariadb pod does not start with the connection to peer timed out exception, repeat the step 2.

  4. Obtain the MariaDB database admin password:

    kubectl get secrets -n kaas mariadb-dbadmin-password \
    -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d ; echo
  5. Log in to MariaDB:

    kubectl exec -it -n kaas mariadb-server-0 -- bash -c 'mysql -uroot -p<mysqlDbadminPassword>'

    Substitute <mysqlDbadminPassword> with the corresponding value obtained in the previous step.

  6. Run the following command:

  7. Manually delete the Keycloak pods:

    kubectl delete po -n kaas iam-keycloak-{0,1,2}


[10029] Authentication fails with the 401 Unauthorized error

Authentication may not work on some controller nodes after a managed cluster creation. As a result, the Kubernetes API operations with the managed cluster kubeconfig fail with Response Status: 401 Unauthorized.

As a workaround, manually restart the ucp-controller and ucp-auth Docker services on the affected node.


Moving forward, the workaround for this issue will be moved from Release Notes to Operations Guide: Troubleshooting.

Management cluster update

[9899] Helm releases get stuck in PENDING_UPGRADE during cluster update

Fixed in 2.14.0

Helm releases may get stuck in the PENDING_UPGRADE status during a management or managed cluster upgrade. The HelmBundle Controller cannot recover from this state and requires manual actions. The workaround below describes the recovery process for the openstack-operator release that stuck during a managed cluster update. Use it as an example for other Helm releases as required.


  1. Log in to the helm-controller pod console:

    kubectl exec -n kube-system -it helm-controller-0 sh -c tiller
  2. Identify the release that stuck in the PENDING_UPGRADE status. For example:

    ./helm --host=localhost:44134 history openstack-operator

    Example of system response:

    REVISION  UPDATED                   STATUS           CHART                      DESCRIPTION
    1         Tue Dec 15 12:30:41 2020  SUPERSEDED       openstack-operator-0.3.9   Install complete
    2         Tue Dec 15 12:32:05 2020  SUPERSEDED       openstack-operator-0.3.9   Upgrade complete
    3         Tue Dec 15 16:24:47 2020  PENDING_UPGRADE  openstack-operator-0.3.18  Preparing upgrade
  3. Roll back the failed release to the previous revision:

    1. Download the Helm v3 binary. For details, see official Helm documentation.

    2. Roll back the failed release:

      helm rollback <failed-release-name>

      For example:

      helm rollback openstack-operator 2

    Once done, the release will be reconciled.


[10060] Ceph OSD node removal fails

Fixed in Container Cloud 2.7.0

A Ceph node removal is not being triggered properly after updating the KaasCephCluster custom resource (CR). Both management and managed clusters are affected.


  1. Remove the parameters for a Ceph OSD from the KaasCephCluster CR as described in Operations Guide: Add, remove, or reconfigure Ceph nodes.

  2. Obtain the IDs of the osd and mon services that are located on the old node:

    1. Obtain the UID of the affected machine:

      kubectl get machine <CephOSDNodeName> -n <ManagedClusterProjectName> -o jsonpath='{.metadata.annotations.kaas\.mirantis\.com\/uid}'
    2. Export kubeconfig of your managed cluster. For example:

      export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
    3. Identify the pods IDs that run the osd and mon services:

      kubectl get pods -o wide -n rook-ceph | grep <affectedMachineUID> | grep -E "mon|osd"

      Example of the system response extract:

      rook-ceph-mon-c-7bbc5d757d-5bpws                              1/1  Running    1  6h1m
      rook-ceph-osd-2-58775d5568-5lklw                              1/1  Running    4  44h
      rook-ceph-osd-prepare-705ae6c647cfdac928c63b63e2e2e647-qn4m9  0/1  Completed  0  94s

      The pods IDs include the osd or mon services IDs. In the example system response above, the osd ID is 2 and the mon ID is c.

  3. Delete the deployments of the osd and mon services obtained in the previous step:

    kubectl delete deployment rook-ceph-osd(mon)-<ID> -n rook-ceph

    For example:

    kubectl delete deployment rook-ceph-mon-c -n rook-ceph
    kubectl delete deployment rook-ceph-osd-2 -n rook-ceph
  4. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0]}') bash
  5. Rebalance the Ceph OSDs:

    ceph osd out osd(s).ID

    Wait for the rebalance to complete.

  6. Rebalance the Ceph data:

    ceph osd purge osd(s).ID

    Wait for the Ceph data to rebalance.

  7. Remove the old node from the Ceph OSD tree:

    ceph osd crush rm <NodeName>
  8. If the removed node contained mon services, remove them:

    ceph mon rm <monID>

[9928] Ceph rebalance during a managed cluster update

Fixed in Container Cloud 2.5.0

During a managed cluster update, Ceph rebalance leading to data loss may occur.


  1. Before updating a managed cluster:

    1. Log in to the ceph-tools pod:

      kubectl -n rook-ceph exec -it <ceph-tools-pod-name> bash
    2. Set the noout flag:

      ceph osd set noout
  2. Update a managed cluster.

  3. After updating a managed cluster:

    1. Log in to the ceph-tools pod:

      kubectl -n rook-ceph exec -it <ceph-tools-pod-name> bash
    2. Unset the noout flag:

      ceph osd unset noout

[7073] Cannot automatically remove a Ceph node

Fixed in 2.16.0

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.

[10050] Ceph OSD pod is in the CrashLoopBackOff state after disk replacement

Fixed in 2.11.0

If you use a custom BareMetalHostProfile, after disk replacement on a Ceph OSD, the Ceph OSD pod switches to the CrashLoopBackOff state due to the Ceph OSD authorization key failing to be created properly.


  1. Export kubeconfig of your managed cluster. For example:

    export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
  2. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0]}') bash
  3. Delete the authorization key for the failed Ceph OSD:

    ceph auth del osd.<ID>
  4. SSH to the node on which the Ceph OSD cannot be created.

  5. Clean up the disk that will be a base for the failed Ceph OSD. For details, see official Rook documentation.


    Ignore failures of the sgdisk --zap-all $DISK and blkdiscard $DISK commands if any.

  6. On the managed cluster, restart Rook Operator:

    kubectl -n rook-ceph delete pod -l app=rook-ceph-operator

Container Cloud web UI

[249] A newly created project does not display in the Container Cloud web UI

Affects only Container Cloud 2.18.0 and earlier

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.