Mirantis Container Cloud Release Notes latest documentation

Mirantis Container Cloud Release Notes

Releases summary

Container Cloud release

Release date

Supported Cluster releases

Summary

2.5.0 (current)

March 1, 2021

  • Support for Mirantis Kubernetes Engine 3.3.6

  • Support for Mirantis OpenStack for Kubernetes 21.1

  • Proxy support for OpenStack and VMWare vSphere providers

  • NTP server configuration on regional clusters

  • Optimized ClusterRelease upgrade process

  • Dedicated network for external connection to the Kubernetes services on bare metal

  • Ceph RADOS Gateway HA

  • Ceph RADOS Gateway check box in Container Cloud web UI

  • Ceph maintenance label

  • Cerebro support for StackLight

  • Proxy support for StackLight

2.4.0

February 2, 2021

  • Support for the updated version of Mirantis Container Runtime 19.03.14

  • Dedicated network for Kubernetes pods traffic on bare metal clusters

  • Improvements for the feedback form in the Container Cloud web UI

  • StackLight enhancements:

    • Alert inhibition rules

    • Integration between Grafana and Kibana

    • New Telegraf alert TelegrafGatherErrors

    • Configuration of Ironic Telegraf input plugin

    • Automatically defined cluster ID

2.3.0

December 23, 2020

  • Support for Mirantis Kubernetes Engine 3.3.4 and Mirantis Container Runtime 19.03.13

  • Support for multiple host-specific L2 templates per a bare metal cluster

  • Additional regional cluster on VMWare vSphere

  • Automated setup of an OVF template for the VMWare vSphere provider

  • StackLight support for VMWare vSphere

  • Improvements in the Container Cloud logs collection

2.2.0

November 5, 2020

  • Support for VMWare vSphere provider on RHEL

  • Kernel parameters management through BareMetalHostProfile

  • Support for multiple subnets per cluster

  • Optimization of the Container Cloud logs collection

  • Container Cloud API documentation for bare metal

* - the Cluster release version is deprecated and will become unsupported in one of the following Container Cloud releases. For details about unsupported releases, see Container Cloud releases history.

Product highlights

Mirantis Container Cloud enables you to create and manage Container Cloud clusters across multiple cloud and infrastructure platforms. Currently supported architectures include OpenStack, bare metal, AWS, and VMWare vSphere Technology Preview with more choice coming in the near future.

Container Cloud provides a web UI, CLI, and API to enable a self-service deployment of Container Cloud with Mirantis Kubernetes Engine (MKE) container clusters.

The major features of the Container Cloud GA release include:

  • Container Cloud with MKE multi-cluster
    • Create new fully managed Container Cloud with MKE container clusters on the Container Cloud supported infrastructure and operating system

    • Attach existing MKE clusters (starting from version 3.3.3) that run on any infrastructure and operating system with a core set of most critical LCM operations

  • Automated software LCM

    Provide automated zero-touch update for Container Cloud and MKE, including Ubuntu operating system updates, with no workload impact

  • Multi-cloud infrastructure support

    Deploy Container Cloud with MKE container clusters through automated provisioning on the following infrastructure platforms: OpenStack, VMWare vSphere Technology Preview, AWS, and bare metal

  • Logging, monitoring, alerting

    Monitor individual managed Container Cloud with MKE container clusters as well as provide cluster metrics and alerts on resource utilization aggregated to the Container Cloud level

  • Bare Metal as a Service

    Provision pools of bare metal machines to use as a bare metal provider, which reduces layers of virtualization and provides efficient support for compute-intensive workloads

  • Ceph storage for bare metal

    Deploy Ceph utilizing Rook to provide and manage a robust and scalable persistent storage that can be used by Kubernetes workloads

  • User management

    Integrate IdP and user RBAC to assimilate with corporate single source of truth and assign role-based access

  • Intuitive web UIs for cluster management

    Provide a single UI for multiple cluster management as well as the Container Cloud with MKE container clusters UI for single cluster management

Mirantis Container Cloud releases

This section outlines the release notes for the Mirantis Container Cloud GA release. Within the scope of the Container Cloud GA release, minor releases are being published continuously with new features, improvements, and critical issues resolutions to enhance the Container Cloud GA version. For details, see Mirantis Container Cloud releases and Cluster releases (managed).

Once a new Container Cloud release is available, a management cluster automatically upgrades to a newer consecutive release unless this cluster contains managed clusters with a Cluster release unsupported by that Container Cloud release. For more details about the Container Cloud release mechanism, see Reference Architecture: Release controller.

2.5.0 (current)

The Mirantis Container Cloud GA release 2.5.0:

  • Introduces support for the Cluster release 5.12.0 that is based on Kubernetes 1.18, Mirantis Container Runtime 19.03.14, and the updated version of Mirantis Kubernetes Engine 3.3.6.

  • Introduces support for the Cluster release 6.12.0 that is based on the Cluster release 5.12.0 and supports Mirantis OpenStack for Kubernetes (MOS) 21.1.

  • Still supports previous Cluster releases 5.11.0 and 6.10.0 that are now deprecated and will become unsupported in one of the following Container Cloud releases.

    Caution

    Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.5.0.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.5.0. For the list of enhancements in the Cluster release 5.12.0 and Cluster release 6.12.0 that are supported by the Container Cloud release 2.5.0, see the 5.12.0 and 6.12.0 sections.


Updated version of Mirantis Kubernetes Engine

Updated the Mirantis Kubernetes Engine (MKE) version to 3.3.6 for the Container Cloud management and managed clusters.

For the MKE release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.


Proxy support for OpenStack and VMWare vSphere providers

Implemented proxy support for OpenStack-based and vSphere-based Technology Preview clusters. If you require all Internet access to go through a proxy server for security and audit purposes, you can now bootstrap management and regional clusters using proxy.

You can also enable a separate proxy access on an OpenStack-based managed cluster using the Container Cloud web UI. This proxy is intended for the end user needs and is not used for a managed cluster deployment or for access to the Mirantis resources.

Note

The proxy support for:

  • The OpenStack provider is generally available.

  • The VMWare vSphere provider is available as Technology Preview. For the Technology Preview feature definition, see Preface.

  • The AWS and bare metal providers is in the development stage and will become available in the future Container Cloud releases.


Artifacts caching

Introduced artifacts caching support for all Container Cloud providers to enable deployment of managed clusters without direct Internet access. The Mirantis artifacts used during managed clusters deployment are downloaded through a cache running on a regional cluster.

The feature is enabled by default on new managed clusters based on the Cluster releases 5.12.0 and 6.12.0 and will be automatically enabled on existing clusters during upgrade to the latest version.


NTP server configuration on regional clusters

Implemented the possibility to configure regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region. The feature is applicable to all supported cloud providers. The NTP server parameters can be added before or after management and regional clusters deployment.


Optimized ClusterRelease upgrade process

Optimized the ClusterRelease upgrade process by enabling the Container Cloud provider to upgrade the LCMCluster components, such as MKE, before the HelmBundle components, such as StackLight or Ceph.


Dedicated network for external connection to the Kubernetes services

Technology Preview

Implemented the k8s-ext bridge in L2 templates that allows you to use a dedicated network for external connection to the Kubernetes services exposed by the cluster. When using such bridge, the MetalLB ranges and the IP addresses provided by the subnet that is associated with the bridge must fit in the same CIDR.

If enabled, MetalLB will listen and respond on the dedicated virtual bridge. Also, you can create additional subnets to configure additional address ranges for MetalLB.

Caution

Using of a dedicated network for Kubernetes pods traffic and using of a dedicated network for external connection to the Kubernetes services exposed by the cluster described above are available as Technology Preview. Use such configurations for testing and evaluation purposes only. For details about the Mirantis Technology Preview support scope, see the Preface section of this guide.

The following feature is still under development and will be announced in one of the following Container Cloud releases:

  • Switching Kubernetes API to listen to the specified IP address on the node

Addressed issues

The following issues have been addressed in the Mirantis Container Cloud release 2.5.0 and the Cluster releases 5.12.0 and 6.12.0:

  • [10453] [LCM] Fixed the issue with time synchronization on nodes that could cause networking issues.

  • [9748] [LCM] Fixed the issue with the false-positive helmRelease success status in HelmBundle during Helm upgrade operations.

  • [9748] [LCM] Fixed the issue with the false-positive helmRelease success status in HelmBundle during Helm upgrade operations.

  • [8464] Fixed the issue with Helm controller and OIDC integration failing to be deleted during detach of an MKE cluster.


  • [9928] [Ceph] Fixed the issue with Ceph rebalance leading to data loss during a managed cluster update by implementing the maintenance label to be set before and unset after the cluster update.

  • [9892] [Ceph] Fixed the issue with Ceph being locked during a managed cluster update by adding the PodDisruptionBudget object that enables minimum 2 Ceph OSD nodes running without rescheduling during update.


  • [6988] [BM] Fixed the issue with LVM failing to deploy on a new disk if an old volume group with the same name already existed on the target hardware node but on the different disk.

  • [8560] [BM] Fixed the issue with manual deletion of BareMetalHost from a managed cluster leading to its silent removal without a power-off and deprovision. The fix adds the admission controller webhook to validate the old BareMetalHost when the deletion is requested.

  • [11102] [BM] Fixed the issue with Keepalived not detecting and restoring a VIP of a managed cluster node after running the netplan apply command.

  • [9905] [9906] [9909] [9914] [9921] [BM] Fixed the following Ubuntu CVEs in the bare metal Docker images:

    • CVE-2019-20477 and CVE-2020-1747 for PyYAML in vbmc:latest-20201029

    • CVE-2020-1971 for OpenSSL in the following images:

      • dnsmasq:bionic-20201105044831

      • rabbitmq-management:3.7.15-bionic-20200812044813

      • kaas-ipam:base-bionic-20201208153852

      • ironic-operator:base-bionic-20201106182102

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.5.0 including the Cluster release 5.12.0 and 6.12.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[6222] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

vSphere
[11633] A vSphere-based project cannot be cleaned up

A vSphere-based managed cluster project can fail to be cleaned up because of stale secret(s) related to the RHEL license object(s). Before you can successfully clean up such project, manually delete the secret using the steps below.

Workaround:

  1. Log in to a local machine where your management cluster kubeconfig is located and where kubectl is installed.

  2. Obtain the list of stale secrets:

    kubectl --kubeconfig <kubeconfigPath> get secrets -n <projectName>
    
  3. Open each secret for editing:

    kubectl --kubeconfig <kubeconfigPath> edit secret <secret name> -n <projectName>
    
  4. Remove the following lines:

    finalizers:
    - kaas.mirantis.com/credentials-secret
    
  5. Remove stale secrets:

    kubectl --kubeconfig <kubeconfigPath> delete secret <secretName> -n <projectName>
    

Bare metal
[7655] Wrong status for an incorrectly configured L2 template

If an L2 template is configured incorrectly, a bare metal cluster is deployed successfully but with the runtime errors in the IpamHost object.

Workaround:

If you suspect that the machine is not working properly because of incorrect network configuration, verify the status of the corresponding IpamHost object. Inspect the l2RenderResult and ipAllocationResult object fields for error messages.

[9875] Full preflight fails with a timeout waiting for BareMetalHost

If you run bootstrap.sh preflight with KAAS_BM_FULL_PREFLIGHT=true, the script fails with the following message:

failed to create BareMetal objects: failed to wait for objects of kinds BareMetalHost
to become available: timed out waiting for the condition

As a workaround, unset full preflight using unset KAAS_BM_FULL_PREFLIGHT to run fast preflight instead.

[11468] Pods using LVP PV are not mounted to LVP disk

The persistent volumes (PVs) that are created using local volume provisioner (LVP), are not mounted on the dedicated disk labeled as local-volume and use the root volume instead. In the workaround below, we use StackLight volumes as an example.

Workaround:

  1. Identify whether your cluster is affected:

    1. Log in to any control plane node on the management cluster.

    2. Run the following command:

      findmnt /mnt/local-volumes/stacklight/elasticsearch-data/vol00
      

      In the output, inspect the SOURCE column. If the path starts with /dev/mapper/lvm_root-root, the host is affected by the issue.

      Example of system response:

      TARGET                                                 SOURCE                                                                                FSTYPE OPTIONS
      /mnt/local-volumes/stacklight/elasticsearch-data/vol00 /dev/mapper/lvm_root-root[/var/lib/local-volumes/stacklight/elasticsearch-data/vol00] ext4   rw,relatime,errors=remount-ro,data=ordered
      
    3. Verify other StackLight directories by replacing elasticsearch-data in the command above with the corresponding folders names.

      If your cluster is affected, follow the steps below to manually move all data for volumes that must be on the dedicated disk to the mounted device.

  2. Identify all nodes that run the elasticsearch-master pod:

    kubectl -n stacklight get pods -o wide | grep elasticsearch-master
    

    Apply the steps below to all nodes provided in the output.

  3. Identify the mount point for the dedicated device /dev/mapper/lvm_lvp-lvp. Typically, this device is mounted as /mnt/local-volumes.

    findmnt /mnt/local-volumes
    

    Verify that SOURCE for the /mnt/local-volumes mount target is /dev/mapper/lvm_lvp-lvp on all the nodes.

  4. Create new source directories for the volumes on the dedicated device /dev/mapper/lvm_lvp-lvp:

    mkdir -p /mnt/local-volumes/src/stacklight/elasticsearch-data/vol00
    
  5. Stop the pods that use the volumes to ensure that the data is not corrupted during the switch. Set the number of replicas in StatefulSet to 0:

    kubectl -n stacklight edit statefulset elasticsearch-master
    

    Wait until all elasticsearch-master pods are stopped.

  6. Move the Elasticsearch data from the current location to the new directory:

    cp -pR /var/lib/local-volumes/stacklight/elasticsearch-data/vol00/** /mnt/local-volumes/src/stacklight/elasticsearch-data/vol00/
    
  7. Unmount the old source directory from the volume mount point:

    umount /mnt/local-volumes/stacklight/elasticsearch-data/vol00
    

    Apply this step and the next one to every node with the /mnt/local-volumes/stacklight/elasticsearch-data/vol00 volume.

  8. Remount the new source directory to the volume mount point:

    mount --bind /mnt/local-volumes/src/stacklight/elasticsearch-data/vol00 /mnt/local-volumes/stacklight/elasticsearch-data/vol00
    
  9. Edit the Cluster object by adding the highlighted parameters below for the StackLight Helm chart:

    kubectl --kubeconfig <mgmtClusterKubeconfig> edit -n <projectName> cluster <managedClusterName>
    
    spec:
      helmReleases:
      - name: stacklight
        values:
          ...
          elasticsearch:
            clusterHealthCheckParams: wait_for_status=red&timeout=1s
    
  10. Start the Elasticsearch pods by setting the number of replicas in StatefulSet to 3:

    kubectl -n stacklight edit statefulset elasticsearch-master
    

    Wait until all elasticsearch-master pods are up and running.

  11. Remove the previously added clusterHealthCheckParams parameters from the Cluster object.

  12. In /etc/fstab on every node that has the volume /mnt/local-volumes/stacklight/elasticsearch-data/vol00, edit the following entry:

    /var/lib/local-volumes/stacklight/elasticsearch-data/vol00 /mnt/local-volumes/stacklight/elasticsearch-data/vol00 none bind 0 0
    

    In this entry, replace the old directory /var/lib/local-volumes/stacklight/elasticsearch-data/vol00 with the new one: /mnt/local-volumes/src/stacklight/elasticsearch-data/vol00.


Storage
[10060] Ceph OSD node removal fails

A Ceph node removal is not being triggered properly after updating the KaasCephCluster custom resource (CR). Both management and managed clusters are affected.

Workaround:

  1. Remove the parameters for a Ceph OSD from the KaasCephCluster CR as described in Operations Guide: Add, remove, or reconfigure Ceph nodes.

  2. Obtain the IDs of the osd and mon services that are located on the old node:

    1. Obtain the UID of the affected machine:

      kubectl get machine <CephOSDNodeName> -n <ManagedClusterProjectName> -o jsonpath='{.metadata.annotations.kaas\.mirantis\.com\/uid}'
      
    2. Export kubeconfig of your managed cluster. For example:

      export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
      
    3. Identify the pods IDs that run the osd and mon services:

      kubectl get pods -o wide -n rook-ceph | grep <affectedMachineUID> | grep -E "mon|osd"
      

      Example of the system response extract:

      rook-ceph-mon-c-7bbc5d757d-5bpws                              1/1  Running    1  6h1m
      rook-ceph-osd-2-58775d5568-5lklw                              1/1  Running    4  44h
      rook-ceph-osd-prepare-705ae6c647cfdac928c63b63e2e2e647-qn4m9  0/1  Completed  0  94s
      

      The pods IDs include the osd or mon services IDs. In the example system response above, the osd ID is 2 and the mon ID is c.

  3. Delete the deployments of the osd and mon services obtained in the previous step:

    kubectl delete deployment rook-ceph-osd(mon)-<ID> -n rook-ceph
    

    For example:

    kubectl delete deployment rook-ceph-mon-c -n rook-ceph
    kubectl delete deployment rook-ceph-osd-2 -n rook-ceph
    
  4. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  5. Rebalance the Ceph OSDs:

    ceph osd out osd(s).ID
    

    Wait for the rebalance to complete.

  6. Rebalance the Ceph data:

    ceph osd purge osd(s).ID
    

    Wait for the Ceph data to rebalance.

  7. Remove the old node from the Ceph OSD tree:

    ceph osd crush rm <NodeName>
    
  8. If the removed node contained mon services, remove them:

    ceph mon rm <monID>
    
[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.

[10050] Ceph OSD pod is in the CrashLoopBackOff state after disk replacement

If you use a custom BareMetalHostProfile, after disk replacement on a Ceph OSD, the Ceph OSD pod switches to the CrashLoopBackOff state due to the Ceph OSD authorization key failing to be created properly.

Workaround:

  1. Export kubeconfig of your managed cluster. For example:

    export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
    
  2. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  3. Delete the authorization key for the failed Ceph OSD:

    ceph auth del osd.<ID>
    
  4. SSH to the node on which the Ceph OSD cannot be created.

  5. Clean up the disk that will be a base for the failed Ceph OSD. For details, see official Rook documentation.

    Note

    Ignore failures of the sgdisk --zap-all $DISK and blkdiscard $DISK commands if any.

  6. On the managed cluster, restart the Rook operator:

    kubectl -n rook-ceph delete pod -l app=rook-ceph-operator
    

IAM
[10829] Keycloak pods fail to start during a management cluster bootstrap

The Keycloak pods may fail to start during a management cluster bootstrap with the Failed to update database exception in logs.

Caution

The following workaround is applicable only to deployments where mariadb-server has started successfully. Otherwise, fix the issues with MariaDB first.

Workaround:

  1. Verify that mariadb-server has started:

    kubectl get po -n kaas | grep mariadb-server
    
  2. Scale down the Keycloak instances:

    kubectl scale sts iam-keycloak --replicas=0 -n kaas
    
  3. Open the iam-keycloak-sh configmap for editing:

    kubectl edit cm -n kaas iam-keycloak-sh
    
  4. On the last line of the configmap, before the $MIGRATION_ARGS variable, add the following parameter:

    -Djboss.as.management.blocking.timeout=<RequiredValue>
    

    The recommended timeout value is minimum 15 minutes set in seconds. For example, -Djboss.as.management.blocking.timeout=900.

  5. Open the iam-keycloak-startup configmap for editing:

    kubectl edit cm -n kaas iam-keycloak-startup
    
  6. In the iam-keycloak-startup configmap, add the following line:

    /subsystem=transactions/:write-attribute(name=default-timeout,value=<RequiredValue>)
    

    The recommended timeout value is minimum 15 minutes set in seconds.

  7. In the Keycloak StatefulSet, adjust liveness probe timeouts:

    kubectl edit sts -n kaas iam-keycloak
    
  8. Scale up the Keycloak instances:

    kubectl scale sts iam-keycloak --replicas=3 -n kaas
    

LCM
[10029] Authentication fails with the 401 Unauthorized error

Authentication may not work on some controller nodes after a managed cluster creation. As a result, the Kubernetes API operations with the managed cluster kubeconfig fail with Response Status: 401 Unauthorized.

As a workaround, manually restart the ucp-controller and ucp-auth Docker services on the affected node.

[9496] Helm releases get stuck in FAILED or UNKNOWN state

During a management, regional, or managed cluster deployment, Helm releases may get stuck in the FAILED or UNKNOWN state although the corresponding machines statuses are Ready in the Container Cloud web UI. For example, if the StackLight Helm release fails, the links to its endpoints are grayed out in the web UI. In the cluster status, providerStatus.helm.ready and providerStatus.helm.releaseStatuses.<releaseName>.success are false.

HelmBundle cannot recover from such states and requires manual actions. The workaround below describes the recovery steps for the stacklight release that got stuck during a cluster deployment. Use this procedure as an example for other Helm releases as required.

Workaround:

  1. Verify the failed release has the UNKNOWN or FAILED status in the HelmBundle object:

    kubectl --kubeconfig <regionalClusterKubeconfigPath> get helmbundle <clusterName> -n <clusterProjectName> -o=jsonpath={.status.releaseStatuses.stacklight}
    
    In the command above and in the steps below, replace the parameters
    enclosed in angle brackets with the corresponding values of your cluster.
    

    Example of system response:

    stacklight:
    attempt: 2
    chart: ""
    finishedAt: "2021-02-05T09:41:05Z"
    hash: e314df5061bd238ac5f060effdb55e5b47948a99460c02c2211ba7cb9aadd623
    message: '[{"occurrence":1,"lastOccurrenceDate":"2021-02-05 09:41:05","content":"error
      updating the release: rpc error: code = Unknown desc = customresourcedefinitions.apiextensions.k8s.io
      \"helmbundles.lcm.mirantis.com\" already exists"}]'
    notes: ""
    status: UNKNOWN
    success: false
    version: 0.1.2-mcp-398
    
  2. Log in to the helm-controller pod console:

    kubectl --kubeconfig <affectedClusterKubeconfigPath> exec -n kube-system -it helm-controller-0 sh -c tiller
    
  3. Remove the failed release. For example:

    ./helm --host=localhost:44134 delete stacklight
    

    Once done, the release triggers for redeployment.


StackLight
[11001] Patroni pod fails to start

After the management cluster update, a Patroni pod may fail to start and remain in the CrashLoopBackOff status. Messages similar to the following ones may be present in Patroni logs:

Local timeline=4 lsn=0/A000000
master_timeline=6
master: history=1 0/1ADEB48       no recovery target specified
2       0/8044500       no recovery target specified
3       0/A0000A0       no recovery target specified
4       0/A1B6CB0       no recovery target specified
5       0/A2C0C80       no recovery target specified

As a workaround, reinitialize the affected pod with a new volume by deleting the pod itself and the associated PersistentVolumeClaim (PVC).

Workaround:

  1. Obtain the PVC of the affected pod:

    kubectl -n stacklight get "pod/${POD_NAME}" -o jsonpath='{.spec.volumes[?(@.name=="storage-volume")].persistentVolumeClaim.claimName}'
    
  2. Delete the affected pod and its PVC:

    kubectl -n stacklight delete "pod/${POD_NAME}" "pvc/${POD_PVC}"
    sleep 3  # wait for StatefulSet to reschedule the pod, but miss dependent PVC creation
    kubectl -n stacklight delete "pod/${POD_NAME}"
    

Management and regional clusters
[9899] Helm releases get stuck in PENDING_UPGRADE during cluster update

Helm releases may get stuck in the PENDING_UPGRADE status during a management or managed cluster upgrade. The HelmBundle controller cannot recover from this state and requires manual actions. The workaround below describes the recovery process for the openstack-operator release that stuck during a managed cluster update. Use it as an example for other Helm releases as required.

Workaround:

  1. Log in to the helm-controller pod console:

    kubectl exec -n kube-system -it helm-controller-0 sh -c tiller
    
  2. Identify the release that stuck in the PENDING_UPGRADE status. For example:

    ./helm --host=localhost:44134 history openstack-operator
    

    Example of system response:

    REVISION  UPDATED                   STATUS           CHART                      DESCRIPTION
    1         Tue Dec 15 12:30:41 2020  SUPERSEDED       openstack-operator-0.3.9   Install complete
    2         Tue Dec 15 12:32:05 2020  SUPERSEDED       openstack-operator-0.3.9   Upgrade complete
    3         Tue Dec 15 16:24:47 2020  PENDING_UPGRADE  openstack-operator-0.3.18  Preparing upgrade
    
  3. Roll back the failed release to the previous revision. For example:

    ./helm --host=localhost:44134 rollback openstack-operator 2
    

    Once done, the release will be reconciled.

[10424] OpenStack-based regional cluster cleanup fails by timeout

An OpenStack-based regional cluster cleanup fails with the timeout error.

Workaround:

  1. Wait for the Cluster object to be deleted in the bootstrap cluster:

    kubectl --kubeconfig <(./bin/kind get kubeconfig --name clusterapi) get cluster
    

    The system output must be empty.

  2. Remove the bootstrap cluster manually:

    ./bin/kind delete cluster --name clusterapi
    

Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.5.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.17.4

aws-credentials-controller

1.17.4

Bare metal

baremetal-operator Updated

3.2.1

baremetal-public-api Updated

3.2.1

baremetal-provider Updated

1.17.6

httpd Updated

1.18.0

ironic Updated

ussuri-bionic-20210202180025

ironic-operator

base-bionic-20210106163336

kaas-ipam Updated

base-bionic-20210218141033

local-volume-provisioner

1.0.4-mcp

mariadb Updated

10.4.17-bionic-20210203155435

IAM

iam Updated

1.3.0

iam-controller Updated

1.17.4

keycloak

9.0.0

Container Cloud Updated

admission-controller

1.17.5

byo-credentials-controller

1.17.4

byo-provider

1.17.4

kaas-public-api

1.17.4

kaas-exporter

1.17.4

kaas-ui

1.17.4

lcm-controller

0.2.0-259-g71792430

mcc-cache New

1.17.4

proxy-controller New

1.17.4

release-controller

1.17.4

squid-proxy New

0.0.1-1

OpenStack Updated

openstack-provider

1.17.4

os-credentials-controller

1.17.4

VMWare vSphere Updated

vsphere-provider

1.17.6

vsphere-credentials-controller

1.17.4

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.5.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-3.2.1.tgz

baremetal-public-api Updated

https://binary.mirantis.com/bm/helm/baremetal-public-api-3.2.1.tgz

ironic-python-agent-bionic.kernel Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-ussuri-bionic-debug-20210204084827

ironic-python-agent-bionic.initramfs Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-ussuri-bionic-debug-20210204084827

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-3.2.1.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20201113171304

httpd Updated

mirantis.azurecr.io/lcm/nginx:1.18.0

ironic Updated

mirantis.azurecr.io/openstack/ironic:ussuri-bionic-20210202180025

ironic-inspector Updated

mirantis.azurecr.io/openstack/ironic-inspector:ussuri-bionic-20210202180025

ironic-operator

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20210106163336

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20210218141033

mariadb Updated

mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20210203155435


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.17.5.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.17.5.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.17.4.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.17.4.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.17.4.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.17.4.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.17.4.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.17.4.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.17.4.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.17.4.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.17.4.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.17.4.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.17.4.tgz

mcc-cache New

https://binary.mirantis.com/core/helm/mcc-cache-1.17.4.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.17.4.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.17.4.tgz

proxy-controller New

https://binary.mirantis.com/core/helm/proxy-controller-1.17.4.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.17.4.tgz

squid-proxy New

https://binary.mirantis.com/core/helm/squid-proxy-1.17.4.tgz

vsphere-credentials-controller

https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.17.4.tgz

vsphere-provider

https://binary.mirantis.com/core/helm/vsphere-provider-1.17.4.tgz

Docker images Updated

admission-controller

mirantis.azurecr.io/core/admission-controller:1.17.5

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.17.4

aws-credentials-controller

mirantis.azurecr.io/core/aws-credentials-controller:1.17.4

byo-credentials-controller

mirantis.azurecr.io/core/byo-credentials-controller:1.17.4

cluster-api-provider-baremetal

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.17.6

frontend

mirantis.azurecr.io/core/frontend:1.17.4

iam-controller

mirantis.azurecr.io/core/iam-controller:1.17.4

lcm-controller

mirantis.azurecr.io/core/lcm-controller:v0.2.0-259-g71792430

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.17.4

os-credentials-controller

mirantis.azurecr.io/core/os-credentials-controller:1.17.4

release-controller

mirantis.azurecr.io/core/release-controller:1.17.4

squid-proxy New

mirantis.azurecr.io/core/squid-proxy:0.0.1-1

vsphere-credentials-controller

mirantis.azurecr.io/core/vsphere-credentials-controller:1.17.6


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin

iamctl-linux

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-linux

iamctl-darwin

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-darwin

iamctl-windows

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-windows

Helm charts Updated

iam

http://binary.mirantis.com/iam/helm/iam-1.3.0.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

keycloak-proxy

http://binary.mirantis.com/core/helm/keycloak_proxy-1.17.4.tgz

Docker images

api Updated

mirantis.azurecr.io/iam/api:0.4.0

auxiliary Updated

mirantis.azurecr.io/iam/auxiliary:0.4.0

kubernetes-entrypoint Updated

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052

keycloak Updated

mirantis.azurecr.io/iam/keycloak:0.4.0

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

Unsupported releases

Unsupported Container Cloud releases history

Version

Release date

Supported Cluster releases

Summary

2.4.0

February 2, 2021

  • Support for the updated version of Mirantis Container Runtime 19.03.14

  • Dedicated network for Kubernetes pods traffic on bare metal clusters

  • Improvements for the feedback form in the Container Cloud web UI

  • StackLight enhancements:

    • Alert inhibition rules

    • Integration between Grafana and Kibana

    • New Telegraf alert TelegrafGatherErrors

    • Configuration of Ironic Telegraf input plugin

    • Automatically defined cluster ID

2.3.0

December 23, 2020

  • Support for Mirantis Kubernetes Engine 3.3.4 and Mirantis Container Runtime 19.03.13

  • Support for multiple host-specific L2 templates per a bare metal cluster

  • Additional regional cluster on VMWare vSphere

  • Automated setup of an OVF template for the VMWare vSphere provider

  • StackLight support for VMWare vSphere

  • Improvements in the Container Cloud logs collection

2.2.0

November 5, 2020

  • Support for VMWare vSphere provider on RHEL

  • Kernel parameters management through BareMetalHostProfile

  • Support of multiple subnets per cluster

  • Optimization of the Container Cloud logs collection

  • Container Cloud API documentation for bare metal

2.1.0

October 19, 2020

  • Node labeling for machines

  • AWS resources discovery in the Container Cloud web UI

  • Credentials statuses for OpenStack and AWS in the Container Cloud web UI

  • StackLight improvements:

    • Grafana upgrade from version 6.6.2 to 7.1.5

    • Grafana Image Renderer pod to offload rendering of images from charts

    • Grafana home dashboard improvements

    • Splitting of the regional and management cluster function in StackLight telemetry to obtain aggregated metrics on the management cluster from regional and managed clusters

    • Amendments to the StackLight alerts

2.0.0

September 16, 2020

5.7.0

First GA release of Container Cloud with the following key features:

  • Container Cloud with Mirantis Kubernetes Engine (MKE) container clusters for the management plane

  • Support for managed Container Cloud with MKE container clusters on top of the AWS, OpenStack, and bare metal cloud providers

  • Support for attaching of the existing MKE standalone clusters

  • Ceph as a Kubernetes storage provider for the bare metal use case

  • Multi-region support for security and scalability

  • IAM integration with MKE container clusters to provide SSO

  • Logging, monitoring, and alerting tuned for MKE with data aggregation to the management cluster and telemetry sent to Mirantis

2.4.0

The Mirantis Container Cloud GA release 2.4.0:

  • Introduces support for the Cluster release 5.11.0 that is based on Kubernetes 1.18, Mirantis Kubernetes Engine 3.3.4, and the updated version of Mirantis Container Runtime 19.03.14.

  • Supports the Cluster release 6.10.0 that is based on the Cluster release 5.10.0 and supports Mirantis OpenStack for Kubernetes (MOS) Ussuri.

  • Still supports previous Cluster releases 5.10.0 and 6.8.1 that are now deprecated and will become unsupported in one of the following Container Cloud releases.

    Caution

    Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.4.0.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.4.0. For the list of enhancements in the Cluster release 5.11.0 and Cluster release 6.10.0 that are supported by the Container Cloud release 2.4.0, see the 5.11.0 and 6.10.0 sections.


Support for the updated version of Mirantis Container Runtime

Updated the Mirantis Container Runtime (MCR) version to 19.03.14 for all types of Container Cloud clusters.

For the Mirantis Container Runtime release highlights, see MKE, MSR, and MCR documentation: Mirantis Container Runtime release notes.

Caution

Due to the development limitations, the MCR upgrade to version 19.03.13 or 19.03.14 on existing Container Cloud clusters is not supported.


Dedicated network for Kubernetes pods traffic on bare metal clusters

Technology Preview

Implemented the k8s-pods bridge in L2 templates that allows you to use a dedicated network for Kubernetes pods traffic. When the k8s-pods bridge is defined in an L2 template, Calico CNI uses that network for routing the pods traffic between nodes.

Caution

Using of a dedicated network for Kubernetes pods traffic described above is available as Technology Preview. Use such configuration for testing and evaluation purposes only. For details about the Mirantis Technology Preview support scope, see the Preface section of this guide.

The following features are still under development and will be announced in one of the following Container Cloud releases:

  • Switching Kubernetes API to listen to the specified IP address on the node

  • Enable MetalLB to listen and respond on the dedicated virtual bridge.


Feedback form improvement in Container Cloud web UI

Extended the functionality of the feedback form for the Container Cloud web UI. Using the Feedback button, you can now provide 5-star product rating and feedback about Container Cloud. If you have an idea or found a bug in Container Cloud, you can create a ticket for the Mirantis support team to help us improve the product.

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.4.0 including the Cluster release 5.11.0 and 6.10.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[6222] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

Bare metal
[9875] Full preflight fails with a timeout waiting for BareMetalHost

If you run bootstrap.sh preflight with KAAS_BM_FULL_PREFLIGHT=true, the script fails with the following message:

failed to create BareMetal objects: failed to wait for objects of kinds BareMetalHost
to become available: timed out waiting for the condition

As a workaround, unset full preflight using unset KAAS_BM_FULL_PREFLIGHT to run fast preflight instead.

[11102] Keepalived does not detect the loss of VIP deleted by netplan

Fixed in Container Cloud 2.5.0

This issue may occur on the baremetal-based managed clusters that are created using L2 templates when network configuration is changed by the user or when Container Cloud is updated from version 2.3.0 to 2.4.0.

Due to the community issue, Keepalived 1.3.9 does not detect and restore a VIP of a managed cluster node after running the netplan apply command. The command is used to apply network configuration changes.

As a result, the Kubernetes API on the affected managed clusters becomes inaccessible.

As a workaround, log in to all nodes of the affected managed clusters and restart Keepalived using systemctl restart keepalived.

[6988] LVM fails to deploy if the volume group name already exists

Fixed in Container Cloud 2.5.0

During a management or managed cluster deployment, LVM cannot be deployed on a new disk if an old volume group with the same name already exists on the target hardware node but on the different disk.

Workaround:

In the bare metal host profile specific to your hardware configuration, add the wipe: true parameter to the device that fails to be deployed. For the procedure details, see Operations Guide: Create a custom host profile.

[7655] Wrong status for an incorrectly configured L2 template

If an L2 template is configured incorrectly, a bare metal cluster is deployed successfully but with the runtime errors in the IpamHost object.

Workaround:

If you suspect that the machine is not working properly because of incorrect network configuration, verify the status of the corresponding IpamHost object. Inspect the l2RenderResult and ipAllocationResult object fields for error messages.

[8560] Manual deletion of BareMetalHost leads to its silent removal

Fixed in Container Cloud 2.5.0

If BareMetalHost is manually removed from a managed cluster, it is silently removed without a power-off and deprovision that leads to a managed cluster failures.

Workaround:

Do not manually delete a BareMetalHost that has the Provisioned status.


Storage
[10060] Ceph OSD node removal fails

A Ceph node removal is not being triggered properly after updating the KaasCephCluster custom resource (CR). Both management and managed clusters are affected.

Workaround:

  1. Remove the parameters for a Ceph OSD from the KaasCephCluster CR as described in Operations Guide: Add, remove, or reconfigure Ceph nodes.

  2. Obtain the IDs of the osd and mon services that are located on the old node:

    1. Obtain the UID of the affected machine:

      kubectl get machine <CephOSDNodeName> -n <ManagedClusterProjectName> -o jsonpath='{.metadata.annotations.kaas\.mirantis\.com\/uid}'
      
    2. Export kubeconfig of your managed cluster. For example:

      export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
      
    3. Identify the pods IDs that run the osd and mon services:

      kubectl get pods -o wide -n rook-ceph | grep <affectedMachineUID> | grep -E "mon|osd"
      

      Example of the system response extract:

      rook-ceph-mon-c-7bbc5d757d-5bpws                              1/1  Running    1  6h1m
      rook-ceph-osd-2-58775d5568-5lklw                              1/1  Running    4  44h
      rook-ceph-osd-prepare-705ae6c647cfdac928c63b63e2e2e647-qn4m9  0/1  Completed  0  94s
      

      The pods IDs include the osd or mon services IDs. In the example system response above, the osd ID is 2 and the mon ID is c.

  3. Delete the deployments of the osd and mon services obtained in the previous step:

    kubectl delete deployment rook-ceph-osd(mon)-<ID> -n rook-ceph
    

    For example:

    kubectl delete deployment rook-ceph-mon-c -n rook-ceph
    kubectl delete deployment rook-ceph-osd-2 -n rook-ceph
    
  4. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  5. Rebalance the Ceph OSDs:

    ceph osd out osd(s).ID
    

    Wait for the rebalance to complete.

  6. Rebalance the Ceph data:

    ceph osd purge osd(s).ID
    

    Wait for the Ceph data to rebalance.

  7. Remove the old node from the Ceph OSD tree:

    ceph osd crush rm <NodeName>
    
  8. If the removed node contained mon services, remove them:

    ceph mon rm <monID>
    
[9928] Ceph rebalance during a managed cluster update

Fixed in Container Cloud 2.5.0

During a managed cluster update, Ceph rebalance leading to data loss may occur.

Workaround:

  1. Before updating a managed cluster:

    1. Log in to the ceph-tools pod:

      kubectl -n rook-ceph exec -it <ceph-tools-pod-name> bash
      
    2. Set the noout flag:

      ceph osd set noout
      
  2. Update a managed cluster.

  3. After updating a managed cluster:

    1. Log in to the ceph-tools pod:

      kubectl -n rook-ceph exec -it <ceph-tools-pod-name> bash
      
    2. Unset the noout flag:

      ceph osd unset noout
      
[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.

[10050] Ceph OSD pod is in the CrashLoopBackOff state after disk replacement

If you use a custom BareMetalHostProfile, after disk replacement on a Ceph OSD, the Ceph OSD pod switches to the CrashLoopBackOff state due to the Ceph OSD authorization key failing to be created properly.

Workaround:

  1. Export kubeconfig of your managed cluster. For example:

    export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
    
  2. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  3. Delete the authorization key for the failed Ceph OSD:

    ceph auth del osd.<ID>
    
  4. SSH to the node on which the Ceph OSD cannot be created.

  5. Clean up the disk that will be a base for the failed Ceph OSD. For details, see official Rook documentation.

    Note

    Ignore failures of the sgdisk --zap-all $DISK and blkdiscard $DISK commands if any.

  6. On the managed cluster, restart the Rook operator:

    kubectl -n rook-ceph delete pod -l app=rook-ceph-operator
    

LCM
[10029] Authentication fails with the 401 Unauthorized error

Authentication may not work on some controller nodes after a managed cluster creation. As a result, the Kubernetes API operations with the managed cluster kubeconfig fail with Response Status: 401 Unauthorized.

As a workaround, manually restart the ucp-controller and ucp-auth Docker services on the affected node.

[9496] Helm releases get stuck in FAILED or UNKNOWN state

During a management, regional, or managed cluster deployment, Helm releases may get stuck in the FAILED or UNKNOWN state although the corresponding machines statuses are Ready in the Container Cloud web UI. For example, if the StackLight Helm release fails, the links to its endpoints are grayed out in the web UI. In the cluster status, providerStatus.helm.ready and providerStatus.helm.releaseStatuses.<releaseName>.success are false.

HelmBundle cannot recover from such states and requires manual actions. The workaround below describes the recovery steps for the stacklight release that got stuck during a cluster deployment. Use this procedure as an example for other Helm releases as required.

Workaround:

  1. Verify the failed release has the UNKNOWN or FAILED status in the HelmBundle object:

    kubectl --kubeconfig <regionalClusterKubeconfigPath> get helmbundle <clusterName> -n <clusterProjectName> -o=jsonpath={.status.releaseStatuses.stacklight}
    
    In the command above and in the steps below, replace the parameters
    enclosed in angle brackets with the corresponding values of your cluster.
    

    Example of system response:

    stacklight:
    attempt: 2
    chart: ""
    finishedAt: "2021-02-05T09:41:05Z"
    hash: e314df5061bd238ac5f060effdb55e5b47948a99460c02c2211ba7cb9aadd623
    message: '[{"occurrence":1,"lastOccurrenceDate":"2021-02-05 09:41:05","content":"error
      updating the release: rpc error: code = Unknown desc = customresourcedefinitions.apiextensions.k8s.io
      \"helmbundles.lcm.mirantis.com\" already exists"}]'
    notes: ""
    status: UNKNOWN
    success: false
    version: 0.1.2-mcp-398
    
  2. Log in to the helm-controller pod console:

    kubectl --kubeconfig <affectedClusterKubeconfigPath> exec -n kube-system -it helm-controller-0 sh -c tiller
    
  3. Remove the failed release. For example:

    ./helm --host=localhost:44134 delete stacklight
    

    Once done, the release triggers for redeployment.


StackLight
[11001] Patroni pod fails to start

After the management cluster update, a Patroni pod may fail to start and remain in the CrashLoopBackOff status. Messages similar to the following ones may be present in Patroni logs:

Local timeline=4 lsn=0/A000000
master_timeline=6
master: history=1 0/1ADEB48       no recovery target specified
2       0/8044500       no recovery target specified
3       0/A0000A0       no recovery target specified
4       0/A1B6CB0       no recovery target specified
5       0/A2C0C80       no recovery target specified

As a workaround, reinitialize the affected pod with a new volume by deleting the pod itself and the associated PersistentVolumeClaim (PVC).

Workaround:

  1. Obtain the PVC of the affected pod:

    kubectl -n stacklight get "pod/${POD_NAME}" -o jsonpath='{.spec.volumes[?(@.name=="storage-volume")].persistentVolumeClaim.claimName}'
    
  2. Delete the affected pod and its PVC:

    kubectl -n stacklight delete "pod/${POD_NAME}" "pvc/${POD_PVC}"
    sleep 3  # wait for StatefulSet to reschedule the pod, but miss dependent PVC creation
    kubectl -n stacklight delete "pod/${POD_NAME}"
    

Management cluster update
[9899] Helm releases get stuck in PENDING_UPGRADE during cluster update

Helm releases may get stuck in the PENDING_UPGRADE status during a management or managed cluster upgrade. The HelmBundle controller cannot recover from this state and requires manual actions. The workaround below describes the recovery process for the openstack-operator release that stuck during a managed cluster update. Use it as an example for other Helm releases as required.

Workaround:

  1. Log in to the helm-controller pod console:

    kubectl exec -n kube-system -it helm-controller-0 sh -c tiller
    
  2. Identify the release that stuck in the PENDING_UPGRADE status. For example:

    ./helm --host=localhost:44134 history openstack-operator
    

    Example of system response:

    REVISION  UPDATED                   STATUS           CHART                      DESCRIPTION
    1         Tue Dec 15 12:30:41 2020  SUPERSEDED       openstack-operator-0.3.9   Install complete
    2         Tue Dec 15 12:32:05 2020  SUPERSEDED       openstack-operator-0.3.9   Upgrade complete
    3         Tue Dec 15 16:24:47 2020  PENDING_UPGRADE  openstack-operator-0.3.18  Preparing upgrade
    
  3. Roll back the failed release to the previous revision. For example:

    ./helm --host=localhost:44134 rollback openstack-operator 2
    

    Once done, the release will be reconciled.


Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Addressed issues

The following issues have been addressed in the Mirantis Container Cloud release 2.4.0 and the Cluster releases 5.11.0 and 6.10.0:

  • [10351] [BM] [IPAM] Fixed the issue with the automatically allocated subnet having the ability to requeue allocation from a SubnetPool in the error state.

  • [10104] [BM] [Ceph] Fixed the issue with OpenStack services failing to access rook-ceph-mon-* pods due to the changed metadata for connection after pods restart if Ceph was deployed without hostNetwork: true.


  • [2757] [IAM] Fixed the issue with IAM failing to start with the IAM pods being in the CrashLoopBackOff state during a management cluster deployment.

  • [7562] [IAM] Disabled the http port in Keycloak to prevent security vulnerabilities.


  • [10108] [LCM] Fixed the issue with accidental upgrade of the docker-ee, docker-ee-cli, and containerd.io packages that must be pinned during the host OS upgrade.

  • [10094] [LCM] Fixed the issue with error handling in the manage-taints Ansible script.

  • [9676] [LCM] Fixed the issue with Keepalived and NGINX being installed on worker nodes instead of being installed on control plane nodes only.


  • [10323] [UI] Fixed the issue with offline tokens being expired over time if fetched using the Container Cloud web UI. The issue occurred if the Log in with Keycloak option was used.

  • [8966] [UI] Fixed the issue with the “invalid_grant”,”error_description”: “Session doesn’t have required client” error occurring over time after logging in to the Container Cloud web UI through Log in with Keycloak.

  • [10180] [UI] Fixed the issue with the SSH Keys dialog becoming blank after the token expiration.

  • [7781] [UI] Fixed the issue with the previously selected Ceph cluster machines disappearing from the drop-down menu of the Create New Ceph Cluster dialog.

  • [7843] [UI] Fixed the issue with Provider Credentials being stuck in the Processing state if created using the Add new credential option of the Create New Cluster dialog.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.4.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.16.1

aws-credentials-controller

1.16.1

Bare metal

baremetal-operator Updated

3.1.7

baremetal-public-api Updated

3.1.7

baremetal-provider Updated

1.16.4

httpd

2.4.46-20201001171500

ironic Updated

ussuri-bionic-20210113180016

ironic-operator Updated

base-bionic-20210106163336

kaas-ipam Updated

base-bionic-20210106163449

local-volume-provisioner

1.0.4-mcp

mariadb Updated

10.4.17-bionic-20210106145941

IAM

iam Updated

1.2.1

iam-controller Updated

1.16.1

keycloak

9.0.0

Container Cloud

admission-controller Updated

1.16.1

byo-credentials-controller Updated

1.16.1

byo-provider Updated

1.16.1

kaas-public-api Updated

1.16.1

kaas-exporter Updated

1.16.1

kaas-ui Updated

1.16.2

lcm-controller

0.2.0-224-g5c413d37

release-controller Updated

1.16.1

OpenStack Updated

openstack-provider

1.16.1

os-credentials-controller

1.16.1

VMWare vSphere Updated

vsphere-provider

1.16.1

vsphere-credentials-controller

1.16.4

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.4.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-3.1.7.tgz

baremetal-public-api Updated

https://binary.mirantis.com/bm/helm/baremetal-public-api-3.1.7.tgz

ironic-python-agent-bionic.kernel Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-ussuri-bionic-debug-20210108095808

ironic-python-agent-bionic.initramfs Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-ussuri-bionic-debug-20210108095808

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-3.1.7.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20201113171304

httpd

mirantis.azurecr.io/bm/external/httpd:2.4.46-20201001171500

ironic Updated

mirantis.azurecr.io/openstack/ironic:ussuri-bionic-20210113180016

ironic-inspector Updated

mirantis.azurecr.io/openstack/ironic-inspector:ussuri-bionic-20210113180016

ironic-operator Updated

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20210106163336

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20210106163449

mariadb Updated

mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20210106145941


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.16.1.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.16.1.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.16.1.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.16.1.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.16.1.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.16.1.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.16.1.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.16.1.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.16.1.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.16.1.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.16.1.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.16.1.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.16.1.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.16.1.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.16.1.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.16.1.tgz

vsphere-credentials-controller

https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.16.1.tgz

vsphere-provider

https://binary.mirantis.com/core/helm/vsphere-provider-1.16.1.tgz

Docker images Updated

admission-controller

mirantis.azurecr.io/core/admission-controller:1.16.1

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.16.1

aws-credentials-controller

mirantis.azurecr.io/core/aws-credentials-controller:1.16.1

byo-credentials-controller

mirantis.azurecr.io/core/byo-credentials-controller:1.16.1

cluster-api-provider-baremetal

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.16.1

frontend

mirantis.azurecr.io/core/frontend:1.16.1

iam-controller

mirantis.azurecr.io/core/iam-controller:1.16.1

lcm-controller

mirantis.azurecr.io/core/lcm-controller:v0.2.0-224-g5c413d37

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.16.1

os-credentials-controller

mirantis.azurecr.io/core/os-credentials-controller:1.16.1

release-controller

mirantis.azurecr.io/core/release-controller:1.16.1

vsphere-credentials-controller

mirantis.azurecr.io/core/vsphere-credentials-controller:1.16.4


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin

iamctl-linux Updated

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-linux

iamctl-darwin Updated

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-darwin

iamctl-windows Updated

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-windows

Helm charts

iam Updated

http://binary.mirantis.com/iam/helm/iam-1.2.1.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.3.tgz

keycloak-proxy Updated

http://binary.mirantis.com/core/helm/keycloak_proxy-1.16.3.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.3.18

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.3.18

kubernetes-entrypoint Updated

mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.0-20200311160233

mariadb Updated

mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052

keycloak

mirantis.azurecr.io/iam/keycloak:0.3.19

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

2.3.0

The Mirantis Container Cloud GA release 2.3.0:

  • Introduces support for the Cluster release 5.10.0 that is based on Kubernetes 1.18 and the updated versions of Mirantis Kubernetes Engine 3.3.4 and Mirantis Container Runtime 19.03.13.

  • Introduces support for the Cluster release 6.10.0 that is based on the Cluster release 5.10.0 and supports Mirantis OpenStack for Kubernetes (MOS) Ussuri.

  • Still supports previous Cluster releases 5.9.0 and 6.8.1 that are now deprecated and will become unsupported in one of the following Container Cloud releases.

    Caution

    Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.3.0.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.3.0. For the list of enhancements in the Cluster release 5.10.0 and Cluster release 6.10.0 introduced by the Container Cloud release 2.3.0, see the 5.10.0 and 6.10.0 sections.


Updated versions of Mirantis Kubernetes Engine and Container Runtime

Updated the Mirantis Kubernetes Engine (MKE) version to 3.3.4 and the Mirantis Container Runtime (MCR) version to 19.03.13 for the Container Cloud management and managed clusters.

For the MKE release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

For the Mirantis Container Runtime release highlights, see MKE, MSR, and MCR documentation: Mirantis Container Runtime release notes.

Caution

Due to the development limitations, the MCR upgrade to version 19.03.13 or 19.03.14 on existing Container Cloud clusters is not supported.


Additional regional cluster on VMWare vSphere

Technical Preview

In scope of Technology Preview support for the VMWare vSphere provider, added the capability to deploy an additional regional vSphere-based cluster on top of the vSphere management cluster to create managed clusters with different configurations if required.


Automated setup of an OVF template for the VMWare vSphere provider

Technical Preview

Automated the process of an OVF template setup for the vSphere-based management and managed clusters deployments. The OVF template is now set up by Packer using the vsphere_template flag that is integrated into bootstrap.sh.


StackLight support for VMWare vSphere

Technical Preview

Added the capability to deploy StackLight on management clusters. However, such deployment has the following limitations:

  • The Kubernetes Nodes and Kubernetes Cluster Grafana dashboards may have empty panels.

  • The DockerNetworkUnhealthy and etcdGRPCRequestsSlow alerts may fail to raise.

  • The CPUThrottlingHigh, CalicoDatapaneIfaceMsgBatchSizeHigh, KubeCPUOvercommitPods, KubeMemOvercommitPods alerts, and the TargetDown alert for the prometheus-node-exporter and calico-node pods may be constantly firing.


Support of multiple host-specific L2 templates per a bare metal cluster

Added support of multiple host-specific L2 templates to be applied to different nodes of the same bare metal cluster. Now, you can use several independent host-specific L2 templates on a cluster to support different hardware configurations. For example, you can create L2 templates with a different number and layout of NICs to be applied to the specific machines of a cluster.


Improvements in the Container Cloud logs collection

Improved user experience with the Container Cloud resources logs collection by implementing collecting of logs on the Mirantis Kubernetes Engine cluster and on all Kubernetes pods, including the ones that were previously removed or failed.

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.3.0 including the Cluster release 5.10.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[6222] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

Bare metal
[6988] LVM fails to deploy if the volume group name already exists

Fixed in Container Cloud 2.5.0

During a management or managed cluster deployment, LVM cannot be deployed on a new disk if an old volume group with the same name already exists on the target hardware node but on the different disk.

Workaround:

In the bare metal host profile specific to your hardware configuration, add the wipe: true parameter to the device that fails to be deployed. For the procedure details, see Operations Guide: Create a custom host profile.

[7655] Wrong status for an incorrectly configured L2 template

If an L2 template is configured incorrectly, a bare metal cluster is deployed successfully but with the runtime errors in the IpamHost object.

Workaround:

If you suspect that the machine is not working properly because of incorrect network configuration, verify the status of the corresponding IpamHost object. Inspect the l2RenderResult and ipAllocationResult object fields for error messages.

[8560] Manual deletion of BareMetalHost leads to its silent removal

Fixed in Container Cloud 2.5.0

If BareMetalHost is manually removed from a managed cluster, it is silently removed without a power-off and deprovision that leads to a managed cluster failures.

Workaround:

Do not manually delete a BareMetalHost that has the Provisioned status.

[9875] Full preflight fails with a timeout waiting for BareMetalHost

If you run bootstrap.sh preflight with KAAS_BM_FULL_PREFLIGHT=true, the script fails with the following message:

failed to create BareMetal objects: failed to wait for objects of kinds BareMetalHost
to become available: timed out waiting for the condition

As a workaround, unset full preflight using unset KAAS_BM_FULL_PREFLIGHT to run fast preflight instead.


IAM
[2757] IAM fails to start during management cluster deployment

Fixed in Container Cloud 2.4.0

During a management cluster deployment, IAM fails to start with the IAM pods being in the CrashLoopBackOff status.

Workaround:

  1. Log in to the bootstrap node.

  2. Remove the iam-mariadb-state configmap:

    kubectl delete cm -n kaas iam-mariadb-state
    
  3. Manually delete the mariadb pods:

    kubectl delete po -n kaas mariadb-server-{0,1,2}
    

    Wait for the pods to start. If the mariadb pod does not start with the connection to peer timed out exception, repeat the step 2.

  4. Obtain the MariaDB database admin password:

    kubectl get secrets -n kaas mariadb-dbadmin-password \
    -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d ; echo
    
  5. Log in to MariaDB:

    kubectl exec -it -n kaas mariadb-server-0 -- bash -c 'mysql -uroot -p<mysqlDbadminPassword>'
    

    Substitute <mysqlDbadminPassword> with the corresponding value obtained in the previous step.

  6. Run the following command:

    DROP DATABASE IF EXISTS keycloak;
    
  7. Manually delete the Keycloak pods:

    kubectl delete po -n kaas iam-keycloak-{0,1,2}
    

LCM
[10029] Authentication fails with the 401 Unauthorized error

Authentication may not work on some controller nodes after a managed cluster creation. As a result, the Kubernetes API operations with the managed cluster kubeconfig fail with Response Status: 401 Unauthorized.

As a workaround, manually restart the ucp-controller and ucp-auth Docker services on the affected node.


Management cluster update
[9899] Helm releases get stuck in PENDING_UPGRADE during cluster update

Helm releases may get stuck in the PENDING_UPGRADE status during a management or managed cluster upgrade. The HelmBundle controller cannot recover from this state and requires manual actions. The workaround below describes the recovery process for the openstack-operator release that stuck during a managed cluster update. Use it as an example for other Helm releases as required.

Workaround:

  1. Log in to the helm-controller pod console:

    kubectl exec -n kube-system -it helm-controller-0 sh -c tiller
    
  2. Identify the release that stuck in the PENDING_UPGRADE status. For example:

    ./helm --host=localhost:44134 history openstack-operator
    

    Example of system response:

    REVISION  UPDATED                   STATUS           CHART                      DESCRIPTION
    1         Tue Dec 15 12:30:41 2020  SUPERSEDED       openstack-operator-0.3.9   Install complete
    2         Tue Dec 15 12:32:05 2020  SUPERSEDED       openstack-operator-0.3.9   Upgrade complete
    3         Tue Dec 15 16:24:47 2020  PENDING_UPGRADE  openstack-operator-0.3.18  Preparing upgrade
    
  3. Roll back the failed release to the previous revision. For example:

    ./helm --host=localhost:44134 rollback openstack-operator 2
    

    Once done, the release will be reconciled.


Storage
[10060] Ceph OSD node removal fails

A Ceph node removal is not being triggered properly after updating the KaasCephCluster custom resource (CR). Both management and managed clusters are affected.

Workaround:

  1. Remove the parameters for a Ceph OSD from the KaasCephCluster CR as described in Operations Guide: Add, remove, or reconfigure Ceph nodes.

  2. Obtain the IDs of the osd and mon services that are located on the old node:

    1. Obtain the UID of the affected machine:

      kubectl get machine <CephOSDNodeName> -n <ManagedClusterProjectName> -o jsonpath='{.metadata.annotations.kaas\.mirantis\.com\/uid}'
      
    2. Export kubeconfig of your managed cluster. For example:

      export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
      
    3. Identify the pods IDs that run the osd and mon services:

      kubectl get pods -o wide -n rook-ceph | grep <affectedMachineUID> | grep -E "mon|osd"
      

      Example of the system response extract:

      rook-ceph-mon-c-7bbc5d757d-5bpws                              1/1  Running    1  6h1m
      rook-ceph-osd-2-58775d5568-5lklw                              1/1  Running    4  44h
      rook-ceph-osd-prepare-705ae6c647cfdac928c63b63e2e2e647-qn4m9  0/1  Completed  0  94s
      

      The pods IDs include the osd or mon services IDs. In the example system response above, the osd ID is 2 and the mon ID is c.

  3. Delete the deployments of the osd and mon services obtained in the previous step:

    kubectl delete deployment rook-ceph-osd(mon)-<ID> -n rook-ceph
    

    For example:

    kubectl delete deployment rook-ceph-mon-c -n rook-ceph
    kubectl delete deployment rook-ceph-osd-2 -n rook-ceph
    
  4. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  5. Rebalance the Ceph OSDs:

    ceph osd out osd(s).ID
    

    Wait for the rebalance to complete.

  6. Rebalance the Ceph data:

    ceph osd purge osd(s).ID
    

    Wait for the Ceph data to rebalance.

  7. Remove the old node from the Ceph OSD tree:

    ceph osd crush rm <NodeName>
    
  8. If the removed node contained mon services, remove them:

    ceph mon rm <monID>
    
[9928] Ceph rebalance during a managed cluster update

Fixed in Container Cloud 2.5.0

During a managed cluster update, Ceph rebalance leading to data loss may occur.

Workaround:

  1. Before updating a managed cluster:

    1. Log in to the ceph-tools pod:

      kubectl -n rook-ceph exec -it <ceph-tools-pod-name> bash
      
    2. Set the noout flag:

      ceph osd set noout
      
  2. Update a managed cluster.

  3. After updating a managed cluster:

    1. Log in to the ceph-tools pod:

      kubectl -n rook-ceph exec -it <ceph-tools-pod-name> bash
      
    2. Unset the noout flag:

      ceph osd unset noout
      
[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.

[10050] Ceph OSD pod is in the CrashLoopBackOff state after disk replacement

If you use a custom BareMetalHostProfile, after disk replacement on a Ceph OSD, the Ceph OSD pod switches to the CrashLoopBackOff state due to the Ceph OSD authorization key failing to be created properly.

Workaround:

  1. Export kubeconfig of your managed cluster. For example:

    export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
    
  2. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  3. Delete the authorization key for the failed Ceph OSD:

    ceph auth del osd.<ID>
    
  4. SSH to the node on which the Ceph OSD cannot be created.

  5. Clean up the disk that will be a base for the failed Ceph OSD. For details, see official Rook documentation.

    Note

    Ignore failures of the sgdisk --zap-all $DISK and blkdiscard $DISK commands if any.

  6. On the managed cluster, restart the Rook operator:

    kubectl -n rook-ceph delete pod -l app=rook-ceph-operator
    

Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Addressed issues

The following issues have been addressed in the Mirantis Container Cloud release 2.3.0 and the Cluster releases 5.10.0 and 6.10.0:

  • [8869] Upgraded kind from version 0.3.0 to 0.9.0 and the kindest/node image version from 1.14.2 to 1.18.8 to enhance the Container Cloud performance and prevent compatibility issues.

  • [8220] Fixed the issue with failure to switch the default label from one BareMetalHostProfile to another.

  • [7255] Fixed the issue with slow creation of the OpenStack clients and pools by redesigning as well as increasing efficiency and speed of ceph-controller.

  • [8618] Fixed the issue with missing pools during a Ceph cluster deployment.

  • [8111] Fixed the issue with a Ceph cluster being available after deleting it using the Container Cloud web UI or deleting the KaaSCephCluster object from the Kubernetes namespace using CLI.

  • [8409, 3836] Refactored and stabilized the upgrade procedure to prevent locks during the upgrade operations.

  • [8925] Fixed improper handling of errors in lcm-controller that may lead to its panic.

  • [8361] Fixed the issue with admission-controller allowing addition of duplicated node labels per machine.

  • [8402] Fixed the issue with the AWS provider failing during node labeling with the Observed a panic: “invalid memory address or nil pointer dereference” error if privateIP is not set for a machine.

  • [7673] Moved logs collection of the bootstrap cluster to the /bootstrap subdirectory to prevent unintentional erasure of the management and regional cluster logs.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.3.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.15.4

aws-credentials-controller

1.15.4

Bare metal

baremetal-operator Updated

3.1.6

baremetal-public-api Updated

3.1.6

baremetal-provider Updated

1.15.4

httpd

2.4.46-20201001171500

ironic Updated

ussuri-bionic-20201111180110

ironic-operator Updated

base-bionic-20201106182102

kaas-ipam Updated

20201210175212

local-volume-provisioner

1.0.4-mcp

mariadb

10.4.14-bionic-20200812025059

IAM

iam

1.1.22

iam-controller Updated

1.15.4

keycloak

9.0.0

Container Cloud Updated

admission-controller

1.15.4

byo-credentials-controller

1.15.4

byo-provider

1.15.4

kaas-public-api

1.15.4

kaas-exporter

1.15.4

kaas-ui

1.15.4

lcm-controller

0.2.0-224-g5c413d37

release-controller

1.15.4

OpenStack Updated

openstack-provider

1.15.4

os-credentials-controller

1.15.4

VMWare vSphere Updated

vsphere-provider

1.15.4

vsphere-credentials-controller

1.15.4

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.3.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-3.1.6.tgz

baremetal-public-api Updated

https://binary.mirantis.com/bm/helm/baremetal-public-api-3.1.6.tgz

ironic-python-agent.kernel Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-ussuri-bionic-debug-20201119132200

ironic-python-agent.initramfs Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-ussuri-bionic-debug-20201119132200

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-3.1.6.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator Updated

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20201113171304

httpd

mirantis.azurecr.io/bm/external/httpd:2.4.46-20201001171500

ironic Updated

mirantis.azurecr.io/openstack/ironic:ussuri-bionic-20201111180110

ironic-inspector Updated

mirantis.azurecr.io/openstack/ironic-inspector:ussuri-bionic-20201111180110

ironic-operator Updated

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20201106182102

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20201210175212

mariadb

mirantis.azurecr.io/general/mariadb:10.4.14-bionic-20200812025059


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.15.4.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.15.4.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.15.4.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.15.4.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.15.4.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.15.4.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.15.4.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.15.4.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.15.4.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.15.4.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.15.4.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.15.4.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.15.4.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.15.4.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.15.4.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.15.4.tgz

vsphere-credentials-controller

https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.15.4.tgz

vsphere-provider

https://binary.mirantis.com/core/helm/vsphere-provider-1.15.4.tgz

Docker images Updated

admission-controller

mirantis.azurecr.io/core/admission-controller:1.15.4

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.15.4

aws-credentials-controller

mirantis.azurecr.io/core/aws-credentials-controller:1.15.4

byo-credentials-controller

mirantis.azurecr.io/core/byo-credentials-controller:1.15.4

cluster-api-provider-baremetal

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.15.4

frontend

mirantis.azurecr.io/core/frontend:1.15.4

iam-controller

mirantis.azurecr.io/core/iam-controller:1.15.4

lcm-controller

mirantis.azurecr.io/core/lcm-controller:v0.2.0-224-g5c413d37

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.15.4

os-credentials-controller

mirantis.azurecr.io/core/os-credentials-controller:1.15.4

release-controller

mirantis.azurecr.io/core/release-controller:1.15.4

vsphere-credentials-controller

mirantis.azurecr.io/core/vsphere-credentials-controller:1.15.4


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin

iamctl-linux

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-linux

iamctl-darwin

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-darwin

iamctl-windows

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-windows

Helm charts

iam Updated

http://binary.mirantis.com/iam/helm/iam-1.1.22.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.3.tgz

keycloak-proxy Updated

http://binary.mirantis.com/core/helm/keycloak_proxy-1.14.3.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.3.18

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.3.18

kubernetes-entrypoint

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/iam/external/mariadb:10.2.18

keycloak Updated

mirantis.azurecr.io/iam/keycloak:0.3.19

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

2.2.0

This section outlines release notes for the Mirantis Container Cloud GA release 2.2.0. This release introduces support for the Cluster release 5.9.0 that is based on Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18. This release also introduces support for the Cluster release 6.8.1 that introduces the support of the Mirantis OpenStack for Kubernetes (MOS) product.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.2.0. For the list of enhancements in the Cluster release 5.9.0 and Cluster release 6.8.1 introduced by the Container Cloud release 2.2.0, see 5.9.0 and 6.8.1.


Support for VMWare vSphere provider on RHEL

TECHNICAL PREVIEW

Introduced the Technology Preview support for the VMWare vSphere cloud provider on RHEL, including support for creation and operating of managed clusters using the Container Cloud web UI.

Deployment of an additional regional vSphere-based cluster or attaching an existing Mirantis Kubernetes Engine (MKE) cluster to a vSphere-based management cluster is on the development stage and will be announced in one of the following Container Cloud releases.

Note

For the Technology Preview feature definition, refer to the Container Cloud Reference Architecture.


Kernel parameters management through BareMetalHostProfile

Implemented the API for managing kernel parameters typically managed by sysctl for bare metal hosts through the BareMetalHost and BareMetalHostProfile objects fields.


Support of multiple subnets per cluster

Implemented support of multiple subnets per a Container Cloud cluster with an ability to specify a different network type for each subnet. Introduced the SubnetPool object that allows for automatic creation of the Subnet objects. Also, added the L3Layout section to L2Template.spec. The L3Layout configuration allows defining the subnets scopes to be used and to enable auto-creation of subnets from a subnet pool.


Optimization of the Container Cloud logs collection

Optimized user experience with the Container Cloud resources logs collection:

  • Added a separate file with a human-readable table that contains information about cluster events

  • Implemented collecting of system logs from cluster nodes


Container Cloud API documentation for bare metal

On top of continuous improvements delivered to the existing Container Cloud guides, added the Mirantis Container Cloud API section to the Operations Guide. This section is intended only for advanced Infrastructure Operators who are familiar with Kubernetes Cluster API.

Currently, this section contains descriptions and examples of the Container Cloud API resources for the bare metal cloud provider. The API documentation for the OpenStack, AWS, and VMWare vSphere API resources will be added in the upcoming Container Cloud releases.

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.2.0 including the Cluster release 5.9.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[6222] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

Bare metal
[6988] LVM fails to deploy if the volume group name already exists

Fixed in Container Cloud 2.5.0

During a management or managed cluster deployment, LVM cannot be deployed on a new disk if an old volume group with the same name already exists on the target hardware node but on the different disk.

Workaround:

In the bare metal host profile specific to your hardware configuration, add the wipe: true parameter to the device that fails to be deployed. For the procedure details, see Operations Guide: Create a custom host profile.

[7655] Wrong status for an incorrectly configured L2 template

If an L2 template is configured incorrectly, a bare metal cluster is deployed successfully but with the runtime errors in the IpamHost object.

Workaround:

If you suspect that the machine is not working properly because of incorrect network configuration, verify the status of the corresponding IpamHost object. Inspect the l2RenderResult and ipAllocationResult object fields for error messages.

[8560] Manual deletion of BareMetalHost leads to its silent removal

Fixed in Container Cloud 2.5.0

If BareMetalHost is manually removed from a managed cluster, it is silently removed without a power-off and deprovision that leads to a managed cluster failures.

Workaround:

Do not manually delete a BareMetalHost that has the Provisioned status.


IAM
[2757] IAM fails to start during management cluster deployment

Fixed in Container Cloud 2.4.0

During a management cluster deployment, IAM fails to start with the IAM pods being in the CrashLoopBackOff status.

Workaround:

  1. Log in to the bootstrap node.

  2. Remove the iam-mariadb-state configmap:

    kubectl delete cm -n kaas iam-mariadb-state
    
  3. Manually delete the mariadb pods:

    kubectl delete po -n kaas mariadb-server-{0,1,2}
    

    Wait for the pods to start. If the mariadb pod does not start with the connection to peer timed out exception, repeat the step 2.

  4. Obtain the MariaDB database admin password:

    kubectl get secrets -n kaas mariadb-dbadmin-password \
    -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d ; echo
    
  5. Log in to MariaDB:

    kubectl exec -it -n kaas mariadb-server-0 -- bash -c 'mysql -uroot -p<mysqlDbadminPassword>'
    

    Substitute <mysqlDbadminPassword> with the corresponding value obtained in the previous step.

  6. Run the following command:

    DROP DATABASE IF EXISTS keycloak;
    
  7. Manually delete the Keycloak pods:

    kubectl delete po -n kaas iam-keycloak-{0,1,2}
    

Storage
[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.


Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Addressed issues

The following issues have been addressed in the Mirantis Container Cloud release 2.2.0 including the Cluster release 5.9.0:

  • [8012] Fixed the issue with helm-controller pod being stuck in the CrashLoopBackOff state after reattaching of a Mirantis Kubernetes Engine (MKE) cluster.

  • [7131] Fixed the issue with the deployment of a managed cluster failing during the Ceph Monitor or Manager deployment.

  • [6164] Fixed the issue with the number of placement groups (PGs) per Ceph OSD being too small and the Ceph cluster having the HEALTH_WARN status.

  • [8302] Fixed the issue with deletion of a regional cluster leading to the deletion of the related management cluster.

  • [7722] Fixed the issue with the Internal Server Error or similar errors appearing in the HelmBundle controller logs after bootstrapping the management cluster.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.2.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.14.0

aws-credentials-controller

1.14.0

Bare metal

baremetal-operator Updated

3.1.3

baremetal-public-api Updated

3.1.3

baremetal-provider Updated

1.14.0

httpd

2.4.46-20201001171500

ironic Updated

ussuri-bionic-20201021180016

ironic-operator Updated

base-bionic-20201023172943

kaas-ipam Updated

20201026094912

local-volume-provisioner

1.0.4-mcp

mariadb

10.4.14-bionic-20200812025059

IAM

iam Updated

1.1.22

iam-controller Updated

1.14.0

keycloak

9.0.0

Container Cloud Updated

admission-controller

1.14.0

byo-credentials-controller

1.14.0

byo-provider

1.14.3

kaas-public-api

1.14.0

kaas-exporter

1.14.0

kaas-ui

1.14.2

lcm-controller

0.2.0-178-g8cc488f8

release-controller

1.14.0

OpenStack Updated

openstack-provider

1.14.0

os-credentials-controller

1.14.0

VMWare vSphere New

vsphere-provider

1.14.1

vsphere-credentials-controller

1.14.1

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.2.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

Target system image (ubuntu-bionic)

https://binary.mirantis.com/bm/bin/efi/ubuntu/qcow2-bionic-debug-20200730084816

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-3.1.3.tgz

baremetal-public-api Updated

https://binary.mirantis.com/bm/helm/baremetal-public-api-3.1.3.tgz

ironic-python-agent.kernel Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-ussuri-bionic-debug-20201022084817

ironic-python-agent.initramfs Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-ussuri-bionic-debug-20201022084817

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-3.1.3.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator Updated

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20201028131325

httpd

mirantis.azurecr.io/bm/external/httpd:2.4.46-20201001171500

ironic Updated

mirantis.azurecr.io/openstack/ironic:ussuri-bionic-20201021180016

ironic-inspector Updated

mirantis.azurecr.io/openstack/ironic-inspector:ussuri-bionic-20201021180016

ironic-operator Updated

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20201023172943

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20201026094912

mariadb

mirantis.azurecr.io/general/mariadb:10.4.14-bionic-20200812025059


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.14.0.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.14.0.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.14.0.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.14.0.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.14.0.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.14.0.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.14.0.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.14.3.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.14.0.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.14.0.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.14.0.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.14.2.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.14.0.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.14.0.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.14.0.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.14.0.tgz

vsphere-credentials-controller New

https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.14.1.tgz

vsphere-provider New

https://binary.mirantis.com/core/helm/vsphere-provider-1.14.1.tgz

Docker images for Container Cloud deployment

admission-controller Updated

mirantis.azurecr.io/core/admission-controller:1.14.0

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.14.0

aws-credentials-controller Updated

mirantis.azurecr.io/core/aws-credentials-controller:1.14.0

byo-credentials-controller Updated

mirantis.azurecr.io/core/byo-credentials-controller:1.14.0

cluster-api-provider-baremetal Updated

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.14.0

frontend

mirantis.azurecr.io/core/frontend:1.14.2

iam-controller Updated

mirantis.azurecr.io/core/iam-controller:1.14.0

lcm-controller Updated

mirantis.azurecr.io/core/lcm-controller:v0.2.0-178-g8cc488f8

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.14.0

os-credentials-controller Updated

mirantis.azurecr.io/core/os-credentials-controller:1.14.0

release-controller Updated

mirantis.azurecr.io/core/release-controller:1.14.0

vsphere-credentials-controller New

mirantis.azurecr.io/core/vsphere-credentials-controller:1.14.1


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin

iamctl-linux

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-linux

iamctl-darwin

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-darwin

iamctl-windows

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-windows

Helm charts

iam Updated

http://binary.mirantis.com/iam/helm/iam-1.1.22.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

keycloak-proxy Updated

http://binary.mirantis.com/core/helm/keycloak_proxy-1.14.3.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.3.18

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.3.18

kubernetes-entrypoint

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/iam/external/mariadb:10.2.18

keycloak Updated

mirantis.azurecr.io/iam/keycloak:0.3.19

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

2.1.0

This section outlines release notes for the Mirantis Container Cloud GA release 2.1.0. This release introduces support for the Cluster release 5.8.0 that is based on Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.1.0. For the list of enhancements in the Cluster release 5.8.0 introduced by the KaaS release 2.1.0, see 5.8.0.


Node labeling for machines

Implemented the possibility to assign labels to specific machines with dedicated system and hardware resources through the Container Cloud web UI. For example, you can label the StackLight nodes that run Elasticsearch and require more resources than a standard node to run the StackLight components services on the dedicated nodes. You can label a machine before or after it is deployed. The list of available labels is taken from the current Cluster release.

Node labeling greatly improves cluster performance and prevents pods from being quickly exhausted.


AWS resources discovery in Container Cloud web UI

Improved the user experience during a managed cluster creation using the Container Cloud web UI by implementing drop-down menus with available supported values for the following AWS resources:

  • AWS region

  • AWS AMI ID

  • AWS instance type

To apply the feature to existing deployments, update the IAM policies for AWS as described in Apply updates to the AWS-based management clusters.


Credentials statuses for OpenStack and AWS

Implemented the following statuses for the OpenStack-based and AWS-based credentials in the Container Cloud web UI:

  • Ready

    Credentials are valid and ready to be used for a managed cluster creation.

  • In Use

    Credentials are being used by a managed cluster.

  • Error

    Credentials are invalid. You can hover over the Error status to determine the reason of the issue.

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.1.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[6222] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

Bare metal
[6988] LVM fails to deploy if the volume group name already exists

Fixed in Container Cloud 2.5.0

During a management or managed cluster deployment, LVM cannot be deployed on a new disk if an old volume group with the same name already exists on the target hardware node but on the different disk.

Workaround:

In the bare metal host profile specific to your hardware configuration, add the wipe: true parameter to the device that fails to be deployed. For the procedure details, see Operations Guide: Create a custom host profile.


IAM
[2757] IAM fails to start during management cluster deployment

Fixed in Container Cloud 2.4.0

During a management cluster deployment, IAM fails to start with the IAM pods being in the CrashLoopBackOff status.

Workaround:

  1. Log in to the bootstrap node.

  2. Remove the iam-mariadb-state configmap:

    kubectl delete cm -n kaas iam-mariadb-state
    
  3. Manually delete the mariadb pods:

    kubectl delete po -n kaas mariadb-server-{0,1,2}
    

    Wait for the pods to start. If the mariadb pod does not start with the connection to peer timed out exception, repeat the step 2.

  4. Obtain the MariaDB database admin password:

    kubectl get secrets -n kaas mariadb-dbadmin-password \
    -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d ; echo
    
  5. Log in to MariaDB:

    kubectl exec -it -n kaas mariadb-server-0 -- bash -c 'mysql -uroot -p<mysqlDbadminPassword>'
    

    Substitute <mysqlDbadminPassword> with the corresponding value obtained in the previous step.

  6. Run the following command:

    DROP DATABASE IF EXISTS keycloak;
    
  7. Manually delete the Keycloak pods:

    kubectl delete po -n kaas iam-keycloak-{0,1,2}
    

Storage
[6164] Small number of PGs per Ceph OSD

Fixed in 2.2.0

After deploying a managed cluster with Ceph, the number of placement groups (PGs) per Ceph OSD may be too small and the Ceph cluster may have the HEALTH_WARN status:

health: HEALTH_WARN
        too few PGs per OSD (3 < min 30)

The workaround is to enable the PG balancer to properly manage the number of PGs:

kexec -it $(k get pod -l "app=rook-ceph-tools" --all-namespaces -o jsonpath='{.items[0].metadata.name}') -n rook-ceph bash
ceph mgr module enable pg_autoscaler
[7131] rook-ceph-mgr fails during managed cluster deployment

Fixed in 2.2.0

Occasionally, the deployment of a managed cluster may fail during the Ceph Monitor or Manager deployment. In this case, the Ceph cluster may be down and and a stack trace similar to the following one may be present in Ceph Manager logs:

kubectl -n rook-ceph logs rook-ceph-mgr-a-c5dc846f8-k68rs

/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/mon/MonMap.h: In function 'void MonMap::add(const mon_info_t&)' thread 7fd3d3744b80 time 2020-09-03 10:16:46.586388
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/mon/MonMap.h: 195: FAILED ceph_assert(addr_mons.count(a) == 0)
ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fd3ca9b2875]
2: (()+0x253a3d) [0x7fd3ca9b2a3d]
3: (MonMap::add(mon_info_t const&)+0x80) [0x7fd3cad49190]
4: (MonMap::add(std::string const&, entity_addrvec_t const&, int)+0x110) [0x7fd3cad493a0]
5: (MonMap::init_with_ips(std::string const&, bool, std::string const&)+0xc9) [0x7fd3cad43849]
6: (MonMap::build_initial(CephContext*, bool, std::ostream&)+0x314) [0x7fd3cad45af4]
7: (MonClient::build_initial_monmap()+0x130) [0x7fd3cad2e140]
8: (MonClient::get_monmap_and_config()+0x5f) [0x7fd3cad365af]
9: (global_pre_init(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int)+0x524) [0x55ce86711444]
10: (global_init(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, char const*, bool)+0x76) [0x55ce86711b56]
11: (main()+0x136) [0x55ce864ff9a6]
12: (__libc_start_main()+0xf5) [0x7fd3c6e73555]
13: (()+0xfc010) [0x55ce86505010]

The workaround is to start the managed cluster deployment from scratch.

[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.


Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Addressed issues

In the Mirantis Container Cloud release 2.1.0, the following issues have been addressed:

  • [7281] Fixed the issue with a management cluster bootstrap script failing if there was a space in the PATH environment variable.

  • [7205] Fixed the issue with some cluster objects being stuck during deletion of an AWS-based managed cluster due to unresolved VPC dependencies.

  • [7304] Fixed the issue with failure to reattach a Mirantis Kubernetes Engine (MKE) cluster with the same name.

  • [7101] Fixed the issue with the monitoring of Ceph and Ironic being enabled when Ceph and Ironic are disabled on the baremetal-based clusters.

  • [7324] Fixed the issue with the monitoring of Ceph being disabled on the baremetal-based managed clusters due to the missing provider: BareMetal parameter.

  • [7180] Fixed the issue with lcm-controller periodically failing with the invalid memory address or nil pointer dereference runtime error.

  • [7251] Fixed the issue with setting up the OIDC integration on the MKE side.

  • [7326] Fixed the issue with the missing entry for the host itself in etc/hosts causing failure of services that require node FQDN.

  • [6989] Fixed the issue with baremetal-operator ignoring the clean failed provisioning state if a node fails to deploy on a baremetal-based managed cluster.

  • [7231] Fixed the issue with the baremetal-provider pod not restarting after the ConfigMap changes and causing the telemeter-client pod to fail during deployment.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.1.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.12.2

aws-credentials-controller

1.12.2

Bare metal

baremetal-operator Updated

3.1.0

baremetal-public-api New

3.1.0

baremetal-provider Updated

1.12.2

httpd Updated

2.4.46-20201001171500

ironic

train-bionic-20200803180020

ironic-operator

base-bionic-20200805144858

kaas-ipam Updated

20201007180518

local-volume-provisioner

1.0.4-mcp

mariadb Updated

10.4.14-bionic-20200812025059

IAM

iam Updated

1.1.18

iam-controller Updated

1.12.2

keycloak

9.0.0

Container Cloud Updated

admission-controller

1.12.3

byo-credentials-controller

1.12.2

byo-provider

1.12.2

kaas-public-api

1.12.2

kaas-exporter

1.12.2

kaas-ui

1.12.2

lcm-controller

0.2.0-169-g5668304d

release-controller

1.12.2

OpenStack Updated

openstack-provider

1.12.2

os-credentials-controller

1.12.2

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.1.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

Target system image (ubuntu-bionic)

https://binary.mirantis.com/bm/bin/efi/ubuntu/qcow2-bionic-debug-20200730084816

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-3.1.0.tgz

baremetal-public-api New

https://binary.mirantis.com/bm/helm/baremetal-public-api-3.1.0.tgz

ironic-python-agent.kernel

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-train-bionic-debug-20200730084816

ironic-python-agent.initramfs

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-train-bionic-debug-20200730084816

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-3.1.0.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator Updated

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20201005150946

httpd Updated

mirantis.azurecr.io/bm/external/httpd:2.4.46-20201001171500

ironic

mirantis.azurecr.io/openstack/ironic:train-bionic-20200803180020

ironic-inspector

mirantis.azurecr.io/openstack/ironic-inspector:train-bionic-20200803180020

ironic-operator

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20200805144858

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20201007180518

mariadb Updated

mirantis.azurecr.io/general/mariadb:10.4.14-bionic-20200812025059


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.12.2.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.12.2.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.12.3.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.12.2.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.12.2.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.12.2.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.12.2.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.12.2.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.12.2.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.12.2.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.12.2.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.12.2.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.12.2.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.12.2.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.12.2.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.12.2.tgz

Docker images for Container Cloud deployment Updated

admission-controller

mirantis.azurecr.io/core/admission-controller:1.12.3

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.12.2

aws-credentials-controller

mirantis.azurecr.io/core/aws-credentials-controller:1.12.2

byo-credentials-controller

mirantis.azurecr.io/core/byo-credentials-controller:1.12.2

cluster-api-provider-baremetal

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.12.2

frontend

mirantis.azurecr.io/core/frontend:1.12.2

iam-controller

mirantis.azurecr.io/core/iam-controller:1.12.2

lcm-controller

mirantis.azurecr.io/core/lcm-controller:v0.2.0-169-g5668304d

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.12.2

os-credentials-controller

mirantis.azurecr.io/core/os-credentials-controller:1.12.2

release-controller

mirantis.azurecr.io/core/release-controller:1.12.2


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-236-9cea809-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-236-9cea809-darwin

iamctl-linux

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-linux

iamctl-darwin

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-darwin

iamctl-windows

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-windows

Helm charts

iam Updated

http://binary.mirantis.com/iam/helm/iam-1.1.18.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

keycloak-proxy Updated

http://binary.mirantis.com/core/helm/keycloak_proxy-1.12.2.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.3.18

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.3.18

kubernetes-entrypoint

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/iam/external/mariadb:10.2.18

keycloak

mirantis.azurecr.io/iam/keycloak:0.3.18

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

Apply updates to the AWS-based management clusters

To complete the AWS-based management cluster upgrade to version 2.1.0, manually update the IAM policies for AWS before updating your AWS-based managed clusters.

To update the IAM policies for AWS:

  1. Choose from the following options:

    • Update the IAM policies using get_container_cloud.sh:

      1. On any local machine, download and run the latest version of the Container Cloud bootstrap script:

        wget https://binary.mirantis.com/releases/get_container_cloud.sh
        
        chmod 0755 get_container_cloud.sh
        
        ./get_container_cloud.sh
        
      2. Change the directory to the kaas-bootstrap folder created by the get_container_cloud.sh script.

      3. Export the following parameters by adding the corresponding values for the AWS admin credentials:

        export AWS_SECRET_ACCESS_KEY=XXXXXXX
        export AWS_ACCESS_KEY_ID=XXXXXXX
        export AWS_DEFAULT_REGION=us-east-2
        
      4. Update the AWS CloudFormation template for IAM policy:

        ./kaas bootstrap aws policy
        
    • Update the IAM policies using the AWS Management Console:

      1. Log in to your AWS Management Console.

      2. Verify that the controllers.cluster-api-provider-aws.kaas.mirantis.com role or another AWS role that you use for Container Cloud users contains the following permissions:

        "ec2:DescribeRegions", "ec2:DescribeInstanceTypes"
        

        Otherwise, add these permissions manually.

  2. Proceed to updating your AWS-based managed clusters as described in Operations Guide: Update a managed cluster.

2.0.0

This section outlines release notes for the initial Mirantis Container Cloud GA release 2.0.0. This release introduces support for the Cluster release 5.7.0 that is based on Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18.

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.0.0.


AWS
[6222] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

Bare metal
[6988] LVM fails to deploy if the volume group name already exists

Fixed in Container Cloud 2.5.0

During a management or managed cluster deployment, LVM cannot be deployed on a new disk if an old volume group with the same name already exists on the target hardware node but on the different disk.

Workaround:

In the bare metal host profile specific to your hardware configuration, add the wipe: true parameter to the device that fails to be deployed. For the procedure details, see Operations Guide: Create a custom host profile.


IAM
[2757] IAM fails to start during management cluster deployment

Fixed in Container Cloud 2.4.0

During a management cluster deployment, IAM fails to start with the IAM pods being in the CrashLoopBackOff status.

Workaround:

  1. Log in to the bootstrap node.

  2. Remove the iam-mariadb-state configmap:

    kubectl delete cm -n kaas iam-mariadb-state
    
  3. Manually delete the mariadb pods:

    kubectl delete po -n kaas mariadb-server-{0,1,2}
    

    Wait for the pods to start. If the mariadb pod does not start with the connection to peer timed out exception, repeat the step 2.

  4. Obtain the MariaDB database admin password:

    kubectl get secrets -n kaas mariadb-dbadmin-password \
    -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d ; echo
    
  5. Log in to MariaDB:

    kubectl exec -it -n kaas mariadb-server-0 -- bash -c 'mysql -uroot -p<mysqlDbadminPassword>'
    

    Substitute <mysqlDbadminPassword> with the corresponding value obtained in the previous step.

  6. Run the following command:

    DROP DATABASE IF EXISTS keycloak;
    
  7. Manually delete the Keycloak pods:

    kubectl delete po -n kaas iam-keycloak-{0,1,2}
    

StackLight
[7101] Monitoring of disabled components

Fixed in 2.1.0

On the baremetal-based clusters, the monitoring of Ceph and Ironic is enabled when Ceph and Ironic are disabled. The issue with Ceph relates to both management or managed clusters, the issue with Ironic relates to managed clusters only.

Workaround:

  1. Open the StackLight configuration manifest as described in Operations Guide: Configure StackLight.

  2. Add the following parameter to the StackLight helmReleases values of the Cluster object to explicitly disable the required component monitoring:

    • For Ceph:

      helmReleases:
        - name: stacklight
          values:
            ...
            ceph:
              disabledOnBareMetal: true
            ...
      
    • For Ironic:

      helmReleases:
        - name: stacklight
          values:
            ...
            ironic:
              disabledOnBareMetal: true
            ...
      
[7324] Ceph monitoring disabled

Fixed in 2.1.0

Ceph monitoring may be disabled on the baremetal-based managed clusters due to a missing provider: BareMetal parameter.

Workaround:

  1. Open the StackLight configuration manifest as described in Operations Guide: Configure StackLight.

  2. Add the provider: BareMetal parameter to the StackLight helmReleases values of the Cluster object:

    spec:
      providerSpec:
        value:
          helmReleases:
          - name: stacklight
            values:
              ...
              provider: BareMetal
              ...
    

Storage
[6164] Small number of PGs per Ceph OSD

Fixed in 2.2.0

After deploying a managed cluster with Ceph, the number of placement groups (PGs) per Ceph OSD may be too small and the Ceph cluster may have the HEALTH_WARN status:

health: HEALTH_WARN
        too few PGs per OSD (3 < min 30)

The workaround is to enable the PG balancer to properly manage the number of PGs:

kexec -it $(k get pod -l "app=rook-ceph-tools" --all-namespaces -o jsonpath='{.items[0].metadata.name}') -n rook-ceph bash
ceph mgr module enable pg_autoscaler
[7131] rook-ceph-mgr fails during managed cluster deployment

Fixed in 2.2.0

Occasionally, the deployment of a managed cluster may fail during the Ceph Monitor or Manager deployment. In this case, the Ceph cluster may be down and and a stack trace similar to the following one may be present in Ceph Manager logs:

kubectl -n rook-ceph logs rook-ceph-mgr-a-c5dc846f8-k68rs

/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/mon/MonMap.h: In function 'void MonMap::add(const mon_info_t&)' thread 7fd3d3744b80 time 2020-09-03 10:16:46.586388
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/mon/MonMap.h: 195: FAILED ceph_assert(addr_mons.count(a) == 0)
ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fd3ca9b2875]
2: (()+0x253a3d) [0x7fd3ca9b2a3d]
3: (MonMap::add(mon_info_t const&)+0x80) [0x7fd3cad49190]
4: (MonMap::add(std::string const&, entity_addrvec_t const&, int)+0x110) [0x7fd3cad493a0]
5: (MonMap::init_with_ips(std::string const&, bool, std::string const&)+0xc9) [0x7fd3cad43849]
6: (MonMap::build_initial(CephContext*, bool, std::ostream&)+0x314) [0x7fd3cad45af4]
7: (MonClient::build_initial_monmap()+0x130) [0x7fd3cad2e140]
8: (MonClient::get_monmap_and_config()+0x5f) [0x7fd3cad365af]
9: (global_pre_init(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int)+0x524) [0x55ce86711444]
10: (global_init(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, char const*, bool)+0x76) [0x55ce86711b56]
11: (main()+0x136) [0x55ce864ff9a6]
12: (__libc_start_main()+0xf5) [0x7fd3c6e73555]
13: (()+0xfc010) [0x55ce86505010]

The workaround is to start the managed cluster deployment from scratch.

[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.


Bootstrap
[7281] Space in PATH causes failure of bootstrap process

Fixed in 2.1.0

A management cluster bootstrap script fails if there is a space in the PATH environment variable. As a workaround, before running the bootstrap.sh script, verify that there are no spaces in the PATH environment variable.


Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.0.0.

Container Cloud release components versions

Component

Application/Service

Version

AWS

aws-provider

1.10.12

aws-credentials-controller

1.10.12

Bare metal

baremetal-operator

3.0.7

baremetal-provider

1.10.12

httpd

2.4.43-20200710111500

ironic

train-bionic-20200803180020

ironic-operator

base-bionic-20200805144858

kaas-ipam

20200807130953

local-volume-provisioner

1.0.4-mcp

mariadb

10.4.12-bionic-20200803130834

IAM

iam

1.1.16

iam-controller

1.10.12

keycloak

9.0.0

Container Cloud

admission-controller

1.10.12

byo-credentials-controller

1.10.12

byo-provider

1.10.12

kaas-public-api

1.10.12

kaas-exporter

1.10.12

kaas-ui

1.10.12

lcm-controller

0.2.0-149-g412c5a05

release-controller

1.10.12

OpenStack

openstack-provider

1.10.12

os-credentials-controller

1.10.12

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.0.0.


Bare metal artifacts

Artifact

Component

Path

Binaries

Target system image (ubuntu-bionic)

https://binary.mirantis.com/bm/bin/efi/ubuntu/qcow2-bionic-debug-20200730084816

baremetal-operator

https://binary.mirantis.com/bm/helm/baremetal-operator-3.0.7.tgz

ironic-python-agent.kernel

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-train-bionic-debug-20200730084816

ironic-python-agent.initramfs

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-train-bionic-debug-20200730084816

kaas-ipam

https://binary.mirantis.com/bm/helm/kaas-ipam-3.0.7.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20200812172956

httpd

mirantis.azurecr.io/bm/external/httpd:2.4.43-20200710111500

ironic

mirantis.azurecr.io/openstack/ironic:train-bionic-20200803180020

ironic-inspector

mirantis.azurecr.io/openstack/ironic-inspector:train-bionic-20200803180020

ironic-operator

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20200805144858

kaas-ipam

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20200807130953

mariadb

mirantis.azurecr.io/general/mariadb:10.4.12-bionic-20200803130834


Core components artifacts

Artifact

Component

Path

Bootstrap tarball

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.10.12.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.10.12.tar.gz

Helm charts

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.10.12.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.10.12.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.10.12.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.10.12.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.10.12.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.10.12.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.10.12.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.10.12.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.10.12.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.10.12.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.10.12.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.10.12.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.10.12.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.10.12.tgz

Docker images for Container Cloud deployment

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.10.12

aws-credentials-controller

mirantis.azurecr.io/core/aws-credentials-controller:1.10.12

byo-credentials-controller

mirantis.azurecr.io/core/byo-credentials-controller:1.10.12

cluster-api-provider-baremetal

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.10.12

frontend

mirantis.azurecr.io/core/frontend:1.10.12

iam-controller

mirantis.azurecr.io/core/iam-controller:1.10.12

lcm-controller

mirantis.azurecr.io/core/lcm-controller:v0.2.0-149-g412c5a05

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.10.12

os-credentials-controller

mirantis.azurecr.io/core/os-credentials-controller:1.10.12

release-controller

mirantis.azurecr.io/core/release-controller:1.10.12


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-236-9cea809-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-236-9cea809-darwin

iamctl-linux

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-linux

iamctl-darwin

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-darwin

iamctl-windows

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-windows

Helm charts

iam

http://binary.mirantis.com/iam/helm/iam-1.1.16.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

keycloak-proxy

http://binary.mirantis.com/core/helm/keycloak_proxy-1.10.12.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.3.18

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.3.18

kubernetes-entrypoint

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/iam/external/mariadb:10.2.18

keycloak

mirantis.azurecr.io/iam/keycloak:0.3.18

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

Cluster releases (managed)

This section outlines the release notes for the Mirantis Container Cloud Cluster releases that are supported by specific Container Cloud releases. For details about the Container Cloud releases, see: Mirantis Container Cloud releases.

6.12.0

The Cluster release 6.12.0 is introduced in the Mirantis Container Cloud release 2.5.0 and is based on the Cluster release 5.12.0.

The Cluster release 6.12.0 supports:

  • Mirantis OpenStack for Kubernetes (MOS) 21.1. For details, see MOS Release Notes.

  • Updated versions of Mirantis Kubernetes Engine 3.3.6 and Mirantis Container Runtime 19.03.14. For details, see MKE, MSR, and MCR Release Notes.

  • Kubernetes 1.18.

For the list of resolved issues, refer to the Container Cloud releases 2.4.0 and 2.5.0 sections. For the list of known issues, refer to the Container Cloud release 2.5.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 6.12.0.


Alert inhibition rules

Implemented alert inhibition rules to provide a clearer view on the cloud status and simplify troubleshooting. Using alert inhibition rules, Alertmanager decreases alert noise by suppressing dependent alerts notifications. The feature is enabled by default. For details, see Alert dependencies.


Integration between Grafana and Kibana

Implemented integration between Grafana and Kibana by adding a View logs in Kibana link to the majority of Grafana dashboards, which allows you to immediately view contextually relevant logs through the Kibana web UI.


Telegraf alert

Implemented the TelegrafGatherErrors alert that raises if Telegraf fails to gather metrics.

Learn more

Telegraf alerts


Configuration of Ironic Telegraf input plugin

Added the ironic.insecure parameter for enabling or disabling the host and chain verification for bare metal Ironic monitoring.


Automatically defined cluster ID

Enhanced StackLight to automatically set clusterId that defines an ID of a Container Cloud cluster. Now, you do not need to set or modify this parameter manually when configuring the sf-notifier and sf-reporter services.


Cerebro support for StackLight

Enhanced StackLight by adding support for Cerebro, a web UI that visualizes health of Elasticsearch clusters and allows for convenient debugging. Cerebro is disabled by default.


Ceph maintenance label

Implemented the maintenance label to set for Ceph during a managed cluster update. This prevents Ceph rebalance leading to data loss during a managed cluster update.


RGW check box in Container Cloud web UI

Implemented the Enable Object Storage checkbox in the Container Cloud web UI to allow enabling a single-instance RGW Object Storage when creating a Ceph cluster as described in Add a Ceph cluster.


Ceph RGW HA

Enhanced Ceph to support RADOS Gateway (RGW) high availability. Now, you can run multiple instances of Ceph RGW in active/active mode.


StackLight proxy

Added proxy support for Alertmanager, Metric collector, Salesforce notifier and reporter, and Telemeter client. Now, these StackLight components automatically use the same proxy that is configured for Container Cloud clusters.

Note

Proxy handles only the HTTP and HTTPS traffic. Therefore, for clusters with limited or no Internet access, it is not possible to set up Alertmanager email notifications, which use SMTP, when proxy is used.

Note

Due to a limitation, StackLight fails to integrate with an external proxy with authentication handled by a proxy server. In such cases, the proxy server ignores the HTTP Authorization header for basic authentication passed by Prometheus Alertmanager. Therefore, use proxies without authentication or with authentication handled by a reverse proxy.

Components versions

The following table lists the components versions of the Cluster release 6.12.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Component

Application/Service

Version

Distributed storage

Ceph

14.2.12 (Nautilus)

Rook

1.5.5

Container runtime

Mirantis Container Runtime

19.03.14 1

Cluster orchestration

Mirantis Kubernetes Engine Updated

3.3.6 0

LCM

descheduler

0.8.0

Helm

2.16.11-40

helm-controller Updated

0.2.0-258-ga2d72294

lcm-ansible Updated

0.3.0-10-g7c2a87e

lcm-agent Updated

0.2.0-258-ga2d72294

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

8.0.2-20201014133832

Alertmanager

0.21.0

Cerebro New

0.9.3

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector

0.1-20201222100033

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier

0.3-20201216142028

sf-reporter

0.1-20201216142628

Telegraf

1.9.1-20201222194740

Telemeter

4.4.0-20200424

yq

4.2.0

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1

Artifacts

This section lists the components artifacts of the Cluster release 6.12.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

mirantis.azurecr.io/ceph/helm/ceph-controller-1.0.0-mcp-127.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.12

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20210201202754

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.2.1

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v2.1.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v2.1.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v4.0.0

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v3.1.0

csi-resizer

mirantis.azurecr.io/ceph/k8scsi/csi-resizer:v1.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.5.5


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.3.0-10-g7c2a87e/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-258-ga2d72294/lcm-agent

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-258-ga2d72294

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

cerebro New

https://binary.mirantis.com/stacklight/helm/cerebro-0.1.0-mcp-2.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-33.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-89.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-8.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-19.tgz

prometheus

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-114.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-4.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-11.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-10.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-401.tgz

telegraf-ds

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-20.tgz

telegraf-s

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-20.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-12.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-12.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

cerebro New

mirantis.azurecr.io/stacklight/cerebro:0.9.3

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20201222100033

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201216142028

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20201216142628

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20201222194740

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

yq

mirantis.azurecr.io/stacklight/yq:v4.2.0

5.12.0

This section outlines release notes for the Cluster release 5.12.0 that is introduced in the Mirantis Container Cloud release 2.5.0. This Cluster release supports Kubernetes 1.18 and Mirantis Container Runtime 19.03.14 as well as introduces support for the updated version of Mirantis Kubernetes Engine 3.3.6.

For the list of known and resolved issues, refer to the Container Cloud release 2.5.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.12.0.


Ceph maintenance label

Implemented the maintenance label to set for Ceph during a managed cluster update. This prevents Ceph rebalance leading to data loss during a managed cluster update.


RGW check box in Container Cloud web UI

Implemented the Enable Object Storage checkbox in the Container Cloud web UI to allow enabling a single-instance RGW Object Storage when creating a Ceph cluster as described in Add a Ceph cluster.


Ceph RGW HA

Enhanced Ceph to support RADOS Gateway (RGW) high availability. Now, you can run multiple instances of Ceph RGW in active/active mode.


Cerebro support for StackLight

Enhanced StackLight by adding support for Cerebro, a web UI that visualizes health of Elasticsearch clusters and allows for convenient debugging. Cerebro is disabled by default.


StackLight proxy

Added proxy support for Alertmanager, Metric collector, Salesforce notifier and reporter, and Telemeter client. Now, these StackLight components automatically use the same proxy that is configured for Container Cloud clusters.

Note

Proxy handles only the HTTP and HTTPS traffic. Therefore, for clusters with limited or no Internet access, it is not possible to set up Alertmanager email notifications, which use SMTP, when proxy is used.

Note

Due to a limitation, StackLight fails to integrate with an external proxy with authentication handled by a proxy server. In such cases, the proxy server ignores the HTTP Authorization header for basic authentication passed by Prometheus Alertmanager. Therefore, use proxies without authentication or with authentication handled by a reverse proxy.

Components versions

The following table lists the components versions of the Cluster release 5.12.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Components versions of the Cluster release 5.12.0

Component

Application/Service

Version

Distributed storage

Ceph

14.2.12 (Nautilus)

Rook

1.5.5

Container runtime

Mirantis Container Runtime

19.03.14 1

Cluster orchestration

Mirantis Kubernetes Engine Updated

3.3.6 0

LCM

descheduler

0.8.0

Helm

2.16.11-40

helm-controller Updated

0.2.0-258-ga2d72294

lcm-ansible Updated

0.3.0-10-g7c2a87e

lcm-agent Updated

0.2.0-258-ga2d72294

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

8.0.2-20201014133832

Alertmanager

0.21.0

Cerebro New

0.9.3

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector

0.1-20201222100033

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier

0.3-20201216142028

sf-reporter

0.1-20201216142628

Telegraf

1.9.1-20201222194740

Telemeter

4.4.0-20200424

yq

4.2.0

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1

Artifacts

This section lists the components artifacts of the Cluster release 5.12.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

mirantis.azurecr.io/ceph/helm/ceph-controller-1.0.0-mcp-127.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.12

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20210201202754

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.2.1

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v2.1.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v2.1.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v4.0.0

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v3.1.0

csi-resizer

mirantis.azurecr.io/ceph/k8scsi/csi-resizer:v1.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.5.5


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.3.0-10-g7c2a87e/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-258-ga2d72294/lcm-agent

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-258-ga2d72294

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

cerebro New

https://binary.mirantis.com/stacklight/helm/cerebro-0.1.0-mcp-2.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-33.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-89.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-8.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-19.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-119.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-4.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-11.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-10.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-413.tgz

telegraf-ds

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-20.tgz

telegraf-s

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-20.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-12.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-12.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

cerebro New

mirantis.azurecr.io/stacklight/cerebro:0.9.3

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20201222100033

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201216142028

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20201216142628

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20201222194740

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

yq

mirantis.azurecr.io/stacklight/yq:v4.2.0

Deprecated Cluster releases

This section describes the release notes for the deprecated Cluster releases that will become unsupported in one of the following Container Cloud releases. Make sure to update your managed clusters to the latest supported version as described in Operations Guide: Update a managed cluster.

For details about supported Cluster releases, see Cluster releases (managed).

6.10.0

The Cluster release 6.10.0 is introduced in the Mirantis Container Cloud release 2.3.0 and supports:

  • Mirantis OpenStack for Kubernetes (MOS) Ussuri Update. For details, see MOS Release Notes.

  • Updated versions of Mirantis Kubernetes Engine 3.3.4 and Mirantis Container Runtime 19.03.13. For details, see MKE, MSR, and MCR Release Notes.

  • Kubernetes 1.18.

For the list of known and resolved issues, refer to the Container Cloud release 2.3.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 6.10.0.


Ceph Object Storage support

Enhanced Ceph to support RADOS Gateway (RGW) Object Storage.


Ceph state verification

Implemented the capability to obtain detailed information on the Ceph cluster state, including Ceph logs, Ceph OSDs state, and a list of Ceph pools.

Components versions

The following table lists the components versions of the Cluster release 6.10.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Component

Application/Service

Version

Distributed storage

Ceph

14.2.11 (Nautilus)

Rook

1.4.4

Container runtime

Mirantis Container Runtime Updated

19.03.13 1

Cluster orchestration

Mirantis Kubernetes Engine Updated

3.3.4 0

LCM

descheduler

0.8.0

Helm Updated

2.16.11-40

helm-controller Updated

0.2.0-221-g32bd5f56

lcm-ansible Updated

0.2.0-381-g720ec96

lcm-agent Updated

0.2.0-221-g32bd5f56

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

8.0.2-20201014133832

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector Updated

0.1-20201120155524

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus Updated

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus RabbitMQ Exporter Updated

v1.0.0-RC7.1

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier

0.3-20201001081256

sf-reporter

0.1-20200219140217

Telegraf Updated

1.9.1-20201120081248

Telemeter

4.4.0-20200424

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1
Artifacts

This section lists the components artifacts of the Cluster release 6.10.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

mirantis.azurecr.io/ceph/helm/ceph-controller-1.0.0-mcp-95.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.11

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20201215142221

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.1.0

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v1.2.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v1.6.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v2.1.1

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v2.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.4.4


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-381-g720ec96/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-221-g32bd5f56/lcm-agent

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm Updated

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-221-g32bd5f56

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-33.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-74.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector Updated

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-5.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-17.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-102.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-3.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier Updated

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-9.tgz

sf-reporter Updated

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-8.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-354.tgz

telegraf-ds Updated

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-19.tgz

telegraf-s Updated

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-19.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-11.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-11.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector Updated

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20201120155524

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus Updated

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-rabbitmq-exporter Updated

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v1.0.0-RC7.1

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201001081256

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20200219140217

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf Updated

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20201120081248

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

5.11.0

This section outlines release notes for the Cluster release 5.11.0 that is introduced in the Mirantis Container Cloud release 2.4.0. This Cluster release supports Kubernetes 1.18 and Mirantis Kubernetes Engine 3.3.4 as well as introduces support for the updated version of Mirantis Container Runtime 19.03.14.

For the list of known and resolved issues, refer to the Container Cloud release 2.4.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.11.0.


Alert inhibition rules

Implemented alert inhibition rules to provide a clearer view on the cloud status and simplify troubleshooting. Using alert inhibition rules, Alertmanager decreases alert noise by suppressing dependent alerts notifications. The feature is enabled by default. For details, see Alert dependencies.


Integration between Grafana and Kibana

Implemented integration between Grafana and Kibana by adding a View logs in Kibana link to the majority of Grafana dashboards, which allows you to immediately view contextually relevant logs through the Kibana web UI.


Telegraf alert

Implemented the TelegrafGatherErrors alert that raises if Telegraf fails to gather metrics.

Learn more

Telegraf alerts


Configuration of Ironic Telegraf input plugin

Added the ironic.insecure parameter for enabling or disabling the host and chain verification for bare metal Ironic monitoring.


Automatically defined cluster ID

Enhanced StackLight to automatically set clusterId that defines an ID of a Container Cloud cluster. Now, you do not need to set or modify this parameter manually when configuring the sf-notifier and sf-reporter services.

Components versions

The following table lists the components versions of the Cluster release 5.11.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Components versions of the Cluster release 5.11.0

Component

Application/Service

Version

Distributed storage Updated

Ceph

14.2.12 (Nautilus)

Rook

1.5.5

Container runtime

Mirantis Container Runtime Updated

19.03.14 1

Cluster orchestration

Mirantis Kubernetes Engine

3.3.4 0

LCM

descheduler

0.8.0

Helm

2.16.11-40

helm-controller

0.2.0-221-g32bd5f56

lcm-ansible Updated

0.2.0-394-g599b2a1

lcm-agent

0.2.0-221-g32bd5f56

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

8.0.2-20201014133832

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector Updated

0.1-20201222100033

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus RabbitMQ Exporter

1.0.0-RC7.1

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier Updated

0.3-20201216142028

sf-reporter Updated

0.1-20201216142628

Telegraf Updated

1.9.1-20201222194740

Telemeter

4.4.0-20200424

yq New

4.2.0

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1
Artifacts

This section lists the components artifacts of the Cluster release 5.11.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

mirantis.azurecr.io/ceph/helm/ceph-controller-1.0.0-mcp-111.tgz

Docker images Updated

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.12

ceph-controller

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20210120004212

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.2.1

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v2.1.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v2.1.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v4.0.0

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v3.1.0

csi-resizer New

mirantis.azurecr.io/ceph/k8scsi/csi-resizer:v1.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.5.5


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-394-g599b2a1/lcm-ansible.tar.gz

lcm-agent

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-221-g32bd5f56/lcm-agent

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-221-g32bd5f56

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-33.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-81.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector Updated

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-8.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni Updated

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-19.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-114.tgz

prometheus-blackbox-exporter Updated

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-4.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier Updated

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-11.tgz

sf-reporter Updated

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-10.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-398.tgz

telegraf-ds Updated

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-20.tgz

telegraf-s Updated

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-20.tgz

telemeter-server Updated

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-12.tgz

telemeter-client Updated

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-12.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector Updated

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20201222100033

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-rabbitmq-exporter

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v1.0.0-RC7.1

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier Updated

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201216142028

sf-reporter Updated

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20201216142628

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf Updated

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20201222194740

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

yq New

mirantis.azurecr.io/stacklight/yq:v4.2.0

Unsupported Cluster releases

This section describes the release notes for the unsupported Cluster releases. For details about supported Cluster releases, see Cluster releases (managed).

6.x series

This section outlines release notes for the unsupported Cluster releases of the 6.x series.

6.8.1

The Cluster release 6.8.1 is introduced in the Mirantis Container Cloud release 2.2.0. This Cluster release is based on the Cluster release 5.8.0 and the main difference is support of the Mirantis OpenStack for Kubernetes (MOS) product.

For details about MOS, see MOS Release Notes.

For details about the Cluster release 5.8.0, refer to the 5.8.0 section.

5.x series

This section outlines release notes for the unsupported Cluster releases of the 5.x series.

5.10.0

This section outlines release notes for the Cluster release 5.10.0 that is introduced in the Mirantis Container Cloud release 2.3.0. This Cluster release supports Kubernetes 1.18 and introduces support for the latest versions of Mirantis Kubernetes Engine 3.3.4 and Mirantis Container Runtime 19.03.13.

For the list of known and resolved issues, refer to the Container Cloud release 2.3.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.10.0.


Ceph Object Storage support

Enhanced Ceph to support RADOS Gateway (RGW) Object Storage.


Ceph state verification

Implemented the capability to obtain detailed information on the Ceph cluster state, including Ceph logs, Ceph OSDs state, and a list of Ceph pools.

Components versions

The following table lists the components versions of the Cluster release 5.10.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Component

Application/Service

Version

Distributed storage

Ceph

14.2.11 (Nautilus)

Rook

1.4.4

Container runtime

Mirantis Container Runtime Updated

19.03.13 1

Cluster orchestration

Mirantis Kubernetes Engine Updated

3.3.4 0

LCM

descheduler

0.8.0

Helm Updated

2.16.11-40

helm-controller Updated

0.2.0-221-g32bd5f56

lcm-ansible Updated

0.2.0-381-g720ec96

lcm-agent Updated

0.2.0-221-g32bd5f56

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

8.0.2-20201014133832

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector Updated

0.1-20201120155524

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus Updated

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus RabbitMQ Exporter Updated

v1.0.0-RC7.1

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier

0.3-20201001081256

sf-reporter

0.1-20200219140217

Telegraf Updated

1.9.1-20201120081248

Telemeter

4.4.0-20200424

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1
Artifacts

This section lists the components artifacts of the Cluster release 5.10.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

mirantis.azurecr.io/ceph/helm/ceph-controller-1.0.0-mcp-95.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.11

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20201215142221

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.1.0

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v1.2.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v1.6.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v2.1.1

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v2.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.4.4


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-381-g720ec96/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-221-g32bd5f56/lcm-agent

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm Updated

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-221-g32bd5f56

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-33.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-74.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector Updated

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-5.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-17.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-102.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-3.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier Updated

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-9.tgz

sf-reporter Updated

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-8.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-354.tgz

telegraf-ds Updated

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-19.tgz

telegraf-s Updated

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-19.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-11.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-11.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector Updated

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20201120155524

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus Updated

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-rabbitmq-exporter Updated

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v1.0.0-RC7.1

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201001081256

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20200219140217

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf Updated

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20201120081248

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

5.9.0

This section outlines release notes for the Cluster release 5.9.0 that is introduced in the Mirantis Container Cloud release 2.2.0 and supports Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18.

For the list of known and resolved issues, refer to the Container Cloud release 2.2.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.9.0.


Alerta upgrade

Upgraded Alerta from version 7.4.4 to 8.0.2.


File descriptors monitoring

Enhanced StackLight to monitor the number of file descriptors on nodes and raise FileDescriptorUsage* alerts when a node uses 80%, 90%, or 95% of file descriptors.

Learn more

General node alerts


Alerts improvements
  • Added the SSLProbesFailing alert that raises in case of an SSL certificate probes failure.

  • Improved alerts descriptions and raise conditions.

Components versions

The following table lists the components versions of the Cluster release 5.9.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Components versions of the Cluster release 5.9.0

Component

Application/Service

Version

Distributed storage

Ceph Updated

14.2.11 (Nautilus)

Rook Updated

1.4.4

Container runtime

Mirantis Container Runtime

19.03.12 0

Cluster orchestration

Mirantis Kubernetes Engine

3.3.3 1

LCM

ansible-docker Updated

0.3.5-147-g18f3b44

descheduler

0.8.0

Helm

2.16.9-39

helm-controller Updated

0.2.0-178-g8cc488f8

lcm-ansible Updated

0.2.0-132-g49f7591

lcm-agent Updated

0.2.0-178-g8cc488f8

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta Updated

8.0.2-20201014133832

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

MCC Metric Collector

0.1-20201005141816

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus Updated

2.19.3

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus RabbitMQ Exporter

0.29.0

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier Updated

0.3-20201001081256

sf-reporter

0.1-20200219140217

telegraf-ds

1.9.1-20200901112858

telegraf-s

1.9.1-20200901112858

Telemeter

4.4.0-20200424

0

For the Mirantis Container Runtime release highlights, see MKE, MSR, and MCR documentation: Mirantis Container Runtime release notes.

1

For the Mirantis Kubernetes Engine 3.3.3 release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

Artifacts

This section lists the components artifacts of the Cluster release 5.9.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

mirantis.azurecr.io/ceph/helm/ceph-controller-1.0.0-mcp-33.tgz

Docker images

ceph Updated

mirantis.azurecr.io/ceph/ceph:v14.2.11

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20201022081323

cephcsi Updated

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.1.0

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v1.2.0

csi-provisioner Updated

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v1.6.0

csi-snapshotter Updated

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v2.1.1

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v2.1.0

rook Updated

mirantis.azurecr.io/ceph/rook/ceph:v1.4.4


LCM artifacts

Artifact

Component

Path

Binaries

ansible-docker Updated

https://binary.mirantis.com/lcm/bin/ansible-docker/v0.3.5-147-g18f3b44/ansible-docker.tar.gz

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-132-g49f7591-1/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-178-g8cc488f8/lcm-agent

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.9-39

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-178-g8cc488f8

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta Updated

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-20.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-28.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-66.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

mcc-metric-collector

https://binary.mirantis.com/stacklight/helm/mcc-metric-collector-0.1.0-mcp-22.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-17.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-83.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-3.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-5.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-6.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-325.tgz

telegraf-ds

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-16.tgz

telegraf-s

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-16.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-11.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-11.tgz

Docker images

alerta Updated

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests Updated

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl Updated

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

mcc-metric-collector

mirantis.azurecr.io/stacklight/mcc-metric-collector:v0.1-20201005141816

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus Updated

mirantis.azurecr.io/stacklight/prometheus:v2.19.3

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-rabbitmq-exporter

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v0.29.0

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier Updated

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201001081256

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20200219140217

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20200901112858

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

5.8.0

This section outlines release notes for the Cluster release 5.8.0 that is introduced in the Mirantis Container Cloud release 2.1.0 and supports Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18.

For the list of known issues, refer to the Container Cloud release 2.1.0 Known issues.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.8.0.


Grafana improvements
  • Upgraded Grafana from version 6.6.2 to 7.1.5.

  • Introduced Grafana Image Renderer, a separate Grafana container in a pod to offload rendering of images from charts. Grafana Image Renderer is enabled by default.

  • Configured a home dashboard to replace the Installation/configuration panel that opens when you access Grafana. By default, Kubernetes Cluster is set as a home dashboard. However, you can set any of the available Grafana dashboards.


Clusters telemetry improvement in StackLight
  • Split the regional and management cluster function in StackLight telemetry. Now, the metrics from managed clusters are aggregated on regional clusters, then both regional and managed clusters metrics are sent from regional clusters to the management cluster.

  • Added the capability to filter panels by regions in the Clusters Overview and Telemeter Server Grafana dashboards.


Alerts improvements
  • Improved alerts descriptions and raise conditions.

  • Changed severity in some alerts to improve operability.

  • Improved raise conditions of some alerts by adding the for clause and unifying the existing for clauses.

Components versions

The following table lists the components versions of the Cluster release 5.8.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Components versions of the Cluster release 5.8.0

Component

Application/Service

Version

Distributed storage

Ceph

14.2.9 (Nautilus)

Rook

1.3.8

Container runtime

Mirantis Container Runtime

19.03.12 0

Cluster orchestration

Mirantis Kubernetes Engine

3.3.3 1

LCM

ansible-docker Updated

0.3.5-141-g1007cc9

descheduler

0.8.0

Helm Updated

2.16.9-39

helm-controller Updated

0.2.0-169-g5668304d

lcm-ansible Updated

0.2.0-119-g8f05f58-1

lcm-agent

0.2.0-149-g412c5a05

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

7.4.4-20200615123606

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana Updated

7.1.5

Grafana Image Renderer New

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

MCC Metric Collector Updated

0.1-20201005141816

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.19.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter Updated

0.5.1-20201002144823

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter Updated

0.8.0-20201006113956

Prometheus RabbitMQ Exporter

0.29.0

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier Updated

0.3-20200813125431

sf-reporter

0.1-20200219140217

telegraf-ds Updated

1.9.1-20200901112858

telegraf-s Updated

1.9.1-20200901112858

Telemeter

4.4.0-20200424

0

For the Mirantis Container Runtime release highlights, see MKE, MSR, and MCR documentation: Mirantis Container Runtime release notes.

1

For the Mirantis Kubernetes Engine 3.3.3 release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

Artifacts

This section lists the components artifacts of the Cluster release 5.8.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

mirantis.azurecr.io/ceph/helm/ceph-controller-1.0.0-mcp-18.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.9

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20200903151423

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v2.1.2

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v1.2.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v1.4.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v1.2.2

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v2.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.3.8


LCM artifacts

Artifact

Component

Path

Binaries

ansible-docker Updated

https://binary.mirantis.com/lcm/bin/ansible-docker/v0.3.5-141-g1007cc9/ansible-docker.tar.gz

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-119-g8f05f58-1/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-169-g5668304d/lcm-agent

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm Updated

mirantis.azurecr.io/lcm/helm/tiller:v2.16.9-39

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-169-g5668304d

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-12.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-20.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd Updated

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-25.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-64.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana Updated

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

mcc-metric-collector

https://binary.mirantis.com/stacklight/helm/mcc-metric-collector-0.1.0-mcp-22.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-17.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-80.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-3.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter Updated

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-5.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-6.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-312.tgz

telegraf-ds Updated

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-16.tgz

telegraf-s Updated

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-16.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-11.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-11.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:7.4.4-20200615123606

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200320

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana Updated

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer New

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl New

mirantis.azurecr.io/stacklight/kubectl:1.15.3

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

mcc-metric-collector Updated

mirantis.azurecr.io/stacklight/mcc-metric-collector:v0.1-20201005141816

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.19.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter Updated

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter Updated

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-rabbitmq-exporter

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v0.29.0

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier Updated

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20200813125431

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20200219140217

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf Updated

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20200901112858

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

5.7.0

This section outlines release notes for the Cluster release 5.7.0 that is introduced in the Mirantis Container Cloud release 2.0.0 and supports Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18.

For the list of known issues, refer to the Container Cloud release 2.0.0 Known issues.

Components versions

The following table lists the components versions of the Cluster release 5.7.0.

Components versions of the Cluster release 5.7.0

Component

Application/Service

Version

Distributed storage

Ceph

14.2.9 (Nautilus)

Rook

1.3.8

Container runtime

Mirantis Container Runtime

19.03.12 0

Cluster orchestration

Mirantis Kubernetes Engine

3.3.3 1

LCM

ansible-docker

0.3.5-136-g38653c7

descheduler

0.8.0

Helm

2.16.7-38

helm-controller

0.2.0-149-g412c5a05

lcm-ansible

0.2.0-110-g63cf88b

lcm-agent

0.2.0-149-g412c5a05

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

7.4.4-20200615123606

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

6.6.2

IAM Proxy

6.0.1

Kibana

7.6.1

MCC Metric Collector

0.1-20200806113043

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.19.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20200313132957

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20200715102834

Prometheus RabbitMQ Exporter

0.29.0

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier

0.3-20200430122138

sf-reporter

0.1-20200219140217

telegraf-ds

1.9.1-20200806073506

telegraf-s

1.9.1-20200806073506

Telemeter

4.4.0-20200424

0

For the Mirantis Container Runtime release highlights, see MKE, MSR, and MCR documentation: Mirantis Container Runtime release notes.

1

For the Mirantis Kubernetes Engine 3.3.3 release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

Artifacts

This section lists the components artifacts of the Cluster release 5.7.0.


Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller

mirantis.azurecr.io/ceph/helm/ceph-controller-1.0.0-mcp-16.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.9

ceph-controller

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20200805103414

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v2.1.2

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v1.2.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v1.4.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v1.2.2

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v2.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.3.8


LCM artifacts

Artifact

Component

Path

Binaries

ansible-docker

https://binary.mirantis.com/lcm/bin/ansible-docker/v0.3.5-136-g38653c7/ansible-docker.tar.gz

lcm-ansible

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-110-g63cf88b/lcm-ansible.tar.gz

lcm-agent

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-149-g412c5a05/lcm-agent

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.9-39

helm-controller

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-149-g412c5a05

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-12.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-20.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-24.tgz

grafana

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-59.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-19.tgz

mcc-metric-collector

https://binary.mirantis.com/stacklight/helm/mcc-metric-collector-0.1.0-mcp-22.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-17.tgz

prometheus

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-73.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-3.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-2.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-5.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-6.tgz

stacklight

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-285.tgz

telegraf-ds

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-14.tgz

telegraf-s

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-14.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-11.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-11.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:7.4.4-20200615123606

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200320

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:6.6.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

mcc-metric-collector

mirantis.azurecr.io/stacklight/mcc-metric-collector:v0.1-20200806113043

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.19.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20200313132957

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20200715102834

prometheus-rabbitmq-exporter

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v0.29.0

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20200430122138

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20200219140217

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20200806073506

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600