Mirantis Container Cloud Release Notes latest documentation

Mirantis Container Cloud Release Notes

Releases summary

Container Cloud release

Release date

Supported Cluster releases

Summary

2.7.0 (current)

April 22, 2021

  • Full support for the VMWare vSphere provider

  • Universal SSH user

  • Configuration of SSH keys on existing clusters using web UI

  • Cluster and machines live statuses in web UI

  • Enabling of proxy access using web UI for vSphere, AWS, and bare metal

  • Log collection optimization in StackLight

  • Ceph enhancements:

    • Dedicated network for the Ceph distributed storage traffic

    • Ceph Multisite configuration

  • Documentation enhancements:

    • Ceph disaster recovery procedure

    • QuickStart guides

2.6.0

March 24, 2021

  • RHEL license activation using the activation key

  • Support for VMware vSphere Distributed Switch

  • VMWare vSphere provider integration with IPAM controller

  • Proxy support for all Container Cloud providers

  • StackLight logging levels

  • StackLight remote logging to syslog

  • Hyperconverged Ceph

  • Ceph objectStorage section in KaasCephCluster

  • Ceph maintenance orchestration

  • Updated documentation on the bare metal networking

2.5.0

March 1, 2021

  • Support for Mirantis Kubernetes Engine 3.3.6

  • Support for Mirantis OpenStack for Kubernetes 21.1

  • Proxy support for OpenStack and VMWare vSphere providers

  • NTP server configuration on regional clusters

  • Optimized ClusterRelease upgrade process

  • Dedicated network for external connection to the Kubernetes services on bare metal

  • Ceph RADOS Gateway HA

  • Ceph RADOS Gateway check box in Container Cloud web UI

  • Ceph maintenance label

  • Cerebro support for StackLight

  • Proxy support for StackLight

2.4.0

February 2, 2021

  • Support for the updated version of Mirantis Container Runtime 19.03.14

  • Dedicated network for Kubernetes pods traffic on bare metal clusters

  • Improvements for the feedback form in the Container Cloud web UI

  • StackLight enhancements:

    • Alert inhibition rules

    • Integration between Grafana and Kibana

    • New Telegraf alert TelegrafGatherErrors

    • Configuration of Ironic Telegraf input plugin

    • Automatically defined cluster ID

* - the Cluster release is deprecated and will become unsupported in one of the following Container Cloud releases. For details about unsupported releases, see Container Cloud releases history.

** - the Cluster release supports only attachment of existing MKE 3.3.4 clusters. For the deployment of new or attachment of existing MKE 3.3.6 clusters, the latest available Cluster release is used.

Product highlights

Mirantis Container Cloud enables you to create and manage Container Cloud clusters across multiple cloud and infrastructure platforms. Currently supported architectures include OpenStack, bare metal, AWS, and VMWare vSphere Technology Preview with more choice coming in the near future.

Container Cloud provides a web UI, CLI, and API to enable a self-service deployment of Container Cloud with Mirantis Kubernetes Engine (MKE) container clusters.

The major features of the Container Cloud GA release include:

  • Container Cloud with MKE multi-cluster
    • Create new fully managed Container Cloud with MKE container clusters on the Container Cloud supported infrastructure and operating system

    • Attach existing MKE clusters that run on any infrastructure and operating system with a core set of most critical LCM operations

  • Automated software LCM

    Provide automated zero-touch update for Container Cloud and MKE, including Ubuntu operating system updates, with no workload impact

  • Multi-cloud infrastructure support

    Deploy Container Cloud with MKE container clusters through automated provisioning on the following infrastructure platforms: OpenStack, VMWare vSphere Technology Preview, AWS, and bare metal

  • Logging, monitoring, alerting

    Monitor individual managed Container Cloud with MKE container clusters as well as provide cluster metrics and alerts on resource utilization aggregated to the Container Cloud level

  • Bare Metal as a Service

    Provision pools of bare metal machines to use as a bare metal provider, which reduces layers of virtualization and provides efficient support for compute-intensive workloads

  • Ceph storage for bare metal

    Deploy Ceph utilizing Rook to provide and manage a robust and scalable persistent storage that can be used by Kubernetes workloads

  • User management

    Integrate IdP and user RBAC to assimilate with corporate single source of truth and assign role-based access

  • Intuitive web UIs for cluster management

    Provide a single UI for multiple cluster management as well as the Container Cloud with MKE container clusters UI for single cluster management

Mirantis Container Cloud releases

This section outlines the release notes for the Mirantis Container Cloud GA release. Within the scope of the Container Cloud GA release, minor releases are being published continuously with new features, improvements, and critical issues resolutions to enhance the Container Cloud GA version. For details, see Mirantis Container Cloud releases and Cluster releases (managed).

Once a new Container Cloud release is available, a management cluster automatically upgrades to a newer consecutive release unless this cluster contains managed clusters with a Cluster release unsupported by that Container Cloud release. For more details about the Container Cloud release mechanism, see Reference Architecture: Release controller.

2.7.0 (current)

The Mirantis Container Cloud GA release 2.7.0:

  • Introduces support for the Cluster release 5.14.0 that is based on Kubernetes 1.18, Mirantis Container Runtime 19.03.14, and Mirantis Kubernetes Engine 3.3.6.

  • Supports the Cluster release 6.14.0 that is based on the Cluster release 5.14.0 and represents Mirantis OpenStack for Kubernetes (MOS) 21.2.

  • Supports deprecated Cluster releases 5.13.0 and 6.12.0 that will become unsupported in one of the following Container Cloud releases.

  • Supports the Cluster release 5.11.0 only for attachment of existing MKE 3.3.4 clusters. For the deployment of new or attachment of existing MKE 3.3.6 clusters, the latest available Cluster release is used.

    Caution

    Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.7.0.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.7.0. For the list of enhancements in the Cluster release 5.14.0 and Cluster release 6.14.0 that are supported by the Container Cloud release 2.7.0, see the 5.14.0 and 6.14.0 sections.


Full support for the VMWare vSphere provider

Introduced general availability support for the VMWare vSphere provider after completing full integration of the vSphere provider on RHEL with Container Cloud.

During the Container Cloud 2.6.0 - 2.7.0 release cycle, added the following improvements:

  • Removed the StackLight limitations

  • Completed the integration of proxy support for the vSphere-based managed clusters

  • Completed the integration of the non-DHCP support for regional clusters

  • Addressed a number of critical and major issues


Universal SSH user

Implemented a universal SSH user mcc-user to replace the existing default SSH user names. The mcc-user user name is applicable to any Container Cloud provider and node type, including Bastion.

The existing SSH user names are deprecated as of Container Cloud 2.7.0. SSH keys will be managed only for mcc-user as of one of the following Container Cloud releases.


Configuration of SSH keys on existing clusters using web UI

Implemented the possibility to configure SSH keys on existing clusters using the Container Cloud web UI. You can now add or remove SSH keys on running managed clusters using the Configure cluster web UI menu.

After the update of your Cluster release to the latest version supported by 2.7.0 for the OpenStack and AWS-based managed clusters, a one-time redeployment of the Bastion node is required to apply the first configuration change of SSH keys. For this purpose, the Allow Bastion Redeploy one-time check box is added to the Configure Cluster wizard in the Container Cloud web UI.

Note

After the Bastion node redeploys on the AWS-based clusters, its public IP address changes.


Cluster and machines live statuses in web UI

Implemented the possibility to monitor live status of a cluster and machine deployment or update using the Container Cloud web UI. You can now follow the deployment readiness and health of essential cluster components, such as Helm, Kubernetes, kubelet, Swarm, OIDC, StackLight, and others. For machines, you can monitor nodes readiness reported by kubelet and nodes health reported by Swarm.


Enabling of proxy access using web UI for vSphere, AWS, and bare metal

Extended the Container Cloud web UI with the parameters that enable proxy access on managed clusters for the remaining cloud providers: vSphere, AWS, and bare metal.


QuickStart guides

Created a separate QuickStart guides section in the Container Cloud documentation with a set of QuickStart guides that contain only essential lightweight instructions with no additional options to quickly get started with Container Cloud on the AWS, OpenStack, or vSphere providers.

Addressed issues

The following issues have been addressed in the Mirantis Container Cloud release 2.7.0 along with the Cluster releases 5.14.0 and 6.14.0:

  • [13176] [vSphere] Fixed the issue with the cluster network settings related to IPAM disappearing from the cluster provider spec and leading to invalid metadata provided to virtual machines.

  • [12683] [vSphere] Fixed the issue with the kaas-ipam pods being installed and continuously restarted even if IPAM was disabled on the vSphere-based regional cluster deployed on top of an AWS-based management cluster.


  • [12305] [Ceph] Fixed the issue with inability to define the CRUSH map rules through the KaaSCephCluster custom resource. For details, see Operations Guide: Ceph advanced configuration.

  • [10060] [Ceph] Fixed the issue with a Ceph OSD node removal not being triggered properly and failing after updating the KaasCephCluster custom resource (CR).


  • [13078] [StackLight] Fixed the issue with Elasticsearch not receiving data from Fluentd due to the limit of open index shards per node.

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.7.0 including the Cluster release 5.14.0 and 6.14.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[8013] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

OpenStack
[10424] OpenStack-based regional cluster cleanup fails by timeout

An OpenStack-based regional cluster cleanup fails with the timeout error.

Workaround:

  1. Wait for the Cluster object to be deleted in the bootstrap cluster:

    kubectl --kubeconfig <(./bin/kind get kubeconfig --name clusterapi) get cluster
    

    The system output must be empty.

  2. Remove the bootstrap cluster manually:

    ./bin/kind delete cluster --name clusterapi
    

Bare metal
[7655] Wrong status for an incorrectly configured L2 template

If an L2 template is configured incorrectly, a bare metal cluster is deployed successfully but with the runtime errors in the IpamHost object.

Workaround:

If you suspect that the machine is not working properly because of incorrect network configuration, verify the status of the corresponding IpamHost object. Inspect the l2RenderResult and ipAllocationResult object fields for error messages.


Storage
[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.

[10050] Ceph OSD pod is in the CrashLoopBackOff state after disk replacement

If you use a custom BareMetalHostProfile, after disk replacement on a Ceph OSD, the Ceph OSD pod switches to the CrashLoopBackOff state due to the Ceph OSD authorization key failing to be created properly.

Workaround:

  1. Export kubeconfig of your managed cluster. For example:

    export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
    
  2. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  3. Delete the authorization key for the failed Ceph OSD:

    ceph auth del osd.<ID>
    
  4. SSH to the node on which the Ceph OSD cannot be created.

  5. Clean up the disk that will be a base for the failed Ceph OSD. For details, see official Rook documentation.

    Note

    Ignore failures of the sgdisk --zap-all $DISK and blkdiscard $DISK commands if any.

  6. On the managed cluster, restart the Rook operator:

    kubectl -n rook-ceph delete pod -l app=rook-ceph-operator
    
[12723] ceph_role_* labels remain after deleting a node from KaaSCephCluster

The ceph_role_mon and ceph_role_mgr labels that Ceph controller assigns to a node during a Ceph cluster creation are not automatically removed after deleting a node from KaaSCephCluster.

As a workaround, manually remove the labels using the following commands:

kubectl unlabel node <nodeName> ceph_role_mon
kubectl unlabel node <nodeName> ceph_role_mgr

IAM
[13385] MariaDB pods fail to start after SST sync

The MariaDB pods fail to start after MariaDB blocks itself during the State Snapshot Transfers sync.

Workaround:

  1. Verify the failed pod readiness:

    kubectl describe pod -n kaas <failedMariadbPodName>
    

    If the readiness probe failed with the WSREP not synced message, proceed to the next step. Otherwise, assess the MariaDB pod logs to identify the failure root cause.

  2. Obtain the MariaDB admin password:

    kubectl get secret -n kaas mariadb-dbadmin-password -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d ; echo
    
  3. Verify that wsrep_local_state_comment is Donor or Desynced:

    kubectl exec -it -n kaas <failedMariadbPodName> -- mysql -uroot -p<mariadbAdminPassword> -e "SHOW status LIKE \"wsrep_local_state_comment\";"
    
  4. Restart the failed pod:

    kubectl delete pod -n kaas <failedMariadbPodName>
    

LCM
[13845] Cluster update fails during the LCM agent upgrade with x509 error

During update of a managed cluster from the Cluster releases 5.13.0 to 5.14.0 and 6.12.0 to 6.14.0, the LCM agent upgrade fails with the following error in logs:

lcmAgentUpgradeStatus:
    error: 'failed to download agent binary: Get https://<mcc-cache-address>/bin/lcm/bin/lcm-agent/v0.2.0-289-gd7e9fa9c/lcm-agent:
      x509: certificate signed by unknown authority'

Only clusters initially deployed using Container Cloud 2.4.0 or earlier are affected.

As a workaround, restart lcm-agent using the service lcm-agent-* restart command on the affected nodes.

[13381] Management and regional clusters with enabled proxy are unreachable

After bootstrap, requests to apiserver fail on the management and regional clusters with enabled proxy.

As a workaround, before running bootstrap.sh, add the entire range of IP addresses that will be used for floating IPs to the NO_PROXY environment variable.

[13402] Cluster fails with error: no space left on device

If an application running on a Container Cloud management or managed cluster fails frequently, for example, PostgreSQL, it may produce an excessive amount of core dumps. This leads to the no space left on device error on the cluster nodes and, as a result, to the broken Docker Swarm and the entire cluster.

Core dumps are disabled by default on the operating system of the Container Cloud nodes. But since Docker does not inherit the operating system settings, disable core dumps in Docker using the workaround below.

Warning

The workaround below does not apply to the baremetal-based clusters, including MOS deployments, since Docker restart may destroy the Ceph cluster.

Workaround:

  1. SSH to any machine of the affected cluster using mcc-user and the SSH key provided during the cluster creation.

  2. In /etc/docker/daemon.json, add the following parameters:

    {
        ...
        "default-ulimits": {
            "core": {
                "Hard": 0,
                "Name": "core",
                "Soft": 0
            }
        }
    }
    
  3. Restart the Docker daemon:

    systemctl restart docker
    
  4. Repeat the steps above on each machine of the affected cluster one by one.

[8112] Nodes occasionally become Not Ready on long-running clusters

On long-running Container Cloud clusters, one or more nodes may occasionally become Not Ready with different errors in the ucp-kubelet containers of failed nodes.

Workaround:

  1. Verify that ucp-kubelet has the use of closed network connection error in logs:

    docker logs ucp-kubelet --since 15m 2>&1 | grep 'use of closed network connection'
    

    If the error exists, proceed to the next step. If the error does not exist, further assess the node logs to identify the root cause of failure.

  2. Restart ucp-kubelet on the failed node:

    ctr -n com.docker.ucp snapshot rm ucp-kubelet
    docker rm -f ucp-kubelet
    
[10029] Authentication fails with the 401 Unauthorized error

Authentication may not work on some controller nodes after a managed cluster creation. As a result, the Kubernetes API operations with the managed cluster kubeconfig fail with Response Status: 401 Unauthorized.

As a workaround, manually restart the ucp-controller and ucp-auth Docker services on the affected node.

[6066] Helm releases get stuck in FAILED or UNKNOWN state

During a management, regional, or managed cluster deployment, Helm releases may get stuck in the FAILED or UNKNOWN state although the corresponding machines statuses are Ready in the Container Cloud web UI. For example, if the StackLight Helm release fails, the links to its endpoints are grayed out in the web UI. In the cluster status, providerStatus.helm.ready and providerStatus.helm.releaseStatuses.<releaseName>.success are false.

HelmBundle cannot recover from such states and requires manual actions. The workaround below describes the recovery steps for the stacklight release that got stuck during a cluster deployment. Use this procedure as an example for other Helm releases as required.

Workaround:

  1. Verify the failed release has the UNKNOWN or FAILED status in the HelmBundle object:

    kubectl --kubeconfig <regionalClusterKubeconfigPath> get helmbundle <clusterName> -n <clusterProjectName> -o=jsonpath={.status.releaseStatuses.stacklight}
    
    In the command above and in the steps below, replace the parameters
    enclosed in angle brackets with the corresponding values of your cluster.
    

    Example of system response:

    stacklight:
    attempt: 2
    chart: ""
    finishedAt: "2021-02-05T09:41:05Z"
    hash: e314df5061bd238ac5f060effdb55e5b47948a99460c02c2211ba7cb9aadd623
    message: '[{"occurrence":1,"lastOccurrenceDate":"2021-02-05 09:41:05","content":"error
      updating the release: rpc error: code = Unknown desc = customresourcedefinitions.apiextensions.k8s.io
      \"helmbundles.lcm.mirantis.com\" already exists"}]'
    notes: ""
    status: UNKNOWN
    success: false
    version: 0.1.2-mcp-398
    
  2. Log in to the helm-controller pod console:

    kubectl --kubeconfig <affectedClusterKubeconfigPath> exec -n kube-system -it helm-controller-0 sh -c tiller
    
  3. Remove the failed release. For example:

    ./helm --host=localhost:44134 delete stacklight
    

    Once done, the release triggers for redeployment.


Upgrade
[9899] Helm releases get stuck in PENDING_UPGRADE during cluster update

Helm releases may get stuck in the PENDING_UPGRADE status during a management or managed cluster upgrade. The HelmBundle controller cannot recover from this state and requires manual actions. The workaround below describes the recovery process for the openstack-operator release that stuck during a managed cluster update. Use it as an example for other Helm releases as required.

Workaround:

  1. Log in to the helm-controller pod console:

    kubectl exec -n kube-system -it helm-controller-0 sh -c tiller
    
  2. Identify the release that stuck in the PENDING_UPGRADE status. For example:

    ./helm --host=localhost:44134 history openstack-operator
    

    Example of system response:

    REVISION  UPDATED                   STATUS           CHART                      DESCRIPTION
    1         Tue Dec 15 12:30:41 2020  SUPERSEDED       openstack-operator-0.3.9   Install complete
    2         Tue Dec 15 12:32:05 2020  SUPERSEDED       openstack-operator-0.3.9   Upgrade complete
    3         Tue Dec 15 16:24:47 2020  PENDING_UPGRADE  openstack-operator-0.3.18  Preparing upgrade
    
  3. Roll back the failed release to the previous revision. For example:

    ./helm --host=localhost:44134 rollback openstack-operator 2
    

    Once done, the release will be reconciled.


Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.7.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.19.10

aws-credentials-controller

1.19.10

Bare metal

baremetal-operator Updated

4.0.7

baremetal-public-api Updated

4.0.7

baremetal-provider Updated

1.19.10

httpd

1.18.0

ironic

victoria-bionic-20210302180018

ironic-operator Updated

base-bionic-20210326130922

kaas-ipam Updated

base-bionic-20210329201651

local-volume-provisioner Updated

1.0.5-mcp

mariadb

10.4.17-bionic-20210203155435

IAM

iam Updated

2.2.0

iam-controller Updated

1.19.10

keycloak

9.0.0

Container Cloud

admission-controller Updated

1.19.10

byo-credentials-controller Updated

1.19.10

byo-provider Updated

1.19.10

kaas-public-api Updated

1.19.10

kaas-exporter Updated

1.19.10

kaas-ui Updated

1.19.10

lcm-controller Updated

0.2.0-299-g32c0398a

mcc-cache Updated

1.19.10

proxy-controller Updated

1.19.10

release-controller Updated

1.19.10

rhellicense-controller Updated

1.19.10

squid-proxy

0.0.1-1

OpenStack Updated

openstack-provider

1.19.10

os-credentials-controller

1.19.10

VMWare vSphere Updated

vsphere-provider

1.19.10

vsphere-credentials-controller

1.19.10

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.7.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-4.0.7.tgz

baremetal-public-api Updated

https://binary.mirantis.com/bm/helm/baremetal-public-api-4.0.7.tgz

ironic-python-agent-bionic.kernel

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-victoria-bionic-debug-20210226182519

ironic-python-agent-bionic.initramfs

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-victoria-bionic-debug-20210226182519

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-4.0.7.tgz

local-volume-provisioner Updated

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.5-mcp.tgz

Docker images

baremetal-operator Updated

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20210317164614

httpd

mirantis.azurecr.io/lcm/nginx:1.18.0

ironic

mirantis.azurecr.io/openstack/ironic:victoria-bionic-20210302180018

ironic-inspector

mirantis.azurecr.io/openstack/ironic-inspector:victoria-bionic-20210302180018

ironic-operator

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20210301104323

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20210329201651

mariadb

mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20210203155435


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.19.10.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.19.10.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.19.10.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.19.10.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.19.10.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.19.10.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.19.10.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.19.10.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.19.10.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.19.10.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.19.10.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.19.10.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.19.10.tgz

mcc-cache

https://binary.mirantis.com/core/helm/mcc-cache-1.19.10.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.19.10.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.19.10.tgz

proxy-controller

https://binary.mirantis.com/core/helm/proxy-controller-1.19.10.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.19.10.tgz

rhellicense-controller Updated

https://binary.mirantis.com/core/helm/rhellicense-controller-1.19.10.tgz

squid-proxy

https://binary.mirantis.com/core/helm/squid-proxy-1.19.10.tgz

vsphere-credentials-controller

https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.19.10.tgz

vsphere-provider

https://binary.mirantis.com/core/helm/vsphere-provider-1.19.10.tgz

Docker images

admission-controller Updated

mirantis.azurecr.io/core/admission-controller:1.19.10

aws-cluster-api-controller Updated

mirantis.azurecr.io/core/aws-cluster-api-controller:1.19.10

aws-credentials-controller Updated

mirantis.azurecr.io/core/aws-credentials-controller:1.19.10

byo-cluster-api-controller Updated

mirantis.azurecr.io/core/byo-cluster-api-controller:1.19.10

byo-credentials-controller Updated

mirantis.azurecr.io/core/byo-credentials-controller:1.19.10

cluster-api-provider-baremetal Updated

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.19.10

frontend Updated

mirantis.azurecr.io/core/frontend:1.19.10

iam-controller Updated

mirantis.azurecr.io/core/iam-controller:1.19.10

kproxy Updated

mirantis.azurecr.io/lcm/kproxy:1.19.10

lcm-controller Updated

mirantis.azurecr.io/core/lcm-controller:v0.2.0-299-g32c0398a

nginx

mirantis.azurecr.io/lcm/nginx:1.18.0

openstack-cluster-api-controller Updated

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.19.10

os-credentials-controller Updated

mirantis.azurecr.io/core/os-credentials-controller:1.19.10

registry

mirantis.azurecr.io/lcm/registry:2.7.1

release-controller Updated

mirantis.azurecr.io/core/release-controller:1.19.10

rhellicense-controller Updated

mirantis.azurecr.io/core/rhellicense-controller:1.19.10

squid-proxy

mirantis.azurecr.io/core/squid-proxy:0.0.1-1

vsphere-cluster-api-controller Updated

mirantis.azurecr.io/core/vsphere-api-controller:1.19.10

vsphere-credentials-controller Updated

mirantis.azurecr.io/core/vsphere-credentials-controller:1.19.10


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin

iamctl-linux

http://binary.mirantis.com/iam/bin/iamctl-0.4.0-linux

iamctl-darwin

http://binary.mirantis.com/iam/bin/iamctl-0.4.0-darwin

iamctl-windows

http://binary.mirantis.com/iam/bin/iamctl-0.4.0-windows

Helm charts Updated

iam

http://binary.mirantis.com/iam/helm/iam-2.2.0tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

keycloak-proxy Updated

http://binary.mirantis.com/core/helm/keycloak_proxy-1.19.10.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.4.0

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.4.0

kubernetes-entrypoint

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052

keycloak

mirantis.azurecr.io/iam/keycloak:0.4.0

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

Unsupported releases

Unsupported Container Cloud releases history

Version

Release date

Supported Cluster releases

Summary

2.6.0

March 24, 2021

  • RHEL license activation using the activation key

  • Support for VMware vSphere Distributed Switch

  • VMWare vSphere provider integration with IPAM controller

  • Proxy support for all Container Cloud providers

  • StackLight logging levels

  • StackLight remote logging to syslog

  • Hyperconverged Ceph

  • Ceph objectStorage section in KaasCephCluster

  • Ceph maintenance orchestration

  • Updated documentation on the bare metal networking

2.5.0

March 1, 2021

  • Support for Mirantis Kubernetes Engine 3.3.6

  • Support for Mirantis OpenStack for Kubernetes 21.1

  • Proxy support for OpenStack and VMWare vSphere providers

  • NTP server configuration on regional clusters

  • Optimized ClusterRelease upgrade process

  • Dedicated network for external connection to the Kubernetes services on bare metal

  • Ceph RADOS Gateway HA

  • Ceph RADOS Gateway check box in Container Cloud web UI

  • Ceph maintenance label

  • Cerebro support for StackLight

  • Proxy support for StackLight

2.4.0

February 2, 2021

  • Support for the updated version of Mirantis Container Runtime 19.03.14

  • Dedicated network for Kubernetes pods traffic on bare metal clusters

  • Improvements for the feedback form in the Container Cloud web UI

  • StackLight enhancements:

    • Alert inhibition rules

    • Integration between Grafana and Kibana

    • New Telegraf alert TelegrafGatherErrors

    • Configuration of Ironic Telegraf input plugin

    • Automatically defined cluster ID

2.3.0

December 23, 2020

  • Support for Mirantis Kubernetes Engine 3.3.4 and Mirantis Container Runtime 19.03.13

  • Support for multiple host-specific L2 templates per a bare metal cluster

  • Additional regional cluster on VMWare vSphere

  • Automated setup of an OVF template for the VMWare vSphere provider

  • StackLight support for VMWare vSphere

  • Improvements in the Container Cloud logs collection

2.2.0

November 5, 2020

  • Support for VMWare vSphere provider on RHEL

  • Kernel parameters management through BareMetalHostProfile

  • Support of multiple subnets per cluster

  • Optimization of the Container Cloud logs collection

  • Container Cloud API documentation for bare metal

2.1.0

October 19, 2020

  • Node labeling for machines

  • AWS resources discovery in the Container Cloud web UI

  • Credentials statuses for OpenStack and AWS in the Container Cloud web UI

  • StackLight improvements:

    • Grafana upgrade from version 6.6.2 to 7.1.5

    • Grafana Image Renderer pod to offload rendering of images from charts

    • Grafana home dashboard improvements

    • Splitting of the regional and management cluster function in StackLight telemetry to obtain aggregated metrics on the management cluster from regional and managed clusters

    • Amendments to the StackLight alerts

2.0.0

September 16, 2020

5.7.0

First GA release of Container Cloud with the following key features:

  • Container Cloud with Mirantis Kubernetes Engine (MKE) container clusters for the management plane

  • Support for managed Container Cloud with MKE container clusters on top of the AWS, OpenStack, and bare metal cloud providers

  • Support for attaching of the existing MKE standalone clusters

  • Ceph as a Kubernetes storage provider for the bare metal use case

  • Multi-region support for security and scalability

  • IAM integration with MKE container clusters to provide SSO

  • Logging, monitoring, and alerting tuned for MKE with data aggregation to the management cluster and telemetry sent to Mirantis

2.6.0

The Mirantis Container Cloud GA release 2.6.0:

  • Introduces support for the Cluster release 5.13.0 that is based on Kubernetes 1.18, Mirantis Container Runtime 19.03.14, and Mirantis Kubernetes Engine 3.3.6.

  • Supports the Cluster release 6.12.0 that is based on the Cluster release 5.12.0 and represents Mirantis OpenStack for Kubernetes (MOS) 21.1.

  • Still supports deprecated Cluster releases 5.12.0 and 6.10.0 that will become unsupported in one of the following Container Cloud releases.

  • Supports the Cluster release 5.11.0 only for attachment of existing MKE 3.3.4 clusters. For the deployment of new or attachment of existing MKE 3.3.6 clusters, the latest available Cluster release is used.

    Caution

    Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.6.0.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.6.0. For the list of enhancements in the Cluster release 5.13.0 and Cluster release 6.12.0 that are supported by the Container Cloud release 2.6.0, see the 5.13.0 and 6.12.0 sections.


RHEL license activation using the activation key

Technology Preview

In the scope of Technology Preview support for the VMWare vSphere cloud provider on RHEL, added an additional RHEL license activation method that uses the activation key through RedHat Customer Portal or RedHat Satellite server.

The Satellite configuration on the hosts is done by installing a specific pre-generated RPM package from the Satellite package URL provided by the user through API. The activation key is provided by the user through API.

Along with the new activation method, you can still use the existing method that is adding of your RHEL subscription with the user name and password of your RedHat Customer Portal account associated with your RHEL license for Virtual Datacenters.


Support for VMware vSphere Distributed Switch

Technology Preview

In the scope of Technology Preview support for the VMWare vSphere cloud provider on RHEL, added support for VMware vSphere Distributed Switch (VDS) to provide networking to the vSphere virtual machines. This is an alternative to the vSphere Standard Switch with network on top of it. A VM is attached to a VDS port group. You can specify the path to the port group using the NetworkPath parameter in VsphereClusterProviderSpec.


VMWare vSphere provider integration with IPAM controller

Technology Preview

In the scope of Technology Preview support for the VMWare vSphere cloud provider on RHEL, enabled the vSphere provider to use IPAM controller to assign IP addresses to VMs automatically, without an external DHCP server. If the IPAM controller is not enabled in the bootstrap template, the vSphere provider must rely on external provisioning of the IP addresses by a DHCP server of the user infrastructure.


Proxy support for all Container Cloud providers

Extended proxy support by enabling the feature for the remaining supported AWS and bare metal cloud providers. If you require all Internet access to go through a proxy server for security and audit purposes, you can now bootstrap management and regional clusters of any cloud provider type using proxy.

You can also enable a separate proxy access on the OpenStack-based managed clusters using the Container Cloud web UI. This proxy is intended for the end user needs and is not used for a managed cluster deployment or for access to the Mirantis resources.

Caution

Enabling of proxy access using the Container Cloud web UI for the vSphere, AWS, and baremetal-based managed clusters is on the final development stage and will become available in the next release.


Updated documentation on the bare metal networking

Expanded and restructured the bare metal networking documentation that now contains the following subsections with a detailed description of every bare metal network type:

  • IPAM network

  • Management network

  • Cluster network

  • Host network

Addressed issues

The following issues have been addressed in the Mirantis Container Cloud release 2.6.0 and the Cluster release 5.13.0:

  • [11302] [LCM] Fixed the issue with inability to delete a Container Cloud project with attached MKE clusters that failed to be cleaned up properly.

  • [11967] [LCM] Added vrrp_script chk_myscript to the Keepalived configuration to prevent issues with VIP (Virtual IP) pointing to a node with broken Kubernetes API.

  • [10491] [LCM] Fixed the issue with kubelet being randomly stuck, for example, after a management cluster upgrade. The fix enables automatic restart of kubelet in case of failures.

  • [7782] [bootstrap] Renamed the SSH key used during bootstrap for every cloud provider from openstack_tmp to an accurate and clear ssh_key.

  • [11927] [StackLight] Fixed the issue with StackLight failing to integrate with an external proxy with authentication handled by a proxy server and ignoring the HTTP Authorization header for basic authentication passed by Prometheus Alertmanager.

  • [11001] [StackLight] Fixed the issue with Patroni pod failing to start and remaining in the CrashLoopBackOff status after the management cluster update.

  • [10829] [IAM] Fixed the issue with the Keycloak pods failing to start during a management cluster bootstrap with the Failed to update database exception in logs.

  • [11468] [BM] Fixed the issue with the persistent volumes (PVs) that are created using local volume provisioner (LVP) not being mounted on the dedicated disk labeled as local-volume and using the root volume instead.

  • [9875] [BM] Fixed the issue with the bootstrap.sh preflight script failing with a timeout waiting for BareMetalHost if KAAS_BM_FULL_PREFLIGHT was enabled.

  • [11633] [vSphere] Fixed the issue with the vSphere-based managed cluster projects failing to be cleaned up because of stale secret(s) related to the RHEL license object(s).

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.6.0 including the Cluster release 5.13.0 and 6.12.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[8013] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

vSphere
[12683] The kaas-ipam pods restart on the vSphere region with IPAM disabled

Fixed in Container Cloud 2.7.0

Even though IPAM is disabled on the vSphere-based regional cluster deployed on top of an AWS-based management cluster, the regional cluster still has the kaas-ipam pods installed and continuously restarts them. In this case, the pods logs contain the following exemplary errors:

Waiting for CRDs. [baremetalhosts.metal3.io clusters.cluster.k8s.io machines.cluster.k8s.io
ipamhosts.ipam.mirantis.com ipaddrs.ipam.mirantis.com subnets.ipam.mirantis.com subnetpools.ipam.mirantis.com \
l2templates.ipam.mirantis.com] not found yet
E0318 11:58:21.067502  1 main.go:240] Fetch CRD list failed: \
Object 'Kind' is missing in 'unstructured object has no kind'

As a result, the KubePodCrashLooping StackLight alerts are firing in Alertmanager for kaas-ipam. Disregard these alerts.

[13176] ClusterNetwork settings may disappear from the cluster provider spec

Fixed in Container Cloud 2.7.0

A vSphere-based cluster with IPAM enabled may lose cluster network settings related to IPAM leading to invalid metadata provided to virtual machines. As a result, virtual machines can not obtain assigned IP addresses. The issue occurs during a management cluster bootstrap or a managed cluster creation.

Workaround:

  • If the management cluster with IPAM enabled is not deployed yet, follow the steps below before launching the bootstrap.sh script:

    1. Open kaas-bootstrap/releases/kaas/2.6.0.yaml for editing.

    2. Change the release-controller version from 1.18.1 to 1.18.3:

      - name: release-controller
        version: 1.18.3
        chart: kaas-release/release-controller
        namespace: kaas
        values:
          image:
            tag: 1.18.3
      

    Now, proceed with the management cluster bootstrap.

  • If the management cluster is already deployed, and you want to create a vSphere-based managed cluster with IPAM enabled:

    1. Log in to a local machine where your management or regional cluster kubeconfig is located and export it:

      export KUBECONFIG=kaas-bootstrap/kubeconfig
      
    2. Edit the kaasrelease object by updating the release-controller chart and image version from 1.18.1 to 1.18.3:

      kubectl edit  kaasrelease kaas-2-6-0
      
      - chart: kaas-release/release-controller
        name: release-controller
        namespace: kaas
        values:
          image:
            tag: 1.18.3
        version: 1.18.3
      
    3. Verify that the release-controller deployment is ready with 3/3 replicas:

      kubectl get deployment release-controller-release-controller -n kaas -o=jsonpath='{.status.readyReplicas}/{.status.replicas}'
      

    Now, you can deploy managed clusters with IPAM enabled. For details, see Operations Guide: Create a vSphere-based managed cluster.


Bare metal
[7655] Wrong status for an incorrectly configured L2 template

If an L2 template is configured incorrectly, a bare metal cluster is deployed successfully but with the runtime errors in the IpamHost object.

Workaround:

If you suspect that the machine is not working properly because of incorrect network configuration, verify the status of the corresponding IpamHost object. Inspect the l2RenderResult and ipAllocationResult object fields for error messages.


StackLight
[13078] Elasticsearch does not receive data from Fluentd

Fixed in Container Cloud 2.7.0

Elasticsearch may stop receiving new data from Fluentd. In such case, error messages similar to the following will be present in fluentd-elasticsearch logs:

ElasticsearchError error="400 - Rejected by Elasticsearch [error type]:
illegal_argument_exception [reason]: 'Validation Failed: 1: this action would
add [15] total shards, but this cluster currently has [2989]/[3000] maximum
shards open;'" location=nil tag="ucp-kubelet"

The workaround is to manually increase the limit of open index shards per node:

kubectl -n stacklight exec -ti elasticsearch-master-0 -- \
curl -XPUT -H "content-type: application/json" \
-d '{"persistent":{"cluster.max_shards_per_node": 20000}}' \
http://127.0.0.1:9200/_cluster/settings

Storage
[10060] Ceph OSD node removal fails

Fixed in Container Cloud 2.7.0

A Ceph node removal is not being triggered properly after updating the KaasCephCluster custom resource (CR). Both management and managed clusters are affected.

Workaround:

  1. Remove the parameters for a Ceph OSD from the KaasCephCluster CR as described in Operations Guide: Add, remove, or reconfigure Ceph nodes.

  2. Obtain the IDs of the osd and mon services that are located on the old node:

    1. Obtain the UID of the affected machine:

      kubectl get machine <CephOSDNodeName> -n <ManagedClusterProjectName> -o jsonpath='{.metadata.annotations.kaas\.mirantis\.com\/uid}'
      
    2. Export kubeconfig of your managed cluster. For example:

      export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
      
    3. Identify the pods IDs that run the osd and mon services:

      kubectl get pods -o wide -n rook-ceph | grep <affectedMachineUID> | grep -E "mon|osd"
      

      Example of the system response extract:

      rook-ceph-mon-c-7bbc5d757d-5bpws                              1/1  Running    1  6h1m
      rook-ceph-osd-2-58775d5568-5lklw                              1/1  Running    4  44h
      rook-ceph-osd-prepare-705ae6c647cfdac928c63b63e2e2e647-qn4m9  0/1  Completed  0  94s
      

      The pods IDs include the osd or mon services IDs. In the example system response above, the osd ID is 2 and the mon ID is c.

  3. Delete the deployments of the osd and mon services obtained in the previous step:

    kubectl delete deployment rook-ceph-osd(mon)-<ID> -n rook-ceph
    

    For example:

    kubectl delete deployment rook-ceph-mon-c -n rook-ceph
    kubectl delete deployment rook-ceph-osd-2 -n rook-ceph
    
  4. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  5. Rebalance the Ceph OSDs:

    ceph osd out osd(s).ID
    

    Wait for the rebalance to complete.

  6. Rebalance the Ceph data:

    ceph osd purge osd(s).ID
    

    Wait for the Ceph data to rebalance.

  7. Remove the old node from the Ceph OSD tree:

    ceph osd crush rm <NodeName>
    
  8. If the removed node contained mon services, remove them:

    ceph mon rm <monID>
    
[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.

[10050] Ceph OSD pod is in the CrashLoopBackOff state after disk replacement

If you use a custom BareMetalHostProfile, after disk replacement on a Ceph OSD, the Ceph OSD pod switches to the CrashLoopBackOff state due to the Ceph OSD authorization key failing to be created properly.

Workaround:

  1. Export kubeconfig of your managed cluster. For example:

    export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
    
  2. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  3. Delete the authorization key for the failed Ceph OSD:

    ceph auth del osd.<ID>
    
  4. SSH to the node on which the Ceph OSD cannot be created.

  5. Clean up the disk that will be a base for the failed Ceph OSD. For details, see official Rook documentation.

    Note

    Ignore failures of the sgdisk --zap-all $DISK and blkdiscard $DISK commands if any.

  6. On the managed cluster, restart the Rook operator:

    kubectl -n rook-ceph delete pod -l app=rook-ceph-operator
    
[12723] ceph_role_* labels remain after deleting a node from KaaSCephCluster

The ceph_role_mon and ceph_role_mgr labels that Ceph controller assigns to a node during a Ceph cluster creation are not automatically removed after deleting a node from KaaSCephCluster.

As a workaround, manually remove the labels using the following commands:

kubectl unlabel node <nodeName> ceph_role_mon
kubectl unlabel node <nodeName> ceph_role_mgr

LCM
[13402] Cluster fails with error: no space left on device

If an application running on a Container Cloud management or managed cluster fails frequently, for example, PostgreSQL, it may produce an excessive amount of core dumps. This leads to the no space left on device error on the cluster nodes and, as a result, to the broken Docker Swarm and the entire cluster.

Core dumps are disabled by default on the operating system of the Container Cloud nodes. But since Docker does not inherit the operating system settings, disable core dumps in Docker using the workaround below.

Warning

The workaround below does not apply to the baremetal-based clusters, including MOS deployments, since Docker restart may destroy the Ceph cluster.

Workaround:

  1. SSH to any machine of the affected cluster using mcc-user and the SSH key provided during the cluster creation.

  2. In /etc/docker/daemon.json, add the following parameters:

    {
        ...
        "default-ulimits": {
            "core": {
                "Hard": 0,
                "Name": "core",
                "Soft": 0
            }
        }
    }
    
  3. Restart the Docker daemon:

    systemctl restart docker
    
  4. Repeat the steps above on each machine of the affected cluster one by one.

[10029] Authentication fails with the 401 Unauthorized error

Authentication may not work on some controller nodes after a managed cluster creation. As a result, the Kubernetes API operations with the managed cluster kubeconfig fail with Response Status: 401 Unauthorized.

As a workaround, manually restart the ucp-controller and ucp-auth Docker services on the affected node.

[6066] Helm releases get stuck in FAILED or UNKNOWN state

During a management, regional, or managed cluster deployment, Helm releases may get stuck in the FAILED or UNKNOWN state although the corresponding machines statuses are Ready in the Container Cloud web UI. For example, if the StackLight Helm release fails, the links to its endpoints are grayed out in the web UI. In the cluster status, providerStatus.helm.ready and providerStatus.helm.releaseStatuses.<releaseName>.success are false.

HelmBundle cannot recover from such states and requires manual actions. The workaround below describes the recovery steps for the stacklight release that got stuck during a cluster deployment. Use this procedure as an example for other Helm releases as required.

Workaround:

  1. Verify the failed release has the UNKNOWN or FAILED status in the HelmBundle object:

    kubectl --kubeconfig <regionalClusterKubeconfigPath> get helmbundle <clusterName> -n <clusterProjectName> -o=jsonpath={.status.releaseStatuses.stacklight}
    
    In the command above and in the steps below, replace the parameters
    enclosed in angle brackets with the corresponding values of your cluster.
    

    Example of system response:

    stacklight:
    attempt: 2
    chart: ""
    finishedAt: "2021-02-05T09:41:05Z"
    hash: e314df5061bd238ac5f060effdb55e5b47948a99460c02c2211ba7cb9aadd623
    message: '[{"occurrence":1,"lastOccurrenceDate":"2021-02-05 09:41:05","content":"error
      updating the release: rpc error: code = Unknown desc = customresourcedefinitions.apiextensions.k8s.io
      \"helmbundles.lcm.mirantis.com\" already exists"}]'
    notes: ""
    status: UNKNOWN
    success: false
    version: 0.1.2-mcp-398
    
  2. Log in to the helm-controller pod console:

    kubectl --kubeconfig <affectedClusterKubeconfigPath> exec -n kube-system -it helm-controller-0 sh -c tiller
    
  3. Remove the failed release. For example:

    ./helm --host=localhost:44134 delete stacklight
    

    Once done, the release triggers for redeployment.


Management and regional clusters
[9899] Helm releases get stuck in PENDING_UPGRADE during cluster update

Helm releases may get stuck in the PENDING_UPGRADE status during a management or managed cluster upgrade. The HelmBundle controller cannot recover from this state and requires manual actions. The workaround below describes the recovery process for the openstack-operator release that stuck during a managed cluster update. Use it as an example for other Helm releases as required.

Workaround:

  1. Log in to the helm-controller pod console:

    kubectl exec -n kube-system -it helm-controller-0 sh -c tiller
    
  2. Identify the release that stuck in the PENDING_UPGRADE status. For example:

    ./helm --host=localhost:44134 history openstack-operator
    

    Example of system response:

    REVISION  UPDATED                   STATUS           CHART                      DESCRIPTION
    1         Tue Dec 15 12:30:41 2020  SUPERSEDED       openstack-operator-0.3.9   Install complete
    2         Tue Dec 15 12:32:05 2020  SUPERSEDED       openstack-operator-0.3.9   Upgrade complete
    3         Tue Dec 15 16:24:47 2020  PENDING_UPGRADE  openstack-operator-0.3.18  Preparing upgrade
    
  3. Roll back the failed release to the previous revision. For example:

    ./helm --host=localhost:44134 rollback openstack-operator 2
    

    Once done, the release will be reconciled.

[10424] OpenStack-based regional cluster cleanup fails by timeout

An OpenStack-based regional cluster cleanup fails with the timeout error.

Workaround:

  1. Wait for the Cluster object to be deleted in the bootstrap cluster:

    kubectl --kubeconfig <(./bin/kind get kubeconfig --name clusterapi) get cluster
    

    The system output must be empty.

  2. Remove the bootstrap cluster manually:

    ./bin/kind delete cluster --name clusterapi
    

Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.6.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.18.4

aws-credentials-controller

1.18.1

Bare metal

baremetal-operator Updated

4.0.4

baremetal-public-api Updated

4.0.4

baremetal-provider Updated

1.18.6

httpd

1.18.0

ironic Updated

victoria-bionic-20210302180018

ironic-operator Updated

base-bionic-20210301104323

kaas-ipam Updated

base-bionic-20210304134548

local-volume-provisioner

1.0.4-mcp

mariadb

10.4.17-bionic-20210203155435

IAM

iam Updated

2.0.0

iam-controller Updated

1.18.1

keycloak

9.0.0

Container Cloud

admission-controller Updated

1.18.1

byo-credentials-controller Updated

1.18.1

byo-provider Updated

1.18.4

kaas-public-api Updated

1.18.1

kaas-exporter Updated

1.18.1

kaas-ui Updated

1.18.3

lcm-controller Updated

0.2.0-289-gd7e9fa9c

mcc-cache Updated

1.18.1

proxy-controller Updated

1.18.1

release-controller Updated

1.18.1

rhellicense-controller New

1.18.1

squid-proxy

0.0.1-1

OpenStack Updated

openstack-provider

1.18.4

os-credentials-controller

1.18.1

VMWare vSphere Updated

vsphere-provider

1.18.7

vsphere-credentials-controller

1.18.1

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.6.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-4.0.4.tgz

baremetal-public-api Updated

https://binary.mirantis.com/bm/helm/baremetal-public-api-4.0.4.tgz

ironic-python-agent-bionic.kernel Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-victoria-bionic-debug-20210226182519

ironic-python-agent-bionic.initramfs Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-victoria-bionic-debug-20210226182519

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-4.0.4.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator Updated

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20210216135743

httpd

mirantis.azurecr.io/lcm/nginx:1.18.0

ironic Updated

mirantis.azurecr.io/openstack/ironic:victoria-bionic-20210302180018

ironic-inspector Updated

mirantis.azurecr.io/openstack/ironic-inspector:victoria-bionic-20210302180018

ironic-operator Updated

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20210301104323

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20210304134548

mariadb

mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20210203155435


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.18.6.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.18.6.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.18.1.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.18.1.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.18.4.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.18.6.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.18.1.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.18.4.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.18.1.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.18.1.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.18.1.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.18.3.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.18.1.tgz

mcc-cache

https://binary.mirantis.com/core/helm/mcc-cache-1.18.1.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.18.4.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.18.1.tgz

proxy-controller

https://binary.mirantis.com/core/helm/proxy-controller-1.18.1.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.18.1.tgz

rhellicense-controller New

https://binary.mirantis.com/core/helm/rhellicense-controller-1.18.1.tgz

squid-proxy

https://binary.mirantis.com/core/helm/squid-proxy-1.18.1.tgz

vsphere-credentials-controller

https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.18.1.tgz

vsphere-provider

https://binary.mirantis.com/core/helm/vsphere-provider-1.18.7.tgz

Docker images

admission-controller Updated

mirantis.azurecr.io/core/admission-controller:1.18.1

aws-cluster-api-controller Updated

mirantis.azurecr.io/core/aws-cluster-api-controller:1.18.4

aws-credentials-controller Updated

mirantis.azurecr.io/core/aws-credentials-controller:1.18.1

byo-cluster-api-controller Updated

mirantis.azurecr.io/core/byo-cluster-api-controller:1.18.4

byo-credentials-controller Updated

mirantis.azurecr.io/core/byo-credentials-controller:1.18.1

cluster-api-provider-baremetal Updated

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.18.6

frontend Updated

mirantis.azurecr.io/core/frontend:1.18.3

iam-controller Updated

mirantis.azurecr.io/core/iam-controller:1.18.1

kproxy Updated

mirantis.azurecr.io/lcm/kproxy:1.18.1

lcm-controller Updated

mirantis.azurecr.io/core/lcm-controller:v0.2.0-289-gd7e9fa9c

nginx

mirantis.azurecr.io/lcm/nginx:1.18.0

openstack-cluster-api-controller Updated

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.18.4

os-credentials-controller Updated

mirantis.azurecr.io/core/os-credentials-controller:1.18.1

registry

mirantis.azurecr.io/lcm/registry:2.7.1

release-controller Updated

mirantis.azurecr.io/core/release-controller:1.18.1

rhellicense-controller New

mirantis.azurecr.io/core/rhellicense-controller:1.18.1

squid-proxy

mirantis.azurecr.io/core/squid-proxy:0.0.1-1

vsphere-cluster-api-controller Updated

mirantis.azurecr.io/core/vsphere-api-controller:1.18.7

vsphere-credentials-controller Updated

mirantis.azurecr.io/core/vsphere-credentials-controller:1.18.1


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin

iamctl-linux Updated

http://binary.mirantis.com/iam/bin/iamctl-0.4.0-linux

iamctl-darwin Updated

http://binary.mirantis.com/iam/bin/iamctl-0.4.0-darwin

iamctl-windows Updated

http://binary.mirantis.com/iam/bin/iamctl-0.4.0-windows

Helm charts Updated

iam

http://binary.mirantis.com/iam/helm/iam-2.0.0tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

keycloak-proxy Updated

http://binary.mirantis.com/core/helm/keycloak_proxy-1.18.7.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.4.0

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.4.0

kubernetes-entrypoint

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052

keycloak

mirantis.azurecr.io/iam/keycloak:0.4.0

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

2.5.0

The Mirantis Container Cloud GA release 2.5.0:

  • Introduces support for the Cluster release 5.12.0 that is based on Kubernetes 1.18, Mirantis Container Runtime 19.03.14, and the updated version of Mirantis Kubernetes Engine 3.3.6.

  • Introduces support for the Cluster release 6.12.0 that is based on the Cluster release 5.12.0 and supports Mirantis OpenStack for Kubernetes (MOS) 21.1.

  • Still supports previous Cluster releases 5.11.0 and 6.10.0 that are now deprecated and will become unsupported in one of the following Container Cloud releases.

    Caution

    Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.5.0.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.5.0. For the list of enhancements in the Cluster release 5.12.0 and Cluster release 6.12.0 that are supported by the Container Cloud release 2.5.0, see the 5.12.0 and 6.12.0 sections.


Updated version of Mirantis Kubernetes Engine

Updated the Mirantis Kubernetes Engine (MKE) version to 3.3.6 for the Container Cloud management and managed clusters.

For the MKE release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.


Proxy support for OpenStack and VMWare vSphere providers

Implemented proxy support for OpenStack-based and vSphere-based Technology Preview clusters. If you require all Internet access to go through a proxy server for security and audit purposes, you can now bootstrap management and regional clusters using proxy.

You can also enable a separate proxy access on an OpenStack-based managed cluster using the Container Cloud web UI. This proxy is intended for the end user needs and is not used for a managed cluster deployment or for access to the Mirantis resources.

Note

The proxy support for:

  • The OpenStack provider is generally available.

  • The VMWare vSphere provider is available as Technology Preview. For the Technology Preview feature definition, see Preface.

  • The AWS and bare metal providers is in the development stage and will become available in the future Container Cloud releases.


Artifacts caching

Introduced artifacts caching support for all Container Cloud providers to enable deployment of managed clusters without direct Internet access. The Mirantis artifacts used during managed clusters deployment are downloaded through a cache running on a regional cluster.

The feature is enabled by default on new managed clusters based on the Cluster releases 5.12.0 and 6.12.0 and will be automatically enabled on existing clusters during upgrade to the latest version.


NTP server configuration on regional clusters

Implemented the possibility to configure regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region. The feature is applicable to all supported cloud providers. The NTP server parameters can be added before or after management and regional clusters deployment.


Optimized ClusterRelease upgrade process

Optimized the ClusterRelease upgrade process by enabling the Container Cloud provider to upgrade the LCMCluster components, such as MKE, before the HelmBundle components, such as StackLight or Ceph.


Dedicated network for external connection to the Kubernetes services

Technology Preview

Implemented the k8s-ext bridge in L2 templates that allows you to use a dedicated network for external connection to the Kubernetes services exposed by the cluster. When using such bridge, the MetalLB ranges and the IP addresses provided by the subnet that is associated with the bridge must fit in the same CIDR.

If enabled, MetalLB will listen and respond on the dedicated virtual bridge. Also, you can create additional subnets to configure additional address ranges for MetalLB.

Caution

Using of a dedicated network for Kubernetes pods traffic, for external connection to the Kubernetes services exposed by the cluster, and for the Ceph cluster access and replication traffic is available as Technology Preview. Use such configurations for testing and evaluation purposes only. For details about the Mirantis Technology Preview support scope, see the Preface section of this guide.

The following feature is still under development and will be announced in one of the following Container Cloud releases:

  • Switching Kubernetes API to listen to the specified IP address on the node

Addressed issues

The following issues have been addressed in the Mirantis Container Cloud release 2.5.0 and the Cluster releases 5.12.0 and 6.12.0:

  • [10453] [LCM] Fixed the issue with time synchronization on nodes that could cause networking issues.

  • [9748] [LCM] Fixed the issue with the false-positive helmRelease success status in HelmBundle during Helm upgrade operations.

  • [9748] [LCM] Fixed the issue with the false-positive helmRelease success status in HelmBundle during Helm upgrade operations.

  • [8464] Fixed the issue with Helm controller and OIDC integration failing to be deleted during detach of an MKE cluster.


  • [9928] [Ceph] Fixed the issue with Ceph rebalance leading to data loss during a managed cluster update by implementing the maintenance label to be set before and unset after the cluster update.

  • [9892] [Ceph] Fixed the issue with Ceph being locked during a managed cluster update by adding the PodDisruptionBudget object that enables minimum 2 Ceph OSD nodes running without rescheduling during update.


  • [6988] [BM] Fixed the issue with LVM failing to deploy on a new disk if an old volume group with the same name already existed on the target hardware node but on the different disk.

  • [8560] [BM] Fixed the issue with manual deletion of BareMetalHost from a managed cluster leading to its silent removal without a power-off and deprovision. The fix adds the admission controller webhook to validate the old BareMetalHost when the deletion is requested.

  • [11102] [BM] Fixed the issue with Keepalived not detecting and restoring a VIP of a managed cluster node after running the netplan apply command.

  • [9905] [9906] [9909] [9914] [9921] [BM] Fixed the following Ubuntu CVEs in the bare metal Docker images:

    • CVE-2019-20477 and CVE-2020-1747 for PyYAML in vbmc:latest-20201029

    • CVE-2020-1971 for OpenSSL in the following images:

      • dnsmasq:bionic-20201105044831

      • rabbitmq-management:3.7.15-bionic-20200812044813

      • kaas-ipam:base-bionic-20201208153852

      • ironic-operator:base-bionic-20201106182102

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.5.0 including the Cluster release 5.12.0 and 6.12.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[8013] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

vSphere
[11633] A vSphere-based project cannot be cleaned up

Fixed in Container Cloud 2.6.0

A vSphere-based managed cluster project can fail to be cleaned up because of stale secret(s) related to the RHEL license object(s). Before you can successfully clean up such project, manually delete the secret using the steps below.

Workaround:

  1. Log in to a local machine where your management cluster kubeconfig is located and where kubectl is installed.

  2. Obtain the list of stale secrets:

    kubectl --kubeconfig <kubeconfigPath> get secrets -n <projectName>
    
  3. Open each secret for editing:

    kubectl --kubeconfig <kubeconfigPath> edit secret <secret name> -n <projectName>
    
  4. Remove the following lines:

    finalizers:
    - kaas.mirantis.com/credentials-secret
    
  5. Remove stale secrets:

    kubectl --kubeconfig <kubeconfigPath> delete secret <secretName> -n <projectName>
    

Bare metal
[7655] Wrong status for an incorrectly configured L2 template

If an L2 template is configured incorrectly, a bare metal cluster is deployed successfully but with the runtime errors in the IpamHost object.

Workaround:

If you suspect that the machine is not working properly because of incorrect network configuration, verify the status of the corresponding IpamHost object. Inspect the l2RenderResult and ipAllocationResult object fields for error messages.

[9875] Full preflight fails with a timeout waiting for BareMetalHost

Fixed in Container Cloud 2.6.0

If you run bootstrap.sh preflight with KAAS_BM_FULL_PREFLIGHT=true, the script fails with the following message:

failed to create BareMetal objects: failed to wait for objects of kinds BareMetalHost
to become available: timed out waiting for the condition

As a workaround, unset full preflight using unset KAAS_BM_FULL_PREFLIGHT to run fast preflight instead.

[11468] Pods using LVP PV are not mounted to LVP disk

Fixed in Container Cloud 2.6.0

The persistent volumes (PVs) that are created using local volume provisioner (LVP), are not mounted on the dedicated disk labeled as local-volume and use the root volume instead. In the workaround below, we use StackLight volumes as an example.

Workaround:

  1. Identify whether your cluster is affected:

    1. Log in to any control plane node on the management cluster.

    2. Run the following command:

      findmnt /mnt/local-volumes/stacklight/elasticsearch-data/vol00
      

      In the output, inspect the SOURCE column. If the path starts with /dev/mapper/lvm_root-root, the host is affected by the issue.

      Example of system response:

      TARGET                                                 SOURCE                                                                                FSTYPE OPTIONS
      /mnt/local-volumes/stacklight/elasticsearch-data/vol00 /dev/mapper/lvm_root-root[/var/lib/local-volumes/stacklight/elasticsearch-data/vol00] ext4   rw,relatime,errors=remount-ro,data=ordered
      
    3. Verify other StackLight directories by replacing elasticsearch-data in the command above with the corresponding folders names.

      If your cluster is affected, follow the steps below to manually move all data for volumes that must be on the dedicated disk to the mounted device.

  2. Identify all nodes that run the elasticsearch-master pod:

    kubectl -n stacklight get pods -o wide | grep elasticsearch-master
    

    Apply the steps below to all nodes provided in the output.

  3. Identify the mount point for the dedicated device /dev/mapper/lvm_lvp-lvp. Typically, this device is mounted as /mnt/local-volumes.

    findmnt /mnt/local-volumes
    

    Verify that SOURCE for the /mnt/local-volumes mount target is /dev/mapper/lvm_lvp-lvp on all the nodes.

  4. Create new source directories for the volumes on the dedicated device /dev/mapper/lvm_lvp-lvp:

    mkdir -p /mnt/local-volumes/src/stacklight/elasticsearch-data/vol00
    
  5. Stop the pods that use the volumes to ensure that the data is not corrupted during the switch. Set the number of replicas in StatefulSet to 0:

    kubectl -n stacklight edit statefulset elasticsearch-master
    

    Wait until all elasticsearch-master pods are stopped.

  6. Move the Elasticsearch data from the current location to the new directory:

    cp -pR /var/lib/local-volumes/stacklight/elasticsearch-data/vol00/** /mnt/local-volumes/src/stacklight/elasticsearch-data/vol00/
    
  7. Unmount the old source directory from the volume mount point:

    umount /mnt/local-volumes/stacklight/elasticsearch-data/vol00
    

    Apply this step and the next one to every node with the /mnt/local-volumes/stacklight/elasticsearch-data/vol00 volume.

  8. Remount the new source directory to the volume mount point:

    mount --bind /mnt/local-volumes/src/stacklight/elasticsearch-data/vol00 /mnt/local-volumes/stacklight/elasticsearch-data/vol00
    
  9. Edit the Cluster object by adding the highlighted parameters below for the StackLight Helm chart:

    kubectl --kubeconfig <mgmtClusterKubeconfig> edit -n <projectName> cluster <managedClusterName>
    
    spec:
      helmReleases:
      - name: stacklight
        values:
          ...
          elasticsearch:
            clusterHealthCheckParams: wait_for_status=red&timeout=1s
    
  10. Start the Elasticsearch pods by setting the number of replicas in StatefulSet to 3:

    kubectl -n stacklight edit statefulset elasticsearch-master
    

    Wait until all elasticsearch-master pods are up and running.

  11. Remove the previously added clusterHealthCheckParams parameters from the Cluster object.

  12. In /etc/fstab on every node that has the volume /mnt/local-volumes/stacklight/elasticsearch-data/vol00, edit the following entry:

    /var/lib/local-volumes/stacklight/elasticsearch-data/vol00 /mnt/local-volumes/stacklight/elasticsearch-data/vol00 none bind 0 0
    

    In this entry, replace the old directory /var/lib/local-volumes/stacklight/elasticsearch-data/vol00 with the new one: /mnt/local-volumes/src/stacklight/elasticsearch-data/vol00.


Storage
[10060] Ceph OSD node removal fails

Fixed in Container Cloud 2.7.0

A Ceph node removal is not being triggered properly after updating the KaasCephCluster custom resource (CR). Both management and managed clusters are affected.

Workaround:

  1. Remove the parameters for a Ceph OSD from the KaasCephCluster CR as described in Operations Guide: Add, remove, or reconfigure Ceph nodes.

  2. Obtain the IDs of the osd and mon services that are located on the old node:

    1. Obtain the UID of the affected machine:

      kubectl get machine <CephOSDNodeName> -n <ManagedClusterProjectName> -o jsonpath='{.metadata.annotations.kaas\.mirantis\.com\/uid}'
      
    2. Export kubeconfig of your managed cluster. For example:

      export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
      
    3. Identify the pods IDs that run the osd and mon services:

      kubectl get pods -o wide -n rook-ceph | grep <affectedMachineUID> | grep -E "mon|osd"
      

      Example of the system response extract:

      rook-ceph-mon-c-7bbc5d757d-5bpws                              1/1  Running    1  6h1m
      rook-ceph-osd-2-58775d5568-5lklw                              1/1  Running    4  44h
      rook-ceph-osd-prepare-705ae6c647cfdac928c63b63e2e2e647-qn4m9  0/1  Completed  0  94s
      

      The pods IDs include the osd or mon services IDs. In the example system response above, the osd ID is 2 and the mon ID is c.

  3. Delete the deployments of the osd and mon services obtained in the previous step:

    kubectl delete deployment rook-ceph-osd(mon)-<ID> -n rook-ceph
    

    For example:

    kubectl delete deployment rook-ceph-mon-c -n rook-ceph
    kubectl delete deployment rook-ceph-osd-2 -n rook-ceph
    
  4. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  5. Rebalance the Ceph OSDs:

    ceph osd out osd(s).ID
    

    Wait for the rebalance to complete.

  6. Rebalance the Ceph data:

    ceph osd purge osd(s).ID
    

    Wait for the Ceph data to rebalance.

  7. Remove the old node from the Ceph OSD tree:

    ceph osd crush rm <NodeName>
    
  8. If the removed node contained mon services, remove them:

    ceph mon rm <monID>
    
[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.

[10050] Ceph OSD pod is in the CrashLoopBackOff state after disk replacement

If you use a custom BareMetalHostProfile, after disk replacement on a Ceph OSD, the Ceph OSD pod switches to the CrashLoopBackOff state due to the Ceph OSD authorization key failing to be created properly.

Workaround:

  1. Export kubeconfig of your managed cluster. For example:

    export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
    
  2. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  3. Delete the authorization key for the failed Ceph OSD:

    ceph auth del osd.<ID>
    
  4. SSH to the node on which the Ceph OSD cannot be created.

  5. Clean up the disk that will be a base for the failed Ceph OSD. For details, see official Rook documentation.

    Note

    Ignore failures of the sgdisk --zap-all $DISK and blkdiscard $DISK commands if any.

  6. On the managed cluster, restart the Rook operator:

    kubectl -n rook-ceph delete pod -l app=rook-ceph-operator
    

IAM
[10829] Keycloak pods fail to start during a management cluster bootstrap

Fixed in Container Cloud 2.6.0

The Keycloak pods may fail to start during a management cluster bootstrap with the Failed to update database exception in logs.

Caution

The following workaround is applicable only to deployments where mariadb-server has started successfully. Otherwise, fix the issues with MariaDB first.

Workaround:

  1. Verify that mariadb-server has started:

    kubectl get po -n kaas | grep mariadb-server
    
  2. Scale down the Keycloak instances:

    kubectl scale sts iam-keycloak --replicas=0 -n kaas
    
  3. Open the iam-keycloak-sh configmap for editing:

    kubectl edit cm -n kaas iam-keycloak-sh
    
  4. On the last line of the configmap, before the $MIGRATION_ARGS variable, add the following parameter:

    -Djboss.as.management.blocking.timeout=<RequiredValue>
    

    The recommended timeout value is minimum 15 minutes set in seconds. For example, -Djboss.as.management.blocking.timeout=900.

  5. Open the iam-keycloak-startup configmap for editing:

    kubectl edit cm -n kaas iam-keycloak-startup
    
  6. In the iam-keycloak-startup configmap, add the following line:

    /subsystem=transactions/:write-attribute(name=default-timeout,value=<RequiredValue>)
    

    The recommended timeout value is minimum 15 minutes set in seconds.

  7. In the Keycloak StatefulSet, adjust liveness probe timeouts:

    kubectl edit sts -n kaas iam-keycloak
    
  8. Scale up the Keycloak instances:

    kubectl scale sts iam-keycloak --replicas=3 -n kaas
    

LCM
[10029] Authentication fails with the 401 Unauthorized error

Authentication may not work on some controller nodes after a managed cluster creation. As a result, the Kubernetes API operations with the managed cluster kubeconfig fail with Response Status: 401 Unauthorized.

As a workaround, manually restart the ucp-controller and ucp-auth Docker services on the affected node.

[6066] Helm releases get stuck in FAILED or UNKNOWN state

During a management, regional, or managed cluster deployment, Helm releases may get stuck in the FAILED or UNKNOWN state although the corresponding machines statuses are Ready in the Container Cloud web UI. For example, if the StackLight Helm release fails, the links to its endpoints are grayed out in the web UI. In the cluster status, providerStatus.helm.ready and providerStatus.helm.releaseStatuses.<releaseName>.success are false.

HelmBundle cannot recover from such states and requires manual actions. The workaround below describes the recovery steps for the stacklight release that got stuck during a cluster deployment. Use this procedure as an example for other Helm releases as required.

Workaround:

  1. Verify the failed release has the UNKNOWN or FAILED status in the HelmBundle object:

    kubectl --kubeconfig <regionalClusterKubeconfigPath> get helmbundle <clusterName> -n <clusterProjectName> -o=jsonpath={.status.releaseStatuses.stacklight}
    
    In the command above and in the steps below, replace the parameters
    enclosed in angle brackets with the corresponding values of your cluster.
    

    Example of system response:

    stacklight:
    attempt: 2
    chart: ""
    finishedAt: "2021-02-05T09:41:05Z"
    hash: e314df5061bd238ac5f060effdb55e5b47948a99460c02c2211ba7cb9aadd623
    message: '[{"occurrence":1,"lastOccurrenceDate":"2021-02-05 09:41:05","content":"error
      updating the release: rpc error: code = Unknown desc = customresourcedefinitions.apiextensions.k8s.io
      \"helmbundles.lcm.mirantis.com\" already exists"}]'
    notes: ""
    status: UNKNOWN
    success: false
    version: 0.1.2-mcp-398
    
  2. Log in to the helm-controller pod console:

    kubectl --kubeconfig <affectedClusterKubeconfigPath> exec -n kube-system -it helm-controller-0 sh -c tiller
    
  3. Remove the failed release. For example:

    ./helm --host=localhost:44134 delete stacklight
    

    Once done, the release triggers for redeployment.


StackLight
[11001] Patroni pod fails to start

Fixed in Container Cloud 2.6.0

After the management cluster update, a Patroni pod may fail to start and remain in the CrashLoopBackOff status. Messages similar to the following ones may be present in Patroni logs:

Local timeline=4 lsn=0/A000000
master_timeline=6
master: history=1 0/1ADEB48       no recovery target specified
2       0/8044500       no recovery target specified
3       0/A0000A0       no recovery target specified
4       0/A1B6CB0       no recovery target specified
5       0/A2C0C80       no recovery target specified

As a workaround, reinitialize the affected pod with a new volume by deleting the pod itself and the associated PersistentVolumeClaim (PVC).

Workaround:

  1. Obtain the PVC of the affected pod:

    kubectl -n stacklight get "pod/${POD_NAME}" -o jsonpath='{.spec.volumes[?(@.name=="storage-volume")].persistentVolumeClaim.claimName}'
    
  2. Delete the affected pod and its PVC:

    kubectl -n stacklight delete "pod/${POD_NAME}" "pvc/${POD_PVC}"
    sleep 3  # wait for StatefulSet to reschedule the pod, but miss dependent PVC creation
    kubectl -n stacklight delete "pod/${POD_NAME}"
    

Management and regional clusters
[9899] Helm releases get stuck in PENDING_UPGRADE during cluster update

Helm releases may get stuck in the PENDING_UPGRADE status during a management or managed cluster upgrade. The HelmBundle controller cannot recover from this state and requires manual actions. The workaround below describes the recovery process for the openstack-operator release that stuck during a managed cluster update. Use it as an example for other Helm releases as required.

Workaround:

  1. Log in to the helm-controller pod console:

    kubectl exec -n kube-system -it helm-controller-0 sh -c tiller
    
  2. Identify the release that stuck in the PENDING_UPGRADE status. For example:

    ./helm --host=localhost:44134 history openstack-operator
    

    Example of system response:

    REVISION  UPDATED                   STATUS           CHART                      DESCRIPTION
    1         Tue Dec 15 12:30:41 2020  SUPERSEDED       openstack-operator-0.3.9   Install complete
    2         Tue Dec 15 12:32:05 2020  SUPERSEDED       openstack-operator-0.3.9   Upgrade complete
    3         Tue Dec 15 16:24:47 2020  PENDING_UPGRADE  openstack-operator-0.3.18  Preparing upgrade
    
  3. Roll back the failed release to the previous revision. For example:

    ./helm --host=localhost:44134 rollback openstack-operator 2
    

    Once done, the release will be reconciled.

[10424] OpenStack-based regional cluster cleanup fails by timeout

An OpenStack-based regional cluster cleanup fails with the timeout error.

Workaround:

  1. Wait for the Cluster object to be deleted in the bootstrap cluster:

    kubectl --kubeconfig <(./bin/kind get kubeconfig --name clusterapi) get cluster
    

    The system output must be empty.

  2. Remove the bootstrap cluster manually:

    ./bin/kind delete cluster --name clusterapi
    

Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.5.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.17.4

aws-credentials-controller

1.17.4

Bare metal

baremetal-operator Updated

3.2.1

baremetal-public-api Updated

3.2.1

baremetal-provider Updated

1.17.6

httpd Updated

1.18.0

ironic Updated

ussuri-bionic-20210202180025

ironic-operator

base-bionic-20210106163336

kaas-ipam Updated

base-bionic-20210218141033

local-volume-provisioner

1.0.4-mcp

mariadb Updated

10.4.17-bionic-20210203155435

IAM

iam Updated

1.3.0

iam-controller Updated

1.17.4

keycloak

9.0.0

Container Cloud Updated

admission-controller

1.17.5

byo-credentials-controller

1.17.4

byo-provider

1.17.4

kaas-public-api

1.17.4

kaas-exporter

1.17.4

kaas-ui

1.17.4

lcm-controller

0.2.0-259-g71792430

mcc-cache New

1.17.4

proxy-controller New

1.17.4

release-controller

1.17.4

squid-proxy New

0.0.1-1

OpenStack Updated

openstack-provider

1.17.4

os-credentials-controller

1.17.4

VMWare vSphere Updated

vsphere-provider

1.17.6

vsphere-credentials-controller

1.17.4

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.5.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-3.2.1.tgz

baremetal-public-api Updated

https://binary.mirantis.com/bm/helm/baremetal-public-api-3.2.1.tgz

ironic-python-agent-bionic.kernel Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-ussuri-bionic-debug-20210204084827

ironic-python-agent-bionic.initramfs Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-ussuri-bionic-debug-20210204084827

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-3.2.1.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20201113171304

httpd Updated

mirantis.azurecr.io/lcm/nginx:1.18.0

ironic Updated

mirantis.azurecr.io/openstack/ironic:ussuri-bionic-20210202180025

ironic-inspector Updated

mirantis.azurecr.io/openstack/ironic-inspector:ussuri-bionic-20210202180025

ironic-operator

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20210106163336

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20210218141033

mariadb Updated

mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20210203155435


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.17.5.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.17.5.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.17.4.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.17.4.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.17.4.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.17.4.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.17.4.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.17.4.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.17.4.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.17.4.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.17.4.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.17.4.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.17.4.tgz

mcc-cache New

https://binary.mirantis.com/core/helm/mcc-cache-1.17.4.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.17.4.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.17.4.tgz

proxy-controller New

https://binary.mirantis.com/core/helm/proxy-controller-1.17.4.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.17.4.tgz

squid-proxy New

https://binary.mirantis.com/core/helm/squid-proxy-1.17.4.tgz

vsphere-credentials-controller

https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.17.4.tgz

vsphere-provider

https://binary.mirantis.com/core/helm/vsphere-provider-1.17.4.tgz

Docker images Updated

admission-controller

mirantis.azurecr.io/core/admission-controller:1.17.5

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.17.4

aws-credentials-controller

mirantis.azurecr.io/core/aws-credentials-controller:1.17.4

byo-cluster-api-controller

mirantis.azurecr.io/core/byo-cluster-api-controller:1.17.4

byo-credentials-controller

mirantis.azurecr.io/core/byo-credentials-controller:1.17.4

cluster-api-provider-baremetal

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.17.6

frontend

mirantis.azurecr.io/core/frontend:1.17.4

iam-controller

mirantis.azurecr.io/core/iam-controller:1.17.4

kproxy New

mirantis.azurecr.io/lcm/kproxy:1.17.4

lcm-controller

mirantis.azurecr.io/core/lcm-controller:v0.2.0-259-g71792430

nginx New

mirantis.azurecr.io/lcm/nginx:1.18.0

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.17.4

os-credentials-controller

mirantis.azurecr.io/core/os-credentials-controller:1.17.4

registry New

mirantis.azurecr.io/lcm/registry:2.7.1

release-controller

mirantis.azurecr.io/core/release-controller:1.17.4

squid-proxy New

mirantis.azurecr.io/core/squid-proxy:0.0.1-1

vsphere-cluster-api-controller

mirantis.azurecr.io/core/vsphere-api-controller:1.17.6

vsphere-credentials-controller

mirantis.azurecr.io/core/vsphere-credentials-controller:1.17.6


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin

iamctl-linux

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-linux

iamctl-darwin

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-darwin

iamctl-windows

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-windows

Helm charts Updated

iam

http://binary.mirantis.com/iam/helm/iam-1.3.0.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

keycloak-proxy

http://binary.mirantis.com/core/helm/keycloak_proxy-1.17.4.tgz

Docker images

api Updated

mirantis.azurecr.io/iam/api:0.4.0

auxiliary Updated

mirantis.azurecr.io/iam/auxiliary:0.4.0

kubernetes-entrypoint Updated

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052

keycloak Updated

mirantis.azurecr.io/iam/keycloak:0.4.0

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

2.4.0

The Mirantis Container Cloud GA release 2.4.0:

  • Introduces support for the Cluster release 5.11.0 that is based on Kubernetes 1.18, Mirantis Kubernetes Engine 3.3.4, and the updated version of Mirantis Container Runtime 19.03.14.

  • Supports the Cluster release 6.10.0 that is based on the Cluster release 5.10.0 and supports Mirantis OpenStack for Kubernetes (MOS) Ussuri.

  • Still supports previous Cluster releases 5.10.0 and 6.8.1 that are now deprecated and will become unsupported in one of the following Container Cloud releases.

    Caution

    Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.4.0.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.4.0. For the list of enhancements in the Cluster release 5.11.0 and Cluster release 6.10.0 that are supported by the Container Cloud release 2.4.0, see the 5.11.0 and 6.10.0 sections.


Support for the updated version of Mirantis Container Runtime

Updated the Mirantis Container Runtime (MCR) version to 19.03.14 for all types of Container Cloud clusters.

For the Mirantis Container Runtime release highlights, see MKE, MSR, and MCR documentation: Mirantis Container Runtime release notes.

Caution

Due to the development limitations, the MCR upgrade to version 19.03.13 or 19.03.14 on existing Container Cloud clusters is not supported.


Dedicated network for Kubernetes pods traffic on bare metal clusters

Technology Preview

Implemented the k8s-pods bridge in L2 templates that allows you to use a dedicated network for Kubernetes pods traffic. When the k8s-pods bridge is defined in an L2 template, Calico CNI uses that network for routing the pods traffic between nodes.

Caution

Using of a dedicated network for Kubernetes pods traffic described above is available as Technology Preview. Use such configuration for testing and evaluation purposes only. For details about the Mirantis Technology Preview support scope, see the Preface section of this guide.

The following features are still under development and will be announced in one of the following Container Cloud releases:

  • Switching Kubernetes API to listen to the specified IP address on the node

  • Enable MetalLB to listen and respond on the dedicated virtual bridge.


Feedback form improvement in Container Cloud web UI

Extended the functionality of the feedback form for the Container Cloud web UI. Using the Feedback button, you can now provide 5-star product rating and feedback about Container Cloud. If you have an idea or found a bug in Container Cloud, you can create a ticket for the Mirantis support team to help us improve the product.

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.4.0 including the Cluster release 5.11.0 and 6.10.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[8013] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

Bare metal
[9875] Full preflight fails with a timeout waiting for BareMetalHost

Fixed in Container Cloud 2.6.0

If you run bootstrap.sh preflight with KAAS_BM_FULL_PREFLIGHT=true, the script fails with the following message:

failed to create BareMetal objects: failed to wait for objects of kinds BareMetalHost
to become available: timed out waiting for the condition

As a workaround, unset full preflight using unset KAAS_BM_FULL_PREFLIGHT to run fast preflight instead.

[11102] Keepalived does not detect the loss of VIP deleted by netplan

Fixed in Container Cloud 2.5.0

This issue may occur on the baremetal-based managed clusters that are created using L2 templates when network configuration is changed by the user or when Container Cloud is updated from version 2.3.0 to 2.4.0.

Due to the community issue, Keepalived 1.3.9 does not detect and restore a VIP of a managed cluster node after running the netplan apply command. The command is used to apply network configuration changes.

As a result, the Kubernetes API on the affected managed clusters becomes inaccessible.

As a workaround, log in to all nodes of the affected managed clusters and restart Keepalived using systemctl restart keepalived.

[6988] LVM fails to deploy if the volume group name already exists

Fixed in Container Cloud 2.5.0

During a management or managed cluster deployment, LVM cannot be deployed on a new disk if an old volume group with the same name already exists on the target hardware node but on the different disk.

Workaround:

In the bare metal host profile specific to your hardware configuration, add the wipe: true parameter to the device that fails to be deployed. For the procedure details, see Operations Guide: Create a custom host profile.

[7655] Wrong status for an incorrectly configured L2 template

If an L2 template is configured incorrectly, a bare metal cluster is deployed successfully but with the runtime errors in the IpamHost object.

Workaround:

If you suspect that the machine is not working properly because of incorrect network configuration, verify the status of the corresponding IpamHost object. Inspect the l2RenderResult and ipAllocationResult object fields for error messages.

[8560] Manual deletion of BareMetalHost leads to its silent removal

Fixed in Container Cloud 2.5.0

If BareMetalHost is manually removed from a managed cluster, it is silently removed without a power-off and deprovision that leads to a managed cluster failures.

Workaround:

Do not manually delete a BareMetalHost that has the Provisioned status.


Storage
[10060] Ceph OSD node removal fails

Fixed in Container Cloud 2.7.0

A Ceph node removal is not being triggered properly after updating the KaasCephCluster custom resource (CR). Both management and managed clusters are affected.

Workaround:

  1. Remove the parameters for a Ceph OSD from the KaasCephCluster CR as described in Operations Guide: Add, remove, or reconfigure Ceph nodes.

  2. Obtain the IDs of the osd and mon services that are located on the old node:

    1. Obtain the UID of the affected machine:

      kubectl get machine <CephOSDNodeName> -n <ManagedClusterProjectName> -o jsonpath='{.metadata.annotations.kaas\.mirantis\.com\/uid}'
      
    2. Export kubeconfig of your managed cluster. For example:

      export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
      
    3. Identify the pods IDs that run the osd and mon services:

      kubectl get pods -o wide -n rook-ceph | grep <affectedMachineUID> | grep -E "mon|osd"
      

      Example of the system response extract:

      rook-ceph-mon-c-7bbc5d757d-5bpws                              1/1  Running    1  6h1m
      rook-ceph-osd-2-58775d5568-5lklw                              1/1  Running    4  44h
      rook-ceph-osd-prepare-705ae6c647cfdac928c63b63e2e2e647-qn4m9  0/1  Completed  0  94s
      

      The pods IDs include the osd or mon services IDs. In the example system response above, the osd ID is 2 and the mon ID is c.

  3. Delete the deployments of the osd and mon services obtained in the previous step:

    kubectl delete deployment rook-ceph-osd(mon)-<ID> -n rook-ceph
    

    For example:

    kubectl delete deployment rook-ceph-mon-c -n rook-ceph
    kubectl delete deployment rook-ceph-osd-2 -n rook-ceph
    
  4. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  5. Rebalance the Ceph OSDs:

    ceph osd out osd(s).ID
    

    Wait for the rebalance to complete.

  6. Rebalance the Ceph data:

    ceph osd purge osd(s).ID
    

    Wait for the Ceph data to rebalance.

  7. Remove the old node from the Ceph OSD tree:

    ceph osd crush rm <NodeName>
    
  8. If the removed node contained mon services, remove them:

    ceph mon rm <monID>
    
[9928] Ceph rebalance during a managed cluster update

Fixed in Container Cloud 2.5.0

During a managed cluster update, Ceph rebalance leading to data loss may occur.

Workaround:

  1. Before updating a managed cluster:

    1. Log in to the ceph-tools pod:

      kubectl -n rook-ceph exec -it <ceph-tools-pod-name> bash
      
    2. Set the noout flag:

      ceph osd set noout
      
  2. Update a managed cluster.

  3. After updating a managed cluster:

    1. Log in to the ceph-tools pod:

      kubectl -n rook-ceph exec -it <ceph-tools-pod-name> bash
      
    2. Unset the noout flag:

      ceph osd unset noout
      
[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.

[10050] Ceph OSD pod is in the CrashLoopBackOff state after disk replacement

If you use a custom BareMetalHostProfile, after disk replacement on a Ceph OSD, the Ceph OSD pod switches to the CrashLoopBackOff state due to the Ceph OSD authorization key failing to be created properly.

Workaround:

  1. Export kubeconfig of your managed cluster. For example:

    export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
    
  2. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  3. Delete the authorization key for the failed Ceph OSD:

    ceph auth del osd.<ID>
    
  4. SSH to the node on which the Ceph OSD cannot be created.

  5. Clean up the disk that will be a base for the failed Ceph OSD. For details, see official Rook documentation.

    Note

    Ignore failures of the sgdisk --zap-all $DISK and blkdiscard $DISK commands if any.

  6. On the managed cluster, restart the Rook operator:

    kubectl -n rook-ceph delete pod -l app=rook-ceph-operator
    

LCM
[10029] Authentication fails with the 401 Unauthorized error

Authentication may not work on some controller nodes after a managed cluster creation. As a result, the Kubernetes API operations with the managed cluster kubeconfig fail with Response Status: 401 Unauthorized.

As a workaround, manually restart the ucp-controller and ucp-auth Docker services on the affected node.

[6066] Helm releases get stuck in FAILED or UNKNOWN state

During a management, regional, or managed cluster deployment, Helm releases may get stuck in the FAILED or UNKNOWN state although the corresponding machines statuses are Ready in the Container Cloud web UI. For example, if the StackLight Helm release fails, the links to its endpoints are grayed out in the web UI. In the cluster status, providerStatus.helm.ready and providerStatus.helm.releaseStatuses.<releaseName>.success are false.

HelmBundle cannot recover from such states and requires manual actions. The workaround below describes the recovery steps for the stacklight release that got stuck during a cluster deployment. Use this procedure as an example for other Helm releases as required.

Workaround:

  1. Verify the failed release has the UNKNOWN or FAILED status in the HelmBundle object:

    kubectl --kubeconfig <regionalClusterKubeconfigPath> get helmbundle <clusterName> -n <clusterProjectName> -o=jsonpath={.status.releaseStatuses.stacklight}
    
    In the command above and in the steps below, replace the parameters
    enclosed in angle brackets with the corresponding values of your cluster.
    

    Example of system response:

    stacklight:
    attempt: 2
    chart: ""
    finishedAt: "2021-02-05T09:41:05Z"
    hash: e314df5061bd238ac5f060effdb55e5b47948a99460c02c2211ba7cb9aadd623
    message: '[{"occurrence":1,"lastOccurrenceDate":"2021-02-05 09:41:05","content":"error
      updating the release: rpc error: code = Unknown desc = customresourcedefinitions.apiextensions.k8s.io
      \"helmbundles.lcm.mirantis.com\" already exists"}]'
    notes: ""
    status: UNKNOWN
    success: false
    version: 0.1.2-mcp-398
    
  2. Log in to the helm-controller pod console:

    kubectl --kubeconfig <affectedClusterKubeconfigPath> exec -n kube-system -it helm-controller-0 sh -c tiller
    
  3. Remove the failed release. For example:

    ./helm --host=localhost:44134 delete stacklight
    

    Once done, the release triggers for redeployment.


StackLight
[11001] Patroni pod fails to start

Fixed in Container Cloud 2.6.0

After the management cluster update, a Patroni pod may fail to start and remain in the CrashLoopBackOff status. Messages similar to the following ones may be present in Patroni logs:

Local timeline=4 lsn=0/A000000
master_timeline=6
master: history=1 0/1ADEB48       no recovery target specified
2       0/8044500       no recovery target specified
3       0/A0000A0       no recovery target specified
4       0/A1B6CB0       no recovery target specified
5       0/A2C0C80       no recovery target specified

As a workaround, reinitialize the affected pod with a new volume by deleting the pod itself and the associated PersistentVolumeClaim (PVC).

Workaround:

  1. Obtain the PVC of the affected pod:

    kubectl -n stacklight get "pod/${POD_NAME}" -o jsonpath='{.spec.volumes[?(@.name=="storage-volume")].persistentVolumeClaim.claimName}'
    
  2. Delete the affected pod and its PVC:

    kubectl -n stacklight delete "pod/${POD_NAME}" "pvc/${POD_PVC}"
    sleep 3  # wait for StatefulSet to reschedule the pod, but miss dependent PVC creation
    kubectl -n stacklight delete "pod/${POD_NAME}"
    

Management cluster update
[9899] Helm releases get stuck in PENDING_UPGRADE during cluster update

Helm releases may get stuck in the PENDING_UPGRADE status during a management or managed cluster upgrade. The HelmBundle controller cannot recover from this state and requires manual actions. The workaround below describes the recovery process for the openstack-operator release that stuck during a managed cluster update. Use it as an example for other Helm releases as required.

Workaround:

  1. Log in to the helm-controller pod console:

    kubectl exec -n kube-system -it helm-controller-0 sh -c tiller
    
  2. Identify the release that stuck in the PENDING_UPGRADE status. For example:

    ./helm --host=localhost:44134 history openstack-operator
    

    Example of system response:

    REVISION  UPDATED                   STATUS           CHART                      DESCRIPTION
    1         Tue Dec 15 12:30:41 2020  SUPERSEDED       openstack-operator-0.3.9   Install complete
    2         Tue Dec 15 12:32:05 2020  SUPERSEDED       openstack-operator-0.3.9   Upgrade complete
    3         Tue Dec 15 16:24:47 2020  PENDING_UPGRADE  openstack-operator-0.3.18  Preparing upgrade
    
  3. Roll back the failed release to the previous revision. For example:

    ./helm --host=localhost:44134 rollback openstack-operator 2
    

    Once done, the release will be reconciled.


Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Addressed issues

The following issues have been addressed in the Mirantis Container Cloud release 2.4.0 and the Cluster releases 5.11.0 and 6.10.0:

  • [10351] [BM] [IPAM] Fixed the issue with the automatically allocated subnet having the ability to requeue allocation from a SubnetPool in the error state.

  • [10104] [BM] [Ceph] Fixed the issue with OpenStack services failing to access rook-ceph-mon-* pods due to the changed metadata for connection after pods restart if Ceph was deployed without hostNetwork: true.


  • [2757] [IAM] Fixed the issue with IAM failing to start with the IAM pods being in the CrashLoopBackOff state during a management cluster deployment.

  • [7562] [IAM] Disabled the http port in Keycloak to prevent security vulnerabilities.


  • [10108] [LCM] Fixed the issue with accidental upgrade of the docker-ee, docker-ee-cli, and containerd.io packages that must be pinned during the host OS upgrade.

  • [10094] [LCM] Fixed the issue with error handling in the manage-taints Ansible script.

  • [9676] [LCM] Fixed the issue with Keepalived and NGINX being installed on worker nodes instead of being installed on control plane nodes only.


  • [10323] [UI] Fixed the issue with offline tokens being expired over time if fetched using the Container Cloud web UI. The issue occurred if the Log in with Keycloak option was used.

  • [8966] [UI] Fixed the issue with the “invalid_grant”,”error_description”: “Session doesn’t have required client” error occurring over time after logging in to the Container Cloud web UI through Log in with Keycloak.

  • [10180] [UI] Fixed the issue with the SSH Keys dialog becoming blank after the token expiration.

  • [7781] [UI] Fixed the issue with the previously selected Ceph cluster machines disappearing from the drop-down menu of the Create New Ceph Cluster dialog.

  • [7843] [UI] Fixed the issue with Provider Credentials being stuck in the Processing state if created using the Add new credential option of the Create New Cluster dialog.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.4.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.16.1

aws-credentials-controller

1.16.1

Bare metal

baremetal-operator Updated

3.1.7

baremetal-public-api Updated

3.1.7

baremetal-provider Updated

1.16.4

httpd

2.4.46-20201001171500

ironic Updated

ussuri-bionic-20210113180016

ironic-operator Updated

base-bionic-20210106163336

kaas-ipam Updated

base-bionic-20210106163449

local-volume-provisioner

1.0.4-mcp

mariadb Updated

10.4.17-bionic-20210106145941

IAM

iam Updated

1.2.1

iam-controller Updated

1.16.1

keycloak

9.0.0

Container Cloud

admission-controller Updated

1.16.1

byo-credentials-controller Updated

1.16.1

byo-provider Updated

1.16.1

kaas-public-api Updated

1.16.1

kaas-exporter Updated

1.16.1

kaas-ui Updated

1.16.2

lcm-controller

0.2.0-224-g5c413d37

release-controller Updated

1.16.1

OpenStack Updated

openstack-provider

1.16.1

os-credentials-controller

1.16.1

VMWare vSphere Updated

vsphere-provider

1.16.1

vsphere-credentials-controller

1.16.4

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.4.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-3.1.7.tgz

baremetal-public-api Updated

https://binary.mirantis.com/bm/helm/baremetal-public-api-3.1.7.tgz

ironic-python-agent-bionic.kernel Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-ussuri-bionic-debug-20210108095808

ironic-python-agent-bionic.initramfs Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-ussuri-bionic-debug-20210108095808

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-3.1.7.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20201113171304

httpd

mirantis.azurecr.io/bm/external/httpd:2.4.46-20201001171500

ironic Updated

mirantis.azurecr.io/openstack/ironic:ussuri-bionic-20210113180016

ironic-inspector Updated

mirantis.azurecr.io/openstack/ironic-inspector:ussuri-bionic-20210113180016

ironic-operator Updated

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20210106163336

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20210106163449

mariadb Updated

mirantis.azurecr.io/general/mariadb:10.4.17-bionic-20210106145941


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.16.1.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.16.1.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.16.1.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.16.1.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.16.1.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.16.1.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.16.1.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.16.1.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.16.1.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.16.1.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.16.1.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.16.1.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.16.1.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.16.1.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.16.1.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.16.1.tgz

vsphere-credentials-controller

https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.16.1.tgz

vsphere-provider

https://binary.mirantis.com/core/helm/vsphere-provider-1.16.1.tgz

Docker images Updated

admission-controller

mirantis.azurecr.io/core/admission-controller:1.16.1

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.16.1

aws-credentials-controller

mirantis.azurecr.io/core/aws-credentials-controller:1.16.1

byo-cluster-api-controller

mirantis.azurecr.io/core/byo-cluster-api-controller:1.16.1

byo-credentials-controller

mirantis.azurecr.io/core/byo-credentials-controller:1.16.1

cluster-api-provider-baremetal

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.16.1

frontend

mirantis.azurecr.io/core/frontend:1.16.1

iam-controller

mirantis.azurecr.io/core/iam-controller:1.16.1

lcm-controller

mirantis.azurecr.io/core/lcm-controller:v0.2.0-224-g5c413d37

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.16.1

os-credentials-controller

mirantis.azurecr.io/core/os-credentials-controller:1.16.1

release-controller

mirantis.azurecr.io/core/release-controller:1.16.1

vsphere-cluster-api-controller

mirantis.azurecr.io/core/vsphere-api-controller:1.16.1

vsphere-credentials-controller

mirantis.azurecr.io/core/vsphere-credentials-controller:1.16.4


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin

iamctl-linux Updated

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-linux

iamctl-darwin Updated

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-darwin

iamctl-windows Updated

http://binary.mirantis.com/iam/bin/iamctl-0.3.19-windows

Helm charts

iam Updated

http://binary.mirantis.com/iam/helm/iam-1.2.1.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.3.tgz

keycloak-proxy Updated

http://binary.mirantis.com/core/helm/keycloak_proxy-1.16.3.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.3.18

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.3.18

kubernetes-entrypoint Updated

mirantis.azurecr.io/openstack/extra/kubernetes-entrypoint:v1.0.0-20200311160233

mariadb Updated

mirantis.azurecr.io/general/mariadb:10.4.16-bionic-20201105025052

keycloak

mirantis.azurecr.io/iam/keycloak:0.3.19

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

2.3.0

The Mirantis Container Cloud GA release 2.3.0:

  • Introduces support for the Cluster release 5.10.0 that is based on Kubernetes 1.18 and the updated versions of Mirantis Kubernetes Engine 3.3.4 and Mirantis Container Runtime 19.03.13.

  • Introduces support for the Cluster release 6.10.0 that is based on the Cluster release 5.10.0 and supports Mirantis OpenStack for Kubernetes (MOS) Ussuri.

  • Still supports previous Cluster releases 5.9.0 and 6.8.1 that are now deprecated and will become unsupported in one of the following Container Cloud releases.

    Caution

    Make sure to update the Cluster release version of your managed cluster before the current Cluster release version becomes unsupported by a new Container Cloud release version. Otherwise, Container Cloud stops auto-upgrade and eventually Container Cloud itself becomes unsupported.

This section outlines release notes for the Container Cloud release 2.3.0.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.3.0. For the list of enhancements in the Cluster release 5.10.0 and Cluster release 6.10.0 introduced by the Container Cloud release 2.3.0, see the 5.10.0 and 6.10.0 sections.


Updated versions of Mirantis Kubernetes Engine and Container Runtime

Updated the Mirantis Kubernetes Engine (MKE) version to 3.3.4 and the Mirantis Container Runtime (MCR) version to 19.03.13 for the Container Cloud management and managed clusters.

For the MKE release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

For the Mirantis Container Runtime release highlights, see MKE, MSR, and MCR documentation: Mirantis Container Runtime release notes.

Caution

Due to the development limitations, the MCR upgrade to version 19.03.13 or 19.03.14 on existing Container Cloud clusters is not supported.


Additional regional cluster on VMWare vSphere

Technical Preview

In scope of Technology Preview support for the VMWare vSphere provider, added the capability to deploy an additional regional vSphere-based cluster on top of the vSphere management cluster to create managed clusters with different configurations if required.


Automated setup of an OVF template for the VMWare vSphere provider

Technical Preview

Automated the process of an OVF template setup for the vSphere-based management and managed clusters deployments. The OVF template is now set up by Packer using the vsphere_template flag that is integrated into bootstrap.sh.


StackLight support for VMWare vSphere

Technical Preview

Added the capability to deploy StackLight on management clusters. However, such deployment has the following limitations:

  • The Kubernetes Nodes and Kubernetes Cluster Grafana dashboards may have empty panels.

  • The DockerNetworkUnhealthy and etcdGRPCRequestsSlow alerts may fail to raise.

  • The CPUThrottlingHigh, CalicoDatapaneIfaceMsgBatchSizeHigh, KubeCPUOvercommitPods, KubeMemOvercommitPods alerts, and the TargetDown alert for the prometheus-node-exporter and calico-node pods may be constantly firing.


Support of multiple host-specific L2 templates per a bare metal cluster

Added support of multiple host-specific L2 templates to be applied to different nodes of the same bare metal cluster. Now, you can use several independent host-specific L2 templates on a cluster to support different hardware configurations. For example, you can create L2 templates with a different number and layout of NICs to be applied to the specific machines of a cluster.


Improvements in the Container Cloud logs collection

Improved user experience with the Container Cloud resources logs collection by implementing collecting of logs on the Mirantis Kubernetes Engine cluster and on all Kubernetes pods, including the ones that were previously removed or failed.

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.3.0 including the Cluster release 5.10.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[8013] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

Bare metal
[6988] LVM fails to deploy if the volume group name already exists

Fixed in Container Cloud 2.5.0

During a management or managed cluster deployment, LVM cannot be deployed on a new disk if an old volume group with the same name already exists on the target hardware node but on the different disk.

Workaround:

In the bare metal host profile specific to your hardware configuration, add the wipe: true parameter to the device that fails to be deployed. For the procedure details, see Operations Guide: Create a custom host profile.

[7655] Wrong status for an incorrectly configured L2 template

If an L2 template is configured incorrectly, a bare metal cluster is deployed successfully but with the runtime errors in the IpamHost object.

Workaround:

If you suspect that the machine is not working properly because of incorrect network configuration, verify the status of the corresponding IpamHost object. Inspect the l2RenderResult and ipAllocationResult object fields for error messages.

[8560] Manual deletion of BareMetalHost leads to its silent removal

Fixed in Container Cloud 2.5.0

If BareMetalHost is manually removed from a managed cluster, it is silently removed without a power-off and deprovision that leads to a managed cluster failures.

Workaround:

Do not manually delete a BareMetalHost that has the Provisioned status.

[9875] Full preflight fails with a timeout waiting for BareMetalHost

Fixed in Container Cloud 2.6.0

If you run bootstrap.sh preflight with KAAS_BM_FULL_PREFLIGHT=true, the script fails with the following message:

failed to create BareMetal objects: failed to wait for objects of kinds BareMetalHost
to become available: timed out waiting for the condition

As a workaround, unset full preflight using unset KAAS_BM_FULL_PREFLIGHT to run fast preflight instead.


IAM
[2757] IAM fails to start during management cluster deployment

Fixed in Container Cloud 2.4.0

During a management cluster deployment, IAM fails to start with the IAM pods being in the CrashLoopBackOff status.

Workaround:

  1. Log in to the bootstrap node.

  2. Remove the iam-mariadb-state configmap:

    kubectl delete cm -n kaas iam-mariadb-state
    
  3. Manually delete the mariadb pods:

    kubectl delete po -n kaas mariadb-server-{0,1,2}
    

    Wait for the pods to start. If the mariadb pod does not start with the connection to peer timed out exception, repeat the step 2.

  4. Obtain the MariaDB database admin password:

    kubectl get secrets -n kaas mariadb-dbadmin-password \
    -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d ; echo
    
  5. Log in to MariaDB:

    kubectl exec -it -n kaas mariadb-server-0 -- bash -c 'mysql -uroot -p<mysqlDbadminPassword>'
    

    Substitute <mysqlDbadminPassword> with the corresponding value obtained in the previous step.

  6. Run the following command:

    DROP DATABASE IF EXISTS keycloak;
    
  7. Manually delete the Keycloak pods:

    kubectl delete po -n kaas iam-keycloak-{0,1,2}
    

LCM
[10029] Authentication fails with the 401 Unauthorized error

Authentication may not work on some controller nodes after a managed cluster creation. As a result, the Kubernetes API operations with the managed cluster kubeconfig fail with Response Status: 401 Unauthorized.

As a workaround, manually restart the ucp-controller and ucp-auth Docker services on the affected node.


Management cluster update
[9899] Helm releases get stuck in PENDING_UPGRADE during cluster update

Helm releases may get stuck in the PENDING_UPGRADE status during a management or managed cluster upgrade. The HelmBundle controller cannot recover from this state and requires manual actions. The workaround below describes the recovery process for the openstack-operator release that stuck during a managed cluster update. Use it as an example for other Helm releases as required.

Workaround:

  1. Log in to the helm-controller pod console:

    kubectl exec -n kube-system -it helm-controller-0 sh -c tiller
    
  2. Identify the release that stuck in the PENDING_UPGRADE status. For example:

    ./helm --host=localhost:44134 history openstack-operator
    

    Example of system response:

    REVISION  UPDATED                   STATUS           CHART                      DESCRIPTION
    1         Tue Dec 15 12:30:41 2020  SUPERSEDED       openstack-operator-0.3.9   Install complete
    2         Tue Dec 15 12:32:05 2020  SUPERSEDED       openstack-operator-0.3.9   Upgrade complete
    3         Tue Dec 15 16:24:47 2020  PENDING_UPGRADE  openstack-operator-0.3.18  Preparing upgrade
    
  3. Roll back the failed release to the previous revision. For example:

    ./helm --host=localhost:44134 rollback openstack-operator 2
    

    Once done, the release will be reconciled.


Storage
[10060] Ceph OSD node removal fails

Fixed in Container Cloud 2.7.0

A Ceph node removal is not being triggered properly after updating the KaasCephCluster custom resource (CR). Both management and managed clusters are affected.

Workaround:

  1. Remove the parameters for a Ceph OSD from the KaasCephCluster CR as described in Operations Guide: Add, remove, or reconfigure Ceph nodes.

  2. Obtain the IDs of the osd and mon services that are located on the old node:

    1. Obtain the UID of the affected machine:

      kubectl get machine <CephOSDNodeName> -n <ManagedClusterProjectName> -o jsonpath='{.metadata.annotations.kaas\.mirantis\.com\/uid}'
      
    2. Export kubeconfig of your managed cluster. For example:

      export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
      
    3. Identify the pods IDs that run the osd and mon services:

      kubectl get pods -o wide -n rook-ceph | grep <affectedMachineUID> | grep -E "mon|osd"
      

      Example of the system response extract:

      rook-ceph-mon-c-7bbc5d757d-5bpws                              1/1  Running    1  6h1m
      rook-ceph-osd-2-58775d5568-5lklw                              1/1  Running    4  44h
      rook-ceph-osd-prepare-705ae6c647cfdac928c63b63e2e2e647-qn4m9  0/1  Completed  0  94s
      

      The pods IDs include the osd or mon services IDs. In the example system response above, the osd ID is 2 and the mon ID is c.

  3. Delete the deployments of the osd and mon services obtained in the previous step:

    kubectl delete deployment rook-ceph-osd(mon)-<ID> -n rook-ceph
    

    For example:

    kubectl delete deployment rook-ceph-mon-c -n rook-ceph
    kubectl delete deployment rook-ceph-osd-2 -n rook-ceph
    
  4. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  5. Rebalance the Ceph OSDs:

    ceph osd out osd(s).ID
    

    Wait for the rebalance to complete.

  6. Rebalance the Ceph data:

    ceph osd purge osd(s).ID
    

    Wait for the Ceph data to rebalance.

  7. Remove the old node from the Ceph OSD tree:

    ceph osd crush rm <NodeName>
    
  8. If the removed node contained mon services, remove them:

    ceph mon rm <monID>
    
[9928] Ceph rebalance during a managed cluster update

Fixed in Container Cloud 2.5.0

During a managed cluster update, Ceph rebalance leading to data loss may occur.

Workaround:

  1. Before updating a managed cluster:

    1. Log in to the ceph-tools pod:

      kubectl -n rook-ceph exec -it <ceph-tools-pod-name> bash
      
    2. Set the noout flag:

      ceph osd set noout
      
  2. Update a managed cluster.

  3. After updating a managed cluster:

    1. Log in to the ceph-tools pod:

      kubectl -n rook-ceph exec -it <ceph-tools-pod-name> bash
      
    2. Unset the noout flag:

      ceph osd unset noout
      
[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.

[10050] Ceph OSD pod is in the CrashLoopBackOff state after disk replacement

If you use a custom BareMetalHostProfile, after disk replacement on a Ceph OSD, the Ceph OSD pod switches to the CrashLoopBackOff state due to the Ceph OSD authorization key failing to be created properly.

Workaround:

  1. Export kubeconfig of your managed cluster. For example:

    export KUBECONFIG=~/Downloads/kubeconfig-test-cluster.yml
    
  2. Log in to the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
    
  3. Delete the authorization key for the failed Ceph OSD:

    ceph auth del osd.<ID>
    
  4. SSH to the node on which the Ceph OSD cannot be created.

  5. Clean up the disk that will be a base for the failed Ceph OSD. For details, see official Rook documentation.

    Note

    Ignore failures of the sgdisk --zap-all $DISK and blkdiscard $DISK commands if any.

  6. On the managed cluster, restart the Rook operator:

    kubectl -n rook-ceph delete pod -l app=rook-ceph-operator
    

Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Addressed issues

The following issues have been addressed in the Mirantis Container Cloud release 2.3.0 and the Cluster releases 5.10.0 and 6.10.0:

  • [8869] Upgraded kind from version 0.3.0 to 0.9.0 and the kindest/node image version from 1.14.2 to 1.18.8 to enhance the Container Cloud performance and prevent compatibility issues.

  • [8220] Fixed the issue with failure to switch the default label from one BareMetalHostProfile to another.

  • [7255] Fixed the issue with slow creation of the OpenStack clients and pools by redesigning as well as increasing efficiency and speed of ceph-controller.

  • [8618] Fixed the issue with missing pools during a Ceph cluster deployment.

  • [8111] Fixed the issue with a Ceph cluster being available after deleting it using the Container Cloud web UI or deleting the KaaSCephCluster object from the Kubernetes namespace using CLI.

  • [8409, 3836] Refactored and stabilized the upgrade procedure to prevent locks during the upgrade operations.

  • [8925] Fixed improper handling of errors in lcm-controller that may lead to its panic.

  • [8361] Fixed the issue with admission-controller allowing addition of duplicated node labels per machine.

  • [8402] Fixed the issue with the AWS provider failing during node labeling with the Observed a panic: “invalid memory address or nil pointer dereference” error if privateIP is not set for a machine.

  • [7673] Moved logs collection of the bootstrap cluster to the /bootstrap subdirectory to prevent unintentional erasure of the management and regional cluster logs.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.3.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.15.4

aws-credentials-controller

1.15.4

Bare metal

baremetal-operator Updated

3.1.6

baremetal-public-api Updated

3.1.6

baremetal-provider Updated

1.15.4

httpd

2.4.46-20201001171500

ironic Updated

ussuri-bionic-20201111180110

ironic-operator Updated

base-bionic-20201106182102

kaas-ipam Updated

20201210175212

local-volume-provisioner

1.0.4-mcp

mariadb

10.4.14-bionic-20200812025059

IAM

iam

1.1.22

iam-controller Updated

1.15.4

keycloak

9.0.0

Container Cloud Updated

admission-controller

1.15.4

byo-credentials-controller

1.15.4

byo-provider

1.15.4

kaas-public-api

1.15.4

kaas-exporter

1.15.4

kaas-ui

1.15.4

lcm-controller

0.2.0-224-g5c413d37

release-controller

1.15.4

OpenStack Updated

openstack-provider

1.15.4

os-credentials-controller

1.15.4

VMWare vSphere Updated

vsphere-provider

1.15.4

vsphere-credentials-controller

1.15.4

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.3.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-3.1.6.tgz

baremetal-public-api Updated

https://binary.mirantis.com/bm/helm/baremetal-public-api-3.1.6.tgz

ironic-python-agent.kernel Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-ussuri-bionic-debug-20201119132200

ironic-python-agent.initramfs Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-ussuri-bionic-debug-20201119132200

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-3.1.6.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator Updated

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20201113171304

httpd

mirantis.azurecr.io/bm/external/httpd:2.4.46-20201001171500

ironic Updated

mirantis.azurecr.io/openstack/ironic:ussuri-bionic-20201111180110

ironic-inspector Updated

mirantis.azurecr.io/openstack/ironic-inspector:ussuri-bionic-20201111180110

ironic-operator Updated

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20201106182102

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20201210175212

mariadb

mirantis.azurecr.io/general/mariadb:10.4.14-bionic-20200812025059


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.15.4.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.15.4.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.15.4.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.15.4.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.15.4.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.15.4.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.15.4.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.15.4.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.15.4.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.15.4.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.15.4.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.15.4.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.15.4.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.15.4.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.15.4.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.15.4.tgz

vsphere-credentials-controller

https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.15.4.tgz

vsphere-provider

https://binary.mirantis.com/core/helm/vsphere-provider-1.15.4.tgz

Docker images Updated

admission-controller

mirantis.azurecr.io/core/admission-controller:1.15.4

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.15.4

aws-credentials-controller

mirantis.azurecr.io/core/aws-credentials-controller:1.15.4

byo-cluster-api-controller

mirantis.azurecr.io/core/byo-cluster-api-controller:1.15.4

byo-credentials-controller

mirantis.azurecr.io/core/byo-credentials-controller:1.15.4

cluster-api-provider-baremetal

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.15.4

frontend

mirantis.azurecr.io/core/frontend:1.15.4

iam-controller

mirantis.azurecr.io/core/iam-controller:1.15.4

lcm-controller

mirantis.azurecr.io/core/lcm-controller:v0.2.0-224-g5c413d37

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.15.4

os-credentials-controller

mirantis.azurecr.io/core/os-credentials-controller:1.15.4

release-controller

mirantis.azurecr.io/core/release-controller:1.15.4

vsphere-cluster-api-controller

mirantis.azurecr.io/core/vsphere-api-controller:1.15.4

vsphere-credentials-controller

mirantis.azurecr.io/core/vsphere-credentials-controller:1.15.4


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin

iamctl-linux

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-linux

iamctl-darwin

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-darwin

iamctl-windows

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-windows

Helm charts

iam Updated

http://binary.mirantis.com/iam/helm/iam-1.1.22.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.3.tgz

keycloak-proxy Updated

http://binary.mirantis.com/core/helm/keycloak_proxy-1.14.3.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.3.18

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.3.18

kubernetes-entrypoint

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/iam/external/mariadb:10.2.18

keycloak Updated

mirantis.azurecr.io/iam/keycloak:0.3.19

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

2.2.0

This section outlines release notes for the Mirantis Container Cloud GA release 2.2.0. This release introduces support for the Cluster release 5.9.0 that is based on Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18. This release also introduces support for the Cluster release 6.8.1 that introduces the support of the Mirantis OpenStack for Kubernetes (MOS) product.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.2.0. For the list of enhancements in the Cluster release 5.9.0 and Cluster release 6.8.1 introduced by the Container Cloud release 2.2.0, see 5.9.0 and 6.8.1.


Support for VMWare vSphere provider on RHEL

TECHNICAL PREVIEW

Introduced the Technology Preview support for the VMWare vSphere cloud provider on RHEL, including support for creation and operating of managed clusters using the Container Cloud web UI.

Deployment of an additional regional vSphere-based cluster or attaching an existing Mirantis Kubernetes Engine (MKE) cluster to a vSphere-based management cluster is on the development stage and will be announced in one of the following Container Cloud releases.

Note

For the Technology Preview feature definition, refer to the Container Cloud Reference Architecture.


Kernel parameters management through BareMetalHostProfile

Implemented the API for managing kernel parameters typically managed by sysctl for bare metal hosts through the BareMetalHost and BareMetalHostProfile objects fields.


Support of multiple subnets per cluster

Implemented support of multiple subnets per a Container Cloud cluster with an ability to specify a different network type for each subnet. Introduced the SubnetPool object that allows for automatic creation of the Subnet objects. Also, added the L3Layout section to L2Template.spec. The L3Layout configuration allows defining the subnets scopes to be used and to enable auto-creation of subnets from a subnet pool.


Optimization of the Container Cloud logs collection

Optimized user experience with the Container Cloud resources logs collection:

  • Added a separate file with a human-readable table that contains information about cluster events

  • Implemented collecting of system logs from cluster nodes


Container Cloud API documentation for bare metal

On top of continuous improvements delivered to the existing Container Cloud guides, added the Mirantis Container Cloud API section to the Operations Guide. This section is intended only for advanced Infrastructure Operators who are familiar with Kubernetes Cluster API.

Currently, this section contains descriptions and examples of the Container Cloud API resources for the bare metal cloud provider. The API documentation for the OpenStack, AWS, and VMWare vSphere API resources will be added in the upcoming Container Cloud releases.

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.2.0 including the Cluster release 5.9.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[8013] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

Bare metal
[6988] LVM fails to deploy if the volume group name already exists

Fixed in Container Cloud 2.5.0

During a management or managed cluster deployment, LVM cannot be deployed on a new disk if an old volume group with the same name already exists on the target hardware node but on the different disk.

Workaround:

In the bare metal host profile specific to your hardware configuration, add the wipe: true parameter to the device that fails to be deployed. For the procedure details, see Operations Guide: Create a custom host profile.

[7655] Wrong status for an incorrectly configured L2 template

If an L2 template is configured incorrectly, a bare metal cluster is deployed successfully but with the runtime errors in the IpamHost object.

Workaround:

If you suspect that the machine is not working properly because of incorrect network configuration, verify the status of the corresponding IpamHost object. Inspect the l2RenderResult and ipAllocationResult object fields for error messages.

[8560] Manual deletion of BareMetalHost leads to its silent removal

Fixed in Container Cloud 2.5.0

If BareMetalHost is manually removed from a managed cluster, it is silently removed without a power-off and deprovision that leads to a managed cluster failures.

Workaround:

Do not manually delete a BareMetalHost that has the Provisioned status.


IAM
[2757] IAM fails to start during management cluster deployment

Fixed in Container Cloud 2.4.0

During a management cluster deployment, IAM fails to start with the IAM pods being in the CrashLoopBackOff status.

Workaround:

  1. Log in to the bootstrap node.

  2. Remove the iam-mariadb-state configmap:

    kubectl delete cm -n kaas iam-mariadb-state
    
  3. Manually delete the mariadb pods:

    kubectl delete po -n kaas mariadb-server-{0,1,2}
    

    Wait for the pods to start. If the mariadb pod does not start with the connection to peer timed out exception, repeat the step 2.

  4. Obtain the MariaDB database admin password:

    kubectl get secrets -n kaas mariadb-dbadmin-password \
    -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d ; echo
    
  5. Log in to MariaDB:

    kubectl exec -it -n kaas mariadb-server-0 -- bash -c 'mysql -uroot -p<mysqlDbadminPassword>'
    

    Substitute <mysqlDbadminPassword> with the corresponding value obtained in the previous step.

  6. Run the following command:

    DROP DATABASE IF EXISTS keycloak;
    
  7. Manually delete the Keycloak pods:

    kubectl delete po -n kaas iam-keycloak-{0,1,2}
    

Storage
[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.


Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Addressed issues

The following issues have been addressed in the Mirantis Container Cloud release 2.2.0 including the Cluster release 5.9.0:

  • [8012] Fixed the issue with helm-controller pod being stuck in the CrashLoopBackOff state after reattaching of a Mirantis Kubernetes Engine (MKE) cluster.

  • [7131] Fixed the issue with the deployment of a managed cluster failing during the Ceph Monitor or Manager deployment.

  • [6164] Fixed the issue with the number of placement groups (PGs) per Ceph OSD being too small and the Ceph cluster having the HEALTH_WARN status.

  • [8302] Fixed the issue with deletion of a regional cluster leading to the deletion of the related management cluster.

  • [7722] Fixed the issue with the Internal Server Error or similar errors appearing in the HelmBundle controller logs after bootstrapping the management cluster.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.2.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.14.0

aws-credentials-controller

1.14.0

Bare metal

baremetal-operator Updated

3.1.3

baremetal-public-api Updated

3.1.3

baremetal-provider Updated

1.14.0

httpd

2.4.46-20201001171500

ironic Updated

ussuri-bionic-20201021180016

ironic-operator Updated

base-bionic-20201023172943

kaas-ipam Updated

20201026094912

local-volume-provisioner

1.0.4-mcp

mariadb

10.4.14-bionic-20200812025059

IAM

iam Updated

1.1.22

iam-controller Updated

1.14.0

keycloak

9.0.0

Container Cloud Updated

admission-controller

1.14.0

byo-credentials-controller

1.14.0

byo-provider

1.14.3

kaas-public-api

1.14.0

kaas-exporter

1.14.0

kaas-ui

1.14.2

lcm-controller

0.2.0-178-g8cc488f8

release-controller

1.14.0

OpenStack Updated

openstack-provider

1.14.0

os-credentials-controller

1.14.0

VMWare vSphere New

vsphere-provider

1.14.1

vsphere-credentials-controller

1.14.1

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.2.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

Target system image (ubuntu-bionic)

https://binary.mirantis.com/bm/bin/efi/ubuntu/qcow2-bionic-debug-20200730084816

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-3.1.3.tgz

baremetal-public-api Updated

https://binary.mirantis.com/bm/helm/baremetal-public-api-3.1.3.tgz

ironic-python-agent.kernel Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-ussuri-bionic-debug-20201022084817

ironic-python-agent.initramfs Updated

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-ussuri-bionic-debug-20201022084817

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-3.1.3.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator Updated

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20201028131325

httpd

mirantis.azurecr.io/bm/external/httpd:2.4.46-20201001171500

ironic Updated

mirantis.azurecr.io/openstack/ironic:ussuri-bionic-20201021180016

ironic-inspector Updated

mirantis.azurecr.io/openstack/ironic-inspector:ussuri-bionic-20201021180016

ironic-operator Updated

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20201023172943

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20201026094912

mariadb

mirantis.azurecr.io/general/mariadb:10.4.14-bionic-20200812025059


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.14.0.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.14.0.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.14.0.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.14.0.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.14.0.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.14.0.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.14.0.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.14.3.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.14.0.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.14.0.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.14.0.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.14.2.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.14.0.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.14.0.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.14.0.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.14.0.tgz

vsphere-credentials-controller New

https://binary.mirantis.com/core/helm/vsphere-credentials-controller-1.14.1.tgz

vsphere-provider New

https://binary.mirantis.com/core/helm/vsphere-provider-1.14.1.tgz

Docker images for Container Cloud deployment

admission-controller Updated

mirantis.azurecr.io/core/admission-controller:1.14.0

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.14.0

aws-credentials-controller Updated

mirantis.azurecr.io/core/aws-credentials-controller:1.14.0

byo-cluster-api-controller

mirantis.azurecr.io/core/byo-cluster-api-controller:1.14.3

byo-credentials-controller Updated

mirantis.azurecr.io/core/byo-credentials-controller:1.14.0

cluster-api-provider-baremetal Updated

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.14.0

frontend

mirantis.azurecr.io/core/frontend:1.14.2

iam-controller Updated

mirantis.azurecr.io/core/iam-controller:1.14.0

lcm-controller Updated

mirantis.azurecr.io/core/lcm-controller:v0.2.0-178-g8cc488f8

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.14.0

os-credentials-controller Updated

mirantis.azurecr.io/core/os-credentials-controller:1.14.0

release-controller Updated

mirantis.azurecr.io/core/release-controller:1.14.0

vsphere-cluster-api-controller New

mirantis.azurecr.io/core/vsphere-api-controller:1.14.1

vsphere-credentials-controller New

mirantis.azurecr.io/core/vsphere-credentials-controller:1.14.1


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-268-3cf7f17-darwin

iamctl-linux

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-linux

iamctl-darwin

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-darwin

iamctl-windows

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-windows

Helm charts

iam Updated

http://binary.mirantis.com/iam/helm/iam-1.1.22.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

keycloak-proxy Updated

http://binary.mirantis.com/core/helm/keycloak_proxy-1.14.3.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.3.18

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.3.18

kubernetes-entrypoint

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/iam/external/mariadb:10.2.18

keycloak Updated

mirantis.azurecr.io/iam/keycloak:0.3.19

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

2.1.0

This section outlines release notes for the Mirantis Container Cloud GA release 2.1.0. This release introduces support for the Cluster release 5.8.0 that is based on Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18.

Enhancements

This section outlines new features and enhancements introduced in the Mirantis Container Cloud release 2.1.0. For the list of enhancements in the Cluster release 5.8.0 introduced by the KaaS release 2.1.0, see 5.8.0.


Node labeling for machines

Implemented the possibility to assign labels to specific machines with dedicated system and hardware resources through the Container Cloud web UI. For example, you can label the StackLight nodes that run Elasticsearch and require more resources than a standard node to run the StackLight components services on the dedicated nodes. You can label a machine before or after it is deployed. The list of available labels is taken from the current Cluster release.

Node labeling greatly improves cluster performance and prevents pods from being quickly exhausted.


AWS resources discovery in Container Cloud web UI

Improved the user experience during a managed cluster creation using the Container Cloud web UI by implementing drop-down menus with available supported values for the following AWS resources:

  • AWS region

  • AWS AMI ID

  • AWS instance type

To apply the feature to existing deployments, update the IAM policies for AWS as described in Apply updates to the AWS-based management clusters.


Credentials statuses for OpenStack and AWS

Implemented the following statuses for the OpenStack-based and AWS-based credentials in the Container Cloud web UI:

  • Ready

    Credentials are valid and ready to be used for a managed cluster creation.

  • In Use

    Credentials are being used by a managed cluster.

  • Error

    Credentials are invalid. You can hover over the Error status to determine the reason of the issue.

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.1.0.

Note

This section also outlines still valid known issues from previous Container Cloud releases.


AWS
[8013] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

Bare metal
[6988] LVM fails to deploy if the volume group name already exists

Fixed in Container Cloud 2.5.0

During a management or managed cluster deployment, LVM cannot be deployed on a new disk if an old volume group with the same name already exists on the target hardware node but on the different disk.

Workaround:

In the bare metal host profile specific to your hardware configuration, add the wipe: true parameter to the device that fails to be deployed. For the procedure details, see Operations Guide: Create a custom host profile.


IAM
[2757] IAM fails to start during management cluster deployment

Fixed in Container Cloud 2.4.0

During a management cluster deployment, IAM fails to start with the IAM pods being in the CrashLoopBackOff status.

Workaround:

  1. Log in to the bootstrap node.

  2. Remove the iam-mariadb-state configmap:

    kubectl delete cm -n kaas iam-mariadb-state
    
  3. Manually delete the mariadb pods:

    kubectl delete po -n kaas mariadb-server-{0,1,2}
    

    Wait for the pods to start. If the mariadb pod does not start with the connection to peer timed out exception, repeat the step 2.

  4. Obtain the MariaDB database admin password:

    kubectl get secrets -n kaas mariadb-dbadmin-password \
    -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d ; echo
    
  5. Log in to MariaDB:

    kubectl exec -it -n kaas mariadb-server-0 -- bash -c 'mysql -uroot -p<mysqlDbadminPassword>'
    

    Substitute <mysqlDbadminPassword> with the corresponding value obtained in the previous step.

  6. Run the following command:

    DROP DATABASE IF EXISTS keycloak;
    
  7. Manually delete the Keycloak pods:

    kubectl delete po -n kaas iam-keycloak-{0,1,2}
    

Storage
[6164] Small number of PGs per Ceph OSD

Fixed in 2.2.0

After deploying a managed cluster with Ceph, the number of placement groups (PGs) per Ceph OSD may be too small and the Ceph cluster may have the HEALTH_WARN status:

health: HEALTH_WARN
        too few PGs per OSD (3 < min 30)

The workaround is to enable the PG balancer to properly manage the number of PGs:

kexec -it $(k get pod -l "app=rook-ceph-tools" --all-namespaces -o jsonpath='{.items[0].metadata.name}') -n rook-ceph bash
ceph mgr module enable pg_autoscaler
[7131] rook-ceph-mgr fails during managed cluster deployment

Fixed in 2.2.0

Occasionally, the deployment of a managed cluster may fail during the Ceph Monitor or Manager deployment. In this case, the Ceph cluster may be down and and a stack trace similar to the following one may be present in Ceph Manager logs:

kubectl -n rook-ceph logs rook-ceph-mgr-a-c5dc846f8-k68rs

/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/mon/MonMap.h: In function 'void MonMap::add(const mon_info_t&)' thread 7fd3d3744b80 time 2020-09-03 10:16:46.586388
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/mon/MonMap.h: 195: FAILED ceph_assert(addr_mons.count(a) == 0)
ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fd3ca9b2875]
2: (()+0x253a3d) [0x7fd3ca9b2a3d]
3: (MonMap::add(mon_info_t const&)+0x80) [0x7fd3cad49190]
4: (MonMap::add(std::string const&, entity_addrvec_t const&, int)+0x110) [0x7fd3cad493a0]
5: (MonMap::init_with_ips(std::string const&, bool, std::string const&)+0xc9) [0x7fd3cad43849]
6: (MonMap::build_initial(CephContext*, bool, std::ostream&)+0x314) [0x7fd3cad45af4]
7: (MonClient::build_initial_monmap()+0x130) [0x7fd3cad2e140]
8: (MonClient::get_monmap_and_config()+0x5f) [0x7fd3cad365af]
9: (global_pre_init(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int)+0x524) [0x55ce86711444]
10: (global_init(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, char const*, bool)+0x76) [0x55ce86711b56]
11: (main()+0x136) [0x55ce864ff9a6]
12: (__libc_start_main()+0xf5) [0x7fd3c6e73555]
13: (()+0xfc010) [0x55ce86505010]

The workaround is to start the managed cluster deployment from scratch.

[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.


Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Addressed issues

In the Mirantis Container Cloud release 2.1.0, the following issues have been addressed:

  • [7281] Fixed the issue with a management cluster bootstrap script failing if there was a space in the PATH environment variable.

  • [7205] Fixed the issue with some cluster objects being stuck during deletion of an AWS-based managed cluster due to unresolved VPC dependencies.

  • [7304] Fixed the issue with failure to reattach a Mirantis Kubernetes Engine (MKE) cluster with the same name.

  • [7101] Fixed the issue with the monitoring of Ceph and Ironic being enabled when Ceph and Ironic are disabled on the baremetal-based clusters.

  • [7324] Fixed the issue with the monitoring of Ceph being disabled on the baremetal-based managed clusters due to the missing provider: BareMetal parameter.

  • [7180] Fixed the issue with lcm-controller periodically failing with the invalid memory address or nil pointer dereference runtime error.

  • [7251] Fixed the issue with setting up the OIDC integration on the MKE side.

  • [7326] Fixed the issue with the missing entry for the host itself in etc/hosts causing failure of services that require node FQDN.

  • [6989] Fixed the issue with baremetal-operator ignoring the clean failed provisioning state if a node fails to deploy on a baremetal-based managed cluster.

  • [7231] Fixed the issue with the baremetal-provider pod not restarting after the ConfigMap changes and causing the telemeter-client pod to fail during deployment.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.1.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Container Cloud release components versions

Component

Application/Service

Version

AWS Updated

aws-provider

1.12.2

aws-credentials-controller

1.12.2

Bare metal

baremetal-operator Updated

3.1.0

baremetal-public-api New

3.1.0

baremetal-provider Updated

1.12.2

httpd Updated

2.4.46-20201001171500

ironic

train-bionic-20200803180020

ironic-operator

base-bionic-20200805144858

kaas-ipam Updated

20201007180518

local-volume-provisioner

1.0.4-mcp

mariadb Updated

10.4.14-bionic-20200812025059

IAM

iam Updated

1.1.18

iam-controller Updated

1.12.2

keycloak

9.0.0

Container Cloud Updated

admission-controller

1.12.3

byo-credentials-controller

1.12.2

byo-provider

1.12.2

kaas-public-api

1.12.2

kaas-exporter

1.12.2

kaas-ui

1.12.2

lcm-controller

0.2.0-169-g5668304d

release-controller

1.12.2

OpenStack Updated

openstack-provider

1.12.2

os-credentials-controller

1.12.2

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.1.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Bare metal artifacts

Artifact

Component

Path

Binaries

Target system image (ubuntu-bionic)

https://binary.mirantis.com/bm/bin/efi/ubuntu/qcow2-bionic-debug-20200730084816

baremetal-operator Updated

https://binary.mirantis.com/bm/helm/baremetal-operator-3.1.0.tgz

baremetal-public-api New

https://binary.mirantis.com/bm/helm/baremetal-public-api-3.1.0.tgz

ironic-python-agent.kernel

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-train-bionic-debug-20200730084816

ironic-python-agent.initramfs

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-train-bionic-debug-20200730084816

kaas-ipam Updated

https://binary.mirantis.com/bm/helm/kaas-ipam-3.1.0.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator Updated

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20201005150946

httpd Updated

mirantis.azurecr.io/bm/external/httpd:2.4.46-20201001171500

ironic

mirantis.azurecr.io/openstack/ironic:train-bionic-20200803180020

ironic-inspector

mirantis.azurecr.io/openstack/ironic-inspector:train-bionic-20200803180020

ironic-operator

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20200805144858

kaas-ipam Updated

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20201007180518

mariadb Updated

mirantis.azurecr.io/general/mariadb:10.4.14-bionic-20200812025059


Core artifacts

Artifact

Component

Path

Bootstrap tarball Updated

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.12.2.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.12.2.tar.gz

Helm charts Updated

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.12.3.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.12.2.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.12.2.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.12.2.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.12.2.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.12.2.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.12.2.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.12.2.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.12.2.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.12.2.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.12.2.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.12.2.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.12.2.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.12.2.tgz

Docker images for Container Cloud deployment Updated

admission-controller

mirantis.azurecr.io/core/admission-controller:1.12.3

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.12.2

byo-cluster-api-controller

mirantis.azurecr.io/core/byo-cluster-api-controller:1.12.2

aws-credentials-controller

mirantis.azurecr.io/core/aws-credentials-controller:1.12.2

byo-credentials-controller

mirantis.azurecr.io/core/byo-credentials-controller:1.12.2

cluster-api-provider-baremetal

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.12.2

frontend

mirantis.azurecr.io/core/frontend:1.12.2

iam-controller

mirantis.azurecr.io/core/iam-controller:1.12.2

lcm-controller

mirantis.azurecr.io/core/lcm-controller:v0.2.0-169-g5668304d

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.12.2

os-credentials-controller

mirantis.azurecr.io/core/os-credentials-controller:1.12.2

release-controller

mirantis.azurecr.io/core/release-controller:1.12.2


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-236-9cea809-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-236-9cea809-darwin

iamctl-linux

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-linux

iamctl-darwin

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-darwin

iamctl-windows

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-windows

Helm charts

iam Updated

http://binary.mirantis.com/iam/helm/iam-1.1.18.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

keycloak-proxy Updated

http://binary.mirantis.com/core/helm/keycloak_proxy-1.12.2.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.3.18

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.3.18

kubernetes-entrypoint

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/iam/external/mariadb:10.2.18

keycloak

mirantis.azurecr.io/iam/keycloak:0.3.18

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

Apply updates to the AWS-based management clusters

To complete the AWS-based management cluster upgrade to version 2.1.0, manually update the IAM policies for AWS before updating your AWS-based managed clusters.

To update the IAM policies for AWS:

  1. Choose from the following options:

    • Update the IAM policies using get_container_cloud.sh:

      1. On any local machine, download and run the latest version of the Container Cloud bootstrap script:

        wget https://binary.mirantis.com/releases/get_container_cloud.sh
        
        chmod 0755 get_container_cloud.sh
        
        ./get_container_cloud.sh
        
      2. Change the directory to the kaas-bootstrap folder created by the get_container_cloud.sh script.

      3. Export the following parameters by adding the corresponding values for the AWS admin credentials:

        export AWS_SECRET_ACCESS_KEY=XXXXXXX
        export AWS_ACCESS_KEY_ID=XXXXXXX
        export AWS_DEFAULT_REGION=us-east-2
        
      4. Update the AWS CloudFormation template for IAM policy:

        ./kaas bootstrap aws policy
        
    • Update the IAM policies using the AWS Management Console:

      1. Log in to your AWS Management Console.

      2. Verify that the controllers.cluster-api-provider-aws.kaas.mirantis.com role or another AWS role that you use for Container Cloud users contains the following permissions:

        "ec2:DescribeRegions", "ec2:DescribeInstanceTypes"
        

        Otherwise, add these permissions manually.

  2. Proceed to updating your AWS-based managed clusters as described in Operations Guide: Update a managed cluster.

2.0.0

This section outlines release notes for the initial Mirantis Container Cloud GA release 2.0.0. This release introduces support for the Cluster release 5.7.0 that is based on Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18.

Known issues

This section lists known issues with workarounds for the Mirantis Container Cloud release 2.0.0.


AWS
[8013] Managed cluster deployment requiring PVs may fail

On a management cluster with multiple AWS-based managed clusters, some clusters fail to complete the deployments that require persistent volumes (PVs), for example, Elasticsearch. Some of the affected pods get stuck in the Pending state with the pod has unbound immediate PersistentVolumeClaims and node(s) had volume node affinity conflict errors.

Warning

The workaround below applies to HA deployments where data can be rebuilt from replicas. If you have a non-HA deployment, back up any existing data before proceeding, since all data will be lost while applying the workaround.

Workaround:

  1. Obtain the persistent volume claims related to the storage mounts of the affected pods:

    kubectl get pod/<pod_name1> pod/<pod_name2> \
    -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}'
    

    Note

    In the command above and in the subsequent steps, substitute the parameters enclosed in angle brackets with the corresponding values.

  2. Delete the affected Pods and PersistentVolumeClaims to reschedule them: For example, for StackLight:

    kubectl -n stacklight delete \
    
      pod/<pod_name1> pod/<pod_name2> ...
      pvc/<pvc_name2> pvc/<pvc_name2> ...
    

Bare metal
[6988] LVM fails to deploy if the volume group name already exists

Fixed in Container Cloud 2.5.0

During a management or managed cluster deployment, LVM cannot be deployed on a new disk if an old volume group with the same name already exists on the target hardware node but on the different disk.

Workaround:

In the bare metal host profile specific to your hardware configuration, add the wipe: true parameter to the device that fails to be deployed. For the procedure details, see Operations Guide: Create a custom host profile.


IAM
[2757] IAM fails to start during management cluster deployment

Fixed in Container Cloud 2.4.0

During a management cluster deployment, IAM fails to start with the IAM pods being in the CrashLoopBackOff status.

Workaround:

  1. Log in to the bootstrap node.

  2. Remove the iam-mariadb-state configmap:

    kubectl delete cm -n kaas iam-mariadb-state
    
  3. Manually delete the mariadb pods:

    kubectl delete po -n kaas mariadb-server-{0,1,2}
    

    Wait for the pods to start. If the mariadb pod does not start with the connection to peer timed out exception, repeat the step 2.

  4. Obtain the MariaDB database admin password:

    kubectl get secrets -n kaas mariadb-dbadmin-password \
    -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d ; echo
    
  5. Log in to MariaDB:

    kubectl exec -it -n kaas mariadb-server-0 -- bash -c 'mysql -uroot -p<mysqlDbadminPassword>'
    

    Substitute <mysqlDbadminPassword> with the corresponding value obtained in the previous step.

  6. Run the following command:

    DROP DATABASE IF EXISTS keycloak;
    
  7. Manually delete the Keycloak pods:

    kubectl delete po -n kaas iam-keycloak-{0,1,2}
    

StackLight
[7101] Monitoring of disabled components

Fixed in 2.1.0

On the baremetal-based clusters, the monitoring of Ceph and Ironic is enabled when Ceph and Ironic are disabled. The issue with Ceph relates to both management or managed clusters, the issue with Ironic relates to managed clusters only.

Workaround:

  1. Open the StackLight configuration manifest as described in Operations Guide: Configure StackLight.

  2. Add the following parameter to the StackLight helmReleases values of the Cluster object to explicitly disable the required component monitoring:

    • For Ceph:

      helmReleases:
        - name: stacklight
          values:
            ...
            ceph:
              disabledOnBareMetal: true
            ...
      
    • For Ironic:

      helmReleases:
        - name: stacklight
          values:
            ...
            ironic:
              disabledOnBareMetal: true
            ...
      
[7324] Ceph monitoring disabled

Fixed in 2.1.0

Ceph monitoring may be disabled on the baremetal-based managed clusters due to a missing provider: BareMetal parameter.

Workaround:

  1. Open the StackLight configuration manifest as described in Operations Guide: Configure StackLight.

  2. Add the provider: BareMetal parameter to the StackLight helmReleases values of the Cluster object:

    spec:
      providerSpec:
        value:
          helmReleases:
          - name: stacklight
            values:
              ...
              provider: BareMetal
              ...
    

Storage
[6164] Small number of PGs per Ceph OSD

Fixed in 2.2.0

After deploying a managed cluster with Ceph, the number of placement groups (PGs) per Ceph OSD may be too small and the Ceph cluster may have the HEALTH_WARN status:

health: HEALTH_WARN
        too few PGs per OSD (3 < min 30)

The workaround is to enable the PG balancer to properly manage the number of PGs:

kexec -it $(k get pod -l "app=rook-ceph-tools" --all-namespaces -o jsonpath='{.items[0].metadata.name}') -n rook-ceph bash
ceph mgr module enable pg_autoscaler
[7131] rook-ceph-mgr fails during managed cluster deployment

Fixed in 2.2.0

Occasionally, the deployment of a managed cluster may fail during the Ceph Monitor or Manager deployment. In this case, the Ceph cluster may be down and and a stack trace similar to the following one may be present in Ceph Manager logs:

kubectl -n rook-ceph logs rook-ceph-mgr-a-c5dc846f8-k68rs

/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/mon/MonMap.h: In function 'void MonMap::add(const mon_info_t&)' thread 7fd3d3744b80 time 2020-09-03 10:16:46.586388
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/mon/MonMap.h: 195: FAILED ceph_assert(addr_mons.count(a) == 0)
ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fd3ca9b2875]
2: (()+0x253a3d) [0x7fd3ca9b2a3d]
3: (MonMap::add(mon_info_t const&)+0x80) [0x7fd3cad49190]
4: (MonMap::add(std::string const&, entity_addrvec_t const&, int)+0x110) [0x7fd3cad493a0]
5: (MonMap::init_with_ips(std::string const&, bool, std::string const&)+0xc9) [0x7fd3cad43849]
6: (MonMap::build_initial(CephContext*, bool, std::ostream&)+0x314) [0x7fd3cad45af4]
7: (MonClient::build_initial_monmap()+0x130) [0x7fd3cad2e140]
8: (MonClient::get_monmap_and_config()+0x5f) [0x7fd3cad365af]
9: (global_pre_init(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int)+0x524) [0x55ce86711444]
10: (global_init(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, char const*, bool)+0x76) [0x55ce86711b56]
11: (main()+0x136) [0x55ce864ff9a6]
12: (__libc_start_main()+0xf5) [0x7fd3c6e73555]
13: (()+0xfc010) [0x55ce86505010]

The workaround is to start the managed cluster deployment from scratch.

[7073] Cannot automatically remove a Ceph node

When removing a worker node, it is not possible to automatically remove a Ceph node. The workaround is to manually remove the Ceph node from the Ceph cluster as described in Operations Guide: Add, remove, or reconfigure Ceph nodes before removing the worker node from your deployment.


Bootstrap
[7281] Space in PATH causes failure of bootstrap process

Fixed in 2.1.0

A management cluster bootstrap script fails if there is a space in the PATH environment variable. As a workaround, before running the bootstrap.sh script, verify that there are no spaces in the PATH environment variable.


Container Cloud web UI
[249] A newly created project does not display in the Container Cloud web UI

A project that is newly created in the Container Cloud web UI does not display in the Projects list even after refreshing the page. The issue occurs due to the token missing the necessary role for the new project. As a workaround, relogin to the Container Cloud web UI.

Components versions

The following table lists the major components and their versions of the Mirantis Container Cloud release 2.0.0.

Container Cloud release components versions

Component

Application/Service

Version

AWS

aws-provider

1.10.12

aws-credentials-controller

1.10.12

Bare metal

baremetal-operator

3.0.7

baremetal-provider

1.10.12

httpd

2.4.43-20200710111500

ironic

train-bionic-20200803180020

ironic-operator

base-bionic-20200805144858

kaas-ipam

20200807130953

local-volume-provisioner

1.0.4-mcp

mariadb

10.4.12-bionic-20200803130834

IAM

iam

1.1.16

iam-controller

1.10.12

keycloak

9.0.0

Container Cloud

admission-controller

1.10.12

byo-credentials-controller

1.10.12

byo-provider

1.10.12

kaas-public-api

1.10.12

kaas-exporter

1.10.12

kaas-ui

1.10.12

lcm-controller

0.2.0-149-g412c5a05

release-controller

1.10.12

OpenStack

openstack-provider

1.10.12

os-credentials-controller

1.10.12

Artifacts

This section lists the components artifacts of the Mirantis Container Cloud release 2.0.0.


Bare metal artifacts

Artifact

Component

Path

Binaries

Target system image (ubuntu-bionic)

https://binary.mirantis.com/bm/bin/efi/ubuntu/qcow2-bionic-debug-20200730084816

baremetal-operator

https://binary.mirantis.com/bm/helm/baremetal-operator-3.0.7.tgz

ironic-python-agent.kernel

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/kernel-train-bionic-debug-20200730084816

ironic-python-agent.initramfs

https://binary.mirantis.com/bm/bin/ironic/ipa/ubuntu/initramfs-train-bionic-debug-20200730084816

kaas-ipam

https://binary.mirantis.com/bm/helm/kaas-ipam-3.0.7.tgz

local-volume-provisioner

https://binary.mirantis.com/bm/helm/local-volume-provisioner-1.0.4-mcp.tgz

Docker images

baremetal-operator

mirantis.azurecr.io/bm/baremetal-operator:base-bionic-20200812172956

httpd

mirantis.azurecr.io/bm/external/httpd:2.4.43-20200710111500

ironic

mirantis.azurecr.io/openstack/ironic:train-bionic-20200803180020

ironic-inspector

mirantis.azurecr.io/openstack/ironic-inspector:train-bionic-20200803180020

ironic-operator

mirantis.azurecr.io/bm/ironic-operator:base-bionic-20200805144858

kaas-ipam

mirantis.azurecr.io/bm/kaas_ipam:base-bionic-20200807130953

mariadb

mirantis.azurecr.io/general/mariadb:10.4.12-bionic-20200803130834


Core components artifacts

Artifact

Component

Path

Bootstrap tarball

bootstrap-linux

https://binary.mirantis.com/core/bin/bootstrap-linux-1.10.12.tar.gz

bootstrap-darwin

https://binary.mirantis.com/core/bin/bootstrap-darwin-1.10.12.tar.gz

Helm charts

admission-controller

https://binary.mirantis.com/core/helm/admission-controller-1.10.12.tgz

aws-credentials-controller

https://binary.mirantis.com/core/helm/aws-credentials-controller-1.10.12.tgz

aws-provider

https://binary.mirantis.com/core/helm/aws-provider-1.10.12.tgz

baremetal-provider

https://binary.mirantis.com/core/helm/baremetal-provider-1.10.12.tgz

byo-credentials-controller

https://binary.mirantis.com/core/helm/byo-credentials-controller-1.10.12.tgz

byo-provider

https://binary.mirantis.com/core/helm/byo-provider-1.10.12.tgz

iam-controller

https://binary.mirantis.com/core/helm/iam-controller-1.10.12.tgz

kaas-exporter

https://binary.mirantis.com/core/helm/kaas-exporter-1.10.12.tgz

kaas-public-api

https://binary.mirantis.com/core/helm/kaas-public-api-1.10.12.tgz

kaas-ui

https://binary.mirantis.com/core/helm/kaas-ui-1.10.12.tgz

lcm-controller

https://binary.mirantis.com/core/helm/lcm-controller-1.10.12.tgz

openstack-provider

https://binary.mirantis.com/core/helm/openstack-provider-1.10.12.tgz

os-credentials-controller

https://binary.mirantis.com/core/helm/os-credentials-controller-1.10.12.tgz

release-controller

https://binary.mirantis.com/core/helm/release-controller-1.10.12.tgz

Docker images for Container Cloud deployment

aws-cluster-api-controller

mirantis.azurecr.io/core/aws-cluster-api-controller:1.10.12

aws-credentials-controller

mirantis.azurecr.io/core/aws-credentials-controller:1.10.12

byo-cluster-api-controller

mirantis.azurecr.io/core/byo-cluster-api-controller:1.10.12

byo-credentials-controller

mirantis.azurecr.io/core/byo-credentials-controller:1.10.12

cluster-api-provider-baremetal

mirantis.azurecr.io/core/cluster-api-provider-baremetal:1.10.12

frontend

mirantis.azurecr.io/core/frontend:1.10.12

iam-controller

mirantis.azurecr.io/core/iam-controller:1.10.12

lcm-controller

mirantis.azurecr.io/core/lcm-controller:v0.2.0-149-g412c5a05

openstack-cluster-api-controller

mirantis.azurecr.io/core/openstack-cluster-api-controller:1.10.12

os-credentials-controller

mirantis.azurecr.io/core/os-credentials-controller:1.10.12

release-controller

mirantis.azurecr.io/core/release-controller:1.10.12


IAM artifacts

Artifact

Component

Path

Binaries

hash-generate-linux

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-236-9cea809-linux

hash-generate-darwin

http://binary.mirantis.com/iam/bin/hash-generate-0.0.1-236-9cea809-darwin

iamctl-linux

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-linux

iamctl-darwin

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-darwin

iamctl-windows

http://binary.mirantis.com/iam/bin/iamctl-0.3.18-windows

Helm charts

iam

http://binary.mirantis.com/iam/helm/iam-1.1.16.tgz

iam-proxy

http://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

keycloak-proxy

http://binary.mirantis.com/core/helm/keycloak_proxy-1.10.12.tgz

Docker images

api

mirantis.azurecr.io/iam/api:0.3.18

auxiliary

mirantis.azurecr.io/iam/auxiliary:0.3.18

kubernetes-entrypoint

mirantis.azurecr.io/iam/external/kubernetes-entrypoint:v0.3.1

mariadb

mirantis.azurecr.io/iam/external/mariadb:10.2.18

keycloak

mirantis.azurecr.io/iam/keycloak:0.3.18

keycloak-gatekeeper

mirantis.azurecr.io/iam/keycloak-gatekeeper:6.0.1

Cluster releases (managed)

This section outlines the release notes for the Mirantis Container Cloud Cluster releases that are supported by specific Container Cloud releases. For details about the Container Cloud releases, see: Mirantis Container Cloud releases.

6.14.0

The Cluster release 6.14.0 is introduced in the Mirantis Container Cloud release 2.7.0. This Cluster release is based on the Cluster release 5.14.0.

The Cluster release 6.14.0 supports:

For the list of resolved issues, refer to the Container Cloud releases 2.6.0 and 2.7.0 sections. For the list of known issues, refer to the Container Cloud releases 2.7.0.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 6.14.0.


StackLight logging levels

Significantly enhanced the StackLight log collection mechanism to avoid collecting and keeping an excessive amount of log messages when it is not essential. Now, during or after deployment of StackLight, you can select one of the 9 available logging levels depending on the required severity. The default logging level is INFO.


Remote logging to syslog

Implemented the capability to configure StackLight to forward all logs to an external syslog server. In this case, StackLight will send logs both to the syslog server and to Elasticsearch, which is the default target.


Log collection optimization

Improved the log collection mechanism by optimizing the existing and adding new log parsers for multiple Container Cloud components.


Hyperconverged Ceph

Technology Preview

Implemented the capability to configure Ceph Controller to start pods on the taint nodes and manage the resources of Ceph nodes. Now, when bootstrapping a new management or managed cluster, you can specify requests, limits, or tolerations for Ceph resources. You can also configure resource management for an existing Ceph cluster. However, such approach may cause downtime.


Ceph objectStorage section in KaasCephCluster

Improved user experience by moving the rgw section of the KaasCephCluster CR to a common objectStorage section that now includes all RADOS Gateway configurations of a Ceph cluster. The spec.rgw section is deprecated. However, if you continue using spec.rgw, it will be automatically translated into the new objectStorage.rgw section during the Container Cloud update to 2.6.0.


Ceph maintenance orchestration

Implemented the capability to enable Ceph maintenance mode using the maintenance flag not only during a managed cluster update but also when required. However, Mirantis does not recommend enabling maintenance on production deployments other than during update.


Dedicated network for the Ceph distributed storage traffic

TECHNOLOGY PREVIEW

Added the possibility to configure dedicated networks for the Ceph cluster access and replication traffic using dedicated subnets. Container Cloud automatically configures Ceph to use the addresses from the dedicated subnets after you assign the corresponding addresses to the storage nodes.


Ceph Multisite configuration

Technology Preview

Implemented the capability to enable the Ceph Multisite configuration that allows object storage to replicate its data over multiple Ceph clusters. Using Multisite, such object storage is independent and isolated from another object storage in the cluster.


Ceph troubleshooting documentation

On top of continuous improvements delivered to the existing Container Cloud guides, added the Troubleshoot Ceph section to the Operations Guide. This section now contains a detailed procedure on a failed or accidentally removed Ceph cluster recovery.

Components versions

The following table lists the components versions of the Cluster release 6.14.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Component

Application/Service

Version

Distributed storage

Ceph

14.2.12 (Nautilus)

Rook

1.5.5

Container runtime

Mirantis Container Runtime

19.03.14 1

Cluster orchestration

Mirantis Kubernetes Engine Updated

3.3.6 0

LCM

descheduler

0.8.0

Helm

2.16.11-40

helm-controller Updated

0.2.0-297-g8c87ad67

lcm-ansible Updated

0.5.0-10-gdd307e6

lcm-agent Updated

0.2.0-300-ga874e0df

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta Updated

8.4.1-20210312131419

Alertmanager

0.21.0

Cerebro

0.9.3

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd Updated

1.10.2-20210301155825

Grafana Updated

7.3.7

Grafana Image Renderer Updated

2.0.1

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector Updated

0.1-20210219112938

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter Updated

0.5.1-20210323132924

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus Relay Updated

0.3-20210317133316

Pushgateway

1.2.0

sf-notifier Updated

0.3-20210323132354

sf-reporter

0.1-20201216142628

Telegraf Updated

1.9.1-20210225142050

Telemeter

4.4.0-20200424

yq

4.2.0

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1

Artifacts

This section lists the components artifacts of the Cluster release 6.14.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

https://binary.mirantis.com/ceph/helm/ceph-operator-1.0.0-mcp-177.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.12

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20210322210534

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.2.1

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v2.1.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v2.1.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v4.0.0

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v3.1.0

csi-resizer

mirantis.azurecr.io/ceph/k8scsi/csi-resizer:v1.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.5.5


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.5.0-10-gdd307e6/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-300-ga874e0df/lcm-agent

Helm charts

descheduler Updated

https://binary.mirantis.com/core/helm/descheduler-1.19.1.tgz

managed-lcm-api Updated

https://binary.mirantis.com/core/helm/managed-lcm-api-1.19.1.tgz

metallb Updated

https://binary.mirantis.com/core/helm/metallb-1.19.1.tgz

metrics-server Updated

https://binary.mirantis.com/core/helm/metrics-server-1.19.1.tgz

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-297-g8c87ad67

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta Updated

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-15.tgz

cerebro

https://binary.mirantis.com/stacklight/helm/cerebro-0.1.0-mcp-2.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd Updated

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-17.tgz

fluentd-elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-61.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-93.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-8.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni Updated

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-20.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-124.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-4.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-11.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-10.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-438.tgz

telegraf-ds

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-20.tgz

telegraf-s

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-20.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-12.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-12.tgz

Docker images

alerta Updated

mirantis.azurecr.io/stacklight/alerta-web:8.4.1-20210312131419

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

cerebro

mirantis.azurecr.io/stacklight/cerebro:0.9.3

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd Updated

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20210301155825

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana Updated

mirantis.azurecr.io/stacklight/grafana:7.3.7

grafana-image-renderer Updated

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.1

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar Updated

mirantis.azurecr.io/stacklight/k8s-sidecar:1.10.8

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector Updated

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20210219112938

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter Updated

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20210323132924

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-relay Updated

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20210317133316

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier Updated

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20210323132354

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20201216142628

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf Updated

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20210225142050

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

yq

mirantis.azurecr.io/stacklight/yq:v4.2.0

5.14.0

This section outlines release notes for the Cluster release 5.14.0 that is introduced in the Mirantis Container Cloud release 2.7.0. This Cluster release supports Mirantis Kubernetes Engine 3.3.6, Mirantis Container Runtime 19.03.14, and Kubernetes 1.18.

For the list of known and resolved issues, refer to the Container Cloud release 2.7.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.14.0.


Log collection optimization

Improved the log collection mechanism by optimizing the existing and adding new log parsers for multiple Container Cloud components.


Dedicated network for the Ceph distributed storage traffic

TECHNOLOGY PREVIEW

Added the possibility to configure dedicated networks for the Ceph cluster access and replication traffic using dedicated subnets. Container Cloud automatically configures Ceph to use the addresses from the dedicated subnets after you assign the corresponding addresses to the storage nodes.


Ceph Multisite configuration

TECHNOLOGY PREVIEW

Implemented the capability to enable the Ceph Multisite configuration that allows object storage to replicate its data over multiple Ceph clusters. Using Multisite, such object storage is independent and isolated from another object storage in the cluster.


Ceph troubleshooting documentation

On top of continuous improvements delivered to the existing Container Cloud guides, added the Troubleshoot Ceph section to the Operations Guide. This section now contains a detailed procedure to recover a failed or accidentally removed Ceph cluster.

Components versions

The following table lists the components versions of the Cluster release 5.14.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Components versions of the Cluster release 5.14.0

Component

Application/Service

Version

Distributed storage

Ceph

14.2.12 (Nautilus)

Rook

1.5.5

Cluster orchestration

Mirantis Kubernetes Engine

3.3.6 0

Container runtime

Mirantis Container Runtime

19.03.14 1

LCM

descheduler

0.8.0

Helm

2.16.11-40

helm-controller Updated

0.2.0-297-g8c87ad67

lcm-ansible Updated

0.5.0-10-gdd307e6

lcm-agent Updated

0.2.0-300-ga874e0df

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta Updated

8.4.1-20210312131419

Alertmanager

0.21.0

Cerebro

0.9.3

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20210301155825

Grafana

7.3.7

Grafana Image Renderer

2.0.1

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector

0.1-20210219112938

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter Updated

0.5.1-20210323132924

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus Relay Updated

0.3-20210317133316

Pushgateway

1.2.0

sf-notifier Updated

0.3-20210323132354

sf-reporter

0.1-20201216142628

Telegraf

1.9.1-20210225142050

Telemeter

4.4.0-20200424

yq

4.2.0

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1

Artifacts

This section lists the components artifacts of the Cluster release 5.14.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

https://binary.mirantis.com/ceph/helm/ceph-operator-1.0.0-mcp-177.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.12

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20210322210534

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.2.1

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v2.1.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v2.1.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v4.0.0

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v3.1.0

csi-resizer

mirantis.azurecr.io/ceph/k8scsi/csi-resizer:v1.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.5.5


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.5.0-10-gdd307e6/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-300-ga874e0df/lcm-agent

Helm charts

descheduler Updated

https://binary.mirantis.com/core/helm/descheduler-1.19.1.tgz

managed-lcm-api Updated

https://binary.mirantis.com/core/helm/managed-lcm-api-1.19.1.tgz

metallb Updated

https://binary.mirantis.com/core/helm/metallb-1.19.1.tgz

metrics-server Updated

https://binary.mirantis.com/core/helm/metrics-server-1.19.1.tgz

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-297-g8c87ad67

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-15.tgz

cerebro

https://binary.mirantis.com/stacklight/helm/cerebro-0.1.0-mcp-2.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd Updated

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-17.tgz

fluentd-elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-61.tgz

grafana

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-93.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-8.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-20.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-124.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-4.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-11.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-10.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-438.tgz

telegraf-ds

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-20.tgz

telegraf-s

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-20.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-12.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-12.tgz

Docker images

alerta Updated

mirantis.azurecr.io/stacklight/alerta-web:8.4.1-20210312131419

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

cerebro

mirantis.azurecr.io/stacklight/cerebro:0.9.3

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20210301155825

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.3.7

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.1

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar Updated

mirantis.azurecr.io/stacklight/k8s-sidecar:1.10.8

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20210219112938

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter Updated

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20210323132924

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-relay Updated

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20210317133316

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier Updated

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20210323132354

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20201216142628

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20210225142050

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

yq

mirantis.azurecr.io/stacklight/yq:v4.2.0

5.11.0

This section outlines release notes for the Cluster release 5.11.0 that is introduced in the Mirantis Container Cloud release 2.4.0. This Cluster release supports Kubernetes 1.18 and Mirantis Kubernetes Engine 3.3.4 as well as introduces support for the updated version of Mirantis Container Runtime 19.03.14.

Note

The Cluster release 5.11.0 supports only attachment of existing MKE 3.3.4 clusters. For the deployment of new or attachment of existing MKE 3.3.6 clusters, the latest available Cluster release is used.

For the list of known and resolved issues, refer to the Container Cloud release 2.4.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.11.0.


Alert inhibition rules

Implemented alert inhibition rules to provide a clearer view on the cloud status and simplify troubleshooting. Using alert inhibition rules, Alertmanager decreases alert noise by suppressing dependent alerts notifications. The feature is enabled by default. For details, see Alert dependencies.


Integration between Grafana and Kibana

Implemented integration between Grafana and Kibana by adding a View logs in Kibana link to the majority of Grafana dashboards, which allows you to immediately view contextually relevant logs through the Kibana web UI.


Telegraf alert

Implemented the TelegrafGatherErrors alert that raises if Telegraf fails to gather metrics.

Learn more

Telegraf alerts


Configuration of Ironic Telegraf input plugin

Added the ironic.insecure parameter for enabling or disabling the host and chain verification for bare metal Ironic monitoring.


Automatically defined cluster ID

Enhanced StackLight to automatically set clusterId that defines an ID of a Container Cloud cluster. Now, you do not need to set or modify this parameter manually when configuring the sf-notifier and sf-reporter services.

Components versions

The following table lists the components versions of the Cluster release 5.11.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Components versions of the Cluster release 5.11.0

Component

Application/Service

Version

Distributed storage Updated

Ceph

14.2.12 (Nautilus)

Rook

1.5.5

Container runtime

Mirantis Container Runtime Updated

19.03.14 1

Cluster orchestration

Mirantis Kubernetes Engine

3.3.4 0

LCM

descheduler

0.8.0

Helm

2.16.11-40

helm-controller

0.2.0-221-g32bd5f56

lcm-ansible Updated

0.2.0-394-g599b2a1

lcm-agent

0.2.0-221-g32bd5f56

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

8.0.2-20201014133832

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector Updated

0.1-20201222100033

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus RabbitMQ Exporter

1.0.0-RC7.1

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier Updated

0.3-20201216142028

sf-reporter Updated

0.1-20201216142628

Telegraf Updated

1.9.1-20201222194740

Telemeter

4.4.0-20200424

yq New

4.2.0

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1

Artifacts

This section lists the components artifacts of the Cluster release 5.11.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

https://binary.mirantis.com/ceph/helm/ceph-operator-1.0.0-mcp-111.tgz

Docker images Updated

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.12

ceph-controller

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20210120004212

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.2.1

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v2.1.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v2.1.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v4.0.0

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v3.1.0

csi-resizer New

mirantis.azurecr.io/ceph/k8scsi/csi-resizer:v1.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.5.5


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-394-g599b2a1/lcm-ansible.tar.gz

lcm-agent

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-221-g32bd5f56/lcm-agent

Helm charts

descheduler Updated

https://binary.mirantis.com/core/helm/descheduler-1.16.0.tgz

managed-lcm-api Updated

https://binary.mirantis.com/core/helm/managed-lcm-api-1.16.0.tgz

metallb Updated

https://binary.mirantis.com/core/helm/metallb-1.16.0.tgz

metrics-server Updated

https://binary.mirantis.com/core/helm/metrics-server-1.16.0.tgz

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-221-g32bd5f56

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-33.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-81.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector Updated

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-8.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni Updated

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-19.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-114.tgz

prometheus-blackbox-exporter Updated

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-4.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier Updated

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-11.tgz

sf-reporter Updated

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-10.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-398.tgz

telegraf-ds Updated

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-20.tgz

telegraf-s Updated

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-20.tgz

telemeter-server Updated

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-12.tgz

telemeter-client Updated

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-12.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector Updated

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20201222100033

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-rabbitmq-exporter

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v1.0.0-RC7.1

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier Updated

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201216142028

sf-reporter Updated

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20201216142628

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf Updated

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20201222194740

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

yq New

mirantis.azurecr.io/stacklight/yq:v4.2.0

Deprecated Cluster releases

This section describes the release notes for the deprecated Cluster releases that will become unsupported in one of the following Container Cloud releases. Make sure to update your managed clusters to the latest supported version as described in Operations Guide: Update a managed cluster.

For details about supported Cluster releases, see Cluster releases (managed).

6.12.0

The Cluster release 6.12.0 is introduced in the Mirantis Container Cloud release 2.5.0 and is supported by 2.6.0. This Cluster release is based on the Cluster release 5.12.0.

The Cluster release 6.12.0 supports:

  • Mirantis OpenStack for Kubernetes (MOS) 21.1. For details, see MOS Release Notes.

  • Updated versions of Mirantis Kubernetes Engine (MKE) 3.3.6 and Mirantis Container Runtime (MCR) 19.03.14. For details, see MKE Release Notes and MCR Release Notes.

  • Kubernetes 1.18.

For the list of resolved issues, refer to the Container Cloud releases 2.4.0 and 2.5.0 sections. For the list of known issues, refer to the Container Cloud release 2.5.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 6.12.0.


Alert inhibition rules

Implemented alert inhibition rules to provide a clearer view on the cloud status and simplify troubleshooting. Using alert inhibition rules, Alertmanager decreases alert noise by suppressing dependent alerts notifications. The feature is enabled by default. For details, see Alert dependencies.


Integration between Grafana and Kibana

Implemented integration between Grafana and Kibana by adding a View logs in Kibana link to the majority of Grafana dashboards, which allows you to immediately view contextually relevant logs through the Kibana web UI.


Telegraf alert

Implemented the TelegrafGatherErrors alert that raises if Telegraf fails to gather metrics.

Learn more

Telegraf alerts


Configuration of Ironic Telegraf input plugin

Added the ironic.insecure parameter for enabling or disabling the host and chain verification for bare metal Ironic monitoring.


Automatically defined cluster ID

Enhanced StackLight to automatically set clusterId that defines an ID of a Container Cloud cluster. Now, you do not need to set or modify this parameter manually when configuring the sf-notifier and sf-reporter services.


Cerebro support for StackLight

Enhanced StackLight by adding support for Cerebro, a web UI that visualizes health of Elasticsearch clusters and allows for convenient debugging. Cerebro is disabled by default.


Ceph maintenance label

Implemented the maintenance label to set for Ceph during a managed cluster update. This prevents Ceph rebalance leading to data loss during a managed cluster update.


RGW check box in Container Cloud web UI

Implemented the Enable Object Storage checkbox in the Container Cloud web UI to allow enabling a single-instance RGW Object Storage when creating a Ceph cluster as described in Add a Ceph cluster.


Ceph RGW HA

Enhanced Ceph to support RADOS Gateway (RGW) high availability. Now, you can run multiple instances of Ceph RGW in active/active mode.


StackLight proxy

Added proxy support for Alertmanager, Metric collector, Salesforce notifier and reporter, and Telemeter client. Now, these StackLight components automatically use the same proxy that is configured for Container Cloud clusters.

Note

Proxy handles only the HTTP and HTTPS traffic. Therefore, for clusters with limited or no Internet access, it is not possible to set up Alertmanager email notifications, which use SMTP, when proxy is used.

Note

Due to a limitation, StackLight fails to integrate with an external proxy with authentication handled by a proxy server. In such cases, the proxy server ignores the HTTP Authorization header for basic authentication passed by Prometheus Alertmanager. Therefore, use proxies without authentication or with authentication handled by a reverse proxy.

Components versions

The following table lists the components versions of the Cluster release 6.12.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Component

Application/Service

Version

Distributed storage

Ceph

14.2.12 (Nautilus)

Rook

1.5.5

Container runtime

Mirantis Container Runtime

19.03.14 1

Cluster orchestration

Mirantis Kubernetes Engine Updated

3.3.6 0

LCM

descheduler

0.8.0

Helm

2.16.11-40

helm-controller Updated

0.2.0-258-ga2d72294

lcm-ansible Updated

0.3.0-10-g7c2a87e

lcm-agent Updated

0.2.0-258-ga2d72294

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

8.0.2-20201014133832

Alertmanager

0.21.0

Cerebro New

0.9.3

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector

0.1-20201222100033

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier

0.3-20201216142028

sf-reporter

0.1-20201216142628

Telegraf

1.9.1-20201222194740

Telemeter

4.4.0-20200424

yq

4.2.0

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1
Artifacts

This section lists the components artifacts of the Cluster release 6.12.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

https://binary.mirantis.com/ceph/helm/ceph-operator-1.0.0-mcp-127.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.12

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20210201202754

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.2.1

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v2.1.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v2.1.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v4.0.0

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v3.1.0

csi-resizer

mirantis.azurecr.io/ceph/k8scsi/csi-resizer:v1.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.5.5


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.3.0-10-g7c2a87e/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-258-ga2d72294/lcm-agent

Helm charts

descheduler Updated

https://binary.mirantis.com/core/helm/descheduler-1.17.4.tgz

managed-lcm-api Updated

https://binary.mirantis.com/core/helm/managed-lcm-api-1.17.4.tgz

metallb Updated

https://binary.mirantis.com/core/helm/metallb-1.17.4.tgz

metrics-server Updated

https://binary.mirantis.com/core/helm/metrics-server-1.17.4.tgz

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-258-ga2d72294

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

cerebro New

https://binary.mirantis.com/stacklight/helm/cerebro-0.1.0-mcp-2.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-33.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-89.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-8.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-19.tgz

prometheus

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-114.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-4.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-11.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-10.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-401.tgz

telegraf-ds

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-20.tgz

telegraf-s

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-20.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-12.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-12.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

cerebro New

mirantis.azurecr.io/stacklight/cerebro:0.9.3

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20201222100033

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201216142028

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20201216142628

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20201222194740

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

yq

mirantis.azurecr.io/stacklight/yq:v4.2.0

5.13.0

This section outlines release notes for the Cluster release 5.13.0 that is introduced in the Mirantis Container Cloud release 2.6.0. This Cluster release supports Mirantis Kubernetes Engine 3.3.6, Mirantis Container Runtime 19.03.14, and Kubernetes 1.18.

For the list of known and resolved issues, refer to the Container Cloud release 2.6.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.13.0.


StackLight logging levels

Significantly enhanced the StackLight log collection mechanism to avoid collecting and keeping an excessive amount of log messages when it is not essential. Now, during or after deployment of StackLight, you can select one of the 9 available logging levels depending on the required severity. The default logging level is INFO.


Remote logging to syslog

Implemented the capability to configure StackLight to forward all logs to an external syslog server. In this case, StackLight will send logs both to the syslog server and to Elasticsearch, which is the default target.


Hyperconverged Ceph

Technology Preview

Implemented the capability to configure Ceph Controller to start pods on the taint nodes and manage the resources of Ceph nodes. Now, when bootstrapping a new management or managed cluster, you can specify requests, limits, or tolerations for Ceph resources. You can also configure resource management for an existing Ceph cluster. However, such approach may cause downtime.


Ceph objectStorage section in KaasCephCluster

Improved user experience by moving the rgw section of the KaasCephCluster CR to a common objectStorage section that now includes all RADOS Gateway configurations of a Ceph cluster. The spec.rgw section is deprecated. However, if you continue using spec.rgw, it will be automatically translated into the new objectStorage.rgw section during the Container Cloud update to 2.6.0.


Ceph maintenance orchestration

Implemented the capability to enable Ceph maintenance mode using the maintenance flag not only during a managed cluster update but also when required. However, Mirantis does not recommend enabling maintenance on production deployments other than during update.

Components versions

The following table lists the components versions of the Cluster release 5.13.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Components versions of the Cluster release 5.13.0

Component

Application/Service

Version

Distributed storage

Ceph

14.2.12 (Nautilus)

Rook

1.5.5

Cluster orchestration

Mirantis Kubernetes Engine

3.3.6 0

Container runtime

Mirantis Container Runtime

19.03.14 1

LCM

descheduler

0.8.0

Helm

2.16.11-40

helm-controller Updated

0.2.0-289-gd7e9fa9c

lcm-ansible Updated

0.4.0-4-ga2bb104

lcm-agent Updated

0.2.0-289-gd7e9fa9c

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

8.0.2-20201014133832

Alertmanager

0.21.0

Cerebro

0.9.3

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd Updated

1.10.2-20210301155825

Grafana Updated

7.3.7

Grafana Image Renderer Updated

2.0.1

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector Updated

0.1-20210219112938

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier

0.3-20201216142028

sf-reporter

0.1-20201216142628

Telegraf Updated

1.9.1-20210225142050

Telemeter

4.4.0-20200424

yq

4.2.0

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1
Artifacts

This section lists the components artifacts of the Cluster release 5.13.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

https://binary.mirantis.com/ceph/helm/ceph-operator-1.0.0-mcp-165.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.12

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20210309160354

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.2.1

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v2.1.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v2.1.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v4.0.0

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v3.1.0

csi-resizer

mirantis.azurecr.io/ceph/k8scsi/csi-resizer:v1.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.5.5


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.4.0-4-ga2bb104/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-289-gd7e9fa9c/lcm-agent

Helm charts

descheduler Updated

https://binary.mirantis.com/core/helm/descheduler-1.18.1.tgz

managed-lcm-api Updated

https://binary.mirantis.com/core/helm/managed-lcm-api-1.18.1.tgz

metallb Updated

https://binary.mirantis.com/core/helm/metallb-1.18.1.tgz

metrics-server Updated

https://binary.mirantis.com/core/helm/metrics-server-1.18.1.tgz

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-289-gd7e9fa9c

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta Updated

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-15.tgz

cerebro

https://binary.mirantis.com/stacklight/helm/cerebro-0.1.0-mcp-2.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd Updated

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-16.tgz

fluentd-elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-44.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-93.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-8.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni Updated

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-20.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-121.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-4.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-11.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-10.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-426.tgz

telegraf-ds

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-20.tgz

telegraf-s

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-20.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-12.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-12.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

cerebro

mirantis.azurecr.io/stacklight/cerebro:0.9.3

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd Updated

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20210301155825

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana Updated

mirantis.azurecr.io/stacklight/grafana:7.3.7

grafana-image-renderer Updated

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.1

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector Updated

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20210219112938

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201216142028

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20201216142628

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf Updated

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20210225142050

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

yq

mirantis.azurecr.io/stacklight/yq:v4.2.0

Unsupported Cluster releases

This section describes the release notes for the unsupported Cluster releases. For details about supported Cluster releases, see Cluster releases (managed).

6.x series

This section outlines release notes for the unsupported Cluster releases of the 6.x series.

6.10.0

The Cluster release 6.10.0 is introduced in the Mirantis Container Cloud release 2.3.0 and supports:

  • Mirantis OpenStack for Kubernetes (MOS) Ussuri Update. For details, see MOS Release Notes.

  • Updated versions of Mirantis Kubernetes Engine 3.3.4 and Mirantis Container Runtime 19.03.13. For details, see MKE Release Notes and MCR Release Notes.

  • Kubernetes 1.18.

For the list of known and resolved issues, refer to the Container Cloud release 2.3.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 6.10.0.


Ceph Object Storage support

Enhanced Ceph to support RADOS Gateway (RGW) Object Storage.


Ceph state verification

Implemented the capability to obtain detailed information on the Ceph cluster state, including Ceph logs, Ceph OSDs state, and a list of Ceph pools.

Components versions

The following table lists the components versions of the Cluster release 6.10.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Component

Application/Service

Version

Distributed storage

Ceph

14.2.11 (Nautilus)

Rook

1.4.4

Container runtime

Mirantis Container Runtime Updated

19.03.13 1

Cluster orchestration

Mirantis Kubernetes Engine Updated

3.3.4 0

LCM

descheduler

0.8.0

Helm Updated

2.16.11-40

helm-controller Updated

0.2.0-221-g32bd5f56

lcm-ansible Updated

0.2.0-381-g720ec96

lcm-agent Updated

0.2.0-221-g32bd5f56

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

8.0.2-20201014133832

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector Updated

0.1-20201120155524

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus Updated

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus RabbitMQ Exporter Updated

v1.0.0-RC7.1

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier

0.3-20201001081256

sf-reporter

0.1-20200219140217

Telegraf Updated

1.9.1-20201120081248

Telemeter

4.4.0-20200424

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1
Artifacts

This section lists the components artifacts of the Cluster release 6.10.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

https://binary.mirantis.com/ceph/helm/ceph-operator-1.0.0-mcp-95.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.11

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20201215142221

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.1.0

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v1.2.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v1.6.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v2.1.1

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v2.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.4.4


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-381-g720ec96/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-221-g32bd5f56/lcm-agent

Helm charts

descheduler Updated

https://binary.mirantis.com/core/helm/descheduler-1.15.1.tgz

managed-lcm-api New

https://binary.mirantis.com/core/helm/managed-lcm-api-1.15.1.tgz

metallb Updated

https://binary.mirantis.com/core/helm/metallb-1.15.1.tgz

metrics-server Updated

https://binary.mirantis.com/core/helm/metrics-server-1.15.1.tgz

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm Updated

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-221-g32bd5f56

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-33.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-74.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector Updated

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-5.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-17.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-102.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-3.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier Updated

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-9.tgz

sf-reporter Updated

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-8.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-354.tgz

telegraf-ds Updated

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-19.tgz

telegraf-s Updated

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-19.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-11.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-11.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector Updated

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20201120155524

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus Updated

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-rabbitmq-exporter Updated

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v1.0.0-RC7.1

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201001081256

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20200219140217

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf Updated

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20201120081248

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

6.8.1

The Cluster release 6.8.1 is introduced in the Mirantis Container Cloud release 2.2.0. This Cluster release is based on the Cluster release 5.8.0 and the main difference is support of the Mirantis OpenStack for Kubernetes (MOS) product.

For details about MOS, see MOS Release Notes.

For details about the Cluster release 5.8.0, refer to the 5.8.0 section.

5.x series

This section outlines release notes for the unsupported Cluster releases of the 5.x series.

Note

The Cluster release 5.11.0 supports only attachment of existing MKE 3.3.4 clusters. For the deployment of new or attachment of existing MKE 3.3.6 clusters, the latest available Cluster release is used.

5.12.0

This section outlines release notes for the Cluster release 5.12.0 that is introduced in the Mirantis Container Cloud release 2.5.0. This Cluster release supports Kubernetes 1.18 and Mirantis Container Runtime 19.03.14 as well as introduces support for the updated version of Mirantis Kubernetes Engine 3.3.6.

For the list of known and resolved issues, refer to the Container Cloud release 2.5.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.12.0.


Ceph maintenance label

Implemented the maintenance label to set for Ceph during a managed cluster update. This prevents Ceph rebalance leading to data loss during a managed cluster update.


RGW check box in Container Cloud web UI

Implemented the Enable Object Storage checkbox in the Container Cloud web UI to allow enabling a single-instance RGW Object Storage when creating a Ceph cluster as described in Add a Ceph cluster.


Ceph RGW HA

Enhanced Ceph to support RADOS Gateway (RGW) high availability. Now, you can run multiple instances of Ceph RGW in active/active mode.


Cerebro support for StackLight

Enhanced StackLight by adding support for Cerebro, a web UI that visualizes health of Elasticsearch clusters and allows for convenient debugging. Cerebro is disabled by default.


StackLight proxy

Added proxy support for Alertmanager, Metric collector, Salesforce notifier and reporter, and Telemeter client. Now, these StackLight components automatically use the same proxy that is configured for Container Cloud clusters.

Note

Proxy handles only the HTTP and HTTPS traffic. Therefore, for clusters with limited or no Internet access, it is not possible to set up Alertmanager email notifications, which use SMTP, when proxy is used.

Note

Due to a limitation, StackLight fails to integrate with an external proxy with authentication handled by a proxy server. In such cases, the proxy server ignores the HTTP Authorization header for basic authentication passed by Prometheus Alertmanager. Therefore, use proxies without authentication or with authentication handled by a reverse proxy.

Components versions

The following table lists the components versions of the Cluster release 5.12.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Components versions of the Cluster release 5.12.0

Component

Application/Service

Version

Distributed storage

Ceph

14.2.12 (Nautilus)

Rook

1.5.5

Container runtime

Mirantis Container Runtime

19.03.14 1

Cluster orchestration

Mirantis Kubernetes Engine Updated

3.3.6 0

LCM

descheduler

0.8.0

Helm

2.16.11-40

helm-controller Updated

0.2.0-258-ga2d72294

lcm-ansible Updated

0.3.0-10-g7c2a87e

lcm-agent Updated

0.2.0-258-ga2d72294

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

8.0.2-20201014133832

Alertmanager

0.21.0

Cerebro New

0.9.3

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector

0.1-20201222100033

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier

0.3-20201216142028

sf-reporter

0.1-20201216142628

Telegraf

1.9.1-20201222194740

Telemeter

4.4.0-20200424

yq

4.2.0

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1
Artifacts

This section lists the components artifacts of the Cluster release 5.12.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

https://binary.mirantis.com/ceph/helm/ceph-operator-1.0.0-mcp-127.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.12

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20210201202754

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.2.1

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v2.1.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v2.1.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v4.0.0

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v3.1.0

csi-resizer

mirantis.azurecr.io/ceph/k8scsi/csi-resizer:v1.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.5.5


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.3.0-10-g7c2a87e/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-258-ga2d72294/lcm-agent

Helm charts

descheduler Updated

https://binary.mirantis.com/core/helm/descheduler-1.17.4.tgz

managed-lcm-api Updated

https://binary.mirantis.com/core/helm/managed-lcm-api-1.17.4.tgz

metallb Updated

https://binary.mirantis.com/core/helm/metallb-1.17.4.tgz

metrics-server Updated

https://binary.mirantis.com/core/helm/metrics-server-1.17.4.tgz

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-258-ga2d72294

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

cerebro New

https://binary.mirantis.com/stacklight/helm/cerebro-0.1.0-mcp-2.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-33.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-89.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-8.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-19.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-119.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-4.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-11.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-10.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-413.tgz

telegraf-ds

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-20.tgz

telegraf-s

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-20.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-12.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-12.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

cerebro New

mirantis.azurecr.io/stacklight/cerebro:0.9.3

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20201222100033

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201216142028

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20201216142628

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20201222194740

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

yq

mirantis.azurecr.io/stacklight/yq:v4.2.0

5.10.0

This section outlines release notes for the Cluster release 5.10.0 that is introduced in the Mirantis Container Cloud release 2.3.0. This Cluster release supports Kubernetes 1.18 and introduces support for the latest versions of Mirantis Kubernetes Engine 3.3.4 and Mirantis Container Runtime 19.03.13.

For the list of known and resolved issues, refer to the Container Cloud release 2.3.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.10.0.


Ceph Object Storage support

Enhanced Ceph to support RADOS Gateway (RGW) Object Storage.


Ceph state verification

Implemented the capability to obtain detailed information on the Ceph cluster state, including Ceph logs, Ceph OSDs state, and a list of Ceph pools.

Components versions

The following table lists the components versions of the Cluster release 5.10.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Component

Application/Service

Version

Distributed storage

Ceph

14.2.11 (Nautilus)

Rook

1.4.4

Container runtime

Mirantis Container Runtime Updated

19.03.13 1

Cluster orchestration

Mirantis Kubernetes Engine Updated

3.3.4 0

LCM

descheduler

0.8.0

Helm Updated

2.16.11-40

helm-controller Updated

0.2.0-221-g32bd5f56

lcm-ansible Updated

0.2.0-381-g720ec96

lcm-agent Updated

0.2.0-221-g32bd5f56

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

8.0.2-20201014133832

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

Metric Collector Updated

0.1-20201120155524

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus Updated

2.22.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus RabbitMQ Exporter Updated

v1.0.0-RC7.1

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier

0.3-20201001081256

sf-reporter

0.1-20200219140217

Telegraf Updated

1.9.1-20201120081248

Telemeter

4.4.0-20200424

0

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

1
Artifacts

This section lists the components artifacts of the Cluster release 5.10.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

https://binary.mirantis.com/ceph/helm/ceph-operator-1.0.0-mcp-95.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.11

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20201215142221

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.1.0

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v1.2.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v1.6.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v2.1.1

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v2.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.4.4


LCM artifacts

Artifact

Component

Path

Binaries

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-381-g720ec96/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-221-g32bd5f56/lcm-agent

Helm charts

descheduler Updated

https://binary.mirantis.com/core/helm/descheduler-1.15.1.tgz

managed-lcm-api New

https://binary.mirantis.com/core/helm/managed-lcm-api-1.15.1.tgz

metallb Updated

https://binary.mirantis.com/core/helm/metallb-1.15.1.tgz

metrics-server Updated

https://binary.mirantis.com/core/helm/metrics-server-1.15.1.tgz

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm Updated

mirantis.azurecr.io/lcm/helm/tiller:v2.16.11-40

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-221-g32bd5f56

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-22.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-33.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-74.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

metric-collector Updated

https://binary.mirantis.com/stacklight/helm/metric-collector-0.2.0-mcp-5.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-17.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-102.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-3.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier Updated

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-9.tgz

sf-reporter Updated

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-8.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-354.tgz

telegraf-ds Updated

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-19.tgz

telegraf-s Updated

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-19.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-11.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-11.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

metric-collector Updated

mirantis.azurecr.io/stacklight/metric-collector:v0.1-20201120155524

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus Updated

mirantis.azurecr.io/stacklight/prometheus:v2.22.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-rabbitmq-exporter Updated

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v1.0.0-RC7.1

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201001081256

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20200219140217

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf Updated

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20201120081248

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

5.9.0

This section outlines release notes for the Cluster release 5.9.0 that is introduced in the Mirantis Container Cloud release 2.2.0 and supports Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18.

For the list of known and resolved issues, refer to the Container Cloud release 2.2.0 section.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.9.0.


Alerta upgrade

Upgraded Alerta from version 7.4.4 to 8.0.2.


File descriptors monitoring

Enhanced StackLight to monitor the number of file descriptors on nodes and raise FileDescriptorUsage* alerts when a node uses 80%, 90%, or 95% of file descriptors.

Learn more

General node alerts


Alerts improvements
  • Added the SSLProbesFailing alert that raises in case of an SSL certificate probes failure.

  • Improved alerts descriptions and raise conditions.

Components versions

The following table lists the components versions of the Cluster release 5.9.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Components versions of the Cluster release 5.9.0

Component

Application/Service

Version

Distributed storage

Ceph Updated

14.2.11 (Nautilus)

Rook Updated

1.4.4

Container runtime

Mirantis Container Runtime

19.03.12 0

Cluster orchestration

Mirantis Kubernetes Engine

3.3.3 1

LCM

ansible-docker Updated

0.3.5-147-g18f3b44

descheduler

0.8.0

Helm

2.16.9-39

helm-controller Updated

0.2.0-178-g8cc488f8

lcm-ansible Updated

0.2.0-132-g49f7591

lcm-agent Updated

0.2.0-178-g8cc488f8

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta Updated

8.0.2-20201014133832

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

7.1.5

Grafana Image Renderer

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

MCC Metric Collector

0.1-20201005141816

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus Updated

2.19.3

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20201002144823

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20201006113956

Prometheus RabbitMQ Exporter

0.29.0

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier Updated

0.3-20201001081256

sf-reporter

0.1-20200219140217

telegraf-ds

1.9.1-20200901112858

telegraf-s

1.9.1-20200901112858

Telemeter

4.4.0-20200424

0

For the Mirantis Container Runtime release highlights, see MKE, MSR, and MCR documentation: Mirantis Container Runtime release notes.

1

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

Artifacts

This section lists the components artifacts of the Cluster release 5.9.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

https://binary.mirantis.com/ceph/helm/ceph-operator-1.0.0-mcp-33.tgz

Docker images

ceph Updated

mirantis.azurecr.io/ceph/ceph:v14.2.11

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20201022081323

cephcsi Updated

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v3.1.0

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v1.2.0

csi-provisioner Updated

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v1.6.0

csi-snapshotter Updated

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v2.1.1

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v2.1.0

rook Updated

mirantis.azurecr.io/ceph/rook/ceph:v1.4.4


LCM artifacts

Artifact

Component

Path

Binaries

ansible-docker Updated

https://binary.mirantis.com/lcm/bin/ansible-docker/v0.3.5-147-g18f3b44/ansible-docker.tar.gz

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-132-g49f7591-1/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-178-g8cc488f8/lcm-agent

Helm charts

descheduler Updated

https://binary.mirantis.com/core/helm/descheduler-1.14.0.tgz

metallb Updated

https://binary.mirantis.com/core/helm/metallb-1.14.0.tgz

metrics-server Updated

https://binary.mirantis.com/core/helm/metrics-server-1.14.0.tgz

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.9-39

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-178-g8cc488f8

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta Updated

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-13.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-20.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch Updated

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-28.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-66.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

mcc-metric-collector

https://binary.mirantis.com/stacklight/helm/mcc-metric-collector-0.1.0-mcp-22.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-17.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-83.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-3.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-5.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-6.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-325.tgz

telegraf-ds

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-16.tgz

telegraf-s

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-16.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-11.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-11.tgz

Docker images

alerta Updated

mirantis.azurecr.io/stacklight/alerta-web:8.0.2-20201014133832

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests Updated

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200618

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl Updated

mirantis.azurecr.io/stacklight/kubectl:1.19.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

mcc-metric-collector

mirantis.azurecr.io/stacklight/mcc-metric-collector:v0.1-20201005141816

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus Updated

mirantis.azurecr.io/stacklight/prometheus:v2.19.3

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-rabbitmq-exporter

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v0.29.0

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier Updated

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20201001081256

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20200219140217

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20200901112858

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

5.8.0

This section outlines release notes for the Cluster release 5.8.0 that is introduced in the Mirantis Container Cloud release 2.1.0 and supports Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18.

For the list of known issues, refer to the Container Cloud release 2.1.0 Known issues.

Enhancements

This section outlines new features and enhancements introduced in the Cluster release 5.8.0.


Grafana improvements
  • Upgraded Grafana from version 6.6.2 to 7.1.5.

  • Introduced Grafana Image Renderer, a separate Grafana container in a pod to offload rendering of images from charts. Grafana Image Renderer is enabled by default.

  • Configured a home dashboard to replace the Installation/configuration panel that opens when you access Grafana. By default, Kubernetes Cluster is set as a home dashboard. However, you can set any of the available Grafana dashboards.


Clusters telemetry improvement in StackLight
  • Split the regional and management cluster function in StackLight telemetry. Now, the metrics from managed clusters are aggregated on regional clusters, then both regional and managed clusters metrics are sent from regional clusters to the management cluster.

  • Added the capability to filter panels by regions in the Clusters Overview and Telemeter Server Grafana dashboards.


Alerts improvements
  • Improved alerts descriptions and raise conditions.

  • Changed severity in some alerts to improve operability.

  • Improved raise conditions of some alerts by adding the for clause and unifying the existing for clauses.

Components versions

The following table lists the components versions of the Cluster release 5.8.0.

Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Components versions of the Cluster release 5.8.0

Component

Application/Service

Version

Distributed storage

Ceph

14.2.9 (Nautilus)

Rook

1.3.8

Container runtime

Mirantis Container Runtime

19.03.12 0

Cluster orchestration

Mirantis Kubernetes Engine

3.3.3 1

LCM

ansible-docker Updated

0.3.5-141-g1007cc9

descheduler

0.8.0

Helm Updated

2.16.9-39

helm-controller Updated

0.2.0-169-g5668304d

lcm-ansible Updated

0.2.0-119-g8f05f58-1

lcm-agent

0.2.0-149-g412c5a05

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

7.4.4-20200615123606

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana Updated

7.1.5

Grafana Image Renderer New

2.0.0

IAM Proxy

6.0.1

Kibana

7.6.1

MCC Metric Collector Updated

0.1-20201005141816

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.19.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter Updated

0.5.1-20201002144823

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter Updated

0.8.0-20201006113956

Prometheus RabbitMQ Exporter

0.29.0

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier Updated

0.3-20200813125431

sf-reporter

0.1-20200219140217

telegraf-ds Updated

1.9.1-20200901112858

telegraf-s Updated

1.9.1-20200901112858

Telemeter

4.4.0-20200424

0

For the Mirantis Container Runtime release highlights, see MKE, MSR, and MCR documentation: Mirantis Container Runtime release notes.

1

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

Artifacts

This section lists the components artifacts of the Cluster release 5.8.0.


Note

The components that are newly added, updated, deprecated, or removed as compared to the previous release version, are marked with a corresponding superscript, for example, lcm-ansible Updated.

Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller Updated

https://binary.mirantis.com/ceph/helm/ceph-operator-1.0.0-mcp-18.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.9

ceph-controller Updated

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20200903151423

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v2.1.2

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v1.2.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v1.4.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v1.2.2

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v2.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.3.8


LCM artifacts

Artifact

Component

Path

Binaries

ansible-docker Updated

https://binary.mirantis.com/lcm/bin/ansible-docker/v0.3.5-141-g1007cc9/ansible-docker.tar.gz

lcm-ansible Updated

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-119-g8f05f58-1/lcm-ansible.tar.gz

lcm-agent Updated

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-169-g5668304d/lcm-agent

Helm charts

descheduler Updated

https://binary.mirantis.com/core/helm/descheduler-1.12.2.tgz

metallb Updated

https://binary.mirantis.com/core/helm/metallb-1.12.2.tgz

metrics-server Updated

https://binary.mirantis.com/core/helm/metrics-server-1.12.2.tgz

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm Updated

mirantis.azurecr.io/lcm/helm/tiller:v2.16.9-39

helm-controller Updated

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-169-g5668304d

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-12.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-20.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd Updated

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-25.tgz

grafana Updated

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-64.tgz

iam-proxy

https://binary.mirantis.com/iam/helm/iam-proxy-0.2.2.tgz

kibana Updated

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-20.tgz

mcc-metric-collector

https://binary.mirantis.com/stacklight/helm/mcc-metric-collector-0.1.0-mcp-22.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-17.tgz

prometheus Updated

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-80.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-3.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter Updated

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-4.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-5.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-6.tgz

stacklight Updated

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-312.tgz

telegraf-ds Updated

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-16.tgz

telegraf-s Updated

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-16.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-11.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-11.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:7.4.4-20200615123606

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200320

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana Updated

mirantis.azurecr.io/stacklight/grafana:7.1.5

grafana-image-renderer New

mirantis.azurecr.io/stacklight/grafana-image-renderer:2.0.0

kubectl New

mirantis.azurecr.io/stacklight/kubectl:1.15.3

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

mcc-metric-collector Updated

mirantis.azurecr.io/stacklight/mcc-metric-collector:v0.1-20201005141816

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.19.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter Updated

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20201002144823

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter Updated

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20201006113956

prometheus-rabbitmq-exporter

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v0.29.0

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier Updated

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20200813125431

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20200219140217

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf Updated

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20200901112858

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600

5.7.0

This section outlines release notes for the Cluster release 5.7.0 that is introduced in the Mirantis Container Cloud release 2.0.0 and supports Mirantis Kubernetes Engine 3.3.3, Mirantis Container Runtime 19.03.12, and Kubernetes 1.18.

For the list of known issues, refer to the Container Cloud release 2.0.0 Known issues.

Components versions

The following table lists the components versions of the Cluster release 5.7.0.

Components versions of the Cluster release 5.7.0

Component

Application/Service

Version

Distributed storage

Ceph

14.2.9 (Nautilus)

Rook

1.3.8

Container runtime

Mirantis Container Runtime

19.03.12 0

Cluster orchestration

Mirantis Kubernetes Engine

3.3.3 1

LCM

ansible-docker

0.3.5-136-g38653c7

descheduler

0.8.0

Helm

2.16.7-38

helm-controller

0.2.0-149-g412c5a05

lcm-ansible

0.2.0-110-g63cf88b

lcm-agent

0.2.0-149-g412c5a05

metallb-controller

0.9.3-1

metrics-server

0.3.6-1

StackLight

Alerta

7.4.4-20200615123606

Alertmanager

0.21.0

Elasticsearch

7.6.1

Elasticsearch curator

5.7.6

Elasticsearch exporter

1.0.2

Fluentd

1.10.2-20200609085335

Grafana

6.6.2

IAM Proxy

6.0.1

Kibana

7.6.1

MCC Metric Collector

0.1-20200806113043

Metricbeat

7.6.1

Netchecker

1.4.1

Patroni

12-1.6p3

Prometheus

2.19.2

Prometheus Blackbox Exporter

0.14.0

Prometheus ES Exporter

0.5.1-20200313132957

Prometheus libvirt Exporter

0.1-20200610164751

Prometheus Memcached Exporter

0.5.0

Prometheus MySQL Exporter

0.11.0

Prometheus Node Exporter

1.0.1

Prometheus NGINX Exporter

0.6.0

Prometheus Patroni Exporter

0.1-20200428121305

Prometheus Postgres Exporter

0.8.0-20200715102834

Prometheus RabbitMQ Exporter

0.29.0

Prometheus Relay

0.3-20200519054052

Pushgateway

1.2.0

sf-notifier

0.3-20200430122138

sf-reporter

0.1-20200219140217

telegraf-ds

1.9.1-20200806073506

telegraf-s

1.9.1-20200806073506

Telemeter

4.4.0-20200424

0

For the Mirantis Container Runtime release highlights, see MKE, MSR, and MCR documentation: Mirantis Container Runtime release notes.

1

For the Mirantis Kubernetes Engine release highlights and components versions, see MKE, MSR, and MCR documentation: Mirantis Kubernetes Engine release notes.

Artifacts

This section lists the components artifacts of the Cluster release 5.7.0.


Ceph artifacts

Artifact

Component

Path

Helm chart

ceph-controller

https://binary.mirantis.com/ceph/helm/ceph-operator-1.0.0-mcp-16.tgz

Docker images

ceph

mirantis.azurecr.io/ceph/ceph:v14.2.9

ceph-controller

mirantis.azurecr.io/ceph/mcp/ceph-controller:v1.0.0-20200805103414

cephcsi

mirantis.azurecr.io/ceph/cephcsi/cephcsi:v2.1.2

csi-node-driver-registrar

mirantis.azurecr.io/ceph/k8scsi/csi-node-driver-registrar:v1.2.0

csi-provisioner

mirantis.azurecr.io/ceph/k8scsi/csi-provisioner:v1.4.0

csi-snapshotter

mirantis.azurecr.io/ceph/k8scsi/csi-snapshotter:v1.2.2

csi-attacher

mirantis.azurecr.io/ceph/k8scsi/csi-attacher:v2.1.0

rook

mirantis.azurecr.io/ceph/rook/ceph:v1.3.8


LCM artifacts

Artifact

Component

Path

Binaries

ansible-docker

https://binary.mirantis.com/lcm/bin/ansible-docker/v0.3.5-136-g38653c7/ansible-docker.tar.gz

lcm-ansible

https://binary.mirantis.com/lcm/bin/lcm-ansible/v0.2.0-110-g63cf88b/lcm-ansible.tar.gz

lcm-agent

https://binary.mirantis.com/lcm/bin/lcm-agent/v0.2.0-149-g412c5a05/lcm-agent

Helm charts

descheduler

https://binary.mirantis.com/core/helm/descheduler-1.10.12.tgz

metallb

https://binary.mirantis.com/core/helm/metallb-1.10.12.tgz

metrics-server

https://binary.mirantis.com/core/helm/metrics-server-1.10.12.tgz

Docker images

descheduler

mirantis.azurecr.io/lcm/descheduler/v0.8.0

helm

mirantis.azurecr.io/lcm/helm/tiller:v2.16.9-39

helm-controller

mirantis.azurecr.io/lcm/lcm-controller:v0.2.0-149-g412c5a05

metallb-controller

mirantis.azurecr.io/lcm/metallb/controller:v0.9.3-1

metallb-speaker

mirantis.azurecr.io/lcm/metallb/speaker:v0.9.3-1

metrics-server

mirantis.azurecr.io/lcm/metrics-server-amd64/v0.3.6-1


StackLight artifacts

Artifact

Component

Path

Helm charts

alerta

https://binary.mirantis.com/stacklight/helm/alerta-0.1.0-mcp-12.tgz

elasticsearch

https://binary.mirantis.com/stacklight/helm/elasticsearch-7.1.1-mcp-20.tgz

elasticsearch-curator

https://binary.mirantis.com/stacklight/helm/elasticsearch-curator-1.5.0-mcp-2.tgz

elasticsearch-exporter

https://binary.mirantis.com/stacklight/helm/elasticsearch-exporter-1.2.0-mcp-2.tgz

fluentd

https://binary.mirantis.com/stacklight/helm/fluentd-2.0.3-mcp-15.tgz

fluentd-elasticsearch

https://binary.mirantis.com/stacklight/helm/fluentd-elasticsearch-3.0.0-mcp-24.tgz

grafana

https://binary.mirantis.com/stacklight/helm/grafana-3.3.10-mcp-59.tgz

kibana

https://binary.mirantis.com/stacklight/helm/kibana-3.2.1-mcp-19.tgz

mcc-metric-collector

https://binary.mirantis.com/stacklight/helm/mcc-metric-collector-0.1.0-mcp-22.tgz

metricbeat

https://binary.mirantis.com/stacklight/helm/metricbeat-1.7.1-mcp-8.tgz

netchecker

https://binary.mirantis.com/core/helm/netchecker-1.4.1.tgz

patroni

https://binary.mirantis.com/stacklight/helm/patroni-0.15.1-mcp-17.tgz

prometheus

https://binary.mirantis.com/stacklight/helm/prometheus-8.11.4-mcp-73.tgz

prometheus-blackbox-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-blackbox-exporter-0.3.0-mcp-3.tgz

prometheus-es-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-es-exporter-1.0.0-mcp-3.tgz

prometheus-libvirt-exporter

https://binary.mirantis.com/stacklight/heprometheus-libvirt-exporter-0.1.0-mcp-2.tgz

prometheus-memcached-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-memcached-exporter-0.1.0-mcp-1.tgz

prometheus-mysql-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-mysql-exporter-0.3.2-mcp-1.tgz

prometheus-nginx-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-nginx-exporter-0.1.0-mcp-2.tgz

prometheus-rabbitmq-exporter

https://binary.mirantis.com/stacklight/helm/prometheus-rabbitmq-exporter-0.4.1-mcp-1.tgz

sf-notifier

https://binary.mirantis.com/stacklight/helm/sf-notifier-0.1.0-mcp-5.tgz

sf-reporter

https://binary.mirantis.com/stacklight/helm/sf-reporter-0.1.0-mcp-6.tgz

stacklight

https://binary.mirantis.com/stacklight/helm/stacklight-0.1.2-mcp-285.tgz

telegraf-ds

https://binary.mirantis.com/stacklight/helm/telegraf-ds-1.1.5-mcp-14.tgz

telegraf-s

https://binary.mirantis.com/stacklight/helm/telegraf-s-1.1.5-mcp-14.tgz

telemeter-server

https://binary.mirantis.com/stacklight/helm/telemeter-server-0.1.0-mcp-11.tgz

telemeter-client

https://binary.mirantis.com/stacklight/helm/telemeter-client-0.1.0-mcp-11.tgz

Docker images

alerta

mirantis.azurecr.io/stacklight/alerta-web:7.4.4-20200615123606

alertmanager

mirantis.azurecr.io/stacklight/alertmanager:v0.21.0

alpine-python3-requests

mirantis.azurecr.io/stacklight/alpine-python3-requests:latest-20200320

busybox

mirantis.azurecr.io/stacklight/busybox:1.30

configmap-reload

mirantis.azurecr.io/stacklight/configmap-reload:v0.3.0

curl

mirantis.azurecr.io/stacklight/curl:7.69.0

elasticsearch

mirantis.azurecr.io/stacklight/elasticsearch:7.6.1

elasticsearch-curator

mirantis.azurecr.io/stacklight/curator:5.7.6

elasticsearch-exporter

mirantis.azurecr.io/stacklight/elasticsearch_exporter:1.0.2

fluentd

mirantis.azurecr.io/stacklight/fluentd:1.10.2-20200609085335

gce-proxy

mirantis.azurecr.io/stacklight/gce-proxy:1.11

grafana

mirantis.azurecr.io/stacklight/grafana:6.6.2

k8s-netchecker-agent

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-agent/2019.1/

k8s-netchecker-server

mirantis.azurecr.io/lcm/kubernetes/k8s-netchecker-server/2019.1/

k8s-sidecar

mirantis.azurecr.io/stacklight/k8s-sidecar:0.1.178

kibana

mirantis.azurecr.io/stacklight/kibana:7.6.1

kube-state-metrics

mirantis.azurecr.io/stacklight/kube-state-metrics:v1.9.2

mcc-metric-collector

mirantis.azurecr.io/stacklight/mcc-metric-collector:v0.1-20200806113043

metricbeat

mirantis.azurecr.io/stacklight/metricbeat:7.6.1

node-exporter

mirantis.azurecr.io/stacklight/node-exporter:v1.0.1

origin-telemeter

mirantis.azurecr.io/stacklight/origin-telemeter:4.4.0-20200424

prometheus

mirantis.azurecr.io/stacklight/prometheus:v2.19.2

prometheus-blackbox-exporter

mirantis.azurecr.io/stacklight/blackbox-exporter:v0.14.0

prometheus-es-exporter

mirantis.azurecr.io/stacklight/prometheus-es-exporter:v0.5.1-20200313132957

prometheus-libvirt-exporter

mirantis.azurecr.io/stacklight/libvirt-exporter:v0.1-20200610164751

prometheus-memcached-exporter

mirantis.azurecr.io/stacklight/memcached-exporter:v0.5.0

prometheus-mysql-exporter

mirantis.azurecr.io/stacklight/mysqld-exporter:v0.11.0

prometheus-nginx-exporter

mirantis.azurecr.io/stacklight/nginx-prometheus-exporter:0.6.0

prometheus-patroni-exporter

mirantis.azurecr.io/stacklight/prometheus-patroni-exporter:v0.1-20200428121305

prometheus-postgres-exporter

mirantis.azurecr.io/stacklight/prometheus-postgres-exporter:v0.8.0-20200715102834

prometheus-rabbitmq-exporter

mirantis.azurecr.io/stacklight/rabbitmq-exporter:v0.29.0

prometheus-relay

mirantis.azurecr.io/stacklight/prometheus-relay:v0.3-20200519054052

pushgateway

mirantis.azurecr.io/stacklight/pushgateway:v1.2.0

sf-notifier

mirantis.azurecr.io/stacklight/sf_notifier:v0.3-20200430122138

sf-reporter

mirantis.azurecr.io/stacklight/sf_reporter:v0.1-20200219140217

spilo

mirantis.azurecr.io/stacklight/spilo:12-1.6p3

telegraf

mirantis.azurecr.io/stacklight/telegraf:v1.9.1-20200806073506

telemeter-token-auth

mirantis.azurecr.io/stacklight/telemeter-token-auth:v0.1-20200406175600